A method and system for processing spatial grid time series data

Abstract

The present application relates to the technical field of data processing method, especially to a kind of space grid time series data processing method and system, original numerical value is converted into difference value data with smaller dynamic range by time difference processing, combined with geographical position block and in-block bit width allocation, so that each difference value data occupies minimum storage space as needed, so as to obtain high compression ratio;Further, the bit packing mode of space grid point index sequence first and time sequence second is adopted, so that all time data of the same space grid point are stored continuously in the data area, which facilitates the cooperation of the start offset recorded in the block index area and the restoration parameters, so that when reading a single-point time series, only one continuous I / O operation is needed to locate and decompress all the required data, without traversing multiple-time files or whole-block decompression. In this way, while reducing storage overhead, millisecond-level single-point time series random reading performance is achieved, effectively balancing compression efficiency and reading speed.

Description

Technical Field

[0001] This invention relates to the field of data processing methods, and in particular to a method and system for processing spatial grid time-series data. Background Technology

[0002] In fields such as meteorology, environment, and marine science, it is frequently necessary to process massive amounts of spatial grid time-series data and perform high-frequency read operations in business systems, including real-time visualization, online analysis, and data service interface calls. This type of data typically contains multiple time steps (time increments), each corresponding to a two-dimensional or three-dimensional spatial grid field, such as global temperature distribution, cumulative rainfall, and ocean salinity. Traditionally, this type of data is often stored using scientific data formats such as NetCDF and HDF5, which support data compression and can reduce storage space to some extent.

[0003] However, the aforementioned storage formats have significant drawbacks when faced with the need to read single-point time series data. When extracting the complete time series of a spatial point (e.g., for curve plotting, profile analysis, or spatiotemporal interpolation), the system must traverse all time-series files or data blocks, resulting in a large number of random I / O operations and extremely low read efficiency. Especially when the number of time-series reaches thousands and the grid size reaches millions, the read latency of a single-point time series can reach tens or even hundreds of seconds, completely failing to meet the requirements of high-concurrency, low-latency interactive applications (such as weather warning platforms and online visualization systems). Furthermore, while general compression algorithms (such as GZIP and LZ4) can reduce storage space, the compressed data lacks random access capability. Reading the time series of a specific data point requires first decompressing the entire file or the complete data block containing that data point, resulting in huge memory and time overhead, and similarly failing to achieve on-demand reading.

[0004] In summary, existing technologies lack a solution that can both significantly compress data volume and support efficient random access (especially single-point time series reading). How to achieve millisecond-level single-point time series reading while maintaining a high compression ratio has become a key technical challenge for improving the real-time analysis and service capabilities of large-scale spatiotemporal data. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a method and system for processing spatial grid time series data.

[0006] In a first aspect, embodiments of the present invention provide a method for processing spatial grid time-series data, the method comprising: Acquire spatial grid data corresponding to multiple consecutive time intervals. The spatial grid data corresponding to each time interval includes multiple spatial grid points and their corresponding original values. For each spatial grid point, the original values ​​of the spatial grid point at each time point are subjected to time difference processing to obtain the difference data and the initial value of the spatial grid point; The data is divided into several continuous and non-overlapping data blocks based on the geographical location of the spatial grid points. Each data block includes the difference data of all time points of multiple spatial grid points and the starting value of each spatial grid point. For each data block, based on the numerical distribution range of all difference data within the data block, allocate the same storage bit width to each difference data within the data block; The starting values ​​of each spatial grid point and the difference data of each spatial grid point in the data block are bit-packed in the order of spatial grid point index first and then the order of time to generate a continuous bit stream corresponding to the data block. A file structure consisting of a header, a block index area, and a data area is established, and the bit stream of each data block is written sequentially into the data area. The header stores global information, and the block index area records the storage location information and restoration parameters of each data block.

[0007] In conjunction with the first aspect, the steps of performing time-difference processing on the original values ​​of each spatial grid point at each time interval to obtain the difference data and initial values ​​of the spatial grid points include: The original value of the first time step of each spatial grid point is used as the starting value, and the starting value is stored with an independent starting bit width; For each spatial grid point, calculate the difference between the first original value corresponding to the current time (excluding the first time) and the second original value corresponding to the previous time. Use the difference as the difference data for the current time to obtain multiple difference data. Each difference data is stored using the storage bit width allocated to the data block; The storage bit width of the difference data is less than or equal to the starting bit width.

[0008] In conjunction with the first aspect, the steps for dividing a spatial grid into several data blocks based on the geographical locations of the spatial grid points include: Calculate the spatial rate of change of spatial grid data; When the rate of change of space is lower than a preset threshold, a data block of the first size is used; When the rate of change of space exceeds a preset threshold, a data block of the second size is used; The first dimension is larger than the second dimension.

[0009] In conjunction with the first aspect, the step of allocating the same storage bit width to each difference data within a data block based on the numerical distribution range of all difference data within each data block includes: Traverse all spatial grid points and time-series difference data within the data block to determine the maximum and minimum values ​​of the difference data; Calculate the difference between the maximum and minimum values, and use it as the range of the data block; Calculate the required storage bit width for each difference data within the range block; The maximum value within the storage bit width is used as the missing value identifier, and the missing value identifier is stored when the difference data is missing.

[0010] In conjunction with the first aspect, the steps of generating a continuous bit stream corresponding to the data block by bit packing the starting values ​​of each spatial grid point and the difference data of each spatial grid point in the data block according to the spatial grid point index order first and the time order second include: Based on the ascending order of the spatial grid point indices, each spatial grid point is processed sequentially; For the current spatial grid point, first write its initial value using the initial bit width, and then write the time difference data of each spatial grid point into the bit stream in the order of increasing time, using the allocated storage bit width. The written bitstream is aligned to byte boundaries to form a continuous bitstream.

[0011] In conjunction with the first aspect, the method also includes: Parse the file header to obtain global information, including the number of times, data block size, and latitude and longitude range. Calculate the data block index and relative position of the target spatial grid point within the data block based on the latitude and longitude of the target spatial grid point and the size of the data block; Based on the data block index, read the metadata of the corresponding data block from the block index area. The metadata includes the starting offset, the minimum difference within the block, and the difference storage bit width. Based on the initial offset and relative position, calculate the starting bit position of all time data of the target space grid point in the data area, and read the continuous bytes starting from the starting bit position at once to obtain all the compressed data of the target space grid point; Based on the difference storage bit width and the minimum difference within the block, the original value of each time step is restored bit by bit from consecutive bytes.

[0012] In conjunction with the first aspect, the steps of restoring the original value of each time interval bit by bit from consecutive bytes based on the difference storage bit width and the minimum difference within the block include: Determine whether the current time slot is the first time slot; If so, use the starting bit width to extract the initial value as the original value for the current time step; If not, use the difference storage bit width to extract the difference data; The extracted difference data is added to the minimum difference within the block to obtain the actual difference. The actual difference is added to the value restored in the previous time step to obtain the original value in the current time step.

[0013] In conjunction with the first aspect, the method also includes: if the extracted difference data is equal to the maximum value under the difference storage bit width, then the current time interval is marked as a missing value.

[0014] Secondly, this application provides a spatial grid temporal data processing system for performing the method described above; the system includes: The acquisition module is used to acquire spatial grid data corresponding to multiple consecutive time points. The spatial grid data corresponding to each time point includes multiple spatial grid points and their corresponding original values. The difference processing module is used to perform time difference processing on the original values ​​of each spatial grid point at each time point to obtain the difference data and the initial value of the spatial grid point. The partitioning module is used to divide the spatial grid points into several continuous and non-overlapping data blocks based on their geographical locations. Each data block includes the difference data of all time points of multiple spatial grid points and the starting value of each spatial grid point. The allocation module is used to allocate the same storage bit width to each difference data in a data block based on the numerical distribution range of all difference data within the data block. The packing module is used to pack the starting values ​​of each spatial grid point and the difference data of each spatial grid point in the data block into bits according to the spatial grid point index order first and the time order second, to generate a continuous bit stream corresponding to the data block. The write module is used to establish a file structure that includes a header, a block index area, and a data area, and to write the bit stream of each data block into the data area in sequence. The header stores global information, and the block index area records the storage location information and restoration parameters of each data block.

[0015] In conjunction with the second aspect, the system also includes: The file parsing module is used to parse the file header and obtain global information, including the number of times, data block size, and latitude and longitude range. The location calculation module is used to calculate the data block index and relative position of the target spatial grid point within the data block based on the latitude and longitude of the target spatial grid point and the data block size. The metadata reading module is used to read the metadata of the corresponding data block from the block index area according to the data block index; the metadata includes the starting offset, the minimum difference within the block, and the difference storage bit width; The continuous reading module is used to calculate the starting bit position of all time data of the target space grid point in the data area according to the starting offset and relative position, and read the continuous bytes starting from the starting bit position at one time to obtain all the compressed data of the target space grid point. The numerical restoration module is used to restore the original value of each time step from consecutive bytes bit by bit based on the difference storage bit width and the minimum difference value within the block.

[0016] The embodiments of the present invention bring the following beneficial effects: This application provides a method and system for processing spatial grid time series data. The method includes: acquiring spatial grid data corresponding to multiple consecutive time periods, wherein the spatial grid data corresponding to each time period includes multiple spatial grid points and their corresponding original values; for each spatial grid point, performing time difference processing on the original values ​​of the spatial grid point at each time period to obtain the difference data and initial value of the spatial grid point; dividing the spatial grid points into several continuous and non-overlapping data blocks according to their geographical locations, each data block including the difference data of all time periods of multiple spatial grid points and their corresponding original values. The starting value of the spatial grid points; for each data block, according to the numerical distribution range of all difference data within the data block, allocate the same storage bit width to each difference data within the data block; the starting value of each spatial grid point and the difference data of each spatial grid point within the data block are bit-packed in the order of spatial grid point index first and then time order to generate a continuous bit stream corresponding to the data block; a file structure containing a header, a block index area and a data area is established, and the bit stream of each data block is written to the data area in sequence; wherein, the header stores global information, and the block index area records the storage location information and restoration parameters of each data block.

[0017] This invention transforms original numerical values ​​into difference data with a smaller dynamic range through time difference processing. Combined with geographic location-based segmentation and intra-block bit width allocation, each difference data point occupies minimal storage space as needed, achieving a high compression ratio. Furthermore, it employs a bit-packing method that prioritizes spatial grid point index order followed by temporal order, ensuring all temporal data from the same spatial grid point are stored contiguously within the data area. This facilitates the use of the starting offset and restoration parameters recorded in the block index area, allowing for the location and decompression of all required data with a single continuous I / O operation when reading a single-point time series, eliminating the need to traverse multiple time files or decompress entire blocks. Thus, while reducing storage overhead, it achieves millisecond-level random read performance for single-point time series, effectively balancing compression efficiency and read speed.

[0018] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0020] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a method for processing spatial grid temporal data according to Embodiment 1 of the present invention. Figure 2 This is a flowchart illustrating another method for processing spatial grid temporal data provided in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the structure of a spatial grid temporal data processing system provided in Embodiment 3 of the present invention; Figure 4 This is a schematic diagram of another spatial grid temporal data processing system provided in Embodiment 4 of the present invention.

[0022] Figure label: 10 - Acquisition module, 20 - Differential processing module, 30 - Partitioning module, 40 - Allocation module, 50 - Packaging module, 60 - Writing module; 100 - File parsing module, 200 - Position calculation module, 300 - Metadata reading module, 400 - Continuous reading module, 500 - Numerical restoration module. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] To facilitate understanding of this embodiment, the technical terms used in this application will be briefly introduced below.

[0025] Spatial grid point: refers to the basic unit in spatial grid data. Each grid point is uniquely determined by its geographic coordinates (longitude and latitude) and corresponds to a numerical value.

[0026] Time interval: refers to a sampling moment or step in the time dimension. Multiple consecutive time intervals constitute time series data.

[0027] Difference data and starting value: For the same spatial grid point, the original value of the first time interval is retained as the starting value, and the difference between the current time interval and the previous time interval is stored in each subsequent time interval. These differences form a difference data sequence in chronological order.

[0028] Data block: A continuous and non-overlapping rectangular sub-region that divides the entire spatial grid according to geographical location, containing the difference data and starting value of all time points of multiple spatial grid points.

[0029] Bit packing: Arranging data continuously in a specified number of bits to form a gapless bit stream.

[0030] Bit stream: A sequence of binary data obtained by packing bits together.

[0031] Block index area: A storage area in the file structure used to record restoration parameters such as the starting offset, minimum difference, and storage width of each data block.

[0032] After introducing the technical terms used in this application, the application scenarios and design concepts of the embodiments of this application will be briefly described below.

[0033] Existing formats such as NetCDF are used to store massive spatiotemporal grid data. Reading a single-point time series requires traversing all time intervals, resulting in random I / O latency of tens of seconds. General compression algorithms do not support on-demand decompression. Therefore, there is an urgent need for a storage method that balances high compression ratios with millisecond-level random access.

[0034] Based on this, embodiments of this application provide a method and system for processing spatial grid temporal data.

[0035] Example 1 This application provides a method for processing spatial grid temporal data, combined with... Figure 1 As shown, the method includes: S110, acquire spatial grid data corresponding to multiple consecutive time intervals, where the spatial grid data corresponding to each time interval includes multiple spatial grid points and their corresponding original values.

[0036] S120: For each spatial grid point, perform time difference processing on the original values ​​of the spatial grid point at each time step to obtain the difference data and the initial value of the spatial grid point.

[0037] S130, based on the geographical location of the spatial grid points, divides the data into several continuous and non-overlapping data blocks. Each data block includes the difference data of all time periods of multiple spatial grid points and the starting value of each spatial grid point.

[0038] S140: For each data block, allocate the same storage bit width to each difference data within the data block according to the numerical distribution range of all difference data within the data block.

[0039] S150, the starting values ​​of each spatial grid point and the difference data of each spatial grid point in the data block are bit-packed in the order of spatial grid point index first and then the order of time, to generate a continuous bit stream corresponding to the data block.

[0040] S160, establish a file structure including a header, a block index area, and a data area, and write the bit stream of each data block into the data area in sequence; wherein, the header stores global information, and the block index area records the storage location information and restoration parameters of each data block.

[0041] This invention transforms original numerical values ​​into difference data with a smaller dynamic range through time difference processing. Combined with geographic location-based segmentation and intra-block bit width allocation, each difference data point occupies minimal storage space as needed, thus achieving a high compression ratio. Furthermore, it employs a bit-packing method that prioritizes spatial grid point index order followed by temporal order, ensuring that all temporal data from the same spatial grid point are stored contiguously within the data area. Combined with the starting offset and restoration parameters recorded in the block index area, reading a single-point time series requires only a single continuous I / O operation to locate and decompress all necessary data, eliminating the need to traverse multiple temporal files or decompress entire blocks. Therefore, this invention achieves millisecond-level random read performance for single-point time series data while reducing storage overhead, effectively balancing compression efficiency and read speed.

[0042] The spatial grid data in step S110 originates from operational systems or numerical models in fields such as meteorology, oceanography, environment, or remote sensing, for example, global temperature forecast fields, ocean salinity analysis fields, or satellite-retrieved rainfall grid products. This type of data has two-dimensional attributes: in the time dimension, it contains multiple consecutive time steps (called "time increments"), and the interval between time increments can be a fixed value (such as 1 hour, 6 hours, or 1 day); in the spatial dimension, each time increment corresponds to a two-dimensional or three-dimensional spatial grid field.

[0043] Each time-series spatial grid data consists of multiple regularly arranged spatial grid points, each uniquely located by its geographic coordinates (longitude and latitude). Each grid point also carries a raw numerical value representing the feature value of that geographic location at that time-series, such as air temperature (in degrees Celsius or Kelvin), air pressure (in hectopascals), cumulative precipitation (in millimeters), or sea surface temperature (in degrees Celsius). These raw values ​​are typically stored as floating-point numbers, and their range and precision vary depending on the feature type (e.g., air temperature ranges from -50°C to +50°C, while cumulative radiation values ​​may exceed millions). In practice, input data can come from multiple independent files (e.g., one NetCDF file per time-series) or a multidimensional array containing all time-series data. It is understood that this invention does not limit the original storage format of the data, as long as the raw values ​​can be read by spatial grid point and time-series index. The acquired data will serve as the basis for subsequent time-difference processing.

[0044] To adapt to the precision requirements of different application scenarios, this invention allows for further scaling factor configuration after acquiring the raw data: based on preset precision requirements (e.g., retaining one decimal place is sufficient for visualization needs), the original floating-point value is multiplied by a scaling factor (e.g., 10.0) and converted to an integer, thereby reducing the dynamic range and providing better integer input for subsequent dynamic bit-width compression. The scaling factor is stored as a global parameter in the header of the final output file for restoration during decompression.

[0045] In conjunction with the first aspect, step S120, which involves performing time-difference processing on the original values ​​of each spatial grid point at each time interval to obtain the difference data and initial values ​​of the spatial grid points, includes: S121, take the original value of the first time step of each spatial grid point as the starting value, and store the starting value with an independent starting bit width.

[0046] S122, For each spatial grid point, calculate the difference between the first original value corresponding to the current time (excluding the first time) and the second original value corresponding to the previous time, and use the difference as the difference data for the current time to obtain multiple difference data.

[0047] S123, each difference data is stored using the storage bit width allocated to the data block.

[0048] The storage bit width of the difference data is less than or equal to the starting bit width.

[0049] After completing the data acquisition in S110, step S120 performs time difference compression on the original numerical sequence of each spatial grid point. Considering that the element values ​​in meteorology, oceanography and other fields change slowly over time (for example, the temperature variation between adjacent time intervals is usually only 0~2K, much smaller than its absolute value of 300K), the original large numerical range can be transformed into small difference data with a significantly reduced dynamic range through difference processing.

[0050] In step S121, for each spatial grid point, the original value at the first time step (t=1) is not differentially analyzed and is directly retained as the starting value. Since the starting value reflects the absolute magnitude of the spatial grid point (which may be large, such as a temperature of 300K or a sea-level pressure of 1013 hPa), a sufficiently large bit width is required to ensure no loss of accuracy. Therefore, this invention assigns an independent starting bit width to the starting value. This bit width is a preset fixed value (e.g., 16 bits or 18 bits) that can cover the possible numerical range of the element. It is understood that the size of the starting bit width can be flexibly configured according to the data characteristics and is usually greater than or equal to the bit width required for subsequent differential data.

[0051] For subsequent time intervals (t≥2) at the same spatial grid point, calculate the difference between adjacent time intervals one by one. Specifically, for the current time interval i (i≥2), take its original value (denoted as V). i ) and the original value of the previous time step (i-1) (denoted as V) i-1 ), calculate the difference ΔV i =V i - V i-1 The difference ΔV i This refers to the difference data at the current time point. For a dataset with T time points, each spatial grid point will generate (T-1) difference data points, which, together with the initial value, constitute the complete compressed representation of that grid point. Because the changes between adjacent time points are slow, the absolute value of the difference is much smaller than the absolute value of the original value (for example, the temperature difference is usually between -5K and +5K), significantly reducing the dynamic range of the value.

[0052] This invention dynamically calculates the required storage bit width based on the statistical distribution (range) of all difference data within a data block, using data blocks as the unit. Because the dynamic range of the differences is significantly reduced, the required bit width is typically much smaller than the initial bit width (e.g., if the initial bit width is 16 bits, the difference bit width may only require 4). (9 bits). This allocation result will be used for write operations during bit packing. It should be noted that in the actual implementation, all difference data within the same data block will share the same bit width (determined by the intra-block range) to ensure the regularity of the storage structure and the simplicity of the index. The storage bit width of the difference data is strictly less than or equal to the initial bit width, thereby achieving asymmetric processing of the data characteristic of large initial values ​​and small changes, which avoids the precision loss of the initial value and compresses the storage space of the difference data to the greatest extent.

[0053] In this way, the original numerical sequence of each spatial grid point in step S120 is converted into a starting value and a set of difference data. An asymmetric differential compression method is used, with the starting value stored independently using a large fixed bit width, and the difference data stored using a small bit width based on dynamic range calculation.

[0054] In conjunction with the first aspect, step S130, which involves dividing the spatial grid into several data blocks based on the geographical locations of the spatial grid points, includes: S131, calculates the spatial rate of change of spatial grid data.

[0055] S132, when the spatial change rate is lower than the preset threshold, a data block of the first size is used.

[0056] S133, when the spatial change rate is higher than the preset threshold, a data block of the second size is used.

[0057] The first dimension is larger than the second dimension.

[0058] After completing the time difference processing in step S120, step S130 divides the entire spatial grid into several rectangular data blocks according to geographical location. Each data block serves as the basic unit for subsequent dynamic bit width allocation and bit packing. The purpose of this division is to utilize spatial locality: the feature values ​​of adjacent geographical locations are usually continuous and similar. This results in the difference data of all spatial grid points within the same data block having similar numerical ranges, thus allowing the use of a uniform storage bit width, ensuring compression efficiency while simplifying the index structure.

[0059] To adapt to the spatial variation characteristics of different elements, this invention allows for dynamic configuration of data block size, which can be determined based on the spatial variation rate in specific implementations. The spatial variation rate characterizes the degree of fluctuation of element values ​​in geographic space, and is measured by calculating the statistical characteristics of the numerical differences between adjacent spatial grid points within the entire grid field. For example, for a temperature field, the mean or variance of the absolute values ​​of the first-order differences of all adjacent meridional and zonal grid points can be calculated; for a precipitation field, due to its discontinuous spatial distribution, its variation rate is usually higher. A higher spatial variation rate indicates more drastic changes in the element over short distances; a lower spatial variation rate indicates a more gradual distribution of the element. This calculation can be based on the original numerical values ​​or data after time difference.

[0060] Based on the aforementioned spatial change rate, when the spatial change rate is below a preset threshold, it indicates that the feature values ​​are spatially continuous and change gradually. In this case, a larger data block (referred to as the first size, such as 64×64 or 128×128 grid points) is used. A larger data block can accommodate more spatial grid points, thereby reducing the total number of data blocks and lowering the storage overhead of the index area. Simultaneously, because the numerical range within the block remains concentrated, the dynamic bit width allocation remains effective. Conversely, when the spatial change rate is above the preset threshold, it indicates that the feature values ​​change drastically in space. In this case, a smaller data block (referred to as the second size, such as 8×8 or 16×16 grid points) is used. A smaller data block ensures that the numerical range of the data within the block is relatively concentrated, avoiding excessively large storage bit widths for the entire block due to extreme values ​​widening the range, thus maintaining high compression efficiency. The first size is larger than the second size.

[0061] The adaptive partitioning strategy described above can adapt to the spatial characteristics of different feature types: large blocks are used for smooth features to reduce management overhead, while small blocks are used for drastic features to maintain compression efficiency, thus achieving a balance between compression ratio and random access performance. The determined data block size is stored as a global parameter in the file header for subsequent parsing.

[0062] In conjunction with the first aspect, step S140 includes: S141: Traverse all spatial grid points and time-based difference data within the data block to determine the maximum and minimum values ​​of the difference data.

[0063] S142, calculate the difference between the maximum and minimum values, and use it as the range of the data block.

[0064] S143, calculate the storage bit width required for each difference data in the data block based on the range.

[0065] S144, use the maximum value under the storage bit width as the missing value identifier, and store the missing value identifier when the difference data is missing.

[0066] After spatial partitioning is completed in step S130, each data block contains the difference data for all time intervals of multiple spatial grid points. Since the value range of the difference data within the same data block is usually relatively concentrated, step S140 performs statistical analysis on a block-by-block basis, uniformly allocating a minimum storage bit width to all the difference data within that block, thereby achieving on-demand compression. Specifically: Iterate through all spatial grid points contained in the current data block and the difference data for each spatial grid point across all time intervals (from the second time interval onwards). During the iteration, compare and record all the difference data, and then filter to obtain the maximum value among all the difference data (denoted as ). ) and minimum value (denoted as The difference data can be positive, negative, or zero (depending on the trend of the feature), and this extreme value reflects the overall limit of data fluctuation within the data block.

[0067] Calculate the maximum value (denoted as ). ) and minimum value (denoted as The difference between the two values ​​is used to obtain the range corresponding to the data block. The range represents the overall width of the difference data distribution within the data block. The smaller the range, the more concentrated the data is within the data block, and the smaller the required storage width; conversely, the larger the range, the larger the required storage width.

[0068] In order to store all the difference data within the data block without loss of quality, the required number of bits should be able to cover the values ​​from... arrive The entire integer range. Considering that the difference data may take any integer value within the range, the storage bit width is calculated using the formula:

[0069] in, The minimum storage bit width uniformly used for each difference data within a data block. For rounding up, It is extremely poor.

[0070] As an example, if the range Then the required storage bit width is 4; if the range If the value is 500, then the required storage width is 9 bits.

[0071] As can be seen, the storage bit width is much smaller than the starting bit width used to store the starting value (usually 16 bits or more), thus achieving efficient compression.

[0072] In a dynamic bit-width storage framework, the maximum value that each bit width can represent (i.e., the all-1 mask) is... This is reserved as a missing value identifier. When difference data at a certain time point is missing due to missing original observations or quality control rejections, the actual difference value is not stored; instead, the missing value identifier is stored directly. This is because the missing value identifier uses the maximum value within the current bit width. And that maximum value This is impossible in normal difference data (because the range of normal difference values ​​is...). ~ , - ≤range, the actual difference plus the offset, the maximum value is range, while the value corresponding to the all-ones mask is (This value is strictly greater than the range), so it can be safely used to mark missing values ​​without requiring additional storage space or adding a marker field.

[0073] Through S140, three key parameters are obtained for each data block: minimum difference ( ), storage bit width ( The parameters include the implicit storage range calculated from the range. These parameters will be stored in the file block index area as necessary restoration parameters for restoring the original difference during decompression. The inclusion of missing value identifiers naturally supports the processing of missing test data while maintaining a high compression ratio.

[0074] In conjunction with the first aspect, step S150 includes: S151, based on the ascending order of the spatial grid point indices, process each spatial grid point sequentially.

[0075] S152, for the current spatial grid point, first write its initial value using the initial bit width, and then write the time difference data of each spatial grid point into the bit stream in the order of increasing time, using the allocated storage bit width.

[0076] S153 aligns the written bit stream according to byte boundaries to form a continuous bit stream.

[0077] After dynamic bit width allocation is completed in step S140, each data block has a determined independent storage bit width for the initial value and a uniform storage bit width for the difference data. Step S150 tightly packs these data at the bit level in a specific order to form a continuous bit stream for writing to the file data area.

[0078] Since the spatial grid points within the data block have a defined index order, this order is determined by the row and column layout of the spatial grid (e.g., a linear index in row-major or column-major order). Step S151 traverses each spatial grid point within the data block in ascending order of index. For the current spatial grid point to be processed, its initial value and subsequent time-series difference data are extracted as the data unit to be packaged.

[0079] For the currently selected spatial grid point, its initial value is first bit-packed according to a pre-set initial bit width (usually 16-18 bits) and written to the current position of the bit stream. The initial bit width is fixed and relatively large, which can preserve the original value of the first time step without loss. Subsequently, in ascending order of time steps (from the second time step to the last time step), each difference data of the spatial grid point is retrieved sequentially, using the uniform storage bit width allocated for this data block in S140 (…). The difference data is packed into packets and written sequentially to the bit stream. Since the bit width of the difference data is much smaller than the starting bit width, and all difference data are tightly packed without extra gaps, storage space is significantly saved. Adjacent difference data are stored consecutively in the bit stream, further facilitating sequential reading during decompression.

[0080] During the bit packing process, data is written bit by bit, but ultimately needs to be stored in the file system in bytes. Therefore, after processing all the data at the current spatial grid point, or after packing the entire data block, the accumulated bit stream is aligned to the byte boundary in groups of 8 bits. If the remaining bits are less than 8 bits, they are padded with 0s at the end (or padded according to preset rules) to form a complete byte sequence. The aligned bit stream no longer has gaps, achieving maximum storage density. The continuous bit stream corresponding to each data block is the final storage form of that block in the file data area.

[0081] In step S150, all starting values ​​and difference data within each data block are organized into a compact, spatial grid-point-first bitstream. The characteristic of continuous storage of all time data at the same spatial grid point enables millisecond-level random access to single-point time series data.

[0082] Example 2 This application provides another method for processing spatial grid temporal data, combined with... Figure 2 As shown, the method also includes: S210, parse the file header to obtain global information; the global information includes the number of times, data block size, and latitude and longitude range.

[0083] S220: Calculate the data block index and relative position within the data block where the target spatial grid point is located, based on the latitude and longitude of the target spatial grid point and the size of the data block.

[0084] S230: Based on the data block index, read the metadata of the corresponding data block from the block index area. The metadata includes the starting offset, the minimum difference within the block, and the difference storage width.

[0085] S240: Based on the initial offset and relative position, calculate the starting bit position of all time data of the target space grid point in the data area, and read the continuous bytes starting from the starting bit position at once to obtain all compressed data of the target space grid point.

[0086] S250, based on the difference storage bit width and the minimum difference within the block, restores the original value of each time step bit by bit from consecutive bytes.

[0087] When it is necessary to read the complete time series of a spatial grid point, first open the compressed file generated using the method provided in Embodiment 1 of this invention. Parse the file header to extract global parameters, including: total number of times (used to determine the amount of data to be read for each spatial grid point), data block size (the number of spatial grid points contained in each data block in the row and column directions, used to locate the data block to which the target point belongs), and latitude and longitude range and interval (used to convert geographic coordinates into a grid index). In addition, the header may also contain global configurations such as scaling factors and starting bit width.

[0088] Using the latitude and longitude range and interval obtained from the header, the latitude and longitude coordinates of the target spatial grid points are converted into row and column numbers in the grid coordinate system. Then, based on the data block size (e.g., each block contains M rows × N columns of grid points), the data block number (block index) to which the target spatial grid point belongs and its relative offset within that block (e.g., row and column number within the block) are calculated. For example, if the global index of the target spatial grid point is... ), block size is ( If ), then the block index is ( , The relative positions within the block are ( , This step maps geographic coordinates to the file's internal storage location.

[0089] The file block index area stores the metadata of each block in the order of data blocks. Based on the block index calculated by S220, the corresponding entry in the index area is directly located, and three key parameters of the data block are read: the starting offset (the starting byte position of the data area of ​​the block, used to locate the data), the minimum difference within the block (used to reconstruct the difference data), and the difference storage bit width (used to parse the difference bit by bit). This metadata is written during compressed storage and can be obtained without decompressing the entire data area during reading.

[0090] Because the compressed storage uses a bit-packing method that prioritizes spatial grid point index order over time order, all time data (starting value + time differences) for the same spatial grid point are stored contiguously within the data area. Based on the relative position within the block obtained in S220 (i.e., the linear index of the target point among all grid points within the block), combined with the starting offset, difference storage bit width, and global starting bit width and time count obtained in S230, the starting bit of the target point data can be accurately calculated. The specific calculation formula is as follows: Start bit = Block start offset × 8 + (Linear index of grid point in block) × (Start bit width + (Number of times - 1) × Difference storage bit width).

[0091] Then, the total number of bits occupied by the data at that point is calculated (starting bit width + (number of times - 1) × difference storage bit width), and rounded up to obtain the required number of bytes. Subsequently, the corresponding byte position in the file is located, and several consecutive bytes are read at once. Since the read operation is continuous, the operating system's page cache and CPU cache can be fully utilized, greatly improving I / O efficiency.

[0092] The read contiguous bytes are treated as a bitstream, and data for each time interval is extracted bit by bit. First, the starting value of the first time interval is extracted using the initial bit width; this value is the original value of the first time interval. Subsequently, for each subsequent time interval, the difference data is extracted sequentially according to the difference storage bit width. If the extracted difference data is equal to the maximum value under that bit width (i.e., a mask of all 1s), then the time interval is determined to be a missing value and marked as missing; otherwise, the minimum difference value within the block is added to the difference data to obtain the actual difference value, and then the value restored from the previous time interval is added to recover the original value of the current time interval. After all time intervals have been restored, if a scaling factor is stored in the header, the obtained integer value is divided by the scaling factor to obtain the final floating-point original value. At this point, the complete time series reading of the target spatial grid points is completed.

[0093] Through steps S210 to S250, the present invention utilizes the continuous storage characteristics of spatial grid points in the data area and the precise positioning information of the block index area to achieve one-time continuous reading and bit-by-bit decompression of a single-point time series, completely avoiding the operation of traversing all time files or decompressing the entire data block.

[0094] The actual test was conducted on a dataset with 1000×1000 spatial grid points and 1000 time intervals, and the test results are shown in Table 1.

[0095] Table 1. Results Summary Table

[0096] The results show that reading a single point time series takes only 0.032 seconds, which is about 250 times faster than the traditional NetCDF scheme (8 seconds); reading a complete time series of 100×100 points (10,000 points) takes about 0.8 seconds, which is about 100 times faster than the NetCDF scheme (80 seconds), providing key technical support for real-time visualization and online analysis of large-scale spatiotemporal data.

[0097] In conjunction with the first aspect, step S250 includes: S251, determine whether the current time slot is the first time slot; If yes, execute S252; otherwise, execute S253~S255.

[0098] S252 uses the starting bit width to extract the initial value as the original value of the current time step.

[0099] S253 uses the difference storage bit width to extract the difference data.

[0100] S254 adds the minimum difference within the block to the extracted difference data to obtain the actual difference.

[0101] S255 adds the actual difference to the value restored in the previous time step to obtain the original value in the current time step.

[0102] After reading all the compressed data (continuous bytes) of the target space grid points at once via S240, this step parses and restores the original value of each time step bit by bit. Starting from the beginning of the bit stream, the data is processed sequentially in ascending order of time steps. First, it is determined whether the current time step to be parsed is the first time step (i.e., the 1st time step).

[0103] If the current time step is the first time step, the integer value corresponding to the preset starting bit width (such as 16 bits or 18 bits) is directly read from the bit stream. This value is the original value of the first time step (if a scaling factor was applied before compression, this value is the scaled integer, which needs to be restored uniformly in subsequent steps). The starting value does not need to be added or subtracted and is directly used as the final result of this time step.

[0104] If the current time step is not the first time step (i.e., the second time step and later), then an integer value is sequentially read from the bitstream according to the uniform difference storage bit width of this data block (calculated by S140, usually between 4 and 9 bits). This value is the difference data after the minimum value offset. During compressed storage, in order to map the difference data to a non-negative interval for storage with a smaller bit width, each difference data is subtracted from the minimum difference value within the block (denoted as ) before being written. Therefore, during decompression, the extracted values ​​(denoted as...) need to be... Adding this minimum value restores the original actual difference:

[0105] Among them, the actual difference This represents the difference between the original value of the current time period and the original value of the previous time period.

[0106] Therefore, the actual difference Add the original values ​​that were restored in the previous moment This will restore the original value for the current time period: .

[0107] Repeat steps S253 to S255 until all time difference data has been processed.

[0108] Through the above steps, this invention can efficiently and accurately reconstruct each value of the complete time series of the target spatial grid points. The entire decompression process involves only integer addition, comparison, and bit operations, without the need for complex floating-point transformations or dictionary lookups, resulting in extremely low computational overhead. In a typical test environment (1000×1000 grid points, 1000 hours of temperature data), the total time for reading a single-point time series using the method of this invention (including S210 to S250) is approximately 0.032 seconds, with the bit extraction and reconstruction steps accounting for a very small percentage of the time, fully verifying the computational efficiency advantage of this invention.

[0109] In conjunction with the first aspect, the method also includes: S260, if the extracted difference data is equal to the maximum value under the difference storage bit width, then mark the current time as a missing value.

[0110] During the restoration process in step S250, for any time interval other than the first (i.e., the second time interval and thereafter), after extracting the difference data from the bitstream according to the difference storage bit width, it is first determined whether the value is equal to the maximum value that the current bit width can represent (i.e., the all-1 mask). Because the maximum value is reserved as a missing value identifier during compressed storage (see S144), the normal difference data, after being offset by the minimum value, ranges from 0 to the range, and the range is strictly less than 0. Therefore, the maximum value will not appear in the normal data. If the extracted difference data is equal to the mask value, it indicates that the original observation data for the current time period is missing (e.g., instrument failure, cloud cover causing satellite inversion failure, etc.). In this case, the addition restoration operation is not performed, but the time period is directly marked as a missing value.

[0111] In practical implementation, a unified missing data identifier (e.g., -999, NaN, or null) can be output according to the application scenario. For visualization scenarios requiring continuous numerical curves, the front end can interpolate or skip the missing point; for statistical analysis scenarios, missing data can be directly filtered. This method achieves lossless coexistence of missing values ​​and normal data without the need for additional storage of marker bits or increased bit width, thus maintaining compression efficiency.

[0112] For example, in meteorological observations, a spatial grid point may experience a missing data point at time 5 due to sensor malfunction. During compression, the difference data at this time point is directly stored as a mask of all 1s for the current bit width (e.g., 15 for a 4-bit width). During decompression, when this mask value is encountered, the restored value from the previous time point is not added to the result; instead, the result for that time point is marked as missing (e.g., -999). This time point is not included in the accumulation of subsequent time points (i.e., the difference for the next time point is still calculated based on the valid values ​​from the previous time point, and the location of the missing value does not affect the correctness of subsequent restorations). This approach ensures data integrity while avoiding error propagation due to missing values. Missing value handling is achieved at zero cost within the dynamic bit-width compression framework, further enhancing the method's practicality and robustness.

[0113] Example 3 Thirdly, this application provides a spatial grid temporal data processing system for performing the method as provided in Embodiment 1. Combined with Figure 3 As shown, the system includes: an acquisition module 10, a differential processing module 20, a partitioning module 30, an allocation module 40, a packaging module 50, and a writing module 60.

[0114] The acquisition module 10 is used to acquire spatial grid data corresponding to multiple consecutive time points. The spatial grid data corresponding to each time point includes multiple spatial grid points and their corresponding original values. The difference processing module 20 is used to perform time difference processing on the original values ​​of the spatial grid point at each time for each spatial grid point, so as to obtain the difference data and the starting value of the spatial grid point. The partitioning module 30 is used to divide the spatial grid points into several continuous and non-overlapping data blocks according to their geographical locations. Each data block includes the difference data of all time periods of multiple spatial grid points and the starting value of each spatial grid point. The allocation module 40 is used to allocate the same storage bit width to each difference data in the data block according to the numerical distribution range of all difference data in the data block; Packaging module 50 is used to pack the starting values ​​of each spatial grid point in the data block and the difference data of each spatial grid point in a manner that first spatial grid point index order and then time order to generate a continuous bit stream corresponding to the data block. The writing module 60 is used to establish a file structure including a header, a block index area, and a data area, and to write the bit stream of each data block sequentially into the data area; wherein, the header stores global information, and the block index area records the storage location information and restoration parameters of each data block.

[0115] Through the synergistic action of the aforementioned modules, this system can progressively compress massive amounts of spatial grid time-series data by performing temporal differencing, spatial partitioning, dynamic bit width allocation, and spatial point priority packing, generating a structured file containing a header, block index area, and data area; thus significantly reducing the data volume. Tests on typical meteorological data (such as temperature and cumulative radiation) show that the compressed file size is reduced by approximately 20% to 30% compared to the original NetCDF file, and by approximately 10% to 15% compared to a NetCDF file compressed with GZIP. Moreover, the compression effect becomes more significant as the data volume increases (e.g., cumulative radiation values ​​exceeding one million).

[0116] Example 4 This application also provides another system for processing spatial grid temporal data, combined with Figure 4 As shown, the system also includes: a file parsing module 100, a location calculation module 200, a metadata reading module 300, a continuous reading module 400, and a numerical restoration module 500.

[0117] The file parsing module 100 is used to parse the file header and obtain global information, including the number of times, data block size, and latitude and longitude range.

[0118] The location calculation module 200 is used to calculate the data block index and relative position within the data block where the target spatial grid point is located, based on the latitude and longitude of the target spatial grid point and the size of the data block.

[0119] The metadata reading module 300 is used to read the metadata of the corresponding data block from the block index area according to the data block index; the metadata includes the starting offset, the minimum difference within the block, and the difference storage bit width.

[0120] The continuous reading module 400 is used to calculate the starting bit of all time data of the target space grid point in the data area according to the starting offset and relative position, and read the continuous bytes starting from the starting bit at one time to obtain all the compressed data of the target space grid point.

[0121] The numerical restoration module 500 is used to restore the original value of each time step from consecutive bytes bit by bit based on the difference storage bit width and the minimum difference value within the block.

[0122] This system, through the coordinated operation of the aforementioned modules, utilizes the spatial grid-point-first contiguous storage structure and precise positioning information in the block index area of ​​the compressed file to achieve continuous one-time reading and bit-by-bit decompression of single-point time series. In a typical test environment (1000×1000 grid points, 1000 hours of temperature data), the system takes only 0.032 seconds to read a complete time series of a single point, approximately 250 times faster than the traditional NetCDF solution (8 seconds); reading a complete time series of 100×100 range points (10000 points) takes only 0.8 seconds, approximately 100 times faster than the NetCDF solution (80 seconds). Furthermore, independent storage of each data block and independent indexing of metadata allow multiple threads and processes to read different data blocks in parallel without lock contention, naturally supporting high-concurrency access.

[0123] To further improve read performance in high-concurrency scenarios, this system also adopts file channel caching and header information caching mechanisms: the file channel cache manages the opened file channels based on the least recently used strategy to avoid the same file being opened and closed frequently; the header information cache caches the global parameters in the file header (such as the number of times, data block size, latitude and longitude range) with the file path as the key, thereby avoiding repeated parsing.

[0124] Furthermore, for datasets containing multiple meteorological elements, the system supports parallel processing of multiple elements: during compression, an independent thread is allocated to each element to execute the compression and storage method provided in Example 1; during reading, a thread pool is used to concurrently call the fast reading method provided in Example 2 to acquire time-series data of multiple elements at the same spatial grid point in parallel. These optimizations enable the system to maintain millisecond-level response speeds even under a large number of concurrent requests, further improving overall throughput. The entire decompression process involves only integer shifting and addition operations, with extremely low computational overhead, thus providing key technical support for real-time visualization, online analysis, and early warning of large-scale spatiotemporal data with millisecond-level response speeds.

[0125] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0126] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0127] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0129] Finally, it should be noted that the above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for processing spatial grid time-series data, characterized in that, The method includes: The system acquires spatial grid data corresponding to multiple consecutive time intervals. Each time interval's spatial grid data includes multiple spatial grid points and their corresponding raw values. The spatial grid data originates from operational systems or numerical models in meteorology, oceanography, environment, or remote sensing fields, and possesses two-dimensional temporal and spatial attributes: in the temporal dimension, it contains multiple consecutive time steps, referred to as time intervals; in the spatial dimension, each time interval corresponds to a two-dimensional or three-dimensional spatial grid field; the spatial grid data for each time interval consists of multiple regularly arranged spatial grid points, each spatial grid point being uniquely located by its geographic coordinates; the geographic coordinates include longitude and latitude, and each spatial grid point carries a raw value representing the element value of the geographic location at that time interval, including air temperature, air pressure, cumulative precipitation, or sea surface temperature. For each spatial grid point, time difference processing is performed on the original values ​​of the spatial grid point at each time step to obtain the difference data and starting value of the spatial grid point. This includes: taking the original value of the first time step of each spatial grid point as the starting value and storing the starting value with an independent starting bit width; for each spatial grid point, calculating the difference between the first original value corresponding to the current time step (excluding the first time step) and the second original value corresponding to the previous time step, and using the difference as the difference data of the current time step to obtain multiple difference data; each difference data is stored using the storage bit width allocated by the data block; wherein, the storage bit width of the difference data is less than or equal to the starting bit width; Based on the geographical locations of the spatial grid points, the spatial grid is divided into several continuous and non-overlapping data blocks. Each data block includes the difference data of all time periods of multiple spatial grid points and the initial value of each spatial grid point. The step of dividing the spatial grid into data blocks based on the geographical locations of the spatial grid points includes: calculating the spatial change rate of the spatial grid data; when the spatial change rate is lower than a preset threshold, using data blocks of a first size; when the spatial change rate is higher than the preset threshold, using data blocks of a second size; wherein, the first size is larger than the second size. For each data block, based on the numerical distribution range of all difference data within the data block, the same storage bit width is allocated to each difference data within the data block; The starting values ​​of each spatial grid point within the data block and the difference data between each spatial grid point are bit-packed according to the spatial grid point index order first, followed by the time order, to generate a continuous bit stream corresponding to the data block. This includes: processing each spatial grid point sequentially based on the ascending order of the spatial grid point indices; for the current spatial grid point, first writing its starting value using the starting bit width, and then writing the difference data of each time of the spatial grid point into the bit stream sequentially using the allocated storage bit width according to the ascending time order; aligning the written bit stream according to byte boundaries to form a continuous bit stream. A file structure comprising a header, a block index area, and a data area is established, and the bit streams of each data block are sequentially written into the data area; wherein, the header stores global information, the block index area records the storage location information and restoration parameters of each data block, and the restoration parameters include the starting offset, minimum difference within the block, and difference storage bit width of each data block.

2. The method according to claim 1, characterized in that, The step of allocating the same storage bit width to each difference data within a data block based on the numerical distribution range of all difference data within each data block includes: Traverse all the spatial grid points and the difference data at all times within the data block to determine the maximum and minimum values ​​of the difference data; The difference between the maximum value and the minimum value is calculated as the range of the data block; Calculate the required storage bit width for each difference data within the data block based on the range; The maximum value under the specified storage width is used as the missing value identifier, and the missing value identifier is stored when the difference data is missing.

3. The method according to claim 1, characterized in that, The method further includes: Parse the file header to obtain global information; wherein, the global information includes the number of times, data block size, and latitude and longitude range; Based on the latitude and longitude of the target spatial grid point and the size of the data block, calculate the data block index where the target spatial grid point is located and its relative position within the data block; Based on the data block index, read the metadata of the corresponding data block from the block index area. The metadata includes the starting offset, the minimum difference within the block, and the difference storage bit width. Based on the starting offset and the relative position, calculate the starting bit position of all time data of the target spatial grid point in the data area, and read the continuous bytes starting from the starting bit position at once to obtain all compressed data of the target spatial grid point; Based on the difference storage bit width and the minimum difference value within the block, the original value of each time step is restored bit by bit from the consecutive bytes.

4. The method according to claim 3, characterized in that, The step of restoring the original value of each time interval bit by bit from the consecutive bytes based on the difference storage width and the minimum difference value within the block includes: Determine whether the current time slot is the first time slot; If so, use the starting bit width to extract the initial value as the original value for the current time step; If not, use the difference storage bit width to extract the difference data; The extracted difference data is added to the minimum difference within the block to obtain the actual difference. The actual difference is added to the value restored in the previous time interval to obtain the original value in the current time interval.

5. The method according to claim 4, characterized in that, The method further includes: if the extracted difference data is equal to the maximum value under the difference storage bit width, then the current time is marked as a missing value.

6. A system for processing spatial grid time-series data, characterized in that, The system is used to perform the method as described in any one of claims 1-2; the system comprises: The acquisition module is used to acquire spatial grid data corresponding to multiple consecutive time points. The spatial grid data corresponding to each time point includes multiple spatial grid points and their corresponding original values. The difference processing module is used to perform time difference processing on the original values ​​of the spatial grid point at each time for each spatial grid point, so as to obtain the difference data and the starting value of the spatial grid point. The partitioning module is used to divide the spatial grid points into several continuous and non-overlapping data blocks according to their geographical locations. Each data block includes the difference data of all time periods of multiple spatial grid points and the starting value of each spatial grid point. The allocation module is used to allocate the same storage bit width to each difference data in the data block according to the numerical distribution range of all difference data within the data block; The packing module is used to pack the starting values ​​of each spatial grid point in the data block and the difference data of each spatial grid point in a bit packing manner, first in the spatial grid point index order and then in the time order, to generate a continuous bit stream corresponding to the data block. The writing module is used to establish a file structure including a header, a block index area, and a data area, and to write the bit stream of each data block sequentially into the data area; wherein, the header stores global information, and the block index area records the storage location information and restoration parameters of each data block.

7. The system according to claim 6, characterized in that, The system also includes: The file parsing module is used to parse the file header and obtain global information, including time count, data block size, and latitude and longitude range. The location calculation module is used to calculate the data block index and relative position within the data block where the target spatial grid point is located, based on the latitude and longitude of the target spatial grid point and the size of the data block. The metadata reading module is used to read the metadata of the corresponding data block from the block index area according to the data block index; the metadata includes the starting offset, the minimum difference within the block, and the difference storage bit width; The continuous reading module is used to calculate the starting bit position of all time data of the target spatial grid point in the data area according to the starting offset and the relative position, and read the continuous bytes starting from the starting bit position at one time to obtain all the compressed data of the target spatial grid point. The numerical restoration module is used to restore the original value of each time interval bit by bit from the continuous bytes based on the difference storage bit width and the minimum difference value within the block.

Images