A storage-computing co-sourcing method and system for multi-band radar reflectivity products

Abstract

The application discloses a storage and calculation homologous method and system for multi-band radar reflectivity products. After receiving the original file, the head file metadata and entity data block are parsed to form an observation timestamp and spatial range combination as a block-level positioning key. Threshold processing, non-0 value proportion calculation and matrix scale determination are performed on each entity data block within the corresponding block-level positioning key limit range, standard row and column matrix encoding or CSR sparse matrix encoding is selected, and compression processing is performed when the conditions are met, matrix form identification and compression identification are generated and written into the head file metadata. When a request comes, the hit entity data block set is determined according to the space-time index and the head file metadata, the decoding path and the decompression path are determined according to the corresponding identification before the hit entity data block is read, and only the hit entity data block is executed in situ calculation and the result is returned. The application can improve the storage efficiency and request response efficiency of the radar reflectivity product.

Description

Technical Field

[0001] This invention relates to the field of meteorological radar data processing and database expansion technology, specifically to a storage and computing method and system for multi-band radar reflectivity products. Background Technology

[0002] The new generation of weather radar observation networks acquires multi-elevation angle scanning data in real time. After quality control processing such as electromagnetic interference filtering and ground clutter suppression, combined reflectivity products corresponding to different band weather radars are generated. The reflectivity factor in the radar combined reflectivity product is used to describe the intensity and distribution of precipitation particles. Its data has natural sparsity: precipitation clouds usually only occupy a small part of the scanning space, with most areas having zero or invalid values. At the same time, in operational use, areas below the reflectivity threshold can be considered as having no significant precipitation, thereby further improving data sparsity.

[0003] However, during periods of heavy rainfall, such as typhoons, large areas of non-zero values ​​may appear. In such cases, uniformly using sparse matrix storage not only fails to save space but may actually increase storage and computational load. Among existing comparative solutions, one type primarily focuses on constructing a storage-computation co-processing scheme for dense data, resulting in storage waste for data with a low proportion of non-zero values; another type uses a ternary sparse matrix to store radar data but does not consider scenarios with large areas of non-zero values ​​and also has limitations in matrix operation efficiency.

[0004] Another common processing method in existing technologies is to first store the original radar file as a whole, and then have the user or business program download it to a local node for decoding, querying, calculation, trimming, rendering, or analysis. Alternatively, a unified logical object covering the entire original file can be constructed in the database before accessing it. This processing method usually requires full reading, unified decoding, unified expansion, and data migration, which leads to problems such as frequent file I / O, irrelevant data being involved in processing, and excessively long request-response paths.

[0005] Therefore, there is an urgent need for a storage-computation homogeneous method and system that can improve storage efficiency and request response efficiency without relying on the construction and decoding of all unified objects, given the mixed distribution characteristics of radar reflectivity products with local sparse and local non-sparse features. Summary of the Invention

[0006] The purpose of this invention is to provide a storage and computation method and system for multi-band radar reflectivity products, in order to solve the problems in the prior art, such as large full-scale construction overhead caused by unified object-oriented processing, difficulty in taking into account mixed distribution scenarios of local sparse and local non-sparse storage, reduced processing efficiency caused by unified decoding and unified relocation, and difficulty in meeting the timeliness requirements of short-term monitoring and disaster early warning scenarios by off-database computation.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: It receives and parses the original files of multi-band radar reflectivity products to form initial header metadata and entity data blocks. Within the parsing results of the same original file, the combination of observation timestamps and spatial ranges in the header metadata is used as the block-level positioning key for each entity data block, ensuring that each entity data block is uniquely associated with its corresponding header metadata. For each entity data block, only the local observation values ​​of the entity data block bound to its corresponding block-level positioning key are used as decision inputs. The observation timestamps and spatial ranges defined by the block-level positioning key are used as the calculation range for the proportion of non-zero values, the matrix size determination range, and the compression range. Thresholding processing, non-zero value proportion calculation, and matrix size determination are performed on the entity data block. Based on the proportion of non-zero values, an encoding method is selected between standard row-column matrix encoding and CSR sparse matrix encoding. When a preset matrix size condition is met, compression processing is performed to generate corresponding matrix shape identifiers and compression identifiers. The matrix shape identifier and compression identifier are written into the corresponding header file metadata. The header file metadata and the encoded entity data blocks, or the encoded and compressed entity data blocks, are separated and written into the spatial analysis library. A spatiotemporal index is established based on the block-level positioning key. When responding to a query or calculation request, the set of hit entity data blocks is first determined based on the spatiotemporal index and header file metadata. Before reading the hit entity data blocks, the corresponding decoding path and decompression path are determined based on the matrix shape identifier and compression identifier recorded in the corresponding header file metadata. Subsequently, only the hit entity data blocks are read, decompressed, decoded, and calculated in situ, and the results are returned. The missing entity data blocks do not participate in the execution chain.

[0008] In a preferred embodiment, the spatial analysis library is an object-relational spatial database. A radar data structure is defined within this database, and read, write, query, calculate, and analyze this data structure are implemented through a database extension plugin. The radar data structure includes the following fields: version, x, y, xres, yres, srid, col, row, type, zero_info, compress, and Data. The type field identifies whether the matrix is ​​encoded in standard row-column or CSR sparse matrix format, and the compress field identifies whether compression is used. The type and compress fields constitute the storage fields corresponding to the matrix shape identifier and compression identifier.

[0009] In a preferred embodiment, the block-level positioning key serves as a unified association field between header file metadata, entity data blocks, and the spatiotemporal index. It is used to limit the local statistical range of the corresponding entity data block, determine the write objects for the matrix form identifier and compression identifier, and identify the corresponding hit entity data block during the request phase using the spatiotemporal index. The determination of the set of hit entity data blocks, as well as the determination of the decoding and decompression paths, are completed before reading the hit entity data blocks. Different hit entity data blocks are allowed to use different decoding paths based on their respective matrix form identifiers and compression identifiers, and different decompression paths when the corresponding entity data block is in a compressed state.

[0010] In a preferred embodiment, when the proportion of non-zero values ​​is greater than a preset proportion threshold, standard row and column matrix encoding is used; when the proportion of non-zero values ​​is less than or equal to the preset proportion threshold, CSR sparse matrix encoding is used. Preferably, the preset proportion threshold is 10%. The CSR sparse matrix encoding includes a data value array, an indices column index array, and an indptr row pointer array, wherein data is used to store the values ​​of non-zero elements in row-major order, indices is used to store the column index of the corresponding non-zero element, and indptr is used to record the starting position of each row's non-zero element in the data value array and the indices column index array.

[0011] In a preferred embodiment, entity data blocks that meet the preset matrix size condition are compressed using a lossless compression method before storage. Preferably, the lossless compression method is the zstd lossless compression method; the preset matrix size condition is that the matrix size exceeds 1000×1000.

[0012] In a preferred embodiment, the radar reflectivity product value is thresholded to be non-negative and stored as an integer. Preferably, it is integerized using dBZ×100 to retain two decimal places and stored as a 2-byte unsigned integer.

[0013] In a preferred embodiment, when reflectivity products generated by weather radars of different bands are input into this method, the original files corresponding to each band are parsed independently; for a single parsing process, the parsing result of the same original file corresponds to a single band reflectivity product.

[0014] In a preferred embodiment, after identifying the hit entity data block, at least one of the following can be performed on the hit entity data block: query, calculation, statistics, cropping, rendering, and analysis. Specifically, calculation includes performing addition, subtraction, multiplication, and division operations on the Radar format data stored in the database; statistics include calculating the maximum, minimum, and average values; cropping includes cropping the Radar format data according to latitude and longitude or polygons; rendering includes mapping the radar reflectivity product to an image based on color bands and outputting it; and analysis includes performing rainfall intensity calculation and echo extrapolation analysis on the Radar format data in the database corresponding to the hit entity data block.

[0015] Compared to existing technologies, this invention does not simply list block storage, encoding switching, compressed storage, and in-library processing side by side. Instead, it uses a block-level positioning key as a unified control relationship, establishing a one-to-one correspondence between local statistical range limitation, encoding / compression decision-making, matrix shape identification and compression identification writing, spatiotemporal index retrieval, and path pre-selection in the request phase. Through this technical solution, the request phase can complete the determination of the hit entity data block and the decoding and decompression paths before reading the entity data block. For different hit entity data blocks, different processing methods can be adopted according to their corresponding identifiers; unhit entity data blocks do not enter the subsequent processing flow. Therefore, the resource consumption caused by unified object construction, full decoding, and full expansion can be reduced, improving the storage efficiency and request response efficiency of radar reflectivity products. Attached Figure Description

[0016] Figure 1 A flowchart of the in-situ computation method for multi-band radar reflectivity products.

[0017] Figure 2 This is a flowchart for data transformation and hybrid matrix storage decision-making.

[0018] Figure 3 This is a functional framework diagram of the PostRadar plugin.

[0019] Figure 4 This is a flowchart of the rainfall intensity analysis process based on the ZR coefficient.

[0020] Figure 5 This is a flowchart of echo extrapolation based on the optical flow analysis module.

[0021] Figure 6 This is a compressed statistical result chart of radar reflectivity product samples. Detailed Implementation

[0022] The technical solution of the present invention will be further described below with reference to the accompanying drawings. It should be noted that the following specific embodiments are used to illustrate the present invention, and not to limit the scope of protection of the present invention.

[0023] like Figure 1 As shown, the method flow of the present invention includes the following steps.

[0024] S101: Receive the raw radar reflectivity product file and read the source data information. The raw file can be in bin or dat format. After real-time data arrives at the data center, the file path is written to a processing queue via a message queue. The decoding program retrieves the path information from the message queue and reads the file. Historical data or supplementary data is read via directory polling. The source data information includes at least spatiotemporal attribute information and observation data content used to form header file metadata and entity data blocks, for subsequent parsing to form header file metadata and entity data blocks. When reflectivity products generated by weather radars of different bands are input into this method, the raw files corresponding to each band are independently parsed to form corresponding header file metadata and entity data blocks.

[0025] S102, the original file is parsed into Radar format data, forming header metadata and entity data blocks. The source data information is split to form initial header metadata, and observations are organized into entity data blocks according to the spatial range corresponding to the same observation timestamp. When the parsing result of the same original file contains multiple different observation timestamps, or multiple different spatial ranges under the same observation timestamp, or both, multiple entity data blocks are formed respectively. The initial header metadata includes at least the radar station code, observation timestamp, spatial range, and resolution, used to record the basic descriptive information corresponding to the entity data blocks. Subsequently, based on the encoding and compression processing results of the entity data blocks, the matrix shape identifier and compression identifier are written into the corresponding header metadata. The entity data blocks, as the smallest spatiotemporal organization unit and the smallest compression unit, participate in subsequent local decision-making, encoding, compression, indexing, and request processing.

[0026] S103. Within the same original file parsing result range, a block-level positioning key is constructed, and a unique correspondence is established between the header file metadata and the entity data blocks. Specifically, the observation timestamp and spatial range combination in the header file metadata is used as the block-level positioning key for the entity data blocks, making each entity data block uniquely associated with the corresponding header file metadata, and performing local decision processing with the entity data block as the smallest spatiotemporal organization unit and the smallest compression unit. The block-level positioning key is used to limit the calculation range of the proportion of non-zero values, the matrix size determination range, and the compression range corresponding to the entity data block. Since the same original file parsing result corresponds to a single band reflectivity product, within the original file parsing result range, using the observation timestamp and spatial range combination as the block-level positioning key can achieve a unique association with the entity data block and the corresponding header file metadata.

[0027] S104, perform spatiotemporal correlation compression processing for each entity data block. Specifically, only the local observations bound to the block-level location key corresponding to the entity data block are used as decision inputs, and thresholding, non-zero value proportion calculation, and matrix size determination are performed within the range defined by the block-level location key. When the proportion of non-zero values ​​is greater than a preset proportion threshold, standard row-column matrix encoding is used; when the proportion of non-zero values ​​is less than or equal to the preset proportion threshold, CSR sparse matrix encoding is used. The standard row-column matrix encoding in this invention refers to a dense matrix encoding method that stores all matrix elements continuously according to the matrix row and column order. When the matrix size meets the preset matrix size condition, further compression processing is performed. After processing, a matrix form identifier and a compression identifier corresponding to the entity data block are generated, and the matrix form identifier and compression identifier are written into the header file metadata corresponding to the entity data block, so that the local encoding compression decision result of the entity data block can be directly called in subsequent request stages.

[0028] S105, the header file metadata and entity data blocks are written into the spatial analysis library, and a spatiotemporal index oriented towards request processing is established. Specifically, the header file metadata, after being written with matrix shape identifiers and compression identifiers, and the encoded entity data blocks are written into the spatial analysis library; for entity data blocks that meet the preset matrix size conditions and have already undergone compression processing, the encoded and compressed entity data blocks are written into the library. At the same time, a spatiotemporal index is established based on the block-level positioning key, so that subsequent requests can first locate and hit the entity data block based on the header file metadata and spatiotemporal index, and then determine the processing path and perform in-library calculations based on the corresponding identifier.

[0029] S106, in response to a query or computation request, perform pre-selection of the read path and in-situ processing of the hit blocks based on header file metadata and spatiotemporal index. Specifically, first, determine the set of hit entity data blocks based on the spatiotemporal index and header file metadata. Before reading the entity data of any hit entity data block, determine the corresponding decoding path and decompression path based on the matrix shape identifier and compression identifier recorded in the header file metadata of each hit entity data block. Specifically, when the matrix shape identifier indicates standard row-column matrix encoding, perform the corresponding standard row-column matrix decoding process; when the matrix shape identifier indicates CSR sparse matrix encoding, perform the corresponding CSR decoding process; when the compression identifier indicates a compressed state, perform the corresponding decompression process first and then the decoding process; when the compression identifier indicates an uncompressed state, perform the decoding process directly. Subsequently, only the hit entity data blocks are read, decompressed, decoded, and computed in-situ, and the results are returned. Different hit entity data blocks are allowed to use different decoding paths based on their respective matrix shape identifiers and compression identifiers, and different decompression paths when the corresponding entity data block is in a compressed state; missing entity data blocks do not undergo reading, decompression, decoding, or in-situ calculation, and do not participate in the execution chain after the path is determined before reading. In-situ calculation in this invention refers to directly performing reading, decompression, decoding, and calculation processing within the spatial analysis library, at the data storage location corresponding to the hit entity data block, without exporting the complete original file to an external node for unified expansion and processing.

[0030] like Figure 2 As shown, in the data conversion and hybrid matrix storage determination process, the original radar reflectivity product file in bin or dat format is first read and the source data information is extracted. Then, the Radar format data is converted and the proportion of non-zero values ​​is calculated. Based on this, the standard row-column matrix encoding or CSR sparse matrix encoding is used according to the proportion of non-zero values. For data that meets the preset matrix size conditions, zstd lossless compression is performed before writing to the database. In engineering implementation, this process simultaneously completes the establishment of a spatiotemporal index based on header file metadata, entity data block encoding, and separate writing of header file metadata and entity data blocks. It also provides support for subsequently locating and filtering target data blocks based on header file metadata, then determining the processing method by combining matrix shape identifiers and compression identifiers, and performing corresponding calculations within the database.

[0031] In one alternative implementation, since the radar reflectivity product ranges from 0 to 100 dBZ, the reflectivity value can be integerized using dBZ × 100, ensuring the encoded result falls within the range of 0 to 10000. This result is then stored as a 2-byte unsigned integer, retaining two decimal places. Compared to using 4-byte floating-point storage, this method reduces the number of bytes required to store a single data value and improves the compression ratio of lossless compression and subsequent calculation speed.

[0032] like Figure 3 As shown, the PostRadar plugin provides query, calculation, statistical, cropping, rendering, and analysis functions for Radar format data, and offers data transformation during the data import stage. For the query function, it first locates and filters based on header metadata, then reads matching entity data blocks. For the calculation function, after filtering based on header metadata and reading only the matched entity data blocks, the query conditions and corresponding processing functions are pushed down to storage nodes or storage subsets to perform addition, subtraction, multiplication, and division operations on the Radar format data in the database. For the statistical function, it performs maximum, minimum, and average value statistics on the Radar format data in the database within the matched entity data block range. For the cropping function, it can crop the Radar format data corresponding to the matched entity data blocks according to latitude and longitude ranges or polygon ranges. For the rendering function, it can map radar reflectivity products into images for output based on color bands. For the analysis function, rainfall intensity calculation and echo extrapolation analysis can be directly performed on the Radar format data corresponding to the hit entity data block in the spatial analysis library. Rainfall intensity calculation is based on the radar reflectivity value corresponding to the hit entity data block and the ZR coefficient input during the call, performing a reflectivity-to-rainfall intensity conversion. Echo extrapolation analysis is based on the current and historical time-based Radar format data stored in the database, combined with the optical flow parameters input during the call to estimate echo motion and generate future time-based extrapolation results. The above query, calculation, statistics, pruning, rendering, and analysis functions are all executed after determining the hit entity data block based on the header file metadata, and the processing objects are limited to the Radar format data corresponding to the hit entity data block in the library. In an optional implementation, the calculation, statistics, pruning, and analysis functions can be encapsulated as database processing functions or user-defined functions, and the corresponding function execution is triggered by the path pre-selection result in the request phase. The storage node or storage subset refers to the data storage location containing the hit entity data block, the database storage node, or a storage fragment obtained by the database based on a spatiotemporal index. Based on the pre-screening results of block-level locator keys, header file metadata, and spatiotemporal indexes, the processing function corresponding to the request can be pushed down to the storage node or storage subset, and the calculation can be directly executed at the data storage location corresponding to the hit entity data block, so as to reduce the overhead of data movement, unified expansion, and irrelevant data processing.

[0033] In this invention, the block-level positioning key is formed by combining the observation timestamp and the spatial range, and establishes a unique correspondence with the corresponding header file metadata and entity data block within the same original file parsing result range. After the matrix morphology identifier and compression identifier are written into the corresponding header file metadata, the request phase can first determine the hit entity data block based on the spatiotemporal index and header file metadata, and then determine the decoding path and decompression path based on the corresponding identifier, so that subsequent in-situ processing is only performed on the hit entity data block.

[0034] like Figure 4 As shown, the rainfall intensity analysis process in the analysis function includes the following steps: S401, reading Radar format data or cropped Radar format data stored in the database; S402, inputting ZR coefficient parameters during function call; S403, calling the preset rainfall intensity calculation method to calculate rainfall intensity; S404, outputting the rainfall intensity calculation result. This part of the analysis capability mainly reflects the spatial analysis library's ability to directly calculate and call the Radar format data corresponding to the hit entity data block within the library. The ZR coefficient is related to regional topography and climate conditions, and the specific coefficient is provided during the call.

[0035] like Figure 5 As shown, the echo extrapolation process in the analysis function includes the following steps: S501, reading the current and historical time-formatted Radar data stored in the database; S502, inputting the parameters required for the optical flow method during function call; S503, calling the optical flow analysis module to perform extrapolation analysis; S504, outputting the extrapolation results for future time-formatted data. This analysis capability mainly manifests as encapsulating the optical flow extrapolation capability into an analysis function within the database. Based on a unified storage structure, it directly performs extrapolation analysis on the current and historical time-formatted Radar data, and the spatial analysis library directly returns the results. Decoding, expansion, or exporting intermediate files is not performed on entities that are not matched.

[0036] Based on the above technical solution, this invention can be applied to business scenarios such as meteorological and weather process database construction, case database construction, and real-time display of radar data. For scenarios that require frequent calls to radar reflectivity products for querying, calculation, rendering, and analysis, this invention, through database expansion, radar data structure, and PostRadar plugin, directly parses the raw radar data in the data center and writes it into the spatial analysis database. The algorithm integrated into the database first locates and filters the target entity data blocks based on the header file metadata, and then selects the appropriate processing path by combining matrix shape identifiers and compression identifiers. The relevant calculations are then directly completed within the database and the results are returned, thereby improving business processing efficiency.

[0037] like Figure 6As shown, statistics were compiled on 14,868 radar reflectivity product samples provided by the customer, recording the number of non-zero elements, total number of elements, sparsity, original size, compressed size, compression ratio, and space-saving percentage. The statistical results show that the minimum space-saving percentage was 67.8%, the maximum was 98.4%, and the average was 92.83%; the minimum sparsity was 0.5268, the maximum was 0.9938, and the average was 0.9232. These results indicate that by employing dual-morphological encoding driven by a preset percentage threshold, CSR sparse matrix encoding, dBZ×100 integerization, and zstd lossless compression, the storage space of radar reflectivity products can be significantly compressed. This also facilitates the spatial analysis library in locating target entity data blocks based on metadata during the request phase, then determining the processing method and performing in-library calculations based on the corresponding identifiers. Furthermore, the aforementioned entity data block compression storage method, combined with the pre-selection mechanism for the path before reading, enables the spatial analysis library to first locate the target entity data block and determine the processing path during the request phase, and then perform in-library calculations on the hit entity data block, thereby achieving a "data remains unchanged, calculation is pushed down" processing method. While maintaining the spatiotemporal organization of radar reflectivity products, this reduces data transmission overhead and irrelevant data processing overhead during the request-response process.

[0038] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. All equivalent substitutions, improvements, or modifications made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for in-situ computation of multi-band radar reflectivity products, characterized in that, include: The system receives and parses the original files of multi-band radar reflectivity products to form initial header metadata and entity data blocks. Within the parsing results of the same original file, the combination of the observation timestamp and spatial range in the initial header metadata is used as the block-level positioning key for each entity data block, so that each entity data block is uniquely associated with the corresponding header metadata. Based only on local observations within the range defined by the corresponding block-level positioning key, thresholding, non-zero value proportion calculation, and matrix size determination are performed on the entity data block. The encoding method is selected between standard row and column matrix encoding and CSR sparse matrix encoding according to the proportion of non-zero values. When the preset matrix size condition is met, compression processing is performed to generate the corresponding matrix shape identifier and compression identifier. Write the matrix shape identifier and compression identifier into the corresponding header file metadata, and separate the header file metadata from the encoded entity data block or the encoded and compressed entity data block and write them into the spatial analysis library, and establish a spatiotemporal index based on the block-level positioning key; Within the same original file parsing result range, the same block-level locator key corresponds to only one entity data block, and the same entity data block corresponds to only one block-level locator key; the block-level locator key serves as a unified association field between header file metadata, entity data blocks, and spatiotemporal index, used to limit the local statistical range of the corresponding entity data block, determine the writing objects of matrix form identifier and compression identifier, and determine the corresponding hit entity data block through spatiotemporal index during the request phase; When responding to a query or calculation request, the set of hit entity data blocks is first determined based on the spatiotemporal index and header file metadata. Then, before reading the hit entity data blocks, the decoding path and decompression path are determined based on the matrix shape identifier and compression identifier in the corresponding header file metadata. The determination of the set of hit entity data blocks, as well as the determination of the decoding path and decompression path, are all completed before reading the hit entity data blocks; different hit entity data blocks are allowed to use different decoding paths based on their respective matrix shape identifiers and compression identifiers, and different decompression paths can be used when the corresponding entity data blocks are in a compressed state; It only performs reading, decompression, decoding, and in-situ computation on the hit entity data blocks, while it does not perform reading, decompression, decoding, and in-situ computation on the missing entity data blocks.

2. The method according to claim 1, characterized in that, The spatial analysis library is an object-relational spatial database; the object-relational spatial database defines a radar data structure for storing multi-band radar reflectivity products, and implements reading, writing, querying, calculation and analysis of the radar data structure through a database extension plugin.

3. The method according to claim 1, characterized in that, The method of selecting the encoding method based on the proportion of non-zero values ​​includes: when the proportion of non-zero values ​​is greater than a preset proportion threshold, standard row and column matrix encoding is used; when the proportion of non-zero values ​​is less than or equal to the preset proportion threshold, CSR sparse matrix encoding is used.

4. The method according to claim 1, characterized in that, For entity data blocks that meet the preset matrix size conditions, lossless compression is used to compress and store the entity data blocks after standard row and column matrix encoding or CSR sparse matrix encoding.

5. The method according to claim 1, characterized in that, The radar reflectivity product values ​​are thresholded to ensure they are non-negative and are stored using an integer format.

6. The method according to claim 1, characterized in that, When reflectivity products generated by weather radars of different bands are input into this method, the original files corresponding to each band are parsed independently; for a single parsing process, the parsing result of the same original file corresponds to a single band reflectivity product.

7. A storage and computing system for multi-band radar reflectivity products, characterized in that, include: The data parsing unit is used to receive the original file of the multi-band radar reflectivity product and parse it to form the initial header file metadata and entity data blocks; The block-level positioning and local decision-making unit is used to determine the block-level positioning key of entity data blocks within the same original file parsing result range, using the combination of observation timestamps and spatial ranges in the header file metadata. Within the same original file parsing result range, the same block-level positioning key corresponds to only one entity data block, and the same entity data block corresponds to only one block-level positioning key. The unit uses only the local observations of the entity data block bound to the corresponding block-level positioning key as decision input to perform thresholding, non-zero value proportion calculation, matrix size determination, encoding method selection, and conditional compression processing on the entity data block, generating matrix shape identifiers and compression identifiers. The identifier writing and index building unit is used to write the matrix shape identifier and the compressed identifier into the corresponding header file metadata, and to separate the header file metadata from the encoded entity data block or the encoded and compressed entity data block and write it into the spatial analysis library. At the same time, it builds a spatiotemporal index based on the block-level positioning key. The block-level positioning key serves as a unified association field between header file metadata, entity data blocks, and spatiotemporal indexes. It is used to limit the local statistical range of the corresponding entity data block, determine the writing objects of the matrix form identifier and compression identifier, and determine the corresponding hit entity data block through the spatiotemporal index during the request phase. The hit block determination unit is used to determine the set of hit entity data blocks based on the spatiotemporal index and header file metadata before reading the hit entity data blocks when a query or calculation request is received. The path pre-selection and in-library execution unit is used to determine the decoding path and decompression path based on the matrix form identifier and compression identifier in the metadata of the header file corresponding to each hit entity data block before reading the hit entity data block. Different hit entity data blocks are allowed to use different decoding paths based on their respective matrix form identifier and compression identifier, and different decompression paths are allowed when the corresponding entity data block is in a compressed state. Reading, decompression, decoding, and in-situ calculation are only performed on hit entity data blocks, while reading, decompression, decoding, and in-situ calculation are not performed on missing entity data blocks.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of any one of claims 1 to 6.

Images