Marine meteorological observation data millisecond query system based on time sequence block index

Abstract

The invention discloses a marine meteorological observation data millisecond query system based on time sequence block indexing. The marine meteorological observation data millisecond query system comprises a data input module, a data preprocessing module, a data hierarchical modeling module, a data storage cluster, a data analysis engine, a data query engine and a data operation interface module. Based on data modeling, data thinning and data partitioning technologies, data preprocessing is carried out for different meteorological ocean space weather data, and distributed storage and access of mass data are achieved; the method comprises the following steps: inputting ocean actual observation data, forecast data or historical data, carrying out model construction and index design according to meteorological ocean space weather data characteristics, completing distributed storage, and applying a data query engine and a data analysis engine to realize rapid aggregation query of the meteorological ocean space weather data; storage and query of lattice point data are realized by using a distributed database, and the problems of scale and query performance of meteorological ocean space weather data are effectively solved.

Description

Technical Field

[0001] The present invention belongs to the field of data processing, and more specifically, relates to a millisecond-level query system for marine meteorological observation data based on time series block indexing. Background Art

[0002] Meteorological, oceanographic, and spatial weather data is a typical type of big data, characterized by large data volume, high timeliness, and rich data variety. A large portion of this data is spatiotemporal, recording observed or simulated quantities of various physical quantities at various points in time and space. The amount of data generated daily often ranges from tens to hundreds of terabytes and is growing explosively. Storing and efficiently querying this data is becoming an increasingly challenging problem.

[0003] Traditional storage solutions often use a relational database plus a file system to implement the storage and real-time query of this type of meteorological and ocean spatial weather data. This solution has some defects in scalability, maintainability and performance. As the data scale increases, the shortcomings become more and more obvious.

[0004] Traditional storage solutions cannot meet the growing application requirements in terms of scale, performance, and targeted application services. In terms of scale, its traditional IoE architecture determines its inherent flaws in horizontal resource expansion. On the one hand, long-term historical data cannot be fully stored and served online, and some commonly used data, such as reanalysis data and numerical forecast products, must be retrieved based on archived data. On the other hand, there is an even greater lack of storage capacity for massive bursts of data resources. In terms of performance, centralized storage based on disk arrays can only provide access services in seconds to minutes, which cannot meet the millisecond-to-second access service requirements required by big data application analysis. Therefore, we propose a millisecond-level query system for marine meteorological observation data based on time-series block indexing. Summary of the Invention

[0005] The purpose of the present invention is to address the shortcomings of the existing technology and propose a millisecond-level query system for marine meteorological observation data based on time series block indexing. Based on data modeling, data rarefaction and data blocking technology, data preprocessing is performed for different meteorological and marine space weather data to achieve distributed storage and access of massive data. At the same time, based on the data analysis engine, multi-level aggregation of meteorological and marine space weather big data is performed to improve data access performance.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] The millisecond-level query system for marine meteorological observation data based on time series block indexing includes: data input module, data preprocessing module, data hierarchical modeling module, data storage cluster, data analysis engine, data query engine and data operation interface module;

[0008] The data input module is configured to obtain meteorological and ocean space weather observation data, forecast data, reanalysis data and basic information data;

[0009] The data pre-processing module is configured to perform standardization processing on the site data, pre-thinning and block processing on the grid data, and establish a data index;

[0010] The data hierarchical modeling module is configured to construct a data cube based on dimensions including time, latitude and longitude, feature type and altitude;

[0011] The data storage cluster is configured to store the data cube in a distributed manner;

[0012] The data analysis engine is configured to perform multi-order aggregation calculations on the data cube;

[0013] The data query engine is configured to encapsulate query requests and convert the query requests into data operations;

[0014] The data operation interface module is configured to provide a standardized query interface and achieve a millisecond-level response speed.

[0015] Preferably, the data preprocessing module includes a site data standardization subunit and a grid data processing subunit; the grid data processing subunit is configured to perform block processing on grid data with a resolution exceeding 500×500, perform layered thinning processing on grid data with a resolution exceeding 1000×1000, and establish a spatiotemporal index.

[0016] Preferably, the data analysis engine is configured to construct a data subject domain based on the data cube and perform the multi-order aggregation calculation by statistical methods, wherein the multi-order aggregation calculation includes maximum, minimum, sum and average operations;

[0017] The data storage cluster is configured to store the data cube using a distributed table database and map the data to an HTable table. The HTable table constructs a Cube segment instance based on a time range.

[0018] Preferably, the spatiotemporal index weights of the block processing are dynamically optimized through the spatiotemporal resolution entropy model, and the expression is:

[0019] , where γ is the time dimension attenuation coefficient, is the time difference between the query time window and the center time of the data block, is the normalization coefficient of the total span of the data period, To query the spatial resolution of the region, is the maximum global grid resolution, ρ is the adjustment factor for the spatial sparsity of marine meteorological data, is the data point density in the query area.

[0020] Preferably, the execution efficiency of multi-order aggregation calculation is predicted by the tensor contraction complexity model, which is expressed as:

[0021] , where K is the aggregation dimension set, is the cardinality of dimension k, r is the multi-order aggregation rank reduction rate, is the cross-storage node data transmission weight matrix, m is the node index, Spatial locality function for Cube segments.

[0022] Preferably, the Cube segmentation instance is configured to support segmentation query operations on multidimensional grid point data sets, and the segmentation query operations include plane query, profile query, point sequence query and three-dimensional or four-dimensional subspace query.

[0023] Preferably, the data operation interface module includes a grid data interface unit and a site data interface unit; the grid data interface unit is configured to provide a RESTful API interface to support writing, updating, deleting, filtering query, profile analysis and multi-level aggregation analysis of grid data; the site data interface unit is configured to provide a JDBC or ODBC interface to support writing, updating, deleting, filtering query and aggregation analysis of site data;

[0024] The data operation interface module obtains the grid cube data dimension configuration information through the data analysis engine. When receiving a profile analysis request, it first parses the longitude limit range, latitude limit range, altitude level sequence and target meteorological element identifier in the query parameters; based on the spatiotemporal index established by the block processing, it determines the data block set that intersects with the query space range; for the levels in the data cube that exceed the resolution of 1000×1000, it triggers the hierarchical thinning process to perform spatial compression, reducing the original four-dimensional data structure to a three-dimensional tensor containing the time dimension, the compressed space dimension and the element dimension; then, according to the average operation in the multi-order aggregation calculation, it calculates the plane data of each altitude layer, and finally generates and returns the two-dimensional profile image data.

[0025] Preferably, the site data normalization subunit implements an outlier filtering process based on spatial grid classification:

[0026] After receiving the raw data of the marine meteorological observation station transmitted by the data input module, matching the station metadata database in the basic information data to extract the geographic longitude and latitude coordinates and historical observation error characteristics;

[0027] Calling the layered thinning processing engine to construct a spatial grid coordinate system;

[0028] For site groups within the same grid, a three-step verification strategy is implemented:

[0029] 1) Perform standard deviation outlier detection on the temperature, air pressure, and wind speed elements in the site data model;

[0030] 2) Verify the rationality of factor value mutation based on time continuity constraints;

[0031] 3) Correct abnormal records by referring to the spatial correlation of adjacent grids;

[0032] The normalized data is injected into the site cube.

[0033] Preferably, the data query engine includes a grid scene operation subunit and a site scene operation subunit; the grid scene operation subunit is configured to generate a grid data operation request; the site scene operation subunit is configured to generate a site data operation request; the site scene operation subunit implements a sliding aggregation engine of a variable-scale time window, and automatically switches the aggregation mode according to the time parameter submitted by the user:

[0034] Requests with minute-level precision directly access the materialized view of the data cube;

[0035] Requests above the daily level trigger the dynamic window generator, dividing the timeline into overlapping sliding blocks;

[0036] Each block calls the extreme value operation unit of multi-order aggregation calculation to perform maximum / minimum detection in parallel;

[0037] At the same time, the time dimension index in the block processing is integrated to skip the blank period with no data;

[0038] Merge and sort the partition results to generate a complete time series extreme value report.

[0039] The technical effects and advantages of the present invention are as follows: Compared with the existing technology, the millisecond-level query system for marine meteorological observation data based on time series block indexing provided by the present invention performs data preprocessing for different meteorological and marine spatial weather data based on data modeling, data rarefaction, and data block technology, thus realizing distributed storage and access of massive data;

[0040] At the same time, by inputting real-time ocean observation data, forecast data or historical data, building models and indexing designs based on the characteristics of meteorological and oceanic space weather data, completing distributed storage, and using data query engines and data analysis engines, rapid aggregation and query of meteorological and oceanic space weather data can be achieved. Based on the data analysis engine, multi-level aggregation of meteorological and oceanic space weather big data is performed to improve data access performance. A distributed database is used to store and query grid data, providing ultra-large-scale storage capacity, supporting ultra-large-scale concurrent access and low-latency performance, and effectively solving the scale and query performance problems of meteorological and oceanic space weather data. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is an architecture diagram of the millisecond-level query system for marine meteorological observation data based on time series block indexing of the present invention;

[0042] Figure 2 This is the data processing flow chart of the millisecond-level query system for marine meteorological observation data based on time series block indexing. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0044] The present invention provides Figure 1-2 The millisecond-level query system for marine meteorological observation data, based on time-series block indexing, addresses the shortcomings of traditional storage solutions, which suffer from a single storage structure, significant performance bottlenecks, and poor scalability. To meet the needs of accessing large amounts of meteorological and marine space weather data, this system preprocesses different types of meteorological and marine space weather data using data modeling, data rarefaction, and data block partitioning techniques, enabling distributed storage and access of massive amounts of data. Furthermore, based on a data analysis engine, it performs multi-level aggregation of meteorological and marine space weather big data to improve data access performance. Using a distributed database for grid data storage and query, it provides ultra-large-scale storage capacity, supports ultra-large-scale concurrent access, and offers low-latency performance, effectively addressing the scale and query performance issues of meteorological and marine space weather data.

[0045] Meteorological and oceanographic space weather data are large in scale and have multiple query dimensions. Traditional meteorological and oceanographic space weather data storage structure is single and lacks targeted storage design for applications with different characteristics of ocean, meteorology, hydrology, and space. As a result, various applications can barely be used, but they do not provide the best access service experience.

[0046] A grid dataset is typically a multidimensional structure. Various query methods essentially slice and dice this multidimensional data, such as querying a specific plane, each section, a point sequence, or a three- or four-dimensional subspace. The present invention is designed to ensure query performance under various query conditions. A site dataset typically consists of hundreds of millions of records, and various query solutions essentially aggregate this dataset, such as querying a specific time period, a specific region, a point sequence, or historical extreme values ​​of a feature. Maintaining query performance under various query conditions is a major technical challenge in data querying.

[0047] By inputting real-time ocean observation data, forecast data or historical data, building models and indexing designs based on the characteristics of meteorological and oceanic space weather data, completing distributed storage, and using data query engines and data analysis engines, rapid aggregated queries of meteorological and oceanic space weather data can be achieved.

[0048] like Figure 1 , a millisecond-level query system for marine meteorological observation data based on time series block indexing, the architecture includes:

[0049] Data input module, data preprocessing module, data hierarchical modeling module, data storage cluster, data analysis engine, data query engine and data operation interface module;

[0050] The data input module is configured to obtain meteorological and ocean space weather observation data, forecast data, reanalysis data and basic information data;

[0051] The data preprocessing module is configured to standardize the site data, pre-thin and block the grid data, and establish a data index. The data preprocessing module includes a site data standardization subunit and a grid data processing subunit. The grid data processing subunit is configured to block the grid data with a resolution greater than 500×500, perform layered thinning on the grid data with a resolution greater than 1000×1000, and establish a spatiotemporal index. Block processing divides the data into blocks based on the time series, and the spatiotemporal index includes a composite index structure of the time dimension and the space dimension. The spatiotemporal index weight of the block processing is dynamically optimized through the spatiotemporal resolution entropy model, and the expression is:

[0052] , where γ is the time dimension attenuation coefficient, is the time difference between the query time window and the center time of the data block, is the normalization coefficient of the total span of the data period, To query the spatial resolution of the region, is the maximum global grid resolution, ρ is the adjustment factor for the spatial sparsity of marine meteorological data, is the data point density in the query area.

[0053] The site data normalization subunit implements an outlier filtering process based on spatial grid classification:

[0054] After receiving the raw data from the marine meteorological observation stations transmitted by the data input module, the system first matches the station metadata database in the basic information data to extract the geographic latitude and longitude coordinates and historical observation error characteristics. A hierarchical thinning processing engine is then invoked to construct a spatial grid coordinate system with 0.1°×0.1° latitude and longitude units. For clusters of stations within the same grid, a three-step verification strategy is implemented: 1) standard deviation outlier detection is performed on the temperature, pressure, and wind speed elements in the station data model; 2) the rationality of element value mutations is verified based on temporal continuity constraints; and 3) abnormal records are corrected by referring to the spatial correlation of adjacent grids.

[0055] The site scenario operation subunit implements a sliding aggregation engine for variable-scale time windows:

[0056] When querying historical extreme values ​​of site data elements, the data query engine automatically switches aggregation modes based on the time parameters (minutes / days / months / years) submitted by the user. Requests with minute-level precision directly access the materialized view of the data cube. Requests with daily precision or higher trigger a dynamic window generator, dividing the time axis into overlapping sliding blocks. Each block calls the extreme value operation unit of multi-order aggregation calculations to perform maximum / minimum detection in parallel. The time dimension indexes used in the block processing are also integrated to skip periods with no data. Finally, the partition results are merged and sorted to generate a complete time series extreme value report.

[0057] The data hierarchical modeling module is configured to construct a data cube based on dimensions including time, latitude and longitude, feature type, and altitude. The data hierarchical modeling module is configured to construct a data cube based on dimensions and metrics, and store aggregation operation results through materialized views, where the aggregation operation includes combination operations on time series data. The dynamic thinning factor of the hierarchical thinning process is calculated through the spectral radius constraint model, and the calculation formula is:

[0058] ,in, is the Frobenius norm of the grid data covariance matrix, is the maximum eigenvalue of the Gram matrix of the latitude and longitude grid, is the resolution scaling factor, α is the temporal enhancement coefficient, is the query dimension reduction indicator,

[0059] The data storage cluster is configured to store data cubes in a distributed manner. The data analysis engine is configured to perform multi-level aggregation calculations on data cubes. The data storage cluster is configured to use a distributed table database to store data cubes and map the data to HTable tables. The HTable tables construct Cube segment instances based on time ranges. Cube segment instances are configured to support segmentation query operations on multidimensional grid datasets, including plane queries, profile queries, point sequence queries, and three-dimensional or four-dimensional subspace queries. The data storage cluster's Cube segment instance maintenance method includes a time window sliding strategy and a hot and cold data separation mechanism:

[0060] For HTable tables stored in distributed table databases, the system regularly scans the time dimension range of the data cube and divides it into 365 basic time units based on a one-year historical data cycle. It dynamically creates new Cube segment instances to cover new time periods based on the real-time data inflow rate of the data input module. It sets hot data identification rules based on access frequency: when the cumulative number of queries in a certain period exceeds the preset threshold, the Cube segment instance associated with the period is marked as a high-temperature state and migrated to the SSD storage pool. It performs compression archiving on Cube segment instances that have not been accessed for more than 12 months, while retaining the metadata directory of the spatiotemporal index to support millisecond-level historical data wake-up.

[0061] The multi-dimensional combination mechanism of materialized views adopts a star-snowflake hybrid topology:

[0062] For the design of the galaxy data storage model, the data hierarchical modeling module defines the core fact table as a collection of measured values ​​of core meteorological elements. Independent dimension tables are constructed for time, longitude and latitude, altitude, and element type. The time dimension table contains natural day / week / month stratification and timely meteorological attributes. Three materialization strategies are predefined for the data cube: 1) full precomputation of high-frequency three-dimensional slices (time × longitude × latitude);

[0063] 2) Four-dimensional sparse combination (height × time) incremental update on demand;

[0064] 3) Dynamic generation of cross-element related views (sea temperature-wind speed coupled field); all views are stored in column family partitions of the HTable table.

[0065] The data query engine is configured to encapsulate query requests and convert the query requests into data operations; the data query engine includes a grid scene operation subunit and a site scene operation subunit; the grid scene operation subunit is configured to generate grid data operation requests; the site scene operation subunit is configured to generate site data operation requests;

[0066] The data operation interface module is configured to provide a standardized query interface and achieve millisecond-level response speed;

[0067] The data analysis engine is configured to construct data subject domains based on data cubes and perform multi-order aggregation calculations using statistical methods. Multi-order aggregation calculations include maximum, minimum, sum, and average operations. The execution efficiency of multi-order aggregation calculations is predicted by the tensor contraction complexity model, which is expressed as:

[0068] , where K is the aggregation dimension set, is the cardinality of dimension k, r is the multi-order aggregation rank reduction rate, is the cross-storage node data transmission weight matrix, m is the node index, Spatial locality function for Cube segments.

[0069] The data operation interface module includes a grid data interface unit and a site data interface unit. The grid data interface unit is configured to provide a RESTful API interface to support the writing, updating, deletion, filtering query, profile analysis, and multi-level aggregation analysis of grid data. The site data interface unit is configured to provide a JDBC or ODBC interface to support the writing, updating, deletion, filtering query, and aggregation analysis of site data. The data operation interface module is configured to achieve concurrent access by encapsulating query requests. Concurrent access corresponds to query scenarios, including fixed-dimensional grid query operations and site aggregation query operations, with a response speed of less than 1 second to support millisecond-level queries. The load balancing of concurrent queries is determined by spatiotemporal query entropy routing, expressed as:

[0070] ,in, is the spatial load unevenness of node k, is the time dimension load deviation, β is the time sensitivity parameter, Node benchmark throughput, γ(t) is the daily period oscillation factor;

[0071] The grid data interface unit is configured as a multi-dimensional dynamic dimensionality reduction mechanism that supports profile analysis operations:

[0072] The data operation interface module obtains the grid cube data dimension configuration information through the data analysis engine. When receiving a profile analysis request, it first parses the longitude limit range, latitude limit range, altitude level sequence and target meteorological element identifier in the query parameters; based on the spatiotemporal index established by block processing, it determines the set of data blocks that intersect with the query space range; for levels in the data cube that exceed a resolution of 1000×1000, it triggers hierarchical thinning processing to perform spatial compression, reducing the original four-dimensional data structure to a three-dimensional tensor containing the time dimension, the compressed space dimension and the element dimension; then, based on the average operation in the multi-order aggregation calculation, it calculates the plane data of each altitude layer, and finally generates and returns the two-dimensional profile image data.

[0073] Concurrent access control integrates a backpressure regulator that responds to latency constraints:

[0074] Monitor the real-time concurrent load of the data operation interface module. When the number of requests per unit time exceeds the concurrency threshold, initiate five levels of traffic control: 1) Dynamically expand the RegionServer nodes of the data storage cluster;

[0075] 2) Applying a smooth incremental thinning strategy to grid data query requests;

[0076] 3) Pause non-core block responses based on the partition priority index of block processing;

[0077] 4) Site query operations are downgraded to baseline aggregation mode;

[0078] 5) Reject requests that exceed the hardware capacity limit; at the same time, mark each query instance with a timestamp, and automatically interrupt deep aggregation when the processing time approaches the 1-second threshold, returning incomplete results with a progress indicator.

[0079] Based on the above, if Figure 2 As shown, the data processing process based on big data distributed storage is as follows:

[0080] 1) Data input: Publish interfaces for users to access various station and grid data, including meteorological and oceanographic space weather observation data, forecast data, reanalysis data, and basic information data;

[0081] 2) Data preprocessing: Standardize site data and establish a site data model; pre-thin the grid points, perform layered thinning on grid point data with a resolution exceeding 1000x1000, and block processing on grid point data with a resolution exceeding 500x500; establish data indexes based on data attributes to obtain a grid data model;

[0082] 3) Data Hierarchical Modeling: Construct a data cube based on defined dimensions and metrics. Design a galaxy data storage model based on factors such as latitude and longitude, time, feature type, and altitude. Perform metric aggregation operations on each dimension and save the results as a materialized view. Combine all dimensions on top of it.

[0083] 4) Data storage cluster: Maps data to an HTable table. Each build task constructs a Cube instance within a time range, i.e., a Cube segment. Finally, the data is converted into two data models, grid cube and site cube, for storage.

[0084] 5) Data Analysis Engine: Constructs data subject domains based on defined dimensions and metrics. Designs a galaxy data storage model based on factors such as latitude and longitude, time, feature type, and altitude, and combines all dimensions on it.

[0085] 6) Data query engine: This encapsulates the semantic conversion of data operations in both site and grid scenarios, organizes and constructs data operation requests that can be recognized by distributed storage, and publishes them as standard data interfaces that can be recognized by data operation interfaces;

[0086] 7) Data operation interface: Provides a Rest API interface for grid data operations, enabling grid data writing, updating, deleting, filtering queries, profile analysis, and multi-level aggregation analysis operations; provides a JDBC / ODBC driver for site data operations, enabling site data writing, updating, deleting, filtering queries, and aggregation analysis operations.

[0087] Based on the above, and targeting the characteristics and application requirements of meteorological and marine space weather data, combined with industry-leading distributed storage technology, a millisecond-level query system for marine meteorological observation data based on time-series block indexing is constructed. This system provides standardized, real-time, and historically integrated online storage of meteorological and marine space weather data, enabling storage and high-concurrency access to massive meteorological and marine space weather grids / sites, solving the storage and real-time query challenges of massive meteorological and marine space weather data, and providing a better human-computer interaction interface. This makes the acquisition of meteorological and marine space weather data more diverse and user-friendly, better meeting business needs.

[0088] The performance test is as follows:

[0089] The system was tested for performance in scenarios such as data writing, data deletion, data query, and data aggregation. The test results are as follows:

[0090] 1) Concurrent write: 20 concurrent writes, 100MB grid data write time: <5s;

[0091] 2) Concurrent deletion: 20 concurrent deletions, 100MB of grid data deletion time: <1s;

[0092] 3) Grid query: 100 concurrent queries, single-layer 10 million grid point data, fixed data time, data type, element, altitude layer, timeliness, return two-dimensional array: <0.4s;

[0093] 4) Grid point sparse query: 100 concurrent queries, 100 million grid point data in a single layer, taking any 10 million grid point data, with fixed data time, data type, element, altitude layer, timeliness, and number of sparse layers, return a two-dimensional array: <1s;

[0094] 5) Grid point aggregation: 100 concurrent queries, 60 single-layer data sets with 1 million grid points. When the element is fixed, one of the other dimensions (data time, data type, altitude, timeliness, longitude, and latitude) is fixed, and three-level aggregation (maximum, minimum, sum, and average) is performed on two latitudes. Returning a one-dimensional array takes less than 0.8 seconds.

[0095] Site query: 100 concurrent queries, 1 billion records in a single table, 3,000 site queries for 10 years of arbitrary factor aggregation (maximum, minimum, sum, accumulation, greater than, less than, equal to) query: less than 1s.

[0096] In summary, compared with the existing technology, the present invention is based on data modeling, data rarefaction and data segmentation technology, and performs data preprocessing for different meteorological and ocean spatial weather data, thus realizing distributed storage and access of massive data;

[0097] At the same time, by inputting real-time ocean observation data, forecast data or historical data, building models and indexing designs based on the characteristics of meteorological and oceanic space weather data, completing distributed storage, and using data query engines and data analysis engines, rapid aggregation and query of meteorological and oceanic space weather data can be achieved. Based on the data analysis engine, multi-level aggregation of meteorological and oceanic space weather big data is performed to improve data access performance. A distributed database is used to store and query grid data, providing ultra-large-scale storage capacity, supporting ultra-large-scale concurrent access and low-latency performance, and effectively solving the scale and query performance problems of meteorological and oceanic space weather data.

[0098] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A millisecond-level query system for marine meteorological observation data based on time series block indexing, characterized by: include: Data input module, data preprocessing module, data hierarchical modeling module, data storage cluster, data analysis engine, data query engine and data operation interface module; The data input module is configured to obtain meteorological and ocean space weather observation data, forecast data, reanalysis data and basic information data; The data pre-processing module is configured to perform standardization processing on the site data, pre-thinning and block processing on the grid data, and establish a data index; The data hierarchical modeling module is configured to construct a data cube based on dimensions including time, latitude and longitude, feature type and altitude; The data storage cluster is configured to store the data cube in a distributed manner; The data analysis engine is configured to perform multi-order aggregation calculations on the data cube; The data query engine is configured to encapsulate query requests and convert the query requests into data operations; The data operation interface module is configured to provide a standardized query interface and achieve a millisecond-level response speed.

2. The millisecond-level query system for marine meteorological observation data based on time series block indexing according to claim 1 is characterized in that: The data preprocessing module includes a site data standardization subunit and a grid data processing subunit; the grid data processing subunit is configured to perform block processing on grid data with a resolution exceeding 500×500, perform layered thinning processing on grid data with a resolution exceeding 1000×1000, and establish a spatiotemporal index.

3. The millisecond-level query system for marine meteorological observation data based on time series block indexing according to claim 1 is characterized in that: The data analysis engine is configured to construct a data subject domain based on the data cube and perform the multi-order aggregation calculation using a statistical method, wherein the multi-order aggregation calculation includes maximum, minimum, sum and average operations; The data storage cluster is configured to store the data cube using a distributed table database and map the data to an HTable table. The HTable table constructs a Cube segment instance based on a time range.

4. The millisecond-level query system for marine meteorological observation data based on time series block indexing according to claim 2 is characterized in that: The spatiotemporal index weights of the block processing are dynamically optimized through the spatiotemporal resolution entropy model, and the expression is: , where γ is the time dimension attenuation coefficient, is the time difference between the query time window and the center time of the data block, is the normalization coefficient of the total span of the data period, To query the spatial resolution of the region, is the maximum global grid resolution, ρ is the adjustment factor for the spatial sparsity of marine meteorological data, is the data point density in the query area.

5. The millisecond-level query system for marine meteorological observation data based on time series block indexing according to claim 3 is characterized in that: The execution efficiency of multi-order aggregation calculations is predicted by the tensor contraction complexity model, which is expressed as: , where K is the aggregation dimension set, is the cardinality of dimension k, r is the multi-order aggregation rank reduction rate, is the cross-storage node data transmission weight matrix, m is the node index, Spatial locality function for cube segments.

6. The millisecond-level query system for marine meteorological observation data based on time series block indexing according to claim 3 is characterized in that: The Cube segmentation instance is configured to support segmentation query operations on a multi-dimensional grid data set, wherein the segmentation query operations include plane query, profile query, point sequence query, and three-dimensional or four-dimensional subspace query.

7. The millisecond-level query system for marine meteorological observation data based on time series block indexing according to claim 1 is characterized in that: The data operation interface module includes a grid data interface unit and a site data interface unit; the grid data interface unit is configured to provide a RESTful API interface to support writing, updating, deleting, filtering query, profile analysis and multi-level aggregation analysis of grid data; the site data interface unit is configured to provide a JDBC or ODBC interface to support writing, updating, deleting, filtering query and aggregation analysis of site data; The data operation interface module obtains grid cube data dimension configuration information through the data analysis engine. When receiving a profile analysis request, it first parses the longitude limit range, latitude limit range, altitude level sequence and target meteorological element identifier in the query parameters; based on the spatiotemporal index established by the block processing, it determines the data block set that intersects the query space range; For the levels in the data cube that exceed a resolution of 1000×1000, the hierarchical thinning process is triggered to perform spatial compression, reducing the original four-dimensional data structure to a three-dimensional tensor containing a time dimension, a compressed space dimension, and an element dimension; then, the plane data of each height layer is calculated according to the average operation in the multi-order aggregation calculation, and finally two-dimensional cross-sectional image data is generated and returned.

8. The millisecond-level query system for marine meteorological observation data based on time series block indexing according to claim 2 is characterized in that: The site data normalization subunit implements an outlier filtering process based on spatial grid classification: After receiving the raw data of the marine meteorological observation station transmitted by the data input module, matching the station metadata database in the basic information data to extract the geographic longitude and latitude coordinates and historical observation error characteristics; Calling the layered thinning processing engine to construct a spatial grid coordinate system; For site groups within the same grid, a three-step verification strategy is implemented: 1) Perform standard deviation outlier detection on the temperature, air pressure, and wind speed elements in the site data model; 2) Verify the rationality of factor value mutation based on time continuity constraints; 3) Correct abnormal records by referring to the spatial correlation of adjacent grids; The normalized data is injected into the site cube.

9. The millisecond-level query system for marine meteorological observation data based on time series block indexing according to claim 1 is characterized in that: The data query engine includes a grid scene operation subunit and a site scene operation subunit; the grid scene operation subunit is configured to generate a grid data operation request; The site scenario operation subunit is configured to generate a site data operation request; The site scenario operation subunit implements a sliding aggregation engine for variable-scale time windows, automatically switching aggregation modes based on the time parameters submitted by the user: Requests with minute-level precision directly access the materialized view of the data cube; Requests above the daily level trigger the dynamic window generator, dividing the timeline into overlapping sliding blocks; Each block calls the extreme value operation unit of multi-order aggregation calculation to perform maximum / minimum detection in parallel; At the same time, the time dimension index in the block processing is integrated to skip the blank period with no data; Merge and sort the partition results to generate a complete time series extreme value report.

Images