Atmospheric and oceanic data retrieval optimization method and device based on column database

Abstract

The present application relates to an atmospheric and ocean data retrieval optimization method and device based on a column-based database. The method includes: performing multi-dimensional on-demand clipping, extraction and data decoding on historical atmospheric and ocean data; in the element dimension, performing data storage in a column-based database and optimizing element dimension data retrieval through a single element data table obtained by element segmentation; in the spatial dimension, performing data storage in a column-based database through metadata information extraction, and optimizing spatial dimension data retrieval based on constructed spatial coding; in the time dimension, performing data storage in a column-based database and optimizing time dimension data retrieval through time coding obtained by time discretization; on the basis of optimizing atmospheric and ocean data retrieval in element dimension, spatial dimension and time dimension, further performing data storage based on fixed-size data blocks. The use of this method can improve the efficiency of retrieval and access to massive historical atmospheric and ocean data.

Description

Technical Field

[0001] The present application relates to the technical field of data processing and analysis, and in particular to a method and device for optimizing atmospheric and ocean data retrieval based on a column-based database. Background Art

[0002] With the rapid development of atmospheric and oceanic environmental forecasting technology and the increasing frequency of atmospheric and oceanic data collection, the historical atmospheric and oceanic data has exploded. The historical atmospheric and oceanic data is an important resource and technical cornerstone for the operational application of numerical forecasting, and is of great significance to the fields of atmospheric and oceanic environmental forecasting and climate research. However, traditional atmospheric and oceanic databases are mainly for data archiving, and face huge challenges in the efficient retrieval of massive historical atmospheric and oceanic data. Summary of the invention

[0003] Based on this, in order to quickly and accurately extract data from specific areas that meet specific conditions from massive historical atmospheric and oceanic data to support refined atmospheric and oceanic environmental forecasts and auxiliary decision-making, it is necessary to address the problem of low retrieval and access efficiency of the above-mentioned massive historical atmospheric and oceanic data, and provide an atmospheric and oceanic data retrieval optimization method and device based on a columnar database.

[0004] A method for optimizing atmospheric and oceanic data retrieval based on a column-based database, the method comprising:

[0005] According to the needs of atmospheric and marine environmental forecast analysis, the historical atmospheric and marine data are clipped and extracted according to the element dimension, spatial dimension and time dimension to obtain multi-dimensional atmospheric and marine grid data, and the data values ​​of each element in the atmospheric and marine grid data are obtained by decoding the multi-dimensional atmospheric and marine grid data;

[0006] For the atmospheric and ocean grid data of the element dimension, the single element data is formed according to the element type, and the atmospheric and ocean grid data of the corresponding type and the element data value are stored in the column database in the form of a single element data table, and the atmospheric and ocean data retrieval of the element dimension is optimized based on the single element data table;

[0007] For the atmospheric and ocean grid data of spatial dimension, metadata information is extracted and stored in a column-based database. Spatial coding is constructed based on the longitude, latitude and altitude data in the metadata information, and spatial dimension atmospheric and ocean data retrieval optimization is performed based on the spatial coding. Among them, the spatial coding includes two-dimensional plane grid coding based on longitude and latitude and three-dimensional space grid coding based on longitude and latitude altitude.

[0008] For the atmospheric and ocean grid data in the time dimension, the time is discretized and divided into time periods of different lengths based on the time segmentation level, and the time periods corresponding to the atmospheric and ocean grid data are time-discretely encoded to obtain time codes, which are stored in a column-based database, and the atmospheric and ocean data retrieval in the time dimension is optimized based on the time codes;

[0009] On the basis of optimizing the retrieval of atmospheric and oceanic data in element dimension, spatial dimension and time dimension, one-dimensional data is formed by hierarchically reading the metadata information in the atmospheric and oceanic grid data, and the one-dimensional data is divided into data blocks of fixed size according to the preset spatial resolution. The atmospheric and oceanic grid data and element data values ​​in the corresponding area of ​​each data block are stored as row elements of the columnar database, and the data block index is constructed in the columnar database based on the global position information of the data block, and the atmospheric and oceanic data retrieval is optimized based on the data block index.

[0010] In one embodiment, according to the forecast analysis requirements of the atmosphere and ocean environment, the historical atmosphere and ocean data are clipped and extracted according to the element dimension, space dimension and time dimension respectively to obtain multi-dimensional atmosphere and ocean grid data, including:

[0011] According to the needs of atmospheric and marine environmental forecast analysis, the CDO clipping and extraction operators of element dimension, spatial dimension and time dimension are used to clip and extract the historical atmospheric and marine data to obtain the atmospheric and marine data with multi-dimensional spectral mode. The CDO data conversion operator is used to convert the atmospheric and marine data with multi-dimensional spectral mode into a grid mode to obtain multi-dimensional atmospheric and marine grid data.

[0012] In one of the embodiments, the CDO clipping and extraction operator of the spatial dimension is used to clip and extract the historical atmosphere and ocean data, including regular clipping and irregular clipping; wherein, regular clipping performs data clipping and extraction in a specific longitude and latitude area by specifying the maximum longitude and latitude value and the minimum longitude and latitude value of the longitude and latitude area; irregular clipping performs irregular clipping of the atmosphere and ocean data by setting the vector boundary of the specific longitude and latitude area and generating a rasterized file with the same resolution as the historical atmosphere and ocean data as a mask, and setting other areas except the required area as no data in the specific longitude and latitude area.

[0013] In one embodiment, data values ​​of various elements in the atmosphere and ocean grid data are obtained by decoding the multi-dimensional atmosphere and ocean grid data, including:

[0014] For the multi-dimensional atmosphere and ocean grid data in GRIB code format, GRIB message acquisition and GRIB element decoding are performed in sequence to obtain the data value of each element in the atmosphere and ocean grid data; wherein, GRIB message acquisition is used to obtain the data structure information of the atmosphere and ocean grid data, including the GRIB encoding version number and the GRIB message body; GRIB element decoding is used to perform segment-by-segment format check, segment-by-segment decoding and element-by-element decoding on the GRIB message body to obtain the data value of each element in the atmosphere and ocean grid data.

[0015] In one embodiment, for the atmospheric and oceanic grid data of the element dimension, single element data is formed according to the element type, the atmospheric and oceanic grid data of the corresponding type and the element data value are stored in a column-based database in the form of a single element data table, and the atmospheric and oceanic data retrieval optimization of the element dimension is performed based on the single element data table, including:

[0016] For the atmospheric and ocean grid data of the element dimension, the single element data is formed according to the element type, and a single element data table with the table name consistent with the element name is constructed. The atmospheric and ocean grid data of the corresponding type and the element data values ​​are stored in the column database in the form of a single element data table;

[0017] When data retrieval is performed based on atmospheric and oceanic elements, data retrieval query is performed by locating a single-element data table in a column-based database with the same name as the atmospheric and oceanic element to be retrieved, thereby achieving optimization of atmospheric and oceanic data retrieval in the element dimension.

[0018] In one of the embodiments, when the atmospheric and ocean grid data are updated, for the same type of atmospheric and ocean elements, the updated data is inserted and merged based on the single-element data table stored in the column-based database; for newly added categories of atmospheric and ocean elements, a new single-element data table is constructed in the column-based database for data writing.

[0019] In one embodiment, for the atmospheric and ocean grid data of the spatial dimension, metadata information is extracted and stored in a column-based database, a spatial code is constructed according to the longitude, latitude and altitude data in the metadata information, and the atmospheric and ocean data retrieval optimization of the spatial dimension is performed based on the spatial code, including:

[0020] Extract metadata information for the atmospheric and oceanic grid data in spatial dimensions; the metadata information includes the longitude, latitude, altitude, feature name, and data time of the atmospheric and oceanic grid data;

[0021] Based on the longitude and latitude data and the binary coding principle of geographic space grid, the longitude and latitude area of ​​the atmospheric and ocean grid data is divided into two-dimensional plane multi-level grids, and in each level of the grid, the reverse Z-order symmetry rule is used for binary coding to obtain a two-dimensional plane grid code based on longitude and latitude.

[0022] According to the vertical stratification characteristics of the atmosphere and ocean, the altitude or depth range of the atmosphere and ocean grid data is non-uniformly divided, and in each vertical layer obtained by the division, the preset subdivision level is used for vertical subdivision, and vertical coding is performed according to the type, level, height reference of each vertical layer and the grid sequence number of the subdivision within the vertical layer. On the basis of the two-dimensional plane grid coding, the vertical coding is further combined to obtain the three-dimensional space grid coding based on the longitude and latitude height; among them, the number of subdivision layers of the longitude and latitude two-dimensional plane is consistent with the number of subdivision layers of the vertical layer, both of which are 32 layers; among them, the vertical layer includes It includes the thermosphere, mesosphere, stratosphere and troposphere of the atmosphere, and the upper seawater layer, middle ocean layer, deep ocean layer, hadal layer and ultra-deep ocean layer of the ocean layer; the vertical code is 32 bits in total, divided into 4 sections. The first section represents the atmosphere or ocean, which occupies 1 bit; the second section represents the level of the vertical layer, which occupies 5 bits; the third section represents the height reference of the vertical layer, which occupies 18 bits, and can be expanded to support 260km height reference; the fourth section represents the grid number of the subdivision within the vertical layer, which occupies 8 bits, and can be expanded to represent 256 vertical grid units within the vertical layer;

[0023] When data retrieval is based on the spatial range, the spatial dimension of atmospheric and ocean data retrieval optimization is achieved by spatially encoding the start and end longitude and latitude coordinates and vertical stratification within the spatial range, and comparing the spatial encoding results with the two-dimensional plane grid code or three-dimensional space grid code in the column database for data retrieval query.

[0024] In one embodiment, for the atmospheric and oceanic grid data in the time dimension, the time is discretized and subdivided into time periods of different lengths based on the time subdivision level, the time periods corresponding to the atmospheric and oceanic grid data are time-discretely encoded to obtain time codes, which are stored in a column-based database, and the atmospheric and oceanic data retrieval optimization in the time dimension is performed based on the time codes, including:

[0025] For the atmospheric and ocean grid data in the time dimension, the time division levels corresponding to different time units are used to discretize time into time periods of different lengths; among them, the time division levels are 1 to 8, and the time units corresponding to the 1st to 8th time division levels are year, season, month, week, day, hour, minute, and second respectively;

[0026] The time period corresponding to the atmospheric and ocean grid data is time-discretely encoded to obtain the time code, which is stored in a column-based database; the time code has a total of 72 bits. In order from the first bit to the seventh bit, it represents the time segmentation level; the eighth bit to the 24th bit represents the encoding of the year; the 25th bit to the 50th bit represents the encoding of the month, day, hour, minute and second, and each time unit occupies 4 bits, 5 bits, 5 bits, 6 bits and 6 bits respectively; the 51st bit to the 70th bit represents the encoding of milliseconds and microseconds, each occupying 10 bits; the 71st bit represents the time mark; the 72nd bit is the AD mark, which is used to identify BC or AD;

[0027] When data retrieval is performed based on a time range, the time range is split into multiple time periods, and time discrete encoding is performed separately. The time discrete encoding results of multiple time periods are compared in parallel with the time encoding in the column database to perform parallel data retrieval queries. The final query result is obtained by merging multiple query results, thereby achieving optimization of atmospheric and ocean data retrieval in the time dimension.

[0028] In one of the embodiments, metadata information in the atmospheric and oceanic grid data is read in layers to form one-dimensional data, and the one-dimensional data is divided into data blocks of fixed size according to a preset spatial resolution, and the atmospheric and oceanic grid data and feature data values ​​within the corresponding area of ​​each data block are stored as row elements of a columnar database, and a data block index is constructed in the columnar database based on the global position information of the data block, and atmospheric and oceanic data retrieval optimization of the data block dimension is performed based on the data block index, including:

[0029] According to the altitude level and time, the atmospheric and oceanic element data value information of all latitude and longitude grid points at each level in the metadata information is read in layers to form one-dimensional data. The one-dimensional data is divided into data blocks of fixed size according to the preset spatial resolution, and the starting and ending longitude and latitude of each data block are recorded;

[0030] A unique identifier is assigned to each data block according to its starting and ending longitudes and latitudes, and the atmospheric and oceanic grid data and feature data values ​​within the corresponding area of ​​each data block assigned with a unique identifier are stored as row elements of a column-based database. In the column-based database, a data block index is constructed based on the global position information of each data block assigned with a unique identifier, and atmospheric and oceanic data retrieval optimization of the data block dimension is performed based on the data block index.

[0031] An atmospheric and oceanic data retrieval optimization device based on a column-based database, the device comprising:

[0032] The data clipping and decoding module is used to clip and extract the historical atmosphere and ocean data according to the element dimension, spatial dimension and time dimension according to the needs of the atmosphere and ocean environment forecast analysis, obtain multi-dimensional atmosphere and ocean grid data, and obtain the data value of each element in the atmosphere and ocean grid data by decoding the multi-dimensional atmosphere and ocean grid data;

[0033] The element dimension retrieval optimization module is used to segment the atmospheric and ocean grid data of the element dimension into single element data according to the element type, store the atmospheric and ocean grid data of the corresponding type and the element data value in the column database in the form of a single element data table, and perform element dimension atmospheric and ocean data retrieval optimization based on the single element data table;

[0034] The spatial dimension retrieval optimization module is used to extract metadata information for the atmospheric and ocean grid data of the spatial dimension and store it in a column-based database, construct spatial coding according to the longitude, latitude and altitude data in the metadata information, and optimize the atmospheric and ocean data retrieval of the spatial dimension based on the spatial coding; wherein the spatial coding includes a two-dimensional plane grid coding based on longitude and latitude and a three-dimensional space grid coding based on longitude and latitude altitude;

[0035] The time dimension retrieval optimization module is used to discretize the time into time periods of different lengths based on the time segmentation level for the atmospheric and ocean grid data in the time dimension, discretize the time periods corresponding to the atmospheric and ocean grid data in time, obtain the time codes, store them in the column-based database, and optimize the atmospheric and ocean data retrieval in the time dimension based on the time codes;

[0036] The data block retrieval optimization module is used to form one-dimensional data by hierarchically reading the metadata information in the atmospheric and ocean grid data on the basis of optimizing the retrieval of atmospheric and ocean data in the element dimension, spatial dimension and time dimension, and divide the one-dimensional data into fixed-size data blocks according to the preset spatial resolution, store the atmospheric and ocean grid data and element data values ​​within the corresponding area of ​​each data block as row elements of the column database, and build a data block index based on the global position information of the data block in the column database, and optimize the retrieval of atmospheric and ocean data based on the data block index.

[0037] Compared with the prior art, the above-mentioned atmospheric and ocean data retrieval optimization method and device based on column-based database have the following technical effects:

[0038] 1. By performing on-demand cutting, extraction and data decoding of multi-dimensional data, the information required for atmospheric and marine environmental forecast analysis can be accurately obtained, avoiding the processing of irrelevant data, enhancing the pertinence and effectiveness of the data, and accelerating the speed of subsequent atmospheric and marine data retrieval.

[0039] 2. For the atmospheric and ocean grid data in the element dimension, the single element data table obtained by element segmentation is used to store the atmospheric and ocean grid data, so that the same type of element data is located in the same single element data table of the column database, so that the element-based data query can be quickly located to the corresponding single element data table, which improves the efficiency of atmospheric and ocean data retrieval and query in the element dimension. In particular, when multiple elements are queried jointly, the query efficiency can be further improved by parallel querying multiple single element data tables.

[0040] 3. For the atmospheric and ocean grid data in the spatial dimension, spatial coding is constructed based on the longitude, latitude and altitude data in the extracted metadata information. The atmospheric and ocean grid data within the target spatial range can be quickly found through spatial coding without traversing the entire data set, which greatly improves the efficiency of retrieval.

[0041] 4. For the atmospheric and ocean grid data in the time dimension, the overall query speed can be accelerated by dividing the time range into multiple time periods for parallel query. In the parallel query process, time coding is used for data retrieval query, which can accurately locate the atmospheric and ocean data in a specific time period and reduce unnecessary global scanning, thereby improving the accuracy and speed of atmospheric and ocean data retrieval in the time dimension.

[0042] 5. On the basis of optimizing the retrieval of atmospheric and ocean data in element dimensions, spatial dimensions, and time dimensions, data blocks of fixed size are formed through layered reading and segmentation. Each data block contains multiple atmospheric and ocean grid data within a certain area. Data is stored in data blocks, which improves data storage efficiency. In addition, by using a column-based database for multi-dimensional data storage and retrieval queries, only single-element data tables, spatial codes, time codes, or data blocks related to the query requirements need to be read in the column-based database without scanning the entire database, which further improves data storage and retrieval reading efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of a flow chart of an atmospheric and ocean data retrieval optimization method based on a column-based database in one embodiment;

[0044] Figure 2 A schematic diagram of the GRIB data organization structure in one embodiment;

[0045] Figure 3 A schematic diagram of reverse Z-order symmetric encoding in an embodiment;

[0046] Figure 4 A schematic diagram of time coding in one embodiment;

[0047] Figure 5A schematic diagram of time segment parallel query in one embodiment;

[0048] Figure 6 FIG. 1 is a schematic diagram of a data block in an embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] In one embodiment, Figure 1 As shown, a method for optimizing atmospheric and ocean data retrieval based on a column-based database is provided, comprising the following steps:

[0051] Step S1, according to the needs of atmospheric and marine environment forecast analysis, the historical atmospheric and marine data are cropped and extracted according to the element dimension, spatial dimension and time dimension to obtain multi-dimensional atmospheric and marine grid data, and the data value of each element in the atmospheric and marine grid data is obtained by decoding the multi-dimensional atmospheric and marine grid data.

[0052] Specifically, according to the needs of atmospheric and marine environmental forecast analysis, the CDO (Climate Data Operator) clipping and extraction operators of the element dimension, spatial dimension, and time dimension are used to clip and extract the historical atmospheric and ocean data to obtain atmospheric and ocean data in a multi-dimensional spectral mode. The CDO data conversion operator is used to convert the atmospheric and ocean data in a multi-dimensional spectral mode into a grid mode to obtain multi-dimensional atmospheric and ocean grid data.

[0053] Among them, the CDO operator and its functional description are shown in Table 1. The CDO clipping and extraction operator can realize the on-demand clipping and extraction of historical atmospheric and oceanic data, effectively reducing the amount of data to be processed, avoiding the processing of unnecessary redundant data, and improving the data extraction speed.

[0054] The CDO data conversion operator performs data mode conversion to obtain atmospheric and ocean grid data. The data in the grid mode is based on geographic space and provides clear longitude and latitude coordinates, which makes the data have an intuitive representation in terms of geographic location. Compared with the spectral mode, the data in the grid mode is easier to visualize and interpret, which is convenient for researchers and decision makers to understand and use. In addition, the grid mode data is usually easier to combine with geographic information, which provides a basis for improving the efficiency of subsequent spatial dimension data retrieval.

[0055] Table 1 CDO operators and their functional descriptions

[0056]

[0057] Furthermore, the CDO clipping and extraction operator of the spatial dimension is used to clip and extract the historical atmosphere and ocean data, including regular clipping and irregular clipping. Among them, regular clipping performs data clipping and extraction in a specific longitude and latitude area by specifying the maximum and minimum longitude and latitude values ​​of the longitude and latitude area; irregular clipping sets the vector boundary of the specific longitude and latitude area, and generates a rasterized file with the same resolution as the historical atmosphere and ocean data as a mask. In the specific longitude and latitude area, other areas except the required area are set as no data, thereby performing irregular clipping of the atmosphere and ocean data.

[0058] It should be noted that, in addition to CDO, this method can also use other data processing tools such as GDAL (Geospatial Data Abstraction Library, an open source raster spatial data conversion library) to achieve data clipping on demand.

[0059] Furthermore, by decoding the multi-dimensional atmospheric and oceanic grid data, the data values ​​of each element in the atmospheric and oceanic grid data are obtained, including:

[0060] For the multi-dimensional atmosphere and ocean grid data in GRIB code format, GRIB message acquisition and GRIB element decoding are performed in sequence to obtain the data values ​​of the atmosphere and ocean elements in the atmosphere and ocean grid data. Among them, GRIB message acquisition is used to obtain the data structure information of the atmosphere and ocean grid data, including the GRIB encoding version number and the GRIB message body; GRIB element decoding is used to perform segment-by-segment format check, segment-by-segment decoding, and element-by-element decoding on the GRIB message body to obtain the data values ​​of each element in the atmosphere and ocean grid data.

[0061] Among them, GRIB code is a compressed binary code that is independent of computers. There are two formats, GRIB1 and GRIB2. The historical atmospheric and oceanic data obtained by this method adopts GRIB2 format, which is divided into 9 segments in the data organization structure. The description of each segment is shown in Table 2. A GRIB2 format information can represent multiple fields, of which segments 4-7 can be combined and reused. For example: the 500hPa height field forecasts of 12, 24, 36 and 48 hours generated by the numerical forecast model can be included in the same GRIB2 information, such as Figure 2 shown.

[0062] Table 2 GRIB2 format data field and description

[0063]

[0064] Step S2, for the atmospheric and ocean grid data of the element dimension, split it into single element data according to the element type, store the atmospheric and ocean grid data of the corresponding type and the element data value in the column database in the form of a single element data table, and optimize the atmospheric and ocean data retrieval of the element dimension based on the single element data table.

[0065] Specifically, for the atmospheric and ocean grid data of the element dimension, single element data is formed according to the element type segmentation, and a single element data table with a table name consistent with the element name is constructed, and the atmospheric and ocean grid data of the corresponding type and the element data value are stored in the column database in the form of a single element data table; when data retrieval is performed based on the atmospheric and ocean elements, the single element data table with the same name as the atmospheric and ocean elements to be retrieved in the column database is located for data retrieval query, so as to realize the optimization of atmospheric and ocean data retrieval of the element dimension. Specifically, the column database in this application adopts the open source ClickHouse.

[0066] It can be understood that the single-element data table obtained by element segmentation is used to store atmospheric and ocean grid data, so that the same element data at different grid points are located in a single-element data table in the column-based database, so that element-based data queries can be quickly located to the corresponding single-element data table, thereby improving the efficiency of atmospheric and ocean data retrieval queries in the element dimension, especially when multiple elements are jointly queried, by parallelly querying multiple single-element data tables, the query efficiency is further improved.

[0067] Among them, when the atmospheric and ocean grid data are updated, for the same type of atmospheric and ocean elements, the updated data is inserted and merged based on the single-element data table stored in the column-based database; for newly added categories of atmospheric and ocean elements, a new single-element data table is constructed in the column-based database for data writing.

[0068] Step S3, for the atmospheric and ocean grid data of the spatial dimension, metadata information is extracted and stored in a column-based database, spatial coding is constructed according to the longitude, latitude and altitude data in the metadata information, and spatial dimension atmospheric and ocean data retrieval optimization is performed based on the spatial coding; wherein the spatial coding includes a two-dimensional plane grid coding based on longitude and latitude and a three-dimensional space grid coding based on longitude and latitude altitude.

[0069] Specifically, for the atmospheric and oceanic grid data of spatial dimension, the metadata information is first extracted, which includes the longitude, latitude, altitude, element name and data time in the atmospheric and oceanic grid data. Then, spatial coding is constructed based on the longitude, latitude and altitude data in the metadata information. Through spatial coding, the data query based on longitude, latitude and altitude multi-fields is converted into a one-dimensional query based on unified spatial coding, thereby optimizing the query performance of atmospheric and oceanic environmental data based on spatial range.

[0070] The two-dimensional plane grid encoding process in step S3 includes: based on the longitude and latitude data and the binary encoding principle of the geographic space grid, the longitude and latitude area where the atmospheric and ocean grid data is located is divided into two-dimensional plane multi-level grids, and in each level of the grid, the reverse Z-order symmetry rule is used for binary encoding to obtain a two-dimensional plane grid encoding based on longitude and latitude. Specifically, based on the binary encoding principle of the geographic space grid and considering the uniformity of different grid codes, this application adopts binary encoding, first expanding the initial longitude interval [-180°, 180°] and latitude interval [-90°, 90°] to the closest 2 n value, which is 2 8 =256, the longitude interval and latitude interval are expanded to [-256°, 256°], forming a 512°×512° level 0 grid. Similarly, the 1°×1° grid is expanded to 64'×64', and the 1'×1' is expanded to 64″×64'. Then the longitude and latitude grids are divided into two equal parts to form level 1, level 2, and up to level 32 grids. The two-dimensional plane grid division levels and corresponding sizes are shown in Table 3.

[0071] Table 3 2D plane meshing levels and corresponding dimensions

[0072]

[0073] In each level of the grid, the reverse Z-order symmetry rule is used for binary encoding, such as Figure 3 shown.

[0074] Specifically, the binary code of the Northeast region in the 1st level grid is 00; based on the 1st level grid, the Northeast region in the 2nd level grid is coded as 0000, 0001, 0010, and 0011 in reverse Z order; the 3rd level grid is coded in the same way; the Southwest region is symmetrically coded based on the reverse Z order. Among them, the grid subdivision levels 1-9 are degree grids, levels 10-15 are graded grids, levels 16-21 are second grids, and levels 22-32 are sub-second grids, which are used to meet the grid coding representation of high-resolution environmental data such as geographic surveying and mapping. Among them, when the 15th level is subdivided, the grid size has reached the second level (60"×60" is expanded to 64"×64"), and the highest horizontal resolution of the currently widely used global atmospheric reanalysis data is 0.125°. The 32-level grid coding can fully meet the needs of improving the resolution of atmospheric and ocean data, and can be uniformly organized and quickly accessed with high-resolution environmental data such as geographic surveying and mapping.

[0075] The three-dimensional space grid encoding process in step S3 includes:

[0076] According to the characteristics of atmospheric and ocean vertical stratification, the altitude or depth range of atmospheric and ocean grid data is non-uniformly divided, and in each vertical stratification obtained by division, the preset stratification level is used for vertical stratification, and vertical coding is performed according to the type, level, height reference and grid number of each vertical stratification. On the basis of two-dimensional plane grid coding, vertical coding is further combined to obtain three-dimensional space grid coding based on longitude and latitude; among them, the number of stratification levels of longitude and latitude two-dimensional plane is consistent with the number of stratification levels of vertical stratification, both of which are 32 layers, and the height size of the 32nd level is consistent with the horizontal 32-level grid size. At the same time, in order to keep the corresponding level size as close as possible, considering the characteristics of atmospheric and ocean vertical stratification and actual business usage habits, the vertical stratification is sequentially divided into grids of the previous layer according to about 2 times the height size, and reaches 1km level at the 6th layer. The vertical stratification level and the corresponding height size are shown in Table 4.

[0077] Table 4 Vertical division levels and corresponding height dimensions

[0078]

[0079] The vertical layers include the thermosphere, mesosphere, stratosphere and troposphere at the atmospheric level, and the upper seawater layer, middle ocean layer, deep ocean layer, hadal layer and ultra-deep ocean layer at the ocean level. This application takes the vertical space from 100km above sea level to 12km in depth as an example. Combined with the requirements for the refinement of atmospheric and marine environmental data at the vertical height layer, the subdivision levels used in each vertical layer can be preset. The corresponding subdivision levels, subdivision unit sizes, and vertical layer numbers are shown in Table 5.

[0080] The vertical code is 32 bits in total and is divided into 4 sections. The first section represents the atmosphere or ocean, which occupies 1 bit; the second section represents the level of the vertical layer, which occupies 5 bits; the third section represents the altitude reference of the vertical layer, which occupies 18 bits and can be expanded to support 260km altitude reference; the fourth section represents the grid number within the vertical layer, which occupies 8 bits and can be expanded to represent 256 vertical grid units within the vertical layer.

[0081] Table 5 Vertical layering example

[0082]

[0083] When data retrieval is based on the spatial range, the spatial dimension of atmospheric and ocean data retrieval optimization is achieved by spatially encoding the start and end longitude and latitude coordinates and vertical stratification within the spatial range, and comparing the spatial encoding results with the two-dimensional plane grid code or three-dimensional space grid code in the column database for data retrieval query.

[0084] Step S4, for the atmospheric and ocean grid data in the time dimension, the time is discretized and divided into time periods of different lengths based on the time segmentation level, the time periods corresponding to the atmospheric and ocean grid data are time-discretely encoded to obtain the time codes, which are stored in a column-based database, and the atmospheric and ocean data retrieval in the time dimension is optimized based on the time codes.

[0085] Specifically, for the atmospheric and ocean grid data of the time dimension, the time segmentation levels corresponding to different time units are used to discretize and segment time into time periods of different lengths. Among them, since the state of atmospheric environmental elements will change with the passage of time. In order to accurately reflect the time evolution characteristics of atmospheric data, it is necessary to discretize and segment time and encode it. The description of environmental elements adopts the Gregorian calendar. The Gregorian calendar uses "year, month, day, hour, minute, second, and millisecond" as the time unit. On this basis, according to the time resolution and time series characteristics of meteorological data, the time segmentation levels used in this application are shown in Table 6, including levels 1 to 8. The time units corresponding to the time segmentation levels of levels 1 to 8 are year, season, month, week, day, hour, minute and second, respectively. Different time units represent different time segmentation levels, and each time segmentation level constitutes a hierarchical relationship, and the time segmentation level corresponds to the time level one by one.

[0086] Table 6 Time segmentation levels

[0087]

[0088] The time period corresponding to the atmospheric and ocean grid data is time-discretely encoded to obtain the time code, which is stored in the column database. The time code is as follows: Figure 4 As shown, there are 72 bits in total. In order from front to back, the 1st bit to the 7th bit represent the time segmentation level. Specifically, 128 time segmentation levels can be represented, which is convenient for subsequent discrete time expansion; the 8th bit to the 24th bit represent the encoding of the year, which can represent 120,000 BC to 13,000 AD; the 25th bit to the 50th bit represent the encoding of the month, day, hour, minute and second, and each time unit occupies 4 bits, 5 bits, 5 bits, 6 bits and 6 bits respectively; the 51st bit to the 70th bit represent the encoding of milliseconds and microseconds, each occupying 10 bits; the 71st bit represents the moment identification; the 72nd bit is the AD identification bit, which is used to identify BC or AD. For example, for the moment "2024-06-20 14:30:59", the time segmentation level selects 5, that is, the "day" level, and its time encoding is:

[0089] 00001010000001111110100001101010000000000000000000000000000000000000000011.

[0090] This time code can be used to query and update the atmospheric and ocean data and other environmental data within the time range of "2024-06-20 00:00:00" to "2024-06-20 23:59:59". Figure 5 As shown, when data retrieval is performed based on a time range, the time range is split into multiple time periods, and time discrete encoding is performed separately. The time discrete encoding results of multiple time periods are compared in parallel with the time encoding in the column database to perform data parallel retrieval queries, and the final query result is obtained by merging multiple query results, thereby realizing the optimization of atmospheric and ocean data retrieval in the time dimension.

[0091] It can be understood that by dividing the time range into multiple time periods for parallel query, the overall query speed can be accelerated. In the parallel query process, time coding is used for data retrieval query. Time coding can accurately locate the atmospheric and ocean data of a specific time period, reduce unnecessary global scanning, and thus improve the accuracy and speed of atmospheric and ocean data retrieval in the time dimension.

[0092] Step S5, on the basis of optimizing the retrieval of atmospheric and oceanic data in element dimension, spatial dimension and time dimension, one-dimensional data is formed by hierarchically reading metadata information in atmospheric and oceanic grid data, and the one-dimensional data is divided into data blocks of fixed size according to a preset spatial resolution, and the atmospheric and oceanic grid data and element data values ​​within the corresponding area of ​​each data block are stored as row elements of a column-based database, and a data block index is constructed in the column-based database based on the global position information of the data block, and the retrieval of atmospheric and oceanic data is optimized based on the data block index.

[0093] Specifically, according to the altitude level and time, the atmospheric and ocean element data value information of all latitude and longitude grid points at each level in the metadata information is read in layers to form one-dimensional data. The one-dimensional data is divided into fixed-size data blocks according to the preset spatial resolution, and the starting and ending longitude and latitude of each data block are recorded. Figure 6 As shown in the figure, at a spatial resolution of 0.125°, the relative humidity field data block obtained by one-dimensional data segmentation includes data of 8 grid points in the longitude and latitude directions, with the longitude and latitude of the upper left cell as the starting longitude and latitude of the data block, and the longitude and latitude of the lower right cell as the ending longitude and latitude of the data block. Its element value is a one-dimensional array containing 64 elements. Data storage based on data blocks can improve data storage efficiency. Figure 6Where startx is the starting longitude, starty is the starting latitude, endx is the ending longitude, and endy is the ending latitude.

[0094] A unique identifier is assigned to each data block according to its starting and ending longitudes and latitudes, and the atmospheric and oceanic grid data and feature data values ​​within the corresponding area of ​​each data block assigned with a unique identifier are stored as row elements of a column-based database. In the column-based database, a data block index is constructed based on the global position information of each data block assigned with a unique identifier, and atmospheric and oceanic data retrieval optimization of the data block dimension is performed based on the data block index.

[0095] The fields of the unique identifier of the data block are shown in Table 7, and are composed of the starting longitude, ending longitude, starting latitude, and ending latitude of the data block.

[0096] Table 7 Unique identifier fields and descriptions of data blocks

[0097]

[0098] It can be understood that on the basis of optimizing the retrieval of atmospheric and ocean data in the element dimension, spatial dimension and time dimension, data blocks of fixed size are formed through layered reading and segmentation of data. Each data block contains multiple atmospheric and ocean grid data within a certain area. Data is stored in units of data blocks, which improves the data storage efficiency. In addition, by adopting a column-based database for multi-dimensional data storage and retrieval query, only the single-element data table, spatial code, time code or data block related to the query requirement needs to be read in the column-based database without scanning the entire database, which further improves the data storage and retrieval reading efficiency.

[0099] In order to verify the effectiveness of the atmospheric and ocean data retrieval optimization method based on a columnar database provided by this application, the global atmospheric reanalysis dataset ERA5 is used as a test dataset. First, a columnar database is created based on ClickHouse. As shown in Table 8, each atmospheric and oceanic element corresponds to a single database table to improve the efficiency of element query access; secondly, the ERA5 dataset is clipped and decoded by element, and stored in the database in data blocks. To improve the retrieval efficiency, indexes are created for the spatial coding geocode, time coding timecode, data block start and end longitude and latitude startx, starty, endx, endy, time time, level and other fields. The generation parameters of the spatial coding geocode and time coding timecode are set as follows: 9 levels of horizontal subdivision, 14 levels of height subdivision, and 5 levels of time subdivision. Other parameters such as longitude, latitude, time, and altitude (mode layer) are obtained through decoding. Finally, the effectiveness of this scheme is evaluated by querying the atmospheric and oceanic element values ​​in different scenarios, including single-element query and multi-element joint query based on longitude and latitude regions, single-element query and multi-element joint query based on spatial coding, query based on time range and time coding, etc. The time consumption of each type of query in this method is the average time consumption of executing 3 queries.

[0100] Table 8 Column database field design description

[0101]

[0102] Specifically, taking the atmospheric elements "temperature" and "relative humidity" as examples, the element query performance is evaluated. The data volume of the element table is shown in Table 9, where the total number of entries indicates the number of data rows stored in the column-based database; the original data size indicates the amount of data stored; and the compressed data size indicates the amount of data after optimization and compression using the column-based database ClickHouse. Among them, ‌GiB (Gibibyte) is a unit in computer storage, and its full name is Gigabit Binary byte, which means binary gigabit byte.

[0103] Table 9 Data volume of element table

[0104]

[0105] First, in order to verify the query performance of atmospheric and oceanic elements based on spatial coding, the temperature, relative humidity and other element values ​​of City A and its surrounding areas (116°~117°E, 39°~40°N) on December 30, 2022 are taken as an example. The time consumption of queries based on longitude and latitude range and based on spatial coding is compared through experiments. The query time condition adopts time coding. The results are shown in Table 10.

[0106] Table 10 Comparison of the time consumption for single region data query

[0107]

[0108] It can be seen from Table 10 that when querying the atmospheric and oceanic element values ​​of a certain area, the query time based on spatial coding is much shorter than the query time based on the longitude and latitude range, and when the temperature and relative humidity are jointly queried, the query performance is improved by 81.7%. This also shows that when using a single-element database table, the query based on spatial coding is more advantageous.

[0109] To further verify the optimization performance of spatial coding query, the temperature, relative humidity and other element values ​​of City A and its surrounding areas (116°~117°E, 39°~40°N) and City B and its surrounding areas (112°~113°E, 28°~29°N) on December 30, 2022 are taken as an example to compare the time consumption of queries based on longitude and latitude range and based on spatial coding. The query time condition adopts time coding. The results are shown in Table 11.

[0110] Table 11 Comparison of the query time of multiple regional data

[0111]

[0112] As shown in Table 11, the query time based on longitude and latitude range, the query time based on spatial coding to query single element temperature and relative humidity in multiple regions, and the query time optimization of meteorological element values ​​such as temperature and relative humidity have all reached more than 80%.

[0113] Secondly, in order to verify the data query performance based on time coding, taking the query of temperature, relative humidity and other element values ​​of City A and surrounding areas (116°~117°E, 39°~40°N) on December 30, 2022 as an example, the query spatial conditions are set to longitude and latitude range and spatial coding respectively. The time consumption of queries based on time range and time coding is compared through experiments. The results are shown in Table 12.

[0114] Table 12 Comparison of query time consumption based on time coding

[0115]

[0116] It can be seen from Table 12 that no matter whether the spatial condition adopts the latitude and longitude range or the spatial coding, and the temporal condition adopts the time coding, the query time performance is improved to varying degrees. On the basis of spatial coding, the query performance is better when the time coding is used.

[0117] Therefore, the atmospheric and ocean data are divided into elements, a single-element data table is constructed based on a column-based database for storage, and spatial and temporal codes are preset according to certain rules, which can effectively improve the query and access efficiency of atmospheric and ocean data.

[0118] In one embodiment, an atmospheric and ocean data retrieval optimization device based on a column-based database is provided, comprising:

[0119] The data clipping and decoding module is used to clip and extract the historical atmosphere and ocean data according to the element dimension, spatial dimension and time dimension according to the needs of the atmosphere and ocean environment forecast analysis, obtain multi-dimensional atmosphere and ocean grid data, and obtain the data value of each element in the atmosphere and ocean grid data by decoding the multi-dimensional atmosphere and ocean grid data;

[0120] The element dimension retrieval optimization module is used to segment the atmospheric and ocean grid data of the element dimension into single element data according to the element type, store the atmospheric and ocean grid data of the corresponding type and the element data value in the column database in the form of a single element data table, and perform element dimension atmospheric and ocean data retrieval optimization based on the single element data table;

[0121] The spatial dimension retrieval optimization module is used to extract metadata information for the atmospheric and ocean grid data of the spatial dimension and store it in a column-based database, construct spatial coding according to the longitude, latitude and altitude data in the metadata information, and optimize the atmospheric and ocean data retrieval of the spatial dimension based on the spatial coding; wherein the spatial coding includes a two-dimensional plane grid coding based on longitude and latitude and a three-dimensional space grid coding based on longitude and latitude altitude;

[0122] The time dimension retrieval optimization module is used to discretize the time into time periods of different lengths based on the time segmentation level for the atmospheric and ocean grid data in the time dimension, discretize the time periods corresponding to the atmospheric and ocean grid data in time, obtain the time codes, store them in the column-based database, and optimize the atmospheric and ocean data retrieval in the time dimension based on the time codes;

[0123] The data block retrieval optimization module is used to form one-dimensional data by hierarchically reading the metadata information in the atmospheric and ocean grid data on the basis of optimizing the retrieval of atmospheric and ocean data in the element dimension, spatial dimension and time dimension, and divide the one-dimensional data into fixed-size data blocks according to the preset spatial resolution, store the atmospheric and ocean grid data and element data values ​​within the corresponding area of ​​each data block as row elements of the column database, and build a data block index based on the global position information of the data block in the column database, and optimize the retrieval of atmospheric and ocean data based on the data block index.

[0124] For the specific definition of the atmospheric and ocean data retrieval optimization device based on a column-based database, please refer to the definition of the atmospheric and ocean data retrieval optimization method based on a column-based database above, which will not be repeated here. Each module in the above-mentioned atmospheric and ocean data retrieval optimization device based on a column-based database can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0125] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for optimizing atmospheric and oceanic data retrieval based on a column-based database, characterized in that: The method comprises: According to the needs of atmospheric and marine environmental forecast analysis, the historical atmospheric and marine data are clipped and extracted according to the element dimension, spatial dimension and time dimension respectively to obtain multi-dimensional atmospheric and marine grid data, and the data value of each element in the atmospheric and marine grid data is obtained by decoding the multi-dimensional atmospheric and marine grid data; For the atmospheric and ocean grid data of the element dimension, the single element data is formed according to the element type, and the atmospheric and ocean grid data of the corresponding type and the element data value are stored in the column database in the form of a single element data table, and the atmospheric and ocean data retrieval of the element dimension is optimized based on the single element data table; For the atmospheric and ocean grid data of spatial dimension, metadata information is extracted and stored in a column-based database, spatial coding is constructed according to the longitude, latitude and altitude data in the metadata information, and spatial dimension atmospheric and ocean data retrieval optimization is performed based on the spatial coding; wherein the spatial coding includes a two-dimensional plane grid coding based on longitude and latitude and a three-dimensional space grid coding based on longitude and latitude altitude; For the atmospheric and oceanic grid data in the time dimension, the time is discretized and subdivided into time periods of different lengths based on the time subdivision level, the time periods corresponding to the atmospheric and oceanic grid data are time-discretely encoded to obtain time codes, which are stored in a column-based database, and the atmospheric and oceanic data retrieval optimization in the time dimension is performed based on the time codes; On the basis of optimizing the retrieval of atmospheric and oceanic data in element dimension, spatial dimension and time dimension, one-dimensional data is formed by hierarchically reading metadata information in atmospheric and oceanic grid data, and the one-dimensional data is divided into data blocks of fixed size according to a preset spatial resolution, and the atmospheric and oceanic grid data and element data values ​​within the corresponding area of ​​each data block are stored as row elements of a column-based database, and a data block index is constructed in the column-based database based on the global position information of the data block, and atmospheric and oceanic data retrieval optimization is performed based on the data block index.

2. The method according to claim 1, characterized in that According to the needs of atmospheric and marine environmental forecast analysis, historical atmospheric and marine data are clipped and extracted according to element dimension, spatial dimension and time dimension to obtain multi-dimensional atmospheric and marine grid data, including: According to the needs of atmospheric and marine environmental forecast analysis, the CDO clipping and extraction operators of element dimension, spatial dimension and time dimension are used to clip and extract historical atmospheric and marine data to obtain atmospheric and marine data with a multi-dimensional spectral pattern. The CDO data conversion operator is used to convert the atmospheric and marine data with a multi-dimensional spectral pattern into a grid pattern to obtain multi-dimensional atmospheric and marine grid data.

3. The method according to claim 2, characterized in that The CDO clipping and extraction operator of spatial dimension is used to clip and extract historical atmosphere and ocean data, including regular clipping and irregular clipping. Among them, the regular clipping performs data clipping and extraction in a specific longitude and latitude area by specifying the maximum longitude and latitude value and the minimum longitude and latitude value of the longitude and latitude area; the irregular clipping sets the vector boundary of the specific longitude and latitude area, and generates a rasterized file with the same resolution as the historical atmosphere and ocean data as a mask. In the specific longitude and latitude area, other areas except the required area are set as no data, so as to perform irregular clipping of atmosphere and ocean data.

4. The method according to claim 3, characterized in that By decoding the multi-dimensional atmosphere and ocean grid data, the data value of each element in the atmosphere and ocean grid data is obtained, including: For multi-dimensional atmosphere and ocean grid data in GRIB code format, GRIB message acquisition and GRIB element decoding are performed in sequence to obtain the data value of each element in the atmosphere and ocean grid data; wherein the GRIB message acquisition is used to obtain data structure information of the atmosphere and ocean grid data, including a GRIB encoding version number and a GRIB message body; the GRIB element decoding is used to perform segment-by-segment format check, segment-by-segment decoding, and element-by-element decoding on the GRIB message body to obtain the data value of each element in the atmosphere and ocean grid data.

5. The method according to claim 1, characterized in that For the atmospheric and oceanic grid data of the element dimension, the single element data is formed according to the element type, and the atmospheric and oceanic grid data of the corresponding type and the element data value are stored in the column database in the form of a single element data table. The atmospheric and oceanic data retrieval of the element dimension is optimized based on the single element data table, including: For the atmospheric and ocean grid data of the element dimension, the single element data is formed according to the element type, and a single element data table with the table name consistent with the element name is constructed. The atmospheric and ocean grid data of the corresponding type and the element data values ​​are stored in the column database in the form of a single element data table; When data retrieval is performed based on atmospheric and oceanic elements, data retrieval query is performed by locating a single-element data table in a column-based database with the same name as the atmospheric and oceanic element to be retrieved, thereby achieving optimization of atmospheric and oceanic data retrieval in the element dimension.

6. The method according to claim 5, characterized in that When the atmospheric and ocean grid data are updated, for the same type of atmospheric and ocean elements, the updated data is inserted and merged based on the single-element data table stored in the column-based database; for newly added categories of atmospheric and ocean elements, a new single-element data table is constructed in the column-based database for data writing.

7. The method according to claim 1, characterized in that For the atmospheric and ocean grid data of the spatial dimension, metadata information is extracted and stored in a column-based database, spatial coding is constructed according to the longitude, latitude and altitude data in the metadata information, and spatial dimension atmospheric and ocean data retrieval optimization is performed based on the spatial coding, including: Extract metadata information of the atmospheric and oceanic grid data of spatial dimension; wherein the metadata information includes the longitude, latitude, altitude, element name and data time of the atmospheric and oceanic grid data; Based on the longitude and latitude data and the binary coding principle of geographic space grid, the longitude and latitude area of ​​the atmospheric and ocean grid data is divided into two-dimensional plane multi-level grids, and in each level of the grid, the reverse Z-order symmetry rule is used for binary coding to obtain a two-dimensional plane grid code based on longitude and latitude. According to the vertical stratification characteristics of the atmosphere and ocean, the altitude or depth range where the atmosphere and ocean grid data are located is non-uniformly divided, and in each vertical layer obtained by the division, a preset subdivision level is used for vertical subdivision, and vertical coding is performed according to the type, level, height reference of each vertical layer and the grid sequence number of the subdivision within the vertical layer. On the basis of the two-dimensional plane grid coding, the vertical coding is further combined to obtain a three-dimensional space grid coding based on longitude and latitude; wherein the number of subdivision layers of the longitude and latitude two-dimensional plane is consistent with the number of subdivision layers of the vertical stratification, both of which are 32 layers; wherein the vertical stratification Including the thermosphere, mesosphere, stratosphere and troposphere of the atmosphere, and the upper seawater layer, middle ocean layer, deep ocean layer, hadal layer and ultra-deep ocean layer of the ocean layer; the vertical code is 32 bits in total, divided into 4 sections, the first section represents the atmosphere or ocean, which occupies 1 bit; the second section represents the level of the vertical layer, which occupies 5 bits; the third section represents the height reference of the vertical layer, which occupies 18 bits, and can be expanded to support 260km height reference; the fourth section represents the grid number of the subdivision within the vertical layer, which occupies 8 bits, and can be expanded to represent 256 vertical grid units within the vertical layer; When data retrieval is based on the spatial range, the spatial dimension of atmospheric and ocean data retrieval optimization is achieved by spatially encoding the start and end longitude and latitude coordinates and vertical stratification within the spatial range, and comparing the spatial encoding results with the two-dimensional plane grid code or three-dimensional space grid code in the column database for data retrieval query.

8. The method according to claim 1, characterized in that For the atmospheric and oceanic grid data in the time dimension, the time is discretized and subdivided into time periods of different lengths based on the time subdivision level, the time periods corresponding to the atmospheric and oceanic grid data are time-discretely encoded to obtain time codes, which are stored in a column-based database, and the atmospheric and oceanic data retrieval optimization in the time dimension is performed based on the time codes, including: For the atmospheric and ocean grid data in the time dimension, the time division levels corresponding to different time units are used to discretize the time into time periods of different lengths; among them, the time division levels are 1 to 8, and the time units corresponding to the 1st to 8th time division levels are year, season, month, week, day, hour, minute, and second respectively; The time period corresponding to the atmospheric and ocean grid data is time-discretely encoded to obtain the time code, which is stored in a column-based database; the time code has a total of 72 bits. In order from the first bit to the seventh bit, it represents the time segmentation level; the eighth bit to the 24th bit represents the encoding of the year; the 25th bit to the 50th bit represents the encoding of the month, day, hour, minute and second, and each time unit occupies 4 bits, 5 bits, 5 bits, 6 bits and 6 bits respectively; the 51st bit to the 70th bit represents the encoding of milliseconds and microseconds, each occupying 10 bits; the 71st bit represents the time mark; the 72nd bit is the AD mark, which is used to identify BC or AD; When data retrieval is performed based on a time range, the time range is split into multiple time periods, and time discrete encoding is performed separately. The time discrete encoding results of multiple time periods are compared in parallel with the time encoding in the column database to perform parallel data retrieval queries. The final query result is obtained by merging multiple query results, thereby achieving optimization of atmospheric and ocean data retrieval in the time dimension.

9. The method according to claim 1, characterized in that: One-dimensional data is formed by hierarchically reading metadata information in atmospheric and oceanic grid data, and the one-dimensional data is divided into data blocks of fixed size according to a preset spatial resolution, and the atmospheric and oceanic grid data and element data values ​​within the corresponding area of ​​each data block are stored as row elements of a column-based database, and a data block index is constructed in the column-based database based on the global position information of the data block, and atmospheric and oceanic data retrieval optimization of the data block dimension is performed based on the data block index, including: According to the altitude level and time, the atmospheric and oceanic element data value information of all latitude and longitude grid points at each level in the metadata information is read in layers to form one-dimensional data, and the one-dimensional data is divided into data blocks of fixed size according to a preset spatial resolution, and the starting and ending longitude and latitude of each data block are recorded; A unique identifier is assigned to each data block according to its starting and ending longitudes and latitudes, and the atmospheric and oceanic grid data and element data values ​​within the corresponding area of ​​each data block assigned with a unique identifier are stored as row elements of a column-based database. In the column-based database, a data block index is constructed based on the global position information of each data block assigned with a unique identifier, and atmospheric and oceanic data retrieval optimization of the data block dimension is performed based on the data block index.

10. An atmospheric and ocean data retrieval optimization device based on a column-based database, characterized in that: The device comprises: A data clipping and decoding module is used to clip and extract historical atmosphere and ocean data according to element dimensions, spatial dimensions, and time dimensions according to the needs of atmosphere and ocean environment forecast analysis, to obtain multi-dimensional atmosphere and ocean grid data, and to obtain data values ​​of each element in the atmosphere and ocean grid data by decoding the multi-dimensional atmosphere and ocean grid data; The element dimension retrieval optimization module is used to segment the atmospheric and ocean grid data of the element dimension into single element data according to the element type, store the atmospheric and ocean grid data of the corresponding type and the element data value in the column database in the form of a single element data table, and perform element dimension atmospheric and ocean data retrieval optimization based on the single element data table; A spatial dimension retrieval optimization module is used to extract metadata information for the atmospheric and ocean grid data of the spatial dimension and store it in a column-based database, construct a spatial code according to the longitude, latitude and altitude data in the metadata information, and perform spatial dimension atmospheric and ocean data retrieval optimization based on the spatial code; wherein the spatial code includes a two-dimensional plane grid code based on longitude and latitude and a three-dimensional space grid code based on longitude and latitude; A time dimension retrieval optimization module is used to discretize the time into time periods of different lengths based on the time segmentation level for the atmospheric and ocean grid data of the time dimension, discretize the time periods corresponding to the atmospheric and ocean grid data of the time dimension, obtain the time codes, store them in a column-based database, and optimize the retrieval of the atmospheric and ocean data of the time dimension based on the time codes; The data block retrieval optimization module is used to form one-dimensional data by hierarchically reading metadata information in the atmospheric and oceanic grid data on the basis of optimizing the retrieval of atmospheric and oceanic data in the element dimension, spatial dimension and time dimension, and divide the one-dimensional data into data blocks of fixed size according to a preset spatial resolution, store the atmospheric and oceanic grid data and element data values ​​within the corresponding area of ​​each data block as row elements of a column-based database, and construct a data block index based on the global position information of the data block in the column-based database, and optimize the retrieval of atmospheric and oceanic data based on the data block index.

Images