A method for refining megabit-level meteorological live grid data service

Abstract

The application discloses a method for refining megabit meteorological live grid data, which comprises the following steps: step P1: discretizing the meteorological live grid data to obtain a one-dimensional array B; step P2: calculating a one-dimensional index array C to define a box; step P3: managing the one-dimensional array B and the one-dimensional index array C of meteorological elements by using a database; step P4: constructing a dictionary table name "element code_GIST", judging whether the table exists in the database, and skipping steps P5 and P6 if the table exists; step P5: calculating a one-dimensional array D for generating the dictionary table; step P6: managing the one-dimensional index array C and the one-dimensional array D by using the database; step P7: repeating steps P1 to P6 for different meteorological elements and / or different time; and step P8: searching the meteorological live grid data. The method can quickly obtain the meteorological grid data of a required position and reduce the learning cost of users using the meteorological grid data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of meteorological grid data processing. Specifically, it is a method for fine service of megabit-level meteorological real-time grid data. BACKGROUND

[0002] Weather conditions are highly related to people's production and life. Mobile Internet technology enables people to check weather conditions anytime and anywhere. Meteorological data has gradually become one of the daily necessities. With the progress of meteorological science and technology, numerical models have become the main direction of development in the field of meteorology. Under the support of new technologies such as high-performance computing, mass storage, and application of various observation devices, China's numerical weather prediction model is in an unprecedented stage of rapid development. Along with this, there is a large amount of meteorological data produced by numerical models. This data is mainly stored in Grib or NC format, which is referred to as meteorological grid data in the industry. Taking China's regional multi-source fusion real-time analysis data as an example, the amount of data produced each day is 15.5 GB.

[0003] However, with the improvement of high-performance computing capabilities, meteorological grid data has also developed towards high resolution. Taking meteorological real-time grid data as an example, in September 2013, China's ground fusion real-time product was released, with a spatial resolution of 6.25 km and a single file size of 0.7 megabytes. In June 2017, the China Meteorological Administration's intelligent grid real-time fusion product was released, with a spatial resolution of 5 km and a single data file size of 4 megabytes. In September 2021, China's regional multi-source fusion real-time analysis product was released, with a spatial resolution of 1 km and a single data file size of 105 megabytes. Meteorological grid products have the characteristics of large data volume, fast update frequency, and strong professionalism, and need to be processed with professional tool software. Meteorological industry researchers generally use meteorological grid data through professional software or development kits. A typical tool software is Panoply, a simple visualization software developed by the National Aeronautics and Space Administration (NASA) based on Java. It can be used to quickly view data in NetCDF, HDF, GRIB, and other formats commonly used in the field of earth science, and to make maps. NASA also released a NetCDF development kit for researchers to develop secondary development of meteorological grid data. In China, desktop software such as MeteoInfo and Micaps provides visualization analysis tools for NetCDF, GRIB, and other grid data. Meteorological grid data not only serves professional personnel, but also serves social industry users. Without the help of tools, industry users need to pay a high learning cost if they want to directly apply meteorological grid data. In the field of meteorological data service technology, there is no good solution to the problem between the strong professionalism of meteorological grid data and the simplification of meteorological data demand.

[0004] Currently, the meteorological grid data service mainly has two modes: mode one: database index plus disk file download, that is, the meteorological grid file metadata information is stored in the database, including file name, size, storage path, file generation time, storage time, update time, etc. When used, the user first retrieves the storage path of the file on the disk through the database, and then downloads through FTP or HTTP mode. Mode two: retrieve meteorological grid files according to latitude and longitude points or according to latitude and longitude rectangular frames. This mode is an improved mode of database index plus disk file download. In the download link, the meteorological grid file is parsed through the NetCDF development kit provided by the server to find the grid data value of the latitude and longitude point or multiple grid data values within the latitude and longitude rectangular frame. The meteorological big data cloud platform developed by the National Meteorological Information Center provides meteorological grid data service by combining the above two modes. From the user's point of view, the above two modes have problems in fine service. For example, the user needs meteorological grid data in a certain area, which may be a certain administrative area or an irregular area. Mode one: the user directly downloads the entire meteorological grid data and then processes it twice to extract the meteorological element value at the required position. This mode has a large amount of work for the user, and there is a problem of data asset loss for the data center. Mode two: there will be redundant data, i.e. dirty data, in the rectangular frame range that the user does not need, and the user also needs to process the data twice; and when retrieving the rectangular frame position, the entire meteorological grid file needs to be read to extract the meteorological element value in the rectangular frame range, which has problems in efficiency.

[0005] Therefore, it is necessary to develop an efficient meteorological grid data fine service method to meet the needs of different industry users for meteorological data fine service. SUMMARY

[0006] Therefore, it is necessary to develop an efficient meteorological grid data fine service method to meet the needs of different industry users for meteorological data fine service.

[0007] To solve the above technical problems, the present application provides the following technical solutions:

[0008] A method for fine service of megascale meteorological live grid data, comprising the following steps:

[0009] Step P1: a two-dimensional matrix of meteorological element values in megascale meteorological live grid data is recorded as array A; the meteorological live grid data is discretized to obtain one-dimensional array B of meteorological elements;

[0010] Step P2: according to the grid size of the two-dimensional meteorological live grid data, a one-dimensional index array C is calculated; each storage unit in the one-dimensional index array C is defined as a box, and the value stored in each unit of the array is the number of each box;

[0011] Step P3: using the PostgreSQL database to manage the one-dimensional array B of meteorological elements and the one-dimensional index array C;

[0012] Step P4: constructing a dictionary table "element code_GIST", judging whether the table exists in the PostgreSQL database, if it exists, skipping step P5 and step P6;

[0013] Step P5: according to the size of the two-dimensional meteorological live grid data, calculating a one-dimensional array D for generating the dictionary table;

[0014] Step P6: using the PostgreSQL database to manage the one-dimensional index array C and the one-dimensional array D;

[0015] Step P7: repeating steps P1 to P6 for different meteorological elements and / or different times;

[0016] Step P8: retrieval of megabit-level meteorological live grid data.

[0017] The above-mentioned megabit-level meteorological live grid data refinement service method, the method of discretization in step P1 is:

[0018] Step P101: analyzing the meteorological live grid data to obtain a two-dimensional matrix of meteorological element values, denoted as array A, obtaining the grid space resolution, data time, element code, starting longitude, starting latitude, observed element grid row number and observed element grid column number;

[0019] Step P102: extracting the meteorological element values in the meteorological live grid data, and converting the meteorological element values into a one-dimensional array B;

[0020] Step P103: deleting the parsed meteorological live grid data file and recycling the storage resources.

[0021] The above-mentioned megabit-level meteorological live grid data refinement service method, in step P102, the algorithm for converting the meteorological element values into a one-dimensional array B is:

[0022] B[i*n+j]=A[i][j] (1);

[0023] In formula (1), A represents the two-dimensional matrix of meteorological element values, i represents the row number of the observed element, j represents the column number of the observed element, B represents the one-dimensional array of observed element values after conversion, and n represents the total number of elements in each row of the meteorological live grid data.

[0024] Through two-dimensional circulation, each value of the observed element in the meteorological live grid data is traversed, and the array B is assigned, i and j start from zero.

[0025] The above-mentioned 100-megabit meteorological live grid data refinement service method, in step P2, the length of the one-dimensional index array C is equal to the length of the one-dimensional array B; the calculation method of each value in the one-dimensional index array C is:

[0026] C[i*n+j]=i*n+j (2);

[0027] In formula (2), i represents the row number of the observed element, j represents the column number of the observed element, and n represents the total number of elements in each row of the meteorological live grid data.

[0028] The management method of step P3 of the above-mentioned 100-megabit meteorological live grid data refinement service method is:

[0029] Step P301: Create a new data table and name it "element code_data time_tmp", define it as an unlogged table;

[0030] Step P302: Write the one-dimensional array B of meteorological elements and the one-dimensional index array C into the table "element code_data time_tmp" in the form of copy batch storage, and define the column of the one-dimensional index array C as the ID column;

[0031] Step P303: Create an index normal index for the ID column;

[0032] Step P304: Modify the name of the unlogged table from "element code_data time_tmp" to "element code_data time";

[0033] Step P305: Delete the data table of the previous time of the current data time, that is, delete the data table "element code_data time-2 hours" to recycle the database storage resources.

[0034] The above-mentioned 100-megabit meteorological live grid data refinement service method, step P5 specifically includes the following steps:

[0035] Step P501: Calculate the longitude and latitude coordinates of each meteorological live grid data in the array A by loop calculation; the coordinates of each meteorological live grid center point are:

[0036] lon[j]=x_lon(0)+j*R (3);

[0037] lat[i]=y_lat(0)+i*R (4);

[0038] In formula (3) and formula (4), the one-dimensional array lon represents the longitude of the position of each meteorological live grid data observation value; the one-dimensional array lat represents the latitude of the position of each meteorological live grid data observation value; x_lon(0) represents the longitude of the position of the first meteorological live grid data observation value, y_lat(0) represents the latitude of the position of the first meteorological live grid data observation value; R represents the spatial resolution of the meteorological live grid data; i represents the row number of the observation element, and j represents the column number of the observation element;

[0039] The longitude and latitude values of each meteorological live grid center point coordinate are calculated by looping through i and j respectively;

[0040] Step P502: Store the coordinates of all grid center points in the one-dimensional array D in the order from west longitude to east longitude and from low latitude to high latitude, and the specific algorithm is as follows:

[0041] D[i*n+j] = "POINT("+lon[j]+" "+lat[i]+")" (5).

[0042] The above-mentioned megascale meteorological live grid data refinement service method, step P6 specifically includes the following steps:

[0043] Step P601: Create a dictionary table "element code_GIST";

[0044] Step P602: Batch write the one-dimensional index array C and the one-dimensional array D into "element code_GIST" in the form of copy, define the column where the one-dimensional index array C is located as the ID column, define the column where the one-dimensional array D is located as the lonlat column, and define the lonlat column as a spatial field point type;

[0045] Step P603: Perform GIST spatial indexing on the lonlat column.

[0046] The above-mentioned megascale meteorological live grid data refinement service method, the meteorological elements in step P7 include 2-meter air temperature, relative humidity, wind direction, wind speed, specific humidity and precipitation.

[0047] The above-mentioned megascale meteorological live grid data refinement service method, step P8 specifically includes the following steps:

[0048] Step P801: The client initiates a request for megascale meteorological live grid data refinement service;

[0049] Step P802: Determine the request type of the request initiated by the user; according to the request type, the box number is calculated;

[0050] Step P803: Construct the data table name "element code_data time" being served according to the current time;

[0051] Step P804: Determine whether the table "element code_data time" exists, if it exists, take the box number calculated in step P802 as a condition to query the meteorological elements stored in the table "element code_data time"; if it does not exist, query the meteorological elements of the table "element code_data time-1";

[0052] Step P805: Return the calculation result to the client.

[0053] The above-mentioned method for refining 100-megabit meteorological live grid data services, the request type in step P802 includes single-point query of accessing meteorological live grid data according to latitude and longitude points and area query of accessing meteorological live grid data according to irregular polygons;

[0054] When the request type is single-point query: convert the latitude and longitude points requested by the user into a box number according to the following formula:

[0055] xVal = (userLon - x(0) + R / 2) / R (6);

[0056] yVal = (userLat - y(0) + R / 2) / R (7);

[0057] box = yVal * n + xVal (8);

[0058] In formula (6), formula (7) and formula (8): R is the grid resolution; n represents the total number of elements of meteorological live grid data per row; x(0) is the longitude of the first grid center point where the meteorological element is located; y(0) is the latitude of the first grid center point where the meteorological element is located; userLon is the longitude input by the user; userLat is the latitude input by the user; box is the box number; xVal and yVal are integers;

[0059] When the request type is area query: query the dictionary table according to the following SQL statement, and retrieve all box numbers contained in the polygon through spatial indexing:

[0060] SELECT box from${tableName}WHERE st_intersects(ST_SetSRID(ST_MakePoint(lonlat),4326),st_geomfromgeojson(#{geojson}));

[0061] In the SQL statement: tableName is the dictionary table name; geojson is the polygon input by the user; lonlat is the spatial index field in the dictionary table; and box is the box number.

[0062] The technical solution of the present application achieves the following beneficial technical effects:

[0063] 1. The method for fine service of megascale meteorological grid data provided by the present application adopts a discretization processing and spatialization management manner for meteorological grid data, and through testing, the single-point query efficiency is in the millisecond level, the regional query efficiency is within 2 seconds, and the meteorological grid data service in the research area range can be accurately provided for the user, without the need for data cleaning. Meanwhile, the fine service processing method fundamentally solves the problem of intangible loss of meteorological grid data assets.

[0064] 2. The processing flow adopted by the present application is compared with the flow of directly storing the small square center point coordinates and meteorological element values into a table, and the application of a dictionary table is increased to realize GIS graphical area query, although the flow seems complex, but the data processing efficiency is obviously improved.

[0065] 3. The method for fine service of megascale meteorological live grid data provided by the present application has a simple and efficient data processing flow. By using the method for data service, the user only needs to input the longitude and latitude coordinate points or irregular polygon coordinate strings on the client side, and the meteorological grid data of the required position can be quickly obtained, which greatly reduces the learning cost of the user in using meteorological grid data.

[0066] 4. On the data processing end, the present application adopts discretization processing for meteorological live grid data, adopts spatial database for spatialization management of meteorological grid points, introduces the concept of box, and creates a grid position dictionary table. In the real-time business processing process, the spatial index is replaced by ordinary index, so as to solve the problem of low processing flow efficiency caused by continuous creation of spatial index in the real-time business processing flow. In addition, on the client side, for the single-point query scene, the present application creates a longitude and latitude point to box number conversion algorithm, compared with spatial operation, because the calculation amount of the algorithm is greatly reduced, the query efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0067] Figure 1 The processing flow schematic diagram of megascale meteorological live grid data in the embodiment of the present application;

[0068] Figure 2 The discretization processing schematic diagram of meteorological live grid data in the embodiment of the present application;

[0069] Figure 3 The schematic diagram of defining the concept of box and numbering each box in the embodiment of the present application;

[0070] Figure 4 A schematic diagram illustrating the batch storage of a one-dimensional array of meteorological elements and a one-dimensional array of container numbers in this embodiment of the invention;

[0071] Figure 5 A schematic diagram illustrating the calculation of a one-dimensional array for generating a dictionary table in this embodiment of the invention;

[0072] Figure 6 A schematic diagram illustrating the generation of the dictionary table "Element Code_GIST" in this embodiment of the invention;

[0073] Figure 7 A schematic diagram of the real-time grid data fine-grained retrieval process in an embodiment of the present invention. Detailed Implementation

[0074] 1. Materials and Methods

[0075] 1.1 Research Data

[0076] The 100-megabyte-scale meteorological real-time grid data used in this embodiment comes from the China Regional Multi-Source Fusion Real-Time Analysis Product released by the National Meteorological Information Center. The meteorological elements covered include: 2-meter air temperature, relative humidity, wind direction, wind speed, specific humidity, and precipitation. This data product is developed by fusing various observation data from ground-based automatic meteorological stations, weather radar, and Fengyun meteorological satellites, employing techniques such as multi-grid variational assimilation, optimal interpolation, probability density function matching, physical inversion, and terrain correction. It covers the Asian region (0°N to 60°N, 70°E to 140°E) and is updated hourly. The development technology and accuracy of this data product are comparable to international counterparts (such as GLDAS and NLDAS), but its quality in the Chinese region surpasses that of international counterparts, with higher spatiotemporal resolution. The data product can be obtained through the China Meteorological Administration Satellite Data Broadcasting System (CMACast).

[0077] 1.2 Research Methods

[0078] The 100-megabyte-level meteorological real-time grid data with a spatial resolution of 1 km at 00:00 on January 10, 2023, was processed to provide customers with refined services. The file name of this data is: Z_NAFP_C_BABJ_20230110000445_P_HRCLDAS_RT_CHN_0P01_HOR-TAIR-2023011000.GRB2.

[0079] like Figure 1 As shown, the method for refining the service of 100-megabyte-level meteorological real-time grid data in this embodiment specifically includes the following steps:

[0080] Step P1: a two-dimensional matrix of meteorological element values in the megabit weather live grid data is denoted as array A; the weather live grid data is discretized to obtain a one-dimensional array B of meteorological elements;

[0081] The discretization process specifically includes the following steps:

[0082] Step P101: the weather live grid data is parsed to obtain a two-dimensional matrix of meteorological element values denoted as array A, and the grid space resolution is 0.01 degrees, the data time is 2023011000, the element code is TRIA, the starting longitude is 70 degrees, the starting latitude is 0 degrees, the number of observation element grid latitude direction rows is 6001, and the number of observation element grid longitude direction columns is 7001;

[0083] Step P102: as shown in Figure 2 , the meteorological element value in the weather live grid data is extracted, and the meteorological element value is converted into a one-dimensional array B; the algorithm for converting the meteorological element value into a one-dimensional array B is:

[0084] B[i*n+j]=A[i][j] (1);

[0085] In formula (1), A represents the two-dimensional matrix of meteorological element values, i represents the row number of the observation element, j represents the column number of the observation element, B represents the one-dimensional array of observation element values after conversion, and n represents the total number of elements in each row of the weather live grid data;

[0086] Through two-dimensional loops, each value of the observation element in the weather live grid data is traversed, and the array B is assigned values, i and j start from zero; that is:

[0087] B[0*7001+0]=A[0][0];

[0088] B[0*7001+1]=A[0][1];

[0089] B[0*7001+2]=A[0][2];

[0090] …

[0091] B[6000*7001+7000]=A

[6000]

[7000] ;

[0092] Step P103: the parsed weather live grid data file is deleted to recycle storage resources.

[0093] Step P2: as shown in Figure 3As shown, a one-dimensional index array C is generated based on the grid size of the two-dimensional meteorological data grid; the length of the one-dimensional index array C is equal to the length of the one-dimensional array B; the calculation method for each value in the one-dimensional index array C is as follows:

[0094] C[i*n+j]=i*n+j (2);

[0095] In equation (2): i represents the row number of the observed element, j represents the column number of the observed element, and n represents the total number of elements in each row of the meteorological data grid.

[0096] The calculation process in this embodiment is as follows:

[0097] C[0*7001+0]=0*7001+0;

[0098] C[0*7001+1]=0*7001+1;

[0099] C[0*7001+2]=0*7001+2;

[0100] ...

[0101] C[6000*7001+7000]=6000*7001+7000;

[0102] Each storage cell in the one-dimensional index array C is defined as a box, and the value stored in each cell of the array is the number of each box.

[0103] Step P3: Manage the one-dimensional array B and the one-dimensional index array C of meteorological elements using a PostgreSQL database; the specific management method is as follows:

[0104] Step P301: Create a new data table and name it "TAIR_2023011000_tmp" according to the format "feature code_data time_tmp", and define the data table as an unlogged table;

[0105] Step P302: As Figure 4 As shown, the one-dimensional array B of meteorological elements in step P102 and the one-dimensional index array C in step P2 are written into the table "TAIR_2023011000_tmp" using the copy batch import method, and the column containing the one-dimensional index array C is defined as the ID column;

[0106] Step P303: Create a regular index on the ID column;

[0107] Step P304: Change the name of the unlogged table from "TAIR_2023011000_tmp" to "TAIR_2023011000";

[0108] Step P305: delete the data table of the last time on the current material time (real-time data is updated hourly), that is, delete the data table "TAIR_2023010922" to recover the database storage resources.

[0109] Step P4: construct the dictionary table name "TAIR_GIST". By searching the PostgreSQL database, it is found that "TAIR_GIST" does not exist, so steps P5 and P6 are continued.

[0110] Step P5: according to the size of the two-dimensional meteorological live grid data, calculate the one-dimensional array D used to generate the dictionary table; specifically including the following steps:

[0111] Step P501: as shown in Figure 5 , the longitude and latitude coordinates of each meteorological live grid data in the array A are calculated by loop calculation; the coordinates of each meteorological live grid center point are:

[0112] lon[j]=x_lon(0)+j*R (3);

[0113] lat[i]=y_lat(0)+i*R (4);

[0114] In formula (3) and formula (4), the one-dimensional array lon represents the longitude of the position of each meteorological live grid data observation value; the one-dimensional array lat represents the latitude of the position of each meteorological live grid data observation value; x_lon(0) represents the longitude of the position of the first meteorological live grid data observation value, y_lat(0) represents the latitude of the position of the first meteorological live grid data observation value; R represents the spatial resolution of the meteorological live grid data; i represents the row number of the observation element, and j represents the column number of the observation element.

[0115] By i, j, the longitude and latitude values of the coordinates of each meteorological live grid center point are calculated; that is:

[0116] lon[0]=70+0*0.01;

[0117] lat[0]=0+0*0.01;

[0118] lon[1]=70+1*0.01;

[0119] lat[1]=0+0*0.01;

[0120] …

[0121] lon[42013000]=70+7000*0.01;

[0122] lat[42013000] = 0 + 6000 * 0.01;

[0123] Step P502: Store the coordinates of all grid center points in a one-dimensional array D in the order from west longitude to east longitude and from low latitude to high latitude, and the specific algorithm is as follows:

[0124] D[i*n+j] = "POINT("+lon[j]+" "+lat[i]+")" (5);

[0125] That is:

[0126] D[0] = "POINT("+lon[0]+" "+lat[0]+")";

[0127] D[1] = "POINT("+lon[0]+" "+lat[1]+")";

[0128] D[2] = "POINT("+lon[0]+" "+lat[2]+")";

[0129] …

[0130] D[42013000] = "POINT("+lon[42013000]+" "+lat[42013000]+")";

[0131] Wherein, lon[j] and lat[i] come from step P501.

[0132] Step P6: Use the PostgreSQL database to manage the one-dimensional index array C and the one-dimensional array D; specifically including the following steps:

[0133] Step P601: Create a dictionary table "TAIR_GIST";

[0134] Step P602: As shown in the following table, write the one-dimensional index array C in step P2 and the one-dimensional array D in step P502 into "TAIR_GIST" in the form of copy, define the column where the one-dimensional index array C is located as the ID column, define the column where the one-dimensional array D is located as the lonlat column, and define the lonlat column as a spatial field point type; Figure 6

[0135] Step P603: Do GIST spatial indexing on the lonlat column.

[0136] Step P7: Repeat steps P1 to P6 for different meteorological elements and / or different times; in this embodiment, it includes the following steps:

[0137] ​Step P701: Repeat the above steps P1 to P6 for different meteorological elements involved: relative humidity, wind direction, wind speed, specific humidity, precipitation.

[0138] Step P702: Repeat the above steps P1 to P6 for different times involved.

[0139] Step P8: As shown in Figure 7 The efficient retrieval of megabit-level meteorological live grid data includes the following steps:

[0140] Step P801: The client initiates a request for the megabit-level meteorological live grid data refinement service;

[0141] Step P802: Determine the request type of the user's request; calculate the box number according to the request type; the request type includes single-point query of meteorological live grid data according to latitude and longitude points and area query of meteorological live grid data according to irregular polygons;

[0142] When the request type is single-point query: convert the latitude and longitude point requested by the user into a box number according to the following formula:

[0143] xVal = (userLon - x(0) + R / 2) / R (6);

[0144] yVal = (userLat - y(0) + R / 2) / R (7);

[0145] box = yVal * n + xVal (8);

[0146] In formula (6), formula (7), and formula (8): R is the grid resolution; n represents the total number of elements in each row of meteorological live grid data; x(0) is the longitude of the center point of the first grid where the meteorological element is located; y(0) is the latitude of the center point of the first grid where the meteorological element is located; userLon is the longitude input by the user; userLat is the latitude input by the user; box is the box number; xVal and yVal are integers;

[0147] Suppose the user needs to access the temperature value at the latitude and longitude location (110.83, 40.79);

[0148] xVal = (110.83 - 70 + 0.01 / 2) / 0.01;

[0149] yVal = (40.79 - 0 + 0.01 / 2) / 0.01;

[0150] box = yVal * 7001 + xVal;

[0151] At this time, box = 28561162 (xVal = 4083 yVal = 4079).

[0152] When the request type is area query: Query the dictionary table according to the following SQL statement, and retrieve all box numbers contained in the polygon through spatial indexing:

[0153] SELECT box from${tableName}WHERE st_intersects(ST_SetSRID(ST_MakePoint(lonlat),4326),st_geomfromgeojson(#{geojson}));

[0154] In the SQL statement: tableName is the dictionary table name; geojson is the polygon input by the user; lonlat is the spatial index field in the dictionary table; box is the box number;

[0155] Suppose the user needs to access the air temperature value at the area {"type":"Polygon","coordinates":[[[108.42,35.40],[108.45,35.40],[108.44,35.41]]]}.

[0156] The following SQL statement is used for query: SELECT box from${TAIR_GIST}WHERE st_intersects(ST_SetSRID(ST_MakePoint(lonlat),4326),st_geomfromgeojson('{"type":"Polygon","coordinates":[[[108.42,35.40],[108.45,35.40],[108.44,35.41]]]}'));

[0157] The calculated box numbers are:

[0158] box1 = 24787383

[0159] box2 = 24787384

[0160] box3 = 24787385

[0161] box4 = 24787386

[0162] box5 = 24794386

[0163] Step P803: Construct the data table name being served according to the current time "TAIR_2023011000";

[0164] Step P804: Determine whether the table "TAIR_2023011000" exists, if it exists, take the box number calculated in step P802 as a condition to query the meteorological elements stored in the table "TAIR_2023011000"; if it does not exist, query the meteorological elements of the table "TAIR_2023010922";

[0165] Step P805: Return the calculation result to the client.

[0166] 2. Results and analysis

[0167] 2.1 Comparison results of tests under different service modes

[0168] There are currently two main ways of grid data service, one is to directly download the entire grid data file, and the other is to provide grid data according to the latitude and longitude rectangular frame range. From the perspective of the data center, the megascale meteorological real-time grid data refinement service method of the present embodiment has a very obvious advantage compared to directly providing the entire grid file download or providing meteorological grid file retrieval according to the latitude and longitude rectangular frame range. Downloading the entire meteorological grid file or retrieving the grid file according to the latitude and longitude rectangular frame range both have dirty data relative to the user's research area, and the user needs to clean the data. Moreover, downloading the entire file or retrieving according to the latitude and longitude rectangular frame range requires reading the entire data file on the disk for the first time, which results in low efficiency. Through testing, the response time for retrieving 1-kilometer grid data according to the latitude and longitude rectangular frame range is more than 4 seconds for the first time. The megascale meteorological grid data refinement service method of the present embodiment uses a discretization processing method for meteorological grid data and a spatial management method. Through testing, the single-point query efficiency is in the millisecond level, the regional query efficiency is within 2 seconds, and the meteorological grid data service for the user's research area range can be accurately provided without the need for data cleaning. At the same time, the data processing method of the present embodiment fundamentally solves the problem of intangible loss of meteorological grid data assets.

[0169] 2.2 Comparison results of tests under different processing flows

[0170] Discretization processing is performed on the grid data, and then a relational database is used to spatially manage the meteorological grid data. The simplest processing flow is to store the center point coordinates and meteorological element values of each small square in the meteorological grid into a table, and then perform GIST spatial indexing on the center point coordinate column of the data table, which can realize grid data refinement service, but the processing efficiency is low. In the method of the present embodiment, the concept of box is introduced, such as Figure 4As shown, the box number and the meteorological element value are stored in a table, and the column in which the box number is located is indexed normally. The creation efficiency of the normal index is much higher than that of the spatial index because the spatial index needs to perform complex calculation on the spatial distance. In the test using the grid data with a spatial resolution of 5 kilometers, the total number of discrete data is 1682601, the creation of the normal index takes 1 second, and the creation of the spatial index takes 85 seconds. In the test using the grid data with a spatial resolution of 1 kilometer, the total number of discrete data is 42013000, the creation of the normal index takes 30 seconds, and the creation of the spatial index takes 645 seconds. The processing flow adopted in this embodiment is compared with the flow of directly storing the small square center point coordinates and the meteorological element value in a table, and the application of the dictionary table is increased to realize the GIS graphical area query. Although the flow seems complex, the data processing efficiency is obviously improved.

[0171] 3. Conclusion and discussion

[0172] The method for the megabit-level meteorological live grid number refinement service in this embodiment has a simple and efficient data processing flow. When the service is performed by using the method, the user only needs to input the latitude and longitude coordinate point or the irregular polygon coordinate string on the client side, and the meteorological grid data at the required position can be quickly obtained, which greatly reduces the learning cost of the user in using the meteorological grid data.

[0173] At the data processing end, the meteorological live grid data is discretized in this embodiment, the spatial database is used to manage the spatialization of the meteorological grid points, the box concept is introduced, and the grid position dictionary table is created. In the real-time business processing process, the spatial index is replaced by the normal index, so that the problem of low processing flow efficiency caused by the continuous creation of the spatial index in the real-time business processing flow is solved. In addition, on the client side, for the single-point query scene, the algorithm for converting the latitude and longitude point to the box number (see Figure 7 ) is created, and compared with the spatial operation, the calculation amount is greatly reduced, so that the query efficiency is improved. The data processing and service method proposed in this embodiment is applied to the WeChat applet of the China Meteorological Data Network, and the test application effect is good in practice.

[0174] The application is suitable for live grid data processing and live grid data refinement query business scene application, including the point, line, surface, multi-point, multi-line and multi-surface query application in the GIS graph. The application is not suitable for the long sequence grid historical data and the super large area grid data query.

[0175] It should be noted that the defects of the prior art mentioned in the application are the results obtained by the inventors after practice and careful study. Therefore, the discovery process of the above problems and the solution proposed by the application for the above problems should be the contribution of the inventors to the application in the process of the application.

Claims

1. A method for refining services of a megameteorological live grid data, characterized in that, Comprise the following steps: Step P1: the two-dimensional matrix of the meteorological element value in the megascale meteorological live grid data is recorded as array A; the meteorological live grid data is discretized to obtain a one-dimensional array B of meteorological elements; Step P2: according to the grid size of the two-dimensional meteorological live grid data, a one-dimensional index array C is calculated; Each storage unit in the one-dimensional index array C is defined as a box, and the value stored in each unit in the array is the number of each box; Step P3: using PostgreSQL database to manage the one-dimensional array B of meteorological elements and the one-dimensional index array C; Step P4: construct the dictionary table "element code_GIST", judge whether the table exists in the PostgreSQL database, if it exists, skip step P5 and step P6; Step P5: according to the size of the two-dimensional meteorological live grid data, calculate a one-dimensional array D for generating the dictionary table; Step P6: using PostgreSQL database to manage the one-dimensional index array C and the one-dimensional array D; Step P7: for different meteorological elements and / or different time, repeat steps P1 to P6; Step P8: retrieval of megascale meteorological live grid data; The method of discretization in step P1 is: Step P101: parse the meteorological live grid data to obtain the two-dimensional matrix of the meteorological element value recorded as array A, and obtain the grid space resolution, data time, element code, starting longitude, starting latitude, observation element grid row number and observation element grid column number; Step P102: extract the meteorological element value in the meteorological live grid data, and convert the meteorological element value into a one-dimensional array B; Step P103: delete the parsed meteorological live grid data file and recycle the storage resources; In step P102, the algorithm for converting the meteorological element value into a one-dimensional array B is: B[i*n+j]=A[i][j] (1); In formula (1), A represents the two-dimensional matrix of the meteorological element value, i represents the row number of the observation element, j represents the column number of the observation element, B represents the one-dimensional array of the observation element value after conversion, and n represents the total number of elements in each row of the meteorological live grid data; Through two-dimensional loop, each value of the observation element in the meteorological live grid data is traversed, and the array B is assigned value, i and j start from zero; In step P2, the length of the one-dimensional index array C is equal to the length of the one-dimensional array B; the calculation method of each value in the one-dimensional index array C is: C[i*n+j]=i*n+j (2); In formula (2), i represents the row number of the observation element, j represents the column number of the observation element, and n represents the total number of elements in each row of the meteorological live grid data; The management method of step P3 is: Step P301: create a new data table and name it "element code_data time_tmp", define it as an unlogged table; Step P302: use the copy batch storage form to write the one-dimensional array B of meteorological elements and the one-dimensional index array C into the table "element code_data time_tmp", and define the column of the one-dimensional index array C as the ID column; Step P303: creating an index common index for the ID column; Step P304: modifying the name of the unlogged table from "element code_data time_tmp" to "element code_data time"; Step P305: deleting the data table of the last time before the current data time, i.e. deleting the data table "element code_data time-2 hours" to recover the database storage resources; Step P5 specifically comprises the following steps: Step P501: calculating the longitude and latitude coordinates of each meteorological live grid data in the array A by means of loop calculation; the coordinates of the center point of each meteorological live grid are as follows: lon[j]=x_lon(0)+j*R (3); lat[i]=y_lat(0)+i*R (4); In formula (3) and formula (4), the one-dimensional array lon represents the longitude of the position of each meteorological live grid data observation value; the one-dimensional array lat represents the latitude of the position of each meteorological live grid data observation value; x_lon(0) represents the longitude of the position of the first meteorological live grid data observation value, y_lat(0) represents the latitude of the position of the first meteorological live grid data observation value; R represents the spatial resolution of the meteorological live grid data; i represents the row number of the observed element, and j represents the column number of the observed element; By means of loop traversal of i and j, the longitude and latitude values of the coordinates of each meteorological live grid center point are calculated; Step P502: storing the coordinates of all grid center points in the one-dimensional array D in the order from west longitude to east longitude and from low latitude to high latitude, and the specific algorithm is as follows: D[i*n+j]= "POINT("+lon[j]+" "+lat[i]+")" (5); Step P6 specifically comprises the following steps: Step P601: creating a dictionary table "element code_GIST"; Step P602: writing the one-dimensional index array C and the one-dimensional array D into "element code_GIST" in the form of copy, defining the column of the one-dimensional index array C as the ID column, and defining the column of the one-dimensional array D as the lonlat column, and the lonlat column is defined as a spatial field point type; Step P603: making a GIST spatial index for the lonlat column; Step P8 specifically comprises the following steps: Step P801: the client initiates a request for a megascale meteorological live grid data refinement service; Step P802: judging the request type initiated by the user; calculating the box number according to the request type; Step P803: constructing the name of the data table being served according to the current time, i.e. "element code_data time"; Step P804: judging whether the table "element code_data time" exists, if it exists, then taking the box number calculated in step P802 as the condition to query the meteorological elements stored in the table "element code_data time"; if it does not exist, then querying the meteorological elements of the table "element code_data time-1"; Step P805: returning the calculation result to the client; The request type in step P802 includes a single-point query for accessing the weather live grid data according to a latitude-longitude point and a region query for accessing the weather live grid data according to an irregular polygon; When the request type is the single-point query: convert the latitude-longitude point input by the user into a box number according to the following formula: xVal=(userLon-x(0)+R / 2) / R(6); yVal=(userLat-y(0)+R / 2) / R(7); box=yVal*n+xVal(8); In formula (6), formula (7) and formula (8): R is the grid resolution; n represents the total number of elements in each row of the weather live grid data; x(0) is the longitude of the center point of the first grid where the weather element is located; y(0) is the latitude of the center point of the first grid where the weather element is located; userLon is the longitude input by the user; userLat is the latitude input by the user; box is the box number; xVal and yVal are integers; When the request type is the region query: query the dictionary table according to the following SQL statement, and retrieve all the box numbers contained in the polygon through spatial indexing: SELECT box from${tableName}WHEREst_intersects(ST_SetSRID(ST_MakePoint(lonlat),4326),st_geomfromgeojson(#{geojson})); In the SQL statement: tableName is the name of the dictionary table; geojson is the polygon input by the user; lonlat is the spatial index field in the dictionary table; and box is the box number.

2. The method of claim 1, wherein, The weather elements in step P7 include: 2-meter air temperature, relative humidity, wind direction, wind speed, specific humidity and precipitation.

Images