Meteorological data processing mode based on IOTDB time sequence database

Abstract

The invention provides a meteorological data processing mode based on an IOTDB time sequence database, and relates to the field of meteorological data processing, which comprises the following steps: S1, downloading NOAA meteorological data, and analyzing to form two-dimensional data, S2, converting the two-dimensional data into a time sequence data storage format, and writing IOTDB data, S3, converting a query statement into a time sequence database query format, and quickly querying, and S4, when the data volume is too large, sending the query statement to the IOTDB database. The method comprises the following steps of: S101, downloading data forecasted in the next 15 days from a NOAA (Not Open Authorization Accounting) website, and quickly expanding an IOTDB node to meet the demand of data growth; according to the meteorological data processing mode based on the IOTDB time sequence database, rapid storage of data can be achieved through the IOTDB, the storage per second can reach the million level, the meteorological data processing speed is remarkably increased, meanwhile, query is combined into a Key according to query conditions, rapid data returning is achieved, under the condition that the data size is increased, the IOTDB is a distributed database, and the data processing speed is greatly improved. Data storage expansion can be completed only by adding nodes, and the large-scale meteorological data processing requirement is met.

Description

Technical Field

[0001] The present invention relates to the field of meteorological data processing, and in particular to a meteorological data processing method based on an IOTDB time series database. Background Art

[0002] IOTDB is a data management engine that integrates time series data collection, storage and analysis. It is lightweight, high-performance and easy to use. It is suitable for the needs of high-speed writing and complex analysis and query of massive time series data in industrial applications, and has data synchronization, load balancing and operation and maintenance monitoring functions.

[0003] NOAA weather data is based on GRIB-2 data format weather data, published 4 times a day, at 00, 06, 12, 18 o'clock. The data range is the weather forecast for the next 15 days, the data accuracy for the first 5 days is 1 hour, and the data accuracy for the next 10 days is 3 hours. The latitude and longitude, altitude, and data category can be downloaded according to customer selection, and each hour of data is a GRIB-2 format software package.

[0004] The existing NOAA meteorological data has a large amount of data updated at a time, and the data is updated frequently, requiring a large storage space and a fast writing speed. The amount of data involved in a single update reaches hundreds of millions, and the data is updated four times a day. The requirements for the writing speed storage component are relatively high. The writing speed of traditional databases is relatively slow, and horizontal expansion is difficult. The query speed is relatively slow as the data volume grows. Summary of the invention

[0005] The purpose of the invention provided by the present invention is to use IOTDB to realize fast storage of data, and the storage per second can reach millions, which significantly improves the processing speed of meteorological data. At the same time, the query is combined into a Key according to the query conditions, and the data is returned quickly. When the amount of data increases, IOTDB is a distributed database, and data storage expansion can be completed by adding nodes to meet the needs of large-scale meteorological data processing.

[0006] In order to solve the above problem, the present invention provides the following technical solution: based on the meteorological data processing method of IOTDB time series database, the method comprises the following steps:

[0007] Step 1: Load NOAA weather data and analyze it to form two-dimensional data.

[0008] Step 2: Convert the two-dimensional data into a time series data storage format and write it into IOTDB data.

[0009] Step 3: Convert the query statement into the time series database query format for quick query.

[0010] Step 4: When the amount of data is too large, quickly expand the IOTDB nodes to meet the needs of data growth.

[0011] Furthermore, the specific steps of step one are as follows:

[0012] S101: Download the forecast data for the next 15 days from the NOAA website, select the longitude and latitude range to query, select 0.25° for data granularity, and select the following enumeration values ​​for altitude layer: lev_200_mb,lev_250_mb,lev_300_mb,lev_350_mb,lev_400_mb,lev_450_mb,lev_500_mb,lev_550_mb,lev_600_mb,lev_650_mb,lev_700_mb,lev_750_mb,lev_800_mb,lev_850_mb,lev_900_mb,lev_950_mb,lev_1000_mb, and obtain var_TMP, var_UGRD, var_VGRD, which represent the vector values ​​of temperature, wind force and wind direction:

[0013] S102: According to the selection conditions, the JavaNIO package is used to download NOAA raw data and obtain the corresponding time series data packets. The first 120 hours are one data packet per hour, and the next 240 hours are one data packet per three hours. The name of the data packet download is gfs.t18z.pgrb2.0p25.f001 type. After downloading, the data is saved in the local disk;

[0014] S103: UCAR using Java GRIB Package Parse the downloaded GRIB-2 format data to obtain plain text data, and then format the parsed data in two dimensions according to the data download conditions. The resulting data format is [time, longitude, latitude, altitude, temperature, wind force, wind direction].

[0015] Furthermore, in step S101, the var_TMP temperature needs to be converted into degrees Celsius. var_UGRD and var_VGRD are vector data, and Math.sqrt and Math.round need to be used to substitute the two parameters to obtain the wind force in knots and the wind direction in 360 degrees.

[0016] Furthermore, the specific steps of step 2 are as follows:

[0017] S201: For a two-dimensional data format, convert all row data at the same time point into a wide data format at one time point;

[0018] S202: According to the IOTDB data format, the data is combined into a batch data format, [time, key1], [time, key2]..., the time point is the same, and the number of key values ​​reaches 100,000;

[0019] S203: Due to the limit on the number of batch writes, select 10,000 Key data at a time for insertion, write in batches in a loop, modulo the total amount of data, obtain the total required batches, and call the IOTDB interface for writing.

[0020] Further, in step S201, the data format is: [time, Key=longitude+latitude+altitude, Value=temperature+wind force+wind direction].

[0021] Further, in step S202, IOTDB accesses iotdb-session in the reference package org.apache.iotdb using Java, and calls the following methods: createMultiTimeseries to create column information, and insertTablet to write data.

[0022] Furthermore, the specific steps of step three are as follows:

[0023] S301: When a new user queries data, the query condition that needs to be entered is [date, longitude, latitude, altitude] to obtain the meteorological data at the current geographical location and altitude;

[0024] S302: According to the user's query conditions, call the data query interface of IOTDB, enter the time, Key = longitude + latitude + altitude, return Value = temperature + wind speed + wind direction, and then parse the Value to obtain the values ​​of the three fields respectively.

[0025] Furthermore, in step S301, the date needs to be regularized to an integer time point, the longitude and latitude need to be regularized to a value of 0.25 degrees, and the altitude needs to be regularized to the altitude layer currently owned by the system.

[0026] Furthermore, the specific steps of step 4 are as follows:

[0027] S401: The amount of weather forecast data updated each time reaches hundreds of millions. The data continues to grow, and the database needs to save historical data for query. According to the growth of data volume, the server capacity is dynamically expanded without stopping the server.

[0028] S402: Add a service, copy the full IOTDB package, modify the contents of iotdb-common.properties and iotdb-datanode.properties, add the node to the centralized environment, and increase the overall data storage capacity.

[0029] Further, in step S201, the configuration of cluster_name in iotdb-common.properties is consistent with the existing cluster, the configuration of cn_target_config_node_list and dn_target_config_node_list is consistent with the existing cluster, the original data is not moved to the new node, and the newly created metadata partition and data partition exist in the new node.

[0030] The present invention provides a meteorological data processing method based on the IOTDB time series database, which has the following beneficial effects: using IOTDB can realize fast storage of data, and the storage per second can reach the million level, which significantly improves the processing speed of meteorological data. At the same time, the query is combined into a Key according to the query conditions, and the data is returned quickly. In the case of growing data volume, IOTDB is a distributed database, and only nodes need to be added to complete data storage expansion, meeting the needs of large-scale meteorological data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flowchart of the steps of the meteorological data processing method based on the IOTDB time series database of the present invention;

[0032] Figure 2 This is a functional diagram of the meteorological data processing method based on the IOTDB time series database of the present invention;

[0033] Figure 3 This is a flow chart of the meteorological data processing method based on the IOTDB time series database of the present invention;

[0034] Figure 4 The present invention provides a flowchart for an implementation example of the meteorological data processing method based on the IOTDB time series database. DETAILED DESCRIPTION

[0035] See also Figure 1 The present invention provides a technical solution: a meteorological data processing method based on an IOTDB time series database, comprising the following steps:

[0036] Step 1: Load NOAA weather data and analyze it to form two-dimensional data.

[0037] Step 2: Convert the two-dimensional data into a time series data storage format and write it into IOTDB data.

[0038] Step 3: Convert the query statement into the time series database query format for quick query.

[0039] Step 4: When the amount of data is too large, quickly expand the IOTDB nodes to meet the needs of data growth.

[0040] Specifically, the specific steps of step one are as follows:

[0041] S101: Download the forecast data for the next 15 days from the NOAA website, select the longitude and latitude range to query, select 0.25° for data granularity, and select the following enumeration values ​​for altitude layer: lev_200_mb,lev_250_mb,lev_300_mb,lev_350_mb,lev_400_mb,lev_450_mb,lev_500_mb,lev_550_mb,lev_600_mb,lev_650_mb,lev_700_mb,lev_750_mb,lev_800_mb,lev_850_mb,lev_900_mb,lev_950_mb,lev_1000_mb, and obtain the parameters var_TMP, var_UGRD, var_VGRD, which represent the vector values ​​of temperature, wind force and wind direction;

[0042] S102: According to the selection conditions, the JavaNIO package is used to download NOAA raw data and obtain the corresponding time series data packets. The first 120 hours are one data packet per hour, and the next 240 hours are one data packet per three hours. The name of the data packet download is gfs.t18z.pgrb2.0p25.f001 type. After downloading, the data is saved in the local disk;

[0043] S103: UCAR using Java GRIB Package Parse the downloaded GRIB-2 format data to obtain plain text data, and then format the parsed data in two dimensions according to the data download conditions. The resulting data format is [time, longitude, latitude, altitude, temperature, wind force, wind direction].

[0044] Specifically, in step S101, the var_TMP temperature needs to be converted into Celsius, var_UGRD and var_VGRD are vector data, and Math.sqrt and Math.round need to be used to substitute the two parameters to obtain the wind force in knots and the wind direction in 360 degrees.

[0045] Specifically, the specific steps of step 2 are as follows:

[0046] S201: For a two-dimensional data format, convert all row data at the same time point into a wide data format at one time point;

[0047] S202: According to the IOTDB data format, the data is combined into a batch data format, [time, key1], [time, key2]..., the time point is the same, and the number of key values ​​reaches 100,000;

[0048] S203: Due to the limit on the number of batch writes, select 10,000 Key data at a time for insertion, write in batches in a loop, modulo the total amount of data, obtain the total required batches, and call the IOTDB interface for writing.

[0049] Specifically, in step S201, the data format is: [time, Key=longitude+latitude+altitude, Value=temperature+wind force+wind direction].

[0050] Specifically, in step S202, IOTDB accesses iotdb-session in the reference package org.apache.iotdb using Java, and calls the following methods: createMultiTimeseries creates column information, and insertTablet writes data.

[0051] Specifically, the specific steps of step three are as follows:

[0052] S301: When a new user queries data, the query condition that needs to be entered is [date, longitude, latitude, altitude] to obtain the meteorological data at the current geographical location and altitude;

[0053] S302: According to the user's query conditions, call the data query interface of IOTDB, enter the time, Key = longitude + latitude + altitude, return Value = temperature + wind speed + wind direction, and then parse the Value to obtain the values ​​of the three fields respectively.

[0054] Specifically, in step S301, the date needs to be regularized to an integer time point, the longitude and latitude need to be regularized to a value of 0.25 degrees, and the altitude needs to be regularized to the altitude layer currently owned by the system.

[0055] Specifically, the specific steps of step 4 are as follows:

[0056] S401: The amount of weather forecast data updated each time reaches hundreds of millions. The data continues to grow, and the database needs to save historical data for query. According to the growth of data volume, the server capacity is dynamically expanded without stopping the server.

[0057] S402: Add a service, copy the full IOTDB package, modify the contents of iotdb-common.properties and iotdb-datanode.properties, add the node to the centralized environment, and increase the overall data storage capacity.

[0058] Specifically, in step S201, the configuration of cluster_name in iotdb-common.properties is consistent with the existing cluster, the configuration of cn_target_config_node_list and dn_target_config_node_list are consistent with the existing cluster, the original data is not moved to the new node, and the newly created metadata partition and data partition have new nodes.

[0059] The method of the embodiment is used for detection and analysis, and compared with the prior art. It can be concluded that the use of IOTDB can realize rapid storage of data, and the storage per second can reach millions, which significantly improves the processing speed of meteorological data. At the same time, the query is combined into a Key according to the query conditions, and the data is returned quickly. In the case of growing data volume, IOTDB is a distributed database, and only nodes need to be added to complete data storage expansion, meeting the needs of large-scale meteorological data processing.

[0060] The present invention provides a meteorological data processing method based on IOTDB time series database, comprising the following steps:

[0061] Step 1: Get the temperature and wind direction from the meteorological data based on the coordinates and altitude of the flight waypoint. The specific steps are as follows:

[0062] Step 101: Get the current flight status, including waypoints, longitude and latitude coordinates, and altitude layer.

[0063] Step 102: normalize the waypoint coordinates to 0.25 degree longitude and latitude values, normalize the altitude layer to the altitude layer obtained from the data in the air pressure layer, and the time is approximately equal to an integer of one hour.

[0064] Step 103: According to the regularized parameters: longitude, latitude, time, altitude, convert them into [time, longitude and latitude_latitude_altitude] as the query condition of IOTDB, and obtain the value of Value according to the interface.

[0065] Step 104: Get the Value value, which is of String type, separated by commas, and in the format of [temperature, wind force, wind direction].

[0066] Step 105: Assign the value of the array to the weather object and return it to the calling end.

[0067] Step 2: The waypoint calculates the time and fuel consumption required for the section based on the weather data.

[0068] Step 201: The meteorological data corresponding to the current latitude, longitude and altitude of the aircraft has been obtained.

[0069] Step 202: Acquire the performance data of the current aircraft.

[0070] Step 203: According to the flight direction of the current aircraft in the flight plan and the current wind direction, the direction of the force acting on the current aircraft at the current altitude is calculated, and then the speed of the current aircraft relative to the ground is determined according to the wind force.

[0071] Step 204: Calculate the actual flight speed of the aircraft based on the aircraft performance data combined with the meteorological data, and calculate the fuel consumption and flight time required for the waypoint.

[0072] Step 205: The flight time and fuel consumption of all waypoints are added together to obtain the required fuel volume and flight time for the route.

[0073] Step 3: Calculate the turbulence index of the waypoint. The specific steps are as follows:

[0074] Step 301: Obtain the latitude and longitude coordinates and flight altitude layer of the waypoint.

[0075] Step 302: Calculate the altitude layer plus or minus 2000 feet of the flight altitude layer, and determine the closest pressure layer above and below based on the addition and subtraction results, which cannot be consistent with the current layer.

[0076] Step 303: Calculate the current turbulence index based on the meteorological data, wind force and wind direction obtained at 2000 feet above and below.

[0077] Step 4: Based on the calculated data results, the flight plan report displays the wind speed and direction, temperature, and turbulence index of each waypoint. Users can intuitively see the weather conditions of each waypoint during a flight plan.

[0078] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. The meteorological data processing method based on IOTDB time series database is characterized by: The following steps are involved: S1: Download NOAA weather data and generate two-dimensional data after parsing; S2: Convert the two-dimensional data into a time series data storage format and write it into IOTDB data; S3: Converts query statements into time series database query format for fast query; S4: When the amount of data is too large, quickly expand the IOTDB nodes to meet the needs of data growth.

2. According to claim 1, the meteorological data processing method based on IOTDB time series database is characterized in that: The specific steps of step S1 are as follows: S101: Download the forecast data for the next 15 days from the NOAA website, select the longitude and latitude range to query, select 0.25° for data granularity, and select the following enumeration values ​​for altitude layer: lev_200_mb,lev_250_mb,lev_300_mb,lev_350_mb,lev_400_mb,lev_450_mb,lev_500_mb,lev_550_mb,lev_600_mb,lev_650_mb,lev_700_mb,lev_750_mb,lev_800_mb,lev_850_mb,lev_900_mb,lev_950_mb,lev_1000_mb, and obtain the parameters var_TMP, var_UGRD, var_VGRD, which represent the vector values ​​of temperature, wind force and wind direction; S102: According to the selection conditions, the JavaNIO package is used to download NOAA raw data and obtain the corresponding time series data packets. The first 120 hours are one data packet per hour, and the next 240 hours are one data packet per three hours. The name of the data packet download is gfs.t18z.pgrb2.0p25.f001 type. After downloading, the data is saved in the local disk; S103: UCAR using Java GRIB Package Parse the downloaded GRIB-2 format data to obtain plain text data, and then format the parsed data in two dimensions according to the data download conditions. The resulting data format is [time, longitude, latitude, altitude, temperature, wind force, wind direction].

3. According to claim 2, the meteorological data processing method based on IOTDB time series database is characterized in that: In step S101, the var_TMP temperature needs to be converted to Celsius. var_UGRD and var_VGRD are vector data. Math.sqrt and Math.round need to be used to substitute the two parameters to obtain the wind force in knots and the wind direction in 360 degrees.

4. The meteorological data processing method based on IOTDB time series database according to claim 1 is characterized in that: The specific steps of step S2 are as follows: S201: For a two-dimensional data format, convert all row data at the same time point into a wide data format at one time point; S202: According to the IOTDB data format, the data is combined into a batch data format, [time, key1], [time, key2]..., the time point is the same, and the number of key values ​​reaches 100,000; S203: Due to the limit on the number of batch writes, select 10,000 Key data at a time for insertion, write in batches in a loop, modulo the total amount of data, obtain the total required batches, and call the IOTDB interface for writing.

5. According to claim 4, the meteorological data processing method based on IOTDB time series database is characterized in that: In step S201, the data format is: [time, Key=longitude+latitude+altitude, Value=temperature+wind force+wind direction].

6. According to claim 4, the meteorological data processing method based on IOTDB time series database is characterized in that: In step S202, IOTDB accesses iotdb-session in the reference package org.apache.iotdb using Java, and calls the following methods: createMultiTimeseries creates column information, and insertTablet writes data.

7. The meteorological data processing method based on IOTDB time series database according to claim 1 is characterized in that: The specific steps of step S3 are as follows: S301: When a new user queries data, the query condition that needs to be entered is [date, longitude, latitude, altitude] to obtain the meteorological data at the current geographical location and altitude; S302: According to the user's query conditions, call the data query interface of IOTDB, enter the time, Key = longitude + latitude + altitude, return Value = temperature + wind speed + wind direction, and then parse the Value to obtain the values ​​of the three fields respectively.

8. The meteorological data processing method based on IOTDB time series database according to claim 7 is characterized in that: In step S301, the date needs to be regularized to an integer time point, the longitude and latitude need to be regularized to a value of 0.25 degrees, and the altitude needs to be regularized to the altitude layer currently owned by the system.

9. The meteorological data processing method based on IOTDB time series database according to claim 1 is characterized in that: The specific steps of step S4 are as follows: S401: The amount of weather forecast data updated each time reaches hundreds of millions. The data continues to grow, and the database needs to save historical data for query. According to the growth of data volume, the server capacity is dynamically expanded without stopping the server. S402: Add a service, copy the full IOTDB package, modify the contents of iotdb-common.properties and iotdb-datanode.properties, add the node to the centralized environment, and increase the overall data storage capacity.

10. The meteorological data processing method based on IOTDB time series database according to claim 9 is characterized in that: In step S402, the configuration of cluster_name in iotdb-common.properties is consistent with the existing cluster, the configuration of cn_target_config_node_list and dn_target_config_node_list is consistent with the existing cluster, the original data is not moved to the new node, and the newly created metadata partition and data partition exist in the new node.

Images