An intelligent meteorological element query platform
Through the intelligent meteorological factor query platform, springboot2.x+vue3.0 and Python3 technology are adopted, combined with the kudu library and MySQL database, the problem of inconvenience in obtaining data on the meteorological platform is solved, and efficient integration and query of meteorological factor data is realized, which meets the needs of the Advanced meteorological service needs.
Patent Information
- Application Number
- CN202110210925.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-02-25
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-02-25
AI Technical Summary
The existing meteorological platforms have shortcomings in structural design and functions, which leads to inconvenience in obtaining and querying meteorological factor data, requiring a lot of manpower and material resources, and cannot meet the needs of advanced meteorological services.
An intelligent meteorological element query platform was designed, using the system architecture of springboot2.x+vue3.0, combined with Python3 technology, and implementing back-end data processing and data display through Java. The business product library uses kudu library and MySQL database to support data collection. , storage and analysis, provide an MPP computing engine and FLINK stream computing based on memory computing, realizing rapid data extraction and batch export.
It realizes efficient integration, processing and query of meteorological factor data, meets the query needs of historical data in multiple regions, provides complete database query and statistical functions, is good inapplicability, and meets the public's needs for personalized meteorological services.
Smart Images

Figure CN113010751B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a meteorological platform, and in particular to an intelligent meteorological element query platform. Background Art
[0002] In recent years, with the support of the mountain torrent project, the Central Meteorological Observatory has continuously improved the monitoring, forecasting and warning business system, and gradually built a business support platform of "two networks and three platforms" including the MICAPS platform, SWAN platform, forecasting business integration platform, and the Northwest Regional Guidance Forecasting Business Product Network and the Northwest Regional Numerical Forecasting Product Network. With the development of social economy and the advancement of the modernization of meteorological business, the public has an increasingly high demand for personalized meteorological services, and meteorological disaster prevention and mitigation has put forward higher and higher requirements for the improvement of the monitoring, forecasting and warning capabilities of disastrous weather and meteorological disasters. Summary of the invention
[0003] The technical problem to be solved by the present invention is that the existing meteorological platform has deficiencies in structural design and function, which makes it inconvenient to obtain and query actual meteorological element data, and requires too much manpower and material resources, which does not meet current needs.
[0004] In order to solve the above technical problems, the technical solution provided by the present invention is: an intelligent meteorological element query platform, including system architecture, business product library, database, business product processing and query module, basic data product drawing module, the system architecture is mainly composed of springboot2.x+vue3.0, combined with Python3 as a supplement, and the background data processing, quality control, screening and interface service are realized through Java technology, Vue realizes the functions of client data display, call and download, and Python technology mainly realizes the graphical product production module; the business product library is realized by combining kudu library and MySQL database, kudu library mainly stores all historical element data, realizes fast data extraction and batch export; MySQL mainly stores precipitation, temperature and disastrous weather product data; The business product library is divided into the following architectural layers: data collection layer, data storage layer, data analysis and computing layer, and data service layer. The data collection layer is responsible for aggregating data from various data sources. The platform supports flume, sqoop, and streamsets collection. The data flow mainly uses the xxljob platform, and the data collection will be migrated to streamsets in the later stage. The data storage layer platform supports three storage methods: HDFS, HBASE, and KUDU, and supports distributed storage of massive data. The data analysis and computing layer provides the MPP computing engine IMPALA, SPARK computing algorithm, and FLINK stream computing based on memory computing. The data service layer relies on the configuration management background to realize data external services. In the later stage, the data service module will be integrated into the CDH platform to realize the unified planning and release of data.
[0005] The database uses MySQL database as the storage carrier for all table structures, storing precipitation, temperature, and severe weather product data;
[0006] The business product processing and query module processes the basic data of meteorological elements such as temperature, precipitation, snowfall, wind speed, wind direction, air pressure, and relative humidity based on the business product database data;
[0007] The basic data product drawing module adopts inverse distance weighted interpolation (IDW) to achieve the displacement of each pixel of the image according to the given control point pair and the displacement vector (direction and distance) of the control point. The inverse distance weighted interpolation method is to obtain the approximate relationship between each pixel and the selected control point pair, as well as the corresponding weight relationship, to obtain the corresponding change relationship of the pixel point. The approximation function can be understood as the degree of influence on the pixel point p, and the weight function can be regarded as the weight of the distance. The farther the distance, the smaller the weight.
[0008]
[0009] The function f(p) takes in the coordinates of a pixel point and calculates the interpolation through the selected control points. The f function returns the coordinates of the pixel point, and the fi function is an approximation function, that is,
[0010] f i (p) = q i +T i (pp i );
[0011] Here, I did not calculate and solve the equations for T. T is a 2*2 matrix, and T can be approximated to 1. Substituting the control point data into each local interpolation function, the wi weight function is understood as the farther the distance, the smaller the weight.
[0012]
[0013]
[0014] Here d(p, pi) is the distance between p and pi, the exponent μ can be a number > 0, and 2 is the optimal value in the test here. The weight function is obtained through the formula, and the sum reflects the joint influence of multiple groups of control points.
[0015] Compared with the prior art, the advantages of the present invention are: integrating historical data of multiple regions within the coverage of the meteorological station's business, establishing a basic data product library, basic data processing, and building functional modules such as database query, statistics, and mapping covering conventional meteorological elements and disastrous weather. It has good applicability, complete functions, and meets usage requirements.
[0016] As an improvement, the data collection layer is divided into two modules: platform internal data and external aggregation data. The external aggregation data module uses the xxljob system to collect and aggregate external data into HDFS.
[0017] As an improvement, the data of the external data aggregation module is divided into structured data and unstructured data. The structured data is stored in txt and parquet files supported by the HDFS underlying storage. Since structured data needs to be synchronized incrementally, the data is incrementally collected and synchronized through online append of txt, and table partitions are made for large tables. For unstructured data, the following two solutions are provided for file data:
[0018] (1) With the help of the scheduled scheduling function of xxljob, the scheduled file merging operation is realized, and small files are merged into large files through the MR program. The file size is controlled by JSON parameters;
[0019] (2) With the help of ES (Elasticsearch) high-speed query hit efficiency and combined with Hbase's batch storage solution, files are transcoded and stored to achieve the storage and use of small files.
[0020] As an improvement, the platform's internal data module processes the internal circulation data of the data warehouse after stratification. The current data warehouse design is divided into two layers, the ODS cache layer and the DW data integration layer. The ODS layer retains all the characteristics of the source data, and the DW data layer is designed using the KUDU+IMPALA storage method. The data columns and values of the ODS layer are refined, split, and converted.
[0021] As an improvement, the data flow of the internal data module of the platform is queried by calling the IMPALA interface, and the result processing flow is transferred to the KUDU database.
[0022] As an improvement, the precipitation products of the business product processing and query module have a total of 8 modules and 61 sub-modules, including sub-functional units such as cumulative precipitation, precipitation anomaly percentage and daily anomaly, regional precipitation product statistics, precipitation extremes and percentiles, and climate data statistics, including the following two calculation standards:
[0023] (1) The calculation standard for soaking rain is that the daily (20:00-20:00) precipitation in the Hexi region is ≥5.0 mm; in the Hedong region, the daily precipitation is ≥10 mm or the precipitation in two days is ≥15.0 mm. If the rainfall in two days reaches the soaking rain standard, the first day of the first soaking rain is regarded as the first day of soaking rain, and the second day of the last soaking rain is regarded as the last day of soaking rain.
[0024] (2) The calculation standard for the rainy season is to take the ten-day period as the unit, and the period when the relative coefficient of precipitation in the ten-day period is ≥ 2 is considered the rainy season. The specific steps are as follows:
[0025] S1. The day with the largest rainfall in the 15 days when Kb≥2 is determined as the pending day for the start of the rainy season. If Kb≥2 occurs again in the following 15 days, the pending day for the start of the rainy season is determined as the start of the rainy season, and the pentad of the start of the rainy season is the start of the rainy season pentad.
[0026] S2. If Kb≥2 does not occur again within the next 15 days, repeat the steps in S1 to re-determine the pending date for the start of the rainy season and the start date of the rainy season;
[0027] S3. If the rainy season start date is calculated, calculate the date that meets the rainy season start date conditions day by day according to step S1.
[0028] As an improvement, the temperature products in the business product processing and query module include the following temperature algorithm formulas:
[0029] (1) The daily change in average temperature is calculated from the difference between the average temperature of the day and the average temperature of the previous day. The calculation formula is:
[0030] T c1 =T0-T1,
[0031] Where Tc1 is the daily change in average temperature, T0 is the average temperature of the day, and T1 is the average temperature of the previous day;
[0032] (2) The 2-day change in average temperature is calculated from the difference between the average temperature of the day and the average temperature of the previous two days. The calculation formula is:
[0033] T c2 =T0-T2,
[0034] Where Tc2 is the average temperature change over 2 days, T0 is the average temperature of the day, and T2 is the average temperature of the previous two days;
[0035] (3) Anomaly of mean temperature of a pentad / ten-day period / month / season. The anomaly of mean temperature of a pentad / ten-day period / month / season is the difference between the mean temperature of the pentad / ten-day period / month / season and the average of the mean temperature of the pentad / ten-day period / month / season in all previous years. The calculation formula is:
[0036]
[0037] Where xi represents the historical data of the i-th year, i takes values of 1, 2, ..., n, n represents the number of years of data, X represents the average temperature of the pentad / decade / month / season, and D represents the anomaly.
[0038] As an improvement, the wind direction products of the business product processing and query module include 17 wind directions: north (N), north northeast (NNE), northeast (NE), east northeast (ENE), east (E), east southeast (ESE), southeast (SE), south southeast (SSE), south (S), south southwest (SSW), southwest (SW), west southwest (WSW), west (W), west northwest (WNW), northwest (NW), north northwest (NNW), and calm wind; the frequency of wind direction in a certain season, ten-day period, month, and season in each direction is obtained by averaging the frequency of the season, ten-day period, month, and season in previous years, and the calculation formula is:
[0039]
[0040] In the formula, xi represents the frequency of the pentad, ten-day period, month or season in the i-th year, i takes the value of 1, 2, ..., n, n represents the number of years of data, and X represents the frequency of a pentad, ten-day period, month or season in the cumulative years.
[0041] As an improvement, the humidity products in the business product processing and query module include the following humidity algorithm formulas:
[0042] (1) Average relative humidity: The daily average relative humidity is calculated by averaging four timed values (02:00, 08:00, 14:00, and 20:00 Beijing time). The calculation formula is:
[0043]
[0044] Where ri represents four timing values, i takes values of 1, 2, 3, and 4, and R represents the daily average value;
[0045] (2) Maximum relative humidity is calculated from the maximum relative humidity of one hour at 1:00 a.m. to the maximum relative humidity of one hour at 24:00 a.m. on the same day. The calculation formula is:
[0046] R max =Max(R1, R2, R i ...R 24 ),
[0047] Where Rmax is the maximum daily relative humidity, R1, R2…Ri are the maximum hourly relative humidity in the i-th hour respectively;
[0048] (3) Minimum relative humidity is calculated from the minimum value of the one-hour minimum relative humidity at 1:00 a.m. to the one-hour minimum relative humidity at 24:00 a.m. on the same day. The calculation formula is:
[0049] R min =Min(R1, R2, R i ...R 24 ),
[0050] Where Rmin is the daily minimum temperature, R1, R2…Ri are the hourly minimum relative humidity in the ith hour. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 It is a system architecture diagram of an intelligent meteorological element query platform.
[0052] Figure 2 This is a diagram of the database architecture.
[0053] Figure 3 It is an algorithm flow chart of the regional rainstorm algorithm of the business product processing and query module in the first embodiment.
[0054] Figure 4 It is an algorithm flow chart of the regional rainstorm algorithm of the business product processing and query module in the second embodiment. DETAILED DESCRIPTION
[0055] The present invention is further described in detail below in conjunction with the accompanying drawings.
[0056] In the specific implementation of the present invention, an intelligent meteorological element query platform includes a system architecture, a business product library, a database, a business product processing and query module, and a basic data product drawing module. The system architecture is mainly composed of springboot2.x+vue3.0, combined with Python3 as a supplement. Java technology is used to realize background data processing, quality control, screening and interface service provision. Vue realizes the functions of client data display, call and download. Python technology mainly realizes the graphical product production module; the business product library is realized by combining kudu library and MySQL database. Kudu library mainly stores all historical element data to realize fast data extraction and batch export; MySQL mainly stores precipitation, temperature, and disastrous weather product data; the business product library is in the framework The structure is divided into: data collection layer, data storage layer, data analysis and computing layer and data service layer. The data collection layer is responsible for gathering data from various data sources. The platform supports flume, sqoop and streamsets collection. The data flow mainly uses the xxljob platform. The data collection will be migrated to streamsets in the later stage. The data storage layer platform supports three storage methods: HDFS, HBASE and KUDU, and supports distributed storage of massive data. The data analysis and computing layer provides the MPP computing engine IMPALA, SPARK computing algorithm and FLINK stream computing based on memory computing. The data service layer relies on the configuration management background to realize data external services. The data service module will be integrated into the CDH platform in the later stage to realize the unified planning and release of data.
[0057] The database uses MySQL database as the storage carrier of all table structures to store precipitation, temperature, and severe weather product data;
[0058] The business product processing and query module processes the basic data of meteorological elements such as temperature, precipitation, snowfall, wind speed, wind direction, air pressure, and relative humidity based on the business product database data;
[0059] The basic data product drawing module adopts inverse distance weighted interpolation (IDW) to realize the displacement of each pixel point of the image according to the given control point pair and the displacement vector (direction and distance) of the control point. The inverse distance weighted interpolation method is to obtain the approximate relationship between each pixel point and the selected control point pair, and the corresponding weight relationship, and obtain the corresponding change relationship of the pixel point. The approximation function can be understood as the degree of influence on the pixel point p, and the weight function can be regarded as the weight of the distance. The farther the distance, the smaller the weight.
[0060]
[0061] The function f(p) takes in the coordinates of a pixel point and calculates the interpolation through the selected control points. The f function returns the coordinates of the pixel point, and the fi function is an approximation function, that is,
[0062] f i (p) = q i +T i (pp i );
[0063] Here, I did not calculate and solve the equations for T. T is a 2*2 matrix, and T can be approximated to 1. Substituting the control point data into each local interpolation function, the wi weight function is understood as the farther the distance, the smaller the weight.
[0064]
[0065]
[0066] Here d(p, pi) is the distance between p and pi, the exponent μ can be a number > 0, and 2 is the optimal value in the test here. The weight function is obtained through the formula, and the sum reflects the joint influence of multiple groups of control points.
[0067] The data collection layer is divided into two modules: platform internal data and external aggregated data. The external aggregated data module uses the xxljob system to collect and aggregate external data into HDFS.
[0068] The data in the external data aggregation module is divided into structured data and unstructured data. The structured data is stored in txt and parquet files supported by the HDFS underlying storage. Since structured data needs to be synchronized incrementally, the data is incrementally collected and synchronized through txt online appending, and table partitioning is performed for large tables. For unstructured data, the following two solutions are provided for file data:
[0069] (1) With the help of the scheduled scheduling function of xxljob, the scheduled file merging operation is realized, and small files are merged into large files through the MR program. The file size is controlled by JSON parameters;
[0070] (2) With the help of ES (Elasticsearch) high-speed query hit efficiency and combined with Hbase's batch storage solution, files are transcoded and stored to achieve the storage and use of small files.
[0071] The platform's internal data module processes the internal circulation data of the data warehouse after stratification. The current data warehouse design is divided into two layers, the ODS cache layer and the DW data integration layer. The ODS layer retains all the characteristics of the source data, and the DW data layer is designed using the KUDU+IMPALA storage method. The data columns and values of the ODS layer are refined, split, and converted.
[0072] The data flow of the internal data module of the platform is transferred by calling the IMPALA interface to query the data, and the result processing flow is transferred to the KUDU database.
[0073] The precipitation products of the business product processing and query module have a total of 8 modules and 61 sub-modules, including sub-functional units such as cumulative precipitation, precipitation anomaly percentage and daily anomaly, regional precipitation product statistics, precipitation extremes and percentiles, and climate data statistics, including the following two calculation standards:
[0074] (1) The calculation standard for soaking rain is that the daily (20:00-20:00) precipitation in the Hexi region is ≥5.0 mm; in the Hedong region, the daily precipitation is ≥10 mm or the precipitation in two days is ≥15.0 mm. If the rainfall in two days reaches the soaking rain standard, the first day of the first soaking rain is regarded as the first day of soaking rain, and the second day of the last soaking rain is regarded as the last day of soaking rain.
[0075] (2) The calculation standard for the rainy season is to take the ten-day period as the unit, and the period when the relative coefficient of precipitation in the ten-day period is ≥ 2 is considered the rainy season. The specific steps are as follows:
[0076] S1. The day with the largest rainfall in the 15 days when Kb≥2 is determined as the pending day for the start of the rainy season. If Kb≥2 occurs again in the following 15 days, the pending day for the start of the rainy season is determined as the start of the rainy season, and the pentad of the start of the rainy season is the start of the rainy season pentad.
[0077] S2. If Kb≥2 does not occur again within the next 15 days, repeat the steps in S1 to re-determine the pending date for the start of the rainy season and the start date of the rainy season;
[0078] S3. If the rainy season start date is calculated, calculate the date that meets the rainy season start date conditions day by day according to step S1.
[0079] The temperature products of the business product processing and query module include the following temperature algorithm formulas:
[0080] (1) The daily change in average temperature is calculated from the difference between the average temperature of the day and the average temperature of the previous day. The calculation formula is:
[0081] T c1 =T0-T1,
[0082] Where Tc1 is the daily change in average temperature, T0 is the average temperature of the day, and T1 is the average temperature of the previous day;
[0083] (2) The 2-day change in average temperature is calculated from the difference between the average temperature of the day and the average temperature of the previous two days. The calculation formula is:
[0084] T c2 =T0-T2,
[0085] Where Tc2 is the average temperature change over 2 days, T0 is the average temperature of the day, and T2 is the average temperature of the previous two days;
[0086] (3) Anomaly of mean temperature of a pentad / ten-day period / month / season. The anomaly of mean temperature of a pentad / ten-day period / month / season is the difference between the mean temperature of the pentad / ten-day period / month / season and the average of the mean temperature of the pentad / ten-day period / month / season in all previous years. The calculation formula is:
[0087]
[0088] Where xi represents the historical data of the i-th year, i takes values of 1, 2, ..., n, n represents the number of years of data, X represents the average temperature of the pentad / decade / month / season, and D represents the anomaly.
[0089] The wind direction product of the business product processing and query module includes 17 wind directions: north (N), north northeast (NNE), northeast (NE), east northeast (ENE), east (E), east southeast (ESE), southeast (SE), south southeast (SSE), south (S), south southwest (SSW), southwest (SW), west southwest (WSW), west (W), west northwest (WNW), northwest (NW), north northwest (NNW), and calm wind; the frequency of wind direction in a certain season, ten-day period, month, and season in each direction is obtained by averaging the frequency of the season, ten-day period, month, and season in previous years, and the calculation formula is:
[0090]
[0091] In the formula, xi represents the frequency of the pentad, ten-day period, month or season in the i-th year, i takes the value of 1, 2, ..., n, n represents the number of years of data, and X represents the frequency of a pentad, ten-day period, month or season in the cumulative years.
[0092] The humidity product of the business product processing and query module includes the following humidity algorithm formulas:
[0093] (1) Average relative humidity: The daily average relative humidity is calculated by averaging four timed values (02:00, 08:00, 14:00, and 20:00 Beijing time). The calculation formula is:
[0094]
[0095] Where ri represents four timing values, i takes values of 1, 2, 3, and 4, and R represents the daily average value;
[0096] (2) Maximum relative humidity is calculated from the maximum relative humidity of one hour at 1:00 a.m. to the maximum relative humidity of one hour at 24:00 a.m. on the same day. The calculation formula is:
[0097] R max =Max(R1, R2, R i ...R 24 ),
[0098] Where Rmax is the maximum daily relative humidity, R1, R2…Ri are the maximum hourly relative humidity in the i-th hour respectively;
[0099] (3) Minimum relative humidity is calculated from the minimum value of the one-hour minimum relative humidity at 1:00 a.m. to the one-hour minimum relative humidity at 24:00 a.m. on the same day. The calculation formula is:
[0100] R min =Min(R1, R2, R i ...R 24 ),
[0101] Where Rmin is the daily minimum temperature, R1, R2…Ri are the hourly minimum relative humidity in the ith hour.
[0102] Working principle of the invention: The technical framework of this system development is mainly based on springboot2.x+vue3.0, combined with Python3 as a supplement, and uses Java technology to realize background data processing, quality control, screening and provide interface services. Vue realizes client data display, call, download and other functions. Python technology mainly realizes the graphical product production module.
[0103] 1. Business Product Library
[0104] The business product library is implemented by combining the Kudu library and the MySQL database. The Kudu library mainly stores all historical element data, and realizes technologies such as fast data extraction and batch export. MySQL mainly stores precipitation, temperature, and disastrous weather product data. According to the continuous growth of data volume in departmental business systems over the years, data processing bottlenecks are becoming more and more obvious. Most of the existing service systems have the problem of inefficient data processing. To address this problem, the big data platform has built and created a data warehouse architecture to support the use of systems with large amounts of data services. The database architecture is divided into the following layers: data collection layer, data storage layer, data analysis and computing layer, data service layer, etc.
[0105] Data collection layer: responsible for aggregating data from various data sources. The platform supports flume, sqoop, and streamsets collection. Currently, data flow mainly uses the self-developed xxljob platform. In the future, it is planned to migrate the entire data collection to streamsets.
[0106] Data storage layer: The platform supports multiple storage methods such as HDFS, HBASE, KUDU, and supports distributed storage of massive data.
[0107] Data analysis layer: The data warehouse system provides the MPP computing engine IMPALA based on in-memory computing, as well as the SPARK computing algorithm and FLINK stream computing.
[0108] Data service layer: Currently, the data service module mainly relies on the self-developed configuration management background to realize external data services. In the future, it is planned to integrate the data service module into the CDH platform to realize unified planning and release of data.
[0109] (1) The data collection layer is divided into two modules: platform internal data and external aggregated data.
[0110] External data: Currently, the xxljob system is used to collect and aggregate external data into HDFS. The data is divided into structured data and unstructured data.
[0111] Structured data: HDFS underlying storage supports txt and parquet file storage. Since structured data requires incremental data synchronization, we use txt online append to perform incremental data collection and synchronization, and partition large tables at the same time.
[0112] Unstructured data: Provides a variety of solutions for file-based data.
[0113] The first one: The HDFS system has a certain pressure on the storage of small files. When there are a lot of small files, the metadata query pressure of the cluster NAMENODE node will also increase synchronously. To solve this problem, with the help of the scheduled scheduling function of xxljob, the scheduled file merging operation is implemented, and the small files are merged into large files through the MR program. The file size is controlled by JSON parameters.
[0114] The second method: With the help of ES (Elasticsearch) high-speed query hit efficiency, combined with Hbase's batch storage solution, files are transcoded and stored to achieve storage and use of small files.
[0115] Internal data: Internal data mainly refers to the internal flow data of the data warehouse after stratification. The current data warehouse design is divided into two layers, the ODS cache layer and the DW data integration layer. The ODS layer retains all the characteristics of the source data and only performs basic data operations, such as misplaced values and garbled values. For the DW data layer, we adopt the KUDU+IMPALA storage method to design, and the data columns and values of the ODS layer are all subjected to data refinement, splitting, conversion and other operations. The internal data flow queries the data by calling the IMPALA interface, and the result processing flow is transferred to the KUDU database.
[0116] (2) Data processing and handling. For the data warehouse platform, a set of standardized and normalized data processing processes should be established, such as how to collect data, how to transfer data between data levels, how to clean, process and handle data through data standardization, etc. In view of such a data processing model, a complete data processing pipeline should be built to control the entire life cycle of data. At present, the solution uses the powerful built-in functions of flink and spark to customize job tasks according to needs, and then schedule them on a regular or quantitative basis.
[0117] 2. Database
[0118] All table structures of MySQL database are used for storage, mainly storing precipitation, temperature, and severe weather product data.
[0119] 3. Business product processing and query module
[0120] Based on the business product database data, the basic data of meteorological elements such as temperature, precipitation, snowfall, wind speed, wind direction, air pressure, relative humidity, etc. are processed. The precipitation product construction has a total of 8 modules and 61 sub-modules, including cumulative precipitation, precipitation anomaly percentage and daily anomaly, regional precipitation product statistics, precipitation extremes and percentiles, climate data statistics, etc., including:
[0121] (1) The calculation standard for soaking rain is that the daily (20:00-20:00) precipitation in the Hexi region is ≥5.0 mm; in the Hedong region, the daily precipitation is ≥10 mm or the precipitation in two days is ≥15.0 mm. If the rainfall in two days reaches the soaking rain standard, the first day of the first soaking rain is regarded as the first day of soaking rain, and the second day of the last soaking rain is regarded as the last day of soaking rain.
[0122] (2) The calculation standard for the rainy season is to take the ten-day period as the unit, and the period when the relative coefficient of precipitation in the ten-day period is ≥ 2 is considered the rainy season. The specific steps are as follows:
[0123] S1. The day with the largest rainfall in the 15 days when Kb≥2 is determined as the pending day for the start of the rainy season. If Kb≥2 occurs again in the following 15 days, the pending day for the start of the rainy season is determined as the start of the rainy season, and the pentad of the start of the rainy season is the start of the rainy season pentad.
[0124] S2. If Kb≥2 does not occur again within the next 15 days, repeat the steps in S1 to re-determine the pending date for the start of the rainy season and the start date of the rainy season;
[0125] S3. If the rainy season start date is calculated, calculate the date that meets the rainy season start date conditions day by day according to step S1.
[0126] The temperature products of the business product processing and query module include the following temperature algorithm formulas:
[0127] (1) The daily change in average temperature is calculated from the difference between the average temperature of the day and the average temperature of the previous day. The calculation formula is:
[0128] T c1 =T0-T1,
[0129] Where Tc1 is the daily change in average temperature, T0 is the average temperature of the day, and T1 is the average temperature of the previous day;
[0130] (2) The 2-day change in average temperature is calculated from the difference between the average temperature of the day and the average temperature of the previous two days. The calculation formula is:
[0131] T c2 =T0-T2,
[0132] Where Tc2 is the average temperature change over 2 days, T0 is the average temperature of the day, and T2 is the average temperature of the previous two days;
[0133] (3) Anomaly of mean temperature of a pentad / ten-day period / month / season. The anomaly of mean temperature of a pentad / ten-day period / month / season is the difference between the mean temperature of the pentad / ten-day period / month / season and the average of the mean temperature of the pentad / ten-day period / month / season in all previous years. The calculation formula is:
[0134]
[0135] Where xi represents the historical data of the i-th year, i takes values of 1, 2, ..., n, n represents the number of years of data, X represents the average temperature of the pentad / decade / month / season, and D represents the anomaly.
[0136] The wind direction product of the business product processing and query module includes 17 wind directions: north (N), north northeast (NNE), northeast (NE), east northeast (ENE), east (E), east southeast (ESE), southeast (SE), south southeast (SSE), south (S), south southwest (SSW), southwest (SW), west southwest (WSW), west (W), west northwest (WNW), northwest (NW), north northwest (NNW), and calm wind; the frequency of wind direction in a certain season, ten-day period, month, and season in each direction is obtained by averaging the frequency of the season, ten-day period, month, and season in previous years, and the calculation formula is:
[0137]
[0138] In the formula, xi represents the frequency of the pentad, ten-day period, month or season in the i-th year, i takes the value of 1, 2, ..., n, n represents the number of years of data, and X represents the frequency of a pentad, ten-day period, month or season in the cumulative years.
[0139] The humidity product of the business product processing and query module includes the following humidity algorithm formulas:
[0140] (1) Average relative humidity: The daily average relative humidity is calculated by averaging four timed values (02:00, 08:00, 14:00, and 20:00 Beijing time). The calculation formula is:
[0141]
[0142] Where ri represents four timing values, i takes values of 1, 2, 3, and 4, and R represents the daily average value;
[0143] (2) Maximum relative humidity is calculated from the maximum relative humidity of one hour at 1:00 a.m. to the maximum relative humidity of one hour at 24:00 a.m. on the same day. The calculation formula is:
[0144] R max =Max(R1, R2, R i ...R 24 ),
[0145] Where Rmax is the maximum daily relative humidity, R1, R2…Ri are the maximum hourly relative humidity in the i-th hour respectively;
[0146] (3) Minimum relative humidity is calculated from the minimum value of the one-hour minimum relative humidity at 1:00 a.m. to the one-hour minimum relative humidity at 24:00 a.m. on the same day. The calculation formula is:
[0147] R min =Min(R1, R2, Ri ...R 24 ),
[0148] Where Rmin is the daily minimum temperature, R1, R2…Ri are the hourly minimum relative humidity in the ith hour.
[0149] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features, and in the description of the present invention, "plurality" means two or more, unless otherwise clearly and specifically defined.
[0150] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0151] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is lower in level than the second feature.
[0152] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0153] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and intent of the present invention.
Claims
1. An intelligent meteorological element query platform, including a system architecture, a business product library, a database, a business product processing and query module, and a basic data product mapping module, characterized by: The system architecture is mainly based on springboot2.x+vue3.0, supplemented by Python3. It uses Java technology to implement background data processing, quality control, screening and provide interface services. Vue realizes the functions of client data display, call and download. Python technology mainly realizes the graphical product production module. The business product library is implemented by combining the kudu library and MySQL database. The kudu library mainly stores all historical factor data to achieve rapid data extraction and batch export. MySQL mainly stores precipitation, temperature, and severe weather product data. The business product library is divided into the following architectural layers: data collection layer, data storage layer, data analysis and calculation layer, and data service layer. The data collection layer is responsible for aggregating data from various data sources. The platform supports flume, sqoop, and streamsets collection. The data flow mainly uses the xxljob platform, and the data collection will be migrated to streamsets in the later stage. The data storage layer supports three storage methods: HDFS, HBASE, and KUDU, and supports distributed storage of massive data. The data analysis and computing layer provides the MPP computing engine IMPALA, SPARK computing algorithm, and FLINK stream computing based on memory computing. The data service layer relies on the configuration management background to realize external data services. In the later stage, the data service module will be integrated into the CDH platform to realize unified planning and release of data. The database uses MySQL database as the storage carrier of all table structures to store precipitation, temperature, and severe weather product data; The business product processing and query module processes the basic data of meteorological elements such as temperature, precipitation, snowfall, wind speed, wind direction, air pressure, and relative humidity based on the business product database data; The basic data product drawing module adopts inverse distance weighted interpolation to realize the displacement of each pixel point of the image according to the given control point pair and the direction and distance displacement vector of the control point. The inverse distance weighted interpolation method is to obtain the approximate relationship between each pixel point and the selected control point pair, as well as the corresponding weight relationship, to obtain the corresponding change relationship of the pixel point. The approximation function can be understood as the degree of influence on the pixel point p, and the weight function can be regarded as the weight of the distance. The farther the distance, the smaller the weight. The function f(p) takes in the coordinates of a pixel point and calculates the interpolation through the selected control points. The f function returns the coordinates of the pixel point, and the fi function is an approximation function, that is, f i (p)=q i T i (p-p i ); T is used to calculate and solve the equations. T is a 2*2 matrix. T takes an approximate value of 1. Substitute the control point data into each local interpolation function. The wi weight function is understood as the farther the distance, the smaller the weight. Here d(p, pi) is the distance between p and pi, the index μ is a number > 0, and the weight function is obtained by the formula, and the sum reflects the joint influence of multiple groups of control points; The data collection layer is divided into two modules: platform internal data and external aggregated data. The external aggregated data module uses the xxljob system to collect and aggregate external data into HDFS. The data in the external data aggregation module is divided into structured data and unstructured data. Structured data is stored in TXT and Parquet files supported by HDFS underlying storage. Since structured data requires incremental data synchronization, incremental data collection and synchronization is performed through TXT online appending, and table partitioning is performed for large tables. For unstructured data, the following two solutions are provided for file-based data: (1) With the help of the scheduled scheduling function of xxljob, the scheduled file merging operation is realized, and small files are merged into large files through the MR program. The file size is controlled by JSON parameters; (2) Leveraging ES's high-speed query hit efficiency and combining it with Hbase's batch storage solution, we can transcode and store files to achieve the storage and use of small files. The platform's internal data module processes the internal flow data of the data warehouse after stratification. The current data warehouse design is divided into two layers: the ODS cache layer and the DW data integration layer. The ODS layer retains all the characteristics of the source data, and the DW data layer is designed with the KUDU+IMPALA storage method. The data columns and values of the ODS layer are refined, split, and converted. The data flow of the internal data module of the platform is through calling the IMPALA interface to query data, and the result processing flow is transferred to the KUDU database.
2. The intelligent meteorological element query platform according to claim 1, characterized in that: The precipitation products of the business product processing and query module have a total of 8 modules and 61 submodules, including cumulative precipitation, precipitation anomaly percentage and daily anomaly, regional precipitation product statistics, precipitation extremes and percentiles, and climate data statistics sub-functional units, including the following two calculation standards: (1) The calculation standard for soaking rain is based on the daily precipitation between 20:00 and 20:00 in the Hexi region, with a daily precipitation ≥ 5.0 mm; in the Hedong region, the daily precipitation is ≥ 10 mm or the precipitation over two days is ≥ 15.0 mm. If the rainfall over two days reaches the soaking rain standard, the first day of the first soaking rain is regarded as the first day of soaking rain, and the second day of the last soaking rain is regarded as the last day of soaking rain. (2) The calculation standard for the rainy season is to use the ten-day period as the unit, and the period when the relative coefficient of precipitation in the ten-day period is ≥2 is considered the rainy season. The specific steps are as follows: S1. The day with the largest rainfall among the 15 days with Kb≥2 is determined as the undetermined start date of the rainy season. If Kb≥2 occurs again within the next 15 days, the undetermined start date of the rainy season is determined as the start date of the rainy season, and the pentad of the rainy season start date is the rainy season start pentad. S2. If Kb ≥ 2 does not occur within the next 15 days, repeat the steps in S1 and re-determine the pending rainy season start date and the rainy season start date; S3. If the rainy season start date is calculated, calculate the date that meets the rainy season start date conditions according to step S1 day by day.
3. The intelligent meteorological element query platform according to claim 1, characterized in that: The temperature products in the business product processing and query module include the following temperature algorithm formulas: (1) The daily change in average temperature is calculated from the difference between the average temperature of the day and the average temperature of the previous day. The calculation formula is: T c1 =T0-T1, Where Tc1 is the daily temperature change of the average temperature, T0 is the average temperature of the day, and T1 is the average temperature of the previous day; (2) The average temperature change over two days is calculated from the difference between the average temperature of the day and the average temperature of the previous two days. The calculation formula is: T c2 =T0-T2, Where Tc2 is the average temperature variation over 2 days, T0 is the average temperature of the day, and T2 is the average temperature of the previous two days; (3) Anomaly of mean temperature for a pentad / ten-day period / month / season. The anomaly of mean temperature for a pentad / ten-day period / month / season is the difference between the mean temperature for that pentad / ten-day period / month / season and the average mean temperature for that pentad / ten-day period / month / season in all previous years. The calculation formula is: Where xi represents the historical data of year i, i takes the value of 1, 2, ..., n, n represents the number of years of data, X represents the average temperature of the pentad / decadal / month / season, and D represents the anomaly.
4. The intelligent meteorological element query platform according to claim 1 is characterized in that The wind direction product of the business product processing and query module includes 17 wind directions: north, north-northeast, northeast, east-northeast, east, east-southeast, southeast, south-southeast, south, south-southwest, southwest, west-southwest, west, west-northwest, northwest, north-northwest, and calm wind. The frequency of wind direction in a certain season, ten-day period, month, or season in each direction is obtained by averaging the frequency of that season, ten-day period, month, or season in previous years. The calculation formula is: Where xi represents the frequency of the pentad, ten-day period, month, or season in the i-th year, i takes the value of 1, 2, ..., n, where n represents the number of years of data, and X represents the frequency of a pentad, ten-day period, month, or season in the cumulative years.
5. The intelligent meteorological element query platform according to claim 1, characterized in that: The humidity product of the business product processing and query module includes the following humidity algorithm formulas: (1) Average relative humidity. The daily average relative humidity is obtained by averaging four timed values. The calculation formula is: Where ri represents the four timed values, i takes the values 1, 2, 3, and 4, and R represents the daily average value; (2) Maximum relative humidity is calculated from the maximum relative humidity of one hour at 1:00 a.m. to the maximum relative humidity of one hour at 24:00 a.m. on the same day. The calculation formula is: R max =Max(R1,R2,R i ...R 24 ), Where Rmax is the maximum daily relative humidity, R1, R2…Ri are the maximum hourly relative humidity in hour i; (3) Minimum relative humidity is calculated from the minimum value of the one-hour minimum relative humidity at 1:00 a.m. to the one-hour minimum relative humidity at 24:00 a.m. on the same day. The calculation formula is: R min =Min(R1, R2, R i ...R 24 ), Where Rmin is the daily minimum temperature, R1, R2…Ri are the hourly minimum relative humidity in the ith hour.
