A Multi-Source Meteorological Data Integration and Quality Control System

Abstract

The present invention relates to the technical field of meteorological data integration control, and particularly relates to a multi-source meteorological data integration and quality control system, which includes a data receiving module, a data processing module, a data quality control module, a data integration module, a meteorological event prediction module, and an output module; among them, the data receiving module: responsible for receiving data from multiple meteorological data sources; the data processing module: preprocessing the received data; the data quality control module: checking the quality of the preprocessed data and making necessary quality corrections; the data integration module: integrating the data that has passed the quality control; the meteorological event prediction module: based on the integrated data, predicting short-term and medium-term meteorological events and providing corresponding reliability or probability assessments. The present invention realizes the efficient integration, standardization, and optimization of multi-source data, provides precise data quality control, and thereby enhances the prediction accuracy and timeliness of meteorological events.

Description

Technical Field

[0001] The present invention relates to the technical field of meteorological data integration control, and particularly to a multi-source meteorological data integration and quality control system. Background Art

[0002] Meteorological forecasting is a highly complex and public safety-related field. With the rapid development of meteorological observation technologies, various meteorological data sources, such as ground stations, meteorological radars, meteorological satellites, etc., can provide a large amount of real-time data. The types, formats, and accuracies of these data may vary greatly. How to effectively integrate these multi-source data and how to perform quality control have become key issues in meteorological forecasting.

[0003] Traditional meteorological data processing mainly relies on manual or semi-automated methods, and there are many challenges in integrating a large amount of heterogeneous data. In addition, simple data fusion methods may not be able to fully utilize the advantages of each data source. For data quality control, especially the processing of abnormal, duplicate, or missing data, traditional methods may introduce errors and affect the accuracy of forecasting.

[0004] With the rise of deep learning technologies, especially long short-term memory networks (LSTM), significant progress has been made in the field of time series forecasting. LSTM has the ability to handle long-term dependencies, making it have great potential in complex meteorological data forecasting.

[0005] In order to make full use of multi-source meteorological data and improve the accuracy of meteorological forecasting, a system that can effectively integrate various data, perform quality control, and support advanced forecasting algorithms is needed. Summary of the Invention

[0006] Based on the above purpose, the present invention provides a multi-source meteorological data integration and quality control system.

[0007] A multi-source meteorological data integration and quality control system includes a data reception module, a data processing module, a data quality control module, a data integration module, a meteorological event prediction module, and an output module; wherein,

[0008] The data reception module: responsible for receiving data from multiple meteorological data sources;

[0009] The data processing module: preprocesses the received data;

[0010] The data quality control module: examines the quality of the preprocessed data and performs necessary quality corrections;

[0011] The data integration module: integrates the data that has passed quality control;

[0012] Meteorological event prediction module: Based on the integrated data, predict short-term and medium-term meteorological events and provide corresponding confidence or probability assessments for this;

[0013] Output module: Output the integrated data and the meteorological event prediction results for user use.

[0014] Furthermore, the data receiving module includes interfaces with various meteorological data sources. The meteorological data sources include ground stations, meteorological radars, meteorological satellites, meteorological sounding balloons, and ocean buoys. Real-time data and historical data from the above data sources are received through the interfaces, and the interfaces support various communication protocols and high-concurrency data transmission.

[0015] Furthermore, the data processing module includes a data format conversion sub-module, a data interpolation sub-module, and a data cleaning sub-module. Specifically:

[0016] Data format conversion sub-module: Used to convert data from various sources into a unified standard format, supporting multiple meteorological data standards;

[0017] Data interpolation sub-module: Interpolate the data space as needed, using Kriging or bilinear interpolation to improve the data space resolution;

[0018] Data cleaning sub-module: Process abnormal data, duplicate data, and missing data. The processing method uses the median method or the average method for repair or elimination.

[0019] Furthermore, the Kriging method adopted by the data interpolation sub-module establishes a semi-variogram based on the spatial relationship of the data and performs data interpolation according to the distance and direction between stations; the bilinear interpolation method uses the values of the four adjacent points above, below, left, and right of the data point for interpolation, which is suitable for the spatial interpolation of grid data. Specifically,

[0020] The Kriging method adopted by the data interpolation sub-module is as follows:

[0021] Let Z(x i ) represent the observed value at position x i , and the predicted value Z * (x0) at position x0 is expressed as:

[0022] Among them, λ i is the weight, and N is the number of data points used for prediction;

[0023] The bilinear interpolation method is as follows:

[0024] If the four adjacent points are Q 11 ,Q 12 ,Q 21,Q 22 , the interpolation point is P, and its coordinates are (x, y). Then the value F(P) of bilinear interpolation is:

[0025]

[0026] Furthermore, when processing abnormal data, the data cleaning sub-module uses the median method to take the median value of multiple data points for repair to avoid the influence of extreme values on the results; the average method directly calculates the average value of multiple data points as the repair value; specifically,

[0027] Median method:

[0028] For a data set \(x - 1, x - 2, …, x - ∩\), if n is odd, the median is: M = x (n+1) / 2 If n is even, the median is:

[0029] Average method:

[0030] For a data set \(x - 1, x - 2, …, x - ∩\), its average value is:

[0031] Furthermore, the data quality control module will check and evaluate the data quality. This check and evaluation includes spatio-temporal consistency, statistical attributes of the data, physical rationality of the data, and data source verification. Specifically:

[0032] Spatio-temporal consistency: Check the timestamps and spatial coordinates of the data to ensure the continuity and consistency of the data in time and space;

[0033] Statistical attributes of the data: Analyze the statistical characteristics of the mean, variance, skewness, and kurtosis of the data, compare with historical data, and identify outliers;

[0034] Physical rationality of the data: Check whether the data conforms to physical laws to ensure the physical authenticity of the data.

[0035] Data source verification: Cross-verify the data from different data sources to ensure the authenticity and accuracy of the data.

[0036] Furthermore, the data integration module uses the method of data assimilation to integrate the data from multiple data sources. The data assimilation is in the form of Kalman filtering or 4D-VAR, combining the observed data and the output of the numerical model to optimize the initial conditions of the model.

[0037] Furthermore, the output module includes graphic output, tabular output, report output, and real-time notification functions. Specifically:

[0038] Graphical output: It can draw various meteorological graphs such as contour maps, color scale maps, and wind field maps according to user needs, and supports interactive viewing and zooming;

[0039] Table output: Organize data into a table, and users can filter, sort, and download according to time, location, or other conditions;

[0040] Report output: Automatically generate reports containing meteorological analysis, prediction trends, and meteorological disaster alerts. Reports are generated daily, weekly, or monthly and support email push;

[0041] Real-time notification function: When the integrated data reaches a predetermined condition, the system automatically sends real-time notifications to users.

[0042] Furthermore, the deep learning method adopted by the meteorological event prediction module automatically learns the occurrence patterns of meteorological events from training data. By analyzing the relationship between recent meteorological changes and historical data, it provides users with predictions of upcoming meteorological events, including the type, intensity, and possible occurrence time of the events.

[0043] Furthermore, the deep learning method uses a long short-term memory network model for prediction, and the long short-term memory network model includes the following formulas:

[0044] f t = σ(W f · [h t-1 , x t + b f )

[0045] i t = σ(W i · [h t-1 , x t + b i )

[0046]

[0047]

[0048] o t = σ(W o · [h t-1 , x t + b o )

[0049] h t = o t × tanh(C t )

[0050] where x t is the input vector at time t; h tis the hidden state at time t; C t is the cell state at time t; f t , i t , o t are the values of the forget gate, input gate, cell state, and output gate respectively; W and b are the learned weights and biases; σ is the Sigmoid activation function; × represents element-wise multiplication;

[0051] The long short-term memory network model is controlled by three gates to inflow, save, and outflow information from the cell state. The above formula describes how to update these gates, as well as the cell state and hidden state, at time point t. Through this structure, the long short-term memory network can capture and remember patterns in long time series.

[0052] Advantages of the present invention:

[0053] The present invention realizes the efficient access to various meteorological data sources, which means that various data from ground stations, weather radars to meteorological satellites can be captured by the system. The data processing module further ensures that all input data is converted into a unified standard format, eliminating problems caused by data heterogeneity. This integration and standardization strategy enables the system to maintain high efficiency and accuracy when processing large amounts of data from different sources.

[0054] The present invention, by adopting the median method, average method or model-based method, enables the system to intelligently repair or eliminate these problematic data. More advanced is that by combining deep learning techniques, especially the long short-term memory network (LSTM), the system provides a method to predict future meteorological events based on historical and current data. This comprehensive data processing strategy significantly improves the accuracy of meteorological prediction. Brief Description of the Drawings

[0055] To more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only those of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0056] Figure 1 is a schematic diagram of the data integration and quality control system according to an embodiment of the present invention;

[0057] Figure 2 is a schematic diagram of the data processing module according to an embodiment of the present invention;

[0058] Figure 3 is a schematic diagram of the output module according to an embodiment of the present invention. Detailed Embodiments

[0059] To make the objectives, technical solutions, and advantages of the present invention more clear and understandable, the present invention will be further described in detail below in conjunction with specific embodiments.

[0060] It should be noted that unless otherwise defined, the technical terms or scientific terms used in the present invention should have the ordinary meanings understood by those of ordinary skill in the field to which the present invention pertains. The "first", "second", and similar terms used in the present invention do not denote any order, quantity, or importance, but are only used to distinguish different components. Terms such as "including" or "comprising" mean that the elements or items appearing before the term cover the elements or items listed after the term and their equivalents, without excluding other elements or items. Terms such as "connected" or "coupled" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. Terms such as "upper", "lower", "left", "right", etc. are only used to represent relative positional relationships, and when the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0061] As Figures 1 - 3 shown, a multi-source meteorological data integration and quality control system includes a data reception module, a data processing module, a data quality control module, a data integration module, a meteorological event prediction module, and an output module; among them,

[0062] The data reception module: responsible for receiving data from multiple meteorological data sources;

[0063] The data processing module: preprocesses the received data;

[0064] The data quality control module: examines the quality of the preprocessed data and makes necessary quality corrections;

[0065] The data integration module: integrates the data that has passed through quality control;

[0066] The meteorological event prediction module: based on the integrated data, predicts short-term and medium-term meteorological events, such as heavy rain, typhoon paths, high temperatures, etc., and provides corresponding reliability or probability assessments for this;

[0067] The output module: outputs the integrated data and the meteorological event prediction results for user use.

[0068] The data reception module includes interfaces with various meteorological data sources, and the meteorological data sources include ground stations, meteorological radars, meteorological satellites, meteorological sounding balloons, and ocean buoys. Real-time data and historical data from the above data sources are received through the interfaces, and the interfaces support various communication protocols and high-concurrency data transmission to ensure real-time performance and data integrity.

[0069] The data processing module includes a data format conversion sub-module, a data interpolation sub-module, and a data cleaning sub-module. Specifically:

[0070] Data format conversion sub-module: Used to convert data from various sources into a unified standard format, supporting multiple meteorological data standards such as GRIB, NetCDF, etc.;

[0071] Data interpolation sub-module: Interpolates the data space as needed, using Kriging or bilinear interpolation to improve the data space resolution;

[0072] Data cleaning sub-module: Processes abnormal data, duplicate data, and missing data, and the processing methods are to repair or eliminate them using the median method or the average method.

[0073] The Kriging method adopted by the data interpolation sub-module establishes a semi-variogram based on the spatial relationship of the data and performs data interpolation according to the distance and direction between stations; the bilinear interpolation method uses the values of the four adjacent points above, below, left, and right of the data point for interpolation, which is suitable for the spatial interpolation of grid data. Specifically,

[0074] The Kriging method adopted by the data interpolation sub-module is as follows:

[0075] Let Z(x i ) represent the observed value at position x i , and the predicted value Z * (x0) at position x0 is expressed as:

[0076] where λ i is the weight and N is the number of data points used for prediction;

[0077] The bilinear interpolation method is as follows:

[0078] If the four adjacent points are Q 11 , Q 12 , Q 21 , Q 22 , and the interpolation point is P with coordinates (x, y), then the bilinear interpolation value F(P) is:

[0079]

[0080] When the data cleaning sub-module processes abnormal data, it uses the median method to take the median value of multiple data points for repair to avoid the influence of extreme values on the results; the average method directly calculates the average value of multiple data points as the repair value, which is suitable for the case where the data does not change much. Specifically,

[0081] Median method:

[0082] For a data set \(x - 1, x - 2, …, x - ∩\), if \(n\) is odd, the median is: \(M = x\) (x+1) / 2 If \(n\) is even, the median is:

[0083] Average method:

[0084] For a data set \(x - 1, x - 2, …, x - ∩\), its average value is:

[0085] The data quality control module will check and evaluate the data quality. This check and evaluation include spatio-temporal consistency, statistical attributes of the data, physical rationality of the data, and data source verification. Specifically:

[0086] Spatio-temporal consistency: Check the timestamps and spatial coordinates of the data to ensure the continuity and consistency of the data in time and space;

[0087] Statistical attributes of the data: Analyze the statistical characteristics of the mean, variance, skewness, and kurtosis of the data, compare with historical data, and identify outliers;

[0088] Physical rationality of the data: Check whether the data conforms to physical laws, such as the relationships between meteorological elements such as temperature, humidity, and air pressure, to ensure the physical authenticity of the data.

[0089] Data source verification: Cross-verify the data from different data sources to ensure the truth and accuracy of the data;

[0090] Through the above meticulous data quality control, ensure the accuracy and reliability of subsequent data integration and meteorological event prediction.

[0091] The data integration module uses the method of data assimilation to integrate the data from multiple data sources. The data assimilation is in the form of Kalman filtering or 4D-VAR, combines the observed data and the output of the numerical model, and optimizes the initial conditions of the model to obtain a more accurate meteorological prediction.

[0092] The output module includes graphic output, table output, report output, and real-time notification functions. Specifically:

[0093] Graphic output: Can draw various meteorological graphics such as contour maps, color scale maps, and wind field maps according to the user's needs, and support interactive viewing and zooming;

[0094] Table output: Organize the data into a table. Users can filter, sort, and download according to time, location, or other conditions. The table supports multiple formats such as CSV, Excel, etc.;

[0095] Report Output: Automatically generate reports containing meteorological analysis, predicted trend meteorological disaster alerts. Reports are generated daily, weekly, or monthly and support email push;

[0096] Real-time Notification Function: When the integrated data reaches a predetermined condition (such as too high temperature, too fast wind speed, etc.), the system automatically sends real-time notifications to users to help them make decisions in a timely manner.

[0097] The deep learning method adopted by the meteorological event prediction module automatically learns the occurrence patterns of meteorological events from the training data. By analyzing the relationship between recent meteorological changes and historical data, it provides users with predictions of upcoming meteorological events, including the type, intensity, and possible occurrence time of the events.

[0098] The deep learning method uses a long short-term memory network model for prediction, and the long short-term memory network model includes the following formulas:

[0099] f t =σ(W f ·[h t-1 ,x t +b f )

[0100] i t =σ(W i ·[h t-1 ,x t +b i )

[0101]

[0102]

[0103] o t =σ(W o ·[h t-1 ,x t +b o )

[0104] h t =o t ×tanh(C t )

[0105] Among them, x t is the input vector at time t; h t is the hidden state at time t; C t is the cell state at time t; f t ,i t , o tare the values of the forget gate, input gate, cell state, and output gate, respectively; W and b are the learned weights and biases; σ is the Sigmoid activation function; × represents element-wise multiplication;

[0106] The long short-term memory network model is controlled by three gates (input gate, output gate, and forget gate) for information inflow, preservation, and outflow of the cell state. The above formula describes how to update these gates, as well as the cell state and hidden state at time point t. Through this structure, the long short-term memory network can capture and remember patterns in long time series, which is particularly crucial for the prediction of meteorological events because meteorological events are often affected by meteorological conditions over a past period of time.

[0107] The present invention aims to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A multi-source meteorological data integration and quality control system, characterized in that, It includes a data receiving module, a data processing module, a data quality control module, a data integration module, a meteorological event prediction module, and an output module. Among them, The data receiving module: responsible for receiving data from multiple meteorological data sources; The data processing module: preprocesses the received data; The data quality control module: examines the quality of the preprocessed data and makes necessary quality corrections; The data integration module: integrates the data that has passed the quality control; The meteorological event prediction module: based on the integrated data, predicts short-term and medium-term meteorological events and provides corresponding reliability or probability assessments for this; The output module: outputs the integrated data and the meteorological event prediction results for user use; The data receiving module includes interfaces with various meteorological data sources. The meteorological data sources include ground stations, meteorological radars, meteorological satellites, meteorological sounding balloons, and ocean buoys. Real-time data and historical data from the above data sources are received through the interfaces, and the interfaces support various communication protocols and high-concurrency data transmission; The data processing module includes a data format conversion sub-module, a data interpolation sub-module, and a data cleaning sub-module. Specifically: The data format conversion sub-module: used to convert data from various sources into a unified standard format, supporting multiple meteorological data standards; The data interpolation sub-module: performs interpolation processing on the data space as needed, using Kriging or bilinear interpolation to improve the data space resolution; The data cleaning sub-module: processes abnormal data, duplicate data, and missing data, and the processing method uses the median method or the average method for repair or elimination; The Kriging method adopted by the data interpolation sub-module establishes a semi-variogram based on the spatial relationship of the data, and performs data interpolation according to the distance and direction between stations; the bilinear interpolation method uses the values of the four adjacent points above, below, left, and right of the data point for interpolation, which is suitable for the spatial interpolation of grid data. Specifically, The Kriging method adopted by the data interpolation sub-module is as follows: Let Z(x i ) denote the observed value at position x i . The predicted value Z * (x0) at position x0 is expressed as: where λ i is the weight and N is the number of data points used for prediction; The bilinear interpolation method is as follows: If the four neighboring points are Q 11 , Q 12 , Q 21 , Q 22 , and the interpolation point is P with coordinates (x, y), then the value F(P) of bilinear interpolation is: When the data cleaning sub-module processes abnormal data, it uses the median method to take the median value of multiple data points for repair to avoid the influence of extreme values on the results; the average method directly calculates the average value of multiple data points as the repair value. Specifically, The median method: For a data set \(x_1, x_2, \ldots, x_n\), if \(n\) is odd, the median is: \(M = x_{(n + 1) / 2}\); if \(n\) is even, the median is: The average method: For a data set \(x_1, x_2, \ldots, x_n\), its mean is: The data quality control module will conduct an inspection and evaluation of the data quality. This inspection and evaluation includes spatio-temporal consistency, statistical attributes of the data, physical rationality of the data, and data source verification. Specifically: Spatio-temporal consistency: checks the timestamps and spatial coordinates of the data to ensure the continuity and consistency of the data in time and space; Statistical attributes of the data: analyzes the statistical characteristics of the mean, variance, skewness, and kurtosis of the data, compares with historical data, and identifies outliers; Physical rationality of the data: checks whether the data conforms to physical laws to ensure the physical authenticity of the data, Data source verification: cross-verifies the data from different data sources to ensure the authenticity and accuracy of the data; The data integration module integrates data from multiple data sources by using the method of data assimilation. The data assimilation is in the form of Kalman filtering or 4D-VAR, which combines the observed data with the output of the numerical model to optimize the initial conditions of the model. The output module includes graphic output, table output, report output, and real-time notification functions. Specifically: Graphic output: It can draw various meteorological graphics such as contour maps, color scale maps, and wind field maps according to the user's needs, and supports interactive viewing and zooming. Table output: It organizes the data into a table, and the user can filter, sort, and download according to time, location, or other conditions. Report output: It automatically generates reports containing meteorological analysis, prediction trends, and meteorological disaster alerts. The reports are generated daily, weekly, or monthly and support email push. Real-time notification function: When the integrated data reaches the predetermined conditions, the system automatically sends real-time notifications to the users. The meteorological event prediction module uses deep learning methods to automatically learn the occurrence patterns of meteorological events from the training data. By analyzing the relationship between recent meteorological changes and historical data, it provides predictions of upcoming meteorological events for users. The predictions include the type, intensity, and possible occurrence time of the events. The deep learning method uses a long short-term memory network model for prediction. The long short-term memory network model includes the following formula: f t = σ(W f · [h t-1 , x t + b f ) i t = σ(W i · [h t-1 'x t + b i ) o t = σ(W o · [h t-1 , x t + b o ) h t = o t × tanh(C t ) where x t is the input vector at time t; h t is the hidden state at time t; C t is the cell state at time t; f t , i t , o t are the values of the forget gate, input gate, cell state, and output gate, respectively; W and b are the learned weights and biases; σ is the Sigmoid activation function; × denotes element-wise multiplication; The long short-term memory network model is controlled by three gates for the inflow, preservation, and outflow of cell states. The above formula describes how to update these gates, as well as the cell state and hidden state at time point t. Through this structure, the long short-term memory network can capture and remember patterns in long time series.

Images