Methods for Building a Flood Forecasting Data Warehouse Based on Real-Time Assessment of Multi-Source Data

Abstract

This invention relates to a method for building a flood forecast data warehouse based on real-time evaluation of multi-source data, comprising the following steps: constructing a flood intelligent classification model considering multiple driving factors based on meteorological and hydrological indicators, and proposing applicable hydrological models; evaluating multi-dimensional heterogeneous data sources and screening usable data; constructing a sub-basin division method, and using this to construct a set of modeling schemes for a conceptual model of flood forecasting; designing a data model for the forecast object according to the applicable hydrological model and sub-basin schemes, and sorting out the core entities and relationships of the flood forecast data model; designing a physical model of the forecast object, constructing a data storage structure, and optimizing storage and querying; and extracting, transforming, and loading the data into the data warehouse. The flood forecast data warehouse constructed by this invention can not only meet the needs of flood forecasting operations for multi-dimensional heterogeneous data, but also automatically design modeling schemes based on data availability and runoff characteristics, providing reference and basis for hydrological forecasting and water resource management in the Yellow River Basin.

Description

Technical Field

[0001] This invention relates to a method for building a flood forecast data warehouse based on real-time assessment of multi-source data, belonging to the field of data warehouse construction technology. Background Technology

[0002] The in-depth application of water conservancy information technology and the continuous innovation in river basin water resources management, particularly in the research on Yellow River flood forecasting, have powerfully promoted the rapid development of digital river basin construction in my country, significantly improving the modernization level of the "four early warnings" (early warning, early warning, early reporting, and early warning) for river basin flood control. This has provided strong technical support for effectively responding to various flood disaster risks and ensuring sustainable economic and social development, achieving significant social and economic benefits. However, rapid economic and social development has also brought enormous pressure to river basin flood control systems, leading to a significant increase in the complexity, suddenness, and uncertainty of flood risks. This presents unprecedented challenges to the implementation of accurate flood forecasting and flood evolution simulation, involving more factors and greater difficulty.

[0003] As is well known, scientific, rapid, and effective flood forecasting must be based on massive, reliable data, enabling professionals to efficiently extract key information. For example, in critical stages such as flood forecasting and early warning, joint reservoir operation, activation of flood storage and detention areas, and emergency response deployment, in-depth analysis of various relevant data and their inherent relationships is necessary to formulate optimal solutions. With the rapid development of the economy, society, and information and communication technologies, data collection and processing methods related to flood monitoring, forecasting, operation, and defense are becoming increasingly diversified, automated, and intelligent. Real-time rainfall, water levels, weather forecasts, water conservancy project operation status, river topography, levee conditions, socio-economic exposure, and watershed spatial geographic information are characterized by their enormous volume, rapid growth, and complex relationships. Simultaneously, higher demands are being placed on the real-time nature, accuracy, and precision of information acquisition.

[0004] Taking Yellow River flood forecasting as an example, Yellow River floods are characterized by high peaks, large volumes, sudden onset, and complex evolution. Accurate forecasting directly impacts the scientific nature of flood control operations, the safety of downstream dikes, and the lives and property of people in the floodplains. It is a core basis for formulating flood control and disaster relief plans and optimizing the joint operation strategy of reservoir groups. However, the current ability to perform high-level, flexible, and efficient analysis, mining, and knowledge discovery on the massive, multi-source, heterogeneous data supporting flood forecasting remains insufficient. The timeliness and accuracy of obtaining key information required for forecasting are difficult to meet operational needs, and cannot effectively address the highly dynamic and complex nature of flood forecasting itself. Furthermore, existing technologies for forecasting peak flow, flood evolution, and inundation extent sometimes rely too heavily on a single fixed model or limited data sources, failing to fully integrate multi-dimensional real-time dynamic information from meteorology, underlying surface, and engineering scheduling for comprehensive analysis. This results in limited forecast accuracy and insufficient lead time, thus affecting the timeliness and scientific rigor of forecasting operations. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a method for building a flood forecast data warehouse based on real-time evaluation of multi-source data. This method comprehensively analyzes the influencing factors of Yellow River flood forecasting, extracts relevant data based on the runoff generation and confluence characteristics of the catchment area of ​​the forecast target, and designs a set of modeling schemes based on the runoff generation and confluence characteristics. This enables forecasters to quickly and scientifically conduct flood forecasting analysis in different regions of the Yellow River.

[0006] This invention provides a technical solution: a method for building a flood forecast data warehouse based on real-time evaluation of multi-source data, comprising the following steps:

[0007] S1. Obtain underlying surface data for multiple flood events within the catchment area of ​​the forecast target, calculate flood characteristic indicators and flood type influencing factors, extract spatiotemporal features of flood type influencing factors using graph attention neural network, classify floods using K-means clustering, calculate the runoff coefficient range for different flood types based on the classification results, identify runoff characteristics for different flood types, and select hydrological models suitable for full-storage runoff characteristics, infiltration-excess runoff characteristics, or mixed runoff characteristics;

[0008] S2. Obtain multidimensional heterogeneous data, statistically calculate the evaluation indicators of each data source, use the fuzzy cognitive graph model to calculate the weight of each evaluation indicator, and evaluate and filter out usability data based on the weights;

[0009] S3. Based on the spatial distribution of national hydrological stations monitoring runoff, the catchment area of ​​the forecast target is divided into multiple preliminary sub-basins; the inverse distance weighting method is used to unify the spatial accuracy of the underlying surface data; based on the underlying surface data with unified spatial accuracy, the DBSCAN clustering algorithm is used to spatially cluster within each preliminary sub-basin, extracting preliminary sub-basins with similar runoff generation and confluence properties to obtain multiple sub-basins; according to the expected rainstorm center in the meteorological forecast data and the availability data obtained in step S2, the sub-basins are merged to obtain multiple suitable sub-basins; a set of modeling schemes for the conceptual model of flood forecasting is constructed.

[0010] S4. Extract the core parameters and input / output data required by the hydrological model in step S1, and convert the data format, accuracy, etc. into the field requirements of the data model; combine the flood forecasting business process, divide into multiple core entities, and clarify the business meaning of each field; clarify the dependency relationship between core entities through "primary key-foreign key" association and business logic association, and present the association logic using an ER diagram;

[0011] S5. Transform the ER diagram from step S4 into a database table structure, determine the field mapping, storage engine selection, partitioning design, table partitioning strategy, and storage parameter configuration for each table, and obtain multiple databases;

[0012] S6. Define the scope of raw data required in the data warehouse, delineate data boundaries and spatiotemporal dimensions, extract raw data from multiple databases, clean and transform the raw data, load the transformed data into the data warehouse, and synchronously generate metadata and store it in the metadata database.

[0013] Furthermore, in step S1, a graph attention neural network is used to extract the spatiotemporal features of flood type influencing factors. The method includes:

[0014] The watershed under study is discretized into several grids, each grid being a node. Based on the discretized digital elevation model data, the D8 single-flow direction algorithm is used to obtain the flow direction relationship between each grid and determine the edge relationship between each node. Based on the nodes and edges, a GAT rainstorm map is constructed.

[0015] The node features are vectorized and expressed by the following formula:

[0016] ,

[0017] in This represents the number of nodes in the graph structure. The dimension of the node features in the graph structure is precipitation × land use index × slope.

[0018] The 3D node feature columns of the GAT rainstorm atlas are Z-score standardized, and the standardized 3D node features are then fed into a graph attention neural network model step by step over time. The attention mechanism dynamically aggregates neighbor information to obtain the aggregated multidimensional features of each node, expressed by the formula:

[0019] ,

[0020] in This represents the number of nodes in the graph structure. This represents the dimension of the node features output by the hidden layer.

[0021] Furthermore, the formula for calculating the Z-score standardization is as follows:

[0022] ,

[0023] in, It is the first The mean of the dimensional features, It is the first Standard deviation of dimensional features This represents the number of time steps, i.e., the number of maps in the GAT rainstorm atlas.

[0024] Furthermore, by dynamically aggregating neighbor information through an attention mechanism, multi-dimensional features of each node are obtained. The methods include:

[0025] For nodes and neighbors First, calculate the attention score. The calculation formula is as follows:

[0026] ,

[0027] in, , They are nodes and neighbors Input features, It is a learnable weight matrix (feature transformation). For learnable attention vectors, This involves concatenating vectors.

[0028] Then, the attention weights are normalized and the neighbor features are aggregated in a weighted manner to obtain the values ​​of each node. Multidimensional spatial aggregation features:

[0029] .

[0030] Furthermore, in step S2, the evaluation indicators for each data source are statistically calculated, and the methods include:

[0031] Evaluation metrics include: outlier rate, error rate, data transmission time, monitoring interval, monitoring coverage period, and monitoring site level. Data transmission time, monitoring interval, monitoring coverage period, and monitoring site level are obtained statistically. Outlier rate and error rate are calculated as follows:

[0032] ,

[0033] in, This represents the outlier rate. This represents the number of times outliers occurred. To monitor the total amount of data, For error value occurrence rate, This represents the number of times the error value occurred.

[0034] Furthermore, in step S2, the weights of each evaluation index are calculated using a fuzzy cognitive graph model, as follows:

[0035] (1) Set the initial values ​​of the evaluation indicators and initial causal weight ;

[0036] (2) Calculate the peak state value of the evaluation index:

[0037] ,

[0038] In the formula, Representing the vertex In time The state value; It is the vertex In time The weights;

[0039] (3) Determine systematic error:

[0040] ,

[0041] In the formula, Represents the model's learned values Compared with the actual system value The difference between them These are the actual system state values;

[0042] (4) Judgment result:

[0043] When satisfied and Stop iterating when the time comes and output the final causal correlation weights. That is, the weight of the evaluation indicators; otherwise, proceed to the next step.

[0044] (5) Adjustment of causal relationship weights:

[0045] ,

[0046] Adjust the causal association weights according to Formula 11 and return to step (2).

[0047] Furthermore, in step S4, the multiple core entities include:

[0048] Basic data source entity: Used to store raw meteorological, hydrological, and underlying surface data, containing fields such as "data collection time", "data source identifier", "rainfall", and "flow".

[0049] Model process entity: Used to store intermediate results of model calculations, containing fields including "sub-basin number", "calculation time step", "runoff coefficient", and "convergence time";

[0050] Forecast Result Entity: Used to store the final forecast data, containing fields including "Flood Event Number", "Forecast Peak Flow", "Forecast Flood Duration", and "Forecast Release Time";

[0051] Management and control entity: Used to store data quality and model version information, including fields such as "data quality level", "model version number" and "update responsible person".

[0052] Furthermore, in step S6, the methods for cleaning and transforming the original data include: removing outliers using the 3σ principle, filling in missing values ​​using linear interpolation, and identifying duplicates by comparing the three fields of "data collection time", "sub-basin ID", and "data type" to remove redundant information; and transforming the cleaned data using methods of format standardization, logical calculation, and aggregation reconstruction.

[0053] Due to the application of the above technical solution, the present invention has the following advantages compared with the prior art:

[0054] 1. This invention classifies flood events based on the spatiotemporal distribution characteristics of multiple driving factors, analyzes the runoff generation and confluence characteristics of different flood types for the forecast object, and proposes an applicable hydrological model, thereby enhancing the understanding of the hydrological characteristics of the forecast object.

[0055] 2. This invention constructs an intelligent evaluation model for data sources based on effectiveness, timeliness, and sustainability. It uses expert experience and knowledge to determine the strength of causal relationships and assess data usability, thereby enhancing the objectivity of data evaluation.

[0056] 3. This invention constructs a sub-basin partitioning method based on the DBSCAN clustering algorithm, which has strong practicality and wide applicability. Attached Figure Description

[0057] Figure 1 This is a node feature map of the GAT graph of flood type influencing factors in the Yiluo River Basin in one embodiment of the present invention;

[0058] Figure 2 This is a preliminary sub-basin modeling scheme diagram of the main runoff-producing area of ​​the Yi-Luo River Basin in one embodiment of the present invention;

[0059] Figure 3 This is a diagram illustrating a sub-basin modeling scheme for the main runoff-producing areas of the Yi-Luo River Basin, according to one embodiment of the present invention.

[0060] Figure 4 This is a diagram illustrating a suitable sub-basin modeling scheme for the main runoff-producing area of ​​the Yi-Luo River Basin in one embodiment of the present invention;

[0061] Figure 5 This is an ER (entity-relationship) diagram of the forecast object data model in one embodiment of the present invention. Detailed Implementation

[0062] The technical solution of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0064] Example 1:

[0065] This embodiment presents a method for building a flood forecast data warehouse based on real-time assessment of multi-source data. The method includes the following steps:

[0066] S1. Construct a smart flood classification model based on meteorological and hydrological indicators, considering multiple driving factors, analyze the runoff generation and confluence characteristics of the forecast object under different flood events, and propose applicable hydrological models. Specifically, this includes:

[0067] S1.1 Obtain underlying surface data for multiple flood events within the catchment area of ​​the forecast target. In this embodiment, underlying surface data for more than 10 flood events are obtained.

[0068] S1.2 Calculate flood characteristic indicators, including peak flow, flood duration, maximum 1-hour flood volume, and total flood volume, and calculate flood type influencing factors, including precipitation, land use, and slope.

[0069] S1.3 employs a Graph Attention Neural Network (GAT) to extract the spatiotemporal features of flood type influencing factors. Specific methods include:

[0070] The watershed under study is discretized into several grids, each grid being a node. Based on the discretized digital elevation model (DEM) data, the D8 single-flow direction algorithm is used to obtain the flow direction relationship between each grid and determine the edge relationship between each node. Based on the nodes and edges, a GAT rainstorm map is constructed.

[0071] The node features are vectorized and expressed by the following formula:

[0072] ,

[0073] in This represents the number of nodes in the graph structure. The dimension of each node feature in the graph structure is calculated as precipitation × comprehensive land use index × slope. The comprehensive land use index categorizes land use into four levels based on the degree of human activity in developing and utilizing land: unused land, forest / grassland / water use, agricultural land, and urban settlement land. This index reflects the strength of human activities' impact on the underlying surface features. Slope refers to the degree of inclination of a point on the land surface relative to the horizontal plane, usually expressed as the ratio of vertical height change (elevation difference) to horizontal distance. It directly affects the flow and distribution of surface water, thus influencing flood events.

[0074] The 3D node feature columns of the GAT rainstorm atlas are standardized using Z-score, and the calculation formula is as follows:

[0075] ,

[0076] in, It is the first The mean of the dimensional features, It is the first Standard deviation of dimensional features Time steps, which is the number of maps in the GAT rainstorm atlas.

[0077] The standardized 3D node features are input into the graph attention neural network model step by step over time. The neighbor information is dynamically aggregated through the attention mechanism to obtain the aggregated multidimensional features of each node, expressed by the formula:

[0078] ,

[0079] in This represents the number of nodes in the graph structure. This represents the dimension of the node features output by the hidden layer.

[0080] The above-mentioned method dynamically aggregates neighbor information through an attention mechanism to obtain multi-dimensional features of each node. The specific method is as follows:

[0081] For nodes and neighbors First, calculate the attention score. The calculation formula is as follows:

[0082] ,

[0083] in, , They are nodes and neighbors Input features, It is a learnable weight matrix (feature transformation). For learnable attention vectors, This involves concatenating vectors.

[0084] Then, the attention weights are normalized and the neighbor features are aggregated in a weighted manner to obtain the values ​​of each node. Multidimensional spatial aggregation features:

[0085] .

[0086] S1.4 uses K-means clustering to classify floods. The specific method is as follows:

[0087] The flood characteristic index values ​​of the multiple flood events calculated in step S1.2 are subjected to Z-score standardization to eliminate the dimensional differences between different flood characteristic indices. Then, the elbow rule is used to observe the clustering error. The trend of change as the K value increases, and the search for the "inflection point" (i.e., the "elbow"). The calculation formula is as follows: ,

[0088] To determine the optimal number of clusters for flooding.

[0089] The K-means clustering method was used to cluster the calculated flood characteristic indicators of historical flood events at hydrological stations. The clustering results were then used as the flood type classification results. The steps of the K-means clustering method are as follows:

[0090] (1) Randomly initialize k points as cluster centroids;

[0091] (2) Assign each point in the sample set to a cluster: calculate the distance between each point and the centroid (usually Euclidean distance and cosine distance), and assign it to the cluster corresponding to the nearest centroid;

[0092] (3) Update the centroid of the cluster: The centroid of each cluster is updated to the average value of all points in the cluster;

[0093] (4) Iterate through steps (2) to (3). If the distance between the new centroid and the previous centroid is less than a certain threshold, it can be considered that the clustering algorithm we performed has achieved the expected result and the algorithm terminates. Otherwise, continue iterating.

[0094] S1.5 Calculate the range of runoff generation coefficients (ratio of runoff to precipitation) for different flood types based on the classification results, identify the runoff generation characteristics of different flood types, and select hydrological models suitable for runoff generation characteristics of full storage, runoff generation characteristics of excessive infiltration, or mixed runoff generation characteristics from existing models in the current technology.

[0095] S2. Acquire multidimensional heterogeneous data, including precipitation, evaporation, soil moisture, and runoff. Statistically calculate the evaluation indicators for each data source, use a fuzzy cognitive graph model (FCM) to calculate the weights of each evaluation indicator, and evaluate and select usable data based on the weights.

[0096] Specifically, the evaluation indicators include: effectiveness indicators: outlier rate (%), error rate (%); timeliness indicators: data transmission time (s), monitoring time interval (s); sustainability indicators: monitoring coverage period, monitoring site level. Among these, data transmission time, monitoring time interval, monitoring coverage period, and monitoring site level are obtained statistically. Data transmission time is the time required for the monitoring business unit to transmit data to the data warehouse; the monitoring time interval is the time interval between monitoring data; the monitoring coverage period is the percentage of the monitoring period covered by the site within the forecast period; the monitoring site level is divided into national, provincial, and other levels, with a score of 1 for national level, 0.95 for provincial level, and 0.7 for other levels. The outlier rate and error rate are calculated as follows:

[0097] ,

[0098] in, This represents the outlier rate. This represents the number of times outliers occurred. To monitor the total amount of data, For error value occurrence rate, This represents the number of times the error value occurred.

[0099] The specific method for calculating the weights of each evaluation index using the Fuzzy Cognitive Graph (FCM) model is as follows:

[0100] (1) Set the initial values ​​of the evaluation indicators and initial causal weight , and The specific value is set according to the actual situation;

[0101] (2) Calculate the peak state value of the evaluation index:

[0102] ,

[0103] In the formula, Representing the vertex In time The state value; It is the vertex In time The weights;

[0104] (3) Determine systematic error:

[0105] ,

[0106] In the formula, Represents the model's learned values Compared with the actual system value The difference between them These are the actual system state values;

[0107] (4) Judgment result:

[0108] When satisfied and Stop iterating when the time comes and output the final causal correlation weights. That is, the weight of the evaluation indicators; otherwise, proceed to the next step.

[0109] (5) Adjustment of causal relationship weights:

[0110] ,

[0111] Adjust the causal association weights according to Formula 11 and return to step (2).

[0112] When screening usability data based on weight assessment, the weight value of the selection is set according to the specific situation. In this embodiment, data with a weight calculation result higher than 0.9 are identified as usability data.

[0113] S3. Based on the spatial distribution of national hydrological stations monitoring runoff, the catchment area of ​​the forecast target is divided into multiple preliminary sub-basins. The inverse distance weighting method is used to unify the spatial accuracy of the underlying surface data. Based on the unified spatial accuracy underlying surface data, the DBSCAN clustering algorithm is used to spatially cluster within each preliminary sub-basin, extracting preliminary sub-basins with similar runoff generation and confluence properties, resulting in multiple sub-basins. The extraction of preliminary sub-basins with similar runoff generation and confluence properties is implemented according to specific circumstances, using existing technologies, which will not be elaborated here. Based on the expected rainfall center from meteorological forecast data and the availability data obtained in step S2, the sub-basins are merged to obtain multiple suitable sub-basins. The merging of sub-basins is also implemented according to specific circumstances, using existing technologies, which will not be elaborated here. The above three sub-basin division schemes construct a set of modeling schemes for the conceptual model of flood forecasting, covering different modeling data requirements.

[0114] Specifically, the inverse distance weighting method is used to unify the spatial precision of the underlying surface data. This is an existing technique, and its specific algorithm will not be described here. The calculation steps of the DBSCAN clustering algorithm described above are as follows:

[0115] Assume there is a sample set Sample set of The neighborhood sample set is defined as the sample set whose distance to the sample point is less than or equal to... The set of sample points:

[0116] ,

[0117] in, This is the distance between two sample points, and any definition of distance can be used. Sample point density is defined as the number of sample points in the neighborhood.

[0118] ,

[0119] Density is a non-integer. Core points are defined as sample points in the dataset that are greater than a specified threshold, i.e., if...

[0120] ,

[0121] Then it is called The core points are regions where the sample distribution is dense. The set of all core points in the sample set is... The set consisting of non-core points is If it is a non-core point, and its neighborhood... If a point exists within a region, it is called a boundary point, which marks the boundary of a dense region. If a point is neither a core point nor a boundary point, it is called a noise point, which represents a sparse region of samples.

[0122] if That's the core point. In its Within the neighborhood, it is called From Directly achievable density. A set of samples in the sample set. .if From If the density is directly attainable, then it is called... From The algorithm is density-reachable. Starting from a core point, it iteratively expands to its density-reachable region, ultimately forming a maximized connected region containing both the core point and boundary points. All points within this region are density-connected to each other. Each such region is a cluster, and points outside these regions are considered noise points. Specifically, the underlying surface data affecting runoff generation and runoff properties, such as land use, slope, and soil type, are treated as sample points, and the DBSCAN algorithm is used to cluster these sample points.

[0123] S4. Extract the core parameters and input / output data required for the hydrological model in step S1. Transform the data format and precision into the field requirements of the data model, i.e., extract key parameters as each list item in the data table, such as rainfall, impermeability, and predicted peak flow. Simultaneously, transform the spatial attributes of the sub-basins (such as sub-basin ID, area, and spatial coordinate range) and underlying surface attributes (such as land use type and soil type code) from step S3 into entity attributes of the data model. Based on the flood forecasting workflow, divide the data into multiple core entities and clarify the business meaning of each field. Clarify the dependencies between core entities through primary key-foreign key relationships and business logic associations, and present the association logic using an ER diagram.

[0124] The aforementioned core entities include:

[0125] Basic data source entity: Used to store raw meteorological, hydrological, and underlying surface data, containing fields such as "data collection time", "data source identifier", "rainfall", and "flow".

[0126] Model process entity: Used to store intermediate results of model calculations, containing fields including "sub-basin number", "calculation time step", "runoff coefficient", and "convergence time";

[0127] Forecast Result Entity: Used to store the final forecast data, containing fields including "Flood Event Number", "Forecast Peak Flow", "Forecast Flood Duration", and "Forecast Release Time";

[0128] Management and control entity: Used to store data quality and model version information, including fields such as "data quality level", "model version number" and "update responsible person".

[0129] S5. Transform the ER diagram from step S4 into a database table structure (i.e., the data model described above), determining the field mapping, storage engine selection, partitioning design, table partitioning strategy, and storage parameter configuration for each table, resulting in multiple databases. Preferably, during the transformation, three principles are followed: time-series priority, read-write separation, and business query orientation. The time-series priority principle considers the time-series attributes of flood forecast data, prioritizing time-related storage optimizations (such as partitioning and table partitioning) in the physical design; the read-write separation principle avoids conflicts arising from simultaneous loading (write operations) and business queries (read operations) through storage structure design; and the business query orientation principle designs targeted indexes and storage structures to improve query efficiency for high-frequency query scenarios in flood forecasting.

[0130] S6. Define the scope of raw data required in the data warehouse, and delineate the data boundaries and spatiotemporal dimensions for meteorological, hydrological, and underlying surface data. Extract raw data from multiple databases obtained in step S5. Extraction methods include real-time extraction of high-frequency data via application programming (API) interface, full extraction of low-frequency data daily at midnight via direct database connection, and extraction of historical archived data via file parsing. Clean the raw data, using the 3σ principle to remove outliers (i.e.,... Outliers are identified as data points (where x is the data value, μ is the data mean, and σ is the data standard deviation). Missing values ​​are filled using linear interpolation. Duplicates are identified by comparing the "data collection time," "sub-basin ID," and "data type" fields to remove redundant information. The cleaned data undergoes transformation using format standardization, logical calculations, and aggregation reconstruction to meet the structured requirements of the data warehouse. These transformation methods are commonly used in existing technologies, and their specific calculation processes will not be described in detail here. The transformed data is loaded into the data warehouse using real-time incremental loading. Metadata is generated synchronously during the loading process. This metadata includes basic data attributes, structural descriptions, processing records, etc., and is stored in a metadata database for subsequent data management and use.

[0131] Example 2:

[0132] This embodiment uses the Yi-Luo River as a case study to illustrate the aforementioned method for building a flood forecasting data warehouse based on real-time assessment of multi-source data. The Yi-Luo River is a major source area for major floods in the lower reaches of the Yellow River, with a total drainage area of ​​18,881 km². 2The upper and middle reaches of the basin are mainly mountainous, while the lower reaches are mainly plains, with a confluence time of only 8-10 hours. The Yi-Luo River basin is dominated by a mixed runoff pattern, with both full-saturation runoff and infiltrative runoff coexisting in the same precipitation event.

[0133] S1. A flood intelligent classification model considering multiple driving factors was constructed based on meteorological and hydrological indicators. The runoff generation and confluence characteristics of the forecast object under different flood events were analyzed, and an applicable hydrological model was proposed. Taking the Yi-Luo River as an example, a graph attention neural network was used to extract the spatiotemporal features of flood type influencing factors, and K-means clustering was used to classify historical floods. Based on data quality, a total of 19402 GAT graph nodes were designed, such as... Figure 1 As shown, this paper analyzes 14 flood events from 2020 to 2024. The K-means++ algorithm was used for classification, and the elbow rule determined the optimal classification into three categories: short-duration low-peak type, long-duration low-peak type, and short-duration high-peak type. The main runoff generation and confluence patterns for different flood types can be identified based on the runoff coefficients calculated from the precipitation-runoff processes of each event. The calculation process is shown in Table 1.

[0134] Table 1

[0135]

[0136] S2. Acquire multidimensional heterogeneous data. The forecast data sources for the Yiluo River Basin include meteorological and hydrological monitoring data from operational units, monitoring units, and newly added station databases, as well as water conservancy project management databases, underlying surface data, professional attribute data, and data in other formats. Evaluation indicators for each data source were statistically calculated, and the weights of each evaluation indicator were calculated using a fuzzy cognitive graph model (FCM). The results are shown in Table 2. After evaluation, 617 rain gauge stations and 18 evaporation stations were selected as the model input data.

[0137] Table 2

[0138]

[0139] S3. Based on data support, underlying surface properties, and expected storm center, a sub-basin delineation method based on the DBSCAN clustering algorithm is constructed, and a set of conceptual modeling schemes for flood forecasting is built accordingly: First, a preliminary sub-basin delineation scheme is designed based on existing hydrological stations, such as... Figure 2 The forecast target catchment area is divided into several preliminary sub-basins. Then, the inverse distance weighting method is used to unify the spatial accuracy of land use, slope, and soil type data to 50m×50m. Based on the underlying surface data, the DBSCAN clustering algorithm is used to spatially cluster within each sub-basin, extracting sub-basin division schemes with similar runoff generation and confluence properties, such as... Figure 3 This results in multiple sub-basins. Then, based on the expected rainfall center and data support, these sub-basins are appropriately merged, and a suitable sub-basin delineation scheme is designed, such as... Figure 4This process yielded several suitable sub-basins. Finally, the division schemes for preliminary sub-basins, sub-sub-basins, and suitable sub-basins were compiled to construct a set of modeling schemes for a conceptual flood forecasting model that covers different modeling data requirements.

[0140] S4. Based on the applicable hydrological model and sub-basin scheme, design the forecast object data model, and sort out the core entities and relationships of the flood forecast data model: For the applicable models of three types of floods in the Yi-Luo River Basin—short-duration low-peak, long-duration low-peak, and short-duration high-peak—extract core parameters (such as soil infiltration rate and basin water storage capacity) and input / output data (input includes rainfall from 617 rain gauge stations and sub-basin slope; output includes peak flow and flood duration), and transform them into data model field requirements; transform the spatial attributes (sub-basin ID, spatial coordinate range within the 18881 km² basin, and area) and underlying surface attributes (land use type with 50m×50m precision and soil type code) of the Yi-Luo River sub-basin defined in step S3 into entity attributes. Combined with the Yi-Luo River flood forecasting business process, divide the basic data source entities, model process entities, forecast result entities, and management control entities, and clarify the business meaning of each field. Avoid redundancy and conflicts by using "primary key-foreign key" associations and business logic associations, and present the association logic using an ER diagram, such as... Figure 5 .

[0141] S5. Design the physical model of the forecast object, construct the data storage structure, and optimize storage and query: Based on the characteristics of flood forecast data in the Yiluo River Basin, and following the three principles of time priority, read-write separation, and business query orientation, the logical entity in step S4 is transformed into a specific database table structure, and the field mapping, storage engine selection, partition design, table partitioning strategy, and storage parameter configuration of each table are determined.

[0142] S6. Define the scope of raw data required in the data warehouse, and extract, transform, and load the data: Define the data boundaries for meteorological, hydrological, and underlying surface data in the Yi-Luo River Basin; extract raw data from meteorological databases and business systems, and clean and process the data to remove redundant information, correct errors, and fill in missing values ​​to ensure the accuracy, completeness, and timeliness of the data; perform data transformation operations, such as format standardization, logical calculation, and aggregation reconstruction, to adapt to the structured requirements of the data warehouse; load the transformed data into the data warehouse and simultaneously generate metadata to describe the data attributes and structure for easy subsequent management and use.

[0143] The above embodiments are only for illustrating the technical concept and features of the present invention. Their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be used to limit the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for building a flood forecasting data warehouse based on real-time assessment of multi-source data, characterized by: Includes the following steps: S1. Obtain underlying surface data for multiple flood events within the catchment area of ​​the forecast target, calculate flood characteristic indicators and flood type influencing factors. The flood characteristic indicators include peak flow, flood duration, maximum 1-hour flood total, and total flood volume. The flood type influencing factors include precipitation, land use, and slope. Use a graph attention neural network to extract the spatiotemporal features of the flood type influencing factors, use K-means clustering to classify floods, calculate the runoff coefficient range for different flood types based on the classification results, identify the runoff characteristics of different flood types, and select a hydrological model suitable for full-storage runoff characteristics, infiltration-excess runoff characteristics, or mixed runoff characteristics. S2. Acquire multidimensional heterogeneous data, statistically calculate the evaluation indicators of each data source, calculate the weight of each evaluation indicator using a fuzzy cognitive graph model, evaluate and filter usability data based on the weights, and the evaluation indicators include: outlier rate, error rate, data transmission time, monitoring time interval, monitoring coverage period and monitoring site level, wherein the data transmission time, monitoring time interval, monitoring coverage period and monitoring site level are obtained through statistics; S3. Based on the spatial distribution of national hydrological stations monitoring runoff, the catchment area of ​​the forecast target is divided into multiple preliminary sub-basins; the inverse distance weighting method is used to unify the spatial accuracy of the underlying surface data; based on the underlying surface data with unified spatial accuracy, the DBSCAN clustering algorithm is used to perform spatial clustering within each of the preliminary sub-basins, extracting preliminary sub-basins with similar runoff generation and confluence properties to obtain multiple sub-basins; according to the expected rainstorm center in the meteorological forecast data and the availability data obtained in step S2, the sub-basins are merged to obtain multiple suitable sub-basins; a set of modeling schemes for the conceptual model of flood forecasting is constructed. S4. Extract the core parameters and input / output data required by the hydrological model described in step S1, and convert the data format and accuracy into the field requirements of the data model; combine the flood forecasting business process, divide into multiple core entities, and clarify the business meaning of each field; clarify the dependency relationship between the core entities through "primary key-foreign key" association and business logic association, and present the association logic using an ER diagram; S5. Transform the ER diagram from step S4 into a database table structure, determine the field mapping, storage engine selection, partitioning design, table partitioning strategy, and storage parameter configuration for each table, and obtain multiple databases; S6. Define the scope of raw data required in the data warehouse, delineate data boundaries and spatiotemporal dimensions, extract raw data from the multiple databases, clean and transform the raw data, load the transformed data into the data warehouse, and synchronously generate metadata and store it in the metadata database.

2. The method for building a flood forecast data warehouse based on real-time evaluation of multi-source data according to claim 1, characterized in that: In step S1, the method for extracting the spatiotemporal features of flood type influencing factors using a graph attention neural network includes: The watershed under study is discretized into several grids, each grid being a node. Based on the discretized digital elevation model data, the D8 single-flow direction algorithm is used to obtain the flow direction relationship between each grid and determine the edge relationship between each node. Based on the nodes and edges, a GAT rainstorm map is constructed. The node features are vectorized and expressed by the following formula: ； in This represents the number of nodes in the graph structure. The dimension of the node features in the graph structure is precipitation × land use index × slope. The 3D node feature columns of the GAT rainstorm atlas are Z-score standardized, and the standardized 3D node features are then fed into a graph attention neural network model step by step over time. The attention mechanism dynamically aggregates neighbor information to obtain the aggregated multidimensional features of each node, expressed by the formula: ； in This represents the number of nodes in the graph structure. This represents the dimension of the output node features of the hidden layer.

3. The method for building a flood forecast data warehouse based on real-time evaluation of multi-source data according to claim 2, characterized in that: The formula for calculating the Z-score normalization is as follows: ； in, It is the first The mean of the dimensional features, It is the first Standard deviation of dimensional features This represents the number of time steps, i.e., the number of maps in the GAT rainstorm atlas.

4. The method for building a flood forecast data warehouse based on real-time evaluation of multi-source data according to claim 2, characterized in that: The method for dynamically aggregating neighbor information through an attention mechanism to obtain multidimensional features of each node includes: For nodes and neighbors First, calculate the attention score. The calculation formula is as follows: ； in, , They are nodes and neighbors Input features, The weight matrix is ​​a learnable matrix. For learnable attention vectors, This involves concatenating vectors. Then, the attention weights are normalized and the neighbor features are aggregated in a weighted manner to obtain the values ​​of each node. Multidimensional spatial aggregation features: 。 5. The method for building a flood forecast data warehouse based on real-time evaluation of multi-source data according to claim 1, characterized in that: In step S2, the method for statistically calculating the evaluation indicators for each data source includes: The outlier rate and error rate are obtained through the following calculation methods: ； in, This represents the outlier rate. This represents the number of times outliers occurred. To monitor the total amount of data, For error value occurrence rate, This represents the number of times the error value occurred.

6. The method for building a flood forecast data warehouse based on real-time evaluation of multi-source data according to claim 1, characterized in that: In step S2, the method for calculating the weights of each evaluation index using the fuzzy cognitive graph model is as follows: (1) Set the initial values ​​of the evaluation indicators and initial causal weight ; (2) Calculate the peak state value of the evaluation index: ； In the formula, Representing the vertex In time The state value; It is the vertex In time The weights; (3) Determine systematic error: ； In the formula, Represents the model's learned values Compared with the actual system value The difference between them These are the actual system state values; (4) Judgment result: When satisfied and Stop iterating when the time comes and output the final causal correlation weights. That is, the weight of the evaluation indicators; otherwise, proceed to the next step. (5) Adjustment of causal relationship weights: ； Adjust the causal association weights according to Formula 11 and return to step (2).

7. The method for building a flood forecast data warehouse based on real-time evaluation of multi-source data according to claim 1, characterized in that: In step S4, the multiple core entities include Basic data source entity: Used to store raw meteorological, hydrological, and underlying surface data, containing fields such as "data collection time", "data source identifier", "rainfall", and "flow". Model process entity: Used to store intermediate results of model calculations, containing fields including "sub-basin number", "calculation time step", "runoff coefficient", and "runoff confluence time"; Forecast Result Entity: Used to store the final forecast data, containing fields including "Flood Event Number", "Forecast Peak Flow", "Forecast Flood Duration", and "Forecast Release Time"; Management and control entity: Used to store data quality and model version information, including fields such as "data quality level", "model version number" and "update responsible person".

8. The method for building a flood forecast data warehouse based on real-time evaluation of multi-source data according to claim 1, characterized in that: In step S6, the method for cleaning and transforming the original data includes: removing outliers using the 3σ principle, filling in missing values ​​using linear interpolation, and identifying duplicates by comparing the three fields of "data collection time", "sub-basin ID" and "data type" to remove redundant information; and transforming the cleaned data using methods of format standardization, logical calculation and aggregation reconstruction.

Images