Water resources bulletin real-time data calculation method based on multi-source heterogeneous data fusion

Abstract

This invention relates to the field of multi-source data processing technology, and more particularly to a method for real-time data calculation of water resources bulletins based on the fusion of multi-source heterogeneous data. The method includes the following steps: constructing a pluggable and high-concurrency unified data access gateway, configuring multiple types of adapters within the unified data access gateway; completing the access and extraction of multi-source heterogeneous data through each adapter, adding spatiotemporal tags and metadata tags to the data extracted by the unified data access gateway to obtain an initial data asset catalog; establishing a multi-dimensional data quality assessment matrix, automating and standardizing the data in the initial data asset catalog, and outputting standardized water resources data. This invention, by constructing a unified access gateway, a programmable computing engine, and a spatiotemporal knowledge fusion system, achieves full-process automation of data processing, multi-source fusion, and dynamic generation of bulletins, thereby improving the processing efficiency, timeliness, and availability of multi-source heterogeneous water resources data.

Description

Technical Field

[0001] This invention relates to the field of multi-source data processing technology, and in particular to a method for calculating real-time data from water resources bulletins based on the fusion of multi-source heterogeneous data. Background Technology

[0002] Water resources information in water resources bulletins is the foundation of water resources planning and supply and demand planning, and also the core data support for implementing the strictest water resources management system. Its traditional compilation work relies heavily on manual data collection, summarization and accounting across departments and multiple levels. Data sources cover multiple key elements such as precipitation and runoff from meteorological, hydrological and other departments. The development of technologies such as the Internet of Things and remote sensing has enriched the means of water resource data observation, resulting in an exponential increase in data volume and types. However, this has also created a multi-source heterogeneous big data environment characterized by diverse sources, heterogeneous structures, inconsistent protocol standards, and imbalanced spatiotemporal scales. This has led to numerous shortcomings in existing compilation techniques: data systems in different departments form independent data silos, resulting in high barriers to cross-system data fusion and high costs for manual intervention; data processing is mainly offline batch processing, which is time-consuming for collection and organization, leading to extremely poor timeliness in bulletin publication and failing to meet real-time scheduling needs; there is a lack of an intelligent data quality governance system, and the handling of outliers and missing values ​​relies heavily on manual experience, making it difficult to guarantee the consistency of results; the calculation models for core indicators are scattered and fixed, lacking standardized and reusable services, as well as an automatic verification mechanism for calculation results; the bulletin results are static documents without interactive query or drill-down analysis capabilities, making it difficult to extract deeper value from the data and failing to meet the actual needs of refined water resource management. Summary of the Invention

[0003] Therefore, it is necessary to provide a method for calculating real-time water resources bulletin data based on the fusion of multi-source heterogeneous data to solve at least one of the above-mentioned technical problems.

[0004] To achieve the above objectives, a method for calculating real-time water resources bulletin data based on multi-source heterogeneous data fusion is provided, the method comprising the following steps: Step S1: Build a pluggable and high-concurrency unified data access gateway, configure multiple types of adapters within the unified data access gateway; complete the access and extraction of multi-source heterogeneous data through each adapter, add spatiotemporal tags and metadata tags to the data extracted by the unified data access gateway to obtain the initial data asset catalog. Step S2: Establish a multi-dimensional data quality assessment matrix, automatically process and standardize the data in the initial data asset catalog, and output standardized water resource data; Step S3: Build the core data lake for spatiotemporal indexing, import standardized water resource data to complete spatiotemporal alignment and entity association, construct a water resource knowledge graph, and output spatiotemporally integrated and associated water resource data; Step S4: Determine the configurable water resources index calculation engine, convert the calculation logic of the preset bulletin compilation into calculation atomic services, build a calculation model library and arrange the index calculation workflow based on the calculation model library, call the spatiotemporally fused and correlated water resources data to complete the calculation, and output the bulletin index calculation results. Step S5: Store the calculation results of the bulletin indicators to a high-performance time series database, visualize and interactively analyze the calculation results of the bulletin indicators, and generate and publish dynamic water resources bulletins.

[0005] The beneficial effects of this invention are as follows: 1. By building a unified access gateway, an intelligent data cleaning pipeline, and a programmable computing engine, the entire data processing process is automated and intelligent, significantly reducing manual intervention and effectively improving the efficiency and accuracy of water resources bulletin data processing.

[0006] 2. Based on the spatiotemporal benchmark and knowledge graph, a multi-source data fusion method is constructed to effectively integrate multi-source heterogeneous water resource data from both semantic and spatiotemporal dimensions, breaking down data silos and fusion barriers, and providing a high-quality data foundation for water resource data analysis.

[0007] 3. Upgrade the water resources data processing mode from offline batch processing to near real-time stream processing to enable on-demand and dynamic generation of water resources bulletins, significantly improving the timeliness of bulletin release and providing direct decision-making data support for water resources emergency management and real-time dispatch.

[0008] 4. The system adopts a modular and microservice-based system design architecture, which enables the system to have good maintainability and scalability. New data sources and computing models can be easily integrated into the existing system framework through plug-in, ensuring the continuous iteration and long-term stable operation of the system.

[0009] 5. By constructing an interactive and visual dynamic water resources bulletin display system, the traditional static bulletin presentation format is changed, creating an immersive data consumption experience, effectively improving the availability and utilization efficiency of water resources data, and strengthening the data's ability to support water resources management decisions. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating the steps of a method for calculating real-time water resources bulletin data based on the fusion of multi-source heterogeneous data. Figure 2 A flowchart illustrating the calculation method for real-time data in water resources bulletins; Figure 3 A diagram illustrating the overall technical architecture of the real-time data calculation method for water resources bulletins; The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0011] The technical method of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0012] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor methods and / or microcontroller methods.

[0013] It should be understood that although the terms "first," "second," etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are used merely to distinguish one unit from another. For example, without departing from the scope of the exemplary embodiments, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0014] To achieve the above objectives, please refer to Figures 1 to 3 A method for calculating real-time water resources bulletin data based on multi-source heterogeneous data fusion, the method comprising the following steps: Preferably, step S1: Build a pluggable and high-concurrency unified data access gateway, configure multiple types of adapters in the unified data access gateway; complete the access and extraction of multi-source heterogeneous data through each adapter, add spatiotemporal tags and metadata tags to the data extracted by the unified data access gateway to obtain the initial data asset catalog. Optionally, step S1 includes the following steps: The various types of adapters configured within the unified data access gateway are dedicated processing modules for adapting streaming data, API interfaces, batch data, and unstructured data. Each dedicated processing module performs protocol parsing and data extraction on the corresponding type of multi-source heterogeneous data; After extraction, each piece of multi-source heterogeneous data is assigned a unique data ID, and then bound with spatiotemporal tags and metadata tags; A structured index is created based on data ID and tag information to obtain the initial data asset catalog.

[0015] In this embodiment, a pluggable and high-concurrency unified data access gateway is configured with processing rules for streaming data, application programming interface data, batch data, and unstructured data. For streaming data, a streaming transmission protocol is used to receive the data. The time-series data streams transmitted by hydrological sensors and IoT terminals are parsed at a second-level time resolution to extract the original monitoring data of hydrological elements in the data stream. For application programming interface (API) data, configure API request parameters and authentication information, initiate API calls at an hourly time resolution, and perform protocol parsing on API data returned by meteorological and environmental protection departments to extract structured monitoring data. For batch data, configure synchronization tasks for heterogeneous data sources, synchronize and parse offline water resource data in relational databases and distributed file systems at a daily time resolution, and extract historical monitoring and statistical data stored in the databases. For unstructured data, use optical character recognition (OCR) technology to perform character recognition and extraction on report images and portable document formats, and then use natural language processing (NLP) technology to perform semantic analysis on the extracted character information to filter and extract key water resource business data.

[0016] It should be noted that for all the multi-source heterogeneous data that have been extracted, a unique 32-character data identifier is generated according to the combination rule of collection timestamp + unique number of collection device + data element type code. A spatiotemporal tag is added to each data based on the actual collection time information and the geographical coordinate information of the collection point. A metadata tag is added to each data based on the data source department, collection device type and water resource element category to which the data belongs. A unique correspondence between the spatiotemporal tag, the metadata tag and the data ontology is established through the data identifier.

[0017] Using data identifiers as primary retrieval identifiers, and geographic coordinates in spatiotemporal tags, collection timestamps, and data source departments and data element types in metadata tags as secondary retrieval identifiers, a tree-like index structure is used to build a structured index for all tagged multi-source heterogeneous data. The index information is associated with and stored with the corresponding data ontology to form an initial data asset catalog containing data ontology, unique data identifiers, multi-dimensional tags, and structured indexes.

[0018] Preferably, step S2: establish a multi-dimensional data quality assessment matrix, automatically process and standardize the data in the initial data asset catalog, and output standardized water resource data; Optionally, step S2 includes the following steps: The established multi-dimensional data quality assessment matrix covers multiple verification dimensions of data format, spatiotemporal, and numerical values. A full scan and dimensional verification of the data in the initial data asset catalog were performed based on a multi-dimensional data quality assessment matrix. Based on the verification results, the data in the initial data asset catalog undergoes automated processing for format correction, spatiotemporal completion, and numerical calibration. The data that has been processed automatically will be converted into a standard time-series format, and the timestamps, spatial coordinates and numerical units will be standardized to output standardized water resources data.

[0019] In this embodiment, a multi-dimensional data quality assessment matrix is ​​constructed. The matrix horizontally includes three verification dimensions: format, spatiotemporal, and numerical. Vertically, it is divided into verification categories based on water resource data types, such as precipitation, runoff, groundwater, and water intake / extraction. Quantitative verification standards are configured for each cross-verification unit in the matrix. For the format dimension, verification thresholds are set at a field missing rate ≤0.05% and a field format matching degree of 100%. For the spatiotemporal dimension, verification thresholds are set at a timestamp validity rate ≥99.9% and a spatial coordinate matching degree ≥99%. For the numerical dimension, verification thresholds are set at a numerical anomaly rate ≤0.1% and numerical precision to two decimal places. Corresponding automated verification algorithms and judgment logic are also configured for each verification dimension. Based on this multi-dimensional data quality assessment matrix, a full scan of the water resource data in the initial data asset catalog is performed. According to the verification dimensions and categories of the matrix, the format integrity, spatiotemporal validity, and numerical compliance of each data entry are automatically verified one by one. The verification results, non-compliance types, and specific anomaly locations for each data entry are recorded simultaneously, forming a data quality verification ledger.

[0020] Specifically, based on the results of the data quality verification log, automated processing operations are performed on the data in the initial data asset catalog. For data that fails format verification, missing fields are automatically filled in and incorrect formats are standardized and corrected according to the predefined water resources metadata ontology. All data structures are uniformly mapped to a custom water resources standard data model. For data that fails spatiotemporal verification, a sliding window algorithm is used to identify invalid timestamps and correct them in conjunction with data collection records. Inappropriate spatial coordinates are filled in using a spatial interpolation algorithm. At the same time, the matching of spatiotemporal information is verified and mismatch calibration is completed. For data that fails numerical verification, invalid values ​​are removed, abnormal values ​​are corrected, and the numerical accuracy of all data is uniformly calibrated to ensure that the numerical logic is consistent with the actual water resources monitoring patterns.

[0021] It should be noted that all water resources data that have undergone format correction, spatiotemporal completion, and numerical calibration are standardized and converted. All data are uniformly converted to the standard time series data format, and all timestamps in the data are calibrated to Coordinated Universal Time (UTC), with time accuracy uniformly retained to the second level. All spatial coordinates are converted to the 2000 National Geodetic Coordinate System, with spatial accuracy uniformly retained to the meter level. The numerical units of all water resources data, such as precipitation, runoff, and water level, are all converted to international standard units such as liters, cubic meters, and millimeters. After completing the standardization and conversion of all dimensions, the data undergoes a second quality verification. After the verification is passed, standardized water resources data is formed and officially output.

[0022] Of particular importance is that step S2 involves uniformly converting the data that has undergone automated processing into a standard time-series format, specifically as follows: During the standardization and conversion phase, the data timestamps are uniformly calibrated to UTC time; Transform the data space coordinates to the CGCS2000 coordinate system; All numerical units in the data will be converted to international standard units. Consistency verification is performed on the data that has completed multi-dimensional standardization. Once the verification is passed, it is output as standardized water resources data.

[0023] In this embodiment, all data that has undergone automated processing is uniformly converted into a standard time-series format, and multi-dimensional standardization conversion is carried out. Timestamp calibration adopts time format conversion technology to convert all timestamps of various formats into Coordinated Universal Time (UTC), and the time accuracy is uniformly retained to the second level. Spatial coordinate conversion adopts geographic coordinate projection conversion technology to convert all spatial coordinates of all coordinate systems to the 2000 National Geodetic Coordinate System, and the coordinate accuracy is uniformly retained to the meter level. Numerical unit conversion is based on the water resources calculation standard, and the units of all water resources data such as precipitation, runoff, and groundwater resources are uniformly converted into international standard units such as millimeters, cubic meters, and cubic meters per second.

[0024] After completing the above standardization transformation, consistency verification is carried out on the data. The verification content covers the uniformity of timestamp format, coordinate system type, and numerical unit. At the same time, the matching degree of data fields, spatiotemporal information, numerical logic and water resources standard data model is verified. The verification adopts the full data sampling verification method with a sampling ratio of 100%. The verified data is integrated and packaged to finally output standardized water resources data.

[0025] Preferably, step S3: build a core data lake with spatiotemporal index, import standardized water resource data to complete spatiotemporal alignment and entity association, construct a water resource knowledge graph, and output spatiotemporally integrated and associated water resource data; Optionally, the spatiotemporal index core data lake built in step S3 is specifically as follows: The system combines a pre-defined standardized spatiotemporal network index and a vector administrative division index, and assigns unique spatiotemporal attribute identifiers to each index node. Extract the spatiotemporal attributes of standardized water resources data collection, and match the standardized water resources data to the corresponding nodes in the dual-index system based on the spatiotemporal attributes of collection. Standardized water resource data within each dual-index system node are resampled and aggregated according to a preset unified time and spatial granularity.

[0026] In this embodiment, when building the core data lake for spatiotemporal indexing, a dual index system of standardized spatiotemporal grid index and vector administrative division index is pre-built. The standardized spatiotemporal grid index is divided into 1km×1km regular grid units according to latitude and longitude, and the vector administrative division index is divided into geographical units according to the administrative levels of province, city, county and township. Each grid unit and administrative unit is configured with a unique spatiotemporal attribute identifier, which includes three types of information: geographic code, hierarchical code and time granularity code, forming a dual index spatial framework with full coverage.

[0027] The pre-defined spatiotemporal attributes of the data collection are extracted from the standardized water resources data. The time attribute includes the specific timestamp and time resolution of the data collection, while the spatial attribute includes the geographic coordinates and administrative region of the data collection point. Through geospatial matching technology, the spatial attributes of the standardized water resources data are accurately matched with the spatiotemporal attribute identifiers of the dual-index system, and each piece of standardized water resources data is directed and collected to the corresponding grid cell and administrative unit index node.

[0028] Standardized water resource data within each node of the dual-index system are resampled and aggregated according to a preset unified time and spatial granularity. The unified time granularity is set to daily level, and the unified spatial granularity is divided into three categories according to administrative level: provincial, municipal, and county. Time series resampling technology is used to interpolate and summarize water resource data with different time resolutions within the node. Spatial aggregation technology is used to calculate the mean and total amount of precipitation, runoff, and groundwater resources with different spatial resolutions within the node, ensuring that the standardized water resource data within each index node has a completely unified spatiotemporal granularity.

[0029] The data of each node in the dual-index system that has completed resampling and aggregation processing are associated and stored. The node data of the standardized spatiotemporal network index and the vector administrative division index are mapped through spatiotemporal attribute identifiers to realize mutual query and communication of water resource data in the same region and the same time dimension within the dual-index system. The construction and data entry of the core data lake of spatiotemporal index are completed, providing a unified spatiotemporal data base for subsequent spatiotemporal alignment and entity association.

[0030] Optionally, step S3, which involves importing standardized water resources data to complete spatiotemporal alignment and entity association, includes: In the entity association matching stage, monitoring stations, water conservancy projects, administrative units, and water resource management objects are taken as core entities; a two-way association relationship of spatial affiliation and hydrological association is established among the core entities; Bind the two-way correlation relationship with the spatiotemporally integrated water resource data; Based on the binding results, a water resources knowledge graph is constructed, and spatiotemporally integrated water resources data is output.

[0031] In this embodiment, during the entity association matching stage, basic entity information for monitoring stations, water conservancy projects, administrative units, and water resource management objects is first extracted from standardized water resource data. A unique entity code is assigned to each core entity, consisting of an entity type code and a sequence code, totaling 18 characters. Simultaneously, attribute information such as the spatial coordinates in the 2000 national geodetic coordinate system, the administrative level to which the entity belongs, and the hydrological association range of each entity are collected to establish a core entity basic information database. Based on spatial topology analysis and hydrological system analysis techniques, bidirectional association relationships are established between the core entities. Spatial affiliation relationships include monitoring stations belonging to corresponding administrative units, water conservancy projects located within corresponding administrative units, and water resource management objects belonging to corresponding administrative units. Hydrological association relationships include monitoring stations covering the monitoring range of corresponding water conservancy projects and water resource management objects, upstream and downstream associations of water conservancy projects with corresponding hydrological monitoring stations, and water intake associations of water resource management objects with corresponding water conservancy projects and monitoring stations.

[0032] After completing the construction of the two-way association, the standardized water resources data is matched with the core entity basic information database. Through the spatial matching technology of the collection point coordinates in the data and the entity spatial coordinates, the corresponding core entity code is bound to each standardized water resources data. Then, the two-way association between core entities is mapped to the standardized water resources data through the entity code, so as to realize the accurate binding of the two-way association with the standardized water resources data after spatiotemporal fusion. The bound data includes the data ontology, spatiotemporal attributes, entity code and full information of entity association.

[0033] It should be noted that the construction of the water resources knowledge graph based on the binding results adopts graph data storage technology. Monitoring stations, water conservancy projects, administrative units, and water resources management objects are used as nodes of the knowledge graph. The node attributes include information such as entity code, spatial coordinates, and basic attributes. The spatial affiliation and hydrological association between core entities are used as edges of the knowledge graph. The edge attributes include information such as association type, association range, and association weight. The association between nodes and edges is constructed according to entity code and association relationship to form a complete water resources knowledge graph.

[0034] The constructed water resources knowledge graph and the bound standardized water resources data are integrated into one. The mutual query association between the knowledge graph and the data is established through entity coding. The integrity of the integrated data is verified, and the entity association coverage rate and the spatiotemporal attribute matching degree reach 100%. After the verification is passed, the data is classified and stored according to administrative region and hydrological basin. Finally, the spatiotemporal fusion associated water resources data containing spatiotemporal attributes, entity association, and knowledge graph association are output.

[0035] Most importantly, based on the binding results, a water resource knowledge graph is constructed, and the output of spatiotemporally integrated water resource data is as follows: The entity association integrity of the water resources knowledge graph is verified, and missing association nodes found during the verification are filled and corrected. The corrected knowledge graph and spatiotemporal fusion data are stored in an integrated manner. A search entry is established according to the spatiotemporal and entity attributes of the knowledge graph and spatiotemporal fusion data, and searchable spatiotemporal fusion associated water resources data is output.

[0036] In this embodiment, after constructing a water resources knowledge graph based on the binding results of bidirectional association and standardized water resources data after spatiotemporal fusion, an entity association integrity check is performed on the water resources knowledge graph. According to the core entity association rules of monitoring stations, water conservancy projects, administrative units, and water resources management objects, the association relationships of all entity nodes in the graph are enumerated one by one. The association coverage rate, association relationship pointing accuracy, and association attribute integrity of the entity nodes are checked. The entity association coverage rate check threshold is set to 100%, the association relationship pointing accuracy rate check threshold is set to 100%, and the association attribute integrity rate check threshold is set to 100%. The specific information of missing association nodes, mismatched association relationships, and missing association attributes found during the check process is recorded.

[0037] For nodes with missing associations discovered during verification, information such as entity spatial coordinates, hierarchical level, and hydrological association range from the core entity basic information database is retrieved. Spatial topology analysis and hydrological system analysis are then re-executed to complete the spatial affiliation and hydrological association bidirectional relationship of the nodes. For mismatched association relationships, the direction and association type of the association edges in the graph are directly corrected. For missing association attributes, information such as the association range and association weight of the association edges are supplemented to complete the full correction of the water resources knowledge graph.

[0038] The revised water resources knowledge graph and the standardized water resources data after spatiotemporal fusion are stored in an integrated manner. The graph-data fusion storage technology is adopted to establish a full-domain mapping between the entity nodes and related edge information of the knowledge graph and the standardized water resources data through entity coding. The data is stored in partitions according to the provincial, municipal and county administrative levels and the time granularity of day, month and year. During the storage process, the full correspondence between the spatiotemporal attributes, entity attributes and related relationship attributes of the data is preserved.

[0039] Based on the data attributes of integrated storage, a multi-condition combined search entry is established according to four dimensions: administrative region code, entity type code, collection timestamp, and hydrological element type. Search rules for precise and fuzzy search are configured for each search dimension. The search entry supports fast search by single dimension and multi-dimensional combination filtering. All standardized water resources data that are spatiotemporally integrated and correlated are connected to the search entry. After completing the association configuration between the search entry and the data, the spatiotemporally integrated and correlated water resources data that can be accurately searched through multiple dimensions is officially output.

[0040] Preferably, step S4: determine the configurable water resources index calculation engine, convert the calculation logic of the preset bulletin compilation into calculation atomic services, build a calculation model library and arrange the index calculation workflow based on the calculation model library, call the spatiotemporally fused and correlated water resources data to complete the calculation, and output the bulletin index calculation results; Optionally, the configurable water resource index calculation engine in step S4 has a modular architecture, including an atomic service management module, a model scheduling module, a calculation execution module, and a result verification module. The atomic service management module is responsible for the registration, configuration, and invocation management of atomic computing services. The model scheduling module is responsible for loading and matching the computational model library. The computation execution module is responsible for driving the parallel and serial execution of computation tasks; The result verification module is responsible for real-time verification of the calculation process and results; Each module achieves data exchange and functional linkage through standardized interfaces.

[0041] In this embodiment, the configurable water resource index calculation engine adopts a modular architecture design, which is divided into an atomic service management module, a model scheduling module, a calculation execution module, and a result verification module. Each module is configured with standardized interface specifications, and the interface data transmission format is uniformly structured water resource data format. The interface interaction response time is set to within 1 second. Through this standardized interface, full data exchange and functional linkage between modules are realized. Each module runs independently and can be individually expanded and its parameters adjusted.

[0042] The atomic service management module manages the entire lifecycle of computational atomic services. First, it registers computational atomic services according to the water resources bulletin compilation specifications, assigning a unique service code to each service. The code contains 16 characters, including the service type and sequence identifier. Then, it configures the parameters of the registered services according to computational needs, covering data input / output formats, computation triggering conditions, and runtime resource allocation standards. Finally, it achieves precise call management of computational atomic services based on the service codes, recording the time, node, and data flow of each call, with a call response success rate of 100%.

[0043] The model scheduling module is responsible for the full-process management of the computational model library. First, it loads sub-model libraries such as areal rainfall and river runoff according to the water resources bulletin indicator types. After loading, it verifies the integrity of the model library, and the model file integrity rate must reach 100%. Then, based on the service code and computational requirements of the computational atomic service, it establishes matching rules between services and models, and completes the automatic matching and scheduling of computational models according to these rules. During the scheduling process, the model running status is monitored in real time, and the model scheduling response time is set to within 0.5 seconds to ensure seamless integration between computational atomic services and matching models.

[0044] The computation execution module drives the execution of computation tasks based on the model scheduling results. It first parses the execution rules of the computation tasks, distinguishing between parallel and serial execution types. For parallelizable tasks such as areal rainfall and evaporation calculations, independent computational resources are allocated, with a maximum of 20 parallel tasks running simultaneously. For tasks with sequential logic, such as surface water resources and total water resources calculations, execution is ordered according to serial rules, strictly adhering to the logic of starting subsequent tasks only after the preceding task is completed. The result verification module performs real-time verification of the computation process and results. It collects process data at each computation node, verifies the integrity of data transmission, sets compliance thresholds for the calculation results according to the water resources bulletin compilation standards, and performs full verification of the calculation results. If verification fails, a recalculation instruction is triggered to ensure that the computation process is error-free and the results are fully compliant.

[0045] Optionally, in step S4, converting the computational logic compiled from the preset bulletin into computational atomic services specifically involves: The calculation logic for compiling the pre-set bulletin is broken down into single-dimensional calculation logic units according to the water resources bulletin indicator compilation specifications, with each calculation logic unit corresponding to a basic calculation operation; Configure adjustable parameters and data input / output formats for each computational logic unit, and encapsulate the configured computational logic unit into an independent computational atomic service. Each computational atomic service can adjust parameters or replace logic individually according to computational needs, thereby achieving flexible configuration of computational logic.

[0046] In this embodiment, the present invention strictly follows the "Water Resources Bulletin Compilation Regulations" for calculation logic conversion. First, based on the core indicator system and corresponding calculation methods specified in the regulations, such as precipitation and water resources, the full calculation logic for bulletin compilation is broken down into single-dimensional calculation logic units. Each unit matches the basic calculation operations specified in the regulations. Then, according to the regulations' requirements for the calculation accuracy, statistical caliber, and output format of each indicator, parameter items and standardized input / output formats are configured for each logic unit, with parameter adjustment strictly limited to the range allowed by the regulations. Finally, the configured logic units are encapsulated as independent calculation atomic services. The calculation rules and parameter settings of these services conform to the requirements of the regulations. Individual services can independently adjust parameters or replace logic, ensuring that the calculation logic of all atomic services is highly consistent with the specifications of the "Water Resources Bulletin Compilation Regulations," laying a compliant foundation for subsequent indicator calculations.

[0047] Specifically, based on the index compilation specifications of the water resources bulletin, the pre-set full-scale calculation logic of the bulletin compilation is decomposed. The calculation dimensions are divided according to core indicators such as areal rainfall, river runoff, groundwater resources, and water use efficiency. Each index dimension is further decomposed into a single-dimensional calculation logic unit. Each calculation logic unit corresponds to only one basic calculation operation. After decomposition, a unique logical code is assigned to each calculation logic unit. The code consists of an index type code and an operation sequence code, totaling 12 characters, to ensure that each calculation logic unit is independent and identifiable, without logical overlap or functional redundancy.

[0048] For each decomposed computational logic unit, adjustable computational parameters and standardized data input / output formats are configured. The parameters cover configurable content such as computational thresholds, accuracy requirements, and spatiotemporal granularity. Each parameter has a reasonable adjustment range. Numerical parameters are retained to three decimal places. Spatiotemporal granularity parameters support options such as day, month, year, location, and administrative region. The data input format is uniformly set to the standard time-series data format, including fixed fields such as data identifier, spatiotemporal attributes, and numerical ontology. The data output format is set according to the water resources bulletin compilation specifications, including mandatory fields such as indicator code, calculation result, and calculation time. This completes the parameter and format configuration for all computational logic units.

[0049] After configuration, the computing logic units are independently packaged to form computing atomic services. During the packaging process, each computing atomic service is configured with a service identifier consistent with the corresponding logic unit, and a one-to-one correspondence between the service identifier and the logic code is established. At the same time, each computing atomic service is configured with an independent operating environment and resource allocation standard. The maximum memory usage of a single service is set to 2GB, and the maximum concurrency of a single service is set to 50 times / second, to ensure that each computing atomic service runs independently without interfering with each other.

[0050] All encapsulated atomic computing services are registered and managed. Information such as service identifier, parameter configuration range, and data input / output format is entered into the atomic service management module. Each atomic computing service can adjust the computing rules by modifying the parameter configuration items in the module, or update the service logic by replacing the corresponding logical code computing logic unit. No other atomic computing services or the overall computing framework need to be modified, thus completing the flexible configuration of computing logic and enabling independent invocation, adjustable parameters, and replaceable logic of atomic computing services.

[0051] Optionally, step S4, which involves building a computational model library and orchestrating an indicator calculation workflow based on that library, includes: The computational model library is divided into a quantum model library for areal rainfall, a sub-model library for river runoff, a quantum model library for groundwater resources, and a sub-model library for water use efficiency, based on the index types in the water resources bulletin. Each sub-model library contains multiple computational models adapted to different computational scenarios. Configure scenario matching calculation rules and calculation parameter thresholds for each calculation model, establish the association mapping relationship between atomic calculation services and sub-model library, and the calculation engine can automatically match the corresponding calculation model according to the characteristics of input data, or manually specify the calculation model to complete the adaptation.

[0052] In this embodiment, a calculation model library is built according to the types of indicators in the water resources bulletin, and divided into sub-model libraries for areal rainfall, river runoff, groundwater resources, and water use efficiency. Each sub-model library contains multiple calculation models adapted to different calculation scenarios. The areal rainfall quantum model library is equipped with Thiessen polygon method, arithmetic mean method, and grid method. The river runoff sub-model library is equipped with cross-sectional flow-runoff conversion model and digital filtering baseflow segmentation model. The groundwater resources quantum model library is equipped with water balance principle calculation model. The water use efficiency sub-model library is equipped with industry water use quota accounting model.

[0053] Configure scenario matching calculation rules and calculation parameter thresholds for each calculation model. Set matching rules for the areal rainfall model according to station density, set a filtering parameter threshold of 0.9-0.95 for the digital filtering method model, set a precipitation infiltration coefficient threshold of 0.1-0.4 for the water balance principle model, and configure quota parameter thresholds for the water use efficiency model according to water use type. Clarify the applicable scenarios and parameter adjustment ranges for each model.

[0054] Based on the computational logic of the atomic computing service and the applicable scenarios of the sub-model library, an association mapping relationship is established between the two. A 20-bit unique identifier containing the service code and the model code is configured for each mapping relationship. The mapping relationship and identifier information are entered into the mapping database of the model scheduling module to achieve accurate association between services and models.

[0055] After receiving spatiotemporally fused water resource data, the computing engine extracts feature information such as monitoring station density, data type, and monitoring range. The model scheduling module matches the scenario calculation rules based on the feature information and automatically retrieves the corresponding calculation model from the mapping database. At the same time, a manual intervention channel is retained, with manually specified models having higher priority than automatically matched models, thus completing the flexible adaptation of the calculation model and forming a complete indicator calculation workflow.

[0056] Of particular importance is that step S4, which involves orchestrating the indicator calculation workflow based on the calculation model library, specifically includes: Retrieve registered computational atom services using a visual directed acyclic graph orchestration tool; According to the calculation level and business logic of the water resources bulletin indicators, the calculation atomic services are dragged and dropped to connect and the execution order is set. Data transmission paths and data format conversion rules are configured for adjacent calculation atomic services. Bind the computing models in the computing model library corresponding to each computing atomic service to generate an automatically executed and traceable indicator calculation workflow. The indicator calculation workflow supports adding, deleting, reordering and replacing nodes.

[0057] In this embodiment, a visual directed acyclic graph orchestration tool is used to retrieve all registered computational atomic services in the atomic service management module. The tool synchronously loads the unique code, computational logic, data input / output fields, and parameter configuration range of each service, and displays them in categories according to indicator types such as areal rainfall and river runoff. A precise service code retrieval function is configured, and the retrieval response time is controlled within 1 second. At the same time, the bound model types and applicable scenarios of each service are displayed.

[0058] Based on the calculation hierarchy and business logic of the water resources bulletin indicators from basic to comprehensive, the calculation atomic services are dragged and connected in the arrangement tool interface. The areal rainfall and evaporation calculation atomic services are set as first-level calculation nodes, the river runoff and groundwater resources calculation atomic services are set as second-level calculation nodes, and the total water resources and water use efficiency calculation atomic services are set as third-level calculation nodes. Dedicated point-to-point data transmission paths are configured for adjacent calculation atomic services, and the peak value of single-path data transmission is set to 10MB / s. At the same time, a unified standard time-series data format conversion rule is configured to realize automatic field matching and format adaptation between the output data of the preceding node and the input data of the following node.

[0059] Based on the pre-defined association mapping between computation atomic services and computation model libraries, each computation atomic service is bound to a corresponding computation model in the sub-model library in the orchestration tool. The binding operation is completed through a 1:1 precise matching of service codes and model codes. After binding, a full set of configuration information is generated, including computation nodes, execution order, transmission path, bound models, and format conversion rules. This information is synchronized to the workflow scheduling module of the computation engine in real time, and a 16-bit unique scheduling code is assigned to each computation node. The code includes node level code, type code, and sequence code, enabling full-process traceability of the computation process.

[0060] The generated indicator calculation workflow supports visual and flexible operation. In the orchestration tool, calculation nodes can be added, deleted, or selected in batches. The execution order of nodes can be adjusted directly by dragging and dropping node links. The calculation models bound to each node can be replaced by re-selecting the model code. All operation instructions are synchronized to the workflow scheduling module in real time and take effect immediately. Operation records are automatically retained by timestamp. The storage time for historical configuration versions is set to 12 months to meet the flexible adjustment needs of water resources bulletin indicator calculation in different scenarios.

[0061] Optionally, in step S4, the calculation is completed by calling the spatiotemporally fused water resources data, and the specific calculation results of the output bulletin indicators are as follows: After calling the spatiotemporally fused water resources data, the data is targeted and sliced ​​according to the calculation requirements of each node in the water resources bulletin's indicator calculation workflow. The sliced ​​data is pushed to the corresponding computing nodes, and the model scheduling module of the computing engine loads the matching computing model, driving the computing execution module to complete the computing of each node according to the workflow execution order; The result verification module performs compliance verification on the intermediate calculation results of each node. After the verification is passed, the intermediate results are passed to the next calculation node. After all nodes have completed their calculations, the final results are integrated. The final results are structured and packaged according to the indicator format requirements of the water resources bulletin, and standardized calculation results of bulletin indicators that can be directly used for bulletin preparation are output.

[0062] In this embodiment, after the computing engine calls the spatiotemporally fused water resources data, it performs targeted filtering and slicing of the data according to the computing needs of each node in the water resources bulletin index calculation workflow, based on the index type, spatiotemporal range, and data dimension. The data is divided into subsets according to index types such as areal rainfall and river runoff, and the data is sliced ​​according to the time granularity of day, month, and year and the spatial range of provincial and municipal administrative regions. The sliced ​​data retains core fields such as data identifier, spatiotemporal attributes, and numerical ontology, and the data size of a single slice is controlled within 5MB.

[0063] The sliced ​​data is pushed to the corresponding computing nodes through a point-to-point transmission path. The model scheduling module of the computing engine loads the matching computing model and initializes the computing parameters according to the mapping relationship bound to the nodes. It drives the computing execution module to perform parallel or serial computing on the data of each node according to the execution order of the workflow, from basic to comprehensive. The maximum number of parallel computing nodes that can be executed at the same time is set to 20. The serial computing nodes strictly follow the execution rule of starting only after the previous node has completed its calculation.

[0064] The result verification module calculates compliance standards according to preset indicators and performs full verification of the intermediate calculation results of each node. The verification content includes numerical range, calculation accuracy, and logical consistency. The numerical accuracy is uniformly retained to two decimal places. The intermediate results that pass the verification are automatically passed to the next calculation node. The node that fails the verification is triggered to recalculate. The maximum number of recalculations is set to 3.

[0065] After all computing nodes have completed their calculations, the system automatically integrates the final calculation results from each node. According to the indicator format requirements of the water resources bulletin, the system configures the necessary attributes such as indicator codes, statistical spatiotemporal ranges, and calculation methods for the results, and performs structured encapsulation. The encapsulated results are stored in a standard time-series data format, with fields including indicator name, indicator value, statistical unit, calculation time, and data source. Finally, the system outputs standardized water resources bulletin indicator calculation results that can be directly used for bulletin compilation.

[0066] Preferably, step S5: store the calculation results of the bulletin indicators to a high-performance time series database, visualize and interactively analyze the calculation results of the bulletin indicators, and generate and publish a dynamic water resources bulletin.

[0067] Optionally, step S5 includes the following steps: The calculation results of the bulletin indicators are partitioned and stored in a high-performance time series database according to the time dimension, regional dimension and indicator dimension. A visualization query engine is built based on the partition index of a high-performance time series database. The query engine retrieves the calculation results of the corresponding dimensions and performs multi-form visualization rendering. An interactive analysis channel is built by combining visualization results, and a dynamic water resources bulletin is generated and published after the analysis is completed.

[0068] In this embodiment, the calculation results of the bulletin indicators are partitioned and stored in a high-performance time-series database according to three dimensions: time, region, and indicator. The time dimension is divided into storage partitions according to seconds, hours, days, months, and years. The region dimension is divided into storage partitions according to the administrative levels of provinces, cities, counties, and townships and hydrological basins. The indicator dimension is divided into storage partitions according to the core indicators of the bulletin, such as precipitation, river runoff, and groundwater resources. Each partition establishes an incremental storage index according to the data generation time. When storing a single data record, it is associated with traceability information such as a unique identifier, calculation node, and data source. The storage capacity threshold for a single table in the database is set to 1000GB.

[0069] A visualization query engine is built based on the partition index of a high-performance time-series database. The engine connects to the partition indexes of various dimensions of the database and is configured with both precise and fuzzy query modes. The query conditions support multiple combinations of filtering based on time range, administrative region, and indicator type. The query response time is controlled within 1 second. After retrieving the calculation results of the corresponding dimensions, the engine performs various forms of visualization rendering according to the display requirements. The rendering types include time series charts, bar charts, spatial heat maps, and GIS thematic maps. The numerical precision and display granularity of the rendered graphics can be adjusted as needed.

[0070] A multi-level interactive analysis channel is built based on the visualization rendering results. The channel supports clicking and querying the visualization graphics and drilling down to analyze them. It can drill down from the provincial level to the county level, and from the annual indicators to the hourly indicators. It also supports cross-dimensional comparative analysis. During the analysis, the source information of indicator calculation can be retrieved in real time. All analysis operations are recorded and the record storage time is set to 12 months.

[0071] After completing the interactive analysis, the system automatically integrates the visualization results and analysis data according to the water resources bulletin compilation specifications, and generates a dynamic water resources bulletin containing indicator data, statistical charts, and text descriptions. The bulletin supports two publishing formats: PDF and web page. The web page version of the bulletin enables real-time data updates. Before publication, the bulletin content is fully verified. After verification, it is synchronously published to the web portal. After publication, the data is updated incrementally on a daily basis to ensure the timeliness and completeness of the bulletin.

[0072] Please see Figure 2 This diagram illustrates the flow of the core calculation method of this invention. The process begins with the access of multi-source data. After data cleaning and standardization, key spatiotemporal fusion and correlation are performed. Then, the core indicator calculation stage begins. The calculation results are subject to compliance judgment (such as whether they conform to business logic). If they do not conform, feedback is provided to adjust the model. If they conform, the results are stored and the final public product is generated. The entire process forms a cyclical and optimizable closed-loop system.

[0073] Please see Figure 3This diagram illustrates the overall technical architecture of the present invention, which adopts a layered design concept and is divided into four logical layers from top to bottom: Application Presentation Layer: Serving as a unified portal to the outside world, it provides web-based querying of dynamic water resources bulletins, multi-dimensional data visualization, interactive analysis, and report generation and download functions, and is the user interface.

[0074] The Compute & Service Layer serves as the core business capability center, encapsulating indicator calculation engines (such as calculation models for precipitation, runoff, and water consumption), data service APIs (providing standardized data via RESTful interfaces), and spatial analysis services (performing spatial queries and calculations based on GIS), providing powerful computing power and data service support for upper-layer applications.

[0075] Data Fusion & Governance Layer: As a data processing plant, it is responsible for extracting value from the raw data received at the bottom layer. This includes multi-source data fusion (especially spatiotemporal alignment), implementing strict data quality checks (handling outliers and missing values), and unified metadata and master data management to ensure data consistency and accuracy.

[0076] Data Access & Storage Layer: This layer is responsible for interfacing with various heterogeneous data sources. Through technologies such as real-time data acquisition (interfacing with APIs and data streams), batch data synchronization (ETL / ELT tools), and message queues (e.g., Kafka), it reliably accesses and stores data from multiple heterogeneous data sources, including meteorological, hydrological, and water usage reporting systems, at the system's underlying layer.

[0077] The data flow proceeds from bottom to top (from the data source through various processing layers to the application layer), progressing layer by layer, reflecting the complete transformation chain from raw data to decision-making knowledge. The modular design of each layer, with clear responsibilities, embodies a complete closed loop from data to value.

[0078] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of the equivalents of the application are intended to be included within the invention.

[0079] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features of the invention herein.

Claims

1. A method for calculating real-time water resources bulletin data based on multi-source heterogeneous data fusion, characterized in that, Includes the following steps: Step S1: Build a pluggable and high-concurrency unified data access gateway, and configure multiple types of adapters within the unified data access gateway; The system completes the access and extraction of multi-source heterogeneous data through each adapter, and adds spatiotemporal tags and metadata tags to the data extracted by the unified data access gateway to obtain the initial data asset catalog. Step S2: Establish a multi-dimensional data quality assessment matrix, automatically process and standardize the data in the initial data asset catalog, and output standardized water resource data; Step S3: Build the core data lake for spatiotemporal indexing, import standardized water resource data to complete spatiotemporal alignment and entity association, construct a water resource knowledge graph, and output spatiotemporally integrated and associated water resource data; Step S4: Determine the configurable water resources index calculation engine, convert the calculation logic of the preset bulletin compilation into calculation atomic services, build a calculation model library and arrange the index calculation workflow based on the calculation model library, call the spatiotemporally fused and correlated water resources data to complete the calculation, and output the bulletin index calculation results. Step S5: Store the calculation results of the bulletin indicators to a high-performance time series database, visualize and interactively analyze the calculation results of the bulletin indicators, and generate and publish dynamic water resources bulletins.

2. The method for calculating real-time water resources bulletin data based on multi-source heterogeneous data fusion according to claim 1, characterized in that, Step S1 includes the following steps: The various types of adapters configured within the unified data access gateway are dedicated processing modules for adapting streaming data, API interfaces, batch data, and unstructured data. Each dedicated processing module performs protocol parsing and data extraction on the corresponding type of multi-source heterogeneous data; After extraction, each piece of multi-source heterogeneous data is assigned a unique data ID, and then bound with spatiotemporal tags and metadata tags; A structured index is created based on data ID and tag information to obtain the initial data asset catalog.

3. The method for calculating real-time water resources bulletin data based on multi-source heterogeneous data fusion according to claim 1, characterized in that, Step S2 includes the following steps: The established multi-dimensional data quality assessment matrix covers multiple verification dimensions of data format, spatiotemporal, and numerical values. A full scan and dimensional verification of the data in the initial data asset catalog were performed based on a multi-dimensional data quality assessment matrix. Based on the verification results, the data in the initial data asset catalog undergoes automated processing for format correction, spatiotemporal completion, and numerical calibration. The data that has been processed automatically will be converted into a standard time-series format, and the timestamps, spatial coordinates and numerical units will be standardized to output standardized water resources data.

4. The method for calculating real-time water resources bulletin data based on multi-source heterogeneous data fusion according to claim 1, characterized in that, The spatiotemporal index core data lake built in step S3 is specifically as follows: The system combines a pre-defined standardized spatiotemporal network index and a vector administrative division index, and assigns unique spatiotemporal attribute identifiers to each index node. Extract the spatiotemporal attributes of standardized water resources data collection, and match the standardized water resources data to the corresponding nodes in the dual-index system based on the spatiotemporal attributes of collection. Standardized water resource data within each dual-index system node are resampled and aggregated according to a preset unified time and spatial granularity.

5. The method for calculating real-time water resources bulletin data based on multi-source heterogeneous data fusion according to claim 1, characterized in that, Step S3, importing standardized water resources data to complete spatiotemporal alignment and entity association, includes: In the entity association matching stage, monitoring stations, water conservancy projects, administrative units, and water resource management objects are taken as core entities; a two-way association relationship of spatial affiliation and hydrological association is established among the core entities; Bind the two-way correlation relationship with the spatiotemporally integrated water resource data; Based on the binding results, a water resources knowledge graph is constructed, and spatiotemporally integrated water resources data is output.

6. The method for calculating real-time water resources bulletin data based on multi-source heterogeneous data fusion according to claim 1, characterized in that, The configurable water resource index calculation engine in step S4 has a modular architecture, including an atomic service management module, a model scheduling module, a calculation execution module, and a result verification module. The atomic service management module is responsible for the registration, configuration, and invocation management of computation atomic services; The model scheduling module is responsible for loading and matching the computational model library. The computation execution module is responsible for driving the parallel and serial execution of computation tasks; The result verification module is responsible for real-time verification of the calculation process and results; Each module achieves data exchange and functional linkage through standardized interfaces.

7. The method for calculating real-time water resources bulletin data based on multi-source heterogeneous data fusion according to claim 1, characterized in that, In step S4, the computational logic compiled from the preset bulletin is converted into computational atomic services as follows: The calculation logic for compiling the pre-set bulletin is broken down into single-dimensional calculation logic units according to the water resources bulletin indicator compilation specifications, with each calculation logic unit corresponding to a basic calculation operation; Configure adjustable parameters and data input / output formats for each computational logic unit, and encapsulate the configured computational logic unit into an independent computational atomic service. Each computational atomic service can adjust parameters or replace logic individually according to computational needs, thereby achieving flexible configuration of computational logic.

8. The method for calculating real-time water resources bulletin data based on multi-source heterogeneous data fusion according to claim 7, characterized in that, Step S4, which involves building a computational model library and orchestrating an index calculation workflow based on that library, includes: The computational model library is divided into a quantum model library for areal rainfall, a sub-model library for river runoff, a quantum model library for groundwater resources, and a sub-model library for water use efficiency, based on the index types in the water resources bulletin. Each sub-model library contains multiple computational models adapted to different computational scenarios. Configure scenario matching calculation rules and calculation parameter thresholds for each calculation model, establish the association mapping relationship between atomic calculation services and sub-model library, and the calculation engine can automatically match the corresponding calculation model according to the characteristics of input data, or manually specify the calculation model to complete the adaptation.

9. The method for calculating real-time water resources bulletin data based on multi-source heterogeneous data fusion according to claim 1, characterized in that, In step S4, the spatiotemporally fused water resources data is used to complete the calculation, and the specific calculation results of the output bulletin indicators are as follows: After calling the spatiotemporally fused water resources data, the data is targeted and sliced ​​according to the calculation requirements of each node in the water resources bulletin's indicator calculation workflow. The sliced ​​data is pushed to the corresponding computing nodes, and the model scheduling module of the computing engine loads the matching computing model, driving the computing execution module to complete the computing of each node according to the workflow execution order; The result verification module performs compliance verification on the intermediate calculation results of each node. After the verification is passed, the intermediate results are passed to the next calculation node. After all nodes have completed their calculations, the final results are integrated. The final results are structured and packaged according to the indicator format requirements of the water resources bulletin, and standardized calculation results of bulletin indicators that can be directly used for bulletin preparation are output.

10. The method for calculating real-time water resources bulletin data based on multi-source heterogeneous data fusion according to claim 1, characterized in that, Step S5 includes the following steps: The calculation results of the bulletin indicators are partitioned and stored in a high-performance time series database according to the time dimension, regional dimension and indicator dimension. A visualization query engine is built based on the partition index of a high-performance time series database. The query engine retrieves the calculation results of the corresponding dimensions and performs multi-form visualization rendering. An interactive analysis channel is built by combining visualization results, and a dynamic water resources bulletin is generated and published after the analysis is completed.

Images