Fusion method, device and equipment based on multi-source water conservancy data and medium

Abstract

The invention discloses a fusion method and device based on multi-source water conservancy data, equipment and a medium, and relates to the technical field of water conservancy data processing, and the technical scheme is characterized in that the method comprises the steps: collecting water conservancy original data, carrying out the classified storage of the water conservancy original data according to a water conservancy data specification, and obtaining a water conservancy classification data set; preprocessing the water conservancy classification data set to obtain a preprocessed water conservancy classification data set, combining similar data of multiple sources, and carrying out missing compensation processing on the similar data to obtain a fused similar data set; and carrying out hierarchical modeling on the fused data sets of the same kind by utilizing distributed processing and a multi-bin hierarchical architecture design to obtain a standardized unified water conservancy data mart. The system has the advantages of high-efficiency water conservancy data acquisition, high-quality similar data fusion, high-standard heterogeneous data fusion, high-availability water conservancy data service and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of water conservancy data processing, more specifically, it relates to a fusion method, device and equipment based on multi-source water conservancy data and a medium. BACKGROUND

[0002] Water business data has the characteristics of multiple sources and multiple types, among which the multiple sources refer to collecting water conservancy data by different systems and devices and storing them into different databases, and the multiple types refer to that the water conservancy data has different structures, formats and manifestations. The water conservancy heterogeneous data is mainly divided into three types of structured data, semi-structured data and unstructured data, among which the structured data is mainly stored in the form of a database table, the semi-structured data is mainly stored by using XML documents and JSON documents, and the unstructured data is mainly stored by using Excel files or PDF files. In view of the characteristics of the water business data being massive, multi-source and heterogeneous, an "information island" is easily formed, so it is necessary to fuse the data layer of the water conservancy data. Therefore, how to efficiently integrate, uniformly manage and fully mine the value of the multi-source heterogeneous water conservancy data has become the core proposition of building the water conservancy data driving capability.

[0003] In the actual fusion process of the water conservancy data, the following four problems are often encountered. First, the water conservancy data format is various: the formats of the water conservancy data from the national hydrology, various river hydrology and various provinces are inconsistent, the structured, semi-structured and unstructured water conservancy data coexist, and the number, name, type and time format of the field data lack unified standards; second, the water conservancy data completeness and accuracy are low: the data quality of the national hydrology, various river hydrology and various provincial hydrology is inconsistent, and there are cases of missing, redundant, contradictory and even conflicting water conservancy field data, thereby affecting the effective use and accurate analysis of the water conservancy data; third, the water conservancy data integration degree is low: various types of water conservancy data are relatively dispersed, there is no unified framework for uniformly managing various types of water conservancy data, and there is no unified water conservancy data governance system; fourth, the query performance is low: the water conservancy data is stored in different databases and file systems, and cross-source query has the conditions of high delay and large resource consumption, and the data is difficult to be accessed and analyzed in real time.

[0004] In view of this, the present application provides a fusion method, device, equipment and medium based on multi-source water conservancy data, establishes a unified multi-source water conservancy data fusion system, forms a water conservancy data analysis chain, so as to provide high-quality water conservancy data services to the outside, and further deeply mine the value of the water conservancy data SUMMARY

[0005] The above technical purpose of the present application is realized by the following technical scheme: The first aspect of the present application provides a fusion method based on multi-source water conservancy data, comprising the following steps: Collect water conservancy original data, store the water conservancy original data according to water conservancy data specification, and obtain water conservancy classification data set; The water conservancy classification data set is preprocessed to obtain a preprocessed water conservancy classification data set, the same kind of data from multiple sources is merged, the same kind of data is compensated for missing, and a fused same kind of data set is obtained. The fused same kind of data set is hierarchically modeled by using distributed processing and a data warehouse hierarchical architecture design, and a standardized unified water conservancy data mart is obtained.

[0006] In combination with the first aspect, the application further provides that the water conservancy original data includes basic data, geographic data and time series data.

[0007] In combination with the first aspect, the application further provides that the water conservancy data specification is a water conservancy industry standard.

[0008] In combination with the first aspect, the application further provides that the preprocessing includes converting data types and data formats, testing abnormal values, and removing and correcting abnormal points, and de-duplicating and fusing data.

[0009] In combination with the first aspect, the application further provides that the abnormal value testing adopts a local outlier factor abnormal value testing algorithm, and the abnormal point removing and correcting adopts a LOF method.

[0010] In combination with the first aspect, the application further provides that the merging of the same kind of data from multiple sources adopts a deep clustering algorithm.

[0011] In combination with the first aspect, the application further provides that it further includes establishing a standardized data external service, the data external service including abstracting and managing water conservancy data, establishing a corresponding data source access method, and constituting an API interface for data access.

[0012] The second aspect of the application further provides a device / apparatus / system based on the fusion of multi-source water conservancy data, including a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to realize the steps of the above method.

[0013] The third aspect of the application further provides a computer readable storage medium having a computer program / instruction stored thereon, and the computer program / instruction is executed by a processor to realize the steps of the above method.

[0014] The fourth aspect of the application further provides a computer program product, including a computer program / instruction, and the computer program / instruction is executed by a processor to realize the steps of the above method.

[0015] In summary, the application has the following advantages: This invention boasts advantages such as high-efficiency water conservancy data acquisition, high-quality fusion of similar data, high-standard fusion of heterogeneous data, and highly available water conservancy data services. First, high-efficiency water conservancy data acquisition primarily refers to the real-time acquisition of structured, semi-structured, and unstructured data from multiple sources using various methods such as website data crawling, OCR recognition, and file uploading. This constructs a standardized water conservancy data system, breaks down data silos, and achieves interconnectivity between water conservancy data. Second, high-quality fusion of similar data addresses the inconsistency in the information levels of data from multiple sources by employing deep clustering algorithms to fuse similar water conservancy data. This minimizes the impact of misinterpretations on the fusion results, making the results more accurate. Third, high-standard heterogeneous data fusion utilizes distributed processing technology and data warehouse hierarchical modeling technology to construct a standardized water conservancy data processing workflow, ultimately forming a standardized data mart, thereby improving the efficiency and flexibility of heterogeneous data fusion. Fourth, highly available water conservancy data services refer to the ability to continuously provide high-quality water conservancy data services while ensuring data security. Attached Figure Description

[0016] Figure 1 This is the target of water conservancy data fusion in the embodiments of the present invention; Figure 2 This is the multi-source water conservancy data fusion process in this embodiment of the invention; Figure 3 This is a diagram of the multi-source water conservancy data fusion architecture in an embodiment of the present invention; Figure 4 This is a flowchart of similar data fusion in an embodiment of the present invention; Figure 5 This is a diagram of the water conservancy big data warehouse architecture in an embodiment of the present invention; Figure 6 This is a flowchart of a method for fusing multi-source water conservancy data in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] This invention aims to construct a multi-source data fusion process and design a multi-source data fusion mechanism that integrates data verification, data processing, and data aggregation. It solves the problems of lack of unified standards, low data accuracy, and difficulty in effective sharing in existing water conservancy data fusion, improves the construction efficiency of water conservancy data fusion and the convenience of data sharing, and thus realizes the effective utilization of water conservancy data.

[0019] The goals of water resources data fusion mainly revolve around five themes: data sources, data management, data application, data efficiency, and data services. It features rapid data collection, intelligent information management, easy-to-use knowledge, widespread application, and social benefits. It can quickly aggregate standardized data into a database and closely integrate water resources data with water resources operations, making data management more efficient, high-quality, and intelligent. It can maximize the value of water resources data, continuously enrich water resources data services and application products, and fully leverage the industry effectiveness of water resources data in public services, such as... Figure 1 As shown in the figure. The following description is based on an embodiment.

[0020] Example 1: A method based on the fusion of multi-source water conservancy data, such as Figure 6 As shown, it includes the following steps: S100. Collect raw water conservancy data, classify and store the raw water conservancy data according to the water conservancy data specifications, and obtain a water conservancy classification dataset; S200. Preprocess the water conservancy classification dataset to obtain the preprocessed water conservancy classification dataset, merge similar data from multiple sources, perform missing data compensation for similar data, and obtain the fused similar dataset. S300. By utilizing distributed processing and a hierarchical data warehouse architecture design, the merged datasets of the same type are hierarchically modeled to obtain a standardized unified water conservancy data mart; S400. Establish standardized data services for external users.

[0021] In step S100 of this embodiment, the raw water conservancy data has the characteristics of multiple sources, multiple storage formats, and multiple usage methods, and can therefore be divided into basic data, geographic data, and time-series data. Basic data mainly includes fundamental data such as rivers, lakes, water conservancy projects, and monitoring stations. This type of data is determined during the design phase and does not change significantly over time; it is generally stored in the databases or file systems of the relevant departments. Basic geographic data mainly involves digital elevation models (DEMs), land use / land cover change (LUCC) data, and remote sensing imagery. This type of data is characterized by different resolutions; resolution is directly proportional to data quality and value, and inversely proportional to data storage size, therefore, the resolution needs to be reasonably determined. Time-series data includes both historical data and real-time data. Historical data mainly refers to historical monitoring data, which is characterized by a large data volume and is generally stored in the database of the hydrological bureau or in the form of statistical yearbooks. Real-time data mainly refers to real-time monitoring data, which is dynamically collected hourly by automated monitoring equipment, and then the collected data is stored in a real-time monitoring database and made public on the corresponding hydrological website. Based on the characteristics of the above data, data on watersheds (BAS), rivers (RV), lakes (LK), reservoirs (RES), reservoir dams (DAM), hydrological monitoring stations (ST), soil and water conservation monitoring stations (WSST), and water diversion projects (WADI) for various provinces can be obtained.

[0022] Because multi-source data suffers from problems such as poor consistency, difficulty in communication, incompatibility, and information silos, it is necessary to formulate corresponding water resources data standards to address these issues. Water resources data standards must adhere to water resources industry standards and mainly involve basic data on water resources objects, real-time rainfall and water level data, dam safety monitoring data, water conservancy project construction and management data, hydrological information forecasting data, and geographic information data.

[0023] The formulation of water resources data specifications primarily defines the data table structure and identifiers. Database naming identifiers consist of uppercase English letters, numbers, and underscores, are unique, and can be divided into table identifiers and field identifiers. Table identifiers consist of three parts connected by underscores: a professional classification code, a subject identifier, and a classification suffix. The format is: Professional Classification Code_Subject Identifier_Classification Suffix, for example, ST_STBPRP_B, where ST is the professional classification code, representing the real-time rainfall and water level database; STBPRP is the table code, representing station information; and B is the table identifier classification suffix, indicating the basic information class. Field identifiers are named according to national water resources industry standards, involving field name, field meaning, unit, data type, length, whether it is a primary key, whether it is nullable, whether it is an index, whether it is a key field, and remarks, etc. Based on this, metadata management can be constructed, such as data standards, data dictionaries, and data lineages, thereby creating corresponding database tables. It should be noted that if the data volume is large enough, database sharding and table partitioning can be constructed for storage to facilitate subsequent queries.

[0024] Because water resources data plays a fundamental role in water resources informatization, it is necessary to collect this data. Based on this, it is necessary to resolve conflicts between heterogeneous water resources data from multiple sources by employing database technologies, in accordance with the aforementioned data specifications and standards. Automated data processing technologies should be used to store the collected data in the target data source, and timestamps and other technologies should be used to update the water resources data, ensuring the final consistency of the data in the target data source. Water resources data sources can be categorized into website data, image data, and file data based on their data storage format. Website data mainly consists of real-time monitoring data (such as hydrological station data and meteorological station data), stored primarily in XML files, HTML pages, or HTTP data packets. Panda, BeautifulSoup, and Requests libraries are used to acquire the data, respectively. This process requires first analyzing the webpage structure, locating and annotating webpage elements, then writing scripts to simulate click operations, perform logical judgments, and loop pagination to acquire the annotated information, and finally configuring the database connection, running the data acquisition task to collect the required data into the database, and then verifying it. Given that the update frequency of the aforementioned data is primarily hourly or daily, a scheduled task needs to be built to promptly retrieve the latest data and store it in the corresponding database tables. A failure delay and retry strategy needs to be implemented to address occasional website access failures. File data mainly consists of historical data, stored primarily in Excel or CSV files, and can therefore be directly written to the database. Image data mainly comprises basic data and geographic information data. Basic data is stored as tabular images; OCR technology is used to scan and recognize text in the images, converting them into CSV files before writing the data to the database. Geographic information data mainly includes digital elevation models (DEMs), administrative division data, and national eight-level river system boundaries and surface data, primarily stored as image files. This method addresses the challenges of large data volume, frequent dynamic updates, and high heterogeneity in multi-source water resources data, providing strong support for convenient management and improved utilization of water resources data.

[0025] In step S200 of this embodiment, due to differences in the informatization levels of multiple water conservancy information data sources, data loss or errors are prone to occur during the actual data collection process. These issues can alter the distribution and characteristics of water conservancy data. Therefore, quality inspection and processing of similar water conservancy data are necessary. The evaluation indicators for data quality include accuracy, consistency, and completeness. Accuracy refers to the significant differences in the same type of data provided by multiple water conservancy data sources due to different data collection and processing methods. Consistency refers to the consistency between the data from the data source and the final acquired data, especially for time-series data, ensuring consistency over time. Completeness refers to the absence of anomalies such as duplication and loss in water conservancy data due to uncertainties during collection, transmission, and storage.

[0026] The preprocessing includes data type and format conversion, outlier detection, outlier removal and correction, and data deduplication and fusion. Specifically: During the water resources data collection process, errors in data type and format may occur. Therefore, it is necessary to convert data types and formats in advance and identify and handle them during data integration. Furthermore, false positives may occur during water resources data collection. Therefore, a Local Outlier Detection (LOF) algorithm is needed to detect outliers. Here, the LOF method in Python is used to remove and correct outliers. Since water resources data is obtained from different sources, duplicate data may exist. Therefore, a temporary database table needs to be created to store data from multiple sources. Then, grouping operations are performed on this temporary table to achieve data deduplication and fusion, before storing the data in a new table. It is important to note that a unique index should be created in the new table to ensure data uniqueness. During this process, due to the lag between website data publication and data acquisition, to ensure eventual data consistency, any updates to this data will be overwritten in the table.

[0027] In practical processing, the accuracy of data fusion results obtained through database operations is not high. Therefore, a deep clustering algorithm is used here to improve the accuracy of data fusion. For example... Figure 2 , 3 As shown in Figure 4, assume the number of similar data is... The collection of similar data is as follows: ; in, Indicates the first One data point, Indicates the first One data point, then and The similarity between them is: ; They can be combined into a similarity matrix as follows: ; in, It is mainly used to describe the similarity between similar data, among which The larger the element in the matrix, the higher the similarity. Sum and normalize each column of data to obtain ,in Given a value Only when the following conditions are met Time Retention All other data points were discarded. The remaining data points were... Calculate the local density for each data point. Distance between high-density points Then satisfy The data points are used to construct the initial cluster centers, where For the remaining data points The mean, For the remaining data points The standard deviation. Finally, select 10 data points were used as the initial cluster centers. .

[0028] use Indicates For a neural network with parameters, the deep clustering loss function is constructed as follows: ,here It is a membership matrix. It is a weight matrix.

[0029] Based on the clustering loss function described above, the fusion of similar data is performed through the following steps: First, the similarity of the given similar data needs to be calculated to obtain a similarity matrix, and then preprocessed; second, local density is utilized... Distance between and high density Number of clusters and cluster center First, initialize; second, adjust the weight matrix. and membership matrix Perform initialization and calculate the corresponding loss function. Fourth, iteratively update the cluster centers. Weight matrix and membership matrix Fifth, calculate the updated loss function. If the difference between the loss function before and after the update is less than a given constant... Or the number of updates exceeds If so, exit the iterative update loop; otherwise, continue the iterative update. Finally, take the cluster center with the most data points in the obtained clustering results as the final fusion result, such as... Figure 4 As shown.

[0030] Since directly minimizing the above loss function It is possible to obtain a degenerate solution, i.e., a neural network. It is possible to include all samples All maps to the same point, resulting in a loss function of 0. Therefore, a network loss needs to be added to eliminate degradation. The optimized loss function is thus: ; in, It is a membership matrix. It is a weight matrix. It is a data matrix. It is an orthogonal matrix. It is a constant. It is represented as the trace of a matrix.

[0031] Data sources may contain missing data due to mechanical reasons such as sensor malfunction preventing data collection, or non-mechanical reasons such as human error. Therefore, it is necessary to compensate for missing values. If a data source has missing data in a certain field, it can be compensated by using data from other data sources where that field is not missing. This helps to avoid the occurrence of null values ​​in a single field in the data fusion results, thus making the data fusion results more accurate.

[0032] In step S300 of this embodiment, in actual water conservancy business scenarios, it is necessary to acquire multiple types of water conservancy data simultaneously. Therefore, it is necessary to fuse multiple types of water conservancy data based on the aforementioned fusion of similar data. The Hadoop platform, as an important component of the water conservancy big data warehouse architecture, mainly provides a distributed storage and processing framework for massive water conservancy data. It not only meets the needs of structured CRUD operations but also has high computing performance and low storage costs. Because water conservancy data is of high value, a distributed strategy and a multi-replica strategy are required for storage to ensure data security. Therefore, an MPP database is used to store petabyte-level data, improving the system's high availability, high openness, and high flexibility. Since the amount of real-time water conservancy data is growing rapidly, it will put significant pressure on database resources. Therefore, a data lifecycle management strategy is adopted. This strategy mainly divides data into young generation and old generation based on the length of data storage time. The young generation has a high access frequency, while the old generation has a low access frequency. Therefore, the data in the old generation is gradually migrated to low-cost and low-performance storage devices, effectively relieving the access resource pressure on the main data warehouse. Based on this, a water conservancy big data warehouse architecture diagram is constructed, as follows:Figure 5 As shown.

[0033] The water resources data system is structured around four layers: a province-specific data collection layer, a data foundation layer, a data service layer, and a product application layer. The province-specific data collection layer primarily collects water resources data according to the dimensions of each province, covering data on river basins, rivers, lakes, reservoirs, dams, hydrological monitoring stations, soil and water conservation monitoring stations, and water diversion projects. The data foundation layer is mainly divided into four layers: ODS, DWD, DWS, and ADS. The ODS layer primarily handles the initial collection of raw data, file data parsing, and image data parsing. Building upon this, the DWD layer performs data cleaning, removing dirty and empty data, translating some enumerated values, and summarizing data according to themes such as station attributes, river water conditions, reservoir water conditions, and reservoir capacity curves. The DWS layer retains fewer dimensions, further summarizing data according to basic data, monitoring data, and business management data, providing common indicators for the upper layers. The ADS layer primarily uses broad tables organized around themes such as watershed flood control and water resource management and allocation to provide data services to the model platform (hydraulic professional models, intelligent models, etc.) and knowledge platform (hydraulic knowledge and knowledge engine, etc.) in the data service layer, indirectly serving the product application layer (watershed flood control and water resource management and allocation applications, etc.). The DIM layer mainly stores dimensional data, divided into high-cardinality dimensional data (mainly basic data tables for rivers, reservoirs, etc.) and low-cardinality dimensional data (mainly configuration tables for enumerated values, etc.). In addition, the operation and management platform provides services such as a unified business data portal and task scheduling management. This layered approach optimizes the heterogeneous data fusion process, improves the management and analysis workflow of water conservancy data, and thus enhances the accuracy of standardized water conservancy data.

[0034] In step S300 of this embodiment, the external data service mainly involves abstracting and managing water conservancy data, establishing corresponding data source access methods, and constructing a data access API interface. This API interface possesses capabilities such as service forwarding and geospatial service support to meet diverse external data needs. Furthermore, the external data service requires interface authentication and encryption of the interface response data. Only users with specific permissions can access the interface, thus ensuring the security of water conservancy data. The external data service utilizes blockchain technology to realize its application in the water conservancy field, simplifying the water conservancy data acquisition process and laying the foundation for the development of smart water conservancy applications. To facilitate daily water conservancy data information query and maintenance, a water conservancy data dashboard is developed using Spring Cloud and JavaScript. This dashboard uses reports to statistically analyze water conservancy data and monitors important indicators, enabling timely handling of any issues and making the external data service capabilities more robust.

[0035] Example 2: The present invention also provides an apparatus / device / system based on the fusion of multi-source water conservancy data, including a memory, a processor and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of the above method.

[0036] The present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0037] The present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0038] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A fusion method based on multi-source water conservancy data, characterized by: Includes the following steps: Collect raw water conservancy data, classify and store the raw water conservancy data according to water conservancy data specifications, and obtain a water conservancy classification dataset; The water conservancy classification dataset is preprocessed to obtain a preprocessed water conservancy classification dataset. Data of the same type from multiple sources are merged, and missing data of the same type are compensated to obtain a fused dataset of the same type. By utilizing distributed processing and a hierarchical data warehouse architecture, the merged datasets of the same type are modeled in a hierarchical manner to obtain a standardized unified water conservancy data mart.

2. The fusion method based on multi-source water conservancy data according to claim 1, characterized in that: The raw water conservancy data includes basic data, geographic data, and time-series data.

3. The fusion method based on multi-source water conservancy data according to claim 1, characterized in that: The water conservancy data specifications mentioned are water conservancy industry standards.

4. The fusion method based on multi-source water conservancy data according to claim 1, characterized in that: The preprocessing includes converting data types and formats, checking outliers, removing and correcting outliers, and deduplicating and merging data.

5. The fusion method based on multi-source water conservancy data according to claim 4, characterized in that: The outlier detection algorithm is used to detect outliers, and the Loopholes-of-Flight (LOF) method is used to remove and correct outliers.

6. The fusion method based on multi-source water conservancy data according to claim 1, characterized in that: The merging of similar data from multiple sources employs a deep clustering algorithm.

7. The fusion method based on multi-source water conservancy data according to claim 1, characterized in that: it also... include: Establish standardized data external services, which include abstracting and managing water conservancy data, establishing corresponding data source access methods, and forming a data access API interface.

8. A device / equipment / system based on the fusion of multi-source water conservancy data, including a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-7.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-7.

Images