Efficient data processing method and system based on multi-source data management and fusion

Abstract

The invention relates to the technical field of data fusion, in particular to an efficient data processing method and system based on multi-source data governance and fusion, and the method comprises the steps: collecting multi-source heterogeneous data; performing data quality evaluation on the collected multi-source heterogeneous data, and checking the integrity, accuracy and timeliness of the data; the method comprises the following steps: preprocessing multi-source heterogeneous data, performing data cleaning on redundant, repeated or inaccurate data after data quality evaluation, and converting the data into a required format or structure; fusing the preprocessed multi-source heterogeneous data to form standardized data; mining total element internal relations of the standardized data to obtain related elements; and constructing a data warehouse, and carrying out distributed storage on the data. According to the method, the database is established to realize multi-source heterogeneous data, and support is provided for digital twinning construction of the watershed.

Description

Technical Field

[0001] The present invention relates to the technical field of data fusion, and in particular to an efficient data processing method and system based on multi-source data management and fusion. Background Art

[0002] At present, my country's smart water conservancy construction has entered a critical period of high-quality development. Among them, the construction of digital twin river basins is the key and core of promoting smart water conservancy construction, and the fusion of multi-source heterogeneous data is the basis and necessary means to support the construction of digital twin river basins.

[0003] Multi-source data fusion technology is a technology that uses relevant means to integrate all the information obtained through investigation and analysis, and conducts a unified evaluation of the information to finally obtain unified information. The purpose of developing this technology is to integrate various different data information, absorb the characteristics of different data sources, and then extract unified, better and richer information from them than single data.

[0004] In existing technologies, multi-source data governance and integration face many challenges. First, data sources are extensive and heterogeneous, including geospatial data, basic data, monitoring data, business management data, and external shared data. These data have significant differences in format, structure, and quality, which increases the difficulty of data integration and processing. Secondly, data quality varies, and there are problems with completeness, accuracy, and timeliness, which directly affect the accuracy and reliability of data analysis and application. In addition, complex data relationships, including linear and nonlinear relationships, need to be processed during the data fusion process, and traditional data processing methods are difficult to effectively deal with these challenges.

[0005] Therefore, the present invention aims to provide an efficient data processing method and system based on multi-source data governance and fusion, so as to solve the above-mentioned problems existing in the prior art, improve the efficiency and accuracy of data processing, and reduce the cost of data processing. Summary of the invention

[0006] The purpose of the present invention is to provide an efficient data processing method and system based on multi-source data governance and fusion, so as to solve the technical problems of existing data governance and fusion methods, most of which have high cost, low efficiency and poor flexibility.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] In a first aspect, the present invention provides an efficient data processing method based on multi-source data management and fusion, comprising:

[0009] Collect heterogeneous data from multiple sources;

[0010] Conduct data quality assessment on the collected multi-source heterogeneous data to check the completeness, accuracy and timeliness of the data;

[0011] Pre-process multi-source heterogeneous data, clean the redundant, duplicate or inaccurate data after data quality assessment, and convert it into the required format or structure;

[0012] Fuse pre-processed multi-source heterogeneous data to form standardized data;

[0013] Mining the internal relationship of all elements of standardized data to obtain relevant elements;

[0014] Build a data warehouse and store data in a distributed manner.

[0015] In some embodiments, the multi-source heterogeneous data includes geospatial data, basic data, monitoring data, business management data, and external shared data.

[0016] In some embodiments, in the collection of multi-source heterogeneous data, an API interface is provided to realize automatic collection and access of data, unify its time, spatial reference and unit, ensure effective integration of data, and classify and label the data according to its source and characteristics.

[0017] In some embodiments, the data cleaning process includes deleting duplicate, erroneous or irrelevant data, correcting errors, performing integrity checks, and verifying the validity of the data.

[0018] In some embodiments, during the data fusion process, ETL technology is used to extract, transform and load data to support complex data fusion requirements.

[0019] In some embodiments, the mining needs to first analyze the statistical characteristics of the time series data, and then mine the intrinsic relationships of all elements of the watershed based on the machine learning model, including mining the nonlinear relationships between elements; finally, perform feature analysis based on the intrinsic relationships of all elements of the watershed to obtain related elements.

[0020] In some embodiments, the data warehouse includes a watershed basic database, a water conservancy subject database, and a metadata database.

[0021] In a second aspect, the present invention provides an efficient data processing system based on multi-source data management and fusion, which is characterized by comprising:

[0022] Data collection module, used to collect multi-source heterogeneous energy data;

[0023] Data governance module, including data quality assessment module, data cleaning module, and data fusion module;

[0024] The data quality assessment module is used to perform data quality assessment on the collected multi-source heterogeneous data; the data quality assessment assesses one or more indicators of data accuracy, data completeness, data consistency, data relevance, and data uniformity of the multi-source heterogeneous energy data to obtain data quality;

[0025] The data cleaning module is used to clean the standardized data, wherein the data cleaning mechanism specifically includes deleting duplicate, erroneous or irrelevant data and correcting errors;

[0026] The data fusion module is used to perform data fusion on the multi-source heterogeneous energy data according to the data quality obtained by the data quality assessment;

[0027] A data fusion module is used to fuse multi-source data of the same target object in the standardized data into unified data;

[0028] The data storage module is used for data storage and management, and data call.

[0029] In a third aspect, the present invention provides an electronic device, comprising:

[0030] A memory for storing executable instructions;

[0031] A processor is used to read and execute executable instructions stored in a memory to implement an efficient data processing method based on multi-source data governance and fusion.

[0032] In a fourth aspect, the present invention provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements an efficient data processing method based on multi-source data governance and fusion.

[0033] Based on the above technical solution, the embodiments of the present invention can at least produce the following technical effects:

[0034] (1) The present invention provides an efficient data processing method based on multi-source data governance and fusion, which collects data and constructs a database covering geographic spatial data, basic data, monitoring data, business management data and external shared data, thereby realizing map services, data resource directory services, data sharing services and data management services, and establishing a database to realize multi-source heterogeneous data, providing support for the construction of digital twins of watersheds.

[0035] (2) The efficient data processing method based on multi-source data governance and fusion provided by the present invention realizes rapid data integration by automatically collecting and accessing multi-source heterogeneous data, unifying time, spatial reference and units. At the same time, ETL technology is used to extract, transform and load data, support complex data fusion requirements, and greatly improve the efficiency of data processing.

[0036] (3) The efficient data processing method based on multi-source data governance and fusion provided by the present invention obtains relevant elements by mining the internal relationship of all elements of standardized data, providing strong support for data analysis and application. Based on the machine learning model, the internal relationship of all elements of the basin is mined, including the nonlinear relationship between elements, which improves the depth and breadth of data value mining.

[0037] (4) The efficient data processing method based on multi-source data management and fusion provided by the present invention builds a data warehouse and stores data in a distributed manner, including a basin basic database, a water conservancy database, and a metadata database, etc., to achieve unified management and call of data. This distributed storage method not only improves the reliability and security of data, but also facilitates data sharing and application.

[0038] (5) The efficient data processing system based on multi-source data governance and fusion provided by the present invention comprehensively evaluates the collected multi-source heterogeneous data through the data quality assessment module, checks the integrity, accuracy and timeliness of the data, and ensures the data quality. In the data preprocessing stage, the data cleaning mechanism deletes duplicate, erroneous or irrelevant data, corrects errors, performs integrity checks, and verifies the validity of the data to further improve the data quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0040] Figure 1 It is an architecture diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0042] This embodiment provides an efficient data processing method based on multi-source data management and fusion. Figure 1 And describe it, including:

[0043] S101. Collect multi-source heterogeneous data, which include geospatial data, basic data, monitoring data, business management data and external shared data.

[0044] It should be noted that geospatial data is the focus of data twin basin construction, mainly including DOM (Digital Orthophoto Map), DEM (Digital Elevation Model), oblique photography model, underwater terrain, BIM (Building Information Model), etc. Basic data include the main attribute data and spatial data of various water conservancy objects such as basins, rivers, lakes, water conservancy projects, hydrology, etc. Monitoring data include rainfall, water conditions, construction conditions, water quality, sediment, disaster conditions, groundwater level, water use, soil moisture, water conservancy project safety operation monitoring, video and other data; business management data include basin water and drought disaster management business application data; external shared data include economic and social, land use, ecological environment, meteorology, remote sensing and other related data that need to be shared from other industry departments. The time base of the data should be Beijing time, the space base should be the 2000 National Geodetic Coordinate System (CGCS2000), and the elevation base should be the 1985 National Elevation Base.

[0045] Specifically, the main methods involved in the process of collecting multi-source heterogeneous data are as follows:

[0046] (1) Data access: With the help of various data interface technologies such as APIs and Webhooks, data collection and access can be automated.

[0047] (2) Data standardization: Because data come from different sources, it is necessary to unify their time, spatial reference, units, etc. to ensure that the data can be effectively integrated.

[0048] (3) Multi-source data identification and classification: Classify and label the data according to its source and characteristics.

[0049] S102. Conduct data quality assessment on the collected multi-source heterogeneous data to check the completeness, accuracy and timeliness of the data.

[0050] S103, pre-processing multi-source heterogeneous data, cleaning redundant, duplicate or inaccurate data after data quality assessment, and converting them into the required format or structure;

[0051] Specifically, the data cleaning process includes deleting duplicate, erroneous or irrelevant data, correcting errors, performing integrity checks, and verifying the validity of data. By preprocessing and cleaning data, problems in multi-source heterogeneous data fusion and the occurrence of poor data quality can be reduced, and the efficiency of data value mining and application can be improved.

[0052] S104, fusing the pre-processed multi-source heterogeneous data to form standardized data;

[0053] Specifically, in the data fusion process, ETL technology is used to extract, transform and load data to support complex data fusion requirements.

[0054] S105, mining the internal relationship of all elements of the standardized data to obtain relevant elements;

[0055] Specifically, mining needs to first analyze the statistical characteristics of time series data, and then mine the intrinsic relationships of all elements of the basin based on the machine learning model, including mining the nonlinear relationship between elements; finally, feature analysis is performed based on the intrinsic relationships of all elements of the basin to obtain relevant elements.

[0056] S106, constructing a data warehouse to store data in a distributed manner, wherein the data warehouse includes a basin basic database, a water conservancy subject database, and a metadata database.

[0057] In this embodiment, storage resources are provided by the Ordos Municipal Government Cloud, and the Kuye River Basin data after data governance is stored in the basin basic spatial database and various water conservancy thematic databases of the Ordos Data Center. At the same time, it is also necessary to construct metadata that describes these geographic spatial information, water conservancy business sharing information, and comprehensive water conservancy management information to achieve unified management of structured data and unstructured data.

[0058] Specifically, regarding the establishment of the basin basic spatial database, a hierarchical database architecture is adopted, and the data is divided into the original data layer, the basic data layer and the application data layer. The original data layer includes the original data from different sources, the basic data layer includes the data that has been cleaned, integrated and fused, and the application data layer includes the application data of various water conservancy databases.

[0059] In order to support the establishment and management of the Kuye River Basin database, it is necessary to select a suitable database management system (DBMS). Common DBMS include relational database management system (RDBMS) and object-oriented database management system (OODBMS). According to the scale and performance requirements of the database, select a suitable system, deploy and configure it to ensure the reliability and performance of the database.

[0060] When constructing a basin basic spatial database, it is also necessary to select a suitable data storage and management method. This embodiment uses a relational database, an object-oriented database, or a distributed database to store data. At the same time, it is necessary to establish a data management strategy, including data backup and recovery, data security, and authority management, to ensure the security and reliability of the data.

[0061] Specifically, regarding the establishment of a water resources database, through data visualization, statistical analysis and model building, we will conduct in-depth research on water resources evaluation, water ecological protection and flood risk analysis in the Kuye River Basin. These analysis results can provide a scientific basis for water resources management and decision-making. They mainly include:

[0062] ① Water resources database: including hydrological data such as water quantity, water quality, water level, precipitation, evaporation, and spatial data of water bodies such as reservoirs, rivers, and lakes. These data are used in the fields of water resources evaluation, water quantity scheduling, and water resources management.

[0063] ②Flood database: including historical flood data, flood process data, flood warning data, etc., as well as spatial data such as flood disaster areas and flood inundation ranges. These data are used in flood warning, flood prevention and control, and disaster risk assessment.

[0064] ③ Water conservancy project database: including the design, construction and operation data of water conservancy projects such as reservoirs, levees, sluices, channels, and spatial data of water conservancy projects. These data are used in the fields of water conservancy project management, water conservancy project design, operation and maintenance.

[0065] ④Land use and soil and water conservation database: including data on land use types, land cover changes, land use evaluation, and spatial data on soil and water conservation projects, vegetation coverage, etc. These data are used in the fields of land resource management, soil and water conservation planning, and ecological restoration.

[0066] ⑤ Hydrometeorological database: including meteorological observation data, climate change data, hydrological model input data, and spatial data such as meteorological stations and hydrological stations. These data are used in fields such as meteorological forecasting, hydrological simulation and climate change research.

[0067] ⑥ Water resources economic database: including water price, water fee, water resource utilization efficiency and other water resources economic data, as well as water resources economic zoning, water resources allocation and other spatial data. These data are used in water resources economic analysis, water resources policy formulation and water resources allocation optimization.

[0068] Specifically, the establishment of a metadata database requires the definition and standardization of metadata content and format. The definition and standardization of metadata can ensure the consistency and understandability of various databases and improve the interoperability and discoverability of data. Metadata is obtained through manual entry and automatic extraction, and is classified, organized and verified to ensure the accuracy and completeness of metadata. Metadata collection and entry can be achieved through metadata management tools or database management systems.

[0069] The schematic diagram of the efficient data processing system based on multi-source data management and fusion provided by the present invention is as follows: Figure 1 As shown, including:

[0070] Data collection module, used to collect multi-source heterogeneous energy data;

[0071] Data governance module, including data quality assessment module, data cleaning module, and data fusion module;

[0072] The data quality assessment module is used to perform data quality assessment on the collected multi-source heterogeneous data; the data quality assessment assesses one or more indicators of data accuracy, data completeness, data consistency, data relevance, and data uniformity of the multi-source heterogeneous energy data to obtain data quality;

[0073] The data cleaning module is used to clean the standardized data, wherein the data cleaning mechanism specifically includes deleting duplicate, erroneous or irrelevant data and correcting errors;

[0074] The data fusion module is used to perform data fusion on the multi-source heterogeneous energy data according to the data quality obtained by the data quality assessment;

[0075] A data fusion module is used to fuse multi-source data of the same target object in the standardized data into unified data;

[0076] The data storage module is used for data storage and management, and data call.

[0077] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0078] This embodiment also provides a computer-readable storage medium. A person of ordinary skill in the art can understand that all or part of the steps in the method for implementing the above embodiment can be completed by a program to instruct the processor, and the computer storage medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be a computer-readable instruction, a data structure, a module of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition herein, machine-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carriers.

[0079] This embodiment may also provide a computer program product, the computer program product including one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present application is generated in whole or in part. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website site, computer or data center to another website site, computer or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode.

[0080] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements fall within the scope of the present invention to be protected. The scope of the present invention to be protected is defined by the attached claims and their equivalents.

Claims

1. An efficient data processing method based on multi-source data governance and fusion, characterized by: include: Collect heterogeneous data from multiple sources; Conduct data quality assessment on the collected multi-source heterogeneous data to check the completeness, accuracy and timeliness of the data; Pre-process multi-source heterogeneous data, clean the redundant, duplicate or inaccurate data after data quality assessment, and convert it into the required format or structure; Fuse pre-processed multi-source heterogeneous data to form standardized data; Mining the internal relationship of all elements of standardized data to obtain relevant elements; Build a data warehouse and store data in a distributed manner.

2. The efficient data processing method based on multi-source data management and fusion according to claim 1 is characterized in that: The multi-source heterogeneous data include geospatial data, basic data, monitoring data, business management data and external shared data.

3. The efficient data processing method based on multi-source data management and fusion according to claim 1 is characterized in that: In the collection of multi-source heterogeneous data, an API interface is provided to realize automatic collection and access of data, unify its time, spatial reference and unit, ensure effective integration of data, and classify and label it according to the source and characteristics of the data.

4. The efficient data processing method based on multi-source data management and fusion according to claim 1 is characterized in that: The data cleaning process includes deleting duplicate, erroneous or irrelevant data, correcting errors, performing integrity checks, and verifying the validity of the data.

5. The efficient data processing method based on multi-source data management and fusion according to claim 1 is characterized in that: In the process of data fusion, ETL technology is used to extract, transform and load data to support complex data fusion requirements.

6. The efficient data processing method based on multi-source data management and fusion according to claim 1 is characterized in that: The mining needs to first analyze the statistical characteristics of the time series data, and then mine the intrinsic relationship of all elements of the basin based on the machine learning model, including mining the nonlinear relationship between elements; finally, based on the intrinsic relationship of all elements of the basin, feature analysis is performed to obtain relevant elements.

7. The efficient data processing method based on multi-source data management and fusion according to claim 1 is characterized in that: The data warehouse includes a basin basic database, a water conservancy subject database and a metadata database.

8. An efficient data processing system based on multi-source data management and fusion, used to implement the efficient data processing method based on multi-source data management and fusion as described in any one of claims 1 to 7, characterized in that: include: Data collection module, used to collect multi-source heterogeneous energy data; Data governance module, including data quality assessment module, data cleaning module, and data fusion module; The data quality assessment module is used to perform data quality assessment on the collected multi-source heterogeneous data; the data quality assessment assesses one or more indicators of data accuracy, data completeness, data consistency, data relevance, and data uniformity of the multi-source heterogeneous energy data to obtain data quality; The data cleaning module is used to clean the standardized data, wherein the data cleaning mechanism specifically includes deleting duplicate, erroneous or irrelevant data and correcting errors; The data fusion module is used to perform data fusion on the multi-source heterogeneous energy data according to the data quality obtained by the data quality assessment; A data fusion module is used to fuse multi-source data of the same target object in the standardized data into unified data; The data storage module is used for data storage and management, and data call.

9. An electronic device, characterized in that: include: A memory for storing executable instructions; A processor, used to read and execute executable instructions stored in a memory to implement an efficient data processing method based on multi-source data governance and fusion as described in any one of claims 1-7.

10. A storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the efficient data processing method based on multi-source data governance and fusion as described in any one of claims 1-7 is implemented.

Images