Design method of data governance resource pool for water conservancy industry based on big data platform

Abstract

The present application provides a method for designing a data governance resource pool for the water conservancy industry based on a big data platform, including: obtaining the correlation relationship and data flow between systems based on the water conservancy business system; formulating water conservancy data standard specifications based on national standards and industry standards; designing a data resource pool based on the water conservancy data standard specifications; and developing a big data platform based on the data resource pool to achieve refined governance of water conservancy data. The present application can improve the data governance capabilities in the process of water conservancy data governance; further explore the value of data, improve data utilization, and the data governance model has high stability. It can solve the problem of inconsistent water conservancy data sources in a complex information system environment, realize one number and one source, and achieve refined governance of water conservancy data.

Description

Technical Field

[0001] The present invention belongs to the technical field of water conservancy engineering, and relates to a data governance technical method, and in particular to a water conservancy industry data governance resource pool design method based on a big data platform. Background Art

[0002] At present, the water conservancy industry is gradually improving the process of informatization and intelligence, using new generation information technologies such as big data, artificial intelligence, the Internet of Things, 5G, remote sensing, edge computing, mobile Internet, and drones, based on the water conservancy intelligent monitoring system, focusing on key water conservancy business scenarios such as flood and drought disaster prevention, water resources development and utilization, water resources three red line control, river and lake chief system performance supervision, Wanli Greenway construction management, and major water conservancy project construction life cycle management, to create intelligent and efficient water conservancy decision-making support services to achieve the overall goal of "access anywhere, analyze at any time, and command on the go". Through data governance, the data application needs of digital twin water conservancy are met, and the value of data is further explored to provide support for decision-making in digital management of water conservancy.

[0003] However, the existing data governance technology in the water conservancy industry has insufficient water conservancy data governance capabilities, cannot achieve one number and one source, and has problems with inconsistent data types and formats. It is unable to solve the problem of inconsistent water conservancy data sources in a complex information system environment, and cannot achieve one number and one source, so that the level of refinement of water conservancy data governance is not enough. Summary of the invention

[0004] In view of the shortcomings of the prior art mentioned above, the purpose of this application is to provide a data governance resource pool design method for the water conservancy industry based on a big data platform, which is used to solve the problems in the prior art that due to insufficient water conservancy data governance capabilities, one number and one source cannot be achieved, and data types and formats are inconsistent. It is impossible to solve the problem of inconsistent water conservancy data sources in a complex information system environment, and one number and one source cannot be achieved, so that the degree of refinement of water conservancy data governance is not enough, which leads to the problems of insufficient data governance capabilities, inconsistent data types or formats, and inconsistent data sources in the existing water conservancy industry data governance process, resulting in excessive extensive water conservancy data governance, inability to further explore data value, low data utilization, and poor stability of data governance models.

[0005] To achieve the above-mentioned objectives and other related objectives, on the first aspect, the present application provides a method for designing a data governance resource pool for the water conservancy industry based on a big data platform, comprising the following steps: based on the water conservancy business system, obtaining the correlation relationship and data flow between the systems; formulating water conservancy data standard specifications based on national standards and industry standards; designing a data resource pool based on the water conservancy data standard specifications; and developing a big data platform based on the data resource pool to achieve refined governance of water conservancy data.

[0006] In an implementation of the first aspect, based on the water conservancy business system, obtaining the correlation relationship and data flow between the systems includes the following steps: establishing a research team to conduct an information system survey and obtain water conservancy system data information from the data level; establishing a business data matrix based on the water conservancy system data information; and using the data exploration tools of the big data platform for automatic scanning and inventory.

[0007] In an implementation of the first aspect, the form of the survey includes: any one or more combinations of visits, questionnaires, and discussions; the water conservancy system data information includes: any one or more combinations of water conservancy system functions, water conservancy data production and utilization status, water conservancy data types, and water conservancy data flows.

[0008] In an implementation of the first aspect, the data resource pool adopts a four-layer design structure of a data warehouse, and the structure includes: an ODS source layer, a DWD detail layer, a DWS theme layer and an ADS subject layer; the ODS source layer is used to divide the data domain of water conservancy data; the DWD detail layer is used to process the divided water conservancy data to obtain standardized water conservancy data; the DWS subject layer is used to fuse the standardized water conservancy data to obtain a service data table for a certain subject domain; the ADS subject layer is used to conduct water conservancy data application analysis for different professions according to user needs, and then organize database construction to design a variety of application topics to generate hydrological models under different application scenarios.

[0009] In an implementation of the first aspect, the ODS source layer includes: water conservancy data system inventory, data combing and data organization.

[0010] In an implementation of the first aspect, the data of the DWD detail layer comes from the ODS source layer; the DWD detail layer includes: a process of docking water conservancy data for data cleaning, data domain division, data fusion and standardization according to standards and rules; the data domain division includes: any one or more combinations of basic data class, spatial remote sensing information class, real-time water and rainfall monitoring class, weather forecast class, and business data class; the data cleaning is used by users to propose specific and clear data requirements and reference standards and specifications, and determine the cleaning rules, which are fed back to the data middle station and data source to improve data quality.

[0011] In an implementation of the first aspect, the DWS theme layer includes the following steps: based on the DWD detail layer data and user needs, integrating and summarizing into a water conservancy service data table for analyzing a certain theme domain in the water conservancy industry; according to different application scenarios, sorting out the theme objects under each application scenario, establishing a logical model of the theme object and a water conservancy theme object relationship table, and obtaining the association relationship of the water conservancy theme object; through the theme object and the association relationship of the water conservancy theme object, obtaining the water conservancy holographic data under a certain application scenario; wherein the logical model includes: any one or more combinations of water conservancy theme object foundation, real-time water and rainfall conditions, meteorology, and spatial dimensional information; the water conservancy theme object includes: any one or more combinations of river basins, rivers, lakes, reservoirs, reservoir dams, hydrological stations, sluices, and pumping stations.

[0012] In an implementation of the first aspect, the ADS thematic layer is designed based on input data calculated by a hydrological model; the hydrological model includes: a forecast model, a one-dimensional evolution model, a two-dimensional evolution model, and an early warning model.

[0013] On the second aspect, the present application provides a water conservancy industry data governance resource pool design system based on a big data platform, including: an acquisition module, used to obtain the relationship and data flow between systems based on the water conservancy business system; a standard formulation module, used to formulate water conservancy data standard specifications based on national standards and industry standards; a design module, used to design a data resource pool based on the water conservancy data standard specifications; a platform development module, used to develop a big data platform based on the data resource pool to achieve refined governance of water conservancy data.

[0014] In the last aspect, the present application provides a water conservancy industry data governance resource pool design device based on a big data platform, comprising: a processor and a memory. The memory is used to store a computer program; the processor is connected to the memory and is used to execute the computer program stored in the memory, so that the water conservancy industry data governance resource pool design device based on a big data platform executes the water conservancy industry data governance resource pool design method based on a big data platform.

[0015] As described above, the water conservancy industry data governance resource pool design method based on the big data platform of the present invention has the following beneficial effects:

[0016] This application provides a method for designing a data governance resource pool for the water conservancy industry based on a big data platform. By establishing a digital twin water conservancy data resource pool, it realizes integrated data services and linkages across businesses and organizations, provides standard and authoritative data for the ever-changing service items and application items of the digital twin project, and ultimately provides a solid foundation for building an authoritative data center. At the same time, it can improve the data governance capabilities in the process of water conservancy data governance; further explore the value of data, improve data utilization, and the data governance model is highly stable. It can solve the problem of inconsistent water conservancy data sources in complex information system environments, realize one number and one source, and realize refined water conservancy data governance. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Shown is a flow chart of a method for designing a data governance resource pool for the water conservancy industry based on a big data platform in one embodiment of the present invention.

[0018] Figure 2 Shown is a flow chart of S11 in the data governance resource pool design method for the water conservancy industry based on the big data platform of the present invention.

[0019] Figure 3 Shown is a flow chart of S13 in the data governance resource pool design method for the water conservancy industry based on the big data platform of the present invention.

[0020] Figure 4 Shown is a schematic diagram of the four-layer design structure of the data warehouse in the water conservancy industry data governance resource pool design method based on the big data platform of the present invention.

[0021] Figure 5 Shown is a schematic diagram of the principle structure of a water conservancy industry data governance resource pool design system based on a big data platform in one embodiment of the present invention.

[0022] Figure 6 Shown is a schematic diagram of the principle structure of a device for designing a data governance resource pool for the water conservancy industry based on a big data platform in one embodiment of the present invention.

[0023] Component number description

[0024] 51 Get Module

[0025] 52 Standard Setting Module

[0026] 53 Design Module

[0027] 54 Platform Development Module

[0028] 61 Processor

[0029] 62 Memory

[0030] Steps S11 to S14 DETAILED DESCRIPTION

[0031] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0032] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0033] The following will be described in detail with reference to the accompanying drawings in the embodiments of the present application to describe the design method of the water conservancy industry data governance resource pool based on the big data platform provided in the embodiments of the present application. The present application is used to solve the problems in the prior art that due to insufficient water conservancy data governance capabilities, one number and one source cannot be achieved, data types and formats are inconsistent, and the problem of inconsistent water conservancy data sources in a complex information system environment cannot be solved, and one number and one source cannot be achieved, so that the degree of refinement of water conservancy data governance is not enough, which leads to the existing water conservancy industry data governance process. Insufficient data governance capabilities, inconsistent data types or formats, inconsistent data sources, and water conservancy data governance is too extensive, unable to further explore data value, low data utilization, and poor stability of data governance models.

[0034] See also Figure 1 , which is a flow chart of a method for designing a water conservancy industry data management resource pool based on a big data platform in one embodiment of the present invention. Figure 1 As shown, this embodiment provides a method for designing a data governance resource pool for the water conservancy industry based on a big data platform.

[0035] The water conservancy industry data governance resource pool design method based on the big data platform specifically includes the following steps:

[0036] S11, based on the water conservancy business system, obtain the association relationship and data flow between each system. Based on the water conservancy business system, obtain the association relationship and data flow between each system.

[0037] The robot formation in this application is fixed, but its position is uncertain. That is, the initial position and arrival position of the formation are not fixed in the Cartesian coordinate system. Therefore, the coordinates need to be determined.

[0038] S111, set up a research team to conduct information system research and obtain water conservancy system data information from the data level.

[0039] In this embodiment, a research team is set up to conduct information system research through various research forms according to the actual situation, and obtain water conservancy system data information from the data level. Among them, the various research forms include: visits, questionnaires, discussions and other different forms; the water conservancy system data information includes: water conservancy system functions, water conservancy data production and utilization status, water conservancy data types, water conservancy data flow and other water conservancy data information.

[0040] Specifically, a research team was set up to conduct information system research, using forms such as visits, questionnaires, and discussions, to grasp information such as system functions, data production and utilization status, data types, and data flows from a data level.

[0041] S112: Establishing a business data matrix based on the water conservancy system data information.

[0042] In this embodiment, a business data matrix is ​​established to provide a basis for the subsequent design of a data resource pool.

[0043] Specifically, according to business needs and goals, determine which data sources need to be obtained from, such as: databases, reports, APIs, etc. According to the determined data sources, formulate a data collection plan, including: data collection scope, collection frequency, collection time, etc. Then, according to the formulated data collection plan, obtain data from the data source and perform necessary cleaning and sorting. According to business needs and goals, determine the model and structure of the data matrix, such as what data the rows and columns of the matrix represent, the relationship between the data, etc. Then, according to the determined data matrix model, fill the collected data into the matrix and perform necessary calculations and processing; after the data matrix is ​​constructed, verification is required to ensure the accuracy and completeness of the data; once the data matrix passes the verification, the data matrix can be published for subsequent data analysis and decision support.

[0044] S113, uses the data exploration tool of the big data platform to perform automatic scanning and inventory.

[0045] In this embodiment, the structured data in the business system is automatically scanned and counted using the data exploration tool of the big data platform.

[0046] Specifically, according to business needs and the amount of data, select appropriate data exploration tools, such as Tableau, PowerBI, Redash, etc. Then, connect the selected data exploration tool to the big data platform to realize data storage and analysis. Create data views on the big data platform to view and analyze data more conveniently. Then use the automatic scanning function in the data exploration tool to automatically scan the structured data to find anomalies and problems in the data. Use the data inventory function in the data exploration tool to inventory the structured data to understand the distribution, trends and correlations of the data. Then, based on the results of the scan and inventory, analyze the problems and opportunities in the data, and put forward corresponding suggestions and measures. Finally, export the analysis results to the specified storage device or shared folder for subsequent use.

[0047] Through the above steps, the data exploration tools of the big data platform can be used to automatically scan and inventory the structured data in the business system, helping enterprises to better understand and utilize data resources.

[0048] S12, formulate water conservancy data standard specifications based on national standards and industry standards.

[0049] In this embodiment, the water conservancy data standard specification is determined, the national standards and industry specifications of the water conservancy industry are collected; the water conservancy data standard is then uniformly formulated; data sharing specifications, data management specifications, and application development specifications are formulated separately; standard specification documents are written, and then the standard specifications are released and promoted. In addition, attention should be paid to the updating of the standard specifications.

[0050] Specifically, clarify the goals of water conservancy data standards and specifications, such as standardizing data formats, improving data quality, and promoting data sharing, and determine the scope of application of standards and specifications, such as applicable regions, applicable business areas, etc. Then, collect national standards and industry standards related to water conservancy data, as well as relevant technical literature and information, and conduct analysis and research. Then, based on the collected information and analysis results, formulate water conservancy data standards, including data format, data type, data accuracy, data element definition, etc. Then, formulate water conservancy data sharing specifications, including: sharing methods, sharing processes, sharing protocols, sharing security, etc.; formulate water conservancy data management specifications, including: data collection, data processing, data storage, data backup, etc.; formulate water conservancy application development specifications, including: application development process, application interface design, application interface design, etc. Next, compile the formulated water conservancy data standards and specifications into documents, including: standard number, name, version number, release date, etc., and typeset and print them according to the document format. Finally, publish and promote standards and specifications: publish and promote the compiled water conservancy data standard specification documents, including: publishing to relevant departments of the water conservancy industry, promoting to relevant enterprises and institutions, etc.

[0051] It should be noted that the standards and specifications should be continuously maintained and updated: based on actual application conditions and feedback, the formulated water conservancy data standards and specifications should be continuously maintained and updated to ensure their adaptability and effectiveness.

[0052] S13, designing a data resource pool based on the water conservancy data standard specification. Figure 3 , which is a flow chart of S13 in the water conservancy industry data management resource pool design method based on a big data platform of the present invention. Figure 3 As shown, the S13 comprises the following steps:

[0053] The data resource pool adopts a typical four-layer design of data warehouse, which is divided into: ODS source layer, DWD detail layer, DWS subject layer, and ADS thematic layer.

[0054] S131, the ODS source layer is used to divide the water conservancy data into data domains.

[0055] In this embodiment, the ODS source layer includes: water conservancy data system inventory, data combing and data sorting.

[0056] Specifically, this layer does not make any changes to the original data in order to solve the possible data traceability issues in the future. For data domain division, refer to the data source.

[0057] This layer accesses all the original data of the existing system and the system to be built. In principle, no changes or processing are made to the accessed data to prevent the subsequent problem of tracing the source of the data. The data in this layer all corresponds to the data source system. The work of the source layer focuses on system inventory and data sorting to sort out the data that is useful for each section of this project.

[0058] S132, the DWD detail layer is used to process the divided water conservancy data to obtain standardized water conservancy data.

[0059] The data comes from the ODS source layer and is cleaned, integrated, and standardized according to standards and rules. Dirty data, junk data, inconsistent data, inconsistent state definitions, and data with irregular naming will all be processed to provide standard data for the DWS theme layer.

[0060] In this embodiment, the data of the DWD detail layer comes from the ODS source layer; the DWD detail layer includes: the process of docking water conservancy data for data cleaning, data domain division, data fusion and standardization according to standards and rules; the data domain division includes: any one or more combinations of basic data class, spatial remote sensing information class, real-time water and rainfall monitoring class, weather forecast class, and business data class; the data cleaning is used by users to propose specific and clear data requirements and reference standards and specifications, and determine the cleaning rules, which are fed back to the data middle station and data source to improve data quality.

[0061] Specifically, the DWD layer cleans, integrates, and normalizes data based on standards and rules. The design focuses on data domain division and the formulation of data cleaning standards.

[0062] The data domain division refers to the "Technical Requirements for Provincial-level Flash Flood Disaster Monitoring, Forecasting and Early Warning Platform (Trial)" (September 2020). It is divided into basic data, spatial remote sensing information, real-time water and rainfall monitoring, meteorological forecast, and business data.

[0063] In addition, the data cleaning standards refer to the "Real-time Rainfall Database Table Structure and Identifier Standard" (SL323-2011), "Mountain Torrent Disaster Investigation and Evaluation Database Table Structure and Identifier Standard" (July 2015), "Flood Control Project Database Table Structure and Identifier" (February 2015), and "Real-time Rainfall Database Table Structure and Identifier Standard" (SL323-2011).

[0064] It should be noted that data cleaning is a long-term task. During the system application process, users need to put forward specific and clear data requirements and reference standard specifications, determine the cleaning rules, and feedback them to the data middle platform and data source to improve data quality.

[0065] The ODS original layer and DWD detailed layer comb and organize data in a bottom-up manner, and the DWS theme layer should start from flash flood disaster monitoring.

[0066] S133, the DWS subject layer is used to fuse the standardized water conservancy data to obtain a service data table for a certain subject domain.

[0067] Based on the DWD detail layer data, it is integrated and summarized into service data for analyzing a certain subject domain, which is generally a wide table.

[0068] In this embodiment, based on the DWD detail layer data and user needs, it is integrated and summarized into a water conservancy service data table for analyzing a certain subject domain in the water conservancy industry; according to different application scenarios, the subject objects under each application scenario are sorted out, and the logical model of the subject object and the water conservancy subject object relationship table are established to obtain the association relationship of the water conservancy subject objects; through the association relationship between the subject object and the water conservancy subject object, the water conservancy holographic data under a certain application scenario is obtained; wherein, the logical model includes: any one or more combinations of water conservancy subject object foundation, real-time water and rainfall conditions, meteorology, and spatial dimensional information; the water conservancy subject objects include: any one or more combinations of river basins, rivers, lakes, reservoirs, reservoir dams, hydrological stations, sluices, and pumping stations.

[0069] Specifically, the ODS original layer and DWD detailed layer comb and organize data in a bottom-up manner. The DWS theme layer should start from the business processes and application scenarios of flash flood disaster monitoring, forecasting and early warning, reverse the data needs, organize the theme objects under various application scenarios, and establish the logical model and object relationship table of the theme objects. The logical model should include the object's basic, real-time water and rainfall conditions, meteorological, and spatial dimensional information, so that the application party can quickly obtain holographic data of a certain application scenario through object IDs and associations.

[0070] The subject division refers to "Basic Database Table Structure and Identifiers of Water Conservancy Objects" (SLT809-2021).

[0071] The water conservancy objects involved in this project include 8 watersheds, rivers, lakes, reservoirs, reservoir dams, hydrological stations, sluice gates, and pumping stations. The design of the theme layer is centered on the water conservancy object, sorting out the basic, spatial, and real-time water and rainfall information of the water conservancy object, and establishing the logical model of the object. In addition, an object relationship table should be established to establish the association relationship between objects.

[0072] S134, the ADS thematic layer is used to perform water conservancy data application analysis for different professions according to user needs, and then organize and build a database to design a variety of application topics to generate hydrological models under different application scenarios.

[0073] The ADS thematic layer meets the needs of specific departments or users, and is oriented towards certain professional application analysis. It reorganizes and builds a database for the data of the DWD detail layer and DWS subject layer. Each application section can put forward its requirements, and the data resource pool will respond.

[0074] In this embodiment, the ADS thematic layer is designed based on the input data calculated by the hydrological model; the hydrological model includes: a forecast model, a one-dimensional evolution model, a two-dimensional evolution model, and an early warning model.

[0075] Specifically, among all the information systems involved in this project, hydrological model calculation is an indispensable part of the "four predictions" business scenarios for flash floods and drought disasters. Therefore, the ADS thematic layer is designed based on the input data requirements of hydrological model calculation. The hydrological models included in the design this time are: forecast model, one-dimensional evolution model, two-dimensional evolution model, and early warning model. More models can be expanded in the future.

[0076] The ADS topic layer can further develop and design a variety of application topics, which requires each bidding section to put forward requirements and the data resource pool to respond. Among them, "Water Conservancy Object Basic Database Table Structure and Identifier" (SLT809-2021).

[0077] S14, developing a big data platform based on the data resource pool to achieve refined management of water conservancy data.

[0078] In this embodiment, the development is completed using a big data platform based on the design scheme. According to the above steps, the water conservancy data resource pool is determined; the development environment is built to construct a water conservancy big data management platform; the water conservancy data model is designed; then, the corresponding data processing program is developed to realize the data visualization process; data sharing is realized; then, water conservancy data is governed; the database of the big data management platform is continuously optimized and updated, etc.

[0079] Specifically, first, determine the required data resource pool, including: data type, data format, data source, etc. Build a development environment suitable for big data platform development, including: hardware equipment, software tools, network environment, etc. Then, according to the characteristics of water conservancy data and business needs, design a suitable data model, including: data structure, data relationship, data flow and other data.

[0080] Then, based on the designed data model, develop the data processing program, including: data collection, cleaning, conversion, analysis, etc.

[0081] Next, the processed data is presented through visualization technology to more intuitively understand the data and discover the laws and trends in the data. The processed data is then shared with other business departments or external units to support decision-making and planning of water conservancy work.

[0082] Then, data governance technology is used to manage and control data in a refined manner, including data quality, data security, data life cycle, etc. Based on actual application conditions and feedback, the big data platform is continuously optimized and updated to improve data processing efficiency and data quality.

[0083] Through the above steps, a big data platform can be developed based on the data resource pool to achieve refined management of water conservancy data and improve the efficiency and accuracy of water conservancy work.

[0084] The water conservancy industry data governance resource pool design method based on the big data platform provided in this application realizes integrated data services and linkages across businesses and organizations by establishing a digital twin water conservancy data resource pool, providing standard and authoritative data for the ever-changing service items and application items of the digital twin project, and ultimately providing a solid foundation for the establishment of an authoritative data center. At the same time, it can enhance the data governance capabilities in the process of water conservancy data governance; further explore the value of data, improve data utilization, and the data governance model is highly stable. It can solve the problem of inconsistent water conservancy data sources in a complex information system environment, realize one number and one source, and realize refined water conservancy data governance.

[0085] The protection scope of the water conservancy industry data governance resource pool design method based on the big data platform described in the embodiment of the present application is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0086] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the following is achieved: Figure 1 The method for designing a data governance resource pool for the water conservancy industry based on a big data platform.

[0087] At any possible level of technical detail combination, the present application may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present application.

[0088] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0089] The computer-readable program described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives a computer-readable program instruction from the network, and forwards the computer-readable program instruction for storage in the computer-readable storage medium in each computing / processing device. The computer program instruction for performing the operation of the present application can be an assembly instruction, an instruction set architecture (ISA) instruction, a machine instruction, a machine-related instruction, a microcode, a firmware instruction, a state setting data, an integrated circuit configuration data, or a source code or object code written in any combination of one or more programming languages, wherein the programming language includes an object-oriented programming language, such as Smalltalk, C++, etc., and a procedural programming language, such as "C" language or similar programming language. Computer readable program instructions can be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a computer, or completely on a computer or server. In the case of a computer, the computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of computer readable program instructions to personalize electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs), the electronic circuits can execute computer readable program instructions, thereby realizing various aspects of the present application.

[0090] An embodiment of the present application also provides a water conservancy industry data governance resource pool design system based on a big data platform. The water conservancy industry data governance resource pool design system based on a big data platform can implement the water conservancy industry data governance resource pool design method based on a big data platform described in the present application, but the implementation device of the water conservancy industry data governance resource pool design method based on a big data platform described in the present application includes but is not limited to the structure of the water conservancy industry data governance resource pool design system based on a big data platform listed in the present embodiment. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.

[0091] The following will describe in detail the water conservancy industry data governance resource pool design system based on the big data platform provided by this embodiment with reference to the diagrams.

[0092] This embodiment provides a water conservancy industry data governance resource pool design system based on a big data platform, including:

[0093] See also Figure 5 , which is a schematic diagram of the principle structure of a water conservancy industry data governance resource pool design system based on a big data platform in one embodiment of the present invention. Figure 5 As shown, the water conservancy industry data governance resource pool design system based on the big data platform includes: an acquisition module 51, a standard setting module 52, a design module 53, and a platform development module 54.

[0094] The acquisition module 51 is used to acquire the association relationship and data flow between various systems based on the water conservancy business system.

[0095] Based on the water conservancy business system, the correlation relationship and data flow between each system are obtained. Based on the water conservancy business system, the correlation relationship and data flow between each system are obtained.

[0096] The robot formation in this application is fixed, but its position is uncertain. That is, the initial position and arrival position of the formation are not fixed in the Cartesian coordinate system. Therefore, the coordinates need to be determined.

[0097] A research team was set up to conduct information system research and obtain water conservancy system data information from the data level.

[0098] In this embodiment, a research team is set up to conduct information system research through various research forms according to the actual situation, and obtain water conservancy system data information from the data level. Among them, the various research forms include: visits, questionnaires, discussions and other different forms; the water conservancy system data information includes: water conservancy system functions, water conservancy data production and utilization status, water conservancy data types, water conservancy data flow and other water conservancy data information.

[0099] A business data matrix is ​​established based on the water conservancy system data information.

[0100] In this embodiment, a business data matrix is ​​established to provide a basis for the subsequent design of a data resource pool.

[0101] Specifically, according to business needs and goals, determine which data sources need to be obtained from, such as: databases, reports, APIs, etc. According to the determined data sources, formulate a data collection plan, including: data collection scope, collection frequency, collection time, etc. Then, according to the formulated data collection plan, obtain data from the data source and perform necessary cleaning and sorting. According to business needs and goals, determine the model and structure of the data matrix, such as what data the rows and columns of the matrix represent, the relationship between the data, etc. Then, according to the determined data matrix model, fill the collected data into the matrix and perform necessary calculations and processing; after the data matrix is ​​constructed, verification is required to ensure the accuracy and completeness of the data; once the data matrix passes the verification, the data matrix can be published for subsequent data analysis and decision support.

[0102] Use the data exploration tools of the big data platform for automatic scanning and inventory.

[0103] In this embodiment, the structured data in the business system is automatically scanned and counted using the data exploration tool of the big data platform.

[0104] Specifically, according to business needs and the amount of data, select appropriate data exploration tools, such as Tableau, PowerBI, Redash, etc. Then, connect the selected data exploration tool to the big data platform to realize data storage and analysis. Create data views on the big data platform to view and analyze data more conveniently. Then use the automatic scanning function in the data exploration tool to automatically scan the structured data to find anomalies and problems in the data. Use the data inventory function in the data exploration tool to inventory the structured data to understand the distribution, trends and correlations of the data. Then, based on the results of the scan and inventory, analyze the problems and opportunities in the data, and put forward corresponding suggestions and measures. Finally, export the analysis results to the specified storage device or shared folder for subsequent use.

[0105] Through the above steps, the data exploration tools of the big data platform can be used to automatically scan and inventory the structured data in the business system, helping enterprises to better understand and utilize data resources.

[0106] The standard formulation module 52 is used to formulate water conservancy data standard specifications based on national standards and industry standards.

[0107] In this embodiment, the water conservancy data standard specification is determined, the national standards and industry specifications of the water conservancy industry are collected; the water conservancy data standard is then uniformly formulated; data sharing specifications, data management specifications, and application development specifications are formulated separately; standard specification documents are written, and then the standard specifications are released and promoted. In addition, attention should be paid to the updating of the standard specifications.

[0108] Specifically, clarify the goals of water conservancy data standards and specifications, such as standardizing data formats, improving data quality, and promoting data sharing, and determine the scope of application of standards and specifications, such as applicable regions, applicable business areas, etc. Then, collect national standards and industry standards related to water conservancy data, as well as relevant technical literature and information, and conduct analysis and research. Then, based on the collected information and analysis results, formulate water conservancy data standards, including data format, data type, data accuracy, data element definition, etc. Then, formulate water conservancy data sharing specifications, including: sharing methods, sharing processes, sharing protocols, sharing security, etc.; formulate water conservancy data management specifications, including: data collection, data processing, data storage, data backup, etc.; formulate water conservancy application development specifications, including: application development process, application interface design, application interface design, etc. Next, compile the formulated water conservancy data standards and specifications into documents, including: standard number, name, version number, release date, etc., and typeset and print them according to the document format. Finally, publish and promote standards and specifications: publish and promote the compiled water conservancy data standard specification documents, including: publishing to relevant departments of the water conservancy industry, promoting to relevant enterprises and institutions, etc.

[0109] It should be noted that the standards and specifications should be continuously maintained and updated: based on actual application conditions and feedback, the formulated water conservancy data standards and specifications should be continuously maintained and updated to ensure their adaptability and effectiveness.

[0110] The design module 53 is used to design a data resource pool based on the water conservancy data standard specification.

[0111] The data resource pool adopts a typical four-layer design of data warehouse, which is divided into: ODS source layer, DWD detail layer, DWS subject layer, and ADS thematic layer.

[0112] The ODS source layer is used to divide the water conservancy data into data domains.

[0113] In this embodiment, the ODS source layer includes: water conservancy data system inventory, data combing and data sorting.

[0114] Specifically, this layer does not make any changes to the original data in order to solve the possible data traceability issues in the future. For data domain division, refer to the data source.

[0115] This layer accesses all the original data of the existing system and the system to be built. In principle, no changes or processing are made to the accessed data to prevent the subsequent problem of tracing the source of the data. The data in this layer all corresponds to the data source system. The work of the source layer focuses on system inventory and data sorting to sort out the data that is useful for each section of this project.

[0116] The DWD detail layer is used to process the divided water conservancy data to obtain standardized water conservancy data.

[0117] The data comes from the ODS source layer and is cleaned, integrated, and standardized according to standards and rules. Dirty data, junk data, inconsistent data, inconsistent state definitions, and data with irregular naming will all be processed to provide standard data for the DWS theme layer.

[0118] In this embodiment, the data of the DWD detail layer comes from the ODS source layer; the DWD detail layer includes: the process of docking water conservancy data for data cleaning, data domain division, data fusion and standardization according to standards and rules; the data domain division includes: any one or more combinations of basic data class, spatial remote sensing information class, real-time water and rainfall monitoring class, weather forecast class, and business data class; the data cleaning is used by users to propose specific and clear data requirements and reference standards and specifications, and determine the cleaning rules, which are fed back to the data middle station and data source to improve data quality.

[0119] Specifically, the DWD layer cleans, integrates, and normalizes data based on standards and rules. The design focuses on data domain division and the formulation of data cleaning standards.

[0120] The data domain division refers to the "Technical Requirements for Provincial-level Flash Flood Disaster Monitoring, Forecasting and Early Warning Platform (Trial)" (September 2020). It is divided into basic data, spatial remote sensing information, real-time water and rainfall monitoring, meteorological forecast, and business data.

[0121] In addition, the data cleaning standards refer to the "Real-time Rainfall Database Table Structure and Identifier Standard" (SL323-2011), "Mountain Torrent Disaster Investigation and Evaluation Database Table Structure and Identifier Standard" (July 2015), "Flood Control Project Database Table Structure and Identifier" (February 2015), and "Real-time Rainfall Database Table Structure and Identifier Standard" (SL323-2011).

[0122] The ODS original layer and DWD detailed layer comb and organize data in a bottom-up manner, and the DWS theme layer should start from flash flood disaster monitoring.

[0123] The DWS subject layer is used to fuse the standardized water conservancy data to obtain a service data table for a certain subject domain.

[0124] Based on the DWD detail layer data, it is integrated and summarized into service data for analyzing a certain subject domain, which is generally a wide table.

[0125] In this embodiment, based on the DWD detail layer data and user needs, it is integrated and summarized into a water conservancy service data table for analyzing a certain subject domain in the water conservancy industry; according to different application scenarios, the subject objects under each application scenario are sorted out, and the logical model of the subject object and the water conservancy subject object relationship table are established to obtain the association relationship of the water conservancy subject objects; through the association relationship between the subject object and the water conservancy subject object, the water conservancy holographic data under a certain application scenario is obtained; wherein, the logical model includes: any one or more combinations of water conservancy subject object foundation, real-time water and rainfall conditions, meteorology, and spatial dimensional information; the water conservancy subject objects include: any one or more combinations of river basins, rivers, lakes, reservoirs, reservoir dams, hydrological stations, sluices, and pumping stations.

[0126] The subject division refers to "Basic Database Table Structure and Identifiers of Water Conservancy Objects" (SLT809-2021).

[0127] The water conservancy objects involved in this project include 8 watersheds, rivers, lakes, reservoirs, reservoir dams, hydrological stations, sluice gates, and pumping stations. The design of the theme layer is centered on the water conservancy object, sorting out the basic, spatial, and real-time water and rainfall information of the water conservancy object, and establishing the logical model of the object. In addition, an object relationship table should be established to establish the association relationship between objects.

[0128] The ADS thematic layer is used to perform water conservancy data application analysis for different professions according to user needs, and then organize database construction to design a variety of application topics to generate hydrological models under different application scenarios.

[0129] The ADS thematic layer meets the needs of specific departments or users, and is oriented towards certain professional application analysis. It reorganizes and builds a database for the data of the DWD detail layer and DWS subject layer. Each application section can put forward its requirements, and the data resource pool will respond.

[0130] In this embodiment, the ADS thematic layer is designed based on the input data calculated by the hydrological model; the hydrological model includes: a forecast model, a one-dimensional evolution model, a two-dimensional evolution model, and an early warning model.

[0131] Specifically, among all the information systems involved in this project, hydrological model calculation is an indispensable part of the "four predictions" business scenarios for flash floods and drought disasters. Therefore, the ADS thematic layer is designed based on the input data requirements of hydrological model calculation. The hydrological models included in the design this time are: forecast model, one-dimensional evolution model, two-dimensional evolution model, and early warning model. More models can be expanded in the future.

[0132] The ADS topic layer can further develop and design a variety of application topics, which requires each bid section to put forward requirements, and the data resource pool will respond. "Hydraulic Object Basic Database Table Structure and Identifier" (SLT809-2021).

[0133] The platform development module 54 is used to develop a big data platform based on the data resource pool to achieve refined management of water conservancy data.

[0134] A big data platform is developed based on the data resource pool to achieve refined management of water conservancy data.

[0135] In this embodiment, the development is completed using a big data platform based on the design scheme. According to the above steps, the water conservancy data resource pool is determined; the development environment is built to construct a water conservancy big data management platform; the water conservancy data model is designed; then, the corresponding data processing program is developed to realize the data visualization process; data sharing is realized; then, water conservancy data is governed; the database of the big data management platform is continuously optimized and updated, etc.

[0136] Specifically, first, determine the required data resource pool, including: data type, data format, data source, etc. Build a development environment suitable for big data platform development, including: hardware equipment, software tools, network environment, etc. Then, according to the characteristics of water conservancy data and business needs, design a suitable data model, including: data structure, data relationship, data flow and other data.

[0137] Then, based on the designed data model, develop the data processing program, including: data collection, cleaning, conversion, analysis, etc.

[0138] Next, the processed data is presented through visualization technology to more intuitively understand the data and discover the laws and trends in the data. The processed data is then shared with other business departments or external units to support decision-making and planning of water conservancy work.

[0139] Then, data governance technology is used to manage and control data in a refined manner, including data quality, data security, data life cycle, etc. Based on actual application conditions and feedback, the big data platform is continuously optimized and updated to improve data processing efficiency and data quality.

[0140] Through the above steps, a big data platform can be developed based on the data resource pool to achieve refined management of water conservancy data and improve the efficiency and accuracy of water conservancy work.

[0141] The water conservancy industry data governance resource pool design system based on the big data platform establishes a digital twin water conservancy data resource pool to achieve integrated data services and linkage across businesses and organizations, provide standard and authoritative data for the ever-changing service items and application items of the digital twin project, and ultimately provide a solid foundation for building an authoritative data center.

[0142] It should be noted that it should be understood that the division of the various modules of the above system is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the x module can be a separately established processing element, or it can be integrated in a certain chip of the above system for implementation. In addition, it can also be stored in the memory of the above system in the form of program code, and called and executed by a certain processing element of the above system. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0143] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0144] See also Figure 6 , which is a schematic diagram of the principle structure of a water conservancy industry data management resource pool design device based on a big data platform in one embodiment of the present invention. Figure 6As shown, this embodiment provides a data governance resource pool design device for the water conservancy industry based on a big data platform, and the data governance resource pool design device for the water conservancy industry based on the big data platform includes: a processor 61 and a memory 62; the memory 62 is used to store computer programs; the processor 61 is connected to the memory 62, and is used to execute the computer program stored in the memory 62, so that the data governance resource pool design device for the water conservancy industry based on the big data platform performs the various steps of the data governance resource pool design method for the water conservancy industry based on the big data platform as described above.

[0145] Preferably, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0146] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0147] In summary, the water conservancy industry data governance resource pool design method based on the big data platform provided by this application has the following beneficial effects:

[0148] The water conservancy industry data governance resource pool design method based on the big data platform provided in this application realizes integrated data services and linkages across businesses and organizations by establishing a digital twin water conservancy data resource pool, providing standard and authoritative data for the ever-changing service items and application items of the digital twin project, and ultimately providing a solid foundation for the establishment of an authoritative data center. At the same time, it can enhance the data governance capabilities in the process of water conservancy data governance; further explore the value of data, improve data utilization, and the data governance model is highly stable. It can solve the problem of inconsistent water conservancy data sources in a complex information system environment, realize one number and one source, and realize refined water conservancy data governance.

[0149] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for designing a data governance resource pool for the water conservancy industry based on a big data platform, characterized in that: The following steps are involved: Based on the water conservancy business system, obtain the relationship and data flow between the systems; Formulate water conservancy data standards and specifications based on national standards and industry standards; including: determine water conservancy data standards and specifications, collect national standards and industry specifications for the water conservancy industry; then formulate water conservancy data standards in a unified manner; formulate data sharing specifications, data management specifications, and application development specifications respectively; compile standard specification documents, and publish and promote standard specifications; The data resource pool is designed based on the water conservancy data standard specification; including: the data resource pool adopts a four-layer design structure of a data warehouse, and the structure includes: ODS source layer, DWD detail layer, DWS theme layer and ADS subject layer; the ODS source layer includes: water conservancy data system inventory, data combing and data sorting, which are used to divide the water conservancy data into data domains; the DWD detail layer is used to process the divided water conservancy data to obtain standardized water conservancy data; the DWS theme layer is used to merge the standardized water conservancy data to obtain a service data table for a certain subject domain; including: based on the DWD detail layer data and user needs, integrating and summarizing into a water conservancy service data table for analyzing a certain subject domain in the water conservancy industry; according to different application scenarios, sorting out the application scenarios The subject object under, establish the logical model of the subject object and the water conservancy subject object relationship table, and obtain the association relationship of the water conservancy subject object; through the association relationship between the subject object and the water conservancy subject object, obtain the water conservancy holographic data in a certain application scenario; wherein, the logical model includes: any one or more combinations of water conservancy subject object foundation, real-time water and rainfall conditions, meteorology, and spatial dimensional information; the water conservancy subject object includes: any one or more combinations of river basins, rivers, lakes, reservoirs, reservoir dams, hydrological stations, sluices, and pumping stations; the ADS thematic layer is used to perform water conservancy data application analysis for different professions according to user needs, and then organize the database to design a variety of application topics to generate hydrological models in different application scenarios; the data of the DWD detail layer comes from the ODS source layer; A big data platform is developed based on the data resource pool to achieve refined management of water conservancy data; including: determining the required data resource pool; developing a data processing program based on a designed data model; using visualization technology to display the processed data; and using data governance technology to perform refined management and control of water conservancy data.

2. According to the method for designing a data management resource pool for the water conservancy industry based on a big data platform according to claim 1, it is characterized in that: Based on the water conservancy business system, obtaining the association relationship and data flow between the systems includes the following steps: Establish a research team to conduct information system research and obtain water conservancy system data information from the data level; Establishing a business data matrix based on the water conservancy system data information; Use the data exploration tools of the big data platform for automatic scanning and inventory.

3. The method for designing a data management resource pool for the water conservancy industry based on a big data platform according to claim 2 is characterized in that: The survey forms include: any one or more combinations of visits, questionnaires, and discussions; The water conservancy system data information includes: any one or more combinations of water conservancy system functions, water conservancy data production and utilization status, water conservancy data type, and water conservancy data flow direction.

4. The method for designing a data management resource pool for the water conservancy industry based on a big data platform according to claim 1 is characterized in that: The DWD detail layer includes: the process of docking water conservancy data for data cleaning, data domain division, data fusion and standardization according to standards and rules; The data domain division includes: any one or more combinations of basic data, space remote sensing information, real-time water and rainfall monitoring, weather forecast, and business data; The data cleaning is used by users to propose specific and clear data requirements and reference standard specifications, and determine the cleaning rules, which are fed back to the data center and data source to improve data quality.

5. The method for designing a data management resource pool for the water conservancy industry based on a big data platform according to claim 1 is characterized in that: The ADS thematic layer is designed for input data calculated by the hydrological model; The hydrological model includes: a forecast model, a one-dimensional evolution model, a two-dimensional evolution model, and an early warning model.

6. A water conservancy industry data governance resource pool design system based on a big data platform, characterized in that: include: The acquisition module is used to obtain the association relationship and data flow between various systems based on the water conservancy business system; The standard formulation module is used to formulate water conservancy data standards and specifications based on national standards and industry standards; including: determining water conservancy data standards and specifications, collecting national standards and industry specifications for the water conservancy industry; then uniformly formulating water conservancy data standards; formulating data sharing specifications, data management specifications, and application development specifications respectively; writing standard specification documents, and publishing and promoting standard specifications; The design module is used to design a data resource pool based on the water conservancy data standard specification; including: the data resource pool adopts a four-layer design structure of a data warehouse, and the structure includes: ODS source layer, DWD detail layer, DWS theme layer and ADS subject layer; the ODS source layer includes: water conservancy data system inventory, data combing and data sorting, which are used to divide the water conservancy data into data domains; the DWD detail layer is used to process the divided water conservancy data to obtain standardized water conservancy data; the DWS theme layer is used to fuse the standardized water conservancy data to obtain a service data table for a certain subject domain; including: based on the DWD detail layer data and user needs, integrating and summarizing into a water conservancy service data table for analyzing a certain subject domain in the water conservancy industry; sorting out the various The subject object in the application scenario, establish the logical model of the subject object and the water conservancy subject object relationship table, and obtain the association relationship of the water conservancy subject object; through the association relationship between the subject object and the water conservancy subject object, obtain the water conservancy holographic data in a certain application scenario; wherein, the logical model includes: any one or more combinations of water conservancy subject object foundation, real-time water and rainfall conditions, meteorology, and spatial dimensional information; the water conservancy subject object includes: any one or more combinations of river basins, rivers, lakes, reservoirs, reservoir dams, hydrological stations, sluices, and pumping stations; the ADS thematic layer is used to conduct water conservancy data application analysis for different professions according to user needs, and then organize the database to design a variety of application topics to generate hydrological models in different application scenarios; the data of the DWD detail layer comes from the ODS source layer; The platform development module is used to develop a big data platform based on the data resource pool to achieve refined management of water conservancy data; it includes: determining the required data resource pool; developing a data processing program based on a designed data model; using visualization technology to display the processed data; and using data governance technology to perform refined management and control of water conservancy data.

7. A water conservancy industry data governance resource pool design device based on a big data platform, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is connected to the memory and is used to execute the computer program stored in the memory so that the water conservancy industry data governance resource pool design device based on the big data platform executes the water conservancy industry data governance resource pool design method based on the big data platform as described in any one of claims 1 to 5.

Images