Data-in-data construction method and device and storage medium

Abstract

The invention discloses a data-in-channel construction method and device and a storage medium. The method comprises the steps that DaaS is integrated, a data-in-channel is in butt joint with the DaaS, and the total amount and the increment of data are synchronized; constructing a time sequence database for storing massive time sequence data; the time sequence database stores and processes data with time labels; developing an equipment object construction tool, and constructing equipment objects of the industrial internet in a visual mode, including equipment modeling and equipment association services; constructing an equipment data mart, wherein the data mart comprises a theme library and a thematic library; constructing an industrial calculation scheduling engine for supporting streaming calculation, batch calculation and interactive calculation; an algorithm execution engine is constructed, a basic operation environment supporting calculation of various industrial models is provided, and the algorithm execution engine is used for rapid deployment, loading and calculation of the industrial models and conducting online tracking and management and control on the calculation process. The problem that the intelligent hydroelectric technology lacks data integration, sharing and multiplexing is solved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent hydropower industrial Internet, and in particular to a data middle platform construction method, device and storage medium. Background Art

[0002] Smart hydropower technology has been widely used in the hydropower industry. Currently, various data from hydropower stations can be collected in real time through sensors and monitoring equipment, and transmitted to the database through the network, combined with machine learning and artificial intelligence algorithms for data analysis.

[0003] However, in the existing technology, data collection and processing of hydropower stations are often scattered among various departments and nodes, which makes it difficult for data to circulate and share between different systems, limiting the effectiveness of data integration and analysis. This makes it difficult for different business departments to share data information and work together, affecting further management and maintenance. Summary of the invention

[0004] The present invention provides a data center construction method, device and storage medium.

[0005] In a first aspect, the present invention provides a method for constructing a data center, the method comprising: Integrate DaaS, connect the data center with DaaS, and synchronize the full and incremental data; wherein, the DaaS includes a distributed file system, a NoSQL database, a relational database, and an MPP database; Building a time series database for storing massive time series data; the time series database stores and processes data with time tags; Develop equipment object construction tools to visually construct equipment objects for the Industrial Internet, including equipment modeling and equipment association services; Constructing a device data mart, which includes a subject library and a special library; the data mart provides external data sharing in a subscription manner; the construction method includes: constructing a device data object directory, a structured view, and a time series view; Build an industrial computing scheduling engine to support streaming computing, batch computing, and interactive computing for model operation, monitoring, scheduling, and statistics, store and manage model analysis results, and provide external data and service interfaces; Build an algorithm execution engine to provide a basic operating environment that supports the calculation of various industrial models. It is used for the rapid deployment, loading and calculation of industrial models, and for online tracking and control of the calculation process.

[0006] Furthermore, the method further comprises: Build an extensible general algorithm library, including: data preprocessing operator set, statistical analysis operator set, machine learning operator set, dedicated signal processing operator set, and decision fusion operator set.

[0007] Furthermore, the method further comprises: Build an expandable hydropower-specific algorithm library, including: operating condition identification library, vibration analysis library, temperature analysis library, liquid level analysis library, oil and gas analysis library, partial discharge analysis library, and sensor fault library.

[0008] Furthermore, the method comprises: Through the data services provided by DaaS, real-time data access, offline data access and calculation result write-back are completed; based on the time series data service, the device data in DaaS is uniformly accessed and managed, so that the data is synchronized online in real time; through the device object model, the device-related business data in DaaS is uniformly accessed and managed, so that the data is synchronized offline at regular intervals; through the computing service of DaaS, computing power is provided for data analysis and model calculation of the industrial Internet platform, and the calculation results are written back to DaaS in real time.

[0009] Furthermore, the device modeling and device association services are specifically: The equipment modeling includes: equipment component library configuration, system equipment library configuration, plant structure modeling, and integration with DaaS data services; The equipment association service is used for unified management of equipment association data; the association data includes work tickets, operation tickets, defect information, inspection information, regular work, test record information, equipment ledgers, equipment technical parameter display, equipment failure accident records, equipment shutdown records, equipment abnormality records, and equipment maintenance records; provides a list of measurement point names and real-time data, and equipment association display; provides defect or fault analysis and diagnosis algorithms, as well as defect or fault instances.

[0010] Furthermore, the data mart directly completes service publishing, review, and sharing of the data service interface through a web interface; supports data users to request corresponding data only through the name of the data mart; and supports third parties to quickly retrieve data from the data mart according to interface specifications.

[0011] Furthermore, the method comprises: When selecting based on the data object directory, the entire data object is placed in the data sandbox without any additional data filtering conditions; when selecting data objects through structured views, the filtering conditions defined in the original view are used; when selecting data through time series views, in addition to the conditions defined in the time series view, the data time range conditions are additionally set.

[0012] In a second aspect, the present invention provides a data middle platform construction device, the device comprising: Integrated DaaS module for: Integrate DaaS, connect the data center with DaaS, and synchronize the full and incremental data; wherein, the DaaS includes a distributed file system, a NoSQL database, a relational database, and an MPP database; Build a time series database module for: Building a time series database to store massive time series data; the time series database stores and processes data with time tags; Develop device object builder modules for: Develop equipment object construction tools to visually construct equipment objects for the Industrial Internet, including equipment modeling and equipment association services; Build a device data mart module for: Constructing a device data mart, which includes a subject library and a special library; the data mart provides external data sharing in a subscription manner; the construction method includes: constructing a device data object directory, a structured view, and a time series view; Build an industrial computing scheduling engine module for: Build an industrial computing scheduling engine that supports streaming computing, batch computing, and interactive computing for model operation, monitoring, scheduling, and statistics, stores and manages model analysis results, and provides external data and service interfaces; Build an algorithm execution engine module for: Build an algorithm execution engine to provide a basic operating environment that supports the calculation of various industrial models. It is used for the rapid deployment, loading and calculation of industrial models, and for online tracking and control of the calculation process.

[0013] In a third aspect, the present invention provides a data middle platform construction device, the device comprising: Memory, used to store programs; The processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the above method.

[0014] In a fourth aspect, the present invention provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, causes the one or more processors to execute the above method.

[0015] Compared with the prior art, the present invention has the following advantages: By building a data middle platform based on the industrial Internet of the power industry, connecting the data middle platform with DaaS, achieving full and incremental synchronization of data, building a time series database to support the effective storage and processing of data with time tags, developing equipment object construction tools, and conveniently building industrial Internet equipment objects in a visual way, building equipment data marts based on equipment objects, providing a highly available and high-performance industrial computing scheduling engine, providing a basic operating environment that supports the calculation of various industrial models, and providing an extensible general algorithm library and an extensible hydropower-specific algorithm library. The problem of lack of data integration, sharing and reuse in smart hydropower technology has been solved.

[0016] Through the data middle platform of this application, core capabilities such as data bus, data storage, and data governance are shared, making data integration, processing, and application more unified and efficient. This solves the problem of data being difficult to integrate and standardize in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a data center construction method provided by an embodiment of the present invention; Figure 2 A schematic diagram of the composition of a data middle platform construction device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0019] Explanation of terms involved in the embodiments of the present invention: DaaS (Data-as-a-service) is a new service concept after IaaS, PaaS, and SaaS. It points the way to improve IT efficiency and system performance through centralized resource management. Its main technologies include data virtualization, data integration, SOA, BPM, and PaaS.

[0020] NoSQL, Not Only SQL non-relational database.

[0021] MPP (Massively Parallel Processing), MPP database is a specific type of database management system, which is designed to process large-scale data sets and complex queries.

[0022] The data middle platform is a sedimentation of existing / new information system business and data. It is an intermediate and supporting platform for realizing data empowerment of new business and new applications.

[0023] A data mart, also known as a data market, is a data mart that meets the needs of specific departments or users and is stored in a multi-dimensional manner, including defining dimensions, indicators that need to be calculated, dimensional levels, etc., to generate data cubes for decision-making analysis needs.

[0024] Figure 1 A schematic diagram of a data center construction method provided in an embodiment of the present invention, the method comprising: Integrate DaaS, connect the data center with DaaS, and synchronize the full and incremental data; wherein, the DaaS includes a distributed file system, a NoSQL database, a relational database, and an MPP database; Building a time series database for storing massive time series data; the time series database stores and processes data with time tags; Develop equipment object construction tools to visually construct equipment objects for the Industrial Internet, including equipment modeling and equipment association services; Constructing a device data mart, which includes a subject library and a special library; the data mart provides external data sharing in a subscription manner; the construction method includes: constructing a device data object directory, a structured view, and a time series view; Build an industrial computing scheduling engine to support streaming computing, batch computing, and interactive computing for model operation, monitoring, scheduling, and statistics, store and manage model analysis results, and provide external data and service interfaces; Build an algorithm execution engine to provide a basic operating environment that supports the calculation of various industrial models. It is used for the rapid deployment, loading and calculation of industrial models, and for online tracking and control of the calculation process.

[0025] In one embodiment, the method further comprises: Build an extensible general algorithm library, including: data preprocessing operator set, statistical analysis operator set, machine learning operator set, dedicated signal processing operator set, and decision fusion operator set.

[0026] In one embodiment, the method further comprises: Build an expandable hydropower-specific algorithm library, including: operating condition identification library, vibration analysis library, temperature analysis library, liquid level analysis library, oil and gas analysis library, partial discharge analysis library, and sensor fault library.

[0027] In one embodiment, the method comprises: Through the data services provided by DaaS, real-time data access, offline data access and calculation result write-back are completed; based on the time series data service, the device data in DaaS is uniformly accessed and managed, so that the data is synchronized online in real time; through the device object model, the device-related business data in DaaS is uniformly accessed and managed, so that the data is synchronized offline at regular intervals; through the computing service of DaaS, computing power is provided for data analysis and model calculation of the industrial Internet platform, and the calculation results are written back to DaaS in real time.

[0028] In one embodiment, the device modeling and device association services are specifically: The equipment modeling includes: equipment component library configuration, system equipment library configuration, plant structure modeling, and integration with DaaS data services; The equipment association service is used for unified management of equipment association data; the association data includes work tickets, operation tickets, defect information, inspection information, regular work, test record information, equipment ledgers, equipment technical parameter display, equipment failure accident records, equipment shutdown records, equipment abnormality records, and equipment maintenance records; provides a list of measurement point names and real-time data, and equipment association display; provides defect or fault analysis and diagnosis algorithms, as well as defect or fault instances.

[0029] In one embodiment, the data mart directly completes the service publishing, review, and sharing of the data service interface through a web interface; supports data users to request corresponding data only through the name of the data mart; and supports third parties to quickly retrieve data from the data mart according to interface specifications.

[0030] In one embodiment, the method comprises: When selecting based on the data object directory, the entire data object is placed in the data sandbox without any additional data filtering conditions; when selecting data objects through structured views, the filtering conditions defined in the original view are used; when selecting data through time series views, in addition to the conditions defined in the time series view, the data time range conditions are additionally set.

[0031] An embodiment of the present invention provides a data platform construction device, the device comprising: Integrated DaaS module for: Integrate DaaS, connect the data center with DaaS, and synchronize the full and incremental data; wherein, the DaaS includes a distributed file system, a NoSQL database, a relational database, and an MPP database; Build a time series database module for: Building a time series database to store massive time series data; the time series database stores and processes data with time tags; Develop device object builder modules for: Develop equipment object construction tools to visually construct equipment objects for the Industrial Internet, including equipment modeling and equipment association services; Build a device data mart module for: Constructing a device data mart, which includes a subject library and a special library; the data mart provides external data sharing in a subscription manner; the construction method includes: constructing a device data object directory, a structured view, and a time series view; Build an industrial computing scheduling engine module for: Build an industrial computing scheduling engine that supports streaming computing, batch computing, and interactive computing for model operation, monitoring, scheduling, and statistics, stores and manages model analysis results, and provides external data and service interfaces; Build an algorithm execution engine module for: Build an algorithm execution engine to provide a basic operating environment that supports the calculation of various industrial models. It is used for the rapid deployment, loading and calculation of industrial models, and for online tracking and control of the calculation process.

[0032] Figure 2 A schematic diagram of a data middle platform construction device provided in an embodiment of the present invention; the device includes: Memory, used to store programs; The processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the above method.

[0033] In addition, an embodiment of the present invention provides a storage medium storing computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the above method.

[0034] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for constructing a data middle platform, characterized in that: The method is constructed based on the industrial Internet of the power industry and includes: Integrate DaaS, connect the data center with DaaS, and synchronize the full and incremental data; wherein, the DaaS includes a distributed file system, a NoSQL database, a relational database, and an MPP database; Building a time series database for storing massive time series data; the time series database stores and processes data with time tags; Develop equipment object construction tools to visually construct equipment objects for the Industrial Internet, including equipment modeling and equipment association services; Constructing a device data mart, which includes a subject library and a special library; the data mart provides external data sharing in a subscription manner; the construction method includes: constructing a device data object directory, a structured view, and a time series view; Build an industrial computing scheduling engine to support streaming computing, batch computing, and interactive computing for model operation, monitoring, scheduling, and statistics, store and manage model analysis results, and provide external data and service interfaces; Build an algorithm execution engine to provide a basic operating environment that supports the calculation of various industrial models. It is used for the rapid deployment, loading and calculation of industrial models, and for online tracking and control of the calculation process.

2. The data middle platform construction method according to claim 1, characterized in that: The method further comprises: Build an extensible general algorithm library, including: data preprocessing operator set, statistical analysis operator set, machine learning operator set, dedicated signal processing operator set, and decision fusion operator set.

3. The data platform construction method according to claim 1, characterized in that: The method further comprises: Build an expandable hydropower-specific algorithm library, including: operating condition identification library, vibration analysis library, temperature analysis library, liquid level analysis library, oil and gas analysis library, partial discharge analysis library, and sensor fault library.

4. The data platform construction method according to claim 1, characterized in that: The method comprises: Through the data services provided by DaaS, real-time data access, offline data access and calculation result write-back are completed; based on the time series data service, the device data in DaaS is uniformly accessed and managed, so that the data is synchronized online in real time; through the device object model, the device-related business data in DaaS is uniformly accessed and managed, so that the data is synchronized offline at regular intervals; through the computing service of DaaS, computing power is provided for data analysis and model calculation of the industrial Internet platform, and the calculation results are written back to DaaS in real time.

5. The data middle platform construction method according to claim 1, characterized in that: The device modeling and device association services are specifically: The equipment modeling includes: equipment component library configuration, system equipment library configuration, plant structure modeling, and integration with DaaS data services; The equipment association service is used for unified management of equipment association data; the association data includes work tickets, operation tickets, defect information, inspection information, regular work, test record information, equipment ledgers, equipment technical parameter display, equipment failure accident records, equipment shutdown records, equipment abnormality records, and equipment maintenance records; provides a list of measurement point names and real-time data, and equipment association display; provides defect or fault analysis and diagnosis algorithms, as well as defect or fault instances.

6. The data platform construction method according to claim 1, characterized in that: The data mart directly completes the service publishing, review and sharing of the data service interface through the web interface; supports data users to request corresponding data only through the name of the data mart; and supports third parties to quickly retrieve data from the data mart according to interface specifications.

7. The data middle platform construction method according to claim 1, characterized in that: The method comprises: When selecting based on the data object directory, the entire data object is placed in the data sandbox without any additional data filtering conditions; when selecting data objects through structured views, the filtering conditions defined in the original view are used; when selecting data through time series views, in addition to the conditions defined in the time series view, the data time range conditions are additionally set.

8. A data platform construction device, characterized in that: The device comprises: Integrated DaaS module for: Integrate DaaS, connect the data center with DaaS, and synchronize the full and incremental data; wherein, the DaaS includes a distributed file system, a NoSQL database, a relational database, and an MPP database; Build a time series database module for: Building a time series database to store massive time series data; the time series database stores and processes data with time tags; Develop device object builder modules for: Develop equipment object construction tools to visually construct equipment objects for the Industrial Internet, including equipment modeling and equipment association services; Build a device data mart module for: Constructing a device data mart, which includes a subject library and a special library; the data mart provides external data sharing in a subscription manner; the construction method includes: constructing a device data object directory, a structured view, and a time series view; Build an industrial computing scheduling engine module for: Build an industrial computing scheduling engine that supports streaming computing, batch computing, and interactive computing for model operation, monitoring, scheduling, and statistics, stores and manages model analysis results, and provides external data and service interfaces; Build an algorithm execution engine module for: Build an algorithm execution engine to provide a basic operating environment that supports the calculation of various industrial models. It is used for the rapid deployment, loading and calculation of industrial models, and for online tracking and control of the calculation process.

9. A data platform construction device, characterized in that: The device comprises: Memory, used to store programs; A processor, configured to execute the program stored in the memory; when the program stored in the memory is executed, the processor is configured to execute the steps of the method as claimed in any one of claims 1 to 7.

10. A storage medium storing computer-readable instructions, wherein when the computer-readable instructions are executed by one or more processors, the one or more processors are caused to perform the steps of the method according to any one of claims 1 to 7.

Images