Method and system for data collection and storage, modeling development and service sharing covering IT and OT scenes

Abstract

The invention discloses a data acquisition and storage, modeling development and sharing service method and system covering IT and OT scenes, and the method comprises the steps: data collection: carrying out the data collection of IT data and OT data according to three data types: structured data, measurement data and unstructured data; data storage and modeling are carried out, data storage and modeling are carried out based on three characteristics of a total data base, a distributed framework and a hierarchical data model, and a data lake and data warehouse integrated structure MRS is built based on a storage assembly and a calculation engine in a distributed system infrastructure; data development: performing data development on different types of data through batch processing and stream processing; data sharing: respectively designing data sharing technical schemes according to two scenes of non-real-time data sharing and real-time data sharing; unified storage of IT and OT data can be realized, an integrated data lake and an integrated data warehouse are built, and a foundation is laid for fusion application of the two types of data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and system for data collection, storage, modeling development and sharing services covering IT and OT scenarios. Background Art

[0002] IT data and OT data are two common types of data with distinctly different characteristics. IT data has a complex structure, low real-time requirements, and a large amount of historical data, while OT data has a simple structure and strong real-time performance. In the existing technical solutions, IT data and OT data are processed separately using two different technical routes. IT data is mostly stored and developed using relational databases, while OT data is mostly stored and developed using time series databases. In terms of data collection and storage, IT data and OT data are physically distributed in different types of storage components, and no unified data lake is formed, which brings difficulties to the integrated application of the two types of data. In terms of data modeling and development, existing technologies generally carry out dimensional modeling based on the source layer, lack a unified model management layer, and the results of data integration and standardization work are difficult to precipitate and standardize. At the same time, both IT data and OT data are mostly completed based on the SQL functions and job functions of database components, lack script dependency management, unified scheduling and monitoring, and are difficult to meet the needs of complex data processing scenarios. In terms of data sharing services, existing technologies are mostly based on authorization and access at the database level, which is easy to form a spider web, and directly carrying out data sharing work at the database level also brings security issues. Therefore, it is necessary to design a method and system for data collection, storage, modeling development and sharing services covering IT and OT scenarios to solve the above problems. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a method and system for data collection, storage, modeling development and sharing services covering IT and OT scenarios, so as to solve the problem in the prior art that IT data and OT data are difficult to perform data fusion processing due to their different characteristics. The method can meet the different needs of IT data and OT data in terms of data structure, update method, production frequency, etc., realize unified storage of the two types of data, build an integrated data lake and data warehouse, and lay the foundation for the integrated application of the two types of data.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: A method for data collection, storage, modeling development and shared services covering IT and OT scenarios, including: Data collection: IT data and OT data are collected according to three data types: structured data, measurement data, and unstructured data; Data storage and modeling: Data storage and modeling are performed based on the three features of full data base, distributed framework and layered data model. Based on the distributed system infrastructure, the Hudi storage component and Spark computing engine in the Hadoop ecosystem are selected to build the MRS, an integrated structure of data lake and data warehouse; Data development: Data development for different types of data through batch processing and stream processing; Data sharing: Data sharing is carried out according to two scenarios: non-real-time data sharing and real-time data sharing.

[0005] Preferably, the method for data collection using structured data is: Determine that the data source of structured data is a relational database; collect structured data through ETL tools, adopt a periodic collection method, set the collection period to T+1, and use CDM and API as the data transmission method; implement full collection for tables with data volumes less than the set threshold, and implement incremental collection for tables with data volumes greater than the set threshold; the data collection path is from the data source to the data lake Hudi.

[0006] Preferably, the method of data collection using measurement data is: Determine the data source of the measurement data as message queue, MQTT, IEC104 or API; use real-time collection method and integrated industrial control data collection components to collect measurement data. The data transmission methods include message queue, MQTT, IEC104 and API, and the collection frequency is in seconds; the data collection path is from data source to time series database to data lake Hudi.

[0007] Preferably, the method for data collection using unstructured data is: There are two ways to collect unstructured data: physical collection and virtual collection. Physical collection refers to the synchronous collection of physical files and metadata, and virtual collection refers to the collection of metadata. Data transmission methods include APIs. The data collection path is from the data source to the data lake OBS.

[0008] Preferably, data storage and modeling are performed based on the three features of full data base, distributed framework and hierarchical data model. Based on the distributed system infrastructure, the Hudi storage component in the Hadoop ecosystem and the Spark computing engine are selected to build the data lake and data warehouse integrated structure MRS. The specific method is as follows: MRS, an integrated structure of data lake and data warehouse, is built through a layered architecture, and data is transmitted between each layer of the architecture through SparkSQL; The integrated structure of data lake and data warehouse MRS is divided into source layer SDI, basic layer DWI, general layer DWR and market layer DM. The data lake is the source layer SDI. The source layer is constructed by source modeling. The source layer data is consistent with the source end data. The data warehouse includes the basic layer DWI, general layer DWR and market layer DM. The data of MRS, an integrated structure of data lake and data warehouse, is transmitted to the temporary table Hive communication connection, and the data of temporary table Hive is transmitted to the data mart DWS through CDM transmission method.

[0009] Preferably, the source data includes OT time series data, IT real-time data and IT non-real-time data; OT time series data is transmitted to the source layer SDI in real time through Kafka and FlinkSQL transmission methods; IT real-time data is transmitted to the source layer SDI in real time through CDL transmission method; IT non-real-time data is transmitted to the temporary table Hive in full and regularly through CDM transmission method, and the temporary table Hive is transmitted to the source layer SDI through SparkSQL; OT time series data and IT real-time data constitute structured data, and IT non-real-time data constitutes measurement data.

[0010] Preferably, during data development, batch processing operations are performed on structured data, specifically: Based on SparkSQL scripts, data is transferred from the source layer SDI to the base layer DWI, and the same type of data is aggregated; Based on the SparkSQL script, data is transferred from the basic layer DWI to the general layer DWR, and the dimension table and fact table of the data are created to calculate the business indicators; Based on the SparkSQL script, data is transferred from the general layer DWR to the market layer, and all business indicators are aggregated into a final summary table; then the data in the summary table is displayed.

[0011] Preferably, in step S3, during data development, a stream processing operation is performed on the measurement data, specifically: Real-time data calculation is performed through Kafka, FlinkSQL and the time series library. The time series library completes, removes duplicates, processes outliers, opens data windows, aggregates data and groups data in real time according to business needs, and displays real-time data analysis through message queues or APIs.

[0012] Preferably, data sharing includes: Share structured data through API service and data push service; API services include: Based on data sharing needs and in accordance with data sharing implementation specifications, develop and configure corresponding API interfaces, and authorize them to relevant application systems. Through clear interface authentication, interface access is monitored to enable data to be transmitted to the demander. Data push services include: Complete data processing according to the needs of the demander, and push data tables regularly through data integration components; Share measurement data through API services and message queues; API services include: API services are published based on time series databases, providing real-time query and consumption of measurement data for demanders; The message queue includes: Publish the required data as a message topic to achieve real-time transmission and consumption of data.

[0013] Preferably, a system for data collection, storage, modeling development and sharing services covering IT and OT scenarios, the system includes a data collection module, a data storage and modeling module, a data development module and a data sharing module; The data collection module is used to collect data according to three data types: structured data, measurement data and unstructured data; The data storage and modeling module is used for data storage and modeling based on the three features of full data base, distributed framework and hierarchical data model. It builds the MRS, an integrated structure of data lake and data warehouse, based on the Hudi storage component and Spark computing engine in the Hadoop ecosystem. Data development module, used for data development of different types of data through batch processing and stream processing; The data sharing module is used to share data according to two scenarios: non-real-time data sharing and real-time data sharing.

[0014] The method and system for data collection, storage, modeling development and sharing services covering IT and OT scenarios provided by the present invention have the following beneficial effects: 1. This method designs two technical paths: periodic collection and real-time collection, to meet the different requirements of IT data and OT data in terms of data structure, update method, production frequency, etc., to achieve unified storage of the two types of data, and to build an integrated data lake and data warehouse, laying the foundation for the integrated application of the two types of data.

[0015] 2. This method designs a four-layer data model with source layer SDI, basic layer DWI, general layer DWR and market layer DM, which not only supports large-scale data concentration, but also meets the modeling needs of data application scenarios; innovatively designs the basic layer, carries out unified relationship modeling and formulates data standards according to business scenarios, recodes and names the source layer data according to the data standards, and converts and merges multi-source data to achieve unified model, unified standard and data integration; at the same time, this solution breaks the "black box" data development idea, designs script orchestration and scheduling based on the basic capabilities of SparkSQL, and realizes dependency management, unified scheduling and monitoring of development jobs.

[0016] 3. This method integrates stream processing and batch processing to design a unified service gateway, realizes unified authentication and authorization of data services in both IT and OT scenarios, and provides guarantee for data security while efficiently carrying out data sharing. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The present invention will be further described below in conjunction with the accompanying drawings and embodiments: Figure 1 It is a schematic diagram of the process of data collection in the present invention; Figure 2 It is a schematic diagram of the process of data storage and modeling in the present invention; Figure 3 It is a schematic diagram of the process of data sharing in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] Embodiment 1: A method for data collection, storage, modeling development and shared services covering IT and OT scenarios, including: Data collection: IT data and OT data are collected according to three data types: structured data, measurement data, and unstructured data; Data storage and modeling: Data storage and modeling are performed based on the three features of full data base, distributed framework and layered data model. Based on the distributed system infrastructure, the Hudi storage component and Spark computing engine in the Hadoop ecosystem are selected to build the MRS, an integrated structure of data lake and data warehouse; Data development: Data development for different types of data through batch processing and stream processing; Data sharing: Data sharing is carried out according to two scenarios: non-real-time data sharing and real-time data sharing.

[0019] like Figure 1 As shown in the figure, the method of data collection based on structured data is: Determine that the data source of structured data is a relational database; collect structured data through ETL tools, adopt a periodic collection method, set the collection period to T+1, and schedule it to 10 minutes or hours for some scenarios with high update frequency. The data transmission method is CDM and API; implement full collection for tables with data volume less than the set threshold, and implement incremental collection for tables with data volume greater than the set threshold; the data collection path is from the data source to the data lake Hudi.

[0020] The method of data collection based on measurement data is: The data sources of the measurement data are determined to be message queues, MQTT, IEC104 and APIs; they have the characteristics of strong timeliness, high data frequency and large data volume; real-time collection is used for collection, and integrated industrial control data collection components are used to collect measurement data. The data transmission methods include message queues, MQTT, IEC104 and APIs, and the collection frequency is in seconds; the data collection path is from the data source to the time series database to the data lake Hudi.

[0021] The method of data collection for unstructured data is: Unstructured data has the characteristics of unfixed data format and large data volume. The collection methods of unstructured data include physical collection and virtual collection. Physical collection refers to the synchronous collection of physical files and metadata, and virtual collection refers to the collection of metadata. Data transmission methods include API. The data collection path is from the data source to the data lake OBS.

[0022] like Figure 2 As shown in the figure, the data lake focuses on the aggregation of raw data and supports data centralization. The data warehouse serves the data application scenario and models, processes and stores data according to analysis requirements. The data is stored and modeled based on the three characteristics of the full data base, distributed framework and hierarchical data model. The specific method of building the MRS structure of data lake and data warehouse based on the Hudi storage component in the Hadoop ecosystem and the Spark computing engine is as follows: The MRS, an integrated structure of data lake and data warehouse, is built through a layered architecture. Data is transmitted between each layer of the architecture through SparkSQL. Data processing is carried out layer by layer to achieve data cleaning and conversion, unified modeling and statistical analysis. The integrated structure of data lake and data warehouse MRS is divided into source layer SDI, basic layer DWI, general layer DWR and market layer DM. The data lake is the source layer SDI. The source layer is constructed by source modeling. The source layer data is consistent with the source end data. The data warehouse includes the basic layer DWI, general layer DWR and market layer DM. The data of MRS, an integrated structure of data lake and data warehouse, is transmitted to the temporary table Hive communication connection, and the data of temporary table Hive is transmitted to the data mart DWS through CDM transmission method.

[0023] Preferably, the source data includes OT time series data, IT real-time data and IT non-real-time data; OT time series data is transmitted to the source layer SDI in real time through Kafka and FlinkSQL transmission methods; IT real-time data is transmitted to the source layer SDI in real time through CDL transmission method; IT non-real-time data is transmitted to the temporary table Hive in full and regularly through CDM transmission method, and the temporary table Hive is transmitted to the source layer SDI through SparkSQL; OT time series data and IT real-time data constitute structured data, and IT non-real-time data constitutes measurement data.

[0024] Preferably, during data development, batch processing operations are performed on structured data, specifically: Based on SparkSQL scripts, data is transferred from the source layer SDI to the base layer DWI, and the same type of data is aggregated; Based on the SparkSQL script, data is transferred from the basic layer DWI to the general layer DWR, and the dimension table and fact table of the data are created to calculate the business indicators; Based on the SparkSQL script, data is transferred from the general layer DWR to the market layer, and all business indicators are aggregated into a final summary table; then the data in the summary table is displayed.

[0025] Preferably, in data development, stream processing operations are performed on the measurement data, specifically: Real-time data calculation is performed through Kafka, FlinkSQL and the time series library. The time series library completes, removes duplicates, processes outliers, opens data windows, aggregates data and groups data in real time according to business needs, and displays real-time data analysis through message queues or APIs.

[0026] Preferably, data sharing includes: Share structured data through API service and data push service; API services include: Based on data sharing needs and in accordance with data sharing implementation specifications, develop and configure corresponding API interfaces, and authorize them to relevant application systems. Through clear interface authentication, interface access is monitored to enable data to be transmitted to the demander. Data push services include: Complete data processing according to the needs of the demander, and push data tables regularly through data integration components; Share measurement data through API services and message queues; API services include: API services are published based on time series databases, providing real-time query and consumption of measurement data for demanders; The message queue includes: Publish the required data as a message topic to achieve real-time transmission and consumption of data.

[0027] Embodiment 2: The source layer SDI records data versions and adds data operability fields; the table structure is consistent with the business data source; the table partition design is increased; the full table design is increased; the archive table design is increased to facilitate data re-running.

[0028] The basic layer DWI is also called the detail layer. It stores data through the 3NF model and can organize and classify data in an object-oriented way to reduce data redundancy and model stability. The basic layer DWI conducts unified relationship modeling and formulates data standards according to business scenarios. It recodes and names the source layer data according to the data standards, and converts and merges multi-source data to achieve unified models, unified standards and data integration.

[0029] The general layer DWR uses dimensional modeling or star modeling according to data analysis requirements to restore business facts and activity processes, design fact tables and dimension tables, form star models or constellation models, and support the design and implementation of indicators; through dimension extraction, reduce the impact of frequent changes in dimensional data on the stability of facts; record the history of dimension and fact changes, and restore the process of business processes and production activities.

[0030] The market-level DM is modeled using star and wide tables, and fact table information is integrated based on business analysis scenarios to ensure dimension separation and improve data reusability for flexible business queries. By building wide tables, application display is directly supported.

[0031] Stream processing is suitable for real-time analysis and shared consumption scenarios of measurement data. It meets the needs of real-time data computing and real-time application through two approaches: Flink+Kafka and time series database. The time series database, based on its own stream processing capabilities, performs data completion, deduplication, outlier processing, data windowing, data aggregation and data grouping processing in real time according to business needs, and at the same time meets the real-time analysis and display of data through message queues or APIs; the Flink+kafka service performs real-time fusion calculations on measurement data and management data, enriching the dimensions of measurement data analysis and meeting the needs of different application scenarios.

[0032] like Figure 3As shown in the figure, the data sharing scheme is designed according to the two scenarios of non-real-time data sharing and real-time data sharing. For non-real-time data sharing, two methods are designed: data API service and data push service. API service is guided by data sharing needs. According to the data sharing implementation specifications, the corresponding API interface is developed and configured, and authorized to the relevant application system. By clarifying the interface authentication method and monitoring the interface access, it ensures that the data is provided to the demand side efficiently, accurately and safely. Data push service refers to the scheduled push of data tables through data integration components after completing data processing according to demand. This method is mainly suitable for data sharing scenarios with low demand for real-time data update, but large data volume and more business tables; for real-time data sharing of measurement data, two methods are designed: API service and message queue. API service aims to provide real-time query and consumption of measurement data for demand side. This method publishes API service based on time series database, which can meet the real-time update query of measurement data and provide second-level query of historical data. Message queue refers to publishing the required data as a message topic to realize real-time transmission and consumption of data.

[0033] Embodiment three: This embodiment discloses a system for data collection, storage, modeling, development and sharing services covering IT and OT scenarios, the system includes a data collection module, a data storage and modeling module, a data development module and a data sharing module; The data collection module is used to collect data according to three data types: structured data, measurement data and unstructured data; The data storage and modeling module is used for data storage and modeling based on the three features of full data base, distributed framework and hierarchical data model. It builds the MRS, an integrated structure of data lake and data warehouse, based on the Hudi storage component and Spark computing engine in the Hadoop ecosystem. Data development module, used for data development of different types of data through batch processing and stream processing; The data sharing module is used to share data according to two scenarios: non-real-time data sharing and real-time data sharing.

[0034] This embodiment also discloses a data transmission module, which is a submodule of the data storage and modeling module and is used for data transmission. OT time series data is transmitted to the source layer SDI in real time through Kafka and FlinkSQL transmission methods; IT real-time data is transmitted to the source layer SDI in real time through CDL transmission method; IT non-real-time data is transmitted to the temporary table Hive in full and regularly through CDM transmission method, and the temporary table Hive is transmitted to the source layer SDI through SparkSQL; OT time series data and IT real-time data constitute structured data, and IT non-real-time data constitutes measurement data.

[0035] This embodiment also discloses a batch processing module, which is a submodule of the data development module and is used to perform batch processing operations on structured data. Based on the SparkSQL script, data is transmitted from the source layer SDI to the base layer DWI, and data of the same type is aggregated; Based on the SparkSQL script, data is transferred from the basic layer DWI to the general layer DWR, and the dimension table and fact table of the data are created to calculate the business indicators; Based on the SparkSQL script, data is transferred from the general layer DWR to the market layer, and all business indicators are aggregated into a final summary table; then the data in the summary table is displayed.

[0036] This embodiment also discloses a stream processing module, which is a submodule of the data development module, and is used to perform stream processing operations on the measurement data, and perform real-time data calculations through Kafka+FlinkSQL and the time series library; wherein the time series library completes, deduplicates, processes outliers, performs data windowing, data aggregation, and data grouping in real time according to business needs, and performs real-time analysis and display of data in the form of message queues or APIs.

[0037] This embodiment also discloses an API service module, which is a submodule of the data sharing module and is used for data sharing of structured data and measurement data. Based on data sharing requirements and in accordance with data sharing implementation specifications, the corresponding API interface is developed and configured, and authorized to related application systems. The interface access is monitored by clarifying interface authentication, so that data can be transmitted to the demander. The real-time sharing API service of measurement data is published based on the time series database.

[0038] This embodiment also discloses a data push service module, which is a submodule of the data sharing module and is used for data sharing of structured data, completing data processing according to the needs of the demander, and realizing the scheduled push of data tables through the data integration component.

[0039] This embodiment also discloses a message queue module, which is a submodule of the data sharing module and is used for data sharing of measurement data. The required data is published as a message topic to realize real-time transmission and consumption of data.

Claims

1. A method for data collection, storage, modeling development and shared services covering IT and OT scenarios, characterized in that: include: Data collection: IT data and OT data are collected according to three data types: structured data, measurement data, and unstructured data; The method of data collection with structured data is: Set the data source of structured data as a relational database; collect structured data through ETL tools, use periodic collection methods, and use CDM or API for data transmission; The method of data collection based on measurement data is: Set the data source of the measurement data to message queue, MQTT, IEC104 or API; use real-time collection method to collect measurement data, and use integrated industrial control data collection components to collect measurement data; The method of data collection based on unstructured data is: Set the collection method of unstructured data to physical collection and virtual collection. Physical collection includes the synchronous collection of physical files and metadata, and virtual collection includes the collection of metadata. Data storage and modeling: Data storage and modeling are performed based on the three features of full data base, distributed framework, and layered data model. An integrated data lake and data warehouse are built based on the storage components and computing engines in the distributed system infrastructure. Data development: Data development of structured data, measurement data and unstructured data through batch processing and stream processing; Data sharing: Data sharing is carried out according to two scenarios: non-real-time data sharing and real-time data sharing.

2. A method for data collection, storage, modeling development and shared services covering IT and OT scenarios according to claim 1, characterized in that: When data is collected with structured data, full data collection is performed for tables with data volumes less than the set threshold, and incremental data collection is performed for tables with data volumes greater than the set threshold. The data collection path is from the data source to the data lake.

3. A method for data collection, storage, modeling development and shared services covering IT and OT scenarios according to claim 2, characterized in that: When data is collected based on measurement data, the data transmission methods include message queues, MQTT, IEC104 and API, and the collection frequency is in seconds; the data collection path is from the data source to the time series database to the data lake.

4. A method for data collection, storage, modeling development and shared services covering IT and OT scenarios according to claim 3, characterized in that: When unstructured data is collected, the data transmission method includes API; the data collection path is from the data source to the data lake.

5. A method for data collection, storage, modeling development and shared services covering IT and OT scenarios according to claim 4, characterized in that: Data storage and modeling are performed based on the three features of full data base, distributed framework and hierarchical data model. An integrated data lake and data warehouse structure MRS is built based on the storage components and computing engines in the distributed system infrastructure. The specific methods are as follows: MRS, an integrated structure of data lake and data warehouse, is built through a layered architecture, and data is transmitted between each layer of the architecture through SparkSQL; The integrated structure of data lake and data warehouse MRS is divided into source layer SDI, basic layer DWI, general layer DWR and market layer DM. The data lake is the source layer SDI. The source layer is constructed by source modeling. The source layer data is consistent with the source end data. The data warehouse includes the basic layer DWI, general layer DWR and market layer DM. The data of MRS, an integrated structure of data lake and data warehouse, is transmitted to the temporary table Hive communication connection, and the data of temporary table Hive is transmitted to the data mart DWS through CDM transmission method.

6. A method for data collection, storage, modeling development and shared services covering IT and OT scenarios according to claim 5, characterized in that: Source data includes OT time series data, IT real-time data and IT non-real-time data; OT time series data is transmitted to the source layer SDI in real time through Kafka and FlinkSQL transmission methods; IT real-time data is transmitted to the source layer SDI in real time through CDL transmission method; IT non-real-time data is transmitted to the temporary table Hive in full and at regular intervals through CDM transmission method, and the temporary table Hive is transmitted to the source layer SDI through SparkSQL; OT time series data and IT real-time data constitute structured data, and IT non-real-time data constitutes measurement data.

7. A method for data collection, storage, modeling development and shared services covering IT and OT scenarios according to claim 6, characterized in that: During data development, batch processing is performed on structured data, specifically: Based on SparkSQL, data is transferred from the source layer SDI to the base layer DWI, and the same type of data is aggregated; Based on SparkSQL, data is transferred from the basic layer DWI to the general layer DWR, and dimension tables and fact tables of the data are created to calculate business indicators; Based on SparkSQL, data is transferred from the general layer DWR to the mart layer, all business indicators are aggregated into the final summary table, and then the data in the summary table is displayed.

8. The method for data collection, storage, modeling development and shared services covering IT and OT scenarios according to claim 7 is characterized in that: During data development, stream processing operations are performed on measurement data, specifically: Real-time data calculation is performed through Kafka+FlinkSQL and the time series library. The time series library completes, removes duplications, processes outliers, performs data windowing, aggregates, and groups data in real time according to business needs, and displays real-time data analysis through message queues or APIs.

9. A method for data collection, storage, modeling development and shared services covering IT and OT scenarios according to claim 8, characterized in that: Data sharing includes: Data sharing of structured data through API services and data push services; API services include: Based on data sharing needs and in accordance with data sharing implementation specifications, develop and configure corresponding API interfaces, and authorize them to relevant application systems. Through clear interface authentication, interface access is monitored to enable data to be transmitted to the demander. Data push services include: Complete data processing according to the needs of the demander, and push data tables regularly through data integration components; Share measurement data through API services and message queues; API services include: API services are published based on time series databases, providing real-time query and consumption of measurement data for demanders; The message queue includes: Publish the required data as a message topic to achieve real-time transmission and consumption of data.

10. A system for data collection, storage, modeling development and shared services covering IT and OT scenarios used in the method for data collection, storage, modeling development and shared services covering IT and OT scenarios as described in claim 9, characterized in that: The system includes data collection module, data storage and modeling module, data development module and data sharing module; The data collection module is used to collect data according to three data types: structured data, measurement data and unstructured data; The data storage and modeling module is used for data storage and modeling based on the three features of full data base, distributed framework and hierarchical data model, and to build an integrated data lake and data warehouse integrated structure MRS based on the storage components and computing engines in the distributed system infrastructure; Data development module, used for data development of different types of data through batch processing and stream processing; The data sharing module is used to share data according to two scenarios: non-real-time data sharing and real-time data sharing.

Images