A method and system for data collection, storage, modeling development, and shared services covering IT and OT scenarios

Abstract

The present invention discloses a method and system for data collection, storage, modeling development and sharing services covering IT and OT scenarios, including data collection, which collects IT data and OT data according to three data types: structured data, measurement data and unstructured data; data storage and modeling, which performs data storage and modeling based on the three characteristics of full data base, distributed framework and hierarchical data model, and builds a data lake and data warehouse integrated structure MRS based on the storage components and computing engine in the distributed system infrastructure; data development, which develops data of different types through batch processing and stream processing; data sharing, which designs data sharing technical solutions according to two scenarios: non-real-time data sharing and real-time data sharing; it can realize unified storage of IT and OT data, build an integrated data lake and data warehouse, and lay the foundation for the integrated application of the two types of data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and system for data collection, storage, modeling development, and sharing services covering IT and OT scenarios. Background Art

[0002] IT data and OT data are two common types of data with significantly different characteristics. IT data has a complex data structure, low real-time requirements, and a large amount of historical data. OT data has a simple data structure and strong real-time performance. In existing technical solutions, IT data and OT data are processed separately using two different technical routes. IT data is mostly stored and developed using relational databases, while OT data is mostly stored and developed using time-series databases. In terms of data acquisition and storage, IT data and OT data are physically distributed across different types of storage components, without forming a unified data lake, which makes it difficult to integrate and apply the two types of data. In terms of data modeling and development, existing technologies generally perform dimensional modeling based on the source layer and lack a unified model management layer, making it difficult to consolidate and manage the results of data integration and standardization work. At the same time, both IT and OT data are mostly completed based on the SQL functions and job functions of database components, lacking script dependency management, unified scheduling, and monitoring, making it difficult to meet the needs of complex data processing scenarios. In terms of data sharing services, existing technologies are mostly based on database-level authorization and access, which easily forms a spider web. Directly sharing data at the database level also brings security issues. Therefore, it is necessary to design a method and system for data acquisition, storage, modeling, development, and sharing services covering IT and OT scenarios to solve the above problems. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method and system for data collection, storage, modeling development and sharing services covering IT and OT scenarios, so as to solve the problem in the existing technology that IT data and OT data are difficult to integrate and process due to their different characteristics. It can meet the different requirements of IT data and OT data in terms of data structure, update method, production frequency, etc., realize unified storage of the two types of data, build an integrated data lake and data warehouse, and lay the foundation for the integrated application of the two types of data.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] A method for data collection, storage, modeling development, and shared services covering IT and OT scenarios, including:

[0006] Data collection: IT data and OT data are collected according to three data types: structured data, measurement data, and unstructured data;

[0007] Data storage and modeling: Data storage and modeling are based on the three characteristics of a full data base, a distributed framework, and a layered data model. Based on the distributed system infrastructure, the Hudi storage component and the Spark computing engine in the Hadoop ecosystem are selected to build the MRS, an integrated data lake and data warehouse structure.

[0008] Data development: Data development for different types of data through batch processing and stream processing;

[0009] Data sharing: Data sharing is carried out according to two scenarios: non-real-time data sharing and real-time data sharing.

[0010] Preferably, the method for data aggregation using structured data is:

[0011] Determine that the data source of structured data is a relational database; collect structured data through ETL tools, adopt a periodic collection method, set the collection period to T+1, and use CDM and API for data transmission; implement full collection for tables with data volume less than the set threshold, and implement incremental collection for tables with data volume greater than the set threshold; the data collection path is from the data source to the data lake Hudi.

[0012] Preferably, the method for data aggregation using measurement data is:

[0013] Determine the data source of the measurement data as a message queue, MQTT, IEC104, or API; use real-time collection methods and integrated industrial control data acquisition components to collect measurement data. Data transmission methods include message queues, MQTT, IEC104, and APIs, with a collection frequency of seconds; the data collection path is from the data source to the time series database to the data lake Hudi.

[0014] Preferably, the method for data aggregation using unstructured data is:

[0015] There are two ways to collect unstructured data: physical collection and virtual collection. Physical collection refers to the synchronous collection of physical files and metadata, while virtual collection refers to the collection of metadata. Data transmission methods include APIs. The data collection path is from the data source to the data lake OBS.

[0016] Preferably, data storage and modeling are performed based on the three features of full data base, distributed framework, and hierarchical data model. Based on the distributed system infrastructure, the Hudi storage component in the Hadoop ecosystem and the Spark computing engine are selected to build the data lake and data warehouse integrated structure MRS. The specific method is as follows:

[0017] MRS, an integrated data lake and data warehouse structure, is built through a layered architecture, with SparkSQL used for data transmission between each layer of the architecture.

[0018] The integrated data lake and data warehouse structure MRS is divided into the source layer SDI, the basic layer DWI, the general layer DWR, and the market layer DM. The data lake serves as the source layer SDI. The source layer is constructed using the source modeling method. The source layer data is consistent with the source end data. The data warehouse includes the basic layer DWI, the general layer DWR, and the market layer DM.

[0019] The data of MRS, an integrated data lake and data warehouse structure, is transferred to the temporary table Hive communication connection. The data of the temporary table Hive is transferred to the data mart DWS through the CDM transmission method.

[0020] Preferably, the source data includes OT time series data, IT real-time data and IT non-real-time data; OT time series data is transmitted to the source layer SDI in real time through Kafka and FlinkSQL transmission methods; IT real-time data is transmitted to the source layer SDI in real time through CDL transmission method; IT non-real-time data is transmitted to the temporary table Hive in full and on a regular basis through CDM transmission method, and the temporary table Hive is transmitted to the source layer SDI through SparkSQL; OT time series data and IT real-time data constitute structured data, and IT non-real-time data constitutes measurement data.

[0021] Preferably, during data development, batch processing operations are performed on structured data, specifically:

[0022] Based on SparkSQL scripts, data is transferred from the source layer SDI to the base layer DWI, and data of the same type is aggregated;

[0023] Based on SparkSQL scripts, data is transferred from the basic layer DWI to the general layer DWR, dimension tables and fact tables of the data are created, and business indicators are calculated;

[0024] Based on the SparkSQL script, data is transferred from the general layer DWR to the market layer, and all business indicators are aggregated into the final summary table; then the data in the summary table is displayed.

[0025] Preferably, in step S3, during data development, stream processing operations are performed on the measurement data, specifically:

[0026] Real-time data calculations are performed through Kafka, FlinkSQL, and the time series library. The time series library completes, deduplicates, processes outliers, performs data windowing, aggregates, and grouping according to business needs, while also performing real-time data analysis and display via message queues or APIs.

[0027] Preferably, data sharing includes:

[0028] Sharing of structured data is done through API services and data push services;

[0029] API services include:

[0030] Based on data sharing needs and in accordance with data sharing implementation specifications, develop and configure corresponding API interfaces, and authorize them to relevant application systems. Through clear interface authentication, monitor interface access and transmit data to the demander.

[0031] Data push services include:

[0032] Complete data processing according to the needs of the demander and implement scheduled push of data tables through data integration components;

[0033] Share measurement data through API services and message queues;

[0034] API services include:

[0035] API services are published based on time series databases, providing real-time query and consumption of measurement data for demanders.

[0036] The message queue includes:

[0037] Publish the required data as a message topic to achieve real-time transmission and consumption of data.

[0038] Preferably, a system covering data collection, storage, modeling, development and sharing services for IT and OT scenarios, including a data collection module, a data storage and modeling module, a data development module and a data sharing module;

[0039] The data collection module is used to collect data according to three data types: structured data, measurement data and unstructured data;

[0040] The data storage and modeling module is used for data storage and modeling based on the three characteristics of a full data base, a distributed framework, and a hierarchical data model. It builds the MRS, an integrated data lake and data warehouse structure, based on the Hudi storage component in the Hadoop ecosystem and the Spark computing engine.

[0041] Data development module, used for data development of different types of data through batch processing and stream processing;

[0042] The data sharing module is used to share data in two scenarios: non-real-time data sharing and real-time data sharing.

[0043] The method and system for data collection, storage, modeling, development, and sharing services covering IT and OT scenarios provided by the present invention have the following beneficial effects:

[0044] 1. This method designs two technical paths: periodic collection and real-time collection, to meet the different requirements of IT data and OT data in terms of data structure, update method, production frequency, etc., to achieve unified storage of the two types of data, build an integrated data lake and data warehouse, and lay the foundation for the integrated application of the two types of data.

[0045] 2. This method designs a four-layer data model consisting of the source layer SDI, the basic layer DWI, the general layer DWR, and the market layer DM, which not only supports large-scale data concentration but also meets the modeling requirements of data application scenarios; innovatively designs the basic layer, carries out unified relational modeling and formulates data standards according to business scenarios, recodes and names the source layer data according to the data standards, and converts and merges multi-source data to achieve a unified model, unified standard, and data integration; at the same time, this solution breaks the "black box" data development idea, and based on the basic capabilities of SparkSQL, designs script orchestration and scheduling to achieve dependency management, unified scheduling, and monitoring of development operations.

[0046] 3. This method integrates stream processing and batch processing to design a unified service gateway, realizing unified authentication and authorization of data services in both IT and OT scenarios, thus ensuring data security while efficiently carrying out data sharing. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The present invention will be further described below with reference to the accompanying drawings and examples:

[0048] Figure 1 It is a schematic diagram of the process of data collection in the present invention;

[0049] Figure 2 It is a schematic diagram of the process of data storage and modeling in the present invention;

[0050] Figure 3 It is a flowchart of data sharing in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] Example 1:

[0052] A method for data collection, storage, modeling development, and shared services covering IT and OT scenarios, including:

[0053] Data collection: IT data and OT data are collected according to three data types: structured data, measurement data, and unstructured data;

[0054] Data storage and modeling: Data storage and modeling are based on the three characteristics of a full data base, a distributed framework, and a layered data model. Based on the distributed system infrastructure, the Hudi storage component and the Spark computing engine in the Hadoop ecosystem are selected to build the MRS, an integrated data lake and data warehouse structure.

[0055] Data development: Data development for different types of data through batch processing and stream processing;

[0056] Data sharing: Data sharing is carried out according to two scenarios: non-real-time data sharing and real-time data sharing.

[0057] like Figure 1 As shown in the figure, the method of data collection based on structured data is:

[0058] Determine that the data source of structured data is a relational database; collect structured data through ETL tools, adopt a periodic collection method, set the collection cycle to T+1, and for some scenarios with high update frequency, it can be scheduled by 10 minutes or hours. The data transmission method is CDM and API; implement full collection for tables with data volume less than the set threshold, and implement incremental collection for tables with data volume greater than the set threshold; the data collection path is from the data source to the data lake Hudi.

[0059] The method of data collection based on measurement data is as follows:

[0060] The data sources of the measurement data are determined to be message queues, MQTT, IEC104, and APIs; they have the characteristics of strong timeliness, high data frequency, and large data volume; real-time collection is used for collection, and integrated industrial control data collection components are used to collect measurement data. The data transmission methods include message queues, MQTT, IEC104, and APIs, and the collection frequency is in seconds; the data collection path is from the data source to the time series database to the data lake Hudi.

[0061] The method for data collection of unstructured data is:

[0062] Unstructured data has the characteristics of non-fixed data format and large data volume. Unstructured data can be collected in two ways: physical collection and virtual collection. Physical collection refers to the synchronous collection of physical files and metadata, while virtual collection refers to the collection of metadata. Data transmission methods include APIs. The data collection path is from the data source to the data lake OBS.

[0063] like Figure 2As shown, the data lake focuses on the aggregation of raw data and supports large-scale data centralization. The data warehouse serves data application scenarios and models, processes, and stores data according to analytical requirements. Data storage and modeling are based on the three characteristics of a full data base, a distributed framework, and a hierarchical data model. The specific method for building a data lake and data warehouse integrated structure MRS based on the Hudi storage component in the Hadoop ecosystem and the Spark computing engine is as follows:

[0064] MRS, an integrated data lake and data warehouse structure, is built through a layered architecture. SparkSQL is used to transfer data between each layer of the architecture. Data processing is carried out layer by layer to achieve data cleaning and conversion, unified modeling, and statistical analysis.

[0065] The integrated data lake and data warehouse structure MRS is divided into the source layer SDI, the basic layer DWI, the general layer DWR, and the market layer DM. The data lake serves as the source layer SDI. The source layer is constructed using the source modeling method. The source layer data is consistent with the source end data. The data warehouse includes the basic layer DWI, the general layer DWR, and the market layer DM.

[0066] The data of MRS, an integrated data lake and data warehouse structure, is transferred to the temporary table Hive communication connection. The data of the temporary table Hive is transferred to the data mart DWS through the CDM transmission method.

[0067] Preferably, the source data includes OT time series data, IT real-time data and IT non-real-time data; OT time series data is transmitted to the source layer SDI in real time through Kafka and FlinkSQL transmission methods; IT real-time data is transmitted to the source layer SDI in real time through CDL transmission method; IT non-real-time data is transmitted to the temporary table Hive in full and on a regular basis through CDM transmission method, and the temporary table Hive is transmitted to the source layer SDI through SparkSQL; OT time series data and IT real-time data constitute structured data, and IT non-real-time data constitutes measurement data.

[0068] Preferably, during data development, batch processing operations are performed on structured data, specifically:

[0069] Based on SparkSQL scripts, data is transferred from the source layer SDI to the base layer DWI, and data of the same type is aggregated;

[0070] Based on SparkSQL scripts, data is transferred from the basic layer DWI to the general layer DWR, dimension tables and fact tables of the data are created, and business indicators are calculated;

[0071] Based on the SparkSQL script, data is transferred from the general layer DWR to the market layer, and all business indicators are aggregated into the final summary table; then the data in the summary table is displayed.

[0072] Preferably, during data development, stream processing operations are performed on the measurement data, specifically:

[0073] Real-time data calculations are performed through Kafka, FlinkSQL, and the time series library. The time series library completes, deduplicates, processes outliers, performs data windowing, aggregates, and grouping according to business needs, while also performing real-time data analysis and display via message queues or APIs.

[0074] Preferably, data sharing includes:

[0075] Sharing of structured data is done through API services and data push services;

[0076] API services include:

[0077] Based on data sharing needs and in accordance with data sharing implementation specifications, develop and configure corresponding API interfaces, and authorize them to relevant application systems. Through clear interface authentication, monitor interface access and transmit data to the demander.

[0078] Data push services include:

[0079] Complete data processing according to the needs of the demander and implement scheduled push of data tables through data integration components;

[0080] Share measurement data through API services and message queues;

[0081] API services include:

[0082] API services are published based on time series databases, providing real-time query and consumption of measurement data for demanders.

[0083] The message queue includes:

[0084] Publish the required data as a message topic to achieve real-time transmission and consumption of data.

[0085] Example 2:

[0086] The source layer SDI records data versions and adds data operability fields; the table structure is consistent with the business data source; table partition design is added; full table design is added; archive table design is added to facilitate data re-running.

[0087] The base layer, DWI, also known as the detail layer, stores data using the 3NF model. This allows for object-oriented data organization and classification, reducing data redundancy and model stability. The base layer DWI implements unified relational modeling and establishes data standards based on business scenarios. It also recodes and names source layer data according to these standards and converts and merges data from multiple sources, achieving unified models, unified standards, and data integration.

[0088] The general layer DWR uses dimensional modeling or star modeling according to data analysis requirements to restore business facts and activity processes, design fact tables and dimension tables, and form star models or constellation models to support the design and implementation of indicators; through dimension refinement, it reduces the impact of frequent changes in dimensional data on the stability of facts; records the history of dimension and fact changes, and restores the process of business processes and production activities.

[0089] The marketplace-level DM is modeled using star schemas and wide tables. It integrates fact table information based on business analysis scenarios, ensures dimension separation, and improves data reusability for flexible business queries. By building wide tables, it directly supports application presentation.

[0090] Stream processing is suitable for real-time analysis and shared consumption of measurement data. It meets the needs of real-time data computing and applications through the combination of Flink + Kafka and time-series databases. Time-series databases, based on their streaming capabilities, perform real-time data completion, deduplication, outlier processing, data windowing, data aggregation, and data grouping according to business needs. They also support real-time data analysis and display through message queues or APIs. Flink + Kafka services integrate measurement and management data for real-time computing, enriching the dimensions of measurement data analysis and meeting the needs of diverse application scenarios.

[0091] like Figure 3As shown, the data sharing solution is designed for both non-real-time and real-time data sharing scenarios. For non-real-time data sharing, two approaches are designed: data API services and data push services. API services are driven by data sharing needs. According to data sharing implementation specifications, corresponding API interfaces are developed and configured, authorized to relevant application systems, and access is monitored through clear interface authentication methods to ensure efficient, accurate, and secure data delivery to the user. Data push services implement scheduled push of data tables through data integration components after data processing is completed as required. This approach is primarily suitable for data sharing scenarios where real-time data updates are not a high requirement, but data volumes and business tables are large. For real-time measurement data sharing, two approaches are designed: API services and message queues. API services aim to provide real-time query and consumption of measurement data for users. This approach, based on a time-series database, publishes API services, enabling both real-time query and query of measurement data and second-level query of historical data. Message queues publish the required data as message topics, enabling real-time data transmission and consumption.

[0092] Example 3:

[0093] This embodiment discloses a system for data collection, storage, modeling, development, and sharing services covering IT and OT scenarios. The system includes a data collection module, a data storage and modeling module, a data development module, and a data sharing module.

[0094] The data collection module is used to collect data according to three data types: structured data, measurement data and unstructured data;

[0095] The data storage and modeling module is used for data storage and modeling based on the three characteristics of a full data base, a distributed framework, and a hierarchical data model. It builds the MRS, an integrated data lake and data warehouse structure, based on the Hudi storage component in the Hadoop ecosystem and the Spark computing engine.

[0096] Data development module, used for data development of different types of data through batch processing and stream processing;

[0097] The data sharing module is used to share data in two scenarios: non-real-time data sharing and real-time data sharing.

[0098] This embodiment also discloses a data transmission module, which is a submodule of the data storage and modeling module and is used for data transmission. OT time series data is transmitted to the source layer SDI in real time through Kafka and FlinkSQL transmission methods; IT real-time data is transmitted to the source layer SDI in real time through CDL transmission method; IT non-real-time data is transmitted to the temporary table Hive in full and on a regular basis through CDM transmission method, and the temporary table Hive is transmitted to the source layer SDI through SparkSQL; OT time series data and IT real-time data constitute structured data, and IT non-real-time data constitutes measurement data.

[0099] This embodiment also discloses a batch processing module, which is a submodule of the data development module and is used to perform batch processing operations on structured data. Based on SparkSQL scripts, data is transferred from the source layer SDI to the base layer DWI, and data of the same type is aggregated.

[0100] Based on SparkSQL scripts, data is transferred from the basic layer DWI to the general layer DWR, dimension tables and fact tables of the data are created, and business indicators are calculated;

[0101] Based on the SparkSQL script, data is transferred from the general layer DWR to the market layer, and all business indicators are aggregated into the final summary table; then the data in the summary table is displayed.

[0102] This embodiment also discloses a stream processing module, which is a submodule of the data development module and is used to perform stream processing operations on measurement data and perform real-time data calculations through Kafka+FlinkSQL and the time series library; the time series library completes, deduplicates, processes outliers, opens data windows, aggregates data, and groups data in real time according to business needs, and performs real-time data analysis and display in the form of message queues or APIs.

[0103] This embodiment also discloses an API service module, which is a submodule of the data sharing module and is used for data sharing of structured data and measurement data. Based on data sharing requirements and in accordance with data sharing implementation specifications, the corresponding API interface is developed and configured, and authorized to the relevant application system. By clearly identifying the interface authentication, the interface access status is monitored to enable data to be transmitted to the demander. The real-time sharing API service of measurement data is published based on the time series database.

[0104] This embodiment also discloses a data push service module, which is a submodule of the data sharing module and is used for data sharing of structured data, completing data processing according to the needs of the demander, and realizing the scheduled push of data tables through the data integration component.

[0105] This embodiment also discloses a message queue module, which is a submodule of the data sharing module and is used for data sharing of measurement data. The required data is published as a message topic to achieve real-time transmission and consumption of data.

Claims

1. A method for data collection, storage, modeling development and shared services covering IT and OT scenarios, characterized by: include: Data collection: IT data and OT data are collected according to three data types: structured data, measurement data, and unstructured data; The method of data collection based on structured data is: Set the data source of structured data to a relational database; collect structured data through ETL tools, adopt a periodic collection method, and transmit data through CDM or API; The method of data collection based on measurement data is as follows: Set the data source of measurement data to message queue, MQTT, IEC104 or API; use real-time acquisition mode and integrated industrial control data acquisition components to collect measurement data; The method of data collection based on unstructured data is: Set the collection method of unstructured data to physical collection and virtual collection. Physical collection includes the simultaneous collection of physical files and metadata, while virtual collection includes the collection of metadata. Data storage and modeling: Data storage and modeling are performed based on the three characteristics of a full data base, a distributed framework, and a layered data model. An integrated data lake and data warehouse are built based on the storage components and computing engines in the distributed system infrastructure. The specific methods are as follows: MRS, an integrated data lake and data warehouse structure, is built through a layered architecture, with SparkSQL used for data transmission between each layer of the architecture. The integrated data lake and data warehouse structure MRS is divided into the source layer SDI, the basic layer DWI, the general layer DWR, and the market layer DM. The data lake serves as the source layer SDI. The source layer is constructed using the source modeling method. The source layer data is consistent with the source end data. The data warehouse includes the basic layer DWI, the general layer DWR, and the market layer DM. The data from the MRS, a data lake and data warehouse integrated structure, is transferred to the temporary table Hive through a communication connection. The data from the temporary table Hive is then transferred to the data mart DWS through the CDM transmission method. Source data includes OT time series data, IT real-time data, and IT non-real-time data. OT time series data is transmitted to the source layer SDI in real time via Kafka and FlinkSQL. IT real-time data is transmitted to the source layer SDI in real time via CDL. IT non-real-time data is transmitted in full and regularly to the temporary table Hive via CDM. The temporary table Hive is transmitted to the source layer SDI via SparkSQL. OT time series data and IT real-time data constitute structured data, while IT non-real-time data constitutes measurement data. Data development: Data development is performed on structured data, measurement data, and unstructured data through batch and stream processing. Batch processing operations are performed on structured data, specifically: Based on SparkSQL, data is transferred from the source layer SDI to the base layer DWI, and the same type of data is aggregated; Based on SparkSQL, data is transferred from the basic layer DWI to the general layer DWR, where dimension tables and fact tables are created to calculate business indicators. Based on SparkSQL, data is transferred from the general layer DWR to the market layer, all business indicators are aggregated into the final summary table, and then the data in the summary table is displayed; Data sharing: Data sharing is carried out according to two scenarios: non-real-time data sharing and real-time data sharing.

2. A method for data collection, storage, modeling development and shared services covering IT and OT scenarios according to claim 1, characterized in that: When performing data aggregation with structured data, full data collection is performed for tables with data volumes less than the set threshold, and incremental data collection is performed for tables with data volumes greater than the set threshold. The data collection path is from the data source to the data lake.

3. The method for data collection, storage, modeling development and shared services covering IT and OT scenarios according to claim 2 is characterized in that: When data is collected using measurement data, data transmission methods include message queues, MQTT, IEC104, and APIs, with a collection frequency of seconds; the data collection path is from the data source to the time series database to the data lake.

4. The method for data collection, storage, modeling development and shared services covering IT and OT scenarios according to claim 3 is characterized in that: When collecting unstructured data, the data transmission method includes API; the data collection path is from the data source to the data lake.

5. The method for data collection, storage, modeling development and shared services covering IT and OT scenarios according to claim 1 is characterized in that: During data development, stream processing operations are performed on measurement data, specifically: Real-time data calculations are performed through Kafka+FlinkSQL and the time series library. The time series library completes, deduplicates, processes outliers, performs data windowing, aggregates, and grouping according to business needs. It also performs real-time data analysis and display through message queues or APIs.

6. The method for data collection, storage, modeling development and shared services covering IT and OT scenarios according to claim 5 is characterized in that: Data sharing includes: Sharing of structured data through API services and data push services; API services include: Based on data sharing needs and in accordance with data sharing implementation specifications, develop and configure corresponding API interfaces, and authorize them to relevant application systems. Through clear interface authentication, monitor interface access and transmit data to the demander. Data push services include: Complete data processing according to the needs of the demander and implement scheduled push of data tables through data integration components; Sharing of measurement data through API services and message queues; API services include: API services are published based on time series databases, providing real-time query and consumption of measurement data for demanders. The message queue includes: Publish the required data as a message topic to achieve real-time transmission and consumption of data.

7. A system for data collection, storage, modeling, development and sharing services covering IT and OT scenarios, used in the method for data collection, storage, modeling, development and sharing services covering IT and OT scenarios as described in claim 6, characterized in that: The system includes data collection module, data storage and modeling module, data development module and data sharing module; The data collection module is used to collect data according to three data types: structured data, measurement data and unstructured data; The data storage and modeling module is used for data storage and modeling based on the three characteristics of full data base, distributed framework and layered data model. It builds an integrated data lake and data warehouse structure MRS based on the storage components and computing engine in the distributed system infrastructure. Data development module, used for data development of different types of data through batch processing and stream processing; The data sharing module is used to share data in two scenarios: non-real-time data sharing and real-time data sharing.

Images