IT and OT data fusion governance and capitalization system on big data platform

Abstract

The invention provides a system for fusion governance and capitalization of IT and OT data on a big data platform, and belongs to the technical field of IT and OT data. Comprising a data integration layer, a data storage layer, a data governance layer, a data encryption layer, an capitalization processing layer and a host life cycle management layer, the data integration layer is used for extracting and integrating IT and OT data, the data storage layer is used for storing the extracted IT and OT data, and the data governance layer is used for processing and managing the data. The IT and OT data are fully fused and processed on the big data platform, the data are subjected to capitalization processing and evaluation, the aspects of data integration, governance, analysis, application and the like are comprehensively considered, and effective fusion and governance of the IT and OT data on the big data platform are ensured through accurate data management and intelligent analysis, so that the data integration and management efficiency is improved. The availability and the business value of the data are improved, an enterprise can effectively utilize IT and OT data, and the business decision-making capability and the operation efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the field of IT and OT data technology, and in particular to a system for integrating management and assetization of IT and OT data on a big data platform. Background Art

[0002] IT refers to information technology, which mainly covers various application systems in the enterprise, such as ERP, MES, EAM, OA, etc. These systems are responsible for internal information management, business processing, data analysis and other tasks in the enterprise. The technology is widely used in production planning management, inventory management, supply chain management and other links in the manufacturing industry.

[0003] OT refers to operational technology, which mainly involves the hardware and software management of the production workshop, including PLC, SCADA, gateways, robots, etc. OT technology is mainly used in on-site operation control, equipment monitoring, data collection and other fields, and is the key to ensuring the smooth progress of the production process.

[0004] The main difference between IT and OT is that IT focuses more on information management and data analysis within the enterprise, while OT focuses more on operational control and equipment monitoring at the production site. However, with the development of Industry 4.0, the boundary between IT and OT has gradually blurred, and the two have begun to show a trend of integration.

[0005] However, the existing fusion methods do not completely integrate IT and OT data, and cannot effectively integrate and govern IT and OT data on the big data platform, and cannot assetize data. Therefore, this application provides a system for the integration, governance and assetization of IT and OT data on the big data platform to meet the needs. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a system for the integration, governance and assetization of IT and OT data on a big data platform to solve existing problems.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: A system for the integrated governance and assetization of IT and OT data on a big data platform comprises: a data integration layer, a data storage layer, a data governance layer, a data encryption layer, an assetization processing layer and a host lifecycle management layer, wherein the data integration layer is used to extract and integrate IT and OT data, the data storage layer is used to store the extracted IT and OT data, the data governance layer is used to process and manage data, the data encryption layer is used to encrypt and manage permissions for IT and OT data, the assetization processing layer is used to assetize data, and the data lifecycle management layer is used to manage the entire life cycle of data from creation to deletion.

[0008] In some examples, the data integration layer includes a data pipeline, which is used to extract, transform and load IT and OT data from different sources into the data storage layer, and the extraction and transformation of the IT and OT data adopts ETL / ELT tools, such as Apache NiFi, Talend, etc.; The data integration layer also includes an API integration module, which is used to implement real-time data stream processing and integration using an API such as a RESTful API and a message middleware such as Apache Kafka.

[0009] In some examples, the data storage layer includes data and data warehouses, and the data lake is used to store raw and multi-structured IT and OT data to support flexible analysis and processing; A data warehouse is used to manage structured data and support complex queries and data analysis. The data lake may adopt Amazon S3 or Azure Data Lake, and the data warehouse may adopt Snowflake or Google BigQuery.

[0010] In some examples, the data governance layer includes a data quality management module and a data catalog building module, wherein the data quality management module is used to implement data cleaning, verification, and standardization to improve data quality; The data catalog building module is used to create a data catalog to record the data source, structure and business meaning, and can be performed using tools such as Apache Atlas or Collibra.

[0011] In some examples, the data cleaning is performed using the following method: A. Process missing values ​​by directly deleting them and filling them with the mean; B. Detect outliers; C. Perform data standardization and normalization; D. Perform logarithmic transformation and check data consistency.

[0012] In some examples, mean imputation of missing values ​​is performed using the following formula:

[0013] Where Q is the imputed value, N is the number of non-missing values, and xi is the non-missing observation value; The outlier detection is performed using the standard deviation method and the box plot method. The specific formula of the standard deviation method is as follows: Outliers

[0014] Where is the mean and is the standard deviation, which is usually 2 or 3; The formula of the box plot method is as follows: Outliers

[0015] Where Q1 and Q3 are the first and third quartiles, and IQR=Q3-Q1 is the interquartile range; The data standardization and normalization are specifically as follows: converting the data into a distribution with a mean of 0 and a standard deviation of 1, and scaling the data to the interval [0, 1], using the following formula;

[0016] Where is the mean and is the standard deviation;

[0017] Where and are the minimum and maximum values ​​of the data respectively; The logarithmic transformation is performed using the following formula

[0018] The addition of 1 is to avoid the logarithm value being zero.

[0019] In some examples, the data encryption layer includes a data encryption module, which is used to perform data encryption and permission management to protect data security, and can be performed using tools such as AWS IAM and Apache Ranger.

[0020] In some examples, the asset processing plant includes a data classification and labeling module, an ROI analysis module, a data productization module, and a data transaction module, wherein the data classification and labeling module is used to classify and label the data according to its business value and purpose, so as to facilitate management and use, for example, labeling according to the sensitivity of the data, business department, or usage scenario; The ROI analysis module is used to analyze the return on investment of data assets and evaluate the contribution of data assets to the business, for example, calculating the cost saved or the increased revenue through data analysis; The data productization module is used to treat data as products, define data usage scenarios and target users, and create data products for sharing and commercialization, for example, providing API interfaces or data subscription services; The data trading module is used to exchange or sell data assets in the data trading market to obtain revenue, which can be achieved through a data trading platform, such as Snowflake Data Marketplace.

[0021] In some examples, the data lifecycle management layer includes a data storage and archiving module, which is used to manage data storage and archiving strategies to ensure the availability and security of data during its life cycle, for example, using a tiered storage strategy to migrate infrequently used data to low-cost storage; It also includes a data elimination and deletion module, which is used to regularly delete data that is no longer needed to meet compliance requirements and release storage resources.

[0022] A method for using a system for integrating IT and OT data management and assetization on a big data platform includes the following steps: S1. Determine the specific systems and data formats of IT and OT data sources, and select appropriate ETL / ELT tools for data extraction; S2. Set up data pipelines to ensure that data flows from source systems into the data lake or data warehouse to achieve real-time data stream processing and batch data processing; S3. Clean the extracted data, remove duplicate data, fill in missing values, and unify the data format; S4. Monitor data quality regularly and use data quality tools to detect and fix data problems; S5. Record detailed information about data assets, including the source, structure, and usage rules of the data, to facilitate data discovery and use; S6. Integrate IT data and OT data according to business needs, label and classify important data sets, define data asset management and usage strategies, and ensure efficient use of data; S7. Conduct ROI analysis on asset data and treat data as products, define data usage scenarios and target users, create data products for sharing and commercialization, and exchange or sell data assets on the data trading market to obtain revenue; S8. Manage data storage and archiving strategies to ensure the availability and security of data throughout its life cycle, regularly delete data that is no longer needed, meet compliance requirements and release storage resources.

[0023] Compared with the prior art, the present invention has at least the following beneficial effects: In the above solution, IT and OT data are fully integrated and processed on the big data platform, and the data is assetized and evaluated. Data integration, governance, analysis and application are comprehensively considered. Through precise data management and intelligent analysis, the effective integration and governance of IT and OT data on the big data platform are ensured, and the availability and business value of data are improved. Enterprises can effectively utilize IT and OT data to improve business decision-making capabilities and operational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated herein and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, further serve to explain the principles of the present disclosure and to enable those skilled in the relevant art to make and use the present disclosure.

[0025] Figure 1 This is a schematic diagram of the system structure for the integrated management and assetization of IT and OT data on a big data platform.

[0026] As shown in the figure, in order to clearly implement the structure of the embodiment of the present invention, specific structures and devices are marked in the figure, but this is only for illustrative purposes and is not intended to limit the present invention to the specific structure, device and environment. According to specific needs, ordinary technicians in this field can adjust or modify these devices and environments, and the adjustments or modifications made are still included in the scope of the attached claims. DETAILED DESCRIPTION

[0027] The following is a detailed description of a system for the integration and assetization of IT and OT data on a big data platform provided by the present invention in conjunction with the accompanying drawings and specific embodiments. At the same time, it is explained here that in order to make the embodiments more detailed, the following implementations are listed as the best and preferred embodiments, and technicians in some known technical fields may also adopt other alternative methods to implement them; and the accompanying drawings are only for a more specific description of the embodiments, and are not intended to specifically limit the present invention.

[0028] It should be noted that the references to "one embodiment", "embodiment", "exemplary embodiment", "some embodiments" and the like in the specification indicate that the described embodiments may include specific features, structures or characteristics, but not every embodiment may include the specific features, structures or characteristics. In addition, when a specific feature, structure or characteristic is described in conjunction with an embodiment, it should be within the knowledge of a person skilled in the art to implement such feature, structure or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0029] In general, a term can be understood, at least in part, from its use in context. For example, depending, at least in part, on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular sense, or can be used to describe a combination of features, structures, or characteristics in the plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, but can instead, depending, at least in part, on the context, allow for the presence of other factors that are not necessarily explicitly described.

[0030] It will be understood that the meaning of “on,” “over,” and “above” in this disclosure should be interpreted in the broadest manner, so that “on” not only means “directly on” something, but also includes the meaning of being “on” something with intervening features or layers therebetween, and “on” or “over” not only means “on” or “above” something, but also includes the meaning of being “on” or “above” something with no intervening features or layers therebetween.

[0031] Additionally, spatially relative terms such as "under," "beneath," "lower," "above," "upper," and the like may be used herein for descriptive convenience to describe the relationship of one element or feature to another element or features, as shown in the accompanying drawings. Spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the accompanying drawings. The device may be oriented in other ways, and the spatially relative descriptors used herein may be similarly interpreted accordingly.

[0032] Example 1 like Figure 1 As shown, an embodiment of the present invention provides a system for the integrated governance and assetization of IT and OT data on a big data platform, including: a data integration layer, a data storage layer, a data governance layer, a data encryption layer, an assetization processing layer and a host lifecycle management layer, wherein the data integration layer is used to extract and integrate IT and OT data, the data storage layer is used to store the extracted IT and OT data, the data governance layer is used to process and manage data, the data encryption layer is used to encrypt and manage permissions for IT and OT data, the assetization processing layer is used to assetize data, and the data lifecycle management layer is used to manage the entire life cycle of data from creation to deletion.

[0033] It should be further explained in this embodiment that the data integration layer includes a data pipeline, which is used to extract, transform and load IT and OT data from different sources into the data storage layer. The extraction and transformation of IT and OT data adopt ETL / ELT tools, such as Apache NiFi, Talend, etc.; The data integration layer also includes an API integration module, which is used to implement real-time data stream processing and integration using an API such as a RESTful API and a message middleware such as Apache Kafka.

[0034] It should be further explained in this embodiment that the data storage layer includes data and data warehouses, and the data lake is used to store raw and multi-structured IT and OT data to support flexible analysis and processing; A data warehouse is used to manage structured data and support complex queries and data analysis. The data lake may adopt Amazon S3 or Azure Data Lake, and the data warehouse may adopt Snowflake or Google BigQuery.

[0035] It should be further explained in this embodiment that the data governance layer includes a data quality management module and a data directory building module. The data quality management module is used to implement data cleaning, verification and standardization to improve data quality; The data catalog building module is used to create a data catalog to record the data source, structure and business meaning, and can be performed using tools such as Apache Atlas or Collibra.

[0036] It should be further explained in this embodiment that the data cleaning is performed using the following method: A. Process missing values ​​by directly deleting them and filling them with the mean; B. Detect outliers; C. Perform data standardization and normalization; D. Perform logarithmic transformation and check data consistency.

[0037] In some examples, mean imputation of missing values ​​is performed using the following formula:

[0038] Where Q is the imputed value, N is the number of non-missing values, and xi is the non-missing observation value; The outlier detection is performed using the standard deviation method and the box plot method. The specific formula of the standard deviation method is as follows: Outliers

[0039] Where is the mean and is the standard deviation, which is usually 2 or 3; The formula of the box plot method is as follows: Outliers

[0040] Where Q1 and Q3 are the first and third quartiles, and IQR = Q3 − Q1 is the interquartile range; The data standardization and normalization are specifically as follows: converting the data into a distribution with a mean of 0 and a standard deviation of 1, and scaling the data to the interval [0, 1], using the following formula;

[0041] Where is the mean and is the standard deviation;

[0042] Where and are the minimum and maximum values ​​of the data respectively; The logarithmic transformation is performed using the following formula

[0043] The addition of 1 is to avoid the logarithm value being zero.

[0044] It should be further explained in this embodiment that the data encryption layer includes a data encryption module, which is used to perform data encryption and permission management to protect data security, and can be performed using tools such as AWS IAM and Apache Ranger.

[0045] It should be further explained in this embodiment that the asset processing plant includes a data classification and labeling module, an ROI analysis module, a data productization module and a data transaction module. The data classification and labeling module is used to classify and label data according to its business value and purpose, so as to facilitate management and use, for example, labeling according to the sensitivity of the data, business department or usage scenario; The ROI analysis module is used to analyze the return on investment of data assets and evaluate the contribution of data assets to the business, for example, calculating the cost saved or the increased revenue through data analysis; The data productization module is used to treat data as products, define data usage scenarios and target users, and create data products for sharing and commercialization, for example, providing API interfaces or data subscription services; The data trading module is used to exchange or sell data assets in the data trading market to obtain revenue, which can be achieved through a data trading platform, such as Snowflake Data Marketplace.

[0046] It should be further explained in this embodiment that the data lifecycle management layer includes a data storage and archiving module, which is used to manage the storage and archiving strategies of data to ensure the availability and security of data during its lifecycle, for example, using a tiered storage strategy to migrate infrequently used data to low-cost storage; It also includes a data elimination and deletion module, which is used to regularly delete data that is no longer needed to meet compliance requirements and release storage resources.

[0047] Example 2 A method for using a system for integrating IT and OT data management and assetization on a big data platform includes the following steps: S1. Determine the specific systems and data formats of IT and OT data sources, and select appropriate ETL / ELT tools for data extraction; S2. Set up data pipelines to ensure that data flows from source systems into the data lake or data warehouse to achieve real-time data stream processing and batch data processing; S3. Clean the extracted data, remove duplicate data, fill in missing values, and unify the data format; S4. Monitor data quality regularly and use data quality tools to detect and fix data problems; S5. Record detailed information about data assets, including the source, structure, and usage rules of the data, to facilitate data discovery and use; S6. Integrate IT data and OT data according to business needs, label and classify important data sets, define data asset management and usage strategies, and ensure efficient use of data; S7. Conduct ROI analysis on asset data and treat data as products, define data usage scenarios and target users, create data products for sharing and commercialization, and exchange or sell data assets on the data trading market to obtain revenue; S8. Manage data storage and archiving strategies to ensure the availability and security of data throughout its life cycle, regularly delete data that is no longer needed, meet compliance requirements and release storage resources. The technical solution provided by the present invention fully integrates and processes IT and OT data on a big data platform, and processes and evaluates the data as an asset, comprehensively considering aspects such as data integration, governance, analysis and application. Through precise data management and intelligent analysis, it ensures the effective integration and governance of IT and OT data on a big data platform, improves the availability and business value of data, and enables enterprises to effectively utilize IT and OT data to improve business decision-making capabilities and operational efficiency.

[0048] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0049] A person skilled in the art will appreciate that all or part of the steps in the above-mentioned embodiment method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc.

[0050] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A system for integrating IT and OT data management and assetization on a big data platform, characterized by: include: Data integration layer, data storage layer, data governance layer, data encryption layer, asset processing layer and host life cycle management layer. The data integration layer is used to extract and integrate IT and OT data, the data storage layer is used to store the extracted IT and OT data, the data governance layer is used to process and manage data, the data encryption layer is used to encrypt and manage permissions for IT and OT data, the asset processing layer is used to assetize data, and the data life cycle management layer is used to manage the entire life cycle of data from creation to deletion.

2. The system for integrating IT and OT data management and assetization on a big data platform according to claim 1 is characterized in that: The data integration layer includes a data pipeline, which is used to extract, transform and load IT and OT data from different sources into the data storage layer. The extraction and transformation of IT and OT data adopt ETL / ELT tools, such as Apache NiFi, Talend, etc.; The data integration layer also includes an API integration module, which is used to implement real-time data stream processing and integration using an API such as a RESTful API and a message middleware such as Apache Kafka.

3. According to claim 1, a system for integrating IT and OT data management and assetization on a big data platform is characterized in that: The data storage layer includes data and data warehouses, and the data lake is used to store raw and multi-structured IT and OT data to support flexible analysis and processing; A data warehouse is used to manage structured data and support complex queries and data analysis. The data lake may adopt Amazon S3 or Azure Data Lake, and the data warehouse may adopt Snowflake or Google BigQuery.

4. According to claim 1, a system for integrating IT and OT data management and assetization on a big data platform is characterized in that: The data governance layer includes a data quality management module and a data catalog building module. The data quality management module is used to implement data cleaning, verification and standardization to improve data quality; The data catalog building module is used to create a data catalog to record the data source, structure and business meaning, and can be performed using tools such as Apache Atlas or Collibra.

5. According to claim 4, a system for integrating management and assetization of IT and OT data on a big data platform is characterized in that: The data cleaning is performed using the following method: A. Process missing values ​​by directly deleting them and filling them with the mean. B. Detect outliers; C. Perform data standardization and normalization; D. Perform logarithmic transformation and check data consistency.

6. The system for integrating IT and OT data management and assetization on a big data platform according to claim 5 is characterized in that: Mean imputation of missing values ​​is performed using the following formula: Where Q is the imputed value, N is the number of non-missing values, and xi is the non-missing observation value; The outlier detection is performed using the standard deviation method and the box plot method. The specific formula of the standard deviation method is as follows: Outliers in is the mean, is the standard deviation, Usually the value is 2 or 3; The formula of the box plot method is as follows: Outliers Where Q1 and Q3 are the first and third quartiles, and IQR = Q3 − Q1 is the interquartile range; The data standardization and normalization are specifically as follows: converting the data into a distribution with a mean of 0 and a standard deviation of 1, and scaling the data to the interval [0, 1], using the following formula; in is the mean, is the standard deviation; in and are the minimum and maximum values ​​of the data respectively; The logarithmic transformation is performed using the following formula The addition of 1 is to avoid the logarithm value being zero.

7. The system for integrating IT and OT data management and assetization on a big data platform according to claim 1 is characterized in that: The data encryption layer includes a data encryption module, which is used to perform data encryption and permission management to protect data security. Tools such as AWS IAM and Apache Ranger can be used for this purpose.

8. The system for integrating IT and OT data management and assetization on a big data platform according to claim 1 is characterized in that: The asset processing plant includes a data classification and labeling module, an ROI analysis module, a data productization module and a data transaction module. The data classification and labeling module is used to classify and label data according to its business value and purpose, so as to facilitate management and use, for example, labeling according to the sensitivity of the data, business department or usage scenario; The ROI analysis module is used to analyze the return on investment of data assets and evaluate the contribution of data assets to the business, for example, calculating the cost saved or the increased revenue through data analysis; The data productization module is used to treat data as products, define data usage scenarios and target users, and create data products for sharing and commercialization, for example, providing API interfaces or data subscription services; The data trading module is used to exchange or sell data assets in the data trading market to obtain revenue, which can be achieved through a data trading platform, such as Snowflake Data Marketplace.

9. The system for integrating IT and OT data management and assetization on a big data platform according to claim 1 is characterized in that: The data lifecycle management layer includes a data storage and archiving module, which is used to manage the storage and archiving strategies of data to ensure the availability and security of data during its lifecycle, for example, using a tiered storage strategy to migrate infrequently used data to low-cost storage; It also includes a data elimination and deletion module, which is used to regularly delete data that is no longer needed to meet compliance requirements and release storage resources.

10. A method for using a system for the integration and assetization of IT and OT data on a big data platform, applied to a system for the integration and assetization of IT and OT data on a big data platform as described in any one of claims 1 to 9, characterized in that: The following steps are involved: S1. Determine the specific systems and data formats of IT and OT data sources, and select appropriate ETL / ELT tools for data extraction; S2. Set up data pipelines to ensure that data flows from source systems into the data lake or data warehouse to achieve real-time data stream processing and batch data processing; S3. Clean the extracted data, remove duplicate data, fill in missing values, and unify the data format; S4. Monitor data quality regularly and use data quality tools to detect and fix data problems; S5. Record detailed information about data assets, including the source, structure, and usage rules of the data, to facilitate data discovery and use; S6. Integrate IT data and OT data according to business needs, label and classify important data sets, define data asset management and usage strategies, and ensure efficient use of data; S7. Conduct ROI analysis on asset data and treat data as products, define data usage scenarios and target users, create data products for sharing and commercialization, and exchange or sell data assets on the data trading market to obtain revenue; S8. Manage data storage and archiving strategies to ensure the availability and security of data throughout its life cycle, regularly delete data that is no longer needed, meet compliance requirements and release storage resources.

Images