The "Dual Data" Middle Platform Data Governance Method for Process Industries

Abstract

A "dual data" middle platform data governance method for process industries, "dual data" includes static data and dynamic data, and the steps of the data governance method include: 1) Define data standards: 1.1) Build data hierarchy; 1.2) Configure data encoding rules; 1.3) Define data entities; 1.4) Determine data labels. 2) Data collection and integration: 2.1) Formulate data collection tasks; 2.2) Data cleaning; 2.3) Data quality verification; 2.4) Data integration. 3) Data security: 3.1) Data classification and grading; 3.2) Formulate security policies; 3.3) Run security tasks. 4) Data application: Provide data services to external systems through API interfaces. This method helps enterprises integrate static and dynamic data resources into the middle platform, provide enterprises with intelligent decision-making, empower data analysis, and help enterprises gradually realize information and digital asset management.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and specifically to a "dual data" middle platform data governance method for process industries. Background Art

[0002] In the existing technology, process industry refers to the production process through physical and chemical changes. Typical process industries include petrochemical, chemical, pharmaceutical and food processing industries. The data center uses data technology to collect, calculate, store and process massive data, unify standards and calibers, form standard data and store them, form a big data asset layer, and provide customers with efficient services.

[0003] The data involved in process industries are mainly divided into static data and dynamic data. Static data mainly includes various documents, drawing data, etc.; dynamic data mainly includes production execution / operation data, IoT / Internet data, inspection and maintenance data, etc.

[0004] Due to the particularity of the process industry, there are problems such as diverse data sources, inconsistent data structures, and system information islands.

[0005] In recent years, with the rapid development of the digital economy and the advancement of Industry 4.0, the demand for asset data governance in the process industry has continued to grow. The implementation of data governance by enterprises is conducive to the effective use of resources, reducing waste, and meeting the requirements of sustainable development, while also enhancing the market competitiveness of enterprises. Implementing data governance in the process industry is a complex project, and the following major problems are encountered:

[0006] 1. Information has data silos. In many process industry companies, data cannot flow between systems, and information silos exist between systems.

[0007] 2. Difficulty in migrating old data. Most factory buildings were designed and constructed a long time ago, and the relevant design drawings are no longer clear. It is necessary to conduct precise data management on these old design drawings;

[0008] 3. Diverse data sources. For process industries, in addition to static data such as documents and drawings, there are also dynamic data such as production execution / operation data, IoT / Internet data, and inspection and maintenance data, which bring great difficulties to data governance;

[0009] 4. It is difficult to unify standards. There may be differences in data definitions and standards between different departments or facilities. How to unify data standards so that enterprises can quickly and conveniently complete data governance?

[0010] 5. Data security issues. During the data governance process, enterprises need to ensure the security of sensitive data to prevent data leakage or illegal access;

[0011] 6. The data structure is chaotic and disordered. Most of the enterprise's data comes from different design software and systems. It is necessary to standardize data governance for this multi-source heterogeneous data to meet the standards for asset storage;

[0012] 7. Data update and maintenance. The update and maintenance of digital assets in the later stage is also a key task, and it is necessary to ensure the upgrade and iteration of online digital assets. Summary of the invention

[0013] In the context of digital asset management, in response to the difficulties in data management in process industries, the present invention proposes a "dual data" middle-end data management method for process industries.

[0014] A data management method for a "dual data" platform in a process industry, wherein the "dual data" includes static data and dynamic data. The steps of the data management method include:

[0015] 1) Define data standards

[0016] 1.1) Build the data hierarchy

[0017] The data layer includes: original (data) layer, cleaning layer, integration layer and application layer

[0018] 1.2)Configure data encoding rules

[0019] Develop specific encoding standards for each data type, including: text data, numeric data, and date and time; and identify data types and fields, including: text strings, numbers, date and time, and Boolean values;

[0020] Select the data type. Different data types have different data encoding rules. Take the date data type as an example: Configure the date data encoding rules. Common date data storage formats in the database include: DATE, DATETIME, etc. This method is defined as: DATE format. Then all subsequent date data will be stored in the format of YYYY-MM-DD.

[0021] 1.3) Define data entities

[0022] According to business requirements, determine the required data entities and their attributes, create entity relationship diagrams, and determine primary and foreign keys;

[0023] Clarify data entities, determine primary keys and foreign keys in entity tables, build associations between entity data, and form blood relationships:

[0024] 3-1 Define data entity names: Name each data entity;

[0025] 3-2 Determine the primary and foreign keys of the entity data table: The primary and foreign keys in each data table are unique; (Take the personnel & department entity data table as an example to explain. In the personnel table, define the personnel name as the primary key, the department name as the foreign key, and associate the primary key department ID in the department table to form an association relationship);

[0026] 3-3 Build primary and foreign key associations to form data blood relationships;

[0027] 1.4) Determine data labels

[0028] According to business needs, formulate a label classification system and create a data label vocabulary. The steps are as follows:

[0029] 4-1 Formulate data classification indicators based on the company's actual business needs;

[0030] 4-2 Assign data indicators to different data and form a data indicator vocabulary;

[0031] 2) Data collection and integration

[0032] 2.1) Formulate data collection tasks

[0033] Establish data collection tasks for static data sources and dynamic data sources respectively, collect data from the original layer, and integrate it into the data warehouse;

[0034] Static data collection: Automatically or manually upload static data in batches; at the same time, connect to industrial design software to extract design data;

[0035] Dynamic data collection: Establish a connection with the database of the external production execution system to collect the required data;

[0036] 2.2) Data cleaning

[0037] Clean the data in the ODS layer of the data warehouse. The methods include:

[0038] Handling missing values: For missing data points in the data set, delete the records with missing values;

[0039] Unified data format: Correct the inconsistent data formats in the data set to make them conform to the unified data encoding rules;

[0040] Deduplication: Identify and remove duplicate records in a data set;

[0041] Data type conversion: For data whose data type does not meet the expected data type, perform data type conversion;

[0042] Data mapping: Match the data fields of other systems with the data fields of the middle platform;

[0043] 2.3) Data quality verification

[0044] After completing data cleaning, regularly evaluate the data to ensure that the data meets the established quality standards;

[0045] 2.4) Data Integration

[0046] After data extraction and conversion, it is stored in the DWD layer of the data warehouse as the data application layer;

[0047] 3) Data security

[0048] 3.1) Data classification and grading

[0049] Classify and grade the stored data, and formulate corresponding data security policies for different levels and types of data;

[0050] The data types are divided into four categories: structured data, unstructured data, semi-structured data and sensitive data, and different security strategies are formulated for the four types of data:

[0051] 1. Structured Data

[0052] Access Control: Implement fine-grained access rights management to ensure that only authorized users can access specific data sets.

[0053] Encryption: Encrypt sensitive information, both during storage and transmission.

[0054] Audit trail: Records all operations on data to facilitate subsequent review and problem tracking.

[0055] 2. Unstructured Data

[0056] Content detection: Use content recognition technology to automatically detect sensitive information in unstructured data, such as personally identifiable information (PII). The method can be a combination of text recognition and keyword matching:

[0057] Through natural language processing (NLP) technology, we conduct in-depth semantic analysis on text content to identify sensitive information such as keywords, phrases, and sentences. We preset a sensitive keyword library to identify sensitive information by matching keywords in the text.

[0058] Data leakage protection: Deploy DLP (Data Loss Prevention) solutions to prevent illegal leakage of sensitive information.

[0059] 3. Semi-structured data

[0060] Schema validation: For semi-structured data such as XML or JSON, the correctness of the data format can be verified through schema definition to avoid malicious input.

[0061] Data cleaning: Removing or converting data elements that may cause security risks, such as script code.

[0062] 4. Sensitive Data

[0063] Anonymization / desensitization: Personal identity information and other sensitive data are anonymized or desensitized without affecting the data’s usefulness.

[0064] Principle of least privilege: Follow the principle of least privilege to ensure that users can access only the data necessary to complete their work.

[0065] 3.2) Develop security strategies

[0066] Formulate data encryption and desensitization algorithms

[0067] Data encryption: Use data encryption algorithms to convert original data in plain text into cipher text data;

[0068] Data desensitization: Using data desensitization algorithms, by changing the sensitive parts of the data, the data no longer contains identifiable personal or organizational information while maintaining its use value;

[0069] 3.3) Run security tasks

[0070] Select the data source and its security policy, edit the security task, and execute the security task; the security policy of the data source is the same as that of the data;

[0071] 4) Data Application

[0072] Provide data services to external systems through API interfaces.

[0073] The "dual data" middle platform that applies this method supports local privatization deployment, refers to the national information security technology level 3 security requirements, and integrates industry information delivery standards such as CFIHOS and ISO15926. It not only ensures the information security of the enterprise, but also helps the enterprise complete data governance in accordance with industry standards, and stores the governed data in the middle platform for data analysis and sharing, breaking the information islands between systems.

[0074] Beneficial effects of the present invention:

[0075] Regarding the data governance of digital assets in process industries, the present invention proposes a "dual data" middle-office data governance method in accordance with the industry standards of process industries, helping enterprises integrate static and dynamic data resources into the middle-office, providing enterprises with intelligent decision-making, empowering data analysis, and helping enterprises gradually realize information-based and digital asset management.

[0076] The benefits of the present invention in implementing digital asset management for process industries are summarized as follows:

[0077] (1) Improve accuracy: Data governance reduces the chances of manually organizing data, thereby reducing the possibility of data errors. The use of automated tools and standardized formats can ensure the consistency and accuracy of data.

[0078] (2) Enhanced collaboration: Digital platforms enable multiple teams (including design, procurement, construction, etc.) to work better together at different stages of the project life cycle.

[0079] (3) Enhanced decision support: Provide data-based insights to support management in making more scientific decisions.

[0080] (4) Improved project execution: Digital tools can more effectively track project progress, monitor costs, and adjust plans in a timely manner to respond to changes. This helps ensure that projects are completed on time and on budget.

[0081] (5) Improve maintenance and support efficiency: During the operation phase, having a complete digital asset retrieval platform can help operators quickly find the required information for daily maintenance or troubleshooting.

[0082] (6) Compliance and security: Digital systems typically have built-in security measures, such as access controls, encryption, and backup capabilities, which help protect sensitive information from unauthorized access or the risk of data breaches. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 It represents the "dual data" middle platform design concept that adopts this data governance method.

[0084] Figure 2 It represents the data middle platform design process using this data governance method.

[0085] Figure 3 Represents the data governance module of the "dual data" middle platform.

[0086] Figure 4 The data governance process of the present invention is shown.

[0087] Figure 5 Represents the "dual data" middle-office system structure that adopts this data governance method. DETAILED DESCRIPTION

[0088] The invention is further described below in conjunction with the accompanying drawings and specific embodiments:

[0089] 1. Data governance process

[0090] like Figure 4As shown, the digital (enterprise digital assets) governance process is as follows:

[0091] (1) Define data standards. This includes: building data hierarchies, stratifying and domain-based data. Clarify the name of each level, configure data encoding rules, and define data entities, related data fields, and data labels;

[0092] (2) Data collection. Establish collection tasks for static data (documents, drawings, etc.) and dynamic data (Internet of Things, production execution data, etc.), collect source layer data, store it in the ODS layer of the data warehouse, and then clean and transform the source layer data according to the defined data standards, perform field relationship mapping and matching, and store it in the DWD layer of the data warehouse as the data source for data applications;

[0093] (3) Data integration. This step needs to be carried out simultaneously with step 2. After completing data collection in step 2, the data is integrated and stored in the data warehouse;

[0094] (4) Data security. To ensure the data security of the enterprise, after the data is stored in the data warehouse, the data is classified and graded, different data security strategies are formulated for different types and levels of data, and encryption and desensitization algorithms are used for the data;

[0095] (5) Data application: Through the API interface, the business data in the middle platform is called as the data basis for enterprise intelligent decision-making.

[0096] 2. The structure of the "dual data" middle platform system using this method is as follows: Figure 5 shown.

[0097] The "dual data" middle platform system structure is mainly divided into three layers, including:

[0098] The basic big data platform includes the relevant data algorithms, databases, etc. used by the system. Because the data volume of the middle platform is massive, Kafka is used to integrate data, and a distributed file system (HDFS) is used to store data. For static and dynamic data, SPARK and Flink are used for calculation and analysis respectively.

[0099] The second layer is the core of the middle platform. After the data is collected and stored through the underlying algorithm, the data is stored in the data warehouse and data mart as the enterprise's data assets. It manages the enterprise's metadata, master data, etc., and performs secondary development on these data. Through API calls, it provides data services for enterprise decision-making, which is the third layer of the system, the data service layer.

[0100] 3. The "dual data" governance process is explained in detail as follows:

[0101] 3.1. Define data standards

[0102] (1) Build a data hierarchy. Divide the data layer into: original layer, cleaning layer, integration layer, service layer, analysis layer, etc. Different data layers can be customized according to the needs of the enterprise;

[0103] (2) Configure data encoding rules. Develop specific encoding standards for each data type, including but not limited to: text data, numeric data, date and time, etc., and determine data types and fields, including text strings, numbers, dates and times, Boolean values, etc.;

[0104] (3) Define data entities. According to business requirements, determine the required data entities and their attributes, create entity relationship diagrams, and determine primary and foreign keys;

[0105] (4) Determine data labels. According to business needs, develop a label classification system and create a data label vocabulary.

[0106] 3.2 Data Collection & Integration

[0107] (1) Formulate data collection tasks. Establish data collection tasks for static data sources (documents, drawings, procurement, construction, etc.) and dynamic data sources (production execution, operations, Internet of Things, etc.);

[0108] Static data collection: supports manual batch uploading of documents, drawings, purchases and other files, and connects to major industrial design software (such as CAD, SPPID, S PI...) to extract design data;

[0109] Dynamic data collection: Establish a connection with the database of the external production execution system (such as: MES, ERP, WMS...) and collect the required data to the original data layer.

[0110] (2) Data cleaning. The data of the ODS layer (source layer) is cleaned, mainly including:

[0111] Handling missing values: For missing data points in the data set, delete the records with missing values;

[0112] Unified data format: Correct the inconsistent data formats in the data set to make them conform to the unified data encoding rules;

[0113] De-duplicate data: Identify and delete duplicate records in the data set to avoid analysis bias caused by duplicate data;

[0114] Data type conversion: Sometimes the data type does not meet expectations, such as a numeric field being incorrectly marked as a string. In this case, data type conversion is required to ensure that the data can be correctly used for subsequent processing;

[0115] Data mapping: Match the data fields of other systems with the data fields of the middle platform to ensure that the data mapping is correct.

[0116] (3) Data quality verification. After completing data cleaning, formulate regular data evaluation tasks to ensure that the data meets the established quality standards;

[0117] (4) Data integration: After data extraction and conversion, it is stored in the data warehouse DWD layer as the data application layer.

[0118] 3.3 Data Security

[0119] (1) Data classification and grading.

[0120] Classify and grade the data stored in the middle platform, and formulate corresponding data security policies for different levels and types of data;

[0121] The data types are divided into four categories: structured data, unstructured data, semi-structured data and sensitive data, and different security strategies are formulated for the four types of data:

[0122] 1. Structured Data

[0123] Access Control: Implement fine-grained access rights management to ensure that only authorized users can access specific data sets.

[0124] Encryption: Encrypt sensitive information, both during storage and transmission.

[0125] Audit trail: Records all operations on data to facilitate subsequent review and problem tracking.

[0126] 2. Unstructured Data

[0127] Content detection: Use content recognition technology to automatically detect sensitive information such as personally identifiable information (PII) in unstructured data.

[0128] Data leakage protection: Deploy DLP (Data Loss Prevention) solutions to prevent illegal leakage of sensitive information.

[0129] 3. Semi-structured data

[0130] Schema validation: For semi-structured data such as XML or JSON, the correctness of the data format can be verified through schema definition to avoid malicious input.

[0131] Data cleaning: Removing or converting data elements that may cause security risks, such as script code.

[0132] 4. Sensitive Data

[0133] Anonymization / desensitization: Personal identity information and other sensitive data are anonymized or desensitized without affecting the data’s usefulness.

[0134] Principle of least privilege: Follow the principle of least privilege to ensure that users can access only the data necessary to complete their work.

[0135] (2) Develop security strategies.

[0136] Formulate data encryption algorithm and data desensitization algorithm:

[0137] Data encryption: Use data encryption algorithms, such as AES, MD5 encryption, etc., to convert the original data (plaintext) into a seemingly meaningless form (ciphertext);

[0138] Data desensitization: Use data desensitization algorithms, such as replacement and truncation, to change the sensitive parts of the data so that the data no longer contains identifiable personal or organizational information while maintaining its use value.

[0139] (3) Run the security task. Select the data source and security policy, edit the security task, and execute the security task. The security policy of the data source is the same as that of the data.

[0140] 3.4. Data application: Through API interfaces, provide data analysis services for external systems, help enterprises make intelligent decisions, provide data intelligent retrieval services, etc.

[0141] The following is an example of applying this data governance method to the "dual data" middle platform.

[0142] refer to Figure 1 , the design concept of the "dual data" middle platform.

[0143] (1) Data standard definition

[0144] In order to meet the requirements of importing multiple data sources, data standards need to be defined in advance, including but not limited to the definition of data standards related to metadata, such as data stratification, domain division, data indicators, and encoding rules.

[0145] (2) Multi-source data integration

[0146] Select the data source to be collected, configure the appropriate database (supports MySQL, Oracle-type relational databases, Hadoop, NoSQL-type non-relational databases), establish a connection with the database, edit the data collection task and execute it.

[0147] (3) Data structure analysis & mapping

[0148] Match the field relationships between the original data fields stored in the database and the data fields in the metadata standard to ensure that the imported field values ​​are successfully stored in the database.

[0149] (4) Develop data security strategies

[0150] In order to ensure that data can be securely called by external systems, the data needs to be desensitized and encrypted, and relevant data security strategies such as encryption algorithms, encrypted data, and desensitization strategies need to be formulated.

[0151] (5)Supports a wide range of interface calls

[0152] After integrating data from multiple systems into the middle platform, rich interfaces are provided for external systems to call data, achieving "large middle platform and small front desk".

[0153] (6) Provide fast data retrieval

[0154] Use MySQL and graph databases to store object data and associated data. The flexibility of graph databases can greatly improve the efficiency of data retrieval.

[0155] refer to Figure 2 ,The design process of the "dual data" middle platform is as follows:

[0156] Define data indicators and select the data sources to be collected. The collected data sources are all related to the process industry, mainly including: various documents, drawing data (static data); production execution / operation data, IoT data, inspection and maintenance data (dynamic data), etc.

[0157] Generate and execute collection tasks. After data extraction is completed, do relationship matching and mapping with fields in data indicators. Then integrate related data and store them in data lakes and data warehouses to complete data governance. Used for data analysis and application.

[0158] refer to Figure 3 , conduct "dual data" mid-table data governance for process industry data assets, mainly including the following modules:

[0159] (1) Data quality management (audit). According to data standards, identify and correct erroneous data, check data regularly as needed, and generate reports to ensure data consistency, uniqueness, integrity, etc.;

[0160] (2) Data standard management. Define data standards, including the meaning, format, length, value range and other attributes of data elements, and continuously optimize them;

[0161] (3) Metadata management. Define the data structure, content, quality, location, usage and other related information of metadata, and create a metadata catalog for users to search and query. At the same time, maintain metadata lineage relationships;

[0162] (4) Master data management. Identify which data entities are master data and define the attributes, relationships, and standards of each master data entity;

[0163] (5) Data security management. Classify data according to its sensitivity and importance, and implement corresponding data security measures for differently labeled data, including data encryption and decryption, desensitization, monitoring, data backup, etc.

[0164] (6) Data lifecycle management: Monitor the entire lifecycle of data, including data creation, storage, use, archiving, and eventual destruction.

Claims

1. A data governance method for a "double data" platform in a process industry, wherein the "double data" includes static data and dynamic data, and wherein the steps of the data governance method include: 1) Define data standards 1.1) Build the data hierarchy The data layer includes: raw data layer, cleaning layer, integration layer and application layer 1.2)Configure data encoding rules Develop specific encoding standards for each data type, including: Text data, numeric data, and date and time; and identify data types and fields, including: text strings, numbers, dates and times, and Boolean values; 1.3) Define data entities According to business requirements, determine the required data entities and their attributes, create entity relationship diagrams, and determine primary and foreign keys; 1.4) Determine data labels Develop a label classification system and create a data label vocabulary based on business needs; 2) Data collection and integration 2.1) Formulate data collection tasks Establish data collection tasks for static data sources and dynamic data sources respectively, collect data from the original layer, and integrate it into the data warehouse; Static data collection: Automatically or manually upload static data in batches; at the same time, connect to industrial design software to extract design data; Dynamic data collection: Establish a connection with the database of the external production execution system to collect the required data; 2.2) Data cleaning Clean the data in the ODS layer of the data warehouse. The methods include: Handling missing values: For missing data points in the data set, delete the records with missing values; Unified data format: Correct the inconsistent data formats in the data set to make them conform to the unified data encoding rules; Deduplication: Identify and remove duplicate records in a data set; Data type conversion: For data whose data type does not meet the expected data type, perform data type conversion; Data mapping: Match the data fields of other systems with the data fields of the middle platform; 2.3) Data quality verification After completing data cleaning, evaluate the data regularly to ensure that the data meets the established quality standards; 2.4) Data Integration After data extraction and conversion, it is stored in the data warehouse DWD layer as the data application layer; 3) Data security 3.1) Data classification and grading The stored data is classified and graded, and corresponding data security strategies are formulated for data of different levels and types. The method is to first divide the data types into four categories: structured data, unstructured data, semi-structured data and sensitive data; then formulate different security strategies for the four types of data: a. Structured data Access control: Implement fine-grained access rights management to ensure that only authorized users can access specific data sets; Encryption: Encrypt sensitive information during storage or transmission; Audit trail: records all operations on data to facilitate subsequent review and problem tracking; b. Unstructured data Content detection: Use content recognition technology to automatically detect sensitive information in unstructured data; Data leakage prevention: Deploy data loss prevention (DLP) solutions to prevent illegal leakage of sensitive information; c. Semi-structured data Schema verification: For semi-structured data, the correctness of the data format is verified through schema definition; Data cleaning: removing or converting data elements that may cause security risks; d. Sensitive data Anonymization / desensitization: Anonymize or desensitize sensitive data without affecting the data's usefulness; Principle of least privilege: Follow the principle of least privilege to ensure that users can only access the data necessary to complete their work; 3.2) Develop security strategies Data encryption: converting original data in plain text into cipher text; Data desensitization: By changing the sensitive parts of the data, the data no longer contains identifiable personal or organizational information while maintaining its use value; 3.3) Run security tasks Select the data source and its security policy, edit the security task, and execute the security task; the security policy of the data source is the same as that of the data; 4) Data Application Provide data services to external systems through API interfaces.

2. The "dual data" middle platform data governance method for process industry according to claim 1 is characterized by: In step 1.3), clarify the data entities, determine the primary key and foreign key in the entity table, build the association relationship between entity data, and form a blood relationship: 3-1 Define data entity names: Name each data entity; 3-2 Determine the primary and foreign keys of the entity data table: The primary and foreign keys in each data table are unique; 3-3 Build primary and foreign key associations to form data blood relationships.

3. The "dual data" middle platform data governance method for process industry according to claim 1 is characterized by the following steps: The steps of 1.4) include: 4-1 Formulate data classification indicators based on the company's actual business needs; 4-2 Assign data indicators to different data and form a data indicator vocabulary.

4. The "dual data" middle platform data governance method for process industry according to claim 1 is characterized by: In step 3.1), the method for implementing content detection on unstructured data is to perform deep semantic analysis on the text content and identify keywords through natural language processing (NLP) technology; then, through a preset keyword library, sensitive information is identified by matching keywords in the text.

Images