A decentralized distributed symbiotic shared data weaving system

Abstract

The application discloses a kind of decentralized distributed symbiotic sharing data weaving system, comprising the following steps: 1) construct basic support platform DataFabric;2) by ApacheKafka, establish an asynchronous data intake process, and data is ingested / received from multiple data sources;3) data extraction, conversion and loading;4) metadata management, including metadata extraction and metadata ID generation;5) ontology-based data governance;6) build industrial data tensor body DIKube.The industrial data tensor body DIKube built by the application can generate specific DIKube according to user demand, and can provide for scenario-oriented industrial data application.The application can realize quick review, optimization, integration and iteration in the case where user demand changes.The metadata index ID created by the application can associate industrial data semantics, can realize information query and sharing of heterogeneous data, can trace and manage the whole life cycle of data, can automatically generate the ID of incremental data, and has self-adapting scalability.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing, and in particular to a decentralized, distributed, symbiotic, and shared data weaving system. Background Technology

[0002] The rapid development of information technology has permeated the industrial sector, leading to an explosive growth of multi-source and heterogeneous industrial data. This has brought about challenges such as difficulty in effectively utilizing internal enterprise data, difficulty in data sharing between upstream and downstream of the industrial chain, and difficulty in obtaining valuable open data on the Internet.

[0003] Existing technologies for managing multi-source heterogeneous data generally fall into two categories: big data technology and data space technology. Big data infrastructure platforms built using big data technology are data sharing platforms, typically serving multiple industries and various types of clients when implementing data management. Data space technology, on the other hand, must be integrated with a specific application; that is, a specific data space must be designed based on the application's needs to manage the multi-source heterogeneous data required by that application. Its main purpose is to provide users with a unified view of heterogeneous data sources and intelligent decision support.

[0004] Existing big data infrastructure platforms typically employ a one-stop deployment approach when managing multi-source heterogeneous data. While this approach eliminates compatibility issues between different software or hardware, saves debugging time, and creates value for customers, it struggles to achieve on-demand customization. Furthermore, when user needs change, upgrades can only be made systematically based on existing products, rather than allowing for rapid optimization and iteration. In contrast, dataspaces, as a solution, have applications in numerous fields such as complex scientific data management, ecological data analysis, environmental observation and prediction, social networks, and intelligent manufacturing. Although dataspaces overcome the poor scalability and versatility issues of existing database management systems, data integration systems, desktop search systems, and search engines in these areas, some remain merely general-purpose data sharing platforms, unable to be directly ported to industry sectors with specific industry mechanisms. Others, while validated in various industrial application scenarios, are still theoretically researched and designed, lacking a true understanding of industrial information systems. For example, issues such as data flow, data security design, access management dilemmas, and potential conflicts and feedback problems during system evolution remain unresolved.

[0005] Therefore, it is necessary to improve such existing technologies to overcome the aforementioned shortcomings. Summary of the Invention

[0006] The purpose of this invention is to provide a decentralized, distributed, symbiotic, and shared data weaving system. Based on an industry data identification system and relying on industrial mechanisms, it constructs an industrial knowledge lexicon and an industrial data tensor, DIKube. DIKube stores only metadata and its corresponding identifiers, which not only enables ubiquitous storage of industrial data, providing technical guarantees for data ownership and data security, but also enables on-demand, scenario-driven, and real-time application tailored to specific industries.

[0007] The above-mentioned technical objective of the present invention is achieved through the following technical solution:

[0008] A decentralized, distributed, symbiotic, and shared data weaving system includes the following steps:

[0009] 1) Construct the basic support platform, Data Fabric;

[0010] 2) Establish an asynchronous data ingestion process using Apache Kafka to ingest / receive data from multiple data sources;

[0011] 3) Data extraction, transformation, and loading;

[0012] 4) Metadata management, which includes metadata extraction and metadata ID generation;

[0013] 5) Ontology-based data governance;

[0014] 6) Construct the industrial data tensor DIKube.

[0015] Furthermore, the construction process of the underlying support platform Data Fabric is as follows:

[0016] 1.1) Deploy a distributed infrastructure platform

[0017] Building upon support for both structured and unstructured data storage, this system optimizes HDFS storage for massive amounts of small files by using minIO to improve storage efficiency and provides unified metadata management for heterogeneous data from multiple sources. It dynamically selects different storage components based on data type and utilizes containerization technology for one-click deployment, establishing a decentralized distributed infrastructure platform.

[0018] 1.2) Install file management and migration components on the distributed infrastructure platform.

[0019] MinIO was chosen as the component for file management and migration; when retrieving a file, it is not necessary to provide the specific location of the file in the file system, but instead a Uniform Resource Locator is obtained by requesting the Object Storage Service;

[0020] 1.3) Deploying Spark on Kubernetes streaming computing on a distributed infrastructure platform

[0021] Spark on Kubernetes streaming computing is a distributed computing framework for Spark big data based on Docker containers. It is designed for Spark clusters deployed on Kubernetes and Spark clusters for big data streaming computing on the Kubernetes platform. It can quickly deploy and horizontally scale Spark clusters and achieve load-based elastic scaling of Spark nodes. For Docker container resource monitoring, it collects container resource usage data on each node and performs corresponding scaling activities on Spark nodes according to real-time load.

[0022] 3. The decentralized distributed symbiotic and shared data weaving system according to claim 1, characterized in that the process of step 2) is as follows:

[0023] 2.1) Install the Kafka component to provide message delivery capabilities through Kafka's publish / subscribe pattern and partitioned messaging mechanism;

[0024] 2.2) Data from various databases, message queues, and file storage can be manually or automatically synchronized into Data Fabric via Apache Kafka for unified data management.

[0025] Furthermore, the data extraction refers to acquiring data from the source system and transmitting it to Data Fabric for processing; Data Fabric's ETL provides various data extraction methods, including:

[0026] File import: Supports importing data in various file formats;

[0027] Database connectivity: Supports multiple database types and connection methods;

[0028] Web API: Supports data retrieval via Web API interface;

[0029] The data transformation refers to the cleaning, processing, and transformation of the extracted data to meet subsequent analysis and application needs. Data Fabric ETL provides various data transformation methods, including:

[0030] Data cleaning: removing duplicate data, filling in missing or erroneous data, and adjusting data format;

[0031] Data preprocessing: performing aggregation, calculation, classification, and filtering operations on the data;

[0032] Data conversion: Transforming raw data, such as date conversion and string format conversion;

[0033] Data loading refers to the process of re-importing transformed data into a target data warehouse or business system; Data Fabric ETL provides various data loading methods, including:

[0034] Data write-back: Writing the transformed data back to the source database or file to ensure the integrity and consistency of the source data;

[0035] Data storage: The transformed data is stored in Data Fabric's internal data lake warehouse for easy subsequent querying and analysis;

[0036] Data export: Exporting the transformed data to other systems.

[0037] Furthermore, the metadata extraction employs both automated and manual methods. The automated method utilizes a self-developed metadata extraction tool to scan and extract metadata information from various data sources. The manual method refers to manually entering the data types, field names, and data formats of different data sources.

[0038] Data Fabric utilizes extracted metadata for data lineage analysis, data quality management, and data security auditing to better understand the data, provide sufficient information support for subsequent data processing and applications, and improve the value and utilization efficiency of the data.

[0039] The metadata ID is generated using an industry data identification coding system, based on the industry data catalog and industry knowledge graph, and encrypted to form a unique metadata ID; the metadata ID not only ensures the unique identification of the metadata, but also carries semantics, lineage information and industry mechanism;

[0040] Data Fabric uses metadata IDs to facilitate data management and querying; because unique data IDs not only better identify, find, and track information in datasets, but also ensure that data from different sources does not conflict.

[0041] Furthermore, the ontology-based data governance method is as follows:

[0042] 1) Constructing an industrial knowledge lexicon

[0043] Constructing an industrial knowledge lexicon requires extracting entities and relationships between entities from metadata;

[0044] Entity extraction: Integrating structured, semi-structured, and unstructured data based on the meaning of industrial data and expert experience; identifying and labeling entities in the data, including people, locations, organizations, and terms;

[0045] Relationship extraction: Relationships between entities include membership, similarity, and association;

[0046] The extracted entities and relationships are combined and integrated to construct an industrial knowledge lexicon. A graph database is used to store and manage the data in the knowledge lexicon. The knowledge graph is then queried and reasoned through a knowledge graph query language or inference engine to support applications in multiple scenarios.

[0047] 2) Constructing a metadata index

[0048] Based on the constructed industrial knowledge graph, an industry data catalog and zyxID are formed as an index to meet the efficiency and scalability requirements of DataFabric in managing and querying data;

[0049] Indexes need to be maintained regularly to ensure the integrity and accuracy of the indexed data. Index maintenance includes data updates, data reconstruction, and fault tolerance operations to ensure the consistency and availability of the index and the data.

[0050] 3) Data source and metadata lineage management

[0051] Data source management

[0052] It is necessary to record the source, format, type, and collection time of the data, and to mark the data collector and the person in charge of the data; to record the lineage of the data source, it is necessary to conduct comprehensive documentation and metadata annotation of the data source, including a comprehensive record of the operation process and processing results of the data source;

[0053] Management of processed data

[0054] It is necessary to record the data processing procedures, algorithms, code, and parameter information, and to associate the processing results with the original data source; to use metadata to describe the source and destination of the data flow, thereby determining the data lineage information;

[0055] Metadata lineage management

[0056] Metadata is data that describes data, including data structure, field definitions, data types, and data quality information; managing the lineage of metadata can help identify and track data derivation, changes, and version variations.

[0057] Furthermore, the construction process of the industrial data tensor volume DIKube is as follows:

[0058] 1) Generation of DIKube

[0059] According to the classification of industry data catalogs, different dimensions of metadata tags are formed to create industrial data tensors DIKube; DIKube is a formalized semantic data space;

[0060] 2) Generation of DIKube based on user needs

[0061] Based on industrial mechanisms and user needs, specific DIKubes that can meet user needs can be pre-generated according to categories, applications, rules, formulas, and scenarios. These pre-generated DIKubes can rely on AI and knowledge graphs to conduct big data governance and analysis to pre-generate optimal solution suggestions that meet the needs of user application scenarios, and have the characteristics of multi-disciplinary selection and comprehensive consideration.

[0062] 3) Generation of DIKube, an open industrial data platform

[0063] The Industrial Open Data DIKube is a DIKube that categorizes open data in the industrial sector according to content, interest, genre, and format, and meets the following characteristics:

[0064] Capable of covering open data across all industries;

[0065] It aligns with the objective existence of open data and is easily accepted by people;

[0066] Machine-readable metadata classification tags enable automation.

[0067] In summary, the present invention has the following beneficial effects:

[0068] 1) The industrial data tensor body DIKube constructed by this invention can generate specific DIKube according to user needs, and can provide scenario-oriented industrial data applications.

[0069] 2) The Data Fabric platform built by this invention utilizes open source technology, enabling rapid review, optimization, integration, and iteration in response to changing user needs.

[0070] 3) The metadata index ID created by this invention can associate with industrial data semantics, enable information query and sharing of heterogeneous data, enable data traceability and full lifecycle management, automatically generate IDs for incremental data, and has adaptive scalability. Attached Figure Description

[0071] Figure 1 This is a schematic diagram of the semantic and structured data governance platform that supports the industrial data weaving system described in this invention.

[0072] Figure 2The industrial quick search developed based on this invention is the first vertical search engine in China applied to the industrial field.

[0073] Figure 3 This is a diagram illustrating the generation process of the scene-oriented DIKube described in this invention. Detailed Implementation

[0074] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below with reference to the figures and specific embodiments.

[0075] like Figure 1 As shown, the present invention proposes a decentralized, distributed, symbiotic, and shared data weaving system, which includes the following steps:

[0076] I. Building the Basic Support Platform Data Fabric

[0077] 1) Deploy a distributed infrastructure platform

[0078] Building upon support for both structured and unstructured data storage, this system optimizes HDFS (Hadoop Distributed File System) for storing massive amounts of small files. It improves storage efficiency by using minIO to store small files and provides unified metadata management for multi-source, heterogeneous data. Different storage components are dynamically selected based on data type. Containerization technology enables one-click deployment of the system, establishing a decentralized, distributed infrastructure platform.

[0079] 2) Install file management and migration components on the distributed infrastructure platform.

[0080] MinIO was chosen as the component for file management and migration. MinIO offers advantages such as reliability (automatic fault tolerance), high availability (service remains available even when half of the nodes fail), and strong scalability (dynamic node expansion). MinIO uses object storage technology, enabling convenient management of massive numbers of small files. When retrieving a file, it's not necessary to provide the file's specific location in the file system; instead, a Uniform Resource Locator (URL) is obtained by requesting the object storage service. MinIO's multi-node nature ensures data security and access speed, its flat structure facilitates rapid data retrieval, and its elastic scaling feature allows for easy expansion of the data governance platform.

[0081] 3) Deploy Spark on Kubernetes streaming computing on a distributed infrastructure platform

[0082] Spark on Kubernetes (KBUNET) streaming computing is a distributed big data computing framework based on Docker containers. Containerized big data platforms can significantly improve resource utilization and computational parallelism, simplify operation and maintenance costs, and handle real-time loads, elastically scaling Spark computing nodes. For Spark clusters deployed on Kubernetes and Spark clusters for big data streaming computing on the Kubernetes platform, it allows for rapid deployment and horizontal scaling of Spark clusters, enabling load-based elastic scaling of Spark nodes. For Docker container resource monitoring, it collects container resource usage data on each node and performs corresponding scaling activities on Spark nodes based on real-time load.

[0083] II. Establish an asynchronous data ingestion process using Apache Kafka to ingest / receive data from multiple data sources.

[0084] 1) Install Kafka components

[0085] Kafka is a widely used distributed messaging system for various large-scale data processing scenarios, characterized by high scalability, fault tolerance, and high throughput. It provides message delivery capabilities by supporting publish / subscribe patterns and partitioned messaging mechanisms. Kafka is widely used in large-scale data processing scenarios such as log collection, real-time stream processing, and event-driven architectures. Its core components are Broker, Topic, Partition, and Consumer Group.

[0086] 2) Through a self-developed connector, data from various databases, message queues, and file storage can be manually or automatically synchronized and then transmitted to Data Fabric via Apache Kafka for unified data management. The following different types of data ingestion are treated the same by Kafka: a) Production line data: including sensor data, equipment interface data, manually entered data, image recognition data, etc.; b) Data from relational databases; c) Data from non-relational databases; d) File data; e) Open data; f) Search engine data.

[0087] III. Data ETL (Extract-Transform-Load)

[0088] 1) Data Extraction

[0089] Data extraction refers to acquiring data from a source system and transferring it to Data Fabric for processing. DataFabric's ETL provides various data extraction methods, including:

[0090] File import: Supports importing data in various file formats, such as CSV, Excel, JSON, etc.

[0091] Database connectivity: Supports various database types and connection methods, such as MySQL, Oracle, and PostgreSQL.

[0092] Web API: Supports data retrieval via Web API interface.

[0093] 2) Data Conversion

[0094] Data transformation refers to the cleaning, processing, and conversion of extracted data to meet subsequent analysis and application needs. Data Fabric ETL provides various data transformation methods, including:

[0095] Data cleaning: removing duplicate data, filling in missing or erroneous data, adjusting data format, etc.

[0096] Data preprocessing: performing operations such as aggregation, calculation, classification, and filtering on the data.

[0097] Data conversion: Transforming raw data, such as date conversion, string format conversion, etc.

[0098] 3) Data loading

[0099] Data loading refers to the process of re-importing transformed data into a target data warehouse or business system. Data Fabric ETL provides various data loading methods, including:

[0100] Data write-back: The transformed data is written back to the source database or file to ensure the integrity and consistency of the source data.

[0101] Data storage: The transformed data is stored in Data Fabric's internal data lake warehouse for easy subsequent querying and analysis.

[0102] Data export: Exporting the transformed data to other systems, such as Hadoop and Spark.

[0103] In addition to using the three data loading methods mentioned above to achieve seamless data transmission and processing, Data Fabric ETL also supports efficient data processing methods such as multi-threading and batch processing, and provides a wealth of data transformation functions and script programming interfaces to support more complex data processing needs.

[0104] IV. Metadata Management

[0105] 1) Metadata extraction

[0106] Metadata extraction employs both automated and manual methods. The automated method utilizes a self-developed metadata extraction tool to scan and extract metadata information from various data sources; the manual method involves manually entering metadata information such as data types, field names, and data formats from different data sources.

[0107] Data Fabric utilizes extracted metadata for data lineage analysis, data quality management, and data security auditing to better understand the data, provide sufficient information support for subsequent data processing and applications, and improve the value and efficiency of data utilization.

[0108] 2) Generation of metadata IDs

[0109] Using an industry data identification and coding system, based on industry data catalogs and industry knowledge graphs, a unique metadata ID is formed through encryption. This ID not only ensures the unique identification of metadata, but also carries semantic, lineage information, and industry mechanisms.

[0110] Data Fabric uses metadata IDs to facilitate data management and querying. Unique data IDs not only better identify, locate, and track information within datasets, but also ensure that data from different sources does not conflict.

[0111] V. Ontology-based Data Governance

[0112] 1) Constructing an industrial knowledge lexicon

[0113] Building an industrial knowledge lexicon requires extracting entities and relationships between entities from metadata.

[0114] Entity extraction: Structured, semi-structured, and unstructured data are integrated based on the meaning of industrial data and expert experience. Entities in the data are identified and labeled, including people, locations, organizations, terms, etc. The entity recognition technologies employed in this invention include Natural Language Processing (NLP) techniques and machine learning algorithms, specifically Named Entity Recognition (NER) models or entity extraction algorithms.

[0115] Relationship extraction: Relationships between entities include membership, similarity, and association. This invention employs relation extraction techniques encompassing NLP (Neural Language Processing) and machine learning algorithms, specifically dependency parsing and machine learning algorithms.

[0116] The extracted entities and relationships are combined and merged to construct an industrial knowledge lexicon. This lexicon is then stored and managed using a graph database, such as Neo4j or ArangoDB. The knowledge graph is then queried and reasoned about using a knowledge graph query language (such as SPARQL) or an inference engine (such as Apache Jena) to support applications across multiple scenarios.

[0117] 2) Constructing a metadata index

[0118] Based on the constructed industrial knowledge graph, an industry data catalog and zyxID are formed as an index to meet the efficiency and scalability requirements of DataFabric in managing and querying data.

[0119] Indexes need to be maintained regularly to ensure the integrity and accuracy of the indexed data. Index maintenance includes operations such as data updates, data reconstruction, and fault tolerance handling to ensure the consistency and availability of the index and the data.

[0120] 3) Data source and metadata lineage management

[0121] Data source management

[0122] It is necessary to record information such as the data source, format, type, and collection time, and to label metadata information such as the data collector and the person in charge of the data. The lineage of the data source must be recorded, requiring comprehensive documentation and metadata annotation of the data source, including a complete record of the data source's operation process and processing results.

[0123] Management of processed data

[0124] It is necessary to record information such as the data processing procedure, algorithm, code, and parameters, and to associate the processing results with the original data source. This association can be real-time or batch-processed, using metadata to describe the source and destination of the data stream, thereby determining the data's lineage information.

[0125] Metadata lineage management

[0126] Metadata is data that describes data, such as data structure, field definitions, data types, and data quality. Managing the lineage of metadata can help identify and track data derivation, changes, and version variations.

[0127] VI. Constructing the Industrial Data Tensor DIKube

[0128] 1) Generation of DIKube

[0129] According to the industry data catalog classification, different dimensions of metadata tags are formed to create an industrial data tensor volume DIKube (Data Information Knowledge k(c)ube). DIKube is a formalized semantic data space.

[0130] 2) Generation of DIKube based on user needs

[0131] Based on industrial mechanisms and user needs, specific DIKubes that meet user requirements can be pre-generated according to categories, applications, rules, formulas, and other scenarios. These pre-generated DIKubes can leverage AI and knowledge graphs through big data governance and analysis to pre-generate optimal solution suggestions that align with the user's application scenario needs, featuring multi-disciplinary selection and comprehensive consideration.

[0132] 3) Generation of DIKube, an open industrial data platform

[0133] The Industrial Open Data DIKube is a DIKube that categorizes open data in the industrial sector according to content, interest, genre, and format, and meets the following characteristics:

[0134] It can cover open data across all industries.

[0135] It aligns with the objective existence of open data and is easily accepted by people.

[0136] Machine-readable metadata classification tags enable automation.

[0137] 1) In this technical solution, the industrial data tensor body DIKube is formed by classifying the industry data catalog and forming metadata tags of different dimensions. Based on industrial mechanisms and user needs, it can pre-generate specific DIKube that can meet user needs according to scenarios such as categories, applications, rules, and formulas, thus solving the problem that existing big data infrastructure platforms are difficult to customize on demand.

[0138] 2) The basic support platform Data Fabric built in this technical solution utilizes open source tools such as HDFS, minIO, Spark, Kubernetes and Apache Kafka. The platform has elastic scaling characteristics, which can facilitate the expansion of the data governance platform and solve the shortcomings of existing big data infrastructure platforms that can only be upgraded in an orderly manner on the basis of existing products and cannot be quickly optimized and iterated.

[0139] 3) The metadata index created in this technical solution ensures the integrity and accuracy of the indexed data, the consistency and availability of the index and the data, and manages the lineage of data sources and metadata. Managing the lineage of metadata can identify and track data derivation, changes, and version variations, solving problems such as data flow issues, data security design issues, access management dilemmas, and potential conflicts and feedback during system evolution when managing multi-source heterogeneous data using existing data space technologies.

[0140] Figure 1 This describes a semantic and structured data governance platform supporting industrial data weaving systems. The platform utilizes open-source technologies including HDFS, minIO, Spark, Kubernetes, and Apache Kafka. These are primarily used to build the foundational computing power for scenario-oriented industrial weaving systems. A self-developed industry data identification system and method within the platform encodes multi-source heterogeneous data within the industrial weaving system and generates a unique zyxID for each piece of metadata, achieving data assimilation. The self-developed industrial lexigraph iLexigraph is used to associate data semantics, achieving ontology-based data enrichment. The self-developed industrial data tensor DIKube is used to realize metadata governance based on data space.

[0141] Figure 2 The industrial quick search developed based on this invention is the first vertical search engine in China applied to the industrial field. It has been used to assist in decision-making for industrial investment, capacity layout, and R&D investment; as well as for potential customer prediction, raw material pre-purchase, capacity planning, and decision-making.

[0142] Figure 3 This paper explains that the industrial data tensor DIKube based on this invention can generate various sub-tensors according to different application scenarios, providing high-quality corpora for large language models and helping small models achieve rapid optimization. Here, ioDIKube represents the industrial open data tensor, and ud1DIKube and ud2DIKube represent tensors generated according to different user needs.

[0143] 1) The industrial data tensor body DIKube constructed by this invention can generate specific DIKube according to user needs, and can provide scenario-oriented industrial data applications.

[0144] 2) The Data Fabric platform built by this invention utilizes open source technology, enabling rapid review, optimization, integration, and iteration in response to changing user needs.

[0145] 3) The metadata index ID created by this invention can associate with industrial data semantics, enable information query and sharing of heterogeneous data, enable data traceability and full lifecycle management, automatically generate IDs for incremental data, and has adaptive scalability.

[0146] In this document, the terms "upper," "lower," "front," "back," "left," "right," "top," "bottom," "inner," "outer," "vertical," and "horizontal," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only used for the clarity of expressing the technical solution and for the convenience of description, and therefore should not be construed as limiting the present invention.

[0147] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, which includes not only the elements listed but also other elements not expressly listed.

[0148] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A decentralized, distributed, symbiotic, and shared data weaving system, characterized in that, Includes the following steps: 1) Construct the basic support platform, Data Fabric; 2) Establish an asynchronous data ingestion process using Apache Kafka to ingest / receive data from multiple data sources; 3) Data extraction, data transformation, and data loading; 4) Metadata management, which includes metadata extraction and metadata ID generation; 5) Ontology-based data governance; 6) Construct the industrial data tensor volume DIKube; The construction process of the industrial data tensor DIKube is as follows: 1) Generation of DIKube According to the classification of industry data catalogs, different dimensions of metadata tags are formed to create industrial data tensors DIKube; DIKube is a formalized semantic data space; 2) Generation of DIKube based on user needs Based on industrial mechanisms and user needs, DIKube can be pre-generated according to categories, applications, rules, formulas, and scenarios to meet user needs. This pre-generated DIKube can rely on AI and knowledge graphs to conduct big data governance and analysis to pre-generate optimal solution suggestions that meet the needs of user application scenarios, and has the characteristics of selecting multiple disciplines and considering all aspects. 3) Generation of DIKube, an open database for industrial applications The Industrial Open Data DIKube is a DIKube that categorizes open data in the industrial sector according to content, interest, genre, and format, and meets the following characteristics: Capable of covering open data across all industries; It aligns with the objective existence of open data and is easily accepted by people; Machine-readable metadata classification tags enable automation; Data Fabric utilizes extracted metadata for data lineage analysis, data quality management, and data security auditing to better understand the data, provide sufficient information support for subsequent data processing and applications, and improve the value and utilization efficiency of the data. The metadata ID is generated using an industry data identification coding system, based on an industry data catalog and an industry knowledge graph, and is encrypted to form a unique metadata ID. Metadata IDs not only guarantee the unique identification of metadata, but also carry semantics, lineage information, and industry mechanisms; Data Fabric uses metadata IDs to facilitate data management and querying; because unique data IDs not only better identify, find, and track information in datasets, but also ensure that data from different sources does not conflict. The ontology-based data governance method is as follows: 1) Constructing an industrial knowledge lexicon Constructing an industrial knowledge lexicon requires extracting entities and relationships between entities from metadata; Entity extraction: Integrating structured, semi-structured, and unstructured data based on the meaning of industrial data and expert experience; identifying and labeling entities in the data, including people, locations, organizations, and terms; Relationship extraction: Relationships between entities include membership, similarity, and association; The extracted entities and relationships are combined and integrated to construct an industrial knowledge lexicon. A graph database is used to store and manage the data in the knowledge lexicon. The knowledge graph is then queried and reasoned through a knowledge graph query language or inference engine to support applications in multiple scenarios. 2) Constructing a metadata index Based on the constructed industrial knowledge graph, an industry data catalog and zyxID are formed as an index to meet the efficiency and scalability requirements of DataFabric in managing and querying data; Indexes need to be maintained regularly to ensure the integrity and accuracy of the indexed data. Index maintenance includes data updates, data reconstruction, and fault tolerance operations to ensure the consistency and availability of the index and the data. 3) Data source and metadata lineage management Data source management The source, format, type, and collection time of the data need to be recorded, and the data collector and the person in charge of the data need to be identified. To record the lineage of data sources, it is necessary to conduct comprehensive documentation and metadata annotation of the data sources, including a comprehensive record of the operation process and processing results of the data sources; Management of processed data It is necessary to record the data processing procedure, algorithm, code, and parameter information, and to associate the processing results with the original data source; Metadata is used to describe the source and destination of data streams, thereby determining the lineage of the data; Metadata lineage management Metadata is data that describes data, including data structure, field definitions, data types, and data quality information; Managing the lineage of metadata can help identify and track data derivation, changes, and version variations.

2. The decentralized, distributed, symbiotic, and shared data weaving system according to claim 1, characterized in that, The construction process of the underlying support platform, Data Fabric, is as follows: 1.1) Deploy a distributed infrastructure platform Building upon support for both structured and unstructured data storage, this system optimizes HDFS storage for massive amounts of small files by using minIO to improve storage efficiency and provides unified metadata management for heterogeneous data from multiple sources. It dynamically selects different storage components based on data type and utilizes containerization technology for one-click deployment, establishing a decentralized distributed infrastructure platform. 1.2) Install file management and migration components on the distributed infrastructure platform. MinIO was chosen as the component for file management and migration; when retrieving a file, it is not necessary to provide the specific location of the file in the file system, but instead a Uniform Resource Locator is obtained by requesting the Object Storage Service; 1.3) Deploying Spark on Kubernetes stream computing on a distributed infrastructure platform Spark on Kubernetes is a distributed big data computing framework based on Docker containers. It enables rapid deployment and horizontal scaling of Spark clusters for big data streaming computing on Kubernetes and Kubernetes platforms, allowing for load-based elastic scaling of Spark nodes. For Docker container resource monitoring, it collects container resource usage data on each node and performs corresponding scaling activities on Spark nodes based on real-time load.

3. The decentralized, distributed, symbiotic, and shared data weaving system according to claim 1, characterized in that, The process of step 2) is as follows: 2.1) Install the Kafka component to provide message delivery capabilities through Kafka's publish / subscribe pattern and partitioned messaging mechanism; 2.2) Data from various databases, message queues, and file storage can be manually or automatically synchronized into Data Fabric via Apache Kafka for unified data management.

4. The decentralized, distributed, symbiotic, and shared data weaving system according to claim 1, characterized in that, Data extraction refers to acquiring data from the source system and transmitting it to Data Fabric for processing; Data Fabric's ETL provides various data extraction methods, including: File import: Supports importing data in various file formats; Database connectivity: Supports multiple database types and connection methods; Web API: Supports data retrieval via Web API interface; The data transformation refers to the cleaning, processing, and transformation of the extracted data to meet subsequent analysis and application needs. Data Fabric ETL provides various data transformation methods, including: Data cleaning: removing duplicate data, filling in missing or erroneous data, and adjusting data format; Data preprocessing: performing aggregation, calculation, classification, and filtering operations on the data; Data conversion: Converting raw data, including date conversion and string format conversion; Data loading refers to the process of re-importing transformed data into a target data warehouse or business system; DataFabric ETL provides various data loading methods, including: Data write-back: Writing the transformed data back to the source database or file to ensure the integrity and consistency of the source data; Data storage: The transformed data is stored in Data Fabric's internal data lake warehouse for easy subsequent querying and analysis; Data export: Exporting the transformed data to other systems.

5. The decentralized, distributed, symbiotic, and shared data weaving system according to claim 1, characterized in that, The metadata extraction employs both automated and manual methods. The automated method utilizes a self-developed metadata extraction tool to scan and extract metadata information from various data sources. Manual method refers to manually entering the data types, field names, and data formats of different data sources.

Images