Metadata management system and method based on data lake general metadata model

Abstract

The invention relates to a metadata management system and method based on a universal metadata model of a data lake, and belongs to the technical field of metadata models, and the system comprises a metadata uptake module which uptakes metadata in the process that multi-source heterogeneous data enters the lake; the modeling module is used for modeling the acquired metadata through a metadata model and supporting dynamic expansion; the metadatabase is used for receiving and storing the ingested metadata; the metadata management module is used for managing metadata and data consanguinity of the multi-source heterogeneous data in the data lake and carrying out visual display in a graph form; wherein modeling is carried out on the acquired metadata through the metadata model, and dynamic expansion is supported; comprising the steps that metadata is modeled in a graph form, nodes are used for representing data entities and metadata entities, and a data structure is self-defined and comprises structured data, semi-structured data and unstructured data; and the incidence relation between the nodes is represented by edges. The method supports multi-source heterogeneous data cases in a data lake.

Description

Technical Field

[0001] The present invention belongs to the technical field of metadata models, and in particular relates to a metadata management system and method based on a general metadata model of a data lake. Background Art

[0002] After multi-source heterogeneous data is stored in the data lake, data governance is required to ensure data quality, consistency, and security. Metadata management is a very important part of the data governance process. During metadata management, a suitable metadata model needs to be used to organize the metadata structure and establish a data catalog to facilitate user query and analysis of data.

[0003] Traditional metadata models focus on structured assets within the data warehouse, such as data tables, field definitions, and attributes, but lack support for semi-structured and unstructured data. Therefore, they can only describe basic hierarchical relationships or simple associations between structured data, such as foreign key relationships between data tables, thereby ignoring the complex multidimensional relationship network between heterogeneous data. The lack of heterogeneous data relationships not only limits the potential of data engineers and analysts to mine data value through association analysis, but also increases the complexity and error risk in the data integration and conversion process.

[0004] However, the survey of existing metadata models shows that none of them has proposed a complete model to adapt to metadata use cases in the data lake, and has not considered the adaptability of metadata models in the face of ever-changing data requirements and application scenarios. Therefore, it is necessary to focus on the dynamic expansion capability of metadata models, dynamically expand the discovered heterogeneous data relationships into metadata models, improve the adaptability of metadata models to different fields and business scenarios, and enable data in the data lake to be efficiently applied to various analysis scenarios. Summary of the invention

[0005] The present invention proposes a metadata management system based on a data lake universal metadata model that supports dynamic expansion.

[0006] In the present invention, the metadata model is modeled in the form of a graph, and nodes are used to represent data entities and metadata entities. The data structure can be customized without being constrained by a fixed pattern to support the metadata of multi-source heterogeneous data; at the same time, the relationship between nodes is represented by edges, which not only includes the simple relationship of the original data itself, but also supports the complex relationship between the dynamic expansion of heterogeneous data, and can update and synchronize metadata in time. The implementation method uses the traditional data modeling method, and uses the conceptual model, logical model and physical model to demonstrate the actual implementation process of the proposed metadata model. The data lake architecture adopts Apache Hudi. First, a conceptual model of the metadata model is constructed, the concepts of entity, relationship, grouping and process are defined, and a UML class diagram is designed; then, a conversion rule is defined to convert the conceptual model into a logical model based on a graph, that is, entities, relationships, groups and processes are converted into nodes and edges for representation; finally, the metadata ingested in the process of multi-source heterogeneous data entering the lake is uniformly stored in the graph database Neo4j for modeling, and the physical model of the metadata model is realized, including metadata nodes and relationship edges. By exploring the association relationship between heterogeneous data, it is dynamically stored in the graph database in the form of data nodes and relationship edges to realize the dynamic expansion of the metadata model. Finally, a metadata management system is proposed. Using the metadata model for modeling, the metadata of multi-source heterogeneous data can be displayed in the form of a graph, making the query of metadata more intuitive and convenient, and improving the efficiency of metadata management.

[0007] The present invention also provides a metadata management method based on a data lake universal metadata model that supports dynamic expansion.

[0008] The technical solution of the present invention is: The metadata management system based on the data lake common metadata model includes: The metadata ingestion module ingests metadata when multi-source heterogeneous data enters the lake; Modeling module, which models the ingested metadata through the metadata model and supports dynamic expansion; Metadata repository for receiving and storing ingested metadata; The metadata management module manages the metadata and data lineage of multi-source heterogeneous data in the data lake and displays them visually in the form of graphs; Among them, the ingested metadata is modeled through the metadata model, and dynamic expansion is supported; including: metadata is modeled in the form of a graph, using nodes to represent data entities and metadata entities, and custom data structures, including structured data, semi-structured data and unstructured data; edges are used to represent the association relationship between nodes.

[0009] Preferably, according to the present invention, the metadata model defines three types of nodes and three types of edges, wherein the three types of nodes include primary nodes, secondary nodes and tertiary nodes, and the three types of edges include primary edges, secondary edges and tertiary edges; The first-level node is used to represent the metadata entity extracted from the original data source. Different types of metadata entities have different first-level node labels. The first-level edge is used to represent the relationship between the first-level nodes; the second-level node is used to represent the data entity containing the relationship metadata, and the attribute is the value of the data. The second-level node is dynamically created by mining the relationship metadata; The secondary edge is used to represent the relationship between the data entity and the metadata entity, and the secondary edge is dynamically created according to the discovered relationship; The third-level nodes are used to represent user entities and process entities; the third-level edges are used to represent the relationships between user, process and data entities.

[0010] Preferably, according to the present invention, the implementation process of the metadata model is as follows: Design a conceptual model of the metadata model, which describes a series of concepts used to model the data lake metadata; a series of concepts include data entities, groups, relationships, and processes, and are presented in the form of UML class diagrams; Design the logical model of the metadata model, define the transformation rules from the conceptual model to the logical model, and transform the conceptual model into the logical model of the graph; The physical model of the metadata model is implemented through a graph database, including: first, storing the metadata collected when multi-source heterogeneous data enters the lake and the metadata generated after entering the lake in the graph database, then performing data relationship mining for different application scenarios to obtain new relationship metadata, and dynamically storing the relationship metadata in the graph database, thereby realizing dynamic expansion of the metadata model.

[0011] Further preferably, the data entity is a basic unit of a metadata model, representing a data object from various data sources; The grouping is to aggregate data entities with common attributes to form a group; The relationships refer to relationships between data entities and relationships between data entity groups, which are used to represent the correlation between data entities or the hierarchical structure between groups; The process refers to any transformation process of a data entity, which generates a new data entity after the transformation; or, it refers to the user's access process to the data entity and the flow of data in the data lake, thereby building data lineage.

[0012] Preferably, according to the present invention, the logical model of the metadata model is designed in the form of a graph, and the corresponding concepts are represented in the form of a graph using the nodes, edges and attributes of the graph according to the defined conversion rules; In the conversion rules, data entities are represented by nodes, different data entities correspond to different node labels, and the relevant attributes of the data entities are represented by the attributes of the nodes; relationships are represented by edges, and the relevant attributes of the relationships are stored in the attributes of the edges; groups and processes are represented by a combination of nodes and edges.

[0013] Preferably, according to the present invention, the physical model is implemented by the graphic database management system Neo4j, and Hudi is used as the data lake management architecture. Metadata is collected in the process of Hudi ingesting multi-source heterogeneous data, and the collected metadata is converted and cleaned and stored in the graphic database management system Neo4j for persistent storage, and a metadata model is established in the form of a graph; the metadata is queried through the cypher query language of the graphic database management system Neo4j, and the metadata information of the multi-source heterogeneous data and its storage information in the data lake Hudi are intuitively displayed.

[0014] Preferably, according to the present invention, ingesting metadata during the process of multi-source heterogeneous data entering the lake includes: Multi-source heterogeneous data is stored in the data lake through the stream data processing engine, including structured data, semi-structured data and unstructured data. During the process of entering the lake, the configured metadata ingestion method is adopted according to the different data types; among them, the metadata of structured data is ingested by calling the Metadata class of the Jar package; the metadata of semi-structured data is ingested through a custom structure parsing program, and the metadata of unstructured data is ingested through Apache Tika; finally, the ingested metadata is uniformly stored in the metadata database.

[0015] The metadata management method based on the data lake common metadata model includes: Step 1: Ingest metadata when multi-source heterogeneous data enters the lake; Step 2: Store the ingested metadata into the graph database; Step 3: Model the ingested metadata through the metadata model and support dynamic expansion; Step 4: Manage the metadata and data lineage of multi-source heterogeneous data in the data lake, and visualize them in the form of graphs; Among them, the ingested metadata is modeled through the metadata model, and dynamic expansion is supported; including: metadata is modeled in the form of a graph, using nodes to represent data entities and metadata entities, and custom data structures, including structured data, semi-structured data and unstructured data; edges are used to represent the association relationship between nodes.

[0016] The beneficial effects of the present invention are: The present invention provides a metadata management system and method based on a universal metadata model for a data lake, which helps to improve metadata management efficiency. Dynamic scalability is conducive to the integration of data from multiple sources in a data lake, meeting different types of analysis requirements for different application scenarios; it can easily adapt to new data types, attributes, and relationships without large-scale redesign or reconstruction, thereby providing more lasting support for metadata management and application; it is conducive to improving data quality, facilitating data docking and integration between different systems, and realizing rapid data flow and sharing.

[0017] Graph-oriented database management systems, such as Neo4j, often process data faster than relational databases because their data models are simpler and, without the constraints imposed by ACID properties, are better suited for relational modeling than traditional relational databases. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a UML class diagram of the metadata management system and method based on the data lake general metadata model of the present invention; Figure 2 A schematic diagram of physical model definition of the metadata model of the present invention; Figure 3 This is an example diagram of a physical model of the metadata model of the present invention; Figure 4 This is a diagram showing a dynamic expansion example of the implementation process of the metadata model of the present invention; Figure 5 An example diagram of data lineage provided by an embodiment of the present invention; Figure 6 A schematic diagram of a metadata management system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The present invention is further described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should also be noted that, for ease of description, the accompanying drawings only show parts related to the present invention rather than all of the contents.

[0020] Example 1 Terminology explanation: 1. Dynamic scalability: Dynamic scalability refers to the ability of a system, model, or architecture to easily adapt to changing needs, environments, and data at runtime without requiring large-scale redesign, reconstruction, or service interruption. In the field of data models, dynamic scalability means the ability to flexibly add, modify, or delete data elements, attributes, and relationships.

[0021] 2. UML: UML class diagram is an important diagram type in UML (Unified Modeling Language). It is mainly used to describe classes in software systems and the relationships between classes.

[0022] 3. Apache Hudi: Hudi (Hadoop Upserts Deletes and Incrementals) is an open source transactional data lake management framework for managing large-scale data sets on distributed file systems or cloud storage services.

[0023] 4. Neo4j: Neo4j is a high-performance NoSQL graph database that can store data on a network (called a graph from a mathematical perspective) instead of a traditional table. With its unique graph data structure and powerful query capabilities, Neo4j excels in processing complex relational data.

[0024] Metadata management system based on the common metadata model of data lake, such as Figure 6 As shown, including: The metadata of multi-source heterogeneous data is modeled using the metadata model, and the metadata is uniformly stored in the graph database, which can be visualized, including the data itself and the relationship between the data, making metadata queries more intuitive and convenient.

[0025] The metadata ingestion module ingests metadata during the process of multi-source heterogeneous data entering the lake. The embodiment of the present invention adopts Hudi as the data lake architecture and Flink as the data processing engine, and uses parsing tools to ingest metadata of multi-source heterogeneous data during the process of data entering the lake.

[0026] The modeling module models the ingested metadata through the metadata model and supports dynamic expansion; the metadata is modeled in the form of a graph, including nodes and edges. Nodes represent data and its metadata, and edges represent relationship metadata. It supports dynamic creation of nodes and edges to achieve dynamic expansion of the metadata model.

[0027] The metadata database is used to receive and store the ingested metadata; the embodiment of the present invention adopts the graph database Neo4j as the metadata database and uses Cypher as the query language.

[0028] The metadata management module manages the metadata and data lineage of multi-source heterogeneous data in the data lake and displays them visually in the form of graphs. Data lineage parses log records to obtain user operation records on data in the data lake, such as upload, modification, and download operations. The data flow process is visualized through a graph database to build the data lineage of the data lake.

[0029] The log records are stored in JSON format and the following relevant information is obtained by parsing: 1) "time": "2024-02-29T19:39:25.744431903Z", records the time when the event occurred in UTC format.

[0030] 2) "api.name": "CompleteMultipartUpload", records the operation type, which is a multipart upload request.

[0031] 3) "api.bucket": "data", specifies the uploaded bucket as data.

[0032] 4) "api.object": "test-data.csv", specifies the name of the uploaded object as test-data.csv.

[0033] 5) "remotehost": "127.0.0.1", indicating the IP address of the client that initiated the request, from which user information can be obtained.

[0034] 6) "requestID": "17B86CB0ED88EBE9", records the unique identifier of the request.

[0035] 7) "requestPath": " / data / test-data.csv", indicating the requested resource path.

[0036] 8) "requestHost: "minio.example.net:9000" indicates the requested host address and port.

[0037] The ingested metadata is modeled through the metadata model, and dynamic expansion is supported. This includes: metadata is modeled in the form of a graph, with nodes representing data entities and metadata entities, and custom data structures that are not constrained by fixed patterns to support metadata for multi-source heterogeneous data, including structured data, semi-structured data, and unstructured data; edges are used to represent the associations between nodes, including simple associations of the original data itself, as well as complex associations between heterogeneous data, supporting dynamic expansion of relational metadata, and enriching the metadata model to provide comprehensive metadata management.

[0038] The present invention provides a data lake universal metadata model that supports dynamic expansion to support multi-source heterogeneous data use cases in a data lake.

[0039] Example 2 The metadata management system based on the data lake general metadata model described in Example 1 is different in that: The metadata model defines three types of nodes and three types of edges. The three types of nodes include primary nodes, secondary nodes, and tertiary nodes, and the three types of edges include primary edges, secondary edges, and tertiary edges. Figure 2 shown.

[0040] The first-level node is used to represent the metadata entity extracted from the original data source. Different types of metadata entities have different first-level node labels. For example, the first-level node label corresponding to a video file is Video, the first-level node label corresponding to a MySQL table is MySQL, and the first-level node label corresponding to a document file is Document. The attributes of the first-level node are the metadata fields and their values ​​of the metadata entity, which are stored in the form of key-value pairs.

[0041] First-level edges are used to represent the relationship between first-level nodes; for example, the foreign key relationship between data tables. This relationship is obtained when ingesting metadata and is stored in the graph database together with the metadata entity. The attributes of the first-level edges are the original relationship metadata.

[0042] The secondary node is used to represent the data entity containing relational metadata. The attribute is the value of the data. The secondary node is created dynamically by mining the relational metadata. Secondary edges are used to represent the relationship between data entities and metadata entities. Secondary edges are created dynamically based on the discovered relationships. For example, in a university education scenario, machine learning algorithms are used to discover the association between MySQL data tables and unstructured data, including lectures, course selections, records, and attribution relationships, all of which are newly discovered relational metadata.

[0043] The third-level node is used to represent the user entity and the process entity; the third-level node label corresponding to the user entity is User, and its attribute is user information; the third-level node label corresponding to the process node is Process, and its attribute is log record information. The third-level node is dynamically created by parsing the audit log.

[0044] The third-level edge is used to represent the relationship between users, processes, and data entities. For example, the user's upload, download, and access process of the data lake file, as well as the data conversion and flow process, can all be represented by the third-level edge, so as to initially build data lineage.

[0045] In addition, nodes and edges can be combined to represent groups. The label of the node is Group. Based on the data format, it can be divided into mp4, ppt, pptx, pdf, json, and can be expanded. Based on the area, it can be divided into the original data area, the processed data area and the access data area.

[0046] The implementation process of the metadata model is as follows: Design a conceptual model of the metadata model, which describes a series of concepts used to model the data lake metadata; a series of concepts include data entities, groups, relationships, and processes, and are presented in the form of UML class diagrams; Figure 1 shown.

[0047] Among them, data entity, process, metadata, user and group are classes in UML elements. The attributes of data entity class include data entity number, data entity name and description, the attributes of process class include process number, process type, start time, end time and description, the attributes of metadata class include number, size, type, creation time, update time and path, the attributes of user class include user number, user name and authority, and the attributes of group class include group number and group type.

[0048] Data entity association, group association and grouping belong to the interface class, which is used to realize relationships and groups. The relationship between data entity class, metadata class, process class and user class is an association relationship, the relationship between data entity and group is a combination relationship, data entities of the same type form a group, the relationship between group class and grouping is a realization relationship, and the relationship between data entity class and data entity association, group class and group association is a dependency relationship.

[0049] Design the logical model of the metadata model, define the transformation rules from the conceptual model to the logical model, and transform the conceptual model into the logical model of the graph; The physical model of the metadata model is implemented through a graph database, including: first, storing the metadata collected when multi-source heterogeneous data enters the lake and the metadata generated after entering the lake in the graph database, then performing data relationship mining for different application scenarios to obtain new relationship metadata, and dynamically storing the relationship metadata in the graph database, thereby realizing dynamic expansion of the metadata model.

[0050] A data entity is the basic unit of the metadata model and represents a data object from various data sources; for example, a tuple, a database table, a semi-structured document, or unstructured data such as text, images, audio, video, etc.

[0051] Grouping is to gather data entities with common attributes into a group; for example, the original data area, processing data area, and access data area in the data lake architecture are groups divided according to the area; MySQL data tables, images, videos, and audio are groups divided according to data types.

[0052] Relationships refer to the relationships between data entities and the relationships between data entity groups. They are used to represent the correlation between data entities or the hierarchy between groups. For example, database-data table-data field belongs to the hierarchy from the whole to the part, and year-month-day belongs to the time hierarchy.

[0053] A process refers to any transformation process performed on a data entity, which generates a new data entity after the transformation; or, it refers to the process of users accessing data entities and the flow of data in the data lake, thereby building data lineage.

[0054] The logical model of the metadata model is designed in the form of a graph. According to the defined transformation rules, the corresponding concepts are represented in the form of a graph using the nodes, edges and attributes of the graph. In the transformation rules, data entities are represented by nodes, different data entities have different corresponding node labels, and the relevant attributes of the data entities are represented by the attributes of the nodes; relationships are represented by edges, and the relevant attributes of the relationships are stored in the attributes of the edges; groups and processes are represented by a combination of nodes and edges.

[0055] The physical model is implemented by the graph database management system Neo4j, which uses Hudi as the data lake management architecture. Metadata is collected during the process of Hudi ingesting multi-source heterogeneous data. The collected metadata is converted and cleaned before being stored in the graph database management system Neo4j for persistent storage, and a metadata model is established in the form of a graph. The metadata is queried through the cypher query language of the graph database management system Neo4j, which intuitively displays the metadata information of multi-source heterogeneous data and its storage information in the data lake Hudi.

[0056] Ingest metadata when multi-source heterogeneous data enters the lake; including: Multi-source heterogeneous data is stored in the data lake through the stream data processing engine, including structured data, semi-structured data and unstructured data. In the process of entering the lake, the configured metadata ingestion method is adopted according to the different data types; among them, the metadata of structured data is ingested by calling the Metadata class of the Jar package; the metadata of semi-structured data is ingested through a custom structure parsing program, and the metadata of unstructured data is ingested through Apache Tika (file content parsing tool); finally, the ingested metadata is uniformly stored in the metadata database.

[0057] Example 3 The metadata management system based on the data lake general metadata model described in Example 2 is different in that: The most classic general metadata model in the literature proposes eight features, namely semantic enrichment (SE), data polymorphism / multi-zone (DP&MZ), data version control (DV), usage tracking (UT), data classification (DC), similarity link (SL), metadata attribute (MP) and multiple granularity level (MGL). These eight features are used to compare data lake metadata models and evaluate their universality, but the data stored in the data lake is multi-source heterogeneous data, which is complex and has correlations with each other. Therefore, the metadata model of the data lake needs to have the ability to dynamically expand and be able to update and improve the metadata model in a timely manner.

[0058] This embodiment provides a universal metadata model for a data lake that supports dynamic expansion, and uses dynamic scalability (DS) as the ninth feature for evaluating the universality of a data lake metadata model. It can be applied to different analysis application scenarios, dynamically synchronize and expand relational metadata, and thus provide more efficient metadata management for the data lake.

[0059] The metadata model is modeled in the form of a graph, with nodes representing data entities and metadata entities. The data structure can be customized without being constrained by a fixed pattern to support metadata of multi-source heterogeneous data, including structured data, semi-structured data, and unstructured data. Edges are used to represent the association between nodes, including both simple associations of the original data itself and complex associations between heterogeneous data. Dynamically extended relational metadata is supported, and the metadata model is enriched to provide comprehensive metadata management.

[0060] Table 1 below compares exemplary features of the classic metadata model and the metadata model proposed in the present invention.

[0061] Table 1 Nine characteristics for evaluating the generality of data lake metadata models;

[0062] The GEMMS model is a pioneering generic metadata model for data lakes. GEMMS has two abstract entities: data files and data units. Data files represent generic data sources. Data units represent identifiable data elements within a data source. Each data file consists of a set of data units. However, GEMM requires information about the structure of the data and is therefore not suitable for unstructured data.

[0063] The Ground model can be used to model metadata in a data lake. The model tracks data context (metadata) at three levels: metadata attributes, data usage history, and data versions.

[0064] The model of Diamantini et al. is based on the data file and data unit concepts of the GEMMS model and adds similarity links between data units to indirectly link data files.

[0065] Ravat and Zhao's model introduces the concept of regional metadata, which specifies the region where the data is located, making many data lake architectures take the existence of regions into account.

[0066] The MEDAl model represents data through three main concepts, including data objects, representations, and versions. It also supports linking metadata through classification and similarity links, but does not support multiple levels of data granularity.

[0067] The HANDLE model uses a generic concept of data entities to represent data files, which helps in dealing with any level of granularity. In the HANDLE model, each data entity is associated with a label representing a region, granularity level, or classification.

[0068] The goldMEDAL model is based on four main concepts, namely data entities, groups, links, and processes. These concepts interact with each other to support the metadata management needs of the data lake. The concept of grouping supports the organization of the data lake in regions. Grouping allows the management of multiple levels of data granularity, and the possibility of data provenance tracing is realized through the concept of process.

[0069] Example 4 The metadata management method based on the data lake common metadata model includes: S1: Design a conceptual model of the metadata model, which describes a series of concepts used to model the data lake metadata; S2: Design the logical model of the metadata model, define the transformation rules from the conceptual data model to the logical data model, and transform the conceptual model into the logical model of the graph; S3: The physical model of the metadata model is implemented through a graph database. The metadata collected when multi-source heterogeneous data enters the lake is stored in the graph database as nodes, and its inherent relationships are stored in the graph database as edges, thereby establishing a metadata model in the form of a graph.

[0070] S4: Conduct dynamic expansion experiments for university education scenarios, explore the relationship between structured data, semi-structured data, and unstructured data, obtain relationship metadata, and dynamically store them in the graph database as new nodes and edges, thereby realizing dynamic expansion of the metadata model. At the same time, parse log records, initially build data lineage, and visualize them through the graph database.

[0071] In step S1, the design of the conceptual model refers to the recent classical metadata model and improves and supplements it. The defined concepts include data entities, groups, relationships and processes, which are formalized for metadata modeling.

[0072] Data entities are the basic units of metadata models and can represent data objects from various data sources, such as tuples, database tables, semi-structured documents, or unstructured data such as text, images, audio, and video. The collection of data entities can be formalized as: ; Grouping is to divide data entities into different groups based on common attributes, and the data entities in each group have common attributes. For example, the common raw data area, processing data area, and access data area in the data lake architecture are grouped according to the area; structured data, semi-structured data, and unstructured data are grouped according to the structural characteristics of the data. The grouping set can be formalized as: ; Combining the above mentioned examples, the grouping set is The regional grouping can be formalized as ,in Respectively represent the original data area, the processed data area, and the access data area; type grouping can be formalized ,in They correspond to the grouping of MySQL data table, image data, video data, and document data respectively.

[0073] Relationships refer to the relationships between data entities and the relationships between data entity groups. They are used to represent the correlation between data entities or the hierarchical structure between groups. For example, database → data table → data field belongs to the hierarchical structure from the whole to the part, and course selection table → course, surveillance video → course, lecturer → course belongs to the association relationship. The relationship set can be formalized as: ; A process refers to all transformation processes performed on data entities to generate new data entities, or it can refer to the user's access process to data entities. The process set can be formalized as: ; The conceptual model is evaluated by comparing it with the concepts proposed by other metadata models, as shown in Table 2. When a specific concept has no corresponding concept, it is marked with a "-". As can be seen from Table 2, the conceptual model is able to summarize the different concepts proposed by other metadata models.

[0074] Table 2 Conceptual comparison with the classic metadata model;

[0075] In step S2, the embodiment of the present invention proposes a logical model, which is designed in the form of a graph. According to the defined conversion rules, the corresponding concepts are represented in the form of a graph using the nodes, edges and attributes of the graph. Table 3 defines the conversion rules for converting from the conceptual data model to the logical data model.

[0076] Table 3 Transformation rules from conceptual model to logical model;

[0077] In step S3, the physical model of the general metadata model is implemented by the graph database Neo4j. Neo4j is based on the graph data structure in graph theory and consists of nodes and edges. Nodes are used to represent data entities, and edges are used to represent the relationship between data entities. Both nodes and edges can have attributes, which are further descriptions of nodes and edges.

[0078] The general metadata model defines three types of nodes and three types of edges, where nodes include primary nodes, secondary nodes, and tertiary nodes, and edges include primary edges, secondary edges, and tertiary edges. Figure 2 shown.

[0079] The first-level node is used to represent the metadata entity extracted from the original data source. Different types of metadata entities have different first-level node labels. For example, the first-level node label corresponding to a video file is Video, the first-level node label corresponding to a MySQL table is MySQL, and the first-level node label corresponding to a document file is Document. The attributes of the first-level node are the metadata fields and their values ​​of the metadata entity, which are stored in the form of key-value pairs.

[0080] The first-level edge is used to represent the relationship between first-level nodes, such as the foreign key relationship between data tables. This relationship is obtained when the metadata is ingested and stored in the graph database together with the metadata entity. The attributes of the first-level edge are the original relationship metadata.

[0081] The secondary node is used to represent the data entity containing relational metadata. Its attribute is the value of the data. The secondary node is created dynamically by discovering the relational metadata.

[0082] Secondary edges are used to represent the relationship between data entities and metadata entities. Secondary edges are created dynamically based on the discovered relationships. For example, in the context of higher education, machine learning algorithms are used to discover the relationship between MySQL data tables and unstructured data, including teaching, course selection, records, and attribution relationships, all of which are newly discovered relational metadata.

[0083] The third-level node is used to represent the user entity and the process entity. The third-level node label corresponding to the user entity is User, and its attribute is user information; the third-level node label corresponding to the process node is Process, and its attribute is log record information. The third-level node is dynamically created by parsing the audit log.

[0084] The third-level edge is used to represent the relationship between users, processes, and data entities. For example, the user's upload, download, and access process of the data lake file, as well as the data conversion and flow process, can all be represented by the third-level edge, so as to initially build data lineage.

[0085] In addition, nodes and edges can be combined to represent groups. The label of the node is Group. Based on the data format, it can be divided into mp4, ppt, pptx, pdf, json, and can be expanded. Based on the area, it can be divided into the original data area, the processed data area and the access data area.

[0086] In the college education application scenario of step S4, unstructured data includes classroom monitoring videos (mp4 files) and teaching courseware (ppt files). The data entry processing engine is Apache Flink, and after entering the lake, it is stored in the object storage MinIO. Structured data includes course information table, faculty information table, course selection record table, and student information table. Semi-structured files include audit logs, which are stored in the distributed file system HDFS after entering the lake.

[0087] In the process of data entering the lake, metadata is collected, and each metadata object is used as a first-level node, its metadata information is used as the attribute of the first-level node, and the relationship metadata of the data itself is used as the first-level edge. They are stored together in the graph database, and a metadata model is initially established. The example diagram is as follows: Figure 3 shown.

[0088] Discover the relationship between surveillance video files and course information tables, obtain new relationship metadata, and expand the metadata model, mainly from the following three dimensions: (1) The first dimension is the person. The identity of the teacher in the surveillance video is identified through image recognition and detection technology, and then associated with the "Teacher" field in the course information table. This example uses the open source deep face detection and recognition tool InsightFacePaddle, which is based on PaddlePaddle and currently provides three pre-trained models, including BlazeFace for face detection, ArcFace and MobileFace for face recognition, which can quickly implement the entire process of face detection and recognition. A self-made teacher face dataset is created, an index file is constructed, and InsightFacePaddle is used to identify and detect the identity of the teacher in the surveillance video, output the identity label, that is, the teacher's name, and thus establish a person connection.

[0089] (2) The second dimension is time. By parsing the metadata of the shooting time of the surveillance video file, the decision tree model is used to predict the shooting time of this video corresponding to the course time information in the course information table. The specific steps include: First, collect and organize the data. The metadata format of the shooting time of the surveillance video is generally "Wednesday March 1611:14:10 +08:00 2024". Convert it to the timestamp format, that is, "20241016111410", for subsequent calculation and analysis. Obtain the course time information in the course information table, including the number of classes and the class cycle, normalize the course time information, and uniformly convert it into a timestamp range to represent the class time period of the course. At the same time, extract other relevant features of the course, such as course name, lecturer, etc., for building the features of the decision tree model.

[0090] Secondly, organize the sorted data into a data set, divided into a training set and a test set, 70% for training and 30% for testing. This example uses the C4.5 algorithm to train the training set, allowing the model to learn the association pattern between the video shooting time feature and other related features and the course time information. Then use the trained decision tree model to predict the data of the test set, and predict whether it corresponds to the course time information in the course information table based on the feature information such as the video shooting time in the test set.

[0091] Finally, we select accuracy, precision, and recall to measure the performance of the model. By calculating these indicators, we can understand the performance of the model on the test set and determine whether the model can accurately predict the correspondence between the video shooting time and the course time information.

[0092] (3) The third dimension is space. We establish a mapping relationship between surveillance cameras and classrooms in the teaching building. We associate the camera number of the surveillance video with the "venue name" field of the course information table. The surveillance camera number includes the classroom number, so we create a mapping table. By parsing the camera number of the surveillance video, we query the corresponding classroom number in the mapping table and obtain the classroom name, thereby establishing a spatial connection.

[0093] Through the above three dimensions, we can determine the one-to-one relationship between classroom surveillance videos and teachers and courses. The courses and teachers related to the surveillance videos are taken as secondary nodes, and their relationships are dynamically stored in the graph database as secondary edges. New relationship metadata is generated to achieve dynamic expansion of the metadata model. The example diagram is as follows: Figure 4 shown.

[0094] Using the text matching algorithm based on keyword extraction, we can discover the relationship between the course information table and the course courseware, obtain new relationship metadata, and expand the metadata model. The steps of the text matching algorithm based on keyword extraction are as follows: First, keywords are extracted from the curriculum and courseware content.

[0095] For the course schedule, obtain the course name, course code, lecturer, course description and other fields. Then perform preprocessing operations, including removing punctuation, converting to lowercase, and word segmentation, and use the processed words as keywords for the course schedule.

[0096] For courseware, we obtain content from the courseware title, chapter title, and key text in the slides. Then we perform preprocessing operations, i.e., removing punctuation, converting to lowercase, and segmenting. We use the TF-IDF algorithm to extract keywords, calculate the term frequency (TF) and inverse document frequency (IDF) of each word in the courseware, and select the top N words with higher TF-IDF values ​​as the keywords of the courseware.

[0097] Secondly, a bag-of-words model is constructed to form a common vocabulary of keywords extracted from the course information table and courseware, with a vocabulary size of M. For each record in the course information table and each courseware, an M-dimensional vector is generated, and the vector elements correspond to the number of occurrences or TF-IDF values ​​of the keywords in the vocabulary in the corresponding text.

[0098] Then use the text matching algorithm to compare the two sets of keyword vectors. The algorithm used in this example is cosine similarity. It measures the cosine value of the angle between two vectors. The value range of cosine similarity is [-1, 1]. The closer the value is to 1, the more similar the two texts are, which means that the course information is more related to the course.

[0099] Through calculation, we can determine the degree of correlation between the courseware and the course at the keyword level. If the cosine similarity is high, it means that the courseware is likely to be related to the course. The courseware with the highest cosine similarity is associated with the course, and the course and the teacher are used as secondary nodes and their relationship is dynamically stored in the graph database as secondary edges to generate new relationship metadata and realize the dynamic expansion of the metadata model. The example diagram is as follows: Figure 4 shown.

[0100] By parsing log records, we can obtain the user's operation records on the data in the data lake, dynamically store the user and operation records as third-level nodes in the graph database, use the operation information as node attributes, and establish third-level edges. Combined with the flow of multi-source heterogeneous data into the lake, we can build data lineage, as shown in the schematic diagram. Figure 5 shown.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and these embodiments are only illustrative and not restrictive. Based on the examples of the present invention, all other examples obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

Claims

1. A metadata management system based on the general metadata model of the data lake, characterized by: include: The metadata ingestion module ingests metadata when multi-source heterogeneous data enters the lake; Modeling module, which models the ingested metadata through the metadata model and supports dynamic expansion; Metadata repository for receiving and storing ingested metadata; The metadata management module manages the metadata and data lineage of multi-source heterogeneous data in the data lake and displays them visually in the form of graphs; Among them, the ingested metadata is modeled through the metadata model, and dynamic expansion is supported; including: metadata is modeled in the form of a graph, using nodes to represent data entities and metadata entities, and custom data structures, including structured data, semi-structured data and unstructured data; edges are used to represent the association relationship between nodes.

2. The metadata management system based on the data lake general metadata model according to claim 1, characterized in that: The metadata model defines three types of nodes and three types of edges. The three types of nodes include primary nodes, secondary nodes, and tertiary nodes, and the three types of edges include primary edges, secondary edges, and tertiary edges. The first-level node is used to represent the metadata entity extracted from the original data source. Different types of metadata entities have different first-level node labels. The first-level edge is used to represent the relationship between the first-level nodes; the second-level node is used to represent the data entity containing the relationship metadata, and the attribute is the value of the data. The second-level node is dynamically created by mining the relationship metadata; The secondary edge is used to represent the relationship between the data entity and the metadata entity, and the secondary edge is dynamically created according to the discovered relationship; The third-level nodes are used to represent user entities and process entities; the third-level edges are used to represent the relationships between user, process and data entities.

3. The metadata management system based on the data lake general metadata model according to claim 1 is characterized in that: The implementation process of the metadata model is as follows: Design a conceptual model of the metadata model, which describes a series of concepts used to model the data lake metadata; a series of concepts include data entities, groups, relationships, and processes, and are presented in the form of UML class diagrams; Design the logical model of the metadata model, define the transformation rules from the conceptual model to the logical model, and transform the conceptual model into the logical model of the graph; The physical model of the metadata model is implemented through a graph database, including: first, storing the metadata collected when multi-source heterogeneous data enters the lake and the metadata generated after entering the lake in the graph database, then performing data relationship mining for different application scenarios to obtain new relationship metadata, and dynamically storing the relationship metadata in the graph database, thereby realizing dynamic expansion of the metadata model.

4. The metadata management system based on the data lake general metadata model according to claim 3 is characterized in that: The data entity is the basic unit of the metadata model, representing a data object from various data sources; The grouping is to aggregate data entities with common attributes to form a group; The relationships refer to the relationships between data entities and the relationships between data entity groups, which are used to represent the correlation between data entities or the hierarchical structure between groups; The process refers to any transformation process of a data entity, which generates a new data entity after the transformation; or, it refers to the user's access process to the data entity and the flow of data in the data lake, thereby building data lineage.

5. The metadata management system based on the data lake general metadata model according to claim 1, characterized in that: The logical model of the metadata model is designed in the form of a graph. According to the defined transformation rules, the corresponding concepts are represented in the form of a graph using the nodes, edges and attributes of the graph. In the conversion rules, data entities are represented by nodes, different data entities have different corresponding node labels, and the relevant attributes of the data entities are represented by the attributes of the nodes; relationships are represented by edges, and the relevant attributes of the relationships are stored in the attributes of the edges; Groups and processes are represented using a combination of nodes and edges.

6. The metadata management system based on the data lake general metadata model according to claim 1, characterized in that: The physical model is implemented by the graphic database management system Neo4j, and Hudi is used as the data lake management architecture. Metadata is collected in the process of Hudi ingesting multi-source heterogeneous data. The collected metadata is converted and cleaned and stored in the graphic database management system Neo4j for persistent storage, and a metadata model is established in the form of a graph; the metadata is queried through the cypher query language of the graphic database management system Neo4j, and the metadata information of the multi-source heterogeneous data and its storage information in the data lake Hudi are intuitively displayed.

7. The metadata management system based on the data lake general metadata model according to any one of claims 1 to 6, characterized in that: Ingest metadata when multi-source heterogeneous data enters the lake; including: Multi-source heterogeneous data is stored in the data lake through the stream data processing engine, including structured data, semi-structured data and unstructured data. During the process of entering the lake, the configured metadata ingestion method is adopted according to the different data types; among them, the metadata of structured data is ingested by calling the Metadata class of the Jar package; the metadata of semi-structured data is ingested through a custom structure parsing program, and the metadata of unstructured data is ingested through Apache Tika; finally, the ingested metadata is uniformly stored in the metadata database.

8. A metadata management method based on a universal metadata model of a data lake, characterized in that: include: Step 1: Ingest metadata when multi-source heterogeneous data enters the lake; Step 2: Store the ingested metadata into the graph database; Step 3: Model the ingested metadata through the metadata model and support dynamic expansion; Step 4: Manage the metadata and data lineage of multi-source heterogeneous data in the data lake, and visualize them in the form of graphs; Among them, the ingested metadata is modeled through the metadata model, and dynamic expansion is supported; including: metadata is modeled in the form of a graph, using nodes to represent data entities and metadata entities, and custom data structures, including structured data, semi-structured data and unstructured data; edges are used to represent the association relationship between nodes.

Images