Distributed metadata engine management system and method oriented to multi-source heterogeneity

Abstract

The invention discloses a distributed metadata engine management system and method oriented to multi-source heterogeneity, and the system specifically comprises an API service gateway layer, a metadata service layer, a metadata security layer, and a zero-trust model for achieving the full-link fine-grained authority control, encryption desensitization and non-tampering auditing, wherein the metadata service layer comprises a metadata collection layer, a metadata processing layer, a metadata storage layer, and a metadata security layer, and the zero-trust model is used for realizing the full-link fine-grained authority control, encryption desensitization and non-tampering auditing. According to the distributed metadata management system, through a gateway unified entrance, push-pull dual-mode structure second-level perception, unified blood relationship intermediate code analysis and zero-trust convergence security, closed-loop refreshing of collection, analysis, processing and storage of metadata at a millisecond level is completed, and the capabilities of high structure consistency, multi-end unified access and traceability are obtained.

Description

Technical Field

[0001] This invention relates to the field of metadata engine technology, and mainly to a distributed metadata engine governance system and method for multi-source heterogeneous systems. Background Technology

[0002] Industries such as government affairs, transportation, finance, and manufacturing are undergoing simultaneous digital upgrades, and the data formats are rapidly expanding from traditional tables to various types such as real-time streaming, semi-structured logs, time-series records, documents, and graph relationships, forming a typical "multi-source heterogeneous" situation.

[0003] Research from Gartner, IDC, and CCID shows that over 70% of government and enterprise platforms consider metadata engines as core infrastructure for data governance. However, mainstream solutions still rely on centralized or single-scenario designs, lacking sufficient horizontal scalability and heterogeneous adaptation capabilities, making it difficult to support governance needs that involve massive scale and millisecond-level structural changes. The latest policy standards, for the first time, propose three key indicators: "cross-domain collaboration, real-time synchronization, and security and controllability," forcing a comprehensive evolution of technology towards distributed, intelligent, and inherently secure approaches.

[0004] Currently, metadata engines face four major bottlenecks: First, insufficient scalability; traditional transactional storage is limited by single-machine performance and struggles to scale horizontally to massive scales. Second, rigid heterogeneous adaptation; various data formats such as streaming, time-series, graphs, and documents must maintain separate metadata models, leading to high costs and errors. Third, lack of real-time performance; offline scheduled scans cannot capture second-level structural changes. Fourth, security and compliance gaps; the lack of field-level anonymization and dynamic encryption methods makes it difficult to meet the requirements of laws such as the Personal Information Protection Law. Although the industry has attempted to alleviate these issues through tiered storage, message queues, or plug-in data collection, these efforts remain at the level of localized optimization and have failed to form a unified solution that achieves horizontal scaling, millisecond synchronization, and inherent security. Summary of the Invention

[0005] To address the aforementioned pain points, this patent proposes four systemic innovations: First, it designs a unified abstract interface to shield the differences in underlying storage, enabling dynamic mapping of the same metadata model across multiple systems; second, it constructs a distributed two-level cache, residing hot data in memory and sinking cold data to the high-speed cache and index, reducing overall query latency by an order of magnitude; third, it integrates real-time push and offline backtracking dual-channel collection, ensuring both millisecond-level structural change awareness and historical data integrity; finally, it introduces a national-level cryptographic field encryption and policy hot update engine to achieve dynamic desensitization of sensitive metadata and zero-trust access control. These four improvements work synergistically, enabling the metadata engine to reach leading domestic and advanced international levels in scalability, compatibility, real-time performance, and security. According to the first aspect of this invention, a distributed metadata governance system for multi-source heterogeneous systems is proposed, with the following specific steps:

[0006] The API service gateway layer is used to uniformly handle external requests and perform attribute-based ABAC dynamic authentication, and complete the three-level routing of tenant-project-field and the merging of high-concurrency requests;

[0007] The metadata service layer includes metadata collection, metadata processing, and metadata storage.

[0008] Metadata collection adopts a push-pull dual-mode architecture that combines real-time push and offline pull, and uniformly converts multi-protocol and multi-dialect structural information into standardized structural events.

[0009] Metadata processing is based on lineage intermediate code, field standardization, lineage chaining, conflict resolution, and versioned snapshots, forming millisecond-level parsing and consistency governance of the structure.

[0010] Metadata storage is based on storage engine abstraction, which distributes snapshots according to their frequency to memory column families, cache clusters, persistent graph storage, relational storage, and column storage;

[0011] The metadata security layer employs a zero-trust model to achieve fine-grained access control, encryption and desensitization, and tamper-proof auditing across the entire chain.

[0012] Among them, the distributed metadata governance system achieves closed-loop refresh of metadata collection-parsing-processing-storage in milliseconds through a unified gateway entry, a push-pull dual-mode structure with second-level perception, unified lineage intermediate code parsing, and zero-trust convergence security, thereby obtaining high structural consistency, unified access from multiple terminals, and traceability capabilities.

[0013] Furthermore, the system also includes a real-time acquisition terminal, which maps multiple protocols to the CloudEvents specification based on a unified event model. These protocols include MySQL binlog, PostgreSQL WAL, MongoDBoplog, Kafka Connect, and Pulsar Functions. The terminal then performs Schemadiff and lineage edge generation in a zero-disk manner within the streaming processing engine, resulting in an end-to-end latency of less than 20ms.

[0014] Furthermore, the system also includes offline data collection, which uses a dialect parser to generate ASTs from relational and semi-structured dialects, performs semantic mapping, and translates them into lineage intermediate codes based on Apache Calcite. Low-frequency or existing data is collected through parallel scanning of [primary key range + timestamp] or [file block + modification time].

[0015] Furthermore, the system also includes: both real-time and offline collected structured events are translated into platform-independent lineage intermediate code (Lineage IR) and pushed down to the message bus in standard JSON format to support lineage resolution and consistency governance shared by multiple modules.

[0016] Furthermore, the metadata processing includes:

[0017] Field standardization model based on standard data metadata library and BERT+CRF;

[0018] Utilizing the lineage chain completion mechanism of multi-hop queries in graph databases;

[0019] An automatic conflict resolution strategy is employed for field types that constitute confidence scores, based on source weights, update times, and manual annotations.

[0020] Furthermore, each time the system undergoes a structural change, it generates a 64-bit globally monotonically increasing Version number and forms a unique key with a millisecond timestamp, which is used to support structural snapshot backtracking at any point in time and consistency verification of multiple storage replicas.

[0021] Furthermore, the metadata storage divides the structural snapshots into: a first-level in-memory column cluster, a second-level Redis cache and persistent graph storage, and a relational database and column storage, based on their popularity. The asynchronous refresh of the graph storage, column storage and full-text index is triggered by the MySQL main write, while simultaneously linking multi-level cache invalidation and backfilling.

[0022] Furthermore, the zero-trust model performs field-level ABAC authentication on the gateway side, and completes encryption or masking using national cryptographic algorithms during the query phase based on the sensitive field markers in the policy center. Access behavior is recorded in chain-like audit logs and supports anomaly export detection.

[0023] Furthermore, the API service gateway layer standardizes and encapsulates backend capabilities and provides a unified REST / GraphQL interface, merging high-concurrency query execution requests and protecting against cache penetration to improve consistency and throughput across multiple access points.

[0024] According to a second aspect of the present invention, a governance method for distributed metadata engines oriented towards multi-source heterogeneity is proposed, specifically including:

[0025] S1. Receive external read and write requests through the API service gateway, perform dynamic permission verification on tenants, projects, tables and fields based on the attribute-based ABAC model, and schedule requests according to the three-level routing of tenant-project-field. At the same time, perform request merging and cache penetration protection for high-concurrency read requests.

[0026] S2. Simultaneously perform real-time push collection and offline pull collection on multi-source heterogeneous data:

[0027] S2.1 Real-time acquisition maps the protocol to CloudEvents events and completes the schema diff and lineage edge generation in the zero-disk memory channel;

[0028] S2.3 Offline data collection generates a syntax tree (AST) through a dialect parser, performs semantic mapping, and translates it into a lineage intermediate code (Lineage IR) based on Apache Calcite. Then, it performs parallel scanning in segments according to the method of [primary key range + timestamp] or [file block + modification time].

[0029] S3. Convert the structural information obtained from real-time and offline collection into platform-independent Lineage IR events, write them to the message bus in JSON format, and ensure the exactly-Once semantics of event delivery.

[0030] S4. Execute on structured events:

[0031] S4.1 Field name standardization: Synonymous fields are normalized using a standard data metadata library and the BERT+CRF model.

[0032] S4.2, Graph pattern matching and lineage complementation for multi-hop queries in graph databases;

[0033] S4.3 Automatically resolve field type conflicts based on confidence scores composed of source weights, update times, and manual annotations;

[0034] S4.4 Generate a structure snapshot containing a globally monotonically increasing Version number and a millisecond timestamp;

[0035] S5. Based on the popularity of the snapshot, write it to the in-memory column family, Redis second-level cache and persistent storage, where the persistent area includes graph storage, relational database and column storage; and use MySQL as the main write source to trigger asynchronous refresh of graph storage, column storage and full text index, while linking the invalidation and backfill of multi-level cache, to ensure strong consistency between replicas through timestamps and checksums.

[0036] S6. Under the zero-trust security architecture, the query process performs field-level ABAC permission verification, encrypts or masks sensitive fields according to national cryptographic algorithms based on the sensitive field markers issued by the policy center, and writes the access behavior into an immutable chained audit log; finally, it returns the metadata view to the caller through a unified REST / GraphQL interface.

[0037] According to a third aspect of the present invention, a computer program product is provided, on which one or more computer programs are stored, which, when executed by a computer processor, implement the method described above.

[0038] The above-described one or more technical solutions in the embodiments of this application have at least one of the following technical effects:

[0039] This method has been implemented in a national-level big data operation platform and has been repeatedly validated and awarded in recent years in ministerial and provincial-level practical modeling projects related to enterprise digitalization, rail transit, smart cities, and anti-fraud. In actual modeling, the distributed metadata engine design and implementation method for multi-source heterogeneity implemented using this patent not only breaks down the fragmented situation of "one model for one type of storage" but also achieves millisecond-level structural change awareness and cross-domain lineage tracing. This improves the efficiency of overall data governance while reducing hardware investment and maintenance manpower. Attached Figure Description

[0040] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0041] Figure 1 A schematic diagram of the framework flow of a distributed metadata engine design and implementation method for multi-source heterogeneity according to an embodiment of the present invention is shown.

[0042] Figure 2 This is a schematic diagram of the structure of a computer system suitable for implementing the electronic devices of the present application embodiments. Detailed Implementation

[0043] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0044] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0045] Figure 1 The diagram illustrates a framework flow of a design and implementation method for a distributed metadata engine oriented towards multi-source heterogeneity according to an embodiment of the present invention, as shown below. Figure 1 As shown:

[0046] Module 1, API Service Gateway;

[0047] It adopts a stateless, multi-active deployment to uniformly handle external read and write requests, and includes built-in protocol conversion, traffic management, circuit breaking, rate limiting, and canary releases. All requests first undergo attribute-based dynamic authentication (ABAC) through the gateway, and then are routed in a three-level manner: tenant-project-field. High-concurrency queries are protected by request merging and cache penetration. The gateway also exposes standardized REST / GraphQL interfaces, shielding backend differences and enabling "one interface, multiple endpoints."

[0048] Module 2, Metadata Services;

[0049] Metadata services mainly include metadata sources, metadata collection, metadata processing, and metadata storage, specifically involving the following:

[0050] 2.1 The metadata source is the sole structural origin of the entire metadata engine, spanning three dimensions: real-time and offline, structured and semi-structured, and internal systems and external ecosystems. It includes both continuously outputting streaming channels and relational repositories storing historical data; it encompasses both well-organized business tables and deeply nested, easily changeable files and interfaces; it includes both centralized governance repositories and edge sensors and third-party open platforms. These declare their location, update frequency, and access protocols through a unified registry center, and are then retrieved losslessly by the collection layer using a push-pull dual-mode approach. This ensures that any newly added fields, changes in lineage, or statistical information are globally perceived within seconds, providing a complete, vibrant, and traceable structural baseline for subsequent parsing, governance, and services.

[0051] 2.2 The metadata acquisition layer, located downstream of the "metadata source," is the "nerve ending" of the engine, responsible for seamlessly and with low latency importing structural information scattered across various sources such as real-time streams, historical databases, files, and interfaces into the system. It adopts a "push-pull dual-mode" architecture, primarily serving the following two scenarios:

[0052] 2.2.1 High-time-efficiency scenarios: The real-time acquisition terminal subscribes to the message bus in an event-driven manner, capturing new fields and table structure changes in milliseconds and immediately generating metadata snapshots to ensure second-level awareness.

[0053] 2.2.1.1 Unified Event Model: Unifies the mapping of protocols such as MySQL binlog, PostgreSQL WAL, MongoDB oplog, Kafka Connect, and Pulsar Functions to the CloudEvents specification, automatically flattens the field hierarchy, and automatically labels primary keys, foreign keys, and partition keys.

[0054] 2.2.1.2 Memory-level transformation engine: Based on Apache Flink CEP, a "zero-disk write" pipeline is built. After a DDL event enters, it first performs version diffing of Avro / Protobuf / JSON Schema through the Schema Registry in memory, and then calls the "dialect translator" to generate standard lineage edges and attributes. The entire process is unserialized and written to disk, with an end-to-end latency of <20ms.

[0055] 2.2.1.3 Hot Data Caching: The latest snapshot is directly written to the distributed memory column cluster (Ignite + self-developed columnar index) and exposed to the outside world via gRPC Stream, which is available to downstream real-time query sides through subscription.

[0056] 2.2.2. Massive existing data or low-frequency change scenarios: The offline collector reverse-engineers bloodline through a dialect parser, supports automatic conversion of various relational and semi-structured dialects, and performs batch scanning in a sharded parallel manner, with a single node processing capacity of up to several billion records per day.

[0057] 2.2.2.1 Dialect Parser Construction:

[0058] 2.2.2.1.1 Syntax Layer: Based on Antlr4's 12 pre-built relational dialects (Oracle, SQL Server, MySQL, PostgreSQL, DM, GBase, KingBase, OceanBase, TiDB, GaussDB, TDSQL, PolarDB) and 6 semi-structured dialects (HiveQL, FlinkSQL, SparkSQL, JSON-Schema, Avro-Schema, Protobuf-IDL), a unified AST is generated.

[0059] 2.2.2.1.2 Semantic Layer: Introducing a data lineage graph template library, AST nodes are mapped to "table-column-function-filter condition" quadruples. Through graph pattern matching, high-risk lineage patterns such as CTAS, nested views, stored procedures, MERGE INTO, and INSERTOVERWRITE are automatically identified with an accuracy of ≥96%.

[0060] This invention relates to a data lineage graph, which is a tracing diagram of data flow and transformation. It shows the process of data from its source to its final destination, including all intermediate operational steps. This graph helps in understanding the source, transformation, and final storage of data, especially in complex databases and data warehouses.

[0061] In the process of quadruple mapping, an AST node (Abstract Syntax Tree node) is mapped to a quadruple. A quadruple contains four key elements:

[0062] Table: The basic unit of data storage, representing a table in a database.

[0063] Column: A specific field in a table that represents the specific content of the data.

[0064] Function: The calculation or transformation operations that may be involved in the query, such as aggregate functions, string manipulation functions, etc.

[0065] Filter Condition: Filter conditions in data operations, such as conditions in the WHERE clause, restrict which rows of data are processed.

[0066] In this way, the system converts database operations (such as query statements) into a structured format that can be processed and analyzed, thereby facilitating subsequent lineage analysis.

[0067] Graph pattern matching identifies specific data manipulation patterns using predefined templates (a lineage graph template library). In this case, the system can identify some high-risk data manipulation patterns, such as: CTAS (Create Table As Select): creating a new table based on query results; nested views: one view nested within another; stored procedures: encapsulated operations within the database; MERGEINTO: merge operations, typically used for updating data; INSERT OVERWRITE: inserting and overwriting existing data.

[0068] Through this quadruple mapping and graph pattern matching technique, the system can automatically identify the aforementioned high-risk data operation patterns and improve the accuracy of identification based on the actual situation, ensuring an accuracy rate of ≥96%.

[0069] 2.2.2.1.3 Transformation Layer: Based on the RexNode and RelNode standards of Apache Calcite, the dialect AST is uniformly translated into a platform-independent "lineage intermediate code" (Lineage IR), and then sent down to Kafka in JSON format for use by both real-time and offline sides.

[0070] 2.2.2.2. Sharded Parallel Scanning: Relational databases are sharded using a two-dimensional approach of [primary key range + incremental timestamp], while HDFS / OSS files are sharded using [file block + modification time]. Each shard starts an independent Spark Task, and the scanning process is automatically downgraded to read-only transactions to avoid table locking. The collection speed is 200MB / s / node, and elastic scaling is supported.

[0071] 2.2.2.3 Unified Data Operation Abstraction Interface: Differences from different sources are transformed into a standard event stream, which is then subjected to triple verification using version number, timestamp, and checksum to ensure consistency in structure, lineage, and statistical information. After collection, the data is pushed to the downstream parsing engine in lossless compression and incremental serialization, achieving a closed-loop collection capability that ensures "no hot data is lost and no cold data is missed."

[0072] 2.3 The metadata processing module is located between the data acquisition exit and the storage entry point, and consists of three levels of sub-modules working together:

[0073] 2.3.1 Notification Module: This module includes built-in change monitoring and a message bus, responsible for real-time broadcasting and exception replay of structured events pushed from the collection layer. It uses a Kafka transactional Producer to ensure exactly-once execution, and the replay strategy supports rollback at three levels: table-level, field-level, and lineage-level.

[0074] 2.3.2 General Processing Module: Integrates metadata model management, impact propagation and operation auditing, and completes field standardization, lineage chaining, conflict resolution, version number + timestamp snapshot generation and chained log recording.

[0075] 2.3.2.1 Field Standardization: Based on GB / T 36326-2018 and ISO / IEC 11179, with over 2000 pre-set standard data elements, BERT+CRF Chinese word segmentation and vectorized matching are used to automatically map over 50 synonymous fields such as "usr_id", "userID", and "user identifier" to the standard element "UserID". The mapping accuracy is 92%, and the amount of manual review is reduced by 80%.

[0076] 2.3.2.2, Lineage Link Repair: For nodes with "broken links" caused by asynchronous parsing in the real-time stream, start the "backtracking link repair" thread, use JanusGraph's Multi-Query to read 2-hop neighbors in batches, fill in the missing edges and write them to the cache, with a repair delay of <1s.

[0077] In real-time streaming data processing, data streams typically undergo a series of asynchronous processing steps (such as parsing and transformation). Because these steps may be executed in parallel or asynchronously, some data points may lack their associated data (i.e., "broken links"), forming what are known as "broken link" nodes. Simply put, a node's data cannot establish the necessary relationships with other nodes or data, resulting in data "fractures" or incompleteness.

[0078] To handle these "broken link" nodes, the system of this invention initiates a "backtracking and link repair" thread. The task is to find and repair these missing connections, ensuring that the complete relationships between the data are restored. This means that when the system detects a "broken link" node, it will asynchronously start a separate thread to resolve the issue, rather than blocking or waiting for other processes.

[0079] Here, "2-hop neighbors" refers to finding the directly connected neighbor nodes (1-hop) starting from the current node, as well as the neighbors of these neighbor nodes (2-hop). This is actually finding the relationship chain between the current node and other nodes.

[0080] When the "backtracking and link completion" thread finds missing associations (i.e., missing edges), the system "fills in" these missing edges. This means that the connections between data points are restored, and the relationships between data points are re-established.

[0081] After the missing edges are filled in, the repaired data will be rewritten to the cache, improving the efficiency of data processing and making subsequent queries and operations more efficient, avoiding repeated queries or processing.

[0082] 2.3.2.3 Conflict Resolution: When the same field appears in different types in multiple sources, the "confidence score" strategy (source weight 40%, update time 30%, manual labeling 30%) is used to make an automatic decision. Conflict records are written to the audit table and pushed to the governance dashboard.

[0083] 2.3.2.4 Version Number + Timestamp Snapshot: Each change generates a globally monotonically increasing 64-bit Version (the high 32 bits are the Epoch Day, and the low 32 bits are the sequence number), which is then combined with the millisecond timestamp to form a unique key "Version@Timestamp", supporting backtracking at any point in time.

[0084] 2.3.3 Parsing Module: It carries synchronous parsing, asynchronous parsing and message queue, and performs low-latency parsing and order guarantee for real-time queries and batch events respectively.

[0085] 2.3.3.1 Real-time query: Uses the "synchronous parsing" thread, returns directly based on memory snapshot, P99 (99th percentile) latency <50ms.

[0086] 2.3.3.2 Batch Events: The "asynchronous parsing" thread is used, and Kafka Partition Key = TableId is used to ensure that events in the same table are strictly ordered. With the help of Flink's watermark mechanism, the out-of-order latency is less than 5 seconds at 100,000 QPS.

[0087] The three types of sub-modules are connected in series through a message bus to form a millisecond-level closed loop of "collection and parsing, parsing and processing, and processing and notification of storage refresh", ensuring that structural assets are written to the storage layer in a highly consistent, traceable, and evolvable manner.

[0088] 2.4. Metadata storage relies on the unified scheduling of "storage engine abstraction" to distribute processed structure snapshots according to their popularity:

[0089] 2.4.1 Latest lineage and field definition: Resident in the first-level cache memory column family, adopting an off-heap+LRU+TTL strategy, with a hit rate of 98% and a read latency of 0.3ms.

[0090] 2.4.2 Warm Data: Lower-level cache (Redis Cluster + SSD hybrid storage), supports 100,000 concurrent users, with a read latency of 2ms.

[0091] 2.4.3 Persistent Region: Parallel Layout;

[0092] 2.4.3.1 Graph Storage Abstraction JanusGraph: Stores relational networks, with vertices and edges indexed by Version@Timestamp, supporting millisecond-level multi-hop queries for 1 billion nodes.

[0093] 2.4.3.2 Relational Storage Abstraction MySQL: Records transaction logs, uses InnoDB partitioned tables + auto-incrementing Version primary key, write TPS 20,000+.

[0094] 2.4.3.3 HBase Column Storage Abstraction: Supports high-frequency statistics, pre-partitioning + Rowkey = Version + TableId, single column write latency <10ms.

[0095] 2.4.3.4 Full-text index abstraction ElasticSearch: Provides keyword retrieval, field-level permission anonymization before writing, and search latency <100ms.

[0096] Each replica maintains strong consistency using timestamps and checksums. When a change event occurs, it first writes to MySQL, then asynchronously refreshes JanusGraph, HBase, and ElasticSearch, and links the two-level cache invalidation and refill to ensure that the service layer reads the latest and traceable metadata view in real time.

[0097] Module 3, Metadata Security;

[0098] Metadata security relies on a zero-trust model to connect the entire chain. The gateway layer performs three-level ABAC authentication for project table fields. Sensitive tags are issued by the policy center in seconds and are encrypted or masked by national cryptographic standards during the query stage. The chained audit logs are tamper-proof and comply with the Level 3 Information Security Protection and the Personal Information Protection Law. Anomaly detection detects high-frequency exports or unauthorized access in real time and triggers gateway rate limiting and circuit breaking, ensuring that every access is verifiable, auditable and blocked.

[0099] Through the unified API service gateway entry in Module 1, the in-depth metadata governance in Module 2, and the end-to-end security protection in Module 3, the system has achieved a complete closed loop of "one interface, multiple terminal calls, instant structure awareness, high storage consistency, and zero-trust security," enabling it to continuously provide fresh, reliable, and compliant metadata services in massive, multi-source, and heterogeneous environments.

[0100] The following is for reference. Figure 2 It shows a schematic diagram of the structure of a computer system 200 suitable for implementing electronic devices according to embodiments of the present application. Figure 2 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0101] like Figure 2 As shown, the computer system 200 includes a central processing unit (CPU) 201, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 202 or programs loaded from storage section 208 into random access memory (RAM) 203. The RAM 203 also stores various programs and data required for the operation of the system 200. The CPU 201, ROM 202, and RAM 203 are interconnected via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0102] The following components are connected to I / O interface 205: an input section 206 including a keyboard, mouse, etc.; an output section 207 including a liquid crystal display (LCD) and speakers, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card and a modem, etc. The communication section 209 performs communication processing via a network such as the Internet. A drive 210 is also connected to I / O interface 205 as needed. A removable medium 211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 210 as needed so that computer programs read from it can be installed into storage section 208 as needed.

[0103] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 209, and / or installed from removable medium 211. When the computer program is executed by central processing unit (CPU) 201, it performs the functions defined in the methods of this application. It should be noted that the computer-readable storage medium of this application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0104] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0105] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0106] The modules described in the embodiments of this application can be implemented in software or in hardware.

[0107] On the other hand, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: 1) have an API service gateway layer, used to uniformly carry external requests and perform attribute-based ABAC dynamic authentication, completing tenant-project-field three-level routing and merging of high-concurrency requests; 2) have a metadata service layer, including metadata collection, metadata processing, and metadata storage: metadata collection adopts a push-pull dual-mode architecture that combines real-time push and offline pull, uniformly converting multi-protocol and multi-dialect structural information into standardized structural events; metadata processing is based on lineage intermediate codes, field standardization, lineage chaining, and... Conflict resolution and versioned snapshots enable millisecond-level parsing and consistency governance of the structure; metadata storage, based on storage engine abstraction, distributes snapshots according to frequency to memory column clusters, cache clusters, persistent graph storage, relational storage, and column storage; the metadata security layer adopts a zero-trust model to achieve fine-grained access control, encryption and desensitization, and tamper-proof auditing across the entire chain; among them, the distributed metadata governance system achieves closed-loop refresh of metadata collection-parsing-processing-storage in milliseconds through a unified gateway entry, a push-pull dual-mode structure with second-level awareness, unified lineage intermediate code parsing, and zero-trust convergence security, resulting in high structural consistency, unified access across multiple terminals, and traceability capabilities.

[0108] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A distributed metadata governance system for multi-source heterogeneous systems, characterized in that, include: The API service gateway layer is used to uniformly handle external requests and perform attribute-based ABAC dynamic authentication, and to complete the three-level routing of tenant-project-field and the merging of high-concurrency requests. The metadata service layer includes metadata collection, metadata processing, and metadata storage. Metadata collection adopts a push-pull dual-mode architecture that combines real-time push and offline pull, and uniformly converts multi-protocol and multi-dialect structural information into standardized structural events. Metadata processing is based on lineage intermediate code, field standardization, lineage chaining, conflict resolution, and versioned snapshots, forming millisecond-level parsing and consistency governance of the structure. Metadata storage is based on storage engine abstraction, which distributes snapshots according to their frequency to memory column families, cache clusters, persistent graph storage, relational storage, and column storage; The metadata security layer employs a zero-trust model to achieve fine-grained access control, encryption and desensitization, and tamper-proof auditing across the entire chain. Among them, the distributed metadata governance system achieves closed-loop refresh of metadata collection-parsing-processing-storage in milliseconds through a unified gateway entry, a push-pull dual-mode structure with second-level perception, unified lineage intermediate code parsing, and zero-trust convergence security, thereby obtaining high structural consistency, unified access from multiple terminals, and traceability capabilities.

2. The distributed metadata governance system according to claim 1, characterized in that, The system also includes a real-time acquisition terminal, which maps multiple protocols to the CloudEvents specification based on a unified event model. These protocols include MySQL binlog, PostgreSQL WAL, MongoDB oplog, Kafka Connect, and PulsarFunctions. The terminal performs schema diffing and lineage edge generation in a zero-disk manner within the streaming processing engine, resulting in an end-to-end latency of less than 20ms.

3. The distributed metadata governance system according to claim 1, characterized in that, The system also includes offline data acquisition, which uses a dialect parser to generate ASTs from relational and semi-structured dialects, performs semantic mapping, and translates them into lineage intermediate codes based on Apache Calcite. Low-frequency or existing data is acquired through parallel scanning of [primary key range + timestamp] or [file block + modification time].

4. The distributed metadata governance system according to claim 1, characterized in that, The system also includes: real-time and offline collected structured events are translated into platform-independent lineage intermediate code Lineage IR, and the message bus is pushed down in standard JSON format to support lineage resolution and consistency governance shared by multiple modules.

5. The distributed metadata governance system according to claim 1, characterized in that, The metadata processing includes: Field standardization model based on standard data metadata library and BERT+CRF; Utilizing the lineage chain completion mechanism of multi-hop queries in graph databases; An automatic conflict resolution strategy is employed for field types that constitute confidence scores, based on source weights, update times, and manual annotations.

6. The distributed metadata governance system according to claim 1, characterized in that, Each time the system undergoes a structural change, it generates a 64-bit globally monotonically increasing Version number and forms a unique key with a millisecond timestamp. This key is used to support structural snapshot backtracking at any point in time and consistency verification of multiple storage replicas.

7. The distributed metadata governance system according to claim 1, characterized in that, The metadata storage divides the structure snapshot into: a first-level in-memory column cluster, a second-level Redis cache and persistent graph storage, a relational database and column storage; and the asynchronous refresh of graph storage, column storage and full-text index is triggered by MySQL main write, while linking multi-level cache invalidation and backfilling.

8. The distributed metadata governance system according to claim 1, characterized in that, The zero-trust model performs field-level ABAC authentication on the gateway side, and completes encryption or masking using national cryptographic algorithms during the query phase based on the sensitive field markers in the policy center. Access behavior is recorded in chain-style audit logs and supports anomaly export detection.

9. The distributed metadata governance system according to claim 1, characterized in that, The API service gateway layer standardizes and encapsulates backend capabilities and provides a unified REST / GraphQL interface. It also merges high-concurrency query execution requests and protects against cache penetration to improve consistency and throughput across multiple devices.

10. A governance method for distributed metadata engines with multi-source heterogeneity, characterized in that, Includes the following steps: S1. Receive external read and write requests through the API service gateway, perform dynamic permission verification on tenants, projects, tables and fields based on the attribute-based ABAC model, and schedule requests according to the three-level routing of tenant-project-field. At the same time, perform request merging and cache penetration protection for high-concurrency read requests. S2. Simultaneously perform real-time push collection and offline pull collection on multi-source heterogeneous data: S2.1 Real-time acquisition maps the protocol to CloudEvents events and completes the generation of Schemadiff and lineage edges in the zero-disk memory channel; S2.3 Offline data collection generates an Abstract Syntax Tree (AST) through a dialect parser, performs semantic mapping, and translates it into a Lineage Intermediate (IR) based on Apache Calcite. Then, it performs parallel scanning in segments according to the method of [primary key range + timestamp] or [file block + modification time]. S3. Convert the structural information obtained from real-time and offline collection into platform-independent Lineage IR events, write them to the message bus in JSON format, and ensure the exactly-Once semantics of event delivery. S4. Execute on structured events: S4.1 Field name standardization: Synonymous fields are normalized using a standard data metadata library and the BERT+CRF model. S4.2, Graph pattern matching and lineage complementation for multi-hop queries in graph databases; S4.3 Automatically resolve field type conflicts based on confidence scores composed of source weights, update times, and manual annotations; S4.4 Generate a structure snapshot containing a globally monotonically increasing Version number and a millisecond timestamp; S5. Based on the popularity of the snapshot, write it to the memory column family, Redis second-level cache and persistent storage, where the persistent area includes graph storage, relational database and column storage; and use MySQL as the main write source to trigger asynchronous refresh of graph storage, column storage and full text index, while linking the invalidation and backfill of multi-level cache, to ensure strong consistency between replicas through timestamps and checksums. S6. Under the zero-trust security architecture, the query process performs field-level ABAC permission verification, encrypts or masks sensitive fields according to national cryptographic algorithms based on the sensitive field markers issued by the policy center, and writes the access behavior into an immutable chained audit log; finally, it returns the metadata view to the caller through a unified REST / GraphQL interface.

Images