System and method for automatically collecting metadata for heterogeneous data sources

Abstract

The invention discloses a system and a method for automatically collecting metadata for a heterogeneous data source, the system and the method for efficiently collecting the metadata for the heterogeneous data source are constructed through a full-automation, unification and intelligentization technical path, and the system and the method have the advantages of comprehensiveness, automation, intelligentization and real-time performance, that is, the system and the method are not limited to a certain type of data sources any more. Comprehensive support for multiple types of data sources is realized through a unified framework; an end-to-end automatic process is constructed, and manual intervention is not needed from source discovery to metadata extraction and storage; natural language processing and machine learning technologies are fused, so that an acquisition result has business semantic comprehension and classification capabilities; through an event-driven dynamic monitoring and incremental updating mechanism, high consistency of metadata and source data is realized.

Description

Technical Field

[0001] This invention relates to a method for collecting metadata, and more particularly to a system and method for automatically collecting metadata from heterogeneous data sources. Background Technology

[0002] In the context of the widespread application of big data and complex information systems, metadata management has become a core aspect of data governance. With the development of big data and data governance, metadata collection and management technologies have gradually formed multiple implementation paths. Currently, the more widely used methods mainly include the following categories: First, manual maintenance, where database administrators or data engineers manually record basic information about the data source, including table structure, field attributes, and file directories. This method was feasible in early environments with small data volumes and simple data source types. Second, relying on tools built into the database system, such as the data dictionary provided by certain relational databases or system views provided by relational databases, to directly extract structured metadata from within the database to support management and querying. Third, utilizing professional ETL tools to automatically generate and collect some metadata information during data extraction, transformation, and loading, thereby reducing manual intervention. Fourth, leveraging open-source metadata management platforms to achieve metadata collection and unified cataloging management of common data sources through plugins or interfaces. In practical applications, these methods provide the foundation for building a metadata system and, to a certain extent, improve the standardization and operability of data governance.

[0003] Although existing methods are widely used in practice, they still have shortcomings in efficiency, compatibility, real-time performance, and intelligence when dealing with complex and diverse heterogeneous data source environments. Specifically:

[0004] Poor compatibility with heterogeneous data sources: In terms of data source coverage, most existing tools are designed for a single or a few types of data sources. They have good support for relational databases, i.e., they are biased towards structured data. However, they lack support for semi-structured data (such as JSON and XML) and unstructured data (such as log files and multimedia files). They also lack a unified framework, making it difficult to form a comprehensive view of data assets.

[0005] Poor adaptability: In terms of scalability, existing tools usually adapt to new data sources through customized plugins, but this development model is time-consuming and costly, which is not conducive to quickly adapting to the needs of business changes.

[0006] Lack of real-time monitoring and synchronization mechanisms: In terms of real-time performance and consistency assurance, existing solutions still rely on periodic batch collection methods, which cannot respond in real time to changes in the structure or content of the data source. The lack of real-time monitoring and incremental update mechanisms for data source changes leads to discrepancies between metadata and the actual data source status, affecting the governance effect and the accuracy of data application.

[0007] Lack of semantic understanding and intelligent governance capabilities: At the level of intelligence, existing technologies mainly focus on extracting metadata (such as tables, fields, and indexes) at the structural level, lacking the ability to understand semantics and automatically identify business context, making it difficult to support the needs of intelligent retrieval, semantic association, and deep data consistency governance.

[0008] Therefore, existing data collection methods still fail to meet the urgent needs of large-scale, heterogeneous, and intelligent data governance, and there is an urgent need to provide a method for automatically collecting metadata from heterogeneous data sources. Summary of the Invention

[0009] To address the shortcomings of the aforementioned technologies, this invention provides a system and method for automatically collecting metadata from heterogeneous data sources.

[0010] To solve the above technical problems, the technical solution adopted by the present invention is: a system for automatically collecting metadata from heterogeneous data sources, comprising a data source layer, a core technology layer and an output layer distributed from bottom to top; The data source layer is used to support the access of multiple types of heterogeneous data sources, including relational databases, NoSQL databases, file systems, streaming data, and API interfaces. The core technology layer consists of four mechanisms that collaboratively enable automatic metadata discovery, unified modeling, real-time synchronization, and intelligent governance. These four mechanisms are: a fully automated metadata collection mechanism, a unified heterogeneous data source processing framework, a dynamic monitoring and real-time update mechanism, and a semantic enhancement and intelligent annotation engine. The output layer includes a unified metadata model, a metadata repository, and intelligent annotation and recommendation services, building an end-to-end automated closed loop.

[0011] Preferably, the fully automated metadata collection mechanism includes: an intelligent source discovery and adaptive access module, a multi-mode fusion collection engine module, a metadata governance process module, and a trusted metadata repository and service publishing module; The intelligent source discovery and adaptive access module automatically identifies and accesses relational databases, file systems, streaming data, and API interfaces through multi-protocol network intelligent detection, cloud resource directory identification, and minimal configuration registration strategy technology. The multi-mode fusion acquisition engine module employs three extraction mechanisms—pattern-driven, rule-driven, and intelligent learning—to operate collaboratively. The metadata governance process module enables the extracted metadata to enter an automated governance pipeline, where it is cleaned, standardized, and integrated in sequence, forming a unified metadata model (UMM). The Trusted Metadata Repository and Service Publishing Module writes the governed metadata into the Trusted Metadata Repository, supports idempotent writing, version control and rollback, and publishes it to the outside world through API and SDK.

[0012] Preferably, in the multi-mode fusion acquisition engine module, mode-driven extraction is used to parse the database dictionary and DDL statements to quickly obtain structural information; Rule-driven extraction identifies and completes primary and foreign key, constraint, and encoding information based on a pre-built rule base. Intelligent learning extraction uses sample data and machine learning models to infer field types, encoding methods, and business semantics.

[0013] Preferably, a unified heterogeneous data source processing framework is used to shield the differences between different data sources, enabling unified access, parsing, and standardized mapping of cross-type data; The unified heterogeneous data source processing framework achieves consistent processing across relational databases, NoSQL databases, file systems, streaming data, and API services through three layers: pluggable connectors, unified access channels, and standardized parsing engines.

[0014] Preferably, pluggable connectors shield underlying protocols, drivers, and version differences through a pluggable connector mechanism; The unified access channel has a unified interface for authentication, access control, and data interaction. Unified authentication unifies identity authentication and access control through a single sign-on method; The data interaction interface is set as a standard interface, and all data source interactions are uniformly abstracted into an interface call of "connect-query-get metadata"; Access control is implemented through security isolation methods, including control traffic, logs, and anomaly isolation.

[0015] The standardized parsing engine is used to map metadata from different sources to a unified metadata model (UMM).

[0016] Preferably, the dynamic monitoring and real-time update mechanism is implemented through a collaborative approach of log probes and Change Capture Center (CDC), event bus and asynchronous transmission, incremental acquisition engine and consistency verification and repository write-back.

[0017] Preferably, the structural changes and configuration changes of the metadata layer are captured in real time by deploying log probes on the data source side or by calling the CDC mechanism. The event bus and asynchronous delivery transform captured change events into a standard message format and push them to the event bus, achieving decoupling and asynchronous delivery. The incremental data collection engine collects and updates only the changed parts through differential calculation. The differential calculation method includes calculating differences based on timestamps, version numbers, and hash values. Before writing the update, the dynamic monitoring and real-time update mechanism performs consistency verification on the incremental patch and writes it to the trusted metadata repository through a transaction mechanism, while generating a new version number to support historical backtracking.

[0018] As a preferred embodiment, the semantic enhancement and intelligent annotation engine combines natural language processing (NLP), machine learning, and knowledge graph technologies to support semantic parsing, intelligent classification, tag generation, and repository write-back of metadata. Semantic parsing techniques in Natural Language Processing (NLP) include word segmentation, part-of-speech tagging, and dependency parsing. These techniques perform semantic parsing on field names, annotations, and sample data to extract potential business meanings. Machine learning classification and clustering uses supervised and unsupervised models to classify and cluster fields, inferring their business categories and data themes; Knowledge graph alignment and semantic annotation align the parsing and classification results with industry knowledge graphs, ontology models and standard terminology databases, generate unified semantic tags, and perform tag inheritance and hierarchical mapping when necessary; The system can recommend and write back to the repository, writing the generated semantic tags and intelligent recommendation results into a trusted metadata repository and displaying them intuitively through the front-end interface.

[0019] A metadata collection method for a system that automatically collects metadata from heterogeneous data sources includes the following steps: S1. Automatic Data Source Discovery and Access: Automatically discovers and accesses multiple heterogeneous data sources through proactive scanning, cloud directory interface calls, and minimal configuration registration; S2, Multi-mode fusion acquisition and parsing: After the access is completed, the acquisition engine is called to perform multi-mode fusion extraction, covering three methods: pattern-driven, rule-driven and intelligent learning, to perform multi-mode fusion acquisition and parsing of multiple heterogeneous data sources; S3. Metadata Cleaning, Standardization and Integration: The raw metadata collected enters the governance pipeline, where it is cleaned, standardized and integrated in sequence to form a consistent unified metadata model (UMM). S4. Dynamic monitoring and real-time updates: Deploy log probes or CDC agents at the source end to capture structure and schema change events in real time and trigger incremental collection and repository updates. S5. Semantic Enhancement and Intelligent Labeling: After the metadata is entered into the database, the semantic engine performs NLP parsing, machine learning classification, and knowledge graph alignment to generate semantic tags and provide recommendations. S6, Metadata Repository Storage and Service-Oriented Deployment: Governed and enhanced metadata is written to a trusted repository, supporting idempotent writing, version management and historical rollback, and providing services to external parties through APIs and SDKs.

[0020] Preferably, the unified processing steps for multiple heterogeneous data sources include: Step 1: Submit a connection request: The data source client first initiates a connection request, submitting its access requirements to the system, which serves as the starting point for the entire data collection process; Step 2: Plug-in Connector Adaptation: Requests are received through plug-in connectors, which are responsible for driver loading and protocol adaptation. Different types of data sources can be accessed through different plug-in components, shielding underlying differences. Step 3: Request Transformation and Authentication: The connector transforms external requests into a unified format that the system can recognize, and performs identity authentication and permission verification; Step 4: Unified access channel processing: The transformed request is sent to the unified access channel, which is responsible for traffic control, error isolation, and unified calling specifications; Step 5: Transmit connection information and metadata: After successful authentication, the unified access channel transmits the connection information and the raw metadata obtained from the data source to the standardized parsing engine; Step 6: Standardized parsing: The metadata is cleaned, its structure is identified, its attributes are mapped, and rules are completed through a standardized parsing engine, and then uniformly converted into an intermediate representation that conforms to the system standard; Step 7: Generate a unified metadata model (UMM): The parsing results are integrated and written into a unified metadata model (UMM), completing the standardized abstraction of attributes from different data sources; Step 8: Write to the metadata repository: The generated UMM is stored in the metadata repository to form a trusted version management system; the system also returns confirmation information and version number to the data source client, completing a complete closed-loop process.

[0021] This invention constructs a system and method for efficient metadata collection from heterogeneous data sources through a fully automated, unified, and intelligent technical approach. It has the advantages of comprehensiveness, automation, intelligence, and real-time performance. That is, it is no longer limited to a certain type of data source, but achieves comprehensive support for multiple types of data sources through a unified framework; it builds an end-to-end automated process, from source discovery to metadata extraction and storage, all without manual intervention; it integrates natural language processing and machine learning technologies, enabling the collection results to have business semantic understanding and classification capabilities; and it achieves a high degree of consistency between metadata and source data through an event-driven dynamic monitoring and incremental update mechanism. Attached Figure Description

[0022] Figure 1This is an architecture diagram of the efficient metadata collection system for heterogeneous data sources according to the present invention.

[0023] Figure 2 This is a schematic diagram of the fully automated metadata collection mechanism of the present invention.

[0024] Figure 3 This is a schematic diagram of the unified heterogeneous data source processing framework of the present invention.

[0025] Figure 4 This is a diagram illustrating the dynamic monitoring and real-time update mechanism of this invention.

[0026] Figure 5 This is a functional module diagram of the semantic enhancement and intelligent annotation engine of the present invention.

[0027] Figure 6 This is a flowchart of the efficient metadata collection method for heterogeneous data sources according to the present invention. Detailed Implementation

[0028] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0029] [Statement of Existing Patent Technologies]

[0030] (1) CN111026796B "Multi-source heterogeneous data acquisition methods, devices, systems, media and equipment"

[0031] The patent proposes a solution that involves creating a keyword table, generating corresponding collection syntax based on the content collected from each data source, establishing data collection rules for each collection syntax, and associating the data collection rules with the corresponding keywords in the keyword table. However, it suffers from a lack of real-time monitoring and automatic update mechanisms, resulting in a discrepancy between metadata and the actual data status.

[0032] (2) CN117708212A "Metadata Acquisition Methods, Devices and Electronic Equipment"

[0033] This patent proposes a parallel processing scheme that obtains the data types, data formats, and the amount of metadata to be collected in the metadata collection task, determines the target metadata collection node based on the data types and data formats, and automatically generates multiple collection instances based on the amount of data. However, it has the problem that it mainly focuses on multi-node parallel processing and does not deeply support semantic parsing and intelligent data content understanding.

[0034] (3) CN118796903B "A metadata management system and method for heterogeneous data sources"

[0035] This patent proposes a solution that combines dynamic adapter selection and version management of the data source by allowing users to select adapters, configure the version and connection parameters of the collected tools or products, and manage and maintain the data source. However, it still has room for improvement in terms of fully automated intelligent data collection and deep semantic understanding, and lacks the ability to respond to changes in the data source in real time.

[0036] (4) CN116431668A "Data lineage analysis method, device and electronic equipment based on metadata collection"

[0037] This patent proposes a metadata-driven lineage analysis scheme that triggers data lineage analysis through metadata collection and constructs a relationship graph between data. However, it suffers from limited lineage tracking accuracy and insufficient dynamic update mechanism in complex data transformation scenarios.

[0038] As can be seen from the existing technologies, current metadata collection methods are insufficient in terms of efficiency, compatibility, real-time performance, and intelligence, making it difficult to meet the management needs of multi-source heterogeneous environments. This invention addresses the pain points of existing technologies by constructing a system and method for automatically collecting metadata from heterogeneous data sources through a fully automated, unified, and intelligent technical approach, solving the following technical problems: Achieve fully automated metadata collection: By using intelligent data source discovery, connection, and metadata extraction technologies, replace manual reliance, improve collection efficiency and accuracy, and reduce maintenance costs.

[0039] Build a unified heterogeneous data source processing framework: Design an extensible connector architecture and a standardized parsing engine to support unified access to multiple data sources such as relational databases, NoSQL, file systems, streaming data, and API services.

[0040] Ensure the real-time nature and consistency of metadata: Introduce an event-driven dynamic monitoring and incremental update mechanism to achieve second-level change capture and synchronous update, ensuring that the metadata repository is consistent with the source.

[0041] Enhance the intelligence of metadata: Integrate semantic analysis, machine learning and knowledge graphs to support automatic identification and classification of business semantics of fields and tables, forming interpretable, inferable and reusable metadata assets.

[0042] First, the overall solution for the system for automatically collecting metadata from heterogeneous data sources disclosed in this invention is divided into three layers: the data source layer, the core technology layer, and the output layer.

[0043] like Figure 1 As can be seen from the technical architecture diagram, it is divided into a data source layer, a core technology layer, and an output layer from bottom to top;

[0044] The data source layer supports access to various heterogeneous data sources, including relational databases, NoSQL databases, file systems, streaming data, and API interfaces, ensuring the diversity of input sources.

[0045] The core technology layer consists of four major mechanisms: a fully automated metadata collection mechanism, a unified heterogeneous data source processing framework, a dynamic monitoring and real-time update mechanism, and a semantic enhancement and intelligent annotation engine. These mechanisms work together to achieve automatic discovery, unified modeling, real-time synchronization, and intelligent governance of metadata.

[0046] The output layer includes a unified metadata model, a metadata repository, and intelligent annotation and recommendation services, which can provide consistent, reliable, and intelligent metadata services to the outside world, thereby building an end-to-end automated closed loop.

[0047] Furthermore, the core technology layer of the system for automatically collecting metadata from heterogeneous data sources is described in more detail.

[0048] The core technology layer replaces the traditional reliance on manual configuration and single tools with a fully automated metadata collection mechanism, constructing a closed-loop process encompassing automatic source discovery and adaptive access, a multimodal collection engine, a metadata governance pipeline, and trusted repository storage and publishing. This mechanism enables rapid, comprehensive, and accurate metadata collection in heterogeneous environments, ensuring the consistency, traceability, and service-oriented output of the collection results.

[0049] like Figure 2 As shown, the fully automated metadata collection mechanism includes an intelligent source discovery and adaptive access module, a multi-mode fusion collection engine module, a metadata governance process module, and a trusted metadata repository and service publishing module.

[0050] (1.1) Intelligent source discovery and adaptive access module

[0051] This module utilizes technologies such as multi-protocol network intelligent detection, cloud resource directory identification, and minimal configuration registration strategy to automatically identify and connect to relational databases, file systems, streaming data, and API interfaces. The system has adaptive capabilities and can select the optimal access strategy and connector based on the data source type and characteristics, significantly reducing manual intervention and configuration overhead.

[0052] Multi-protocol network intelligent detection supports multiple discovery methods such as TCP port scanning, SNMP device discovery, and cloud platform API calls; The minimal configuration registration strategy only requires entering basic information (IP, credentials, region), and the system can automatically complete the driver and instance parameters; Adaptive access algorithm: Automatically selects the optimal connector and acquisition mode based on latency, stability and source type characteristics.

[0053] Example: In the scenario of ecological and environmental monitoring, the platform only needs to input the target network segment and cloud account information, and the system can automatically discover the water quality monitoring database, meteorological API service and pollution source message queue, and automatically select relational database driver, HTTP adapter or streaming connector for different sources, without the need for manual configuration one by one.

[0054] (1.2) Multi-mode fusion acquisition engine module

[0055] The engine employs a collaborative approach of three extraction mechanisms: pattern-driven, rule-driven, and intelligent learning. Among them, pattern-driven extraction: quickly obtains structural information by parsing database dictionaries and DDL statements; Rule-driven extraction: Identifies and completes primary and foreign key, constraint, and encoding information based on a pre-built rule base; Intelligent learning extraction: Using sample data and machine learning models to infer field types, encoding methods, and business semantics.

[0056] The three extraction methods mentioned above complement each other, enabling the data collection engine to handle both traditional structured databases and semi-structured databases such as JSON and XML, as well as unstructured databases such as logs and multimedia.

[0057] In this engine module, its three-mode fusion architecture improves the comprehensiveness and accuracy of metadata acquisition; the adaptive collection strategy can dynamically adjust the extraction method combination according to the complexity of the data source; and the incremental collection optimization supports efficient incremental extraction based on timestamps, version numbers, and hash values.

[0058] Example: In the air quality database, the system uses pattern-driven parsing of table structure and field indexes; it uses rule-driven identification of pollution factor fields such as PM2.5 and PM10 and supplements them with unit information; at the same time, it uses a machine learning model trained on sample data to infer the business semantics of the "site number" field, and finally integrates them to obtain a complete metadata description.

[0059] (1.3) Metadata Governance Process Module

[0060] The extracted metadata enters the automated governance pipeline and completes the following processes sequentially: Cleaning: Remove redundancy, repair missing information, and filter abnormal information; Standardization: Unify field naming, data types, and time and space formats; Integration: Aligning and merging metadata from different sources to form a unified metadata model (UMM).

[0061] This metadata governance process module ensures the controllability and repeatability of the processing through pipeline-style automated governance; it introduces a domain-adaptive rule engine to support governance standards for specific industries (such as ecological environment and financial risk control); and it automatically constructs metadata lineage relationships to improve cross-system traceability and consistency.

[0062] Example: For data from different monitoring stations, the treatment pipeline standardizes the fields "cod", "COD concentration", and "chemical oxygen demand" into "COD_Concentration" and unifies the unit to "mg / L"; at the same time, it establishes the lineage of fields and tracks their transmission path in subsequent reports and model analysis.

[0063] (1.4) Trusted Metadata Repository and Service-Oriented Publishing Module

[0064] The processed metadata is written to a trusted metadata repository, which supports: Idempotent write: avoids duplicate storage; Version control requires backtracking: ensuring that the evolution of metadata is traceable; Integrity verification: Ensures that metadata has not been tampered with, based on hash or signature mechanisms.

[0065] Meanwhile, the system is released to the outside world through APIs and SDKs, providing retrieval, query, semantic tag calling and incremental subscription services to upper-layer systems, meeting the multi-level data governance and application needs.

[0066] This module ensures the integrity and consistency of metadata through a trusted storage mechanism; multi-version concurrency control supports historical version backtracking and differential comparison; and intelligent caching and tiered storage strategies improve access efficiency.

[0067] Example: In the provincial-level ecological and environmental platform, the metadata after treatment is stored in a trusted repository, and the system records multiple versions of the water quality monitoring table. When a user queries "monitoring factor COD" via API, they can obtain the latest version of the field definition or view historical changes by version; at the same time, the system improves query response speed through caching strategies.

[0068] like Figure 3 As shown, a unified heterogeneous data source processing framework is used to shield the differences between different data sources and achieve unified access, parsing, and standardized mapping of cross-type data. This framework, through three layers—plug-in connectors, a unified access channel, and a standardized parsing engine—enables consistent processing of relational databases, NoSQL databases, file systems, streaming data, and API services, thereby ensuring metadata compatibility and scalability.

[0069] (2.1) Plug-in connector

[0070] This unified heterogeneous data source processing framework module provides connectivity and adaptation capabilities for various data sources. Through a pluggable connector mechanism, it shields the underlying protocol, driver, and version differences. Connectors can be dynamically loaded and upgraded at runtime to support rapid access to new databases and proprietary data sources.

[0071] This model supports hot-swapping through a plug-in architecture, allowing for the expansion of new data sources without modifying the core system; it has high version compatibility, supporting the coexistence of multiple versions of the same type of data source; and it features automatic parameter mapping, where the connector automatically maps common parameters to configuration items for specific sources.

[0072] Example: When accessing document-oriented databases and domestically produced databases, the system calls the corresponding connector plugins to automatically complete authentication protocols and version adaptation. For newly introduced time-series databases, only the new plugin package needs to be loaded for access; no modification to the core framework is required.

[0073] (2.2) Unified access channel

[0074] This unified heterogeneous data source processing framework module establishes a unified authentication, access control, and data interaction interface. Regardless of the data source type, all communication is conducted through a unified access channel, ensuring security and consistency.

[0075] The unified access channel uses a single sign-on authentication method to unify identity authentication and access control, avoiding multiple authentication systems running in parallel; it sets up a standard interface, and all data source interactions are uniformly abstracted into an interface call of "connect-query-get metadata"; and it improves system robustness through security isolation methods such as channel control traffic, logs, and anomaly isolation.

[0076] Example: In environmental data acquisition scenarios, users only need to authenticate once to access water quality databases, meteorological APIs, and message queues through a unified access channel. The system automatically completes multi-source permission verification and traffic control.

[0077] (2.3) Standardized parsing engine

[0078] This module maps metadata from different sources to a unified metadata model (UMM). The parsing engine is responsible for extracting and transforming information such as table structure, fields, indexes, file headers, APIs, and schemas, outputting a consistent abstract representation across sources.

[0079] The standardized parsing engine is based on a model abstraction strategy and designs a unified metadata model (UMM) that can cover structured, semi-structured and unstructured features. The mapping rules are based on a mapping library to uniformly convert source attributes (field names, types, units) into standardized definitions. The extended interface supports the connection and expansion of industry standards (such as national standards for ecological and environmental monitoring).

[0080] Example: When accessing meteorological data in JSON format and meteorological tables in a relational database, the parsing engine converts both into UMM expressions: the fields "temp" and "temperature_celsius" are both mapped to the standard field "Temperature", the type is uniformly floating point, and the unit is labeled "℃".

[0081] Therefore, for Figure 3 As shown in the unified heterogeneous data source processing framework diagram, its processing steps include: Step 1: Submit a connection request; The data source client initiates a connection request, submitting its access needs to the system, which serves as the starting point for the entire data collection process.

[0082] Step 2: Adapting plug-in connectors; The system receives requests through plug-in connectors, which are responsible for driver loading and protocol adaptation. Different types of data sources can be accessed through different plug-in components, shielding underlying differences.

[0083] Step 3: Request conversion and authentication; The connector converts external requests into a unified format that the system can recognize, and performs identity authentication and permission verification to ensure that data access is secure and legal.

[0084] Step 4: Unified access channel processing; The transformed request is sent to a unified access channel, which is responsible for traffic control, error isolation, and unified calling specifications, thereby achieving a consistent access standard across sources.

[0085] Step 5: Transmit connection information and metadata; After successful authentication, the unified access channel transmits the connection information and the raw metadata obtained from the data source to the standardized parsing engine.

[0086] Step 6: Standardized parsing; The standardized parsing engine cleans, identifies, maps, and completes rules for metadata, converting it into an intermediate representation that conforms to the system standard.

[0087] Step 7: Generate a unified metadata model (UMM); The parsing results are integrated and written into a unified metadata model (UMM), completing the standardized abstraction of attributes from different data sources.

[0088] Step 8: Write to the metadata repository; The generated UMM is stored in a metadata repository, forming a trusted versioned management system. The system also returns confirmation information and a version number to the data source client, completing a full closed-loop process.

[0089] Dynamic monitoring and real-time update mechanisms, such as Figure 4 As shown, in a multi-source heterogeneous environment, the structure and content of data sources may change frequently. If metadata cannot be synchronized in real time, it can easily lead to a disconnect between the metadata repository and the source, affecting the accuracy of governance and applications. The dynamic monitoring and real-time update mechanism proposed in this invention adopts a collaborative approach of log probes, event buses, incremental collection engines, and consistency checks to achieve second-level change capture, incremental updates, and version rollback, ensuring that the metadata repository and data source maintain a high degree of consistency.

[0090] (3.1) Log probes and change capture (CDC)

[0091] This module deploys log probes or calls the CDC (Change Data Capture) mechanism on the data source side to capture structural changes (such as table additions, field modifications, and index adjustments) and configuration changes (such as API parameter changes) in the metadata layer in real time.

[0092] Multi-source CDC adaptation supports database logs, file change monitoring, and API schema version monitoring; it collects data via a bypass method, enabling lightweight deployment without affecting the performance of the source system; and it captures data at a fine-grained level, capable of identifying metadata changes at both the field and configuration levels.

[0093] Example: In the water quality monitoring database, when the administrator adds the field "ammonia_nitrogen", the log probe immediately captures the event and generates a change record, which is then sent to the event bus.

[0094] (3.2) Event bus and asynchronous transmission

[0095] This module converts captured change events into a standard message format and pushes them to the event bus, achieving decoupling and asynchronous transmission, and ensuring the reliability of the system under high concurrency.

[0096] The standardized message format specifically describes all events as {source ID, change type, timestamp, change details}; events and the acquisition engine are asynchronously decoupled through a bus, improving system scalability; and multi-channel transmission supports message queues and stream processing frameworks, ensuring low latency in event transmission.

[0097] Example: When the schema version of the meteorological API is updated, the change event is formatted as a JSON message and pushed to the message queue event bus for downstream data acquisition engines to subscribe to.

[0098] (3.3) Incremental Acquisition Engine

[0099] This module collects and updates only the changed parts based on event information, avoiding a full scan and improving efficiency.

[0100] Incremental data collection is achieved through differential calculation, specifically based on timestamps, version numbers, and hash values. It only processes newly added or changed fields without affecting unmodified parts, thus achieving accurate updates. Furthermore, in high-frequency change scenarios, small batch events are merged into a single update task.

[0101] Example: When the noise monitoring log file updates the configuration of some stations, the incremental acquisition engine only extracts the newly added field "station_location_code" and appends it to the existing metadata record, without having to rescan the entire log directory.

[0102] (3.4) Consistency verification and repository write-back

[0103] Before updating, this module performs consistency checks on incremental patches to ensure compatibility between old and new metadata, and writes them to a trusted metadata repository through a transaction mechanism. At the same time, it generates a new version number to support historical backtracking.

[0104] This module uses bidirectional verification to check whether changes conflict with existing metadata in the repository; transaction writing uses a transaction mechanism to ensure the atomicity and integrity of the writing process; version management supports generating a new metadata version for each update, and supports traceability and differential analysis.

[0105] Example: In the atmospheric monitoring system, before adding the new field "pm1_concentration", the system automatically checks its type compatibility with the existing field "pm2.5_concentration". After successful verification, version v5 is generated in the repository, and users can retrospectively view the differences between v4 and v5.

[0106] Semantic enhancement and intelligent annotation engine architecture, such as Figure 5 As shown, traditional metadata collection often remains at the technical level (such as tables, fields, and indexes), lacking an understanding of business semantics, which limits cross-system data retrieval and intelligent analysis. The semantic enhancement and intelligent annotation engine proposed in this invention combines natural language processing (NLP), machine learning, and knowledge graph technologies to achieve semantic parsing, intelligent classification, tag generation, and repository write-back of metadata, thereby significantly improving the understandability, reasonability, and reusability of metadata.

[0107] (4.1) Natural Language Processing (NLP) Semantic Parsing

[0108] This module uses techniques such as word segmentation, part-of-speech tagging, and dependency analysis to perform semantic parsing on field names, annotations, and sample data to extract potential business meanings.

[0109] This module supports multiple languages ​​and is compatible with parsing Chinese, English, and other languages; it analyzes the context by combining field context with annotations to perform semantic disambiguation; and it outputs vectorized semantic vectors that can be used in deep learning models.

[0110] Example: The field name "COD_Conc" is parsed and identified as "Chemical Oxygen Demand Concentration", and further converted into a vector representation [0.13, 0.58, ...] for subsequent classification.

[0111] (4.2) Machine Learning Classification and Clustering

[0112] This module uses supervised and unsupervised models to classify and cluster fields, inferring their business category and data theme.

[0113] Combining classification and clustering analysis, it can identify known categories as well as discover potential new categories; it integrates field names, data distribution, and contextual information for semantic feature fusion; and the model can continuously optimize classification accuracy through an adaptive training strategy that enables continuous learning.

[0114] Example: The fields “pm25”, “pm10”, and “o3” in the atmospheric monitoring data table are classified. The model automatically identifies them as “air pollutants” and clusters them into the same topic.

[0115] (4.3) Knowledge Graph Alignment and Semantic Annotation

[0116] This module aligns the parsing and classification results with industry knowledge graphs, ontology models, and standard terminology databases to generate unified semantic tags and perform tag inheritance and hierarchical mapping when necessary.

[0117] It supports multi-industry alignment with knowledge bases in fields such as ecological environment, water conservancy, and finance; it supports hierarchical labeling of primary and secondary indicators; and it automatically selects the result with higher confidence when field labels from different sources conflict, in order to resolve semantic conflict detection.

[0118] Example: The field "NH3N" is aligned with the knowledge graph and identified as "ammonia nitrogen concentration", and inherits the parent label "water quality index" to form a hierarchical label path "water environment → water quality factor → ammonia nitrogen".

[0119] (4.4) Intelligent recommendation and warehouse write-back

[0120] This module writes the generated semantic tags and intelligent recommendation results into a trusted metadata repository, providing support for subsequent retrieval, data services, and decision analysis.

[0121] Similar field recommendation is based on semantic similarity to recommend synonymous fields across systems; the annotation results are directly written to the repository and a new version is generated to achieve an automatic write-back mechanism; the tags and semantic information are displayed intuitively through the front-end interface to achieve a visual presentation.

[0122] Example: In the water environment database, the field “COD_MgL” is labeled as “COD_Concentration”. The system also recommends the field “ChemicalOxygenDemand” as a synonym and writes it back to the repository in a unified manner to facilitate cross-departmental sharing and access.

[0123] Based on the aforementioned system for automatically collecting metadata from heterogeneous data sources, this invention also discloses a method for automatically collecting metadata from heterogeneous data sources, the implementation steps of which are as follows: Figure 6 As shown, the cleanup process consists of six stages, gradually completing a closed loop from data source access, metadata collection, cleaning and governance, to dynamic updates, semantic enhancement, and service-oriented release.

[0124] S1. Automatic Data Source Discovery and Access:

[0125] In the first phase, through proactive scanning, cloud directory API calls, and minimal configuration registration, the system automatically discovers and connects to various heterogeneous data sources, including relational databases, NoSQL databases, file systems, streaming message queues, and API interfaces. The system can automatically select either a plugin adapter or a native driver based on the data source type and complete credential authentication and secure connection establishment, thereby reducing reliance on manual configuration and ensuring the efficiency and security of the connection process.

[0126] Example: In the provincial environmental monitoring platform, the system automatically discovers a water quality monitoring database and a message queue real-time sewage discharge log stream, automatically loads a relational database Connector for the former and a StreamConnector for the latter, without manual intervention.

[0127] The code is shown below:

[0128] for target in NetworkRange: # Iterate through the network range or asset list to be scanned if ScanPort(target, ports=[1521,3306]): # If the target host has common relational database ports open (such as Oracle 1521 / MySQL 3306) RegisterSource(type="RDB", host=target) # Register the target as a relational database data source elif DetectStream(target): # Otherwise, if streaming message characteristics (such as log queues, Kafka streams, etc.) RegisterSource(type="Stream", host=target) # Register the target as a streaming data source

[0129] S2. Multi-mode fusion acquisition and analysis:

[0130] After the connection is completed, the system calls the data acquisition engine to perform multi-mode fusion extraction, covering three methods: pattern-driven, rule-driven, and intelligent learning. These three methods complement each other, ensuring the integrity of structured information while enhancing adaptability to semi-structured and new data sources.

[0131] Example: In the air quality database, the pattern drives the parsing of the table structure; the rule drives the identification of the PM2.5 and PM10 fields and adds the unit μg / m³; the intelligent learning module infers that "station_id" is the monitoring station identifier through samples.

[0132] The code is shown below:

[0133] for field in SourceFields: # Iterate through all fields in the source database schema = ParseDDL(field) # Schema-driven: Parse field definitions or data dictionaries rule = ApplyRuleBase(field) # Rule-driven: Match field scope or unit ml according to the rule base, conf = ML_Infer(field.samples) # Intelligent learning: Infer field meaning and confidence using sample data if conf > 0.8: # If the confidence of the machine learning inference is greater than 80% result[field] = ml # Prioritize the intelligent learning result elif rule: # Otherwise, if there is a rule matching result result[field] = rule # Use the rule base result else: # If neither covers result[field] = schema # Fall back to the DDL parsing result

[0134] S3, metadata cleaning, standardization and integration:

[0135] The collected raw metadata enters the governance pipeline, where it is cleaned, standardized, and integrated in sequence to form a consistent unified metadata model (UMM).

[0136] Example: The system standardizes the fields "cod", "COD concentration", and "chemical oxygen demand" into "COD_Concentration" and unifies the unit to "mg / L"; at the same time, it establishes a lineage relationship to track the transmission path of this field in subsequent reports.

[0137] The code is shown below:

[0138] for record in MetadataSet: # Iterate through the collected metadata set Clean(record) # Cleaning: deduplication, repairing missing fields, removing abnormal data Standardize(record, rules) # Standardization: unifying field naming, units of measurement, spatiotemporal coordinates, etc. UMM = Integrate(MetadataSet) # Integration: unifying the processed metadata into a UMM (Unified Metadata Model).

[0139] S4. Dynamic monitoring and real-time updates:

[0140] The system deploys log probes or CDC agents at the source end to capture structural and schema change events in real time and trigger incremental data collection and repository updates.

[0141] Example: When a new field "pm1_concentration" is added to the atmospheric monitoring database, the probe captures the event, generates an incremental patch, and updates only the newly added field without having to rerun the full database scan.

[0142] The code is shown below:

[0143] if CDC_Event(field_change): # When a data source structure change event is captured (such as adding a field) delta = BuildPatch(change) # Build the corresponding incremental patch (containing only the changed part) if Validate(delta, Warehouse): # Validate the compatibility and consistency between the incremental patch and the existing repository Warehouse.Update(delta) # Update the metadata in the repository and generate the new version

[0144] S5, Semantic Enhancement and Intelligent Annotation:

[0145] After being stored, the metadata is parsed by the semantic engine using NLP, classified by machine learning, and aligned with the knowledge graph to generate semantic tags and provide recommendations.

[0146] Example: The field "NH3N" is automatically identified as "ammonia nitrogen concentration" and labeled with the parent tag "Water Quality Indicators → Ammonia Nitrogen". Simultaneously, the system recommends "ammonia_nitrogen" as a synonym field for easy cross-database retrieval.

[0147] The code is shown below:

[0148] for field in Warehouse: # Iterate through field objects in the warehouse tokens = NLP_Parse(field.name) # Parse the field name using NLP and extract semantic features label, conf = ML_Classify(tokens) # Use machine learning to classify and obtain semantic labels and confidence levels tag = AlignKnowledgeGraph(label) # Align with the industry knowledge graph to generate hierarchical semantic labels Warehouse.Update(field, tag) # Write the label information back to the warehouse to support retrieval and recommendation

[0149] S6, Metadata Repository Storage and Service Deployment:

[0150] Ultimately, the governed and enhanced metadata is written to a trusted repository, supporting idempotent writing, version management, and historical rollback, and is provided to external users through APIs and SDKs.

[0151] Example: When regulators query the COD field via API, they can obtain the latest definition as well as revert to the old version; at the same time, the caching strategy ensures that the query latency is less than 100ms.

[0152] The code is shown below:

[0153] if not Exists(Hash(metadata)): # Verify the hash value to ensure idempotency (avoid duplicate entries) version = CreateVersion(metadata) # Generate a version number for the current metadata, supporting rollback Warehouse.Write(metadata, version) # Write the metadata to the repository and establish version management ExposeAPI(metadata, version) # Publish the metadata service via API for external systems to call

[0154] In summary, the system and method for automatically collecting metadata from heterogeneous data sources disclosed in this invention have the following innovative features:

[0155] (1) End-to-end fully automated data acquisition closed loop

[0156] The unattended pipeline is formed from "source discovery → adaptive access → multi-mode extraction → governance → repository storage → service release", which significantly reduces the cost of manual configuration and script maintenance.

[0157] "Minimized configuration + adaptive access" allows for one-click parallel access in mixed environments such as multi-cloud / local / dedicated line.

[0158] (2) Three-mode fusion acquisition engine

[0159] By combining "pattern-driven (dictionary / DDL parsing) + rule-driven (constraint / caliber rule base) + intelligent learning (sample inference, distribution features, weak supervision)", it takes into account structured, semi-structured and unstructured sources, and improves both coverage and accuracy.

[0160] (3) Unified heterogeneous access and standardized resolution

[0161] The plug-in connectors shield driver / protocol / version differences; the unified access channel is abstracted into a standard interface of "connection-authentication-query-data retrieval"; the standardized parsing engine maps source attributes to a unified metadata model (UMM), naturally supporting extension and cross-source comparison.

[0162] (4) Event-driven dynamic monitoring and near real-time incremental updates

[0163] CDC / log probe → event bus → incremental patch → consistency check → transaction write-back → multi-version management, supports second-level structure change capture, avoids "full rescan", and ensures high consistency between the repository and the source.

[0164] (5) Semantic enhancement and knowledge alignment

[0165] By combining NLP, ML, and industry knowledge graphs, semantic parsing, classification, clustering, and hierarchical labeling are performed on fields / tables; synonym / near-synonym recommendations are provided to support semantic retrieval and definition reuse, significantly improving the understandability of metadata and the level of intelligent governance.

[0166] (6) Trusted warehouse storage and service output

[0167] Idempotent write, integrity verification, version rollback, and differential comparison; provides query, tagging, and incremental notification capabilities to the outside world through API / SDK / subscription channels, facilitating unified consumption by upper-layer governance tools and business systems.

[0168] Therefore, this invention addresses the shortcomings of existing metadata collection methods, such as poor heterogeneous compatibility, reliance on manual configuration, delayed updates, and lack of semantic understanding. It proposes a fully automated metadata collection system and method for heterogeneous data sources. Through a fully automated collection mechanism, a unified heterogeneous data source processing framework, a dynamic monitoring and real-time update mechanism, and the collaborative work of semantic enhancement and intelligent annotation engines, an end-to-end automated closed loop for metadata collection, governance, and application is achieved. Compared with existing technologies, it has the following technical advantages: (1) Automation and high efficiency

[0169] By using intelligent source discovery and adaptive access, the drawbacks of traditional manual configuration are avoided, enabling zero-configuration access and rapid adaptation, and significantly improving the access efficiency of new data sources.

[0170] (2) Comprehensiveness and compatibility

[0171] Based on pluggable connectors and a standardized parsing engine, it achieves unified processing of relational databases, NoSQL, file systems, streaming data, and APIs. Through a three-mode fusion acquisition engine (pattern-driven, rule-driven, and intelligent learning), it covers structured, semi-structured, and unstructured data scenarios, forming a panoramic view of full-domain metadata.

[0172] (3) Real-time performance and consistency

[0173] By leveraging log probes and CDC mechanisms, combined with an event bus and incremental data acquisition engine, second-level capture and updates are achieved. This ensures consistency between the metadata repository and source data, avoiding governance deviations caused by delays or missing data, and reducing system resource consumption in high-frequency change scenarios.

[0174] (4) Intelligence and comprehensibility

[0175] By incorporating NLP parsing, machine learning, and knowledge graph alignment, the system performs semantic annotation and intelligent classification of metadata. It can automatically identify field semantics and business implications, supporting semantic retrieval, tag recommendation, and definition reuse, significantly improving the interpretability and reusability of metadata.

[0176] (5) Credibility and traceability

[0177] The trusted metadata repository supports idempotent writes, version management, and integrity verification, ensuring the security, stability, and traceability of metadata. Users can query field definitions and lineages from different versions at any time, achieving data consistency across systems and departments.

[0178] (6) Application support capabilities

[0179] By outputting the governed metadata through service-oriented interfaces (APIs, SDKs), support is provided for upper-layer data governance tools, analysis platforms, and business systems. In the context of ecological and environmental monitoring, regulatory authorities can use a unified metadata model to compare monitoring factors across departments and regions and analyze trends, thereby improving the scientific rigor and timeliness of decision-making.

[0180] (7) Summary of Results

[0181] In summary, this invention represents a significant improvement over existing technologies in terms of automation, heterogeneous compatibility, real-time update capabilities, semantic intelligence, trusted storage, and service-oriented capabilities. This method not only addresses the pain points of current metadata collection and governance but also possesses excellent scalability and versatility, enabling its widespread application in multi-source heterogeneous data governance scenarios such as ecological environment and natural resources, providing a solid foundation for data asset operation and intelligent decision-making.

[0182] The above embodiments are not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the technical solution of the present invention are also within the protection scope of the present invention.

Claims

1. A system for automatically collecting metadata from heterogeneous data sources, characterized in that: It includes a bottom-up distributed data source layer, core technology layer, and output layer; The data source layer is used to support the access of multiple types of heterogeneous data sources, including relational databases, NoSQL databases, file systems, streaming data, and API interfaces. The core technology layer consists of four mechanisms that collaboratively realize automatic metadata discovery, unified modeling, real-time synchronization and intelligent governance. The four mechanisms are a fully automated metadata collection mechanism, a unified heterogeneous data source processing framework, a dynamic monitoring and real-time update mechanism, and a semantic enhancement and intelligent annotation engine. The output layer includes a unified metadata model, a metadata repository, and intelligent annotation and recommendation services, constructing an end-to-end automated closed loop.

2. The system for automatically collecting metadata from heterogeneous data sources according to claim 1, characterized in that: The fully automated metadata collection mechanism includes: an intelligent source discovery and adaptive access module, a multi-mode fusion collection engine module, a metadata governance process module, and a trusted metadata repository and service publishing module; The intelligent source discovery and adaptive access module automatically identifies and accesses relational databases, file systems, streaming data, and API interfaces through multi-protocol network intelligent detection, cloud resource directory identification, and minimal configuration registration strategy technology. The multi-mode fusion acquisition engine module employs three extraction mechanisms—pattern-driven, rule-driven, and intelligent learning—to operate collaboratively. The metadata governance process module enables the extracted metadata to enter an automated governance pipeline, where it is sequentially cleaned, standardized, and integrated to form a unified metadata model (UMM). The trusted metadata repository and service publishing module writes the governed metadata into the trusted metadata repository, supports idempotent writing, version control and rollback, and publishes it to the outside world through API and SDK.

3. The system for automatically collecting metadata from heterogeneous data sources according to claim 2, characterized in that: In the multi-mode fusion acquisition engine module, mode-driven extraction is used to parse database dictionaries and DDL statements to quickly obtain structural information; Rule-driven extraction identifies and completes primary and foreign key, constraint, and encoding information based on a pre-built rule base. Intelligent learning extraction uses sample data and machine learning models to infer field types, encoding methods, and business semantics.

4. The system for automatically collecting metadata from heterogeneous data sources according to claim 1, characterized in that: The unified heterogeneous data source processing framework is used to shield the differences between different data sources, enabling unified access, parsing, and standardized mapping of cross-type data; The unified heterogeneous data source processing framework achieves consistent processing across relational databases, NoSQL databases, file systems, streaming data, and API services through three layers: pluggable connectors, unified access channels, and standardized parsing engines.

5. The system for automatically collecting metadata from heterogeneous data sources according to claim 4, characterized in that: The pluggable connector shields underlying protocols, drivers, and version differences through a pluggable connector mechanism; The unified access channel establishes a unified authentication, access control, and data interaction interface; Unified authentication unifies identity authentication and access control through a single sign-on method; The data interaction interface is set as a standard interface, and all data source interactions are uniformly abstracted into an interface call of "connect-query-get metadata"; Access control is implemented through security isolation methods, including control traffic, logs, and anomaly isolation. The standardized parsing engine is used to map metadata from different sources into a unified metadata model (UMM).

6. The system for automatically collecting metadata from heterogeneous data sources according to claim 1, characterized in that: The dynamic monitoring and real-time update mechanism is implemented through a collaborative approach of log probes and Change Capture Center (CDC), event bus and asynchronous transmission, incremental acquisition engine and consistency verification and repository write-back.

7. The system for automatically collecting metadata from heterogeneous data sources according to claim 6, characterized in that: By deploying log probes on the data source side or calling the CDC mechanism, structural changes and configuration changes in the metadata layer can be captured in real time. The event bus and asynchronous delivery transform captured change events into a standard message format and push them to the event bus, achieving decoupling and asynchronous delivery. The incremental acquisition engine collects and updates only the changed parts through differential calculation. The differential calculation method includes calculating differences based on timestamps, version numbers, and hash values. Before writing the update, the dynamic monitoring and real-time update mechanism performs consistency verification on the incremental patch and writes it to the trusted metadata repository through a transaction mechanism, while generating a new version number to support historical backtracking.

8. The system for automatically collecting metadata from heterogeneous data sources according to claim 1, characterized in that: The semantic enhancement and intelligent annotation engine, which combines natural language processing (NLP), machine learning, and knowledge graph technologies, supports semantic parsing, intelligent classification, tag generation, and repository write-back of metadata. Semantic parsing techniques in Natural Language Processing (NLP) include word segmentation, part-of-speech tagging, and dependency parsing. These techniques perform semantic parsing on field names, annotations, and sample data to extract potential business meanings. Machine learning classification and clustering uses supervised and unsupervised models to classify and cluster fields, inferring their business categories and data themes; Knowledge graph alignment and semantic annotation align the parsing and classification results with industry knowledge graphs, ontology models and standard terminology databases, generate unified semantic tags, and perform tag inheritance and hierarchical mapping when necessary; The system can recommend and write back to the repository, writing the generated semantic tags and intelligent recommendation results into a trusted metadata repository and displaying them intuitively through the front-end interface.

9. A metadata collection method for a system for automatically collecting metadata from heterogeneous data sources as described in any one of claims 1-8, characterized in that: The metadata collection method includes the following steps: S1. Automatic Data Source Discovery and Access: Automatically discovers and accesses multiple heterogeneous data sources through proactive scanning, cloud directory interface calls, and minimal configuration registration; S2, Multi-mode fusion acquisition and parsing: After the access is completed, the acquisition engine is called to perform multi-mode fusion extraction, covering three methods: pattern-driven, rule-driven and intelligent learning, to perform multi-mode fusion acquisition and parsing of multiple heterogeneous data sources; S3. Metadata Cleaning, Standardization and Integration: The raw metadata collected enters the governance pipeline, where it is cleaned, standardized and integrated in sequence to form a consistent unified metadata model (UMM). S4. Dynamic monitoring and real-time updates: Deploy log probes or CDC agents at the source end to capture structure and schema change events in real time and trigger incremental collection and repository updates. S5. Semantic Enhancement and Intelligent Labeling: After the metadata is entered into the database, the semantic engine performs NLP parsing, machine learning classification, and knowledge graph alignment to generate semantic tags and provide recommendations. S6, Metadata Repository Storage and Service-Oriented Deployment: Governed and enhanced metadata is written to a trusted repository, supporting idempotent writing, version management and historical rollback, and providing services to external parties through APIs and SDKs.

10. The metadata collection method for a system that automatically collects metadata from heterogeneous data sources according to claim 9, characterized in that: The unified processing steps for multiple heterogeneous data sources include: Step 1: Submit a connection request: The data source client first initiates a connection request, submitting its access requirements to the system, which serves as the starting point for the entire data collection process; Step 2: Plug-in Connector Adaptation: Requests are received through plug-in connectors, which are responsible for driver loading and protocol adaptation. Different types of data sources can be accessed through different plug-in components, shielding underlying differences. Step 3: Request Transformation and Authentication: The connector transforms external requests into a unified format that the system can recognize, and performs identity authentication and permission verification; Step 4: Unified access channel processing: The transformed request is sent to the unified access channel, which is responsible for traffic control, error isolation, and unified calling specifications; Step 5: Transmit connection information and metadata: After successful authentication, the unified access channel transmits the connection information and the raw metadata obtained from the data source to the standardized parsing engine; Step 6: Standardized parsing: The metadata is cleaned, its structure is identified, its attributes are mapped, and rules are completed through a standardized parsing engine, and then uniformly converted into an intermediate representation that conforms to the system standard; Step 7: Generate a unified metadata model (UMM): The parsing results are integrated and written into a unified metadata model (UMM), completing the standardized abstraction of attributes from different data sources; Step 8: Write to the metadata repository: The generated UMM is stored in the metadata repository to form a trusted version management system; the system also returns confirmation information and version number to the data source client, completing a complete closed-loop process.

Images