Method and device for collecting global metadata and automatically constructing association relationship of metadata

Abstract

The invention relates to the technical field of big data, in particular to a method and a device for collecting global metadata and automatically constructing a metadata association relationship, which are characterized in that a main control platform and a collector Agent are designed, and metadata collectors of different types are integrated through a new technical architecture; collection of all types of metadata in the whole domain can be achieved only through one technical architecture; in order to solve the problem of constructing the full-link relationship of the metadata, a technology for constructing the full-link relationship for each type of metadata is adopted, and the association relationship of each type of metadata dispersed in a data circulation full chain can be intelligently and automatically established on the basis of a metadata unified identifier generation technology and a cross-modal relationship construction technology; the invention provides a metadata acquisition technology plug-in implementation method, which comprises a plug-in standard interface, centralized configuration management and unified state monitoring, and solves the major engineering problem of seamlessly integrating various heterogeneous acquisition technologies into one platform.

Description

Technical Field

[0001] This invention relates to the field of big data technology, specifically to a method and apparatus for collecting full-domain metadata and automatically constructing metadata relationships. Background Technology

[0002] In the field of public data authorization and operation, data circulation involves multiple stages and participating entities, and its security risks are characterized by concealment, transmissibility, and complexity. To effectively manage the security risks of data in open utilization, fine-grained monitoring of the entire data circulation process is necessary, from data collection, processing, storage, and uploading to cataloging, data application, data approval, data authorization, data product development, and data authorization supply. To achieve fine-grained monitoring of the entire chain, it is essential to explore and collect the metadata information generated throughout the chain, and then analyze the collected metadata. Currently, industry solutions for full-chain metadata collection in public data authorization and operation face the following main challenges: Metadata collection mainly focuses on metadata of key data flow processes in a single link, such as database operation monitoring log metadata, data security classification and grading metadata, and data call status metadata, which is difficult to support risk analysis and monitoring of the entire data flow process.

[0003] Full-chain metadata collection often requires the modification of business systems. Each business system needs to embed metadata collection-related technologies at every stage of the data flow. The modification and implementation costs are high and difficult, and may affect the stability and performance of business systems. It is also difficult to adapt to the heterogeneous and complex environment of public data authorization and operation platforms.

[0004] Different system scenarios and different modalities of data require different technologies for metadata collection. For example, for database or file metadata, SQL parsers or Sidercar technology are needed for collection. For information generated during the approval process, bytecode enhancement technology is needed for metadata collection. A single metadata collection technology alone cannot meet the needs of collecting various types of metadata throughout the entire data flow process.

[0005] A complete business data chain includes data collection, processing, warehousing, cataloging, application, approval, authorization, data product development, and authorization supply. The metadata collected in each process is fragmented, requiring manual effort to establish relationships between them. Summary of the Invention

[0006] According to a first aspect of the present invention, the present invention claims protection for a method for collecting global metadata and automatically constructing metadata association relationships, comprising the following steps: S1: Multiple metadata collection plugins are managed in a unified manner through a main control platform, wherein the main control platform includes a plugin manager, a plugin repository, a status monitoring manager, and a configuration center manager. The plugin repository stores metadata collection plugins that have been modified into components. The plugin manager registers and manages the metadata collection plugins. The configuration center manager assigns the selected metadata collection plugins to the collector agents. The status monitoring manager monitors the running status of the collector agents in real time. S2: The collector agent loads and executes the metadata collection plugin to collect metadata from multiple links in the entire data flow chain. The collector agent includes a configuration client, a heartbeat client, a collection framework, a collector plugin, and a data caching module. The configuration client configures the communication information between the collector agent and the main control platform. The heartbeat client periodically reports the running status of the collector agent. The collection framework loads and executes the collector plugin. The data caching module temporarily stores the collected metadata. S3: Generate a unified identifier for the collected metadata, wherein the unified identifier is composed of key information of the metadata, including timestamp, host ID, database name, field name, business type, operation description, operation user and permissions; S4: The intelligent association analysis engine automatically constructs the association relationship between metadata based on the unified identifier, wherein the intelligent association analysis engine analyzes the metadata identifier content using at least one strategy among rule-based construction, semantic construction, and graph reasoning, and outputs the metadata association relationship; S5: Store the constructed metadata relationships in a metadata repository to support metadata applications.

[0007] Furthermore, S1 also includes: The main control platform reads the metadata collection plugin JAR package from the plugin repository through the plugin manager and completes the plugin registration. The plugin manager displays the plugin list through a graphical interface. The configuration center manager selects the metadata collection plugin based on the plugin list and assigns it to the specified collector agent. The status monitoring manager collects the running logs and performance indicators of the collector agent through a heartbeat mechanism.

[0008] Furthermore, S2 also includes: The collector agent receives collection instructions from the main control platform through a configured client. The collection instructions include the target data source address, authentication information, and collection parameters. The heartbeat client sends running status information to the main control platform at a preset frequency. The collection framework dynamically loads the collector plugin JAR package through a class loader and calls the plugin execution function to start metadata collection. The data caching module temporarily stores the collected metadata in a buffer and triggers the unified metadata identifier generation process.

[0009] Furthermore, in step S3, the specific steps for generating a unified identifier include: S31: Receive metadata from the collector agent through the metadata information receiving module; S32: Read key information from metadata, including timestamp, host ID, database name, field name, business type, operation description, operation user and permissions; S33: Concatenate the key information into a string in a preset order to form a unique metadata identifier; S34: Store the unique identifier and metadata together in the database.

[0010] Furthermore, S4 also includes: When the intelligent association analysis engine adopts a rule-based construction strategy, it uses the Drools rule engine to parse the unique identifier content of metadata and identifies the association relationship of metadata based on the time proximity and attribute overlap rules. When adopting a semantic-based construction strategy, the TF-IDF model is used to calculate the semantic similarity of metadata identifier content, and metadata dependencies are determined based on similarity thresholds. When adopting a graph-based reasoning strategy, the Neo4j graph database is used to store metadata entities and relationships, and reasoning rules are applied to infer metadata relationships that are not directly related.

[0011] Furthermore, in the rule-based construction strategy, the rules include: If the timestamp difference between two metadata items is within a preset range and they share at least one identical attribute, then a relationship is determined to exist. In the semantic-based construction strategy, the semantic similarity threshold is set to 0.9; if the similarity exceeds the threshold, a dependency relationship is determined. In the graph-based reasoning strategy, the reasoning rules include: if metadata A is associated with metadata B and metadata B is associated with metadata C, then it is inferred that metadata A is associated with metadata C.

[0012] Furthermore, S2 also includes: The collector plugin includes at least one of a JDBC probe, an API parser, and an XML parser. The collector plugin is modularized through a standard interface specification, which includes an initialization method, a start collection method, a stop collection method, and a plugin check method.

[0013] Furthermore, S5 is followed by S6: Based on the relationships in the metadata repository, metadata application services are provided, including lineage analysis, end-to-end data flow monitoring, and root cause analysis. S1 also includes: The plugin repository uploads metadata collection plugin JAR packages through a graphical interface, the plugin manager adds plugins to the management list through the JAR package registration function, and the configuration center manager supports batch allocation of plugins to multiple collectors (Agents).

[0014] Furthermore, S4 also includes: The intelligent association analysis engine uses rule-based construction, semantic construction, and graph-based reasoning strategies in synergy. It applies the rule-based construction strategy to identify direct associations, then applies the semantic construction strategy to supplement similar metadata associations, and finally applies the graph-based reasoning strategy to infer indirect associations, thus constructing a complete metadata relationship graph.

[0015] According to a second aspect of the present invention, the present invention claims protection for an apparatus for collecting global metadata and automatically constructing metadata associations, comprising: One or more processors; A memory that stores one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the method for collecting global metadata and automatically constructing metadata associations.

[0016] This invention relates to the field of big data technology, specifically to a method and apparatus for collecting full-domain metadata and automatically constructing metadata relationships. It designs a main control platform and a collector agent, integrating different types of metadata collectors through a new technical architecture, enabling the collection of all types of metadata across the entire domain with only one technical architecture. To address the challenge of constructing full-link relationships for metadata, it employs a technology for building full-link relationships for various types of metadata. Based on unified metadata identifier generation technology and cross-modal relationship construction technology, it enables the intelligent and automatic establishment of relationships between various types of metadata scattered throughout the data flow chain. It provides a plug-in implementation method for metadata collection technology, including plug-in standard interfaces, centralized configuration management, and unified status monitoring, solving the major engineering challenge of seamlessly integrating multiple heterogeneous collection technologies into a single platform. Attached Figure Description

[0017] Figure 1 The flowchart illustrates a method for collecting global metadata and automatically constructing metadata associations as claimed in an embodiment of the present invention. Figure 2 This is a second flowchart of a method for collecting global metadata and automatically constructing metadata associations, as claimed in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0019] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0020] According to a first embodiment of the present invention, the present invention claims protection for a method for collecting global metadata and automatically constructing metadata association relationships, referring to... Figure 1 This includes the following steps: S1: Multiple metadata collection plugins are managed in a unified manner through a main control platform, wherein the main control platform includes a plugin manager, a plugin repository, a status monitoring manager, and a configuration center manager. The plugin repository stores metadata collection plugins that have been modified into components. The plugin manager registers and manages the metadata collection plugins. The configuration center manager assigns the selected metadata collection plugins to the collector agents. The status monitoring manager monitors the running status of the collector agents in real time. S2: The collector agent loads and executes the metadata collection plugin to collect metadata from multiple links in the entire data flow chain. The collector agent includes a configuration client, a heartbeat client, a collection framework, a collector plugin, and a data caching module. The configuration client configures the communication information between the collector agent and the main control platform. The heartbeat client periodically reports the running status of the collector agent. The collection framework loads and executes the collector plugin. The data caching module temporarily stores the collected metadata. S3: Generate a unified identifier for the collected metadata, wherein the unified identifier is composed of key information of the metadata, including timestamp, host ID, database name, field name, business type, operation description, operation user and permissions; S4: The intelligent association analysis engine automatically constructs the association relationship between metadata based on the unified identifier, wherein the intelligent association analysis engine analyzes the metadata identifier content using at least one strategy among rule-based construction, semantic construction, and graph reasoning, and outputs the metadata association relationship; S5: Store the constructed metadata relationships in a metadata repository to support metadata applications.

[0021] Furthermore, S1 also includes: The main control platform reads the metadata collection plugin JAR package from the plugin repository through the plugin manager and completes the plugin registration. The plugin manager displays the plugin list through a graphical interface. The configuration center manager selects the metadata collection plugin based on the plugin list and assigns it to the specified collector agent. The status monitoring manager collects the running logs and performance indicators of the collector agent through a heartbeat mechanism. The overall implementation logic of the main control platform adopts metadata collection technology, which is currently the mainstream metadata collection technology in the industry, including JDBC probes, APIs, XML parsers, etc. In accordance with standards and specifications, this embodiment provides a set of standard plugin interface specifications, which cover component initialization methods, methods for starting metadata collection, methods for stopping metadata collection, and plugin inspection methods; and provides standard modification reference templates for each collection technology in the form of Java classes; Regarding the modular transformation of data acquisition technology, based on the provided standards and specifications, the data acquisition technology is modularized. The modified components are all output as JAR files. For the plugin repository, the modified component JAR files are uploaded to the plugin repository through the main control platform's component repository interface; The plugin manager reads the JAR files in the component repository and registers them through the JAR file registration function. After successful registration, the component information can be viewed in the interface list. Configure the center manager. Based on the components successfully registered in the plugin manager, the center manager selects the required components and assigns them to the data collector agent. Collector Agent. The collector agent performs metadata collection tasks based on the assigned collection components; The Status Monitoring Manager monitors and collects the running status and log information of the Collector Agent in real time.

[0022] Furthermore, S2 also includes: The collector agent receives collection instructions from the main control platform through a configured client. The collection instructions include the target data source address, authentication information, and collection parameters. The heartbeat client sends running status information to the main control platform at a preset frequency. The collection framework dynamically loads the collector plugin JAR package through a class loader and calls the plugin execution function to start metadata collection. The data caching module temporarily stores the collected metadata in a buffer and triggers the unified metadata identifier generation process. The implementation logic of the collector agent is for the configuration client, which configures the communication address, protocol, and JAR path information between the collector agent and the main control platform; Configure the collector agent and the monitoring frequency of the main console collector for the heartbeat client; The data collection framework includes two modules: component loading and component execution. The background of the data collection framework automatically loads the data collection JAR package plugin based on the configuration information of the configured client and the heartbeat client. The data acquisition component executes the data acquisition JAR package plugin; Data caching is used to store the metadata collected by the collector in the cache area; Generate a unified identifier for metadata, creating a unique identifier for the metadata collected by the collector.

[0023] In a real-world scenario, Xiao Wang needs to collect metadata about all operations performed on the production order table in a MySQL database running on 192.168.1.100. This includes collecting data on who performed the operation, when, and what operation was performed. Let's reconstruct the business scenario based on the above sequence: Xiao Wang first accessed the client interface of the data collector on the main control platform, selecting the Agent deployed on the host 192.168.1.100, and then selecting the mysql-jdbc-probe plugin for the Agent. In the mysql-jdbc-probe plugin editing interface selected by the data collector Agent, he configured the target address information for data collection, as shown in the example below: { "jdbc.url": "jdbc:mysql: / / 192.168.1.100:3306 / ecommerce_db?useSSL=false", "jdbc.username": "metamonitor", "jdbc.password": " ", "monitored_tables": "orders_table", "collector_id": "collector_mysql_01" } After configuration, click "Issue" to complete the issuance of the data collection command.

[0024] b) Simultaneously, Xiao Wang also needs to configure the heartbeat client to periodically monitor the Agent's running status. Here, Xiao Wang has set it to monitor every 5 seconds. An example of the configuration information is as follows: { "agent_id": "agent_host_192.168.1.100", "timestamp": 5s, "status": "HEALTHY", "running_plugins": ["mysql-jdbc-probe"], "system_metrics": { "cpu_usage": "12.5%", "memory_usage": "45.8%", "queue_size": 5 / / Current number of data buffers in the queue } } The data collector agent client converts the issued data collection instructions into a format that the data collection framework can recognize, and sends a request to the data collection framework. After receiving the instructions, the data collection framework loads the data collection component using Java's class loader and calls the component execution function to execute the data collection component. An example of the data collection instruction request format is as follows: Agent_id:agent_host_192.168.1.100 Config:{"jdbc.url": "jdbc:mysql: / / 192.168.1.100:3306 / ecommerce_db?useSSL=false", "jdbc.username": "metamonitor", "jdbc.password": " ", "monitored_tables": "orders_table", "collector_id": "collector_mysql_01" } This feature tracks the execution process of components, including corresponding background processes and tasks. It's used to monitor whether components are executing correctly.

[0025] Data caching and data collection using the mysql-jdbc-probe component scan the MySQL database `orders_table` on server 192.168.1.100 for all operations, such as a user query generating an operation record: SELECT The plugin will intercept the SQL statement `FROMorders WHERE user_id = 12345;` and instantly generate a raw metadata record, as shown in the example below: { SQL: SELECT FROM orders WHERE user_id = 12345", "database": "ecommerce_db", "table": "orders", "operation": "SELECT", "source_ip": "192.168.1.50", / / Application server IP "user": "app_user", "timestamp": 1722504620000, "execution_time_ms": 15 }

[0026] The data caching unit temporarily writes the scanned metadata information into the cache database for storage, and at the same time calls the metadata information receiving interface in the metadata unified generation and identification unit to send metadata information to the metadata unified generation and identification unit.

[0027] Furthermore, referring to Figure 2 In step S3, the specific steps for generating a unified identifier include: S31: Receive metadata from the collector agent through the metadata information receiving module; S32: Read key information from metadata, including timestamp, host ID, database name, field name, business type, operation description, operation user and permissions; S33: Concatenate the key information into a string in a preset order to form a unique metadata identifier; S34: Store the unique identifier and metadata together in the database.

[0028] Metadata unified identifier generation generates a unique identifier for each piece of metadata collected by the collector agent. To facilitate the subsequent construction of metadata relationships based on this identifier, we designed the following components for the metadata identifier: timestamp, host ID, database name, field name, business type, operation description, operating user, and permissions. Example: 20250829103129-127.0.0.10-racle001-name.phone.xx-approval-download-admin-query; When receiving metadata information, the metadata collected by the collector Agent will be received into the metadata unified identifier generation framework through the metadata information receiving module.

[0029] When reading key information from metadata, key information is retrieved, including timestamp, host ID, database name, field name, business type, operation description, operation user, and permissions.

[0030] When concatenating metadata key information, the read key information is concatenated into a complete string. Example: 20250829103129-127.0.0.10-racle001-name.phone.xx-approval-download-admin-query.

[0031] The concatenated string is used as a unique identifier for metadata and stored in the database; Continuing with the above embodiment, after receiving metadata information from the metadata cache unit, the metadata information receiving unit extracts the key information from the metadata and concatenates the key information, as shown in the following example: Received information: { SQL: SELECT FROM orders WHERE user_id = 12345", "database": "ecommerce_db", "table": "orders", "operation": "SELECT", "source_ip": "192.168.1.50", / / Application server IP "user": "app_user", "timestamp": 1722504620000, "execution_time_ms": 15 } Metadata Identification Generation: Number: {Time of Action Occurrence + User + Address + Action + Database + Object Table} ID_SEL_987: {1722504620000 + Xiao Wang + 192.168.1.50 + select + ecommerce_db + orders}

[0032] Furthermore, in S4, it further includes: When the intelligent association analysis engine adopts the rule-based construction strategy, it uses the Drools rule engine to parse the content of the metadata unique identifier and identify the metadata association relationship according to the time proximity and attribute overlap rules; When adopting the semantics-based construction strategy, it uses the TF-IDF model to calculate the semantic similarity of the metadata identifier content and judges the metadata dependence relationship based on the similarity threshold; When adopting the graph reasoning strategy, it uses the Neo4j graph database to store metadata entities and relationships, and applies reasoning rules to infer metadata relationships that are not directly associated.

[0033] Among them, in this embodiment, the intelligent association analysis engine analyzes the relationship between characters based on the character information in the metadata unified identifier and constructs the association relationship between metadata. This embodiment provides three construction strategies: Rule-based construction, using the Drools rule engine, reads the content of the metadata unique identifier, and identifies the association relationship between multiple metadata based on rules.

[0034] Semantics-based construction, using the TF-IDF semantic similarity model, reads the content in the metadata unique identifier, and judges the mutual dependence relationship for metadata with higher similarity.

[0035] Graph reasoning-based, using the Neo4j graph database technology, combines the information of metadata entities, and infers the relationship between them.

[0036] The unique identifier is the ID card for constructing the metadata relationship. Based on the information in the ID card, the front and back relationship between each other is constructed. Taking the following actual scenario as an example: Scenario description: Engineer Xiao Wang now needs a report named [Daily Report of VIP Customer Order Amount] to view the total daily order amount of VIP customers.

[0037] The data flow is as follows: detailed data is extracted from the orders table into the warehouse ods_orders table, then summarized into another summary table vip_order_daily_amount, and finally the data analysis tool reads the data from the summary table to generate the final report report_vip_order_daily_amount.

[0038] Xiao Wang needs to establish the metadata relationships between all nodes in the above process.

[0039] Step 1: The collector agent collects metadata from each node and generates a unique identifier for each metadata element, as shown in Table 1. Table 1 Metadata Collection Table

[0040] Step 2: Intelligently associate metadata from different modalities, as shown in the following example: Based on rules: The rule is triggered to establish a relationship between pairs of metadata that are closest in time and contain the most identical attribute information. ID_SEL_987 and ID_ETL_456 are closest in time and both contain "orders," thus triggering the rule. ID_ETL_456. Based on NLP semantics: the similarity score of metadata identifiers, those with a score greater than 0.9 (out of 1.0) are related.

[0041] Graph-based reasoning: Based on the results of the first two strategies, reasoning rules are established. If xx and xxx, then xxx is inferred. The constructed reasoning rules are applied to the metadata relationships that can be identified under the above two strategies.

[0042] The three strategies work together to automatically construct a complete element map.

[0043] Furthermore, in the rule-based construction strategy, the rules include: If the timestamp difference between two metadata items is within a preset range and they share at least one identical attribute, then a relationship is determined to exist. In the semantic-based construction strategy, the semantic similarity threshold is set to 0.9; if the similarity exceeds the threshold, a dependency relationship is determined. In the graph-based reasoning strategy, the reasoning rules include: if metadata A is associated with metadata B and metadata B is associated with metadata C, then it is inferred that metadata A is associated with metadata C.

[0044] Furthermore, S2 also includes: The collector plugin includes at least one of a JDBC probe, an API parser, and an XML parser. The collector plugin is modularized through a standard interface specification, which includes an initialization method, a start collection method, a stop collection method, and a plugin check method.

[0045] Furthermore, S5 is followed by S6: Based on the relationships in the metadata repository, metadata application services are provided, including lineage analysis, end-to-end data flow monitoring, and root cause analysis. S1 also includes: The plugin repository uploads metadata collection plugin JAR packages through a graphical interface, the plugin manager adds plugins to the management list through the JAR package registration function, and the configuration center manager supports batch allocation of plugins to multiple collectors (Agents).

[0046] Furthermore, S4 also includes: The intelligent association analysis engine uses rule-based construction, semantic construction, and graph-based reasoning strategies in synergy. It applies the rule-based construction strategy to identify direct associations, then applies the semantic construction strategy to supplement similar metadata associations, and finally applies the graph-based reasoning strategy to infer indirect associations, thus constructing a complete metadata relationship graph.

[0047] According to a second aspect of the present invention, the present invention claims protection for an apparatus for collecting global metadata and automatically constructing metadata associations, comprising: One or more processors; A memory that stores one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the method for collecting global metadata and automatically constructing metadata associations.

[0048] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0049] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0050] The specific embodiments of the invention have been described in detail above, but they are only examples, and this application is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications or substitutions to the invention are also within the scope of this application. Therefore, all equivalent changes, modifications, and improvements made without departing from the spirit and principles of this application should be covered within the scope of this application.

Claims

1. A method for collecting full-domain metadata and automatically constructing metadata relationships, characterized in that, Includes the following steps: S1: Multiple metadata collection plugins are managed in a unified manner through a main control platform, wherein the main control platform includes a plugin manager, a plugin repository, a status monitoring manager, and a configuration center manager. The plugin repository stores metadata collection plugins that have been modified into components. The plugin manager registers and manages the metadata collection plugins. The configuration center manager assigns the selected metadata collection plugins to the collector agents. The status monitoring manager monitors the running status of the collector agents in real time. S2: The collector agent loads and executes the metadata collection plugin to collect metadata from multiple links in the entire data flow chain. The collector agent includes a configuration client, a heartbeat client, a collection framework, a collector plugin, and a data caching module. The configuration client configures the communication information between the collector agent and the main control platform. The heartbeat client periodically reports the running status of the collector agent. The collection framework loads and executes the collector plugin. The data caching module temporarily stores the collected metadata. S3: Generate a unified identifier for the collected metadata, wherein the unified identifier is composed of key information of the metadata, including timestamp, host ID, database name, field name, business type, operation description, operation user and permissions; S4: The intelligent association analysis engine automatically constructs the association relationship between metadata based on the unified identifier, wherein the intelligent association analysis engine analyzes the metadata identifier content using at least one strategy among rule-based construction, semantic construction, and graph reasoning, and outputs the metadata association relationship; S5: Store the constructed metadata relationships in a metadata repository to support metadata applications.

2. The method for collecting full-domain metadata and automatically constructing metadata association relationships according to claim 1, characterized in that, S1 also includes: The main control platform reads the metadata collection plugin JAR package from the plugin repository through the plugin manager and completes the plugin registration. The plugin manager displays the plugin list through a graphical interface. The configuration center manager selects the metadata collection plugin based on the plugin list and assigns it to the specified collector agent. The status monitoring manager collects the running logs and performance indicators of the collector agent through a heartbeat mechanism.

3. The method for collecting full-domain metadata and automatically constructing metadata association relationships according to claim 1, characterized in that, S2 also includes: The collector agent receives collection instructions from the main control platform through a configured client. The collection instructions include the target data source address, authentication information, and collection parameters. The heartbeat client sends running status information to the main control platform at a preset frequency. The collection framework dynamically loads the collector plugin JAR package through a class loader and calls the plugin execution function to start metadata collection. The data caching module temporarily stores the collected metadata in a buffer and triggers the unified metadata identifier generation process.

4. The method for collecting full-domain metadata and automatically constructing metadata association relationships according to claim 1, characterized in that, In step S3, the specific steps for generating a unified identifier include: S31: Receive metadata from the collector agent through the metadata information receiving module; S32: Read key information from metadata, including timestamp, host ID, database name, field name, business type, operation description, operation user and permissions; S33: Concatenate the key information into a string in a preset order to form a unique metadata identifier; S34: Store the unique identifier and metadata together in the database.

5. The method for collecting full-domain metadata and automatically constructing metadata association relationships according to claim 1, characterized in that, S4 also includes: When the intelligent association analysis engine adopts a rule-based construction strategy, it uses the Drools rule engine to parse the unique identifier content of metadata and identifies the association relationship of metadata based on the time proximity and attribute overlap rules. When adopting a semantic-based construction strategy, the TF-IDF model is used to calculate the semantic similarity of metadata identifier content, and metadata dependencies are determined based on similarity thresholds. When adopting a graph-based reasoning strategy, the Neo4j graph database is used to store metadata entities and relationships, and reasoning rules are applied to infer metadata relationships that are not directly related.

6. The method for collecting full-domain metadata and automatically constructing metadata association relationships according to claim 5, characterized in that, The rules in the rule-based construction strategy include: If the timestamp difference between two metadata items is within a preset range and they share at least one identical attribute, then a relationship is determined to exist. In the semantic-based construction strategy, the semantic similarity threshold is set to 0.9; if the similarity exceeds the threshold, a dependency relationship is determined. In the graph-based reasoning strategy, the reasoning rules include: if metadata A is associated with metadata B and metadata B is associated with metadata C, then it is inferred that metadata A is associated with metadata C.

7. The method for collecting full-domain metadata and automatically constructing metadata association relationships according to claim 1, characterized in that, S2 also includes: The collector plugin includes at least one of a JDBC probe, an API parser, and an XML parser. The collector plugin is modularized through a standard interface specification, which includes an initialization method, a start collection method, a stop collection method, and a plugin check method.

8. The method for collecting full-domain metadata and automatically constructing metadata association relationships according to claim 1, characterized in that, S5 is followed by S6: Based on the relationships in the metadata repository, metadata application services are provided, including lineage analysis, end-to-end data flow monitoring, and root cause analysis. S1 also includes: The plugin repository uploads metadata collection plugin JAR packages through a graphical interface, the plugin manager adds plugins to the management list through the JAR package registration function, and the configuration center manager supports batch allocation of plugins to multiple collectors (Agents).

9. The method for collecting full-domain metadata and automatically constructing metadata association relationships according to claim 1, characterized in that, S4 also includes: The intelligent association analysis engine uses rule-based construction, semantic construction, and graph-based reasoning strategies in synergy. It applies the rule-based construction strategy to identify direct associations, then applies the semantic construction strategy to supplement similar metadata associations, and finally applies the graph-based reasoning strategy to infer indirect associations, thus constructing a complete metadata relationship graph.

10. A device for collecting full-domain metadata and automatically constructing metadata associations, characterized in that, include: One or more processors; A memory having stored one or more programs, which, when executed by one or more processors, cause the one or more processors to implement a method for global metadata collection and automatic construction of metadata associations according to any one of claims 1 to 9.

Images