Method and device for global metadata collection and automatic construction of metadata correlation relationship

Abstract

The present application relates to the technical field of big data, and specifically relates to a method and device for global metadata acquisition and automatic construction of metadata correlation relationship, a main control platform and an acquisition agent are designed, different types of metadata acquisition agents are integrated through a new technical architecture, so that the acquisition of all types of global metadata can be realized through only one technical architecture; in solving the problem of constructing the full-link relationship of metadata, the technology of constructing the full-link relationship of each type of metadata is adopted, based on the metadata unified identification generation technology and the cross-modal relationship construction technology, the correlation relationship of each type of metadata scattered in the whole chain of data circulation can be intelligently and automatically established; a metadata acquisition technology plug-in implementation method is provided, including a plug-in standard interface, centralized configuration management and unified state monitoring, and the major engineering problem of seamlessly integrating multiple heterogeneous acquisition technologies into one platform is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and in particular to a method and device for global metadata acquisition and automatic construction of metadata correlation. BACKGROUND

[0002] In the field of public data authorized operation, data circulation involves multiple links and participating subjects, and its security risks have the characteristics of concealment, transmission and complexity. In order to effectively control the security risks of data in open use, it is necessary to monitor the data flow process in a fine-grained manner from data acquisition, processing, warehousing, data shelving to catalog, data application, data approval, data authorization, data product development, and data authorized supply. To achieve fine-grained monitoring of the whole chain, the metadata information generated by the whole chain must be investigated and collected, and analysis must be performed based on the collected metadata. At present, the industry's solution to the whole-link metadata acquisition for public data authorized operation mainly has the following difficulties:

[0003] The collection of metadata mainly focuses on the metadata of key data flow processes in a single link, such as database operation monitoring log metadata, data security classification metadata, and data call metadata, which is difficult to support the analysis and monitoring of the whole-link data flow process risks.

[0004] The collection of whole-link metadata often requires modification of business systems, and each business system involved in the data flow process needs to embed related metadata collection technologies, which has high modification and implementation costs, is difficult to implement, and may affect the stability and performance of the business system, making it difficult to adapt to the complex environment of the public data authorized operation platform.

[0005] Different system scenarios and different modal data require different technologies for metadata collection. For example, for database or file metadata, SQL parser or Sidercar technology is needed for collection, and for information generated in the approval process, Bytecode enhancement technology is needed for metadata collection. A single metadata collection technology cannot meet the needs of collecting all types of metadata in the whole data circulation process.

[0006] A complete business data whole chain includes data acquisition, processing, warehousing, data shelving to catalog, data application, data approval, data authorization, data product development, and data authorized supply. The metadata collected in each process is fragmented, and the correlation between them needs to be established by relying on manual work. SUMMARY

[0007] According to the first aspect of the present application, the present application claims a method for global metadata acquisition and automatic construction of metadata correlation, comprising the following steps:

[0008] S1: Multiple metadata collection plugins are managed in a unified manner through a main control platform, wherein the main control platform includes a plugin manager, a plugin repository, a status monitoring manager, and a configuration center manager. The plugin repository stores metadata collection plugins that have been modified into components. The plugin manager registers and manages the metadata collection plugins. The configuration center manager assigns the selected metadata collection plugins to the collector agents. The status monitoring manager monitors the running status of the collector agents in real time.

[0009] S2: The collector agent loads and executes the metadata collection plugin to collect metadata from multiple links in the entire data flow chain. The collector agent includes a configuration client, a heartbeat client, a collection framework, a collector plugin, and a data caching module. The configuration client configures the communication information between the collector agent and the main control platform. The heartbeat client periodically reports the running status of the collector agent. The collection framework loads and executes the collector plugin. The data caching module temporarily stores the collected metadata.

[0010] S3: Generate a unified identifier for the collected metadata, wherein the unified identifier is composed of key information of the metadata, including timestamp, host ID, database name, field name, business type, operation description, operation user and permissions;

[0011] S4: The intelligent association analysis engine automatically constructs the association relationship between metadata based on the unified identifier, wherein the intelligent association analysis engine uses at least one strategy among rule-based construction, semantic construction and graph reasoning to analyze the metadata identifier content and output the metadata association relationship;

[0012] S5: Store the constructed metadata relationships in a metadata repository to support metadata applications.

[0013] Furthermore, S1 also includes:

[0014] The main control platform reads the metadata collection plugin JAR package from the plugin repository through the plugin manager and completes the plugin registration. The plugin manager displays the plugin list through a graphical interface. The configuration center manager selects the metadata collection plugin based on the plugin list and assigns it to the specified collector agent. The status monitoring manager collects the running logs and performance indicators of the collector agent through a heartbeat mechanism.

[0015] Furthermore, S2 also includes:

[0016] The collector agent receives collection instructions from the main control platform through a configured client. The collection instructions include the target data source address, authentication information, and collection parameters. The heartbeat client sends running status information to the main control platform at a preset frequency. The collection framework dynamically loads the collector plugin JAR package through a class loader and calls the plugin execution function to start metadata collection. The data caching module temporarily stores the collected metadata in a buffer and triggers the unified metadata identifier generation process.

[0017] Furthermore, in step S3, the specific steps for generating a unified identifier include:

[0018] S31: Receive metadata from the collector agent through the metadata information receiving module;

[0019] S32: Read key information from metadata, including timestamp, host ID, database name, field name, business type, operation description, operation user and permissions;

[0020] S33: Concatenate the key information into a string in a preset order to form a unique metadata identifier;

[0021] S34: Store the unique identifier and metadata together in the database.

[0022] Furthermore, S4 also includes:

[0023] When the intelligent association analysis engine adopts a rule-based construction strategy, it uses the Drools rule engine to parse the unique identifier content of metadata and identifies the association relationship of metadata based on the time proximity and attribute overlap rules.

[0024] When adopting a semantic-based construction strategy, the TF-IDF model is used to calculate the semantic similarity of metadata identifier content, and metadata dependencies are determined based on similarity thresholds.

[0025] When adopting a graph-based reasoning strategy, the Neo4j graph database is used to store metadata entities and relationships, and reasoning rules are applied to infer metadata relationships that are not directly related.

[0026] Furthermore, in the rule-based construction strategy, the rules include:

[0027] If the timestamp difference between two metadata items is within a preset range and they share at least one identical attribute, then a relationship is determined to exist.

[0028] In the semantic-based construction strategy, the semantic similarity threshold is set to 0.9; if the similarity exceeds the threshold, a dependency relationship is determined.

[0029] In the graph-based reasoning strategy, the reasoning rules include: if metadata A is associated with metadata B and metadata B is associated with metadata C, then it is inferred that metadata A is associated with metadata C.

[0030] Furthermore, S2 also includes:

[0031] The collector plugin includes at least one of a JDBC probe, an API parser, and an XML parser. The collector plugin is modularized through a standard interface specification, which includes an initialization method, a start collection method, a stop collection method, and a plugin check method.

[0032] Furthermore, S5 is followed by S6:

[0033] Based on the relationships in the metadata repository, metadata application services are provided, including lineage analysis, end-to-end data flow monitoring, and root cause analysis.

[0034] S1 also includes:

[0035] The plugin repository uploads metadata collection plugin JAR packages through a graphical interface, the plugin manager adds plugins to the management list through the JAR package registration function, and the configuration center manager supports batch allocation of plugins to multiple collectors (Agents).

[0036] Furthermore, S4 also includes:

[0037] The intelligent association analysis engine uses rule-based construction, semantic construction, and graph-based reasoning strategies in synergy. It applies the rule-based construction strategy to identify direct associations, then applies the semantic construction strategy to supplement similar metadata associations, and finally applies the graph-based reasoning strategy to infer indirect associations, thus constructing a complete metadata relationship graph.

[0038] According to a second aspect of the present invention, the present invention claims protection for an apparatus for collecting global metadata and automatically constructing metadata associations, comprising:

[0039] One or more processors;

[0040] A memory that stores one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the method for collecting global metadata and automatically constructing metadata associations.

[0041] This invention relates to the field of big data technology, specifically to a method and apparatus for collecting full-domain metadata and automatically constructing metadata relationships. It designs a main control platform and a collector agent, integrating different types of metadata collectors through a new technical architecture, enabling the collection of all types of metadata across the entire domain with only one technical architecture. To address the challenge of constructing full-link relationships for metadata, it employs a technology for building full-link relationships for various types of metadata. Based on unified metadata identifier generation technology and cross-modal relationship construction technology, it enables the intelligent and automatic establishment of relationships between various types of metadata scattered throughout the data flow chain. It provides a plug-in implementation method for metadata collection technology, including plug-in standard interfaces, centralized configuration management, and unified status monitoring, solving the major engineering challenge of seamlessly integrating multiple heterogeneous collection technologies into a single platform. Attached Figure Description

[0042] Figure 1 The flowchart illustrates a method for collecting global metadata and automatically constructing metadata associations as claimed in an embodiment of the present invention.

[0043] Figure 2 This is a second flowchart of a method for collecting global metadata and automatically constructing metadata associations, as claimed in an embodiment of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0045] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0046] According to a first embodiment of the present invention, the present invention claims protection for a method for collecting global metadata and automatically constructing metadata association relationships, referring to... Figure 1 This includes the following steps:

[0047] S1: Multiple metadata collection plugins are managed in a unified manner through a main control platform, wherein the main control platform includes a plugin manager, a plugin repository, a status monitoring manager, and a configuration center manager. The plugin repository stores metadata collection plugins that have been modified into components. The plugin manager registers and manages the metadata collection plugins. The configuration center manager assigns the selected metadata collection plugins to the collector agents. The status monitoring manager monitors the running status of the collector agents in real time.

[0048] S2: The collector agent loads and executes the metadata collection plugin to collect metadata from multiple links in the entire data flow chain. The collector agent includes a configuration client, a heartbeat client, a collection framework, a collector plugin, and a data caching module. The configuration client configures the communication information between the collector agent and the main control platform. The heartbeat client periodically reports the running status of the collector agent. The collection framework loads and executes the collector plugin. The data caching module temporarily stores the collected metadata.

[0049] S3: Generate a unified identifier for the collected metadata, wherein the unified identifier is composed of key information of the metadata, including timestamp, host ID, database name, field name, business type, operation description, operation user and permissions;

[0050] S4: The intelligent association analysis engine automatically constructs the association relationship between metadata based on the unified identifier, wherein the intelligent association analysis engine uses at least one strategy among rule-based construction, semantic construction and graph reasoning to analyze the metadata identifier content and output the metadata association relationship;

[0051] S5: Store the constructed metadata relationships in a metadata repository to support metadata applications.

[0052] Furthermore, S1 also includes:

[0053] The main control platform reads the metadata collection plugin JAR package from the plugin repository through the plugin manager and completes the plugin registration. The plugin manager displays the plugin list through a graphical interface. The configuration center manager selects the metadata collection plugin based on the plugin list and assigns it to the specified collector agent. The status monitoring manager collects the running logs and performance indicators of the collector agent through a heartbeat mechanism.

[0054] The overall implementation logic of the main control platform adopts metadata collection technology, which is currently the mainstream metadata collection technology in the industry, including JDBC probes, APIs, XML parsers, etc.

[0055] In accordance with standards and specifications, this embodiment provides a set of standard plugin interface specifications, which cover component initialization methods, methods for starting metadata collection, methods for stopping metadata collection, and plugin inspection methods; and provides standard modification reference templates for each collection technology in the form of Java classes;

[0056] Regarding the modular transformation of data acquisition technology, based on the provided standards and specifications, the data acquisition technology is modularized.

[0057] The modified components are all output as JAR files.

[0058] For the plugin repository, the modified component JAR files are uploaded to the plugin repository through the main control platform's component repository interface;

[0059] The plugin manager reads the JAR files in the component repository and registers them through the JAR file registration function. After successful registration, the component information can be viewed in the interface list.

[0060] Configure the center manager. Based on the components successfully registered in the plugin manager, the center manager selects the required components and assigns them to the data collector agent.

[0061] Collector Agent. The collector agent performs metadata collection tasks based on the assigned collection components;

[0062] The Status Monitoring Manager monitors and collects the running status and log information of the Collector Agent in real time.

[0063] Furthermore, S2 also includes:

[0064] The collector agent receives collection instructions from the main control platform through a configured client. The collection instructions include the target data source address, authentication information, and collection parameters. The heartbeat client sends running status information to the main control platform at a preset frequency. The collection framework dynamically loads the collector plugin JAR package through a class loader and calls the plugin execution function to start metadata collection. The data caching module temporarily stores the collected metadata in a buffer and triggers the unified metadata identifier generation process.

[0065] The implementation logic of the collector agent is for the configuration client, which configures the communication address, protocol, and JAR path information between the collector agent and the main control platform;

[0066] Configure the collector agent and the monitoring frequency of the main console collector for the heartbeat client;

[0067] The data collection framework includes two modules: component loading and component execution. The background of the data collection framework automatically loads the data collection JAR package plugin based on the configuration information of the configured client and the heartbeat client.

[0068] The data acquisition component executes the data acquisition JAR package plugin;

[0069] Data caching is used to store the metadata collected by the collector in the cache area;

[0070] Generate a unified identifier for metadata, creating a unique identifier for the metadata collected by the collector.

[0071] In a real-world scenario, Xiao Wang needs to collect metadata about all operations performed on the production order table in a MySQL database running on 192.168.1.100. This includes collecting data on who performed the operation, when, and what operation was performed. Let's reconstruct the business scenario based on the above sequence:

[0072] Xiao Wang first accessed the client interface of the data collector on the main control platform, selecting the Agent deployed on the host 192.168.1.100, and then selecting the mysql-jdbc-probe plugin for the Agent. In the mysql-jdbc-probe plugin editing interface selected by the data collector Agent, he configured the target address information for data collection, as shown in the example below:

[0073] {

[0074] "jdbc.url": "jdbc:mysql: / / 192.168.1.100:3306 / ecommerce_db?useSSL=false",

[0075] "jdbc.username": "metamonitor",

[0076] "jdbc.password": " ",

[0077] "monitored_tables": "orders_table",

[0078] "collector_id": "collector_mysql_01"

[0079] }

[0080] After configuration, click "Issue" to complete the issuance of the data collection command.

[0081] b) Simultaneously, Xiao Wang also needs to configure the heartbeat client to periodically monitor the Agent's running status. Here, Xiao Wang has set it to monitor every 5 seconds. An example of the configuration information is as follows:

[0082] {

[0083] "agent_id": "agent_host_192.168.1.100",

[0084] "timestamp": 5s,

[0085] "status": "HEALTHY",

[0086] "running_plugins": ["mysql-jdbc-probe"],

[0087] "system_metrics": {

[0088] "cpu_usage": "12.5%",

[0089] "memory_usage": "45.8%",

[0090] "queue_size": 5 / / Current number of data buffers in the queue

[0091] }

[0092] }

[0093] The data collector agent client converts the issued data collection instructions into a format that the data collection framework can recognize, and sends a request to the data collection framework. After receiving the instructions, the data collection framework loads the data collection component using Java's class loader and calls the component execution function to execute the data collection component. An example of the data collection instruction request format is as follows:

[0094] Agent_id:agent_host_192.168.1.100

[0095] Config:{"jdbc.url": "jdbc:mysql: / / 192.168.1.100:3306 / ecommerce_db?useSSL=false",

[0096] "jdbc.username": "metamonitor",

[0097] "jdbc.password": " ",

[0098] "monitored_tables": "orders_table",

[0099] "collector_id": "collector_mysql_01"

[0100] }

[0101] This feature tracks the execution process of components, including corresponding background processes and tasks. It's used to monitor whether components are executing correctly.

[0102] Data caching and data collection using the mysql-jdbc-probe component scan the MySQL database `orders_table` on server 192.168.1.100 for all operations, such as a user query generating an operation record: SELECT The plugin will intercept the SQL statement `FROMorders WHERE user_id = 12345;` and instantly generate a raw metadata record, as shown in the example below:

[0103] {

[0104] SQL: SELECT FROM orders WHERE user_id = 12345",

[0105] "database": "ecommerce_db",

[0106] "table": "orders",

[0107] "operation": "SELECT",

[0108] "source_ip": "192.168.1.50", / / Application server IP

[0109] "user": "app_user",

[0110] "timestamp": 1722504620000,

[0111] "execution_time_ms": 15

[0112] }

[0113] The data caching unit temporarily writes the scanned metadata information into the cache database for storage, and at the same time calls the metadata information receiving interface in the metadata unified generation and identification unit to send metadata information to the metadata unified generation and identification unit.

[0114] Furthermore, referring to Figure 2 In step S3, the specific steps for generating a unified identifier include:

[0115] S31: Receive metadata from the collector agent through the metadata information receiving module;

[0116] S32: Read key information from metadata, including timestamp, host ID, database name, field name, business type, operation description, operation user and permissions;

[0117] S33: Concatenate the key information into a string in a preset order to form a unique metadata identifier;

[0118] S34: Store the unique identifier and metadata together in the database.

[0119] Metadata unified identifier generation generates a unique identifier for each piece of metadata collected by the collector agent. To facilitate the subsequent construction of metadata relationships based on this identifier, we designed the following components for the metadata identifier: timestamp, host ID, database name, field name, business type, operation description, operating user, and permissions. Example: 20250829103129-127.0.0.10-racle001-name.phone.xx-approval-download-admin-query;

[0120] When receiving metadata information, the metadata collected by the collector Agent will be received into the metadata unified identifier generation framework through the metadata information receiving module.

[0121] When reading key information from metadata, key information is retrieved, including timestamp, host ID, database name, field name, business type, operation description, operation user, and permissions.

[0122] When concatenating metadata key information, the read key information is concatenated into a complete string. Example:

[0123] 20250829103129-127.0.0.10-racle001-name.phone.xx-approval-download-admin-query.

[0124] The concatenated string is used as a unique identifier for metadata and stored in the database;

[0125] Continuing with the above embodiment, after receiving metadata information from the metadata cache unit, the metadata information receiving unit extracts the key information from the metadata and concatenates the key information, as shown in the following example:

[0126] Received information:

[0127] {

[0128] "sql": "SELECT FROM orders WHERE user_id = 12345",

[0129] "database": "ecommerce_db",

[0130] "table": "orders",

[0131] "operation": "SELECT",

[0132] "source_ip": "192.168.1.50", / / IP of the application server

[0133] "user": "app_user",

[0134] "timestamp": 1722504620000,

[0135] "execution_time_ms": 15

[0136] }

[0137] Metadata identifier generation:

[0138] Number: {Time of action occurrence + User + Address + Action + Database + Object table}

[0139] ID_SEL_987: {1722504620000 + Xiao Wang + 192.168.1.50 + select + ecommerce_db + orders}.

[0140] Furthermore, in S4, it further includes:

[0141] When the intelligent association analysis engine adopts a rule-based construction strategy, it uses the Drools rule engine to parse the content of the metadata unique identifier and identify the metadata association relationship according to the rules of time proximity and attribute overlap;

[0142] When adopting a semantics-based construction strategy, it uses the TF-IDF model to calculate the semantic similarity of the metadata identifier content and judges the metadata dependency relationship based on the similarity threshold;

[0143] When adopting a graph-based reasoning strategy, the Neo4j graph database is used to store metadata entities and relationships, and reasoning rules are applied to infer metadata relationships that are not directly related.

[0144] In this embodiment, the intelligent association analysis engine analyzes the relationships between characters based on the character information in the unified metadata identifier, and constructs association relationships between metadata. This embodiment provides three construction strategies:

[0145] Based on rule construction, the Drools rule engine is used to read the uniquely identified metadata content and identify the relationship between multiple metadata based on the rules.

[0146] Based on semantic construction, the TF-IDF semantic similarity model is used to read the content in the unique identifier of metadata and determine the dependency relationship between metadata with high similarity.

[0147] Based on graph reasoning, Neo4j graph database technology is used to infer the relationships between entities by combining metadata information.

[0148] A unique identifier is like an ID card for building metadata relationships. Based on the information in the ID card, the relationships between elements are constructed. Consider the following practical scenario:

[0149] Scenario Description: Engineer Xiao Wang needs a report titled "Daily Report on Order Amount of Major Customers" to view the total order amount of VIP customers each day.

[0150] The data flow is as follows: detailed data is extracted from the orders table into the warehouse ods_orders table, then summarized into another summary table vip_order_daily_amount, and finally the data analysis tool reads the data from the summary table to generate the final report report_vip_order_daily_amount.

[0151] Xiao Wang needs to establish the metadata relationships between all nodes in the above process.

[0152] Step 1: The collector agent collects metadata from each node and generates a unique identifier for each metadata element, as shown in Table 1.

[0153] Table 1 Metadata Collection Table

[0154]

[0155] Step 2: Intelligently associate metadata from different modalities, as shown in the following example:

[0156] Based on rules: The rule is triggered to establish a relationship between pairs of metadata that are closest in time and contain the most identical attribute information. ID_SEL_987 and ID_ETL_456 are closest in time and both contain "orders," thus triggering the rule. ID_ETL_456.

[0157] Based on NLP semantics: the similarity score of metadata identifiers, those with a score greater than 0.9 (out of 1.0) are related.

[0158] Graph-based reasoning: Based on the results of the first two strategies, reasoning rules are established. If xx and xxx, then xxx is inferred. The constructed reasoning rules are applied to the metadata relationships that can be identified under the above two strategies.

[0159] The three strategies work together to automatically construct a complete element map.

[0160] Furthermore, in the rule-based construction strategy, the rules include:

[0161] If the timestamp difference between two metadata items is within a preset range and they share at least one identical attribute, then a relationship is determined to exist.

[0162] In the semantic-based construction strategy, the semantic similarity threshold is set to 0.9; if the similarity exceeds the threshold, a dependency relationship is determined.

[0163] In the graph-based reasoning strategy, the reasoning rules include: if metadata A is associated with metadata B and metadata B is associated with metadata C, then it is inferred that metadata A is associated with metadata C.

[0164] Furthermore, S2 also includes:

[0165] The collector plugin includes at least one of a JDBC probe, an API parser, and an XML parser. The collector plugin is modularized through a standard interface specification, which includes an initialization method, a start collection method, a stop collection method, and a plugin check method.

[0166] Furthermore, S5 is followed by S6:

[0167] Based on the relationships in the metadata repository, metadata application services are provided, including lineage analysis, end-to-end data flow monitoring, and root cause analysis.

[0168] S1 also includes:

[0169] The plugin repository uploads metadata collection plugin JAR packages through a graphical interface, the plugin manager adds plugins to the management list through the JAR package registration function, and the configuration center manager supports batch allocation of plugins to multiple collectors (Agents).

[0170] Furthermore, S4 also includes:

[0171] The intelligent association analysis engine uses rule-based construction, semantic construction, and graph-based reasoning strategies in synergy. It applies the rule-based construction strategy to identify direct associations, then applies the semantic construction strategy to supplement similar metadata associations, and finally applies the graph-based reasoning strategy to infer indirect associations, thus constructing a complete metadata relationship graph.

[0172] According to a second aspect of the present invention, the present invention claims protection for an apparatus for collecting global metadata and automatically constructing metadata associations, comprising:

[0173] One or more processors;

[0174] A memory that stores one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the method for collecting global metadata and automatically constructing metadata associations.

[0175] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0176] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0177] The specific embodiments of the invention have been described in detail above, but they are only examples, and this application is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications or substitutions to the invention are also within the scope of this application. Therefore, all equivalent changes, modifications, and improvements made without departing from the spirit and principles of this application should be covered within the scope of this application.

Claims

1. A method for collecting full-domain metadata and automatically constructing metadata relationships, characterized in that, Includes the following steps: S1: Multiple metadata collection plugins are managed in a unified manner through a main control platform, wherein the main control platform includes a plugin manager, a plugin repository, a status monitoring manager, and a configuration center manager. The plugin repository stores metadata collection plugins that have been modified into components. The plugin manager registers and manages the metadata collection plugins. The configuration center manager assigns the selected metadata collection plugins to the collector agents. The status monitoring manager monitors the running status of the collector agents in real time. S2: The collector agent loads and executes the metadata collection plugin to collect metadata from multiple links in the entire data flow chain. The collector agent includes a configuration client, a heartbeat client, a collection framework, a collector plugin, and a data caching module. The configuration client configures the communication information between the collector agent and the main control platform. The heartbeat client periodically reports the running status of the collector agent. The collection framework loads and executes the collector plugin. The data caching module temporarily stores the collected metadata. S3: Generate a unified identifier for the collected metadata, wherein the unified identifier is composed of key information of the metadata, including timestamp, host ID, database name, field name, business type, operation description, operation user and permissions; S4: The intelligent association analysis engine automatically constructs the association relationship between metadata based on the unified identifier, wherein the intelligent association analysis engine uses at least one strategy among rule-based construction, semantic construction and graph reasoning to analyze the metadata identifier content and output the metadata association relationship; S5: Store the constructed metadata relationships in the metadata repository to support metadata applications; S4 also includes: When the intelligent association analysis engine adopts a rule-based construction strategy, it uses the Drools rule engine to parse the unique identifier content of metadata and identifies the association relationship of metadata based on the time proximity and attribute overlap rules. When adopting a semantic-based construction strategy, the TF-IDF model is used to calculate the semantic similarity of metadata identifier content, and metadata dependencies are determined based on similarity thresholds. When adopting a graph-based reasoning strategy, the Neo4j graph database is used to store metadata entities and relationships, and reasoning rules are applied to infer metadata relationships that are not directly related. The rules in the rule-based construction strategy include: If the timestamp difference between two metadata items is within a preset range and they share at least one identical attribute, then a relationship is determined to exist. In the semantic-based construction strategy, the semantic similarity threshold is set to 0.9; if the similarity exceeds the threshold, a dependency relationship is determined. In the graph-based reasoning strategy, the reasoning rules include: if metadata A is associated with metadata B and metadata B is associated with metadata C, then it is inferred that metadata A is associated with metadata C.

2. The method for collecting full-domain metadata and automatically constructing metadata association relationships according to claim 1, characterized in that, S1 also includes: The main control platform reads the metadata collection plugin JAR package from the plugin repository through the plugin manager and completes the plugin registration. The plugin manager displays the plugin list through a graphical interface. The configuration center manager selects the metadata collection plugin based on the plugin list and assigns it to the specified collector agent. The status monitoring manager collects the running logs and performance indicators of the collector agent through a heartbeat mechanism.

3. The method for collecting full-domain metadata and automatically constructing metadata association relationships according to claim 1, characterized in that, S2 also includes: The collector agent receives collection instructions from the main control platform through a configured client. The collection instructions include the target data source address, authentication information, and collection parameters. The heartbeat client sends running status information to the main control platform at a preset frequency. The collection framework dynamically loads the collector plugin JAR package through a class loader and calls the plugin execution function to start metadata collection. The data caching module temporarily stores the collected metadata in a buffer and triggers the unified metadata identifier generation process.

4. The method for collecting full-domain metadata and automatically constructing metadata association relationships according to claim 1, characterized in that, In step S3, the specific steps for generating a unified identifier include: S31: Receive metadata from the collector agent through the metadata information receiving module; S32: Read key information from metadata, including timestamp, host ID, database name, field name, business type, operation description, operation user and permissions; S33: Concatenate the key information into a string in a preset order to form a unique metadata identifier; S34: Store the unique identifier and metadata together in the database.

5. The method for collecting full-domain metadata and automatically constructing metadata association relationships according to claim 1, characterized in that, S2 also includes: The collector plugin includes at least one of a JDBC probe, an API parser, and an XML parser. The collector plugin is modularized through a standard interface specification, which includes an initialization method, a start collection method, a stop collection method, and a plugin check method.

6. The method for collecting full-domain metadata and automatically constructing metadata association relationships according to claim 1, characterized in that, S5 is followed by S6: Based on the relationships in the metadata repository, metadata application services are provided, including lineage analysis, end-to-end data flow monitoring, and root cause analysis. S1 also includes: The plugin repository uploads metadata collection plugin JAR packages through a graphical interface, the plugin manager adds plugins to the management list through the JAR package registration function, and the configuration center manager supports batch allocation of plugins to multiple collectors (Agents).

7. The method for collecting full-domain metadata and automatically constructing metadata association relationships according to claim 1, characterized in that, S4 also includes: The intelligent association analysis engine uses rule-based construction, semantic construction, and graph-based reasoning strategies in synergy. It applies the rule-based construction strategy to identify direct associations, then applies the semantic construction strategy to supplement similar metadata associations, and finally applies the graph-based reasoning strategy to infer indirect associations, thus constructing a complete metadata relationship graph.

8. A device for collecting full-domain metadata and automatically constructing metadata associations, characterized in that, include: One or more processors; A memory having stored one or more programs, which, when executed by one or more processors, cause the one or more processors to implement a method for global metadata collection and automatic construction of metadata associations according to any one of claims 1 to 7.

Images