A Method and System for Intelligent Classification and Association of Metadata Based on Knowledge Graphs

Abstract

This invention provides a method and system for intelligent classification and association of metadata based on knowledge graphs. The method includes: acquiring metadata from different business systems of an enterprise using a multi-source metadata access layer; identifying the semantic relationships between metadata based on natural language processing technology; performing semantic alignment processing on the metadata based on an improved graph attention network model to generate a unified semantic mapping table; automatically constructing a metadata knowledge graph using knowledge graph-enhanced retrieval technology; generating descriptive semantic tags for each metadata based on a generative pre-trained language model and semantic information in the metadata knowledge graph; receiving natural language query requests and converting them into knowledge graph query statements using a semantic parsing model; performing semantic retrieval in the metadata knowledge graph using the knowledge graph query statements; and obtaining cross-system associated query results by combining an intelligent recommendation algorithm for association paths. This invention achieves efficient and intelligent management and deep semantic association of metadata.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for intelligent classification and association of metadata based on knowledge graphs. Background Technology

[0002] Existing metadata management system solutions for multiple data sources primarily support metadata access from various data sources (such as Oracle, Greenplum, Hive, etc.), load data source configuration information through the Metacat component, utilize MySQL to store structured metadata, and Elasticsearch to provide fast metadata search capabilities, while simultaneously synchronizing data between the two to ensure consistency. This system introduces the Lineage module, using the Neo4j graph database to establish storage and analysis capabilities for metadata lineage and complex relationships. Leveraging a RESTful API, the system provides standardized interfaces for reading, querying, and managing metadata. Finally, through a user interface (UI), it offers users functions such as data source management, access configuration, metadata search, and visual browsing, meeting the needs for efficient, dynamic, and scalable metadata management. Taking a keyword-matching-based metadata management system as an example, its process is: metadata input → keyword extraction → rule matching and classification → storage in the database.

[0003] Current technology also has the following prominent limitations: (1) Low classification accuracy Rule-based classification methods heavily rely on human experience and subjective definitions, making it difficult to cover complex semantics. As data continues to grow and change, updating pre-defined rules is complex and lags significantly, leading to distorted and inaccurate classification results, thus hindering subsequent data analysis. Therefore, improving classification accuracy and reducing reliance on human intervention are pressing issues that current technology needs to address.

[0004] (2) The depth of metadata association is obviously insufficient. Establishing metadata relationships using static rules presents significant limitations when dealing with massive, diverse, and time-sensitive data scenarios. This static association method struggles to flexibly adapt to data changes, fails to deeply mine the complex semantic meaning of metadata, and also restricts its effective scalability to large-scale data processing environments. Developing a more dynamic and flexible association analysis method has become one of the important research topics in the field of metadata management.

[0005] (3) The system has poor real-time performance. Current classification and association models generally lack adaptive online update mechanisms and rely too much on manual intervention for updates, which cannot meet the high timeliness requirements of metadata for real-time analysis and decision-making. Summary of the Invention

[0006] In view of this, the present invention provides a method and system for intelligent classification and association of metadata based on knowledge graphs to solve the above problems.

[0007] This invention provides a knowledge graph-based intelligent classification and association method for metadata, comprising: acquiring metadata from different business systems of an enterprise using a multi-source metadata access layer, and identifying the semantic relationships of each metadata based on natural language processing technology; performing semantic alignment processing on the metadata based on an improved graph attention network model to generate a unified semantic mapping table, wherein the improved graph attention network model integrates node attention mechanism and semantic attention mechanism, and adjusts the alignment confidence of metadata from different data sources through a dynamic weight adaptive module; automatically constructing a metadata knowledge graph based on the unified semantic mapping table using knowledge graph enhanced retrieval technology; generating descriptive semantic tags for each metadata based on a generative pre-trained language model and in combination with semantic information in the metadata knowledge graph; iteratively optimizing the generated tags through reinforcement learning methods and in combination with user feedback; receiving natural language query requests and converting the query requests into knowledge graph query statements using a semantic parsing model; performing semantic retrieval in the metadata knowledge graph using the knowledge graph query statements, and obtaining cross-system associated query results by combining an intelligent recommendation algorithm for association paths.

[0008] In another implementation of the present invention, the metadata knowledge graph construction process includes knowledge extraction, knowledge fusion, and knowledge representation.

[0009] In another implementation of the present invention, the method further includes: using a few-shot learning mechanism to quickly integrate the metadata of the new data source into the metadata knowledge graph, thereby realizing the dynamic updating and expansion of the knowledge graph.

[0010] In another implementation of the present invention, the node attention mechanism adopts the self-attention mechanism of GAT to calculate the structural correlation weight between the target node and its neighboring nodes, as shown in the following formula: e ij = (W h i W h j ) in, h i , h j Let W be the node feature vector, and W be the feature transformation matrix. The attention scoring function is used; the node-level attention weights are obtained by normalization using the Softmax function, and the features of neighboring nodes are aggregated.

[0011] in, Attention weights at the node level.

[0012] In another implementation of the present invention, the semantic attention mechanism adds a semantic similarity calculation branch. Based on the semantic vector features of the input layer, it calculates the semantic fit between the target node and its neighboring nodes, using cosine similarity as the semantic scoring function.

[0013] in, , This is the semantic vector of the node.

[0014] In another implementation of the present invention, the method further includes: designing dynamic fusion coefficients. Attention fusion is achieved by adaptively adjusting the sparsity of metadata; the higher the sparsity, the better. The larger the value, the higher the weight of semantic attention, and the final aggregated features are:

[0015] in, β ij Semantic weights, adjusted via the Sigmoid function, can enable semantic-level feature selection.

[0016] In another implementation of the present invention, the method further includes: the dynamic weight adaptive module assigns differentiated alignment confidence weights to metadata from different sources based on the type of the system to which the metadata belongs; the semantic alignment probability is output through the fully connected layer; when the probability is higher than a preset threshold, mapping rules are automatically generated and stored in the unified semantic mapping table; when the probability is lower than the preset threshold, it is marked as a suspected association and a manual verification process is triggered to balance automation efficiency and alignment accuracy.

[0017] In another aspect, this invention provides a knowledge graph-based intelligent metadata classification and association system, comprising: a data acquisition module: acquiring metadata from different business systems of an enterprise using a multi-source metadata access layer, and identifying the semantic relationships of each metadata based on natural language processing technology; a semantic mapping table generation module: performing semantic alignment processing on the metadata based on an improved graph attention network model to generate a unified semantic mapping table, wherein the improved graph attention network model integrates node attention mechanism and semantic attention mechanism, and adjusts the alignment confidence of metadata from different data sources through a dynamic weight adaptive module; a knowledge graph construction module: automatically constructing a metadata knowledge graph based on the unified semantic mapping table using knowledge graph enhanced retrieval technology; a tag generation module: generating descriptive semantic tags for each metadata based on a generative pre-trained language model and in combination with semantic information in the metadata knowledge graph; iteratively optimizing the generated tags through reinforcement learning methods and in combination with user feedback; and an intelligent semantic retrieval module: receiving natural language query requests, converting the query requests into knowledge graph query statements using a semantic parsing model; performing semantic retrieval in the metadata knowledge graph using the knowledge graph query statements, and combining with an intelligent recommendation algorithm for associated paths to obtain cross-system associated query results.

[0018] In another aspect, the present invention provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of a knowledge graph-based metadata intelligent classification and association method as described in any of the preceding claims. In another aspect, the present invention provides a computer storage medium storing a computer program, which, when executed by a processor, implements the steps of a knowledge graph-based metadata intelligent classification and association method as described in any of the preceding claims.

[0019] The knowledge graph-based intelligent classification and association method for metadata of this invention realizes automated tag generation, deep semantic association, and real-time intelligent query of metadata through dynamic adaptive classification model, cross-system semantic association network, and natural language semantic retrieval. It effectively solves the bottleneck problems of low classification accuracy, weak association ability, and insufficient support for multimodal data in traditional methods, and realizes efficient intelligent management and deep semantic association of metadata. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. By reading the detailed description of the embodiments below, the advantages and benefits of the solutions will become clear to those skilled in the art. The accompanying drawings are only for illustrating preferred embodiments and are not intended to limit the present invention. In the accompanying drawings: Figure 1 This is a flowchart illustrating the steps of a knowledge graph-based metadata intelligent classification and association method according to an embodiment of the present invention.

[0021] Figure 2 This is a detailed flowchart of a knowledge graph-based metadata intelligent classification and association method according to an embodiment of the present invention.

[0022] Figure 3 This is an overall flowchart of a knowledge graph-based metadata intelligent classification and association method according to an embodiment of the present invention.

[0023] Figure 4 This is a schematic diagram of a rule engine design according to an embodiment of the present invention.

[0024] Figure 5 This is a schematic diagram of database connection and metadata configuration according to an embodiment of the present invention.

[0025] Figure 6 This is a schematic diagram illustrating the generation of a knowledge graph according to an embodiment of the present invention.

[0026] Figure 7 This is a schematic diagram illustrating the automatic generation of metadata tags according to an embodiment of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art should fall within the protection scope of the present invention.

[0028] Figure 1 This is a schematic diagram of a knowledge graph-based metadata intelligent classification and association method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, this embodiment mainly includes: S101. Obtain metadata from different business systems of the enterprise using the multi-source metadata access layer, and identify the semantic relationships of each metadata based on natural language processing technology.

[0029] For example, such as Figure 5 As shown, the multi-source metadata access layer is compatible with metadata formats from different systems, enabling metadata access and compatibility across multiple data sources such as databases, APIs, and file systems. It supports mainstream data formats such as JSON, XML, and RDF. Based on natural language processing technology, it identifies and parses the semantic relationships between different fields, such as the semantic equivalence between "User ID" and "Customer Number".

[0030] In the metadata access phase, natural language processing and graph structure learning methods are used to perform unified semantic expression and mapping of metadata from heterogeneous data such as multi-databases, API interfaces and file systems. This ensures the implementation of unified semantic specifications from the data source and solves the fundamental problem of semantic fusion of multimodal and multi-structured data.

[0031] S102. Based on the improved graph attention network model, the metadata is semantically aligned to generate a unified semantic mapping table. The improved graph attention network model integrates node attention mechanism and semantic attention mechanism, and adjusts the alignment confidence of metadata from different data sources through a dynamic weight adaptive module.

[0032] For example, a graph attention network (GAT) is used to automatically learn semantic alignment rules for cross-system metadata, thereby enabling the automatic generation of a unified semantic mapping table from heterogeneous data.

[0033] AI technology application: Based on NLP to identify field semantics (such as the semantic equivalence of "user ID" and "customer number"), automatic cross-system metadata implicit relationship mining and advanced semantic reasoning are realized through NLP technology, ensuring the intelligent reasoning and deep relationship mining capabilities supported by the knowledge graph after construction, and expanding the application breadth of the graph.

[0034] By employing advanced knowledge graph and AI semantic reasoning methods, significant improvements have been made in cross-system semantic association of metadata. Compared with traditional ETL tools, it can achieve higher semantic association coverage and deeper discovery of implicit relationships.

[0035] S103. Based on the unified semantic mapping table, a metadata knowledge graph is automatically constructed using knowledge graph-enhanced retrieval technology.

[0036] For example, such as Figure 6 As shown, intelligent methods such as graph attention networks are used to automatically achieve semantic consistency recognition and dynamic semantic alignment among metadata from different systems and multiple sources, thereby constructing a unified global knowledge association network for metadata across systems. This solves the problems of difficult cross-system data semantic fusion and the generation of fragmented data silos in existing solutions, and is the technical foundation for achieving deep fusion of cross-system metadata.

[0037] S104. Based on the generative pre-trained language model, and combined with the semantic information in the metadata knowledge graph, generate descriptive semantic tags for each metadata.

[0038] For example, such as Figure 7 As shown, a label generation model is built based on a generative pre-training framework, which outputs a highly accurate set of candidate labels based on the attribute descriptions of metadata. By introducing the synergistic optimization of an advanced generative pre-training model and reinforcement learning algorithms, the system achieves automatic label generation capabilities that are highly consistent with expert manual annotation, significantly improving the accuracy and efficiency of label generation. Furthermore, this system has multilingual label generation capabilities, supporting multiple language scenarios including Chinese and English, and possesses greater data diversity and application flexibility.

[0039] Based on classification and association, the tag generation module automatically generates descriptive tags for metadata through algorithms or natural language processing technology to help users quickly grasp the content and attributes of metadata, and realize the ability to automatically and accurately tag metadata entities (such as automatically generating semantic tags such as "high-value customer table" and "abnormal transaction records").

[0040] S105. The generated labels are iteratively optimized by using reinforcement learning methods and combining user feedback.

[0041] For example, a method of introducing a generative pre-trained language model collaborative reinforcement learning algorithm is used to achieve accurate automated tag generation and proactive iterative optimization of metadata, so as to ensure the accuracy and interpretability of the tag roles. This overcomes the limitations of hard-coded rules and template matching in traditional tag generation schemes, and greatly improves the accuracy, universality, adaptability and scalability of tag generation.

[0042] S106. Receive a natural language query request and use a semantic parsing model to convert the query request into a knowledge graph query statement.

[0043] For example, it supports natural language semantic query services, such as "query transaction records of high-value customers in the past six months", and quickly returns cross-system related data. It uses a semantic parsing model to convert natural language into standardized graph database statements for semantic query.

[0044] S107. Using the knowledge graph query statement, perform semantic retrieval in the metadata knowledge graph, and combine it with the intelligent recommendation algorithm for related paths to obtain cross-system related query results.

[0045] For example, such as Figure 4 As shown, artificial intelligence algorithms are introduced to evaluate and rank the confidence of different related paths involved in the query, thereby optimizing the accuracy of the returned results.

[0046] By integrating knowledge graphs and natural language semantic parsing technologies, it supports users in performing complex cross-system natural language semantic retrieval queries, accurately provides related path recommendations and dynamic path optimization mechanisms, and effectively solves the inherent defects of traditional SQL cross-database queries such as insufficient efficiency and difficulty in semantic understanding. It is a key functional technology for the implementation of the overall application solution.

[0047] Compared to traditional relational database cross-database SQL query solutions, the use of knowledge graph-based natural language semantic retrieval technology effectively shortens the response latency of complex cross-system data queries, achieving millisecond-level fast responses. Simultaneously, the relevance of search results and user satisfaction are significantly improved, effectively meeting users' demands for high accuracy and real-time data retrieval.

[0048] Specifically, such as Figure 2 and Figure 3 As shown, the overall design flow of this invention is as follows: Database connection → Metadata acquisition and import into the knowledge base → Sample data association analysis and import into the knowledge base → Chinese semantic information recognition (based on NLP technology) → Automatic construction of knowledge graph based on graph-augmented generation (Graph RAG) technology → Automated tag generation.

[0049] Once the knowledge graph is constructed, it can support semantic-based intelligent retrieval.

[0050] By constructing a cross-system, multi-source metadata association network, metadata from different databases and formats is semantically integrated to form a unified knowledge graph structure. Fully utilizing AI technologies (including natural language processing and deep learning algorithms), the system automatically learns the potential relationships between metadata, constructing a dynamically adaptable, full-domain semantic association structure, effectively improving the depth and breadth of association analysis. Furthermore, the system possesses a real-time update and dynamic expansion mechanism, autonomously updating the knowledge graph as metadata changes, ensuring the accuracy and timeliness of semantic associations. Through the effective application of AI fusion technology, the system can deeply mine the implicit semantic relationships of heterogeneous data, providing more accurate and comprehensive information services.

[0051] The knowledge graph-based intelligent classification and association method for metadata of this invention realizes automated tag generation, deep semantic association, and real-time intelligent query of metadata through dynamic adaptive classification model, cross-system semantic association network, and natural language semantic retrieval. It effectively solves the bottleneck problems of low classification accuracy, weak association ability, and insufficient support for multimodal data in traditional methods, and realizes efficient intelligent management and deep semantic association of metadata.

[0052] In another implementation of the present invention, the metadata knowledge graph construction process includes knowledge extraction, knowledge fusion, and knowledge representation.

[0053] For example, a global relational network can be constructed among cross-system, multi-source metadata, supporting dynamic expansion, real-time updates, and efficient semantic reasoning.

[0054] Detailed explanation of the application of Graph RAG technology: Knowledge graphs are semantic knowledge representation models composed of entities, attributes, and relationships.

[0055] The knowledge graph construction process includes three main stages: knowledge extraction (entity, relation, and attribute extraction), knowledge fusion (integration of multi-source knowledge and redundancy removal), and knowledge representation (graph storage and presentation, commonly represented by RDF and OWL).

[0056] By employing knowledge representation learning models such as TransE and RotatE, we can automatically mine and infer the implicit relationships between metadata across systems (e.g., identifying the association path between "user order table (from system A)" and "transaction record (from system B)").

[0057] In another implementation of the present invention, the method further includes: using a few-shot learning mechanism to quickly integrate the metadata of the new data source into the metadata knowledge graph, thereby realizing the dynamic updating and expansion of the knowledge graph and enabling the knowledge graph to have continuous expansion and dynamic adaptability.

[0058] For example, by leveraging few-shot learning techniques, the ability to rapidly and efficiently integrate new systems or data sources into an existing global knowledge graph is achieved, and the knowledge graph content can be updated dynamically and adaptively in a timely manner. This significantly improves the real-time performance, scalability, and ease of maintenance of the knowledge graph, laying a crucial foundation for the continuous and dynamic application of the overall method. Furthermore, the knowledge fusion technique based on few-shot learning has significant advantages in the integration of new data sources and the rapid construction of knowledge graphs, effectively shortening the metadata integration cycle of new systems and greatly improving data fusion efficiency.

[0059] In another implementation of the present invention, the node attention mechanism adopts the self-attention mechanism of GAT to calculate the structural correlation weight between the target node and its neighboring nodes, as shown in the following formula: e ij = (W h i W h j ) in, hi , h j Let W be the node feature vector, and W be the feature transformation matrix. The attention scoring function is used; the node-level attention weights are obtained by normalization using the Softmax function, and the features of neighboring nodes are aggregated.

[0060] in, Attention weights at the node level.

[0061] In another implementation of the present invention, the semantic attention mechanism adds a semantic similarity calculation branch. Based on the semantic vector features of the input layer, it calculates the semantic fit between the target node and its neighboring nodes, using cosine similarity as the semantic scoring function.

[0062] in, , This is the semantic vector of the node.

[0063] In another implementation of the present invention, the method further includes: designing dynamic fusion coefficients. Attention fusion is achieved by adaptively adjusting the sparsity of metadata; the higher the sparsity, the better. The larger the value, the higher the weight of semantic attention, and the final aggregated features are:

[0064] in, β ij Semantic weights, adjusted via the Sigmoid function, can enable semantic-level feature selection.

[0065] For example, in response to the core requirements of the semantic mapping of heterogeneous data, a dual attention mechanism fusion and dynamic weight adaptive module design were carried out on the basis of the traditional graph attention network (GAT). This solves the problems of "excessive local dependency" and "mismatch of semantic ambiguity" in the semantic alignment of cross-system metadata of the traditional GAT, and significantly improves the mapping accuracy and network generalization ability.

[0066] This paper analyzes the limitations of the traditional GAT algorithm and addresses the semantic characteristics and sparsity issues of metadata fields by first enhancing the features of the input fields. On one hand, semantic vector features of the fields are extracted using NLP techniques (emphasizing the use of a BERT fine-tuning model to convert field names and annotation text into 768-dimensional vectors to capture deep semantics). On the other hand, field attribute feature vectors (containing structured attributes such as data type, length, system, and frequency of occurrence) are constructed. The two types of vectors are concatenated and used as the initial features of the nodes to solve the problem of insufficient features in sparse fields.

[0067] Specifically, the improved GAT model structure design: By co-computing node attention and semantic attention, accurate aggregation of neighbor node information is achieved. Node attention branch: Using the self-attention mechanism of GAT, the structural correlation weights between the target node and its neighboring nodes are calculated.

[0068] Semantic Attention Branch: A new semantic similarity calculation branch has been added, which calculates the semantic fit between the target node and its neighboring nodes based on the semantic vector features of the input layer.

[0069] Attention Fusion: Designing Dynamic Fusion Coefficients (Adaptively adjusted by metadata sparsity; the higher the sparsity, the better) The larger the value, the higher the weight of semantic attention, resulting in the final aggregated features.

[0070] In another implementation of the present invention, the method further includes: the dynamic weight adaptive module assigns differentiated alignment confidence weights to metadata from different sources based on the type of the system to which the metadata belongs; the semantic alignment probability is output through the fully connected layer; when the probability is higher than a preset threshold, mapping rules are automatically generated and stored in the unified semantic mapping table; when the probability is lower than the preset threshold, it is marked as a suspected association and a manual verification process is triggered to balance automation efficiency and alignment accuracy.

[0071] For example, a new dynamic weight adaptive module is added, which assigns differentiated alignment confidence weights to metadata from different sources based on the type of system to which the metadata belongs (structured database, unstructured file, API interface). The semantic alignment probability is output through the fully connected layer. When the probability is higher than a set threshold (0.85), a mapping rule of "field name - unified semantic label - alignment confidence" is automatically generated and stored in a unified semantic mapping table; when it is lower than the threshold, it is marked as a suspected association and a manual verification process is triggered, balancing automation efficiency and alignment accuracy.

[0072] In the process of implementing the improved DDA-GAT model and building a cross-system metadata association network, the core challenges are resolving semantic ambiguities in cross-system metadata and adapting sparse data. The following section, based on development practice, details the causes of the problems, solutions, and implementation steps.

[0073] (1) Core difficulties: Semantic ambiguity in cross-system metadata mainly manifests in two categories: one is synonymy (such as “user ID”, “customer number”, and “member ID” all referring to the unique identifier of a user), and the other is homonymy (such as “order amount” including shipping costs in the e-commerce system, but not including shipping costs in the financial system).

[0074] Data sparsity stems from the lack of annotations in metadata of some older systems, the limited number of fields in niche systems, and incomplete cross-system data interaction, making it difficult for models to learn stable semantic association rules. Failure to address this issue will directly lead to errors in the semantic mapping table, distorted association network construction, and impact on the accuracy of subsequent information services.

[0075] (2) Technical solutions: To address semantic ambiguity, a dual attention mechanism is used to achieve a dual judgment of "structural association + semantic fit": node attention captures the frequency of association of fields in the graph structure (e.g., "User ID" and "Customer Number" are often associated with the same "Order Information" node, and have high structural weights), while semantic attention accurately identifies synonymous relationships through BERT semantic vectors, while distinguishing semantic differences between fields with the same name (e.g., adding the semantic attribute "whether it includes shipping fee" to "Order Amount" to optimize the semantic vector representation).

[0076] To address data sparsity, a combined strategy of "feature enhancement + transfer learning" is adopted: First, the dual feature vectors of the input layer are concatenated (semantic vector + attribute vector) to supplement the feature dimension of sparse fields; second, the pre-trained BERT model is combined with the DDA-GAT model, utilizing general text semantic knowledge to transfer to the semantic learning of metadata fields, reducing the impact of sparse data on model training. In practice, the BERT model is first pre-trained on a large-scale general text corpus, and then fine-tuned using a small amount of high-quality cross-system metadata corpus, enabling the model to quickly adapt to metadata semantic scenarios.

[0077] It should be understood that traditional cross-system semantic alignment schemes often employ ontology or manually configured mapping tables, which suffer from poor semantic adaptability and limited scalability, making them unsuitable for large-scale heterogeneous data environments. This method innovatively applies graph attention networks to automatically learn the semantic relationships between cross-system data, enabling real-time identification of semantic equivalence between different field names and generating an efficient and stable unified semantic mapping layer. This effectively improves the accuracy, efficiency, and scalability of heterogeneous data semantic alignment.

[0078] Traditional tag generation methods are mostly based on rule templates or keyword matching, which have limited accuracy and lack versatility and adaptability. This system integrates generative language pre-trained models with reinforcement learning methods and incorporates user feedback to achieve a self-optimizing and iterative tag generation mode. This mode effectively solves the pain points of traditional methods, such as insufficient accuracy and difficulty in adapting to complex scenarios, significantly enhancing the reliability and scalability of tag generation.

[0079] This invention focuses on achieving cross-system semantic alignment through Graph Attention Networks (GAT), utilizing generative pre-trained models combined with reinforcement learning to achieve automated label generation, and relying on graph augmented retrieval (Graph RAG) technology to achieve real-time intelligent semantic retrieval. Besides the specific implementation described in this invention, the following alternative solutions also achieve the technical objectives of this invention: (1) Alternatives to the metadata semantic alignment and association module: In addition to using Graph Attention Networks (GAT), other graph neural network variants can be employed, such as Graph Convolutional Networks (GCN), Graph Isomorphism Networks (GIN), or Heterogeneous Graph Attention Networks (HAN).

[0080] Semantic alignment schemes based on knowledge graph embedding methods can be used, such as knowledge graph embedding models, to assist or independently complete the semantic mapping and alignment of cross-system metadata.

[0081] By employing multi-view learning or joint embedding methods, a unified cross-system semantic mapping relationship is formed by integrating metadata and other additional semantic information within the system.

[0082] (2) Alternatives to the automated label generation module: In addition to using generative pre-trained models combined with reinforcement learning, other language model variants and technology combinations can also be considered, such as GPT series models (GPT-2, GPT-3, ChatGPT) combined with unsupervised contrastive learning or semi-supervised learning for label generation and optimization.

[0083] The initial generation of metadata tags is achieved by using sequence labeling-based models or classification models (such as BERT fine-tuning + classifier layer), and the tag results are intelligently optimized and iterated by combining active learning technology or transfer learning technology.

[0084] Selectively add expert rule engines to assist the AI ​​model to ultimately produce more accurate labeling results that meet business needs.

[0085] (3) Alternatives to the semantic intelligent retrieval module: In addition to the proposed graph-based augmented retrieval (graph RAG), a combination of graph database query engines (such as Neo4j) and natural language query and translation tools can also be used.

[0086] By combining semantic parsing with template methods, such as using neural symbolic reasoning to model and reason about query statements, the accuracy of semantic retrieval can be improved.

[0087] (4) Alternatives to knowledge graph construction: In addition to the graph RAG method, extractive methods and open information extraction methods can be used to automatically construct knowledge graphs from structured, semi-structured, or unstructured data; rule-based knowledge extraction methods (such as regular expression rule engines and template extraction engines) can be used in conjunction with machine learning methods to jointly construct preliminary knowledge graphs; ontology-driven knowledge graph construction methods can be used to assist AI technology in achieving more accurate knowledge association networks that better meet domain requirements.

[0088] The application scenario of this invention is enterprise data governance and data platform: metadata of structured and semi-structured data scattered across different business systems (such as CRM, ERP, SCM), databases (Oracle, MySQL), data warehouses, and big data platforms within an enterprise. Examples include table structure, field definitions, lineage relationships, data quality rules, and business terminology. The cross-system semantic alignment, automated tag generation, and intelligent retrieval described in this paper can directly solve problems such as data silos, difficulties in building data catalogs, and low efficiency in finding and using data within enterprises.

[0089] Another aspect of the present invention provides a knowledge graph-based metadata intelligent classification and association system, comprising: Data acquisition module: It uses a multi-source metadata access layer to acquire metadata from different business systems of the enterprise and identifies the semantic relationships of each metadata based on natural language processing technology.

[0090] Semantic Mapping Table Generation Module: Based on an improved graph attention network model, the metadata is semantically aligned to generate a unified semantic mapping table. The improved graph attention network model integrates node attention mechanism and semantic attention mechanism, and adjusts the alignment confidence of metadata from different data sources through a dynamic weight adaptive module.

[0091] Knowledge graph construction module: Based on the unified semantic mapping table, a metadata knowledge graph is automatically constructed using knowledge graph enhanced retrieval technology.

[0092] Tag generation module: Based on a generative pre-trained language model, combined with semantic information in the metadata knowledge graph, descriptive semantic tags for each metadata are generated; the generated tags are iteratively optimized through reinforcement learning methods and user feedback.

[0093] Intelligent semantic retrieval module: Receives natural language query requests, converts the query requests into knowledge graph query statements using a semantic parsing model; performs semantic retrieval on the metadata knowledge graph using the knowledge graph query statements, and combines them with an intelligent recommendation algorithm for related paths to obtain cross-system related query results.

[0094] The knowledge graph-based intelligent metadata classification and association system of this invention achieves automated tag generation, deep semantic association, and real-time intelligent querying of metadata through dynamic adaptive classification models, cross-system semantic association networks, and natural language semantic retrieval. It effectively solves the bottleneck problems of low classification accuracy, weak association ability, and insufficient support for multimodal data in traditional methods, and realizes efficient intelligent management and deep semantic association of metadata.

[0095] In another aspect of the present invention, the electronic device includes: a processor, a memory, and a communication bus and a communication interface.

[0096] in: The processor, memory, and communication interface communicate with each other via a communication bus.

[0097] A communication interface is used to communicate with other electronic devices or servers.

[0098] The processor is used to execute programs, specifically, to perform any of the steps of the knowledge graph-based metadata intelligent classification and association method in the above embodiments.

[0099] Specifically, the program may include program code, which includes computer operation instructions.

[0100] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0101] Memory is used to store programs. Memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive.

[0102] Specifically, the program can be used to cause the processor to execute the steps of any of the knowledge graph-based metadata intelligent classification and association methods described in the embodiments. The specific implementation of each step in the program can be found in the corresponding descriptions of the steps and units executed by any of the knowledge graph-based metadata intelligent classification and association methods described above, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments.

[0103] An exemplary embodiment of this application also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods of various embodiments of this application.

[0104] The methods described above according to embodiments of the present invention can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored on a remote recording medium or a non-transitory machine-readable medium and subsequently stored on a local recording medium, downloaded via a network. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0105] Specific embodiments of the present invention have now been described. Other embodiments are within the scope of the appended claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result.

[0106] It should be noted that all directional indications (such as up, down, left, right, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship between the components in a certain order (as shown in the figure). If the specific order changes, the directional indication will also change accordingly.

[0107] In the description of this invention, the terms "first" and "second" are used only for convenience in describing different components or names, and should not be construed as indicating or implying a sequential relationship, relative importance, or implicitly specifying the number of technical features indicated. Thus, a feature defined with "first" and "second" may explicitly or implicitly include at least one of that feature.

[0108] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0109] It should be noted that although specific embodiments of the present invention have been described in detail with reference to the accompanying drawings, this should not be construed as limiting the scope of protection of the present invention. Various modifications and variations that can be made by those skilled in the art without inventive effort within the scope described in the claims still fall within the scope of protection of the present invention.

[0110] The examples of the embodiments of the present invention are intended to concisely illustrate the technical features of the embodiments of the present invention, so that those skilled in the art can intuitively understand the technical features of the embodiments of the present invention, and are not intended to be an improper limitation of the embodiments of the present invention.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent classification and association of metadata based on knowledge graphs, characterized in that, include: The multi-source metadata access layer is used to obtain metadata from different business systems of an enterprise, and the semantic relationships of each metadata are identified based on natural language processing technology. The metadata is semantically aligned based on an improved graph attention network model to generate a unified semantic mapping table. The improved graph attention network model integrates node attention mechanism and semantic attention mechanism, and adjusts the alignment confidence of metadata from different data sources through a dynamic weight adaptive module. Based on the unified semantic mapping table, a metadata knowledge graph is automatically constructed using knowledge graph-enhanced retrieval technology. Based on the generative pre-trained language model, and combined with the semantic information in the metadata knowledge graph, descriptive semantic tags for each metadata are generated. The generated tags are iteratively optimized by using reinforcement learning methods and incorporating user feedback. Receive natural language query requests and use a semantic parsing model to convert the query requests into knowledge graph query statements; The knowledge graph query statement is used to perform semantic retrieval in the metadata knowledge graph, and combined with the intelligent recommendation algorithm for related paths, to obtain query results related to cross systems.

2. The method according to claim 1, characterized in that, The metadata knowledge graph construction process includes knowledge extraction, knowledge fusion, and knowledge representation.

3. The method according to claim 2, characterized in that, Also includes: By utilizing a few-shot learning mechanism, metadata from newly added data sources is quickly integrated into the metadata knowledge graph, enabling dynamic updates and expansion of the knowledge graph.

4. The method according to claim 1, characterized in that, The node attention mechanism adopts the self-attention mechanism of GAT, and calculates the structural correlation weight between the target node and its neighboring nodes, as shown in the following formula: e ij = (IN h i ,IN h j ) in, h i , h j Let W be the node feature vector, and W be the feature transformation matrix. For attention scoring functions; Attention weights at the node level are obtained by normalization using the Softmax function, and then aggregated with features from neighboring nodes. in, Attention weights at the node level.

5. The method according to claim 4, characterized in that, The semantic attention mechanism adds a semantic similarity calculation branch, which calculates the semantic fit between the target node and its neighboring nodes based on the semantic vector features of the input layer, using cosine similarity as the semantic scoring function. in, , This is the semantic vector of the node.

6. The method according to claim 5, characterized in that, Also includes: Design dynamic fusion coefficient Attention fusion is achieved by adaptively adjusting the sparsity of metadata; the higher the sparsity, the better. The larger the value, the higher the weight of semantic attention, and the final aggregated features are: in, β ij Semantic weights, adjusted via the Sigmoid function, can enable semantic-level feature selection.

7. The method according to claim 1, characterized in that, Also includes: The dynamic weight adaptive module assigns differentiated alignment confidence weights to metadata from different sources based on the type of system to which the metadata belongs. The semantic alignment probability is output through the fully connected layer. When the probability is higher than a preset threshold, a mapping rule is automatically generated and stored in the unified semantic mapping table. When the value is below the preset threshold, it is marked as a suspected association, triggering a manual verification process to balance automation efficiency and alignment accuracy.

8. A knowledge graph-based intelligent metadata classification and association system, characterized in that, include: Data acquisition module: It uses a multi-source metadata access layer to acquire metadata from different business systems of the enterprise and identifies the semantic relationships of each metadata based on natural language processing technology; Semantic mapping table generation module: Based on the improved graph attention network model, the metadata is semantically aligned to generate a unified semantic mapping table. The improved graph attention network model integrates node attention mechanism and semantic attention mechanism, and adjusts the alignment confidence of metadata from different data sources through a dynamic weight adaptive module. Knowledge graph construction module: Based on the unified semantic mapping table, a metadata knowledge graph is automatically constructed using knowledge graph enhanced retrieval technology; Tag generation module: Based on the generative pre-trained language model, combined with the semantic information in the metadata knowledge graph, it generates descriptive semantic tags for each metadata. The generated tags are iteratively optimized by using reinforcement learning methods and incorporating user feedback. Intelligent semantic retrieval module: Receives natural language query requests, converts the query requests into knowledge graph query statements using a semantic parsing model; performs semantic retrieval on the metadata knowledge graph using the knowledge graph query statements, and combines them with an intelligent recommendation algorithm for related paths to obtain cross-system related query results.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the knowledge graph-based metadata intelligent classification and association method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, implements the steps of the knowledge graph-based metadata intelligent classification and association method as described in any one of claims 1 to 7.

Images