Metadata query method based on knowledge graph

Abstract

The invention discloses a metadata query method based on a knowledge graph, which relates to the technical field of data query, and comprises the following steps: collecting metadata from various data sources; then respectively performing knowledge extraction on the data, wherein the knowledge extraction comprises entity extraction, relationship extraction and attribute extraction; constructing a data asset directory according to the metadata after knowledge fusion and a knowledge extraction result of the metadata; then knowledge modeling is carried out, and a knowledge graph is established; and finally, constructing a retrieval enhancement generation process by utilizing a large language model, and retrieving the metadata. According to the method, the natural language can be understood and generated by utilizing the large language model and the retrieval enhancement generation process, and the RAG process retrieves related knowledge and inputs the related knowledge into the language model as a context, so that the processing capability of the model on complex queries is enhanced. According to the method, the extraction accuracy and efficiency can be improved, the method can better adapt to new fields and data types, and the requirement for manual intervention is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data query technology, and in particular to a metadata query method based on a knowledge graph. Background Art

[0002] In the era of big data, information extraction technology extracts valuable knowledge from massive data and has become one of the core technologies in the field of artificial intelligence. Knowledge extraction involves identifying and extracting information such as entities, relationships, and attributes from structured, semi-structured, and unstructured data sources, and converting them into structured knowledge representations for use in building knowledge graphs, intelligent question-answering systems, and other intelligent applications. Structured data extraction usually relies on database query and mapping technology, while unstructured data extraction relies on natural language processing technology, such as entity recognition, relationship extraction, and event extraction.

[0003] Existing information extraction technologies face many challenges when processing unstructured data, such as the diversity and ambiguity of text data, the efficiency of processing large-scale data, real-time and dynamic requirements, and domain dependence. In addition, existing extraction systems may be difficult to adapt to new domains or data types, requiring a lot of manual intervention and maintenance. These challenges limit the accuracy, efficiency, and generalization ability of extraction systems.

[0004] In view of this, a metadata query method based on knowledge graph is needed. Summary of the invention

[0005] In view of the diversity and ambiguity of text data in the prior art, the processing efficiency of large-scale data, the real-time and dynamic requirements, and the domain dependence, the present invention provides a metadata query method based on knowledge graph, which can effectively overcome these shortcomings by utilizing large language models (LLMs) and retrieval enhanced generation (RAG) processes. Large language models can understand and generate natural language, while the RAG process enhances the model's ability to handle complex queries by retrieving relevant knowledge and inputting it as context to the language model. This method can not only improve the accuracy and efficiency of extraction, but also better adapt to new fields and data types, reducing the need for manual intervention. The specific technical solution is as follows:

[0006] A metadata query method based on knowledge graph includes the following steps:

[0007] Collect metadata from various data sources, including structured data, semi-structured data, and unstructured data;

[0008] Extract knowledge from structured data, semi-structured data and unstructured data respectively. Knowledge extraction includes entity extraction, relationship extraction and attribute extraction.

[0009] Build a data asset catalog based on the metadata after knowledge fusion and the knowledge extraction results of the metadata;

[0010] Define entity categories, attribute definitions, and relationship types, perform knowledge modeling, and build a knowledge graph;

[0011] Using large language models, we build a retrieval enhancement generation process to retrieve metadata.

[0012] Preferably, knowledge extraction includes extraction of structured data, specifically as follows:

[0013] Data identification and preprocessing: Clean structured data such as database tables or spreadsheets, including removing duplicate records, correcting errors, and processing missing values ​​to ensure data quality;

[0014] Entity recognition: Use regular expressions, dictionary matching, or machine learning models to identify specific entities from data, including dates, numbers, and categorical labels;

[0015] Relationship extraction: Identify relationships between entities based on relationships between fields or data patterns, including foreign key associations and formula calculation results;

[0016] Entity linking: linking the identified entities to the corresponding entries in the knowledge base;

[0017] Relation type normalization: Map the extracted relations to predefined relation types;

[0018] Data standardization: unify data formats, units, etc.;

[0019] Knowledge fusion: integrating structured data from different sources;

[0020] Knowledge storage: Store the extracted structured data in a knowledge graph or other knowledge base to facilitate subsequent query and analysis.

[0021] Preferably, knowledge extraction includes extraction of unstructured data, specifically as follows:

[0022] Text preprocessing: cleaning text data, including noise removal, language standardization, word segmentation, and part-of-speech tagging;

[0023] Entity recognition: Use natural language processing techniques, including named entity recognition (NER), to identify specific entities from text;

[0024] Relation extraction: Identify the relationships between entities in text through pattern matching, supervised learning or deep learning methods;

[0025] Event extraction: Identify the events and their participants described in the text, and extract the event types and arguments;

[0026] Entity linking: linking entities in the text to corresponding entities in the knowledge base;

[0027] Text summary: summarize the text content and extract key information;

[0028] Sentiment analysis: Analyze the sentiment tendency of the text and extract the sentiment polarity;

[0029] Knowledge storage: Store the extracted entities, relationships, events, etc. into the knowledge base to form a structured knowledge representation.

[0030] Preferably, knowledge extraction includes extraction of semi-structured data, specifically as follows:

[0031] Data identification and preprocessing: standardize the format and clean the semi-structured data such as XML and HTML documents;

[0032] Entity recognition: using labels or attribute information in the data to identify entities in semi-structured data;

[0033] Attribute extraction: extracting attribute information of entities, including class and id in HTML or fields in XML;

[0034] Relationship extraction: Identify the relationship between entities by analyzing the nested relationships and label structure of the data;

[0035] Entity linking: matching extracted entities with entities in the knowledge base;

[0036] Data fusion: integrating information from multiple semi-structured data sources;

[0037] Data standardization: converting the extracted data into a unified format;

[0038] Knowledge storage: Store the extracted entities, attributes and relationships in the knowledge base to form a structured knowledge representation.

[0039] Preferably, the knowledge graph construction process is as follows:

[0040] The entity type and its attributes are defined according to the entity modeling results, and the relationship type and its constraints are defined according to the relationship modeling results. Then these modeling results are integrated to construct the ontology.

[0041] Map the constructed ontology into the graph database and create corresponding nodes, edges and their attributes.

[0042] Using the graph query language, the information in the data asset catalog is imported into the ontology to obtain a knowledge graph that can be used to query metadata.

[0043] Preferably, the specific steps of querying metadata are as follows:

[0044] Select documents and data to be included in the knowledge base, use text segmentation to segment documents according to paragraphs or other logic, and then use the embedding model to generate semantic vectors and store them in the vector database;

[0045] Obtain the user's natural language query. Depending on the application scenario, you may need to use natural language processing technology to convert the query into a specific format;

[0046] Take the transformed query and implement a logical or semantic routing mechanism to direct the query to the most appropriate index or processing unit;

[0047] Get the user's query, calculate the similarity between the query embedding and the embedding of each document block in the knowledge base, and return the most similar result. Then output the document block or information related to the query;

[0048] The user's query and the retrieved related document blocks are obtained, and the query and retrieval results are provided as input to the language model, and the model generates answers based on this information.

[0049] Preferably, the method further comprises the following steps:

[0050] Publish a data asset catalogue so that all departments and units can access and use relevant data resources.

[0051] Preferably, the method further comprises the following steps:

[0052] Publish a knowledge graph for querying metadata so that various departments and units can access and utilize it.

[0053] A computer-readable storage medium includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute the metadata query method based on the knowledge graph as described above.

[0054] A processor is used to run a program, wherein the program executes the metadata query method based on the knowledge graph as described above when running.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] The present invention collects metadata from various data sources, and the metadata includes structured data, semi-structured data and unstructured data; then the structured data, semi-structured data and unstructured data are respectively subjected to knowledge extraction, and the knowledge extraction includes entity extraction, relationship extraction and attribute extraction; a data asset catalog is constructed according to the metadata after knowledge fusion and the knowledge extraction results of the metadata; then entity categories, attribute definitions and relationship types are defined, knowledge modeling is performed, and a knowledge graph is established; finally, a retrieval enhancement generation process is constructed using a large language model to retrieve the metadata. These deficiencies are effectively overcome by using large language models (LLMs) and retrieval enhancement generation (RAG) processes. The large language model can understand and generate natural language, and the RAG process enhances the model's ability to process complex queries by retrieving relevant knowledge and inputting it as context to the language model. This method can not only improve the accuracy and efficiency of extraction, but also better adapt to new fields and data types, reducing the need for manual intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0058] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0061] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0062] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0063] In one embodiment of the present invention, a metadata query method based on a knowledge graph is provided. Figure 1 As shown, the following steps are included:

[0064] Step 1: Collect metadata from various data sources, including equipment management systems, maintenance systems, monitoring systems, etc. These metadata include structured data, semi-structured data and unstructured data.

[0065] Structured data, such as specification parameters of power grid equipment, power transaction records, and user power consumption data, provide clear node and attribute information for the graph. These data are usually stored in relational databases and are easy to query and analyze.

[0066] Semi-structured data, such as power system configuration files and grid topology information in XML format, are not as regular as structured data, but they contain rich information and can provide additional context for the knowledge graph. By parsing this data, the connection relationship between devices can be extracted, providing key side information for graph construction. For example, by analyzing historical electricity consumption data, peak electricity demand can be predicted, thereby optimizing the operating efficiency of the power grid.

[0067] Unstructured data, such as maintenance logs of power systems, expert reports, and discussions on social media, are difficult to use directly to build graphs, but they contain a lot of underutilized knowledge. Using natural language processing technology, key information can be extracted from them, such as fault causes, user feedback, and industry trends, which are essential for enriching the content of graphs and improving their application value. For example, by analyzing users' discussions on power outages on social media, potential weak links in the power grid can be discovered, so that measures can be taken in advance to improve them.

[0068] Step 2: Extract knowledge from structured data, semi-structured data and unstructured data respectively. Knowledge extraction includes entity extraction, relationship extraction and attribute extraction.

[0069] The data extracted from the knowledge graph is mainly divided into the following three types of data:

[0070] (1) Facing structured data

[0071] Power structured data is a form of data storage in a table-like form similar to that in a relational database, in which there are clear relationship names and corresponding relationships between items. This data model is widely used in the power field to describe the entities, attributes and the relationships between them in the power system. The process of processing power structured data includes data modeling, data import, data query and analysis, data cleaning and verification, data indexing and optimization, and data update and maintenance. First, when performing data modeling, we need to identify the entities in the power system, such as power plants, substations, transmission lines, etc., and their attributes, such as name, location, capacity, etc. At the same time, we also need to clarify the relationship between entities, such as connection relationship, power supply relationship, etc. By defining appropriate table structure, primary key, foreign key and constraint, the consistency and integrity of the data can be ensured. By extracting knowledge from power structured data, meaningful information and associations can be extracted from the data to support analysis, decision-making and application development in the power field. This may involve steps such as data understanding, entity extraction, attribute extraction, relationship extraction and knowledge representation to transform data into useful knowledge forms and provide deeper insights and applications for the power industry.

[0072] (2) Facing semi-structured data

[0073] Semi-structured data is a special form of structured data that is incompatible with traditional relational databases or other forms of data table structures, but uses tags or other markers to separate semantic elements and maintain the hierarchical structure of records and data fields. In infrastructure project documents, we can find rich semi-structured table data. The inventory of accepted equipment records the attribute data of various types of equipment used in the project, such as equipment type, material code, purchase price, etc. The test report records the parameter information and test data of the main equipment at the time of acceptance, such as the rated current, DC voltage, manufacturer, loop insulation test results, zero drift test results, etc. of the main transformer. These data have high accuracy and are in a wide range of applications, so they have become an important data source for building large-scale knowledge graphs and an important way to acquire knowledge.

[0074] From the perspective of project management, we can model the equipment used in the power grid infrastructure project as the entity node. At the same time, the descriptive data related to the equipment can also be modeled as the attributes of the entity node, such as the material name, material code, purchase price, etc. of the equipment. Through this modeling method, we can better manage and understand the equipment used in the project and its related information. This modeling method based on semi-structured data helps to improve the efficiency and accuracy of project management, and at the same time provides a valuable basis for further data analysis and decision-making.

[0075] Semi-structured data is a form between structured data and unstructured data. It does not strictly define the data model like the table structure in the traditional relational database, but contains tags or other markers to separate semantic elements to maintain the hierarchical structure of the data. This form of data is common in text documents, log files, web pages, etc. Due to the particularity of semi-structured data, traditional relational database management systems cannot process it directly. Therefore, when processing semi-structured data, the common method is to use specialized tools and technologies, such as XML (Extensible Markup Language) or JSON (JavaScript Object Notation). These tools and technologies can parse and extract information from semi-structured data and convert it into a structured form that can be analyzed and applied. In the field of engineering management, semi-structured data has important application value. The semi-structured table data in infrastructure engineering files contains key information such as equipment attributes, parameter information, and test data.

[0076] By extracting, organizing and analyzing these data, the following knowledge graph construction can be achieved: Semi-structured data is one of the important sources for building large-scale knowledge graphs. By modeling the entity nodes and attribute information in semi-structured data, a knowledge graph in the field of engineering management can be constructed to provide more comprehensive and accurate knowledge representation and query capabilities. Semi-structured data plays an important role in engineering management. They provide key information about equipment, attributes and test data, help optimize resources, support decision-making, and provide an important data foundation for the construction of knowledge graphs. Through appropriate tools and technologies, these data can be effectively extracted, processed and utilized to improve the efficiency and quality of engineering management.

[0077] (3) Unstructured data

[0078] In the unstructured natural language text of the power industry, relation extraction and attribute extraction remain important tasks of information extraction to identify semantic relations between entities and obtain attribute information of entities. The goal of relation extraction is to identify the relations between entities from the text. In the text of the power industry, there may be sentences or paragraphs describing the relations between entities such as equipment, components, suppliers, projects, etc. Through relation extraction, we can identify these relations, such as equipment A is provided by supplier B, equipment C is connected to equipment D, etc. These relations can help us establish connections between entities, thus forming a more complete knowledge graph or semantic network.

[0079] Attribute extraction is the process of extracting specific attribute information of entities from text. In the text of the power industry, entities can be equipment, components, technical specifications, etc. The goal of attribute extraction is to identify and extract the attributes of these entities, such as the model, manufacturer, rated power, etc. of the equipment. Through attribute extraction, we can obtain detailed features about the entity and further enrich the description and semantic information of the entity. The attribute extraction problem can be regarded as a special case of the relationship extraction problem, in which the attribute can be regarded as a relationship between the entity and the attribute value. For example, in a description "the model of equipment A is B", "equipment A" can be regarded as an entity, "model" as an attribute, and "B" as an attribute value. Therefore, attribute extraction can be performed under the framework of relationship extraction.

[0080] Relation extraction and attribute extraction have important application value in natural language texts in the power industry. They can help us extract useful structured information from a large amount of unstructured text and support knowledge management, intelligent analysis and decision-making in the power industry. These tasks can be solved by machine learning and natural language processing techniques, such as entity recognition, relationship classification and attribute extraction. Through relationship and attribute extraction, we can better understand the relationship between entities in the power industry text and obtain richer structured semantic information.

[0081] Specifically, structured data, semi-structured data and unstructured data are subjected to knowledge extraction respectively. Knowledge extraction includes entity extraction, relationship extraction and attribute extraction to generate preliminary knowledge representations, but these representations may still contain duplication, conflict and inconsistency.

[0082] Step 3: Build a data asset catalog based on the metadata after knowledge fusion and the knowledge extraction results of the metadata;

[0083] Knowledge fusion refers to the integration and fusion of knowledge from different sources to form a more comprehensive, consistent and accurate knowledge representation. After extraction, the extracted information is integrated and normalized to resolve redundancy and conflict between data from different sources and ensure that all extracted information is represented under a unified semantic framework.

[0084] Knowledge fusion mainly includes two parts: entity disambiguation and coreference resolution.

[0085] Entity disambiguation refers to solving the problem of entity reference ambiguity in text in natural language processing and information retrieval, that is, determining the specific entity that an entity reference should correspond to in a specific context. In the context of knowledge fusion, entity disambiguation is an important part of the knowledge fusion process, which is used to solve the problem of identifying and linking similar or identical entities in different data sources or knowledge graphs. When an entity reference appears in a text, there may be multiple candidate entities corresponding to it, which is caused by homonymous entities, synonyms, abbreviations, aliases, etc. The goal of entity disambiguation is to determine the correct meaning of the entity reference through contextual information and semantic reasoning technology, thereby eliminating the ambiguity of the entity and ensuring that it is correctly associated with the corresponding entity in the knowledge graph. The process of entity disambiguation usually includes the following steps: 1. Candidate entity generation: Based on the text form of the entity reference, a set of candidate entities are generated from the knowledge graph or other data sources, which may be related to the entity reference. 2. Context feature extraction: Analyze the context environment of the entity reference, including context vocabulary, syntactic structure, entity relationship, etc., and extract features that help distinguish candidate entities. 3. Feature matching and similarity calculation: Match the context features with the features of the candidate entities and calculate similarity to determine which candidate entity best matches the entity reference. 4. Disambiguation decision: Based on the results of matching and similarity calculation, select the best candidate entity as the disambiguation result of the entity reference. For example, the text "The equipment is a facility in the power grid that plays the role of voltage transformation, current transformation, and distribution of electric energy", then the entity candidate is equipment, and the entity in the knowledge graph is "Substation A: located in city X, belongs to the 500 kV level, and has a capacity of 100 MW". Entity disambiguation process: Based on the context information, we can infer that the "equipment" in the text refers to the facilities in the power grid for voltage transformation, current transformation, and distribution of electric energy. Based on this context, we can link the entity candidate "substation" to "substation A" in the knowledge graph, because they are both facilities in the power grid for voltage transformation and distribution of electric energy. The final link result: equipment->substation A.

[0086] Co-reference resolution refers to solving the problem of co-reference relationships in text, that is, determining the same entity referred to by different expressions in the text. The goal of co-reference resolution is to determine the specific entities referred to by different expressions through context analysis and reasoning techniques, and associate them into the same co-reference chain. Co-reference resolution can help understand the complex semantic relationships in the text and improve the accuracy and semantic understanding ability of the natural language processing system.

[0087] In the process of knowledge fusion, coreference resolution is an important link. When there are problems in identifying and linking similar or identical entities in different data sources or knowledge graphs, coreference resolution technology can help determine which entity references are mutually referenced and represent the same entity. By associating entity references with coreference relationships to the same entity representation, the ambiguity of entities can be eliminated and the accuracy and consistency of knowledge fusion can be improved. Coreference resolution is to solve the problem that multiple words (referent terms) correspond to the same entity. For example, the text: Current overload on the transmission line will lead to increased losses, and timely measures need to be taken to prevent damage. In addition, the equipment in the substation also needs to be regularly inspected to ensure its normal operation. Coreference resolution process: In this example, there are multiple coreference relationships that need to be resolved. First, "current overload on the transmission line" and "lead to increased losses" refer to the same entity, that is, the overload condition. Secondly, the phrases "need to take timely measures to prevent damage" and "in addition" refer to the overload condition mentioned above. Finally, "the equipment of the substation" and "its normal operation" refer to the same entity, that is, the equipment of the substation. Final analysis result: current overload on transmission line -> overload condition; resulting in increased losses -> overload condition; timely measures need to be taken to prevent damage -> overload condition; in addition -> overload condition; equipment in substation -> equipment in substation; its normal operation -> equipment in substation.

[0088] Through entity disambiguation and coreference resolution, the extracted entities are aligned.

[0089] Build a data asset catalog: Build a data asset catalog based on the metadata after knowledge fusion and the knowledge extraction results of the metadata. When building a data asset catalog, it is necessary to consider the company's data resource classification and data information description paradigm to formulate catalog structure specifications. This specification can be developed around the "system-function-table-field" hierarchy to ensure the integrity and ease of use of the data asset catalog. At the same time, by obtaining the system function menu from the unified permission system, each function can be combined with the corresponding data table catalog and the data flow relationship between tables to form a "business-data" correspondence.

[0090] Step 4: Define entity categories, attribute definitions, and relationship types, and build a knowledge graph; the details are as follows:

[0091] The following is the general knowledge modeling process:

[0092] Determine the domain and goal: Clarify the domain and goal to be modeled. For example, in the knowledge modeling of the power grid, the goal may be to describe the attributes and relationships of power equipment, substations, lines, etc.

[0093] Entity Modeling: Identify and define the entities that need to be modeled, which can be physical objects, concepts, events, etc. in the power grid. For example, generators, substations, loads, etc. Define unique identifiers and related attributes for each entity.

[0094] Relationship modeling: Define the relationships between entities, which can be connections, dependencies, hierarchies, etc. For example, the connection relationship between generators and substations, the dependency relationship between substations and lines, etc. Define unique identifiers and related attributes for each relationship.

[0095] Attribute modeling: define attributes for entities and relationships to describe their characteristics and properties, such as the power output attribute of a generator, the capacity attribute of a line, etc.

[0096] The following is the process of building a knowledge graph:

[0097] 1. Define entity types and their attributes based on entity modeling results, define relationship types and their constraints based on relationship modeling results, and then integrate these modeling results to build an ontology.

[0098] 2. Map the constructed ontology into the graph database and create corresponding nodes, edges and their attributes to effectively store and manage knowledge data.

[0099] 3. Use a graph query language, such as Cypher or SPARQL, to import the information in the data asset catalog into the ontology to obtain a knowledge graph that can be used to query metadata.

[0100] Step 5: Use the large language model to build a retrieval enhancement generation process to retrieve metadata.

[0101] Retrieval Augmented Generation (RAG) is an artificial intelligence technology that combines information retrieval technology with language generation models. It retrieves relevant information from external knowledge bases and inputs it as prompts to large language models (LLMs) to enhance the model's ability to handle knowledge-intensive tasks such as question answering, text summarization, and content generation.

[0102] In building a RAG-based system, the main steps include:

[0103] 1. Index construction (Indexing), as follows:

[0104] Select the documents and data to be included in the knowledge base, use the text segmenter to segment the documents according to paragraphs or other logic, and then use the embedding model (such as OpenAI's Embeddings) to generate semantic vectors and store them in the vector database. Then output the index library, which contains text blocks and corresponding semantic vectors.

[0105] The purpose of this step is to split the document into multiple text blocks, create a semantic vector for each block, and build an index for subsequent retrieval.

[0106] 2. Query Translation, as follows:

[0107] Get the user's natural language query. Depending on the application scenario, you may need to use natural language processing technology to convert the query into a specific format. Then output the converted query, which can be a SQL statement, design instruction, etc.

[0108] The purpose of this step: Convert user queries into language or instructions that the machine can understand.

[0109] 3. Routing, as follows:

[0110] Takes the transformed query. Implements a logical or semantic routing mechanism to direct the query to the most appropriate index or processing unit. Then outputs the path or pipeline that determines how the query is processed.

[0111] The purpose of this step is to select the appropriate processing flow based on the query content.

[0112] 4. Retrieval, as follows:

[0113] Get the user's query. Calculate the similarity between the query embedding and the embedding of each document block in the knowledge base, and return the most similar result. Then output the document block or information related to the query

[0114] The purpose of this step: Retrieve relevant information from the index library based on the query.

[0115] 5. Generation, as follows:

[0116] Get the user's query and the relevant document blocks retrieved. Provide the query and retrieval results as input to the language model, and the model generates answers based on this information. Finally, output the generated answers or content.

[0117] The purpose of this step: Use the large language model to generate accurate and relevant answers or content.

[0118] The advantage of RAG technology is that it can combine the latest data and information to provide more accurate and timely answers than traditional static knowledge-based systems. It is suitable for complex scenarios that require processing large amounts of data and dynamically updated information, such as power grid fault analysis, academic paper abstracts, and corporate knowledge base queries.

[0119] Step 6: Publish the data asset catalog so that all departments and units can easily access and use relevant data resources. At the same time, the knowledge graph used to query metadata is also published so that all departments and units can easily access and use relevant power knowledge to support the operation and management of the power system.

[0120] Those of ordinary skill in the art will appreciate that the units of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition of each example has been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0121] In the embodiments provided by the present invention, it should be understood that the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored, etc.

[0122] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0123] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-0nlyMemory), random access memory (RAM, RandomAccessMemory), mobile hard disk, magnetic disk or optical disk, etc., which can store program code.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A metadata query method based on knowledge graph, characterized in that: The following steps are involved: Collect metadata, which includes structured data, semi-structured data, and unstructured data; Extract knowledge from structured data, semi-structured data and unstructured data respectively. Knowledge extraction includes entity extraction, relationship extraction and attribute extraction. Build a data asset catalog based on the metadata after knowledge fusion and the knowledge extraction results of the metadata; Define entity categories, attribute definitions, and relationship types, perform knowledge modeling, and build a knowledge graph; Using large language models, we build a retrieval enhancement generation process to retrieve metadata.

2. According to the metadata query method based on knowledge graph according to claim 1, it is characterized in that: Knowledge extraction includes the extraction of structured data, as follows: Data identification and preprocessing: Clean structured data such as database tables or spreadsheets, including removing duplicate records, correcting errors, and processing missing values ​​to ensure data quality; Entity recognition: Use regular expressions, dictionary matching, or machine learning models to identify specific entities from data, including dates, numbers, and categorical labels; Relationship extraction: Identify relationships between entities based on relationships between fields or data patterns, including foreign key associations and formula calculation results; Entity linking: linking the identified entities to the corresponding entries in the knowledge base; Relation type normalization: Map the extracted relations to predefined relation types; Data standardization: unify data formats, units, etc.; Knowledge fusion: integrating structured data from different sources; Knowledge storage: Store the extracted structured data in a knowledge graph or other knowledge base to facilitate subsequent query and analysis.

3. According to the metadata query method based on knowledge graph according to claim 1, it is characterized in that: Knowledge extraction includes the extraction of unstructured data, as follows: Text preprocessing: cleaning text data, including noise removal, language standardization, word segmentation, and part-of-speech tagging; Entity recognition: Use natural language processing techniques, including named entity recognition (NER), to identify specific entities from text; Relation extraction: Identify the relationships between entities in text through pattern matching, supervised learning or deep learning methods; Event extraction: Identify the events and their participants described in the text, and extract the event types and arguments; Entity linking: linking entities in the text to corresponding entities in the knowledge base; Text summary: summarize the text content and extract key information; Sentiment analysis: Analyze the sentiment tendency of the text and extract the sentiment polarity; Knowledge storage: Store the extracted entities, relationships, events, etc. into the knowledge base to form a structured knowledge representation.

4. According to the metadata query method based on knowledge graph according to claim 1, it is characterized in that: Knowledge extraction includes the extraction of semi-structured data, as follows: Data identification and preprocessing: standardize the format and clean the semi-structured data such as XML and HTML documents; Entity recognition: using labels or attribute information in the data to identify entities in semi-structured data; Attribute extraction: extracting attribute information of entities, including class and id in HTML or fields in XML; Relationship extraction: Identify the relationship between entities by analyzing the nested relationships and label structure of the data; Entity linking: matching extracted entities with entities in the knowledge base; Data fusion: integrating information from multiple semi-structured data sources; Data standardization: converting the extracted data into a unified format; Knowledge storage: Store the extracted entities, attributes and relationships in the knowledge base to form a structured knowledge representation.

5. According to the metadata query method based on knowledge graph according to claim 1, it is characterized in that: The knowledge graph construction process is as follows: Define entity types and their attributes based on entity modeling results, define relationship types and their constraints based on relationship modeling results, and then integrate these modeling results to build an ontology; Map the constructed ontology into the graph database and create corresponding nodes, edges and their attributes; Using the graph query language, the information in the data asset catalog is imported into the ontology to obtain a knowledge graph that can be used to query metadata.

6. According to the metadata query method based on knowledge graph according to claim 1, it is characterized in that: The specific steps for querying metadata are as follows: Select documents and data to be included in the knowledge base, use text segmentation to segment documents according to paragraphs or other logic, and then use the embedding model to generate semantic vectors and store them in the vector database; Obtain the user's natural language query. Depending on the application scenario, you may need to use natural language processing technology to convert the query into a specific format; Take the transformed query and implement a logical or semantic routing mechanism to direct the query to the most appropriate index or processing unit; Get the user's query, calculate the similarity between the query embedding and the embedding of each document block in the knowledge base, return the most similar result, and then output the document block or information related to the query; The user's query and the retrieved relevant document blocks are obtained, and the query and retrieval results are provided as input to the language model, and the model generates answers based on this information.

7. The metadata query method based on knowledge graph according to claim 1 is characterized in that: The following steps are also included: Publish a data asset catalogue so that all departments and units can access and use relevant data resources.

8. The metadata query method based on knowledge graph according to claim 1, characterized in that: The following steps are also included: Publish a knowledge graph for querying metadata so that various departments and units can access and utilize it.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the metadata query method based on the knowledge graph as described in any one of claims 1 to 8.

10. A processor, characterized in that: The processor is used to run a program, wherein the program executes the metadata query method based on the knowledge graph described in any one of claims 1 to 8 when running.

Images