Knowledge graph construction method and device

Abstract

The invention discloses a knowledge graph construction method and device. The method comprises the following steps: acquiring multiple groups of initial text data of a target technical field from multiple data sources; for each group of initial text data, performing word segmentation preprocessing on the initial text data to obtain standard text data; performing semantic analysis on the standard text data to obtain multi-dimensional text semantic information; analyzing the multi-dimensional text semantic information by using a plurality of agents to obtain a knowledge entity set, a relationship set and an attribute set; and constructing a knowledge graph of the target technical field based on the knowledge entity set, the relationship set and the attribute set corresponding to the multiple groups of initial text data. The technical problem that a traditional knowledge graph construction method is poor in performance when coping with large-scale unstructured data is solved.

Description

Technical Field

[0001] The present application relates to the field of knowledge graph technology, and more specifically, to a method and device for constructing a knowledge graph. Background Art

[0002] In recent years, with the advent of the big data era, unstructured text data has seen explosive growth. This data contains a wealth of information, but it often exists in the form of free text, lacking a clear structure and format, making it difficult to use directly. To better utilize the information contained in this data, knowledge graph technology has emerged. Knowledge graphs are effective tools for representing information in a structured form. They can graphically display complex real-world concepts, entities, and their relationships, making them easier for computers to understand and process.

[0003] Currently, the main methods for constructing knowledge graphs include rule-based methods, machine learning methods, and hybrid methods. Among them, rule-based methods rely on a series of pre-defined patterns or rules to identify entities and relationships in text. Although this method performs well in certain specific fields, it is difficult to cope with the challenges of large-scale and diverse data sets due to the high complexity of rule formulation and high maintenance costs. On the other hand, although the development of machine learning, especially deep learning technology, has provided new ways to automatically extract knowledge, its training process usually requires a large amount of labeled data support, which is not only time-consuming and labor-intensive, but also may be difficult to obtain sufficient training samples in some emerging or niche fields. In addition, existing knowledge extraction technologies perform poorly when faced with linguistic phenomena with significant long-tail effects, that is, they are not accurate enough in identifying some less common but equally important concepts or relationships.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0005] The embodiments of the present application provide a knowledge graph construction method and device to at least solve the technical problem that traditional knowledge graph construction methods perform poorly when dealing with large-scale unstructured data.

[0006] According to one aspect of an embodiment of the present application, a knowledge graph construction method is provided, including: obtaining multiple groups of initial text data in the target technology field from multiple data sources; for each group of initial text data, performing word segmentation preprocessing on the initial text data to obtain standard text data; performing semantic analysis on the standard text data to obtain multi-dimensional text semantic information; using multiple intelligent agents to analyze the multi-dimensional text semantic information to obtain a knowledge entity set, a relationship set and an attribute set; and constructing a knowledge graph in the target technology field based on the knowledge entity set, relationship set and attribute set corresponding to the multiple groups of initial text data.

[0007] Optionally, multiple groups of initial text data in the target technical field are obtained from multiple data sources, including: obtaining multiple first files from the first data source using an application program interface provided by the first data source, and taking each first file as a group of initial text data; and / or, receiving a text data stream transmitted in real time by a second data source through a message queue, and dividing the text data stream with a preset data volume into a group of initial text data; and / or, receiving multiple second files uploaded by the target object, and taking each second file as a group of initial text data; wherein the types of the first data source and the second data source include at least one of the following: an internal enterprise document system, a social media platform, and a professional database; and the formats of the first file and the second file include at least one of the following: TXT format, DOCX format, and PDF format.

[0008] Optionally, the initial text data is pre-processed for word segmentation to obtain standard text data, including: performing data cleaning processing, text standardization processing, stop word filtering processing and sentence segmentation processing on the initial text data in sequence to obtain standard text data corresponding to the initial text data, wherein the data cleaning processing includes: non-text information denoising processing, and the text standardization processing includes at least one of the following: field format standardization processing and encoding format standardization processing.

[0009] Optionally, semantic analysis is performed on standard text data to obtain multi-dimensional text semantic information, including: performing word segmentation processing on the standard text data, and using the obtained text segmentation data as text semantic information of the first dimension, wherein the word segmentation processing, in addition to text segmentation, also includes at least one of the following: part-of-speech tagging and word form restoration; using a pre-trained large language model to perform named entity recognition on the text segmentation data to obtain multiple named entities and corresponding entity categories, and using the multiple named entities and corresponding entity categories as text semantic information of the second dimension; using a dependency syntax analysis tool to analyze the text segmentation data to obtain dependency relationships between each segmentation in the text segmentation data, and constructing a logical relationship chain based on the dependency relationship, and using the dependency relationship and the logical relationship chain as text semantic information of the third dimension; using a sentiment analysis tool to analyze the text segmentation data to obtain the sentiment polarity of each segmentation in the text segmentation data, and using the sentiment polarity as text semantic information of the fourth dimension; using a topic modeling algorithm to analyze the text segmentation data to obtain the topic distribution status of each segmentation in the text segmentation data, and using the topic distribution status as text semantic information of the fifth dimension.

[0010] Optionally, multiple agents are used to analyze multi-dimensional text semantic information to obtain a knowledge entity set, a relationship set and an attribute set, including: creating multiple knowledge extraction tasks based on multi-dimensional text semantic information, wherein the type of knowledge extraction task includes at least one of the following: knowledge entity extraction task, relationship extraction task, attribute extraction task; distributing each knowledge entity extraction task to multiple first-class agents based on load balancing technology, distributing each relationship extraction task to multiple second-class agents, and distributing each attribute extraction task to multiple third-class agents; the knowledge entities extracted by each first-class agent are organized into a knowledge entity set, the relationships extracted by each second-class agent are organized into a relationship set, and the attributes extracted by each third-class agent are organized into an attribute set, wherein the attributes include at least one of the following: knowledge entity attributes, relationship attributes.

[0011] Optionally, a knowledge graph of the target technology field is constructed based on the knowledge entity sets, relationship sets and attribute sets corresponding to multiple groups of initial text data, including: taking each knowledge entity in the knowledge entity set as each node in the knowledge graph; defining edges between corresponding nodes based on each relationship in the relationship set; filling each attribute in the attribute set into the corresponding knowledge entity or edge to obtain a knowledge graph of the target technology field; and storing the knowledge graph in a preset format in a graph database.

[0012] Optionally, the above method also includes: responding to a knowledge query instruction input in the human-computer interaction interface, determining the target knowledge corresponding to the knowledge query instruction from the knowledge graph, and feeding back the target knowledge in the human-computer interaction interface, wherein the type of the target knowledge includes at least one of the following: knowledge entity, edge; responding to a knowledge graph display instruction input in the human-computer interaction interface, displaying the knowledge graph in the human-computer interaction interface.

[0013] Optionally, the above method also includes: periodically counting the query frequency of each knowledge entity and each edge in the knowledge graph; determining that the knowledge entities with a query frequency higher than a preset threshold are hot knowledge entities, and determining that the edges with a query frequency higher than a preset threshold are hot edges; and creating indexes for hot knowledge entities and hot edges.

[0014] Optionally, the above method also includes: real-time monitoring of a distributed system integrating multiple intelligent agents and each intelligent agent to obtain multiple performance indicators, wherein the types of performance indicators include at least one of the following: system CPU utilization, memory usage, processing delay, and intelligent agent operating status; when any performance indicator does not meet the preset normal indicator range, the distributed system or intelligent agent is optimized using a fault-tolerant mechanism, wherein the fault-tolerant mechanism includes at least one of the following: load balancing, intelligent degradation, and fault restart.

[0015] According to another aspect of an embodiment of the present application, a knowledge graph construction device is also provided, including: an acquisition module for acquiring multiple groups of initial text data in the target technology field from multiple data sources; a processing module for performing word segmentation preprocessing on the initial text data for each group of initial text data to obtain standard text data; performing semantic analysis on the standard text data to obtain multi-dimensional text semantic information; using multiple intelligent agents to analyze the multi-dimensional text semantic information to obtain a knowledge entity set, a relationship set and an attribute set; a construction module for constructing a knowledge graph in the target technology field based on the knowledge entity set, relationship set and attribute set corresponding to the multiple groups of initial text data.

[0016] According to another aspect of an embodiment of the present application, a computer program product is also provided, which includes: a computer program, wherein the computer program implements the above-mentioned knowledge graph construction method when executed by a processor.

[0017] According to another aspect of an embodiment of the present application, an electronic device is also provided, which includes: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned knowledge graph construction method through the computer program.

[0018] In the embodiment of the present application, by performing semantic analysis on the standard text after word segmentation preprocessing, the efficiency and accuracy of text annotation can be effectively improved, the dependence on manual annotation is reduced, and the analysis of multi-dimensional text semantic information can be realized by using multiple intelligent agents working together. It can more comprehensively, accurately and efficiently extract knowledge entities, relationships and attributes from large amounts of text data, thereby constructing a high-quality knowledge graph. This method not only improves the efficiency of knowledge graph construction, but also enhances the integrity and accuracy of the knowledge graph, making it show significant advantages in knowledge retrieval, analysis and understanding in the technical field, thereby solving the technical problem that traditional knowledge graph construction methods perform poorly when dealing with large-scale unstructured data. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 is a schematic diagram of an optional knowledge graph construction method according to an embodiment of the present application;

[0021] Figure 2 This is a flowchart of an optional complete knowledge graph construction method according to an embodiment of the present application;

[0022] Figure 3This is a schematic structural diagram of an optional knowledge graph construction device according to an embodiment of the present application;

[0023] Figure 4 It is a schematic structural diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0025] It should be noted that the terms "first", "second", etc. in the specification, claims, and drawings of the present application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0026] In order to better understand the embodiments of the present application, some nouns or terms that appear in the description of the embodiments of the present application are first translated and explained as follows:

[0027] Agent: An agent is a software entity capable of autonomously performing specific tasks, making decisions, and interacting with other systems or users. In this invention, agent t is designed to process semantically parsed text data and perform knowledge extraction tasks such as entity recognition, relationship extraction, and attribute extraction. The agent's multitasking and adaptive learning capabilities enable the automated construction and maintenance of knowledge graphs.

[0028] Large language models: Large language models, such as GPT-3 and BERT, are natural language processing models with a large number of parameters and trained on large datasets. These models can understand and generate complex natural language and perform tasks such as text generation, translation, summarization, and semantic parsing. Large language models can be used to perform in-depth semantic analysis of unstructured text, improving the accuracy and comprehensiveness of knowledge extraction.

[0029] Knowledge graph: A knowledge graph is a graphical representation of knowledge that uses nodes and edges to represent entities and their relationships. Knowledge graphs can be used to organize and store knowledge elements extracted from unstructured text, such as disease entities and drug entities, as well as their relationships, such as therapeutic relationships. The construction of knowledge graphs facilitates systematic information management and efficient querying, supporting subsequent data analysis and intelligent decision-making.

[0030] Entity recognition, also known as named entity recognition, involves identifying meaningful entities in text, such as names of people, places, organizations, and diseases. Entity recognition is the foundation of knowledge graph construction, helping the system understand key information points in text.

[0031] Dependency parsing: Dependency parsing is a grammatical analysis method that focuses on the dependencies between words within a sentence and can reveal the deep structure and meaning of a sentence. Through dependency parsing, the system can understand the logical connections between words in a sentence, thereby more accurately extracting the relationships between entities. In this invention, dependency parsing is a key step in building the knowledge graph relationship chain, which enables the system to identify complex associations between entities, such as the relationship between diseases and symptoms.

[0032] Graph databases: Graph databases are database systems specifically designed for storing and querying graph data, efficiently processing complex relational data. In this paper, graph databases such as Neo4j and JanusGraph are used to store the constructed knowledge graph. Their powerful data models and query languages ​​enable efficient storage and flexible querying of the knowledge graph. The use of graph databases ensures high performance and scalability of the knowledge graph, meeting the needs of large-scale knowledge storage and real-time knowledge retrieval.

[0033] Example 1

[0034] According to an embodiment of the present application, a knowledge graph construction method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0035] Figure 1 This is a flow chart of a knowledge graph construction method provided in accordance with an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0036] Step S102, acquiring multiple groups of initial text data in the target technical field from multiple data sources;

[0037] Step S104: For each set of initial text data, perform word segmentation preprocessing on the initial text data to obtain standard text data; perform semantic analysis on the standard text data to obtain multi-dimensional text semantic information; and use multiple intelligent agents to analyze the multi-dimensional text semantic information to obtain a knowledge entity set, a relationship set, and an attribute set.

[0038] Step S106: construct a knowledge graph in the target technology field based on the knowledge entity sets, relationship sets, and attribute sets corresponding to the multiple groups of initial text data.

[0039] The following describes the various steps of the knowledge graph construction method in combination with the specific implementation process.

[0040] Figure 2 A more complete flow chart of the knowledge graph construction method is shown in Figure 2 It can be seen that obtaining multiple sets of initial text data in the target technology field from multiple data sources is the beginning of the entire knowledge graph construction method process. Specifically, multiple sets of initial text data can be obtained in the following ways: using the application interface provided by the first data source to obtain multiple first files from the first data source, and taking each first file as a set of initial text data; and / or, receiving the text data stream transmitted in real time by the second data source through the message queue, and dividing the text data stream of a preset data volume into a set of initial text data; and / or, receiving multiple second files uploaded by the target object, and taking each second file as a set of initial text data; wherein the types of the first data source and the second data source include at least one of the following: internal enterprise document system, social media platform, professional database; the formats of the first file and the second file include at least one of the following: TXT format, DOCX format, PDF format.

[0041] The acquired text data may be a file or a text data stream. Specifically, when the acquired text data is a file, it can be acquired through an application document interface provided by different channels such as an internal enterprise document system, a social media platform, a professional database, etc., or it can be acquired by receiving files uploaded by the user himself. When the acquired text data is a text data stream, it can be acquired in real time through a message queue associated with different data channels such as an internal enterprise document system, a social media platform, a professional database, etc. Furthermore, when the acquired text data is text, the text format includes common formats such as TXT format, DOCX format, PDF format, etc.

[0042] Acquiring data from data sources of different channels ensures the diversity of data acquisition. For example, using application programming interfaces to obtain data from the company's internal document system can update the knowledge graph in real time to reflect the latest technological developments within the company; receiving data streams through message queues can process large-scale real-time data, which is suitable for scenarios with frequent dynamic updates such as social media; receiving files uploaded by target objects can integrate in-depth information in specific fields in real time. These multi-channel and multi-type acquisition modes not only improve the efficiency and flexibility of data acquisition, but also solve the problems of single data source and slow update in traditional knowledge graph construction, so that the constructed knowledge graph can more quickly and comprehensively reflect the latest developments in the target technology field.

[0043] As an optional implementation, for the multiple groups of initial texts obtained, word segmentation preprocessing operations can be performed on them in the following ways to obtain standard text data: data cleaning processing, text standardization processing, stop word filtering processing and sentence segmentation processing are performed on the initial text data in turn to obtain standard text data corresponding to the initial text data, wherein the data cleaning processing includes: non-text information denoising processing, and the text standardization processing includes at least one of the following: field format standardization processing, encoding format standardization processing.

[0044] In the above process, word segmentation preprocessing is the foundation of knowledge graph construction. Data cleaning removes irrelevant non-text information, such as images, tables, HTML tags, reference lists, and other non-textual information, ensuring the accuracy of subsequent analysis. Text standardization, such as standardizing field formats, encoding formats, date formats, and unit representations, resolves format inconsistencies caused by diverse data sources and improves data processing efficiency. Stop word filtering removes common stop words and reduces noise interference. Sentence segmentation further optimizes text structure by dividing sentences according to rules, making subsequent intelligent agent analysis more accurate. The combined application of these preprocessing steps addresses the uneven quality of raw text data, providing high-quality input for subsequent semantic parsing and intelligent agent analysis, thereby laying the foundation for building a more accurate and efficient knowledge graph.

[0045] As an optional implementation, after obtaining the standard text data, the standard text data can be semantically parsed in the following ways to obtain multi-dimensional text semantic information: perform word segmentation on the standard text data, and use the obtained text word segmentation data as the text semantic information of the first dimension. Among them, the word segmentation process includes not only text word segmentation but also at least one of the following:词性标注、词形还原;use the pre-trained large language model to perform named entity recognition on the text word segmentation data to obtain multiple named entities and their corresponding entity categories, and use the multiple named entities and their corresponding entity categories as the text semantic information of the second dimension; use the dependency syntax analysis tool to analyze the text word segmentation data to obtain the dependency relationships between the individual words in the text word segmentation data, and construct a logical relationship chain based on the dependency relationships, and use the dependency relationships and the logical relationship chain as the text semantic information of the third dimension; use the sentiment analysis tool to analyze the text word segmentation data to obtain the sentiment polarity of each word in the text word segmentation data, and use the sentiment polarity as the text semantic information of the fourth dimension; use the topic modeling algorithm to analyze the text word segmentation data to obtain the topic distribution status of each word in the text word segmentation data, and use the topic distribution status as the text semantic information of the fifth dimension.

[0046] Among them, text word segmentation is the process of splitting continuous natural language text into independent lexical units. This step is the basis of natural language processing tasks because most natural language processing algorithms need to break down the text into the smallest understandable units to work. By performing word segmentation, it is easier to identify the word boundaries in a sentence, which is crucial for subsequent processes such as词性标注、命名实体识别、语法分析. The accuracy of word segmentation directly affects the efficiency and effectiveness of subsequent processing. The word segmentation process usually also involves removing punctuation marks and stop words (such as words like "的", "是", "和" that frequently appear in the text but carry less semantic information). This helps reduce the noise in the text and makes subsequent analysis more focused on key vocabulary.

[0047] Among them,词性标注 can determine the grammatical role of a word and assign a词性标签 (such as noun, verb, adjective, etc.) to each word in the text. This helps understand the grammatical function and role of the word in the sentence. Through词性标注, it is possible to more accurately identify the entities, attributes, and relationships in the text, which is crucial for constructing the nodes and edges in a knowledge graph and can improve the accuracy and integrity of the knowledge graph. In addition,词性标注 is a prerequisite for performing dependency syntax analysis or constituency syntax analysis, which can help the system understand the sentence structure, identify components such as the subject, verb, and object, and then infer the deep meaning of the sentence.

[0048] It should be noted that the Chinese terms "词性标注" and "词形还原" in the original text are not translated as there is no clear indication of their English equivalents in the rules. You may need to provide the correct English translations for these terms according to the actual situation.Among them, word form restoration refers to restoring the vocabulary in a sentence to its basic form in the dictionary, that is, the stem or root. This is critical for processing word inflections (such as tense changes, plural forms, etc.), and can ensure that different forms of the same word are regarded as the same concept. When constructing a knowledge graph, word form restoration can avoid missing the recognition of entities or relationships due to different forms of words. For example, whether "run" appears in a sentence as "runs", "running" or "ran", after word form restoration, it will be regarded as "run", thereby reducing the complexity of knowledge extraction and improving the efficiency of information retrieval. Word form restoration helps to simplify text analysis and statistical processes because analysts or algorithms can focus on the basic form of words without having to write additional processing logic for the multiple variations of words.

[0049] In summary, text segmentation, part-of-speech tagging, and lemmatization play a vital role in natural language processing. They work together to provide powerful tools for understanding and utilizing unstructured text data. Especially in the process of building knowledge graphs, these steps can significantly improve the accuracy and efficiency of knowledge extraction.

[0050] Multi-dimensional semantic parsing technology not only includes the above-mentioned word segmentation, part-of-speech tagging, and word form restoration, but also includes named entity recognition, dependency syntax analysis, sentiment analysis, and topic modeling. It can deeply understand the text content from different angles and facilitate the extraction of key information such as knowledge entities, relationships, and attributes.

[0051] In specific implementation, tools such as Stanford POS Tagger or SpaCy can be used to perform part-of-speech tagging on the word segmentation results. Part-of-speech tagging and word form restoration help understand the role and original form of words in sentences. The text segmentation data obtained through word segmentation processing can further lay the foundation for subsequent processing processes such as named entity recognition, dependency syntax analysis, sentiment analysis, and topic modeling.

[0052] Combined with a pre-trained large language model, named entity recognition can assist in the automatic identification of specific entities in text segmentation data, such as names of people, places, and technical terms, providing a foundation for the construction of knowledge graphs. The logical relationship chain constructed by dependency syntactic analysis can reveal the complex relationships between entities. Sentiment analysis and topic modeling enrich the content of the knowledge graph from the perspectives of sentimental tendencies and topic distribution, respectively. In specific implementations, tools such as TexBlob can be used as sentiment analysis tools to analyze the polarity of text segmentation data. The comprehensive application of these technical features solves the problems of traditional text analysis methods in terms of insufficient semantic understanding and incomplete information extraction. The constructed knowledge graph not only contains basic information about entities and relationships, but also incorporates multi-dimensional semantic information such as sentiment and topic, providing richer and more accurate data support for subsequent intelligent applications.

[0053] As an optional implementation method, after obtaining multi-dimensional text semantic information, the multi-dimensional text semantic information can be analyzed using multiple agents in the following manner to obtain a knowledge entity set, a relationship set, and an attribute set: creating multiple knowledge extraction tasks based on the multi-dimensional text semantic information, wherein the types of knowledge extraction tasks include at least one of the following: knowledge entity extraction task, relationship extraction task, attribute extraction task; distributing each knowledge entity extraction task to multiple first-class agents based on load balancing technology, distributing each relationship extraction task to multiple second-class agents, and distributing each attribute extraction task to multiple third-class agents; organizing the knowledge entities extracted by each first-class agent into a knowledge entity set, organizing the relationships extracted by each second-class agent into a relationship set, and organizing the attributes extracted by each third-class agent into an attribute set, wherein the attributes include at least one of the following: knowledge entity attributes, relationship attributes.

[0054] Among them, different types of analysis tasks are distributed to specialized intelligent agents through load balancing technology, which not only improves the efficiency of task processing but also ensures the accuracy and depth of analysis.

[0055] For example, knowledge entity extraction tasks are handled by the first type of agents, which may have been specially trained to more accurately identify and classify entities; relationship extraction tasks are handled by the second type of agents, which can more effectively analyze the relationships between entities; and attribute extraction tasks are performed by the third type of agents, which can deeply explore the characteristics of entities and relationships. This division of labor and cooperation among agents constructs a knowledge graph that is not only large-scale, comprehensive, and highly accurate, but can better support knowledge retrieval and analysis in technical fields, thereby solving the problems of low task processing efficiency and insufficient analytical depth in traditional knowledge graph construction.

[0056] As an optional implementation method, after determining the knowledge entity set, relationship set and attribute set, the knowledge graph of the target technology field can be constructed in the following way: each knowledge entity in the knowledge entity set is used as a node in the knowledge graph; the edges between the corresponding nodes are defined based on each relationship in the relationship set; each attribute in the attribute set is filled into the corresponding knowledge entity or edge to obtain the knowledge graph of the target technology field; and the knowledge graph is stored in a preset format in a graph database.

[0057] The above process reflects the construction and storage process of a knowledge graph. By integrating knowledge entities, relationships, and attributes into nodes, edges, and attributes, a structured knowledge network topology is formed. This structured storage method not only facilitates knowledge query and analysis but also leverages the characteristics of graph databases, such as efficient query and association analysis, to further improve the efficiency of knowledge graph utilization. For example, using the query language of a graph database, relationships with specific attributes can be quickly located, which is of great significance for scenarios such as technology trend analysis and innovation discovery. The combination of these technical features addresses the difficulties of information retrieval and the complexity of association analysis in traditional knowledge storage methods. The constructed graph can more efficiently and intuitively display the relationships between knowledge, providing users with more convenient knowledge retrieval and analysis services.

[0058] Furthermore, the query and display of the graph database can be achieved in the following ways: in response to the knowledge query instruction input in the human-computer interaction interface, the target knowledge corresponding to the knowledge query instruction is determined from the knowledge graph, and the target knowledge is fed back in the human-computer interaction interface, wherein the types of target knowledge include but are not limited to: knowledge entities, edges; in response to the knowledge graph display instruction input in the human-computer interaction interface, the knowledge graph is displayed in the human-computer interaction interface.

[0059] Among them, the query and display functions of the knowledge graph enable users to intuitively obtain and understand the information in the knowledge graph. For example, by entering specific query instructions, users can quickly find knowledge entities and relationships related to technology trends, innovations, etc., which is of great value for technology research and decision support. The display function can present the knowledge graph in a graphical manner, making the connections and structure between knowledge clear at a glance, improving the readability and comprehension of the information. The implementation of these functions solves the problem of users having difficulty quickly locating and understanding key knowledge in massive amounts of information, making the knowledge graph an effective tool for users to obtain and analyze knowledge in the field of technology.

[0060] Furthermore, in order to optimize the query efficiency of the knowledge graph, the query frequency of each knowledge entity and each edge in the knowledge graph can be periodically counted; the knowledge entities with a query frequency higher than a preset threshold are determined as hot knowledge entities, and the edges with a query frequency higher than the preset threshold are determined as hot edges; and indexes are created for hot knowledge entities and hot edges.

[0061] By counting and analyzing the query frequency of entities and edges in the knowledge graph, we can identify hot topics of interest to users and create indexes for these topics, significantly improving query efficiency. For example, by creating indexes for frequently queried technical terms or technical relationships, users can access relevant information more quickly, which is crucial for scenarios such as real-time decision support and rapid knowledge retrieval. This optimization strategy addresses the low query efficiency of knowledge graphs in large-scale data, enabling them to better serve users' real-time needs.

[0062] Furthermore, since the construction of the knowledge graph depends on different types of intelligent agents, it is necessary to implement corresponding monitoring measures and fault recovery measures for the intelligent agents. Specifically, this can be done in the following ways: real-time monitoring of the distributed system that integrates multiple intelligent agents and each intelligent agent to obtain multiple performance indicators, where the types of performance indicators include but are not limited to: system CPU utilization, memory usage, processing delay, and intelligent agent operating status; when any performance indicator does not meet the preset normal indicator range, the distributed system or intelligent agent is optimized using a fault-tolerant mechanism, where the fault-tolerant mechanism includes but is not limited to: load balancing, intelligent degradation, and fault restart.

[0063] Specifically, Prometheus or Grafana can be used as real-time monitoring tools to monitor different types of indicators. The introduction of real-time monitoring and fault-tolerance mechanisms in the above process ensures the stability and reliability of the knowledge graph construction process. For example, monitoring the system's CPU utilization and memory usage can promptly identify resource bottlenecks, optimize through load balancing technology, and avoid system overload. Monitoring the operating status and processing delays of intelligent agents can ensure the timeliness of data processing and the healthy operation of intelligent agents, and improve the system's fault tolerance through mechanisms such as intelligent degradation and fault restart. These monitoring and optimization measures address common performance bottlenecks and failure issues in distributed systems, ensuring the efficiency and stability of knowledge graph construction, thereby providing users with a high-quality user experience.

[0064] Through the above steps, semantic parsing of the standard text after word segmentation preprocessing can effectively improve the efficiency and accuracy of text annotation, reduce reliance on manual annotation, and utilize multiple intelligent agents working together to analyze multi-dimensional text semantic information. This can more comprehensively, accurately, and efficiently extract knowledge entities, relationships, and attributes from large amounts of text data, thereby constructing a high-quality knowledge graph. This approach not only improves the efficiency of knowledge graph construction, but also enhances the integrity and accuracy of the knowledge graph, giving it significant advantages in knowledge retrieval, analysis, and understanding within the technical field. This solves the technical problem that traditional knowledge graph construction methods perform poorly when dealing with large-scale unstructured data.

[0065] Example 2

[0066] According to the embodiment of the present application, a knowledge graph construction device for implementing the knowledge graph construction method in embodiment 1 is also provided. Figure 3 As shown, the knowledge graph construction device includes at least: an acquisition module 31, a processing module 32 and a construction module 33, wherein:

[0067] An acquisition module 31 can acquire multiple sets of initial text data in the target technical field from multiple data sources;

[0068] The processing module 32 is used to perform word segmentation preprocessing on each set of initial text data to obtain standard text data; perform semantic analysis to obtain multi-dimensional text semantic information; and use multiple intelligent agents to analyze the multi-dimensional text semantic information to obtain knowledge entity sets, relationship sets, and attribute sets.

[0069] The construction module 33 is used to construct a knowledge graph in the target technology field based on the knowledge entity set, relationship set and attribute set corresponding to multiple groups of initial text data.

[0070] The following describes the functions of each module of the knowledge graph construction device in conjunction with the specific implementation process.

[0071] The acquisition module acquires multiple sets of initial text data in the target technology field from multiple data sources, which is the beginning of the entire knowledge graph construction method process. Specifically, the multiple sets of initial text data can be acquired in the following ways: using the application interface provided by the first data source to acquire multiple first files from the first data source, and taking each first file as a set of initial text data; and / or, receiving the text data stream transmitted in real time by the second data source through the message queue, and dividing the text data stream of a preset data volume into a set of initial text data; and / or, receiving multiple second files uploaded by the target object, and taking each second file as a set of initial text data; wherein the types of the first data source and the second data source include at least one of the following: internal enterprise document system, social media platform, professional database; the formats of the first file and the second file include at least one of the following: TXT format, DOCX format, PDF format.

[0072] As an optional implementation, for the multiple groups of initial texts acquired by the acquisition module, the processing module can first perform word segmentation preprocessing operations on them in the following ways to obtain standard text data: perform data cleaning processing, text standardization processing, stop word filtering processing and sentence segmentation processing on the initial text data in turn to obtain standard text data corresponding to the initial text data, wherein the data cleaning processing includes: non-text information denoising processing, and the text standardization processing includes at least one of the following: field format standardization processing, encoding format standardization processing.

[0073] As an optional implementation method, after obtaining the standard text data, the processing module can perform semantic analysis on the standard text data in the following ways to obtain multi-dimensional text semantic information: perform word segmentation processing on the standard text data, and use the obtained text word segmentation data as the text semantic information of the first dimension, wherein the word segmentation processing includes not only text word segmentation, but also at least one of the following: part-of-speech tagging, word form restoration; perform named entity recognition on the text word segmentation data using a pre-trained large language model to obtain multiple named entities and corresponding entity categories, and use the multiple named entities and corresponding entity categories as the text semantic information of the second dimension. semantic information; use dependency syntax analysis tools to analyze text segmentation data, obtain the dependency relationship between each segmentation in the text segmentation data, and build a logical relationship chain based on the dependency relationship, and use the dependency relationship and logical relationship chain as the third dimension of text semantic information; use sentiment analysis tools to analyze text segmentation data, obtain the sentiment polarity of each segmentation in the text segmentation data, and use the sentiment polarity as the fourth dimension of text semantic information; use topic modeling algorithm to analyze text segmentation data, obtain the topic distribution status of each segmentation in the text segmentation data, and use the topic distribution status as the fifth dimension of text semantic information.

[0074] As an optional implementation method, after obtaining multi-dimensional text semantic information, the processing module can also use multiple agents to analyze the multi-dimensional text semantic information in the following manner to obtain a knowledge entity set, a relationship set, and an attribute set: create multiple knowledge extraction tasks based on the multi-dimensional text semantic information, wherein the type of knowledge extraction task includes at least one of the following: knowledge entity extraction task, relationship extraction task, attribute extraction task; based on load balancing technology, distribute each knowledge entity extraction task to multiple first-class agents, distribute each relationship extraction task to multiple second-class agents, and distribute each attribute extraction task to multiple third-class agents; organize the knowledge entities extracted by each first-class agent into a knowledge entity set, organize the relationships extracted by each second-class agent into a relationship set, and organize the attributes extracted by each third-class agent into an attribute set, wherein the attributes include at least one of the following: knowledge entity attributes, relationship attributes.

[0075] As an optional implementation, after determining the knowledge entity set, relationship set and attribute set, the construction module can construct a knowledge graph in the target technology field in the following way: use each knowledge entity in the knowledge entity set as each node in the knowledge graph; define the edges between corresponding nodes based on each relationship in the relationship set; fill each attribute in the attribute set into the corresponding knowledge entity or edge to obtain the knowledge graph of the target technology field; store the knowledge graph in a preset format in the graph database.

[0076] Optionally, the knowledge graph construction device may also include a query display module and a monitoring optimization module.

[0077] Furthermore, the query and display module of the graph database can be implemented in the following ways: in response to the knowledge query instruction input in the human-computer interaction interface, the target knowledge corresponding to the knowledge query instruction is determined from the knowledge graph, and the target knowledge is fed back in the human-computer interaction interface, wherein the types of target knowledge include but are not limited to: knowledge entities, edges; in response to the knowledge graph display instruction input in the human-computer interaction interface, the knowledge graph is displayed in the human-computer interaction interface.

[0078] Furthermore, in order to optimize the query efficiency of the knowledge graph, the construction module can also periodically count the query frequency of each knowledge entity and each edge in the knowledge graph; determine the knowledge entities with a query frequency higher than a preset threshold as hot knowledge entities, and determine the edges with a query frequency higher than a preset threshold as hot edges; create indexes for hot knowledge entities and hot edges.

[0079] Furthermore, since the construction of the knowledge graph depends on different types of intelligent agents, it is necessary for the monitoring and optimization module to implement corresponding monitoring measures and fault recovery measures for the intelligent agents. Specifically, this can be done in the following ways: real-time monitoring of the distributed system that integrates multiple intelligent agents and each intelligent agent to obtain multiple performance indicators, where the types of performance indicators include but are not limited to: system CPU utilization, memory usage, processing delay, and intelligent agent operating status; when any performance indicator does not meet the preset normal indicator range, the distributed system or intelligent agent is optimized using a fault-tolerant mechanism, where the fault-tolerant mechanism includes but is not limited to: load balancing, intelligent degradation, and fault restart.

[0080] It should be noted that each module in the knowledge graph construction device in the embodiment of the present application corresponds one-to-one to each implementation step of the knowledge graph construction method in Example 1. Since a detailed description has been given in Example 1, some details not reflected in this embodiment can be referred to Example 1 and will not be elaborated here.

[0081] Example 3

[0082] According to an embodiment of the present application, a computer program product is also provided, which includes a computer program, wherein when the computer program is executed by a processor, it implements the knowledge graph construction method in Example 1.

[0083] According to an embodiment of the present application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the knowledge graph construction method in Example 1 by running the computer program.

[0084] According to an embodiment of the present application, a processor is also provided, which is used to run a computer program, wherein the knowledge graph construction method in Example 1 is executed when the computer program is running.

[0085] According to an embodiment of the present application, an electronic device is also provided, which includes: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the knowledge graph construction method in Example 1 through the computer program.

[0086] Specifically, when the computer program is running, the following steps are executed: multiple groups of initial text data in the target technology field are obtained from multiple data sources; for each group of initial text data, the initial text data is preprocessed with word segmentation to obtain standard text data; the standard text data is semantically parsed to obtain multi-dimensional text semantic information; the multi-dimensional text semantic information is analyzed by multiple intelligent agents to obtain knowledge entity sets, relationship sets and attribute sets; and a knowledge graph of the target technology field is constructed based on the knowledge entity sets, relationship sets and attribute sets corresponding to the multiple groups of initial text data.

[0087] As an optional implementation, the electronic device may be in the form of a mobile terminal, a computer terminal or a similar computing device. Figure 4 The figure shows a hardware structure block diagram of an electronic device for implementing a knowledge graph construction method. Figure 4 As shown, the electronic device 40 may include one or more (402a, 402b, ..., 402n are shown in the figure) processors 402 (the processor 402 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 404 for storing data, and a transmission device 406 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 4 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 4 More or fewer components than shown, or with Figure 4 Different configurations shown.

[0088] It should be noted that the one or more processors 402 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single, independent processing module, or may be incorporated in whole or in part into any of the other components of the electronic device 40. As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0089] The memory 404 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the knowledge graph construction method in the embodiment of the present application. The processor 402 executes various functional applications and data processing by running the software programs and modules stored in the memory 404, that is, implementing the vulnerability detection method of the above-mentioned application. The memory 404 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 404 may further include a memory remotely located relative to the processor 402, and these remote memories may be connected to the electronic device 40 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0090] Transmission device 406 is used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by the communications provider of electronic device 40. In one embodiment, transmission device 406 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 406 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0091] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the electronic device 40 .

[0092] The serial numbers of the above embodiments are for description only and do not represent the advantages or disadvantages of the embodiments.

[0093] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0094] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0095] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected to achieve the purpose of the present embodiment according to actual needs.

[0096] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0097] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program code.

[0098] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A knowledge graph construction method, characterized in that: include: Acquire multiple sets of initial text data in the target technical field from multiple data sources; For each set of initial text data, the initial text data is pre-processed with word segmentation to obtain standard text data; the standard text data is semantically parsed to obtain multi-dimensional text semantic information; the multi-dimensional text semantic information is analyzed by multiple intelligent agents to obtain a knowledge entity set, a relationship set, and an attribute set; A knowledge graph of the target technology field is constructed based on the knowledge entity sets, relationship sets and attribute sets corresponding to multiple groups of the initial text data.

2. The method according to claim 1, characterized in that Acquire multiple sets of initial text data in the target technology area from multiple data sources, including: Obtaining a plurality of first files from a first data source using an application program interface provided by the first data source, and using each of the first files as a set of initial text data; and / or, receiving a text data stream transmitted in real time by a second data source through a message queue, and dividing a preset amount of the text data stream into a group of initial text data; and / or, receiving a plurality of second files uploaded by a target object, and treating each of the second files as a set of initial text data; The types of the first data source and the second data source include at least one of the following: an internal enterprise document system, a social media platform, and a professional database; the formats of the first file and the second file include at least one of the following: TXT format, DOCX format, and PDF format.

3. The method according to claim 1, characterized in that Performing word segmentation preprocessing on the initial text data to obtain standard text data includes: The initial text data is sequentially subjected to data cleaning processing, text standardization processing, stop word filtering processing and sentence segmentation processing to obtain standard text data corresponding to the initial text data, wherein the data cleaning processing includes: non-text information denoising processing, and the text standardization processing includes at least one of the following: field format standardization processing and encoding format standardization processing.

4. The method according to claim 1, wherein Perform semantic analysis on the standard text data to obtain multi-dimensional text semantic information, including: Performing word segmentation processing on the standard text data, and using the obtained text word segmentation data as text semantic information of the first dimension, wherein the word segmentation processing includes at least one of the following in addition to text word segmentation: part-of-speech tagging and word form restoration; Performing named entity recognition on the text segmentation data using a pre-trained large language model to obtain a plurality of named entities and corresponding entity categories, and using the plurality of named entities and corresponding entity categories as text semantic information of a second dimension; Analyzing the text segmentation data using a dependency syntax analysis tool to obtain dependency relationships between the individual segmentations in the text segmentation data, and constructing a logical relationship chain based on the dependency relationships, using the dependency relationships and the logical relationship chain as text semantic information of the third dimension; Analyzing the text segmentation data using a sentiment analysis tool to obtain the sentiment polarity of each segmentation in the text segmentation data, and using the sentiment polarity as text semantic information of the fourth dimension; The text segmentation data is analyzed using a topic modeling algorithm to obtain a topic distribution state of each segmentation in the text segmentation data, and the topic distribution state is used as the text semantic information of the fifth dimension.

5. The method according to claim 1, wherein Analyze the multi-dimensional text semantic information using multiple agents to obtain a knowledge entity set, a relationship set, and an attribute set, including: Creating a plurality of knowledge extraction tasks based on the multi-dimensional text semantic information, wherein the types of the knowledge extraction tasks include at least one of the following: a knowledge entity extraction task, a relationship extraction task, and an attribute extraction task; Based on load balancing technology, each knowledge entity extraction task is distributed to multiple first-class agents, each relationship extraction task is distributed to multiple second-class agents, and each attribute extraction task is distributed to multiple third-class agents; The knowledge entities extracted by each of the first-category intelligent agents are grouped into the knowledge entity set, the relationships extracted by each of the second-category intelligent agents are grouped into the relationship set, and the attributes extracted by each of the third-category intelligent agents are grouped into the attribute set, wherein the attributes include at least one of the following: knowledge entity attributes, relationship attributes.

6. The method according to claim 1, wherein Constructing a knowledge graph in the target technology field based on the knowledge entity sets, relationship sets, and attribute sets corresponding to multiple groups of the initial text data includes: Taking each knowledge entity in the knowledge entity set as each node in the knowledge graph; defining edges between corresponding nodes based on each relationship in the relationship set; Fill each attribute in the attribute set into the corresponding knowledge entity or edge to obtain the knowledge graph of the target technology field; The knowledge graph is stored in a preset format in a graph database.

7. The method according to claim 6, characterized in that The method further comprises: In response to a knowledge query instruction input in a human-computer interaction interface, target knowledge corresponding to the knowledge query instruction is determined from the knowledge graph, and the target knowledge is fed back in the human-computer interaction interface, wherein the type of the target knowledge includes at least one of the following: knowledge entity, edge; In response to a knowledge graph display instruction input in a human-computer interaction interface, the knowledge graph is displayed in the human-computer interaction interface.

8. The method according to claim 7, characterized in that The method further comprises: Periodically counting the query frequencies of each knowledge entity and each edge in the knowledge graph; Determine a knowledge entity whose query frequency is higher than a preset threshold as a hot knowledge entity, and determine an edge whose query frequency is higher than a preset threshold as a hot edge; An index is created for the hot knowledge entity and the hot edge.

9. The method according to claim 1, characterized in that The method further comprises: Monitor the distributed system integrating the multiple agents and each agent in real time to obtain a plurality of performance indicators, wherein the performance indicators include at least one of the following: system CPU utilization, memory usage, processing delay, and agent operation status; When any of the performance indicators does not meet the preset normal indicator range, the distributed system or the intelligent agent is optimized using a fault-tolerant mechanism, wherein the fault-tolerant mechanism includes at least one of the following: load balancing, intelligent degradation, and fault restart.

10. A knowledge graph construction device, characterized in that: include: An acquisition module, configured to acquire multiple sets of initial text data in a target technical field from multiple data sources; The processing module is configured to perform word segmentation preprocessing on each set of initial text data to obtain standard text data; perform semantic analysis on the standard text data to obtain multi-dimensional text semantic information; and respectively analyze the multi-dimensional text semantic information using multiple intelligent agents to obtain a knowledge entity set, a relationship set, and an attribute set; A construction module is used to construct a knowledge graph in the target technology field based on the knowledge entity sets, relationship sets and attribute sets corresponding to multiple groups of the initial text data.

11. A computer program product, characterized in that include: A computer program, wherein when the computer program is executed by a processor, it implements the knowledge graph construction method described in any one of claims 1 to 9.

12. An electronic device, characterized in that: include: A memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the knowledge graph construction method described in any one of claims 1 to 9 through the computer program.

Images