Knowledge base index construction optimization method and device based on large model

Abstract

The invention discloses a knowledge base index construction optimization method and device based on a large model, and the method comprises the steps: dividing a text document into a plurality of configuration blocks, carrying out the vectorization embedding of text blocks to obtain a vector database, and carrying out the extraction of graph elements and the combination of entities to obtain a combination result. Performing entity analysis on a merging result to obtain a structured entity and a relational data graph, grouping based on a relationship closeness degree between the structured entity and the relational data graph to obtain a community structure with similar characteristics, generating a community abstract report by utilizing a large model, integrating the community abstract report into a graph database, and finally, constructing a graph database based on a naive RAG method and a knowledge graph technology. And combining the vector database and the graph database to obtain a knowledge base multi-dimensional index. According to the method, the knowledge graph is adopted to construct a knowledge base index through complex connections and invisible relations in knowledge, meanwhile, a large model technology is adopted to intelligently assist the knowledge graph generation process, a knowledge community is constructed to supplement dominant and invisible relations between the knowledge, and the efficiency and accuracy of knowledge retrieval and generation are improved.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a method and device for optimizing knowledge base index construction based on a large model. Background Art

[0002] With the development of deep learning and natural language processing technology, large-scale pre-trained models have shown excellent performance in many fields, but they still face many challenges in practical applications, such as hallucination phenomena, limitations of professional knowledge, knowledge timeliness and unexplainability. These problems limit the widespread application of large models in specific industries and professional fields. To this end, knowledge retrieval enhanced generation (RAG) technology has gradually attracted attention as an effective solution. RAG technology achieves effective use and rapid response of knowledge through steps such as document segmentation, vectorization embedding, retrieval recall, formulation of context generation strategy, and selection of model generation response. However, RAG technology performs poorly in dealing with global, general and relatively complex problems, especially when dealing with large-scale heterogeneous multi-source unstructured knowledge, its retrieval efficiency and accuracy are difficult to meet actual needs.

[0003] In response to the above problems, researchers have proposed a variety of optimization solutions, such as adjusting the size of document blocks and their overlap, expanding the question description, and optimizing the sorting of retrieval results. Although these measures can improve the performance of the RAG system to a certain extent, they do not fundamentally solve the problem. In order to further improve the practicality and effectiveness of RAG technology, it is necessary to explore new knowledge indexing methods.

[0004] In this context, knowledge graph technology has become an ideal choice for building an efficient knowledge base due to its powerful graph structure expression ability, multi-level associations, and clear traceability. By building a knowledge graph, private knowledge within the enterprise can be better organized and managed, and the speed and accuracy of knowledge retrieval can be improved. However, the construction of knowledge graphs also faces a series of challenges, mainly including a large amount of manual annotation work, complex entity recognition and relationship extraction tasks, and semantic ambiguity and context understanding problems when processing natural language. In addition, the general generalization ability of knowledge graphs is relatively weak, which limits its application scope in different scenarios. Summary of the invention

[0005] The purpose of this application is to overcome the existing technical defects and provide a knowledge base index construction optimization method and device based on a large model. The knowledge graph technology is used to enrich the reasoning context of the large model, and the knowledge graph generation process is intelligently realized using the large model technology. At the same time, a community retrieval step is added to build a knowledge community, and the explicit and implicit relationships between knowledge are completed. Examples of related prompt word design are also given.

[0006] The purpose of this application is achieved through the following technical solutions:

[0007] In a first aspect, the present application proposes a knowledge base index construction optimization method based on a large model, the method comprising:

[0008] Divide the text document into multiple configuration blocks according to the block configuration block settings;

[0009] The configuration blocks are respectively subjected to text block vectorization embedding to obtain a vector database, and the configuration blocks are subjected to graph element extraction and entity merging to obtain a merging result;

[0010] The semantic recognition capability of the large model and the external synonym library are used to perform entity parsing on the merged results to obtain a structured entity and relationship data graph;

[0011] Using the recursive community clustering algorithm, the entities and relationship data graphs are grouped based on the closeness of the relationships between them to obtain a community structure with similar characteristics.

[0012] Based on the entity-relationship function graph, similarity feature community structure and node vector, a large model is used to generate a community summary report, and the community summary report is integrated into the graph database;

[0013] Based on the naive RAG method and knowledge graph technology, the vector database and graph database are combined to obtain a multidimensional index of the knowledge base.

[0014] In a possible implementation, the step of extracting graph elements and merging entities from the configuration blocks to obtain a merged result includes:

[0015] Combined with the prompt words, the large model is called and the thinking chain technology is used to extract the basic units of the graphic elements to obtain the graphics of each text block;

[0016] The same entities are extracted from the graph of each text block, and merged by constructing a description array according to the entity name and entity type to obtain the merged result.

[0017] In a possible implementation, the basic units of graph elements include entities, relationships, and declarations.

[0018] In a possible implementation, the steps of using a recursive community clustering algorithm to group entities based on the closeness of relationships between structured entity and relationship data graphs to obtain a community structure with similar characteristics include:

[0019] Set the community size threshold to determine the maximum number of nodes in each community;

[0020] Find the largest connected subgraph from the structured entity and relationship data graph based on the community size threshold, and standardize the node names in the largest connected subgraph;

[0021] The undirected graph is converted into a directed graph, and then the recursive community clustering algorithm is used to hierarchically cluster the graph to obtain a community structure with similar characteristics.

[0022] In a possible implementation, the method further includes: generating a vector representation of the graph using a Node2Vec algorithm.

[0023] In a possible implementation, based on the entity relationship function graph, the similarity characteristic community structure and the node vector, a large model is used to generate a community summary report, and the steps of integrating the community summary report into the graph database include:

[0024] A summary report for each community is generated using a large model based on entity-relationship function graphs, similarity feature community structures, and node vectors;

[0025] Use the large model to get reports of the entire graph from multiple granularity points, or perform local cluster summarization to generate community summary reports, and integrate the community summary reports into the graph database.

[0026] In a second aspect, the present application proposes a knowledge base index construction optimization device based on a large model, the device comprising:

[0027] A setting module, used for dividing the text document into a plurality of configuration blocks according to the block configuration block setting;

[0028] A processing module is used to respectively perform text block vectorization embedding on the configuration blocks to obtain a vector database, and perform graph element extraction and entity merging on the configuration blocks to obtain a merged result;

[0029] The entity resolution module is used to resolve the entity of the merged result through the semantic recognition capability of the large model and the external synonym library to obtain a structured entity and relationship data graph;

[0030] A grouping module is used to use a recursive community clustering algorithm to group entities based on the closeness of the relationships between structured entity and relationship data graphs to obtain a community structure with similar characteristics;

[0031] A generation module is used to generate a community summary report based on the entity relationship function graph, similarity feature community structure and node vector using a large model, and integrate the community summary report into the graph database;

[0032] The combination module is used to combine the vector database and the graph database to obtain a multidimensional index of the knowledge base based on the naive RAG method and knowledge graph technology.

[0033] In a possible implementation, the processing module is used to:

[0034] Combined with the prompt words, the large model is called and the thinking chain technology is used to extract the basic units of the graphic elements to obtain the graphics of each text block;

[0035] The same entities are extracted from the graph of each text block, and merged by constructing a description array according to the entity name and entity type to obtain the merged result.

[0036] In a possible implementation, the grouping module is used to:

[0037] Set the community size threshold to determine the maximum number of nodes in each community;

[0038] Find the largest connected subgraph from the structured entity and relationship data graph based on the community size threshold, and standardize the node names in the largest connected subgraph;

[0039] The undirected graph is converted into a directed graph, and then the recursive community clustering algorithm is used to hierarchically cluster the graph to obtain a community structure with similar characteristics.

[0040] In a possible implementation, a generating module is used to:

[0041] A summary report for each community is generated using a large model based on entity-relationship function graphs, similarity feature community structures, and node vectors;

[0042] Use the large model to get reports of the entire graph from multiple granularity points, or perform local cluster summarization to generate community summary reports, and integrate the community summary reports into the graph database.

[0043] The above-mentioned main scheme of the present application and its further options can be freely combined to form multiple schemes, all of which are schemes that can be adopted and claimed for protection in the present application; and in the present application, (non-conflicting options) options and other options can also be freely combined. After understanding the scheme of the present application, those skilled in the art can understand that there are multiple combinations based on the prior art and common knowledge, all of which are technical schemes to be protected by the present application, and they are not exhaustively listed here.

[0044] The present application discloses a knowledge base index construction optimization method and device based on a large model, which divides a text document into multiple configuration blocks, performs vectorization embedding of the text blocks to obtain a vector database, and performs graph element extraction and entity merging to obtain a merged result, performs entity parsing on the merged result to obtain a structured entity and relationship data graph, and groups the structured entity and relationship data graph based on the closeness of the relationship between the structured entity and relationship data graph to obtain a community structure with similar characteristics, uses a large model to generate a community summary report and integrates it into a graph database, and finally combines the vector database and the graph database based on a naive RAG method and knowledge graph technology to obtain a multi-dimensional index of the knowledge base. The knowledge graph is used to construct the complex connections and invisible relationships in the knowledge into a knowledge base index, and the large model technology is used to intelligently assist the knowledge graph generation process, and a knowledge community is constructed to fill in the explicit and invisible relationships between knowledge, thereby improving the efficiency and accuracy of knowledge retrieval and generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A flowchart of a large-model-based knowledge base index construction optimization method proposed in an embodiment of the present application is shown.

[0047] Figure 2 The index construction process of the knowledge graph proposed in the embodiment of the present application is shown.

[0048] Figure 3 A knowledge base index construction method proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0049] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0050] Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of this application.

[0051] In existing technologies, knowledge is systematic, and this feature is more prominent when it comes to private knowledge within an enterprise. Knowledge graph technology naturally has the advantages of powerful graph structure expression, multi-level association capabilities, and clear traceability, and should naturally be applied to knowledge base construction. However, knowledge graphs also have problems such as requiring a large amount of manually annotated corpus, entity extraction, lack of effective solutions to semantic ambiguity and context understanding in natural language, and poor generalization capabilities.

[0052] Therefore, in order to solve the above-mentioned technical problems, the embodiment of the present application proposes a knowledge base index construction optimization method and device based on a big model, which integrates the big model technology and the knowledge graph technology, and utilizes the big model's powerful natural language understanding ability, context understanding ability, intelligent learning and reasoning ability, etc., which can complement and enhance the knowledge graph technology to a large extent. It is described in detail below.

[0053] Please refer to Figure 1 , Figure 1 The flowchart of a knowledge base index construction optimization method based on a large model proposed in an embodiment of the present application is shown, including the following steps:

[0054] Step S1, dividing the text document into a plurality of configuration blocks according to the block configuration block setting;

[0055] Chunking is the process of dividing a long text document into multiple smaller, manageable parts, with each chunk having a unique identifier (ID) containing metadata information.

[0056] First, you can set the default value based on experience, and adjust the block size according to the actual text content and needs. For example, for documents with denser content, you can appropriately reduce the block size; for documents with sparser content, you can appropriately increase the block size. Then the input document is segmented according to the set block size. Each block will generate a unique ID number and will be stored in the database together with its metadata information (such as the block start position, end position, document ID, etc.). The segmented text and its metadata are stored in the database for subsequent retrieval and processing, and the segmented text and its metadata will be passed to the next processing stage.

[0057] Step S2, respectively performing text block vectorization embedding on the configuration blocks to obtain a vector database, and performing graph element extraction and entity merging on the configuration blocks to obtain a merged result;

[0058] Graph element extraction refers to extracting the basic components of graphs from text, including entities, relationships, and statements. These basic units are used to build knowledge graphs and support various advanced applications such as semantic search, recommendation systems, and intelligent question answering.

[0059] Text block vectorization converts each text block into a vector representation, usually using a pre-trained word embedding model (such as Word2Vec, GloVe, BERT, etc.). The vector representation can capture the semantic information of the text block, which is convenient for subsequent graph element extraction and similarity calculation. After that, the vector representation of all text blocks is stored in the vector database for fast retrieval and comparison. The vector database supports efficient approximate nearest neighbor search (ANN), which can quickly find similar text blocks in large-scale data sets.

[0060] Step S2, step, comprising:

[0061] Combined with the prompt words, the large model is called and the thinking chain technology is used to extract the basic units of the graphic elements to obtain the graphics of each text block;

[0062] The same entities are extracted from the graph of each text block, and merged by constructing a description array according to the entity name and entity type to obtain the merged result.

[0063] The basic units of graph elements include entities, relationships, and declarations.

[0064] First, design appropriate prompt words to guide the large model to extract graph elements. The prompt words should clearly specify the entity type, relationship type and statement type to be extracted. Then use the large model (such as BERT, T5, GPT, etc.) to process each text block to extract entities, relationships and statements. The large model can capture complex semantic information through multi-layer neural networks to improve the accuracy and robustness of extraction. Finally, the Chain-of-Thought (CoT) technology is used to gradually reason and verify the extraction results. Through multi-step reasoning, it is ensured that the extracted graph elements are logically consistent and in line with common sense.

[0065] Then, a graph of each text block is output. The entity list in the graph includes entity name, entity type and entity description. The entity type can be selected from the predefined entity type list entity_types. The relationship list includes source entity, target entity, relationship description and relationship strength. The relationship description can be a specific predicate and the relationship strength can be a confidence score.

[0066] The same entity may be extracted in different text blocks. The same entity is identified by matching the entity name and entity type, and the description information of the same entity is merged into a description array. The description array can contain multiple descriptions, each from a different text block. Finally, the merged entity list and relationship list are output to form the smallest subgraph and obtain the merged result.

[0067] In a possible embodiment, the present application embodiment proposes a guidance template for guiding knowledge graph construction experts on how to identify entities and their relationships from a given text. Specifically, this template defines a process divided into four main steps:

[0068] Image extraction prompt words:

[0069] #You are an expert in building knowledge graphs and can identify all types of entities and relationships between entities from the text based on the input text and a list of entity types.

[0070] ##You need to follow the steps below step by step:

[0071] 1. Identify all entities. Extract the following information for each entity:

[0072] -Entity name entity_name: the name of the entity, with the first letter capitalized;

[0073] - entity_type: one of the following types: [{entity_types}];

[0074] -Entity description entity_description: Entity attributes and comprehensive description.

[0075] --The output format requirements for each entity are as follows:

[0076] "entity"{tuple_delimiter};

[0077] <entity_name> {tuple_delimiter};

[0078] <entity_type> {tuple_delimiter};

[0079] <entity_description> ;

[0080] 2. From the identified entities, identify all source entity / target entity pairs that are clearly related. For each pair of related entities, extract the following information:

[0081] -Source entity: the name of the source entity, which is the identified entity;

[0082] -Target entity: the name of the target entity, which is the identified entity;

[0083] - Relationship description: explain why you think the source entity and the target entity are related;

[0084] -Relationship Strength: Gives a numerical score of the strength of the relationship between the source and target entities.

[0085] 3. All the identified entities and relations are put into a single list, using {record_delimiter} as the delimiter.

[0086] 4. After completing the above steps, output {completion_delimiter}.

[0087] Step S3: Analyze the entity of the merged result through the semantic recognition capability of the large model and the external synonym library to obtain a structured entity and relationship data graph.

[0088] In order to achieve entity resolution of the merged results through the semantic recognition capability of the big model and the external synonym library, we first clean and standardize the input text, remove irrelevant characters, and unify the format. Then, we use the semantic recognition capability of the big model to identify all potential entities from the text. The identified entities are matched with the external synonym library to reduce entity redundancy. Then, the relationship between entities is determined, and the relationship description is generated. Finally, the entities and relationships are organized into a structured data graph.

[0089] Step S4: using a recursive community clustering algorithm, grouping is performed based on the closeness of the relationship between the structured entity and relationship data graph to obtain a community structure with similar characteristics;

[0090] The community structure of the graph is constructed by applying a recursive community clustering algorithm to the subgraph until a community size threshold is reached, and finally the community attributes are added to the graph.

[0091] Step S4 includes:

[0092] Set the community size threshold to determine the maximum number of nodes in each community;

[0093] Find the largest connected subgraph from the structured entity and relationship data graph based on the community size threshold, and standardize the node names in the largest connected subgraph;

[0094] The undirected graph is converted into a directed graph, and then the recursive community clustering algorithm is used to hierarchically cluster the graph to obtain a community structure with similar characteristics.

[0095] First, set the community size threshold to ensure the reasonable size of the community and avoid communities that are too large or too small. Specifically, define the minimum and maximum number of nodes in each community, for example, the minimum community size is 2 and the maximum community size is 5. Then find the largest connected component to ensure that a connected subgraph is processed. Extract the largest connected subgraph from the graph, that is, the part where all nodes are connected to each other. Standardize the graph node names to ensure the consistency of node names and avoid errors caused by naming differences. Check and unify the name format of all nodes to ensure that there are no duplications or inconsistencies.

[0096] Then the undirected graph is converted into a directed graph, specifically by converting each undirected edge into two directed edges in opposite directions. Hierarchical clustering uses a recursive clustering algorithm to group the nodes in the graph into communities with similar characteristics. A recursive community clustering algorithm (such as the Leiden algorithm) is used to hierarchically cluster the graph, gradually merging nodes until the set community size threshold is reached. Finally, each node is assigned to the corresponding community, and the community attributes are added to the graph to the similar characteristic community structure.

[0097] The knowledge base index construction optimization method also includes: using the Node2Vec algorithm to generate a vector representation of the graph.

[0098] Graph embedding is to use the Node2Vec algorithm to generate a vector representation of the graph. Node2Vec is a graph embedding algorithm that generates a sequence of nodes through random walks, and then uses a method similar to Word2Vec to train the low-dimensional vector representation of the nodes. These vectors capture the structure and semantic relationship between nodes and can be used for tasks such as node classification, link prediction, and community detection. By using the Node2Vec algorithm, the implicit structure of the graph can be understood and additional vector space can be provided.

[0099] Step S5: Based on the entity relationship function graph, the similarity feature community structure and the node vector, a community summary report is generated using the large model, and the community summary report is integrated into the graph database.

[0100] Based on entity-relationship functional graphs, similarity feature community structures, and node vectors, a large model is used to generate community summary reports, which are then integrated into a graph database to systematically analyze and understand the community structure in the network and provide support for decision making.

[0101] The steps of step S5 include:

[0102] A summary report for each community is generated using a large model based on entity-relationship function graphs, similarity feature community structures, and node vectors;

[0103] Use the large model to get reports of the entire graph from multiple granularity points, or perform local cluster summarization to generate community summary reports, and integrate the community summary reports into the graph database.

[0104] First, a functional graph that reflects the relationships between entities is constructed. Entities include people, organizations, products, etc., while relationships reflect the interactions or connections between them. By analyzing the entity relationship functional graph, different communities or clusters are identified. Communities may form a hierarchical structure, some of which may contain smaller sub-communities. Then, algorithms such as Node2Vec are used to generate a vector representation of each node (i.e., entity), which can capture the structure and semantic relationships between nodes.

[0105] Based on the above data, a large model (such as a deep learning model) is used to generate a summary report for each community. The summary report summarizes the main information of the community, such as the main entities in the community, their relationships, the characteristics of the community, etc. For higher-level communities, reports of the entire chart can be generated from multiple granularity points to provide a macro perspective. For low-level communities, local cluster summaries are performed to provide a micro perspective. The generated community reports, summaries, and titles are then converted into text embeddings to form a vector representation of the community, which is convenient for subsequent analysis and application. Finally, these community summary reports and their vector representations are integrated into the graph database for query and further data mining.

[0106] In a possible implementation, the present application embodiment proposes a prompt word template for generating a community summary report. This template instructs an analysis assistant on how to write a comprehensive community summary report based on given community data (including entity lists, relationship lists, and related declaration information). The specific format and requirements are as follows:

[0107] Community report generation prompt words:

[0108] #Role: You are an analytical assistant who can identify and evaluate information related to certain entities and perform information discovery.

[0109] ##Objective: Write a comprehensive community summary report based on the entity list, relationship list and related declaration information of the community.

[0110] ###Rules: The report format is as follows:

[0111] - Name: The name of the community. The title should be concise and may include the name of a representative entity;

[0112] - Summary: An overview of the overall structure of the community, including the relationships between entities and the salient information associated with those entities.

[0113] -Importance: A floating point score between 0 and 10 is given to represent the impact of the entity in the community.

[0114] -Summary information: A summary of about 5 key information about the community, which should be concise and representative. And list its 5 most relevant document IDs and text block IDs.

[0115] Use the following inputs and do not make up your own.

[0116] Step S6: Based on the naive RAG method and knowledge graph technology, the vector database and the graph database are combined to obtain a multidimensional index of the knowledge base.

[0117] Figure 2 The index construction process of the knowledge graph proposed in the embodiment of the present application is shown. First, the text is divided into blocks and the block size is configured. Entities, relationships and declarations are extracted from the text blocks, and entity merging is performed. The extracted entities are parsed, community detection is performed using a community detection algorithm, and graph embedding is performed. A community report is generated and community embedding is performed. Finally, text block linking and document embedding are performed.

[0118] based on Figure 2 Please refer to Figure 3 , Figure 3 A knowledge base index construction method proposed in an embodiment of the present application is shown. First, a local document is uploaded to the system, and the uploaded document is divided into multiple text blocks. Each text block is converted into a vector representation for subsequent processing. The vectorized text blocks are stored in a vector database. The data in the vector database constitutes a part of the knowledge base. Based on the knowledge base, a knowledge graph is constructed, community detection is performed on the knowledge graph, different communities are identified, and the community graph is converted into a vector representation. Finally, the vectorized community graph is stored in a function library.

[0119] First, we establish the association between documents and text blocks, and make it clear which text blocks belong to which documents and vice versa, which helps maintain the integrity and traceability of the data in subsequent processing. We use the average embedding of text blocks to generate a vector representation of the document, divide the document into non-overlapping blocks, generate an embedding vector for each block, calculate the average of these blocks, and weight them by the tag count to generate the embedding vector of the document. We build a document graph to generate a logical diagram that shows the implicit relationship between documents, helping users better understand the association and structure between documents.

[0120] The naive RAG (Retrieval-Augmented Generation) method is used to combine retrieval and generation techniques to improve the accuracy and efficiency of knowledge retrieval. The RAG method enhances the performance of the system by retrieving relevant documents and generating answers. It uses graph database technology to build a knowledge graph, visualize the relationship between documents and text blocks, and form a structured knowledge system, in which the knowledge graph can effectively represent the complex relationship between entities.

[0121] On the basis of traditional knowledge graph, we add community detection step and use recursive community clustering algorithm to hierarchically cluster knowledge. The increased hierarchy and systematization of knowledge can make knowledge retrieval more efficient and accurate. Finally, we embed the community detection results into the graph database to add community attributes to the graph. In this way, the graph database contains not only entities and relationships, but also community information, which enhances the navigation function of knowledge retrieval.

[0122] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0123] First, through knowledge graph technology, the complex connections and invisible relationships in the knowledge are constructed into a knowledge base index, so that the large model can obtain richer and more accurate context information when reasoning. The large model itself has powerful natural language understanding and generation capabilities, and combined with the knowledge graph, its reasoning ability is further improved, making it more accurate and efficient when dealing with complex problems.

[0124] Second, by utilizing the intelligent capabilities of the big model, the generation process of the knowledge graph can be automatically assisted. The big model can automatically identify and extract entities and their relationships by reading large amounts of text data to generate high-quality knowledge graphs. This not only greatly reduces the workload of manual labeling, but also improves the accuracy and completeness of the knowledge graph.

[0125] Third, through the community detection algorithm, communities or clusters in the knowledge graph can be identified, which can help users better understand and manage knowledge and provide more fine-grained knowledge retrieval and navigation. The community retrieval step can not only reveal explicit entity relationships, but also discover and supplement invisible associations, making the knowledge graph more complete and comprehensive.

[0126] Fourth, by designing specific prompt words, the large model is guided to generate high-quality community summary reports. The prompt word design can be adjusted and optimized according to different application scenarios, so that the large model can better adapt to different knowledge management and retrieval needs.

[0127] The following is a possible implementation of a knowledge base index construction optimization device based on a large model, which is used to execute the various execution steps and corresponding technical effects of the knowledge base index construction optimization method shown in the above embodiment and possible implementation. The device includes:

[0128] A setting module, used for dividing the text document into a plurality of configuration blocks according to the block configuration block setting;

[0129] A processing module is used to respectively perform text block vectorization embedding on the configuration blocks to obtain a vector database, and perform graph element extraction and entity merging on the configuration blocks to obtain a merged result;

[0130] The entity resolution module is used to resolve the entity of the merged result through the semantic recognition capability of the large model and the external synonym library to obtain a structured entity and relationship data graph;

[0131] A grouping module is used to use a recursive community clustering algorithm to group entities based on the closeness of the relationships between structured entity and relationship data graphs to obtain a community structure with similar characteristics;

[0132] A generation module is used to generate a community summary report based on the entity relationship function graph, similarity feature community structure and node vector using a large model, and integrate the community summary report into the graph database;

[0133] The combination module is used to combine the vector database and the graph database to obtain a multidimensional index of the knowledge base based on the naive RAG method and knowledge graph technology.

[0134] In a possible implementation, the processing module is used to:

[0135] Combined with the prompt words, the large model is called and the thinking chain technology is used to extract the basic units of the graphic elements to obtain the graphics of each text block;

[0136] The same entities are extracted from the graph of each text block, and merged by constructing a description array according to the entity name and entity type to obtain the merged result.

[0137] In a possible implementation, the grouping module is used to:

[0138] Set the community size threshold to determine the maximum number of nodes in each community;

[0139] Find the largest connected subgraph from the structured entity and relationship data graph based on the community size threshold, and standardize the node names in the largest connected subgraph;

[0140] The undirected graph is converted into a directed graph, and then the recursive community clustering algorithm is used to hierarchically cluster the graph to obtain a community structure with similar characteristics.

[0141] In a possible implementation, a generating module is used to:

[0142] A summary report for each community is generated using a large model based on entity-relationship function graphs, similarity feature community structures, and node vectors;

[0143] Use the large model to get reports of the entire graph from multiple granularity points, or perform local cluster summarization to generate community summary reports, and integrate the community summary reports into the graph database.

[0144] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A knowledge base index construction optimization method based on a large model, characterized in that: The method comprises: Divide the text document into multiple configuration blocks according to the block configuration block settings; The configuration blocks are respectively subjected to text block vectorization embedding to obtain a vector database, and the configuration blocks are subjected to graph element extraction and entity merging to obtain a merging result; The semantic recognition capability of the large model and the external synonym library are used to perform entity parsing on the merged results to obtain a structured entity and relationship data graph; Using the recursive community clustering algorithm, the entities and relationship data graphs are grouped based on the closeness of the relationships between them to obtain a community structure with similar characteristics. Based on the entity-relationship function graph, similarity feature community structure and node vector, a large model is used to generate a community summary report, and the community summary report is integrated into the graph database; Based on the naive RAG method and knowledge graph technology, the vector database and graph database are combined to obtain a multidimensional index of the knowledge base.

2. The knowledge base index construction optimization method according to claim 1, characterized in that: The steps of extracting graph elements and merging entities from the configuration blocks to obtain a merged result include: Combined with the prompt words, the large model is called and the thinking chain technology is used to extract the basic units of the graphic elements to obtain the graphics of each text block; The same entities are extracted from the graph of each text block, and merged by constructing a description array according to the entity name and entity type to obtain the merged result.

3. The knowledge base index construction optimization method according to claim 2, characterized in that: The basic units of graph elements include entities, relationships, and declarations.

4. The knowledge base index construction optimization method according to claim 1, characterized in that: Using the recursive community clustering algorithm, the steps of grouping based on the closeness of the relationship between the structured entity and relationship data graph to obtain the community structure with similar characteristics include: Set the community size threshold to determine the maximum number of nodes in each community; Find the largest connected subgraph from the structured entity and relationship data graph based on the community size threshold, and standardize the node names in the largest connected subgraph; The undirected graph is converted into a directed graph, and then the recursive community clustering algorithm is used to hierarchically cluster the graph to obtain a community structure with similar characteristics.

5. The knowledge base index construction optimization method according to claim 1, characterized in that: The method also includes: generating a vector representation of the graph using a Node2Vec algorithm.

6. The knowledge base index construction optimization method according to claim 1, characterized in that: The steps of generating a community summary report based on the entity relationship function graph, the similarity feature community structure and the node vector using the large model and integrating the community summary report into the graph database include: A summary report for each community is generated using a large model based on entity-relationship functional graphs, similarity feature community structures, and node vectors; Use the large model to get reports of the entire graph from multiple granularity points, or perform local cluster summarization to generate community summary reports, and integrate the community summary reports into the graph database.

7. A knowledge base index construction optimization device based on a large model, characterized in that: The device comprises: A setting module, used for dividing the text document into a plurality of configuration blocks according to the block configuration block setting; A processing module is used to respectively perform text block vectorization embedding on the configuration blocks to obtain a vector database, and perform graph element extraction and entity merging on the configuration blocks to obtain a merged result; The entity resolution module is used to resolve the entity of the merged result through the semantic recognition capability of the large model and the external synonym library to obtain a structured entity and relationship data graph; A grouping module is used to use a recursive community clustering algorithm to group entities based on the closeness of the relationships between structured entity and relationship data graphs to obtain a community structure with similar characteristics; A generation module is used to generate a community summary report based on the entity relationship function graph, similarity feature community structure and node vector using a large model, and integrate the community summary report into the graph database; The combination module is used to combine the vector database and the graph database to obtain a multidimensional index of the knowledge base based on the naive RAG method and knowledge graph technology.

8. The knowledge base index construction optimization device according to claim 7, characterized in that: Processing modules for: Combined with the prompt words, the large model is called and the thinking chain technology is used to extract the basic units of the graphic elements to obtain the graphics of each text block; The same entities are extracted from the graph of each text block, and merged by constructing a description array according to the entity name and entity type to obtain the merged result.

9. The knowledge base index construction optimization device according to claim 7, characterized in that: Grouping module for: Set the community size threshold to determine the maximum number of nodes in each community; Find the largest connected subgraph from the structured entity and relationship data graph based on the community size threshold, and standardize the node names in the largest connected subgraph; The undirected graph is converted into a directed graph, and then the recursive community clustering algorithm is used to hierarchically cluster the graph to obtain a community structure with similar characteristics.

10. The knowledge base index construction optimization device according to claim 7, characterized in that: Generate modules for: A summary report for each community is generated using a large model based on entity-relationship functional graphs, similarity feature community structures, and node vectors; Use the large model to get reports of the entire graph from multiple granularity points, or perform local cluster summarization to generate community summary reports, and integrate the community summary reports into the graph database.

Images