A large model-based knowledge base index construction optimization method and device

Abstract

This application discloses a method and apparatus for optimizing knowledge base index construction based on a large model. The method divides text documents into multiple configuration blocks, performs text block vectorization and embedding to obtain a vector database, and extracts graph elements and merges entities to obtain a merged result. Entity parsing is performed on the merged result to obtain a structured entity and relational data graph. Based on the closeness of relationships between the structured entity and relational data graphs, similar characteristic community structures are obtained by grouping them. A community summary report is generated using a large model and integrated into the graph database. Finally, based on the naive RAG method and knowledge graph technology, the vector database and graph database are combined to obtain a multidimensional knowledge base index. The method uses knowledge graphs to construct a knowledge base index from complex connections and implicit relationships in knowledge. Simultaneously, large model technology intelligently assists the knowledge graph generation process, constructing knowledge communities to fill in the explicit and implicit relationships between knowledge points, thereby improving the efficiency and accuracy of knowledge retrieval and generation.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and more specifically, to a method and apparatus for optimizing the construction of a knowledge base index based on a large model. Background Technology

[0002] With the development of deep learning and natural language processing technologies, large-scale pre-trained models have demonstrated outstanding performance in many fields. However, they still face numerous challenges in practical applications, such as illusion phenomena, limitations of professional knowledge, knowledge timeliness, and lack of interpretability. These issues limit the widespread application of large models in specific industries and professional fields. Therefore, knowledge retrieval augmented generation (RAG) technology, as an effective solution, has gradually gained attention. RAG technology achieves effective utilization and rapid response of knowledge through steps such as document segmentation, vectorized embedding, retrieval recall, formulation of context generation strategies, and selection of a model to generate responses. However, RAG technology performs poorly when dealing with global, generalized, and complex problems, especially when handling large-scale heterogeneous, multi-source unstructured knowledge, where its retrieval efficiency and accuracy fail to meet practical needs.

[0003] To address the aforementioned issues, researchers have proposed various optimization schemes, such as adjusting document chunk sizes and their overlap, expanding question descriptions, and optimizing the sorting of search results. While these measures can improve the performance of RAG systems to some extent, they do not fundamentally solve the problems. To further enhance the practicality and effectiveness of RAG technology, it is necessary to explore new knowledge indexing methods.

[0004] Against this backdrop, knowledge graph technology, with its powerful graph structure representation capabilities, multi-level connections, and clear traceability, has become an ideal choice for building efficient knowledge bases. By constructing knowledge graphs, enterprises can better organize and manage their internal private knowledge, improving the speed and accuracy of knowledge retrieval. However, the construction of knowledge graphs also faces a series of challenges, mainly including a large amount of manual annotation work, complex entity recognition and relation extraction tasks, and semantic ambiguity and contextual understanding difficulties encountered when processing natural language. Furthermore, the generalization ability of knowledge graphs is relatively weak, limiting their application scope in different scenarios. Summary of the Invention

[0005] The purpose of this application is to overcome the shortcomings of existing technologies and provide an optimized method and apparatus for constructing a knowledge base index based on a large model. This method enriches the reasoning context of the large model by using knowledge graph technology, intelligently realizes the knowledge graph generation process by using large model technology, adds a community retrieval step to construct a knowledge community, fills in the explicit and implicit relationships between knowledge, and provides relevant prompt word design examples.

[0006] The objective of this application is achieved through the following technical solution:

[0007] Firstly, this application proposes an optimization method for constructing a knowledge base index based on a large model, the method comprising:

[0008] Divide the text document into multiple configuration blocks according to the block configuration settings;

[0009] The configuration blocks are vectorized and embedded to obtain a vector database, and graph elements are extracted and entities are merged to obtain a merged result.

[0010] By leveraging the semantic recognition capabilities of the large model and an external thesaurus, the merged results are parsed to obtain a structured entity and relational data graph.

[0011] Using a recursive community clustering algorithm, groups are formed based on the tightness of relationships between structured entities and relational data graphs to obtain community structures with similar characteristics;

[0012] Based on entity relationship function graphs, similarity characteristic community structures, and node vectors, a large model is used to generate community summary reports, which are then integrated into a graph database.

[0013] Based on the Naive RAG method and knowledge graph technology, a multidimensional index for the knowledge base is obtained by combining vector databases and graph databases.

[0014] In one possible implementation, the steps of extracting graph elements and merging entities from the configuration blocks to obtain the merged result include:

[0015] By combining prompt words to call the large model and using the mind chain technique to extract graph elements from the basic units of the graph, the graph of each text block is obtained;

[0016] Extract the same entities from the graph of each text block, and merge them according to entity name and entity type by constructing a description array to obtain the merged result.

[0017] In one possible implementation, the basic units of a graph element include entities, relations, and declarations.

[0018] In one possible implementation, the step of using a recursive community clustering algorithm to group entities based on the tightness of relationships between structured entities and relational data graphs to obtain a community structure with similar characteristics includes:

[0019] Set a community size threshold to determine the maximum number of nodes in each community;

[0020] Find the largest connected subgraph from the structured entity and relation data graph based on the community size threshold, and standardize the node names in the largest connected subgraph.

[0021] The undirected graph is transformed into a directed graph, and then a recursive community clustering algorithm is used to perform hierarchical clustering of the graph to obtain a community structure with similar characteristics.

[0022] In one possible implementation, the method further includes generating a vector representation of the graph using the Node2Vec algorithm.

[0023] In one possible implementation, the steps of generating a community summary report using a large model based on entity relationship functional graphs, similarity characteristic community structures, and node vectors, and integrating the community summary report into a graph database, include:

[0024] Based on entity relationship function graphs, similarity characteristic community structures, and node vectors, a summary report for each community is generated using a large model.

[0025] The report of the entire graph can be obtained from multiple granular points using a large model, or a community summary report can be generated by local cluster summarization, and the community summary report can be integrated into the graph database.

[0026] Secondly, this application proposes a knowledge base index construction optimization device based on a large model, the device comprising:

[0027] The settings module is used to divide a text document into multiple configuration blocks according to the block configuration settings.

[0028] The processing module is used to perform text block vectorization embedding on the configuration blocks to obtain a vector database, and to extract graph elements and merge entities on the configuration blocks to obtain a merged result.

[0029] The entity parsing module is used to parse the merged results using the semantic recognition capabilities of the large model and an external thesaurus to obtain a structured entity and relational data graph.

[0030] The grouping module is used to group entities based on the tightness of relationships between structured entities and relational data graphs using a recursive community clustering algorithm, thereby obtaining community structures with similar characteristics.

[0031] The generation module is used to generate community summary reports based on entity relationship function graphs, similarity characteristic community structures, and node vectors using a large model, and integrate the community summary reports into the graph database;

[0032] The module combines vector databases and graph databases to obtain a multidimensional index for the knowledge base, based on the Naive RAG method and knowledge graph technology.

[0033] In one possible implementation, the processing module is configured to:

[0034] By combining prompt words to call the large model and using the mind chain technique to extract graph elements from the basic units of the graph, the graph of each text block is obtained;

[0035] Extract the same entities from the graph of each text block, and merge them according to entity name and entity type by constructing a description array to obtain the merged result.

[0036] In one possible implementation, the grouping module is used for:

[0037] Set a community size threshold to determine the maximum number of nodes in each community;

[0038] Find the largest connected subgraph from the structured entity and relation data graph based on the community size threshold, and standardize the node names in the largest connected subgraph.

[0039] The undirected graph is transformed into a directed graph, and then a recursive community clustering algorithm is used to perform hierarchical clustering of the graph to obtain a community structure with similar characteristics.

[0040] In one possible implementation, the generation module is used for:

[0041] Based on entity relationship function graphs, similarity characteristic community structures, and node vectors, a summary report for each community is generated using a large model.

[0042] The report of the entire graph can be obtained from multiple granular points using a large model, or a community summary report can be generated by local cluster summarization, and the community summary report can be integrated into the graph database.

[0043] The main solution and its various further alternatives described above can be freely combined to form multiple solutions, all of which are solutions that can be adopted and are claimed in this application; furthermore, the (non-conflicting alternatives) can also be freely combined with each other and with other alternatives. Those skilled in the art, after understanding the solution of this application, will realize from the prior art and common general knowledge that there are many combinations, all of which are technical solutions to be protected in this application, and will not be exhaustively listed here.

[0044] This application discloses a method and apparatus for optimizing knowledge base index construction based on a large model. The method divides text documents into multiple configuration blocks, performs text block vectorization and embedding to obtain a vector database, and extracts graph elements and merges entities to obtain a merged result. Entity parsing is performed on the merged result to obtain a structured entity and relational data graph. Based on the closeness of relationships between the structured entity and relational data graphs, similar characteristic community structures are obtained by grouping them. A community summary report is generated using a large model and integrated into the graph database. Finally, based on the naive RAG method and knowledge graph technology, the vector database and graph database are combined to obtain a multidimensional knowledge base index. The method uses knowledge graphs to construct a knowledge base index from complex connections and implicit relationships in knowledge. Simultaneously, large model technology intelligently assists the knowledge graph generation process, constructing knowledge communities to fill in the explicit and implicit relationships between knowledge points, thereby improving the efficiency and accuracy of knowledge retrieval and generation. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 The flowchart illustrates an optimization method for constructing a knowledge base index based on a large model, as proposed in an embodiment of this application.

[0047] Figure 2 The indexing process of the knowledge graph proposed in this application embodiment is illustrated.

[0048] Figure 3 This paper illustrates a knowledge base index construction method proposed in an embodiment of this application. Detailed Implementation

[0049] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0050] Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] In existing technologies, knowledge is systematic, and this characteristic is more pronounced in private knowledge within an enterprise. Knowledge graph technology naturally possesses advantages such as powerful graph structure representation capabilities, multi-level association capabilities, and clear traceability, making it a natural fit for knowledge base construction. However, knowledge graphs also face challenges such as the need for large amounts of manually annotated corpora, entity extraction, a lack of effective solutions for semantic ambiguity and contextual understanding in natural language, and poor generalization ability.

[0052] Therefore, in order to solve the above-mentioned technical problems, this application proposes a knowledge base index construction optimization method and apparatus based on a large model, which integrates large model technology and knowledge graph technology. By utilizing the powerful natural language understanding ability, context understanding ability, and intelligent learning and reasoning ability of the large model, it can complement and enhance knowledge graph technology to a large extent. The following is a detailed description of the method.

[0053] Please refer to Figure 1 , Figure 1 The flowchart of an optimization method for constructing a knowledge base index based on a large model, as proposed in an embodiment of this application, is shown, including the following steps:

[0054] Step S1: Divide the text document into multiple configuration blocks according to the block configuration settings;

[0055] Configuration chunking refers to dividing a long text document into multiple smaller, manageable parts, with each chunk having a unique identifier (ID) containing metadata information.

[0056] First, a default value can be set based on experience. The chunk size can then be adjusted according to the actual text content and requirements. For example, for densely packed documents, the chunk size can be appropriately reduced; for sparsely packed documents, the chunk size can be appropriately increased. Next, the input document is divided according to the set chunk size. Each chunk generates a unique ID number and its metadata information (such as the chunk's start and end positions, and its associated document ID) and is stored in the database. The chunked text and its metadata are stored in the database for subsequent retrieval and processing. The chunked text and its metadata are then passed to the next processing stage.

[0057] Step S2: Perform text block vectorization embedding on the configuration blocks to obtain a vector database, and extract graph elements and merge entities on the configuration blocks to obtain a merged result;

[0058] Graph element extraction refers to extracting the basic building blocks of a graph from text, including entities, relations, and declarations. These basic units are used to construct knowledge graphs, supporting various advanced applications such as semantic search, recommendation systems, and intelligent question answering.

[0059] Text block vectorization converts each text block into a vector representation, typically using pre-trained word embedding models (such as Word2Vec, GloVe, BERT, etc.). Vector representations capture the semantic information of text blocks, facilitating subsequent graph element extraction and similarity calculation. All text block vector representations are then stored in a vector database for rapid retrieval and comparison. This vector database supports efficient Approximate Nearest Neighbor (ANN) search, enabling the rapid finding of similar text blocks in large-scale datasets.

[0060] Step S2, the steps include:

[0061] By combining prompt words to call the large model and using the mind chain technique to extract graph elements from the basic units of the graph, the graph of each text block is obtained;

[0062] Extract the same entities from the graph of each text block, and merge them according to entity name and entity type by constructing a description array to obtain the merged result.

[0063] The basic units of a graph element include entities, relations, and declarations.

[0064] First, design appropriate prompts to guide the large model in extracting graph elements. The prompts should clearly specify the entity types, relation types, and declaration types to be extracted. Then, use a large model (such as BERT, T5, GPT, etc.) to process each text block and extract entities, relations, and declarations. The large model can capture complex semantic information through multi-layer neural networks, improving the accuracy and robustness of extraction. Finally, use Chain-of-Thought (CoT) technology to step-by-step reason and verify the extraction results. Through multi-step reasoning, ensure that the extracted graph elements are logically consistent and conform to common sense.

[0065] The graph for each text block is then output. The entity list in the graph includes entity name, entity type, and entity description. The entity type can be selected from the predefined entity type list entity_types. The relation list includes source entity, target entity, relation description, and relation strength. The relation description can be a specific predicate, and the relation strength can be a confidence score.

[0066] The same entities may be extracted from different text blocks. By matching the entity name and entity type, the same entities are identified and their descriptions are merged into a description array. The description array can contain multiple descriptions, each from a different text block. Finally, the merged entity list and relationship list are output to form the smallest subgraph, thus obtaining the merged result.

[0067] In one possible embodiment, this application proposes a guidance template to instruct knowledge graph construction experts on how to identify entities and their relationships from given text. Specifically, this template defines a process consisting of four main steps:

[0068] Image-based keyword extraction:

[0069] #You are a knowledge graph building expert who can identify all types of entities and relationships between entities from the text based on the input text and a list of entity types.

[0070] You need to follow these steps step by step:

[0071] 1. Identify all entities. Extract the following information for each entity:

[0072] -Entity_name: The name of the entity, with the first letter capitalized;

[0073] - Entity type: one of the following types: [{entity_types}];

[0074] - Entity description: Entity attributes and full description.

[0075] --The output format requirements for each entity are as follows:

[0076] "entity"{tuple_delimiter};

[0077] <entity_name> {tuple_delimiter};

[0078] <entity_type> {tuple_delimiter};

[0079] <entity_description> ;

[0080] 2. From the identified entities, identify all source entity / target entity pairs that are clearly related. For each related entity pair, extract the following information:

[0081] - Source Entity: The name of the source entity, which is the identified entity;

[0082] -Target Entity: The name of the target entity, which is the identified entity;

[0083] - Relationship description: Explain why you believe the source entity and the target entity are related;

[0084] - Relationship Strength: Provides a numerical score of the relationship strength between the source entity and the target entity.

[0085] 3. Compile all identified entities and relationships into a single list, using {record_delimiter} as the delimiter.

[0086] 4. After completing the above steps, output {completion_delimiter}.

[0087] Step S3: Using the semantic recognition capabilities of the large model and the external thesaurus, perform entity parsing on the merged results to obtain a structured entity and relational data graph.

[0088] To enable entity parsing of the merged results using the semantic recognition capabilities of a large model and an external thesaurus, the input text is first cleaned and standardized by removing irrelevant characters and unifying the format. The semantic recognition capabilities of the large model are then used to identify all potential entities from the text. The identified entities are matched with the external thesaurus to reduce entity redundancy. Next, the relationships between entities are determined and relationship descriptions are generated. Finally, the entities and relationships are organized into a structured data graph.

[0089] Step S4: Using a recursive community clustering algorithm, groups entities based on the closeness of relationships between structured entities and relational data graphs to obtain community structures with similar characteristics;

[0090] The community structure of the graph is constructed by applying a recursive community clustering algorithm to the subgraph until a community size threshold is reached, and finally the community attributes are added to the graph.

[0091] Step S4 includes:

[0092] Set a community size threshold to determine the maximum number of nodes in each community;

[0093] Find the largest connected subgraph from the structured entity and relation data graph based on the community size threshold, and standardize the node names in the largest connected subgraph.

[0094] The undirected graph is transformed into a directed graph, and then a recursive community clustering algorithm is used to perform hierarchical clustering of the graph to obtain a community structure with similar characteristics.

[0095] First, set community size thresholds to ensure reasonable community sizes, avoiding communities that are too large or too small. Specifically, define the minimum and maximum number of nodes for each community; for example, the minimum community size is 2, and the maximum community size is 5. Next, find the largest connected component to ensure that the process deals with a connected subgraph. Extract the largest connected subgraph from the graph, i.e., the part where all nodes are interconnected. Standardize graph node names to ensure consistency and avoid errors caused by naming differences. Check and unify the name format of all nodes to ensure there are no duplicates or inconsistencies.

[0096] The undirected graph is then transformed into a directed graph, specifically by converting each undirected edge into two directed edges in opposite directions. Hierarchical clustering, using a recursive clustering algorithm, groups the nodes in the graph into communities with similar characteristics. A recursive community clustering algorithm (such as Leiden's algorithm) is used to perform hierarchical clustering of the graph, gradually merging nodes until a set community size threshold is reached. Finally, each node is assigned to its corresponding community, and the community attribute is added to the similarity-characteristic community structure in the graph.

[0097] Other optimization methods for knowledge base index construction include: using the Node2Vec algorithm to generate vector representations of graphs.

[0098] Graph embedding uses the Node2Vec algorithm to generate vector representations of graphs. Node2Vec is a graph embedding algorithm that generates a sequence of nodes through random walks and then trains low-dimensional vector representations of the nodes using a method similar to Word2Vec. These vectors capture the structural and semantic relationships between nodes and can be used for tasks such as node classification, link prediction, and community detection. By using the Node2Vec algorithm, the implicit structure of the graph can be understood, and additional vector space is provided.

[0099] Step S5: Based on the entity relationship function graph, similar characteristic community structure, and node vectors, generate a community summary report using the large model, and integrate the community summary report into the graph database.

[0100] Based on entity relationship functional graphs, similarity characteristic community structures, and node vectors, a large model is used to generate community summary reports, which are then integrated into a graph database to systematically analyze and understand the community structure in the network, providing support for decision-making.

[0101] Step S5 includes:

[0102] Based on entity relationship function graphs, similarity characteristic community structures, and node vectors, a summary report for each community is generated using a large model.

[0103] The report of the entire graph can be obtained from multiple granular points using a large model, or a community summary report can be generated by local cluster summarization, and the community summary report can be integrated into the graph database.

[0104] First, a functional graph reflecting the relationships between entities is constructed. Entities include people, organizations, products, etc., while relationships reflect the interactions or connections between them. By analyzing the entity relationship functional graph, different communities or clusters are identified. Communities may form a hierarchical structure, and some communities may contain smaller sub-communities. Then, algorithms such as Node2Vec are used to generate a vector representation of each node (i.e., entity), which can capture the structural and semantic relationships between nodes.

[0105] Based on the above data, a large model (such as a deep learning model) is used to generate a summary report for each community. The summary report summarizes the main information of the community, such as the main entities within the community, their relationships, and the characteristics of the community. For higher-level communities, a report of the entire graph can be generated from multiple granular points to provide a macro perspective. For lower-level communities, a local cluster summary is performed to provide a micro perspective. The generated community reports, summaries, and titles are then converted into text embeddings to form a vector representation of the community, which facilitates subsequent analysis and application. Finally, these community summary reports and their vector representations are integrated into a graph database for easy querying and further data mining.

[0106] In one possible implementation, this application provides a prompt template for generating a community summary report. This template guides an analytics assistant on how to compile a comprehensive community summary report based on given community data (including entity lists, relationship lists, and related claims). The specific format and requirements are as follows:

[0107] Community report generation prompt:

[0108] #Role: You are an analytics assistant who can identify and evaluate information related to certain entities and perform information discovery.

[0109] ##Objective: To compile a comprehensive community summary report based on the entity list, relationship list, and related statements of the community.

[0110] ###Rules: The report format is as follows:

[0111] - Name: Community name. The title should be concise and summary, and may include representative entity names;

[0112] - Summary: An overview of the overall structure of the community, including the relationships between entities and significant information related to these entities.

[0113] -Importance: A floating-point score between 0 and 10 is given, representing the degree of influence of an entity within the community.

[0114] -Summary Information: A concise and representative summary of approximately five key pieces of information about the community. Include the five most relevant document IDs and text block IDs.

[0115] Step S6: Based on the Naive RAG method and knowledge graph technology, combine the vector database and the graph database to obtain a multidimensional index for the knowledge base.

[0116] Figure 2 The document illustrates the indexing process of the knowledge graph proposed in this application. First, the text is divided into blocks and the block size is configured. Entities, relations, and declarations are extracted from the text blocks and entities are merged. The extracted entities are parsed, community detection algorithms are used for community detection, and graph embedding is performed. A community report is generated and community embedding is performed. Finally, text block linking and document embedding are performed.

[0117] based on Figure 2 Please refer to Figure 3 , Figure 3 This paper illustrates a knowledge base index construction method proposed in an embodiment of this application. First, a local document is uploaded to the system. The uploaded document is divided into multiple text blocks, and each text block is converted into a vector representation for subsequent processing. The vectorized text blocks are stored in a vector database, and the data in the vector database constitutes part of the knowledge base. Based on the knowledge base, a knowledge graph is constructed. Community detection is performed on the knowledge graph to identify different communities. The community graph is converted into a vector representation, and finally, the vectorized community graph is stored in a function library.

[0118] First, establish the relationships between documents and text blocks, clarifying which text blocks belong to which documents, and vice versa. This helps maintain data integrity and traceability in subsequent processing. Use the average embedding of text blocks to generate a vector representation of the document. Divide the document into non-overlapping blocks, generate an embedding vector for each block, calculate the average of these blocks, and weight them by label count to generate the document's embedding vector. Construct a document graph, generating a logical diagram that shows the implicit relationships between documents, helping users better understand the connections and structure between documents.

[0119] By employing the Naive RAG (Retrieval-Augmented Generation) method, combined with retrieval and generation techniques, the accuracy and efficiency of knowledge retrieval are improved. The RAG method enhances system performance by retrieving relevant documents and generating answers. Graph database technology is used to construct a knowledge graph, visualizing the relationships between documents and text blocks to form a structured knowledge system. This knowledge graph can effectively represent complex relationships between entities.

[0120] Building upon traditional knowledge graphs, a community detection step is added, employing a recursive community clustering algorithm to perform hierarchical clustering of knowledge. This increased hierarchy and systematization of knowledge makes knowledge retrieval more efficient and accurate. Finally, the community detection results are embedded into a graph database, supplementing the graph with community attributes. In this way, the graph database not only contains entities and relationships but also community information, enhancing the navigation capabilities of knowledge retrieval.

[0121] Compared with the prior art, the embodiments of this application have the following beneficial effects:

[0122] First, by using knowledge graph technology, complex connections and implicit relationships within knowledge are constructed into a knowledge base index, enabling large models to obtain richer and more accurate contextual information during reasoning. The powerful natural language understanding and generation capabilities of the large models themselves, combined with knowledge graphs, further enhance their reasoning abilities, making them more accurate and efficient in handling complex problems.

[0123] Secondly, by leveraging the intelligent capabilities of large models, the knowledge graph generation process can be automatically assisted. Large models can automatically identify and extract entities and their relationships by reading large amounts of text data, thereby generating high-quality knowledge graphs. This not only greatly reduces the workload of manual annotation but also improves the accuracy and completeness of the knowledge graph.

[0124] Third, by using community detection algorithms to identify communities or clusters in the knowledge graph, users can better understand and manage knowledge, and provide more granular knowledge retrieval and navigation. The community retrieval steps can not only reveal explicit entity relationships, but also discover and supplement implicit connections, making the knowledge graph more complete and comprehensive.

[0125] Fourth, by designing specific prompts, the large model can be guided to generate high-quality community summary reports. The prompt design can be adjusted and optimized according to different application scenarios, so that the large model can better adapt to different knowledge management and retrieval needs.

[0126] The following describes a possible implementation of a knowledge base index construction optimization device based on a large model, which is used to execute the various execution steps and corresponding technical effects of the knowledge base index construction optimization method shown in the above embodiments and possible implementations. The device includes:

[0127] The settings module is used to divide a text document into multiple configuration blocks according to the block configuration settings.

[0128] The processing module is used to perform text block vectorization embedding on the configuration blocks to obtain a vector database, and to extract graph elements and merge entities on the configuration blocks to obtain a merged result.

[0129] The entity parsing module is used to parse the merged results using the semantic recognition capabilities of the large model and an external thesaurus to obtain a structured entity and relational data graph.

[0130] The grouping module is used to group entities based on the tightness of relationships between structured entities and relational data graphs using a recursive community clustering algorithm, thereby obtaining community structures with similar characteristics.

[0131] The generation module is used to generate community summary reports based on entity relationship function graphs, similarity characteristic community structures, and node vectors using a large model, and integrate the community summary reports into the graph database;

[0132] The module combines vector databases and graph databases to obtain a multidimensional index for the knowledge base, based on the Naive RAG method and knowledge graph technology.

[0133] In one possible implementation, the processing module is configured to:

[0134] By combining prompt words to call the large model and using the mind chain technique to extract graph elements from the basic units of the graph, the graph of each text block is obtained;

[0135] Extract the same entities from the graph of each text block, and merge them according to entity name and entity type by constructing a description array to obtain the merged result.

[0136] In one possible implementation, the grouping module is used for:

[0137] Set a community size threshold to determine the maximum number of nodes in each community;

[0138] Find the largest connected subgraph from the structured entity and relation data graph based on the community size threshold, and standardize the node names in the largest connected subgraph.

[0139] The undirected graph is transformed into a directed graph, and then a recursive community clustering algorithm is used to perform hierarchical clustering of the graph to obtain a community structure with similar characteristics.

[0140] In one possible implementation, the generation module is used for:

[0141] Based on entity relationship function graphs, similarity characteristic community structures, and node vectors, a summary report for each community is generated using a large model.

[0142] The report of the entire graph can be obtained from multiple granular points using a large model, or a community summary report can be generated by local cluster summarization, and the community summary report can be integrated into the graph database.

[0143] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for optimizing the construction of a knowledge base index based on a large model, characterized in that, The method includes: Divide the text document into multiple configuration blocks according to the block configuration settings; The configuration blocks are vectorized and embedded to obtain a vector database, and graph elements are extracted and entities are merged to obtain a merged result. The steps for extracting graph elements and merging entities from configuration blocks to obtain the merged result include: By combining prompt words to call the large model and using the mind chain technique to extract graph elements from the basic units of graph elements, the graph of each text block is obtained. The basic units of graph elements include entities, relations and declarations. Extract the same entities from the graph of each text block, and merge them according to entity name and entity type by constructing a description array to obtain the merged result; By leveraging the semantic recognition capabilities of the large model and an external thesaurus, the merged results are parsed to obtain a structured entity and relational data graph. Using a recursive community clustering algorithm, groups are formed based on the tightness of relationships between structured entities and relational data graphs to obtain community structures with similar characteristics; Based on entity relationship function graphs, similarity characteristic community structures, and node vectors, a large model is used to generate community summary reports, which are then integrated into a graph database. Based on the Naive RAG method and knowledge graph technology, a multidimensional index for the knowledge base is obtained by combining vector databases and graph databases.

2. The knowledge base index construction optimization method as described in claim 1, characterized in that, The steps for obtaining a community structure with similar characteristics by using a recursive community clustering algorithm to group entities based on the tightness of relationships between structured entities and relational data graphs include: Set a community size threshold to determine the maximum number of nodes in each community; Find the largest connected subgraph from the structured entity and relation data graph based on the community size threshold, and standardize the node names in the largest connected subgraph. The undirected graph is transformed into a directed graph, and then a recursive community clustering algorithm is used to perform hierarchical clustering of the graph to obtain a community structure with similar characteristics.

3. The knowledge base index construction optimization method as described in claim 1, characterized in that, The method also includes generating a vector representation of the graph using the Node2Vec algorithm.

4. The knowledge base index construction optimization method as described in claim 1, characterized in that, Based on entity relationship functional graphs, similarity characteristic community structures, and node vectors, the steps for generating community summary reports using a large model and integrating these reports into a graph database include: Based on entity relationship function graphs, similarity characteristic community structures, and node vectors, a summary report for each community is generated using a large model. The report of the entire graph can be obtained from multiple granular points using a large model, or a community summary report can be generated by local cluster summarization, and the community summary report can be integrated into the graph database.

5. A knowledge base index construction optimization device based on a large model, characterized in that, The device includes: The settings module is used to divide a text document into multiple configuration blocks according to the block configuration settings. The processing module is used to perform text block vectorization embedding on the configuration blocks to obtain a vector database, and to extract graph elements and merge entities on the configuration blocks to obtain a merged result. By combining prompt words to call the large model and using the mind chain technique to extract graph elements from the basic units of graph elements, the graph of each text block is obtained. The basic units of graph elements include entities, relations and declarations. Extract the same entities from the graph of each text block, and merge them according to entity name and entity type by constructing a description array to obtain the merged result; The entity parsing module is used to parse the merged results using the semantic recognition capabilities of the large model and an external thesaurus to obtain a structured entity and relational data graph. The grouping module is used to group entities based on the tightness of relationships between structured entities and relational data graphs using a recursive community clustering algorithm, thereby obtaining community structures with similar characteristics. The generation module is used to generate community summary reports based on entity relationship function graphs, similarity characteristic community structures, and node vectors using a large model, and integrate the community summary reports into the graph database; The module combines vector databases and graph databases to obtain a multidimensional index for the knowledge base, based on the Naive RAG method and knowledge graph technology.

6. The knowledge base index construction and optimization apparatus as described in claim 5, characterized in that, Grouping module, used for: Set a community size threshold to determine the maximum number of nodes in each community; Find the largest connected subgraph from the structured entity and relation data graph based on the community size threshold, and standardize the node names in the largest connected subgraph. The undirected graph is transformed into a directed graph, and then a recursive community clustering algorithm is used to perform hierarchical clustering of the graph to obtain a community structure with similar characteristics.

7. The knowledge base index construction and optimization apparatus as described in claim 5, characterized in that, Generate modules for: Based on entity relationship function graphs, similarity characteristic community structures, and node vectors, a summary report for each community is generated using a large model. The report of the entire graph can be obtained from multiple granular points using a large model, or a community summary report can be generated by local cluster summarization, and the community summary report can be integrated into the graph database.

Images