Large model knowledge base construction method based on multi-granularity retrieval

Abstract

The invention discloses a large model knowledge base construction method based on multi-granularity retrieval. The method comprises the steps of obtaining a target text and analyzing a document structure, splitting and reorganizing text data according to three levels of titles, fragments and sentences, and constructing three levels of knowledge of the titles, the fragments and the sentences; using a large language model to construct a generative index as question-level knowledge according to the sentence-level knowledge; using a vector model to generate a first vector representing index keys of fragment-level knowledge and / or sentence-level knowledge, and a second vector representing index keys of title-level knowledge and / or question-level knowledge; and respectively storing the first vector and the second vector into a vector database, and storing the index values and the index types corresponding to the first vector and the second vector into a relational database. According to the method, multi-granularity document knowledge base construction is achieved, the expression modes of knowledge under different granularities are expanded, the accuracy of user query and knowledge matching is improved, and the problem that information is lost or incomplete in an existing method is solved.

Description

Technical Field

[0001] The present invention relates to the field of large model technology, and in particular to a large model knowledge base construction method based on multi-granularity retrieval. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the widespread application of large language models (LLM), the accuracy and reliability of generated content has become an important research direction. Retrieval-Augmented Generation (RAG) technology significantly improves the answer quality of the generation model by combining an external knowledge base with a large language model. In traditional RAG systems, knowledge retrieval and content generation are two key stages. Among them, the knowledge retrieval stage usually uses semantic similarity retrieval to convert document content into vector representation and store it in a vector database. However, the existing technology still has the following problems and challenges: 1. Single granularity of knowledge retrieval: Current knowledge retrieval methods mostly use a single-granularity document vectorization strategy, which is difficult to fully capture the hierarchical semantic structure of the document. For example, a large amount of key information may be scattered in different levels of content such as titles, paragraphs, or sentences. Single-granularity representation often leads to incomplete information or missing context, thus limiting the quality of retrieval results.

[0003] 2. Loss or redundancy of context information: In the existing retrieval logic, the sliding window method of expanding context can make up for the problem of short segments to a certain extent, but it still cannot fully capture deep semantic relationships, especially in information processing that requires contextual reasoning or multi-segment association. In addition, the introduction of too much context may cause the model to generate hallucinations, further reducing the accuracy and credibility of the answer.

[0004] 3. Insufficient relevance of generated content: In retrieval-based generation tasks, the diversity and specialized needs of user queries often lead to inaccurate results of vector model retrieval. In addition, vector models have limited knowledge representation capabilities in specific fields or complex semantic scenarios, especially when faced with domain terminology or deep problems, and may not match user needs. Summary of the invention

[0005] In order to solve the above problems, the present invention aims to provide a large-model knowledge base construction and retrieval enhancement generation method based on multi-granularity retrieval. Through multi-granularity analysis and representation of documents, combined with generative index construction and knowledge rearrangement strategy, multi-granularity document knowledge base construction is realized, and the expression of knowledge at different granularities is expanded, which facilitates the subsequent use of multi-granularity knowledge retrieval and rearrangement retrieval strategies to complete query retrieval, and helps to achieve accurate knowledge positioning to improve the model's answer effect.

[0006] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention includes: The method for constructing a large model knowledge base based on multi-granularity retrieval includes the following steps: S1. Obtain the target text and parse the document structure, split and reorganize the text data according to the three levels of title, fragment and sentence, and construct knowledge at the three levels of title, fragment and sentence; each level of knowledge includes index key, index value and index type; S2. Use a large language model to build a generative index based on sentence-level knowledge as question-level knowledge; S3. Using the vector model to generate a first vector of an index key representing segment-level knowledge and / or sentence-level knowledge, and a second vector of an index key representing title-level knowledge and / or question-level knowledge; S4. Store the first vector and the second vector in a vector database respectively, and store the index values ​​and index types corresponding to the first vector and the second vector in a relational database to complete the construction of the target large model knowledge base.

[0007] Preferably, step S1 comprises: S101. Obtain the target text and parse the document structure, traversing to obtain the type and content of each paragraph; S102. For paragraphs of the title type, the title content is used as the index key, and the text contained in the title, the sub-title, and the text contained in the sub-title are sequentially spliced ​​as the index value to construct title-level knowledge; S103. For paragraphs of type body, the body content is used as index key and index value respectively to construct fragment-level knowledge; S104. The paragraphs of the text type are segmented by periods to obtain sentences, and the content of each sentence is used as an index key and an index value to construct sentence-level knowledge.

[0008] Preferably, step S102 further includes: for the table structures existing before and after the text, after parsing, splicing them in the order of row name, column name and cell content, and adding them to the text of the current title.

[0009] Preferably, step S104 further includes: after parsing the table structure, sequentially concatenating the row names, column names and cell contents as index keys and index values ​​respectively to construct sentence-level knowledge.

[0010] Preferably, step S2 comprises: S201. Use the large language model to extract each knowledge point in the given text sentence sample and generate several related questions; S202. Filter the best question with the highest matching degree with the knowledge point, and concatenate the given text sentence sample and the best question corresponding to each knowledge point and add them to the prompt word as an example; S203. Use the prompt words to guide the large language model to output a number of generative indexes based on each sentence-level knowledge, use the generative index as the index key, and use the content of the sentence-level knowledge as the index value to construct question-level knowledge.

[0011] Preferably, for a target text that does not contain title information, the construction of title level knowledge is skipped.

[0012] Preferably, for text in a cross-platform portable document format, if a title block is identified before a text block, the recognition results of the title block and the text block are concatenated in sequence and used as index keys and index values ​​respectively to construct fragment-level knowledge.

[0013] Preferably, for text in a cross-platform portable document format, if no title block is identified before the text block, the text block recognition result is used as an index key and an index value, respectively, to construct fragment-level knowledge.

[0014] Preferably, if there is a table structure in the recognition result, after parsing the table structure, the row name, column name and cell content are concatenated in order and used as index keys and index values ​​respectively to construct fragment-level knowledge.

[0015] Beneficial Effects 1. Improve the accuracy and comprehensiveness of knowledge retrieval: Through multi-granular analysis and representation of documents, the text content is organized into different levels of knowledge at the title, fragment and sentence levels, fully capturing the hierarchical semantic structure of the document. Combined with the construction of generative indexes, it significantly improves the accuracy of user query and knowledge matching, and solves the problem of information loss or incompleteness in existing methods.

[0016] 2. Improve the flexibility and adaptability of knowledge base construction: Support multiple types of target texts, including structured documents with preset titles, non-titled texts, and cross-platform portable document formats (PDF). For different document types, the present invention can flexibly parse and construct corresponding knowledge levels, ensuring the versatility and applicability of the method.

[0017] 3. Reduce the cost of knowledge construction and maintenance: By introducing the large language model (LLM) and in-context learning (ICL) technology, question-level knowledge in the knowledge base is automatically generated, reducing the workload of manual annotation and maintenance, while improving the generalization ability of the knowledge base in diverse scenarios.

[0018] 4. Optimize the storage and management of multi-granularity knowledge: Adopt a storage architecture that combines a vector model and a relational database to manage the vector representation and index values ​​of knowledge of different granularities separately, implement efficient query and retrieval processes, and further improve the system's storage efficiency and retrieval performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of a flow chart of a method for constructing a large model knowledge base based on multi-granularity retrieval provided in a preferred embodiment of the present invention; Figure 2 A schematic diagram of a data transmission structure of a method for constructing a large model knowledge base based on multi-granularity retrieval provided in a preferred embodiment of the present invention; DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described below in conjunction with the accompanying drawings. In the description of the present invention, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention.

[0021] Embodiment 1 like Figure 1 , Figure 2 As shown, this embodiment provides a method for constructing a large model knowledge base based on multi-granularity retrieval, including the steps of: S1. Obtain the target text and parse the document structure, split and reorganize the text data according to the three levels of title, fragment and sentence, and construct knowledge at three levels: title, fragment and sentence; each level of knowledge includes index key, index value and index type.

[0022] The large model knowledge base refers to a special knowledge storage system built to support the large language model (LLM) to generate high-quality content. It stores and organizes knowledge in a form suitable for large models, so that the large model can efficiently retrieve the required information and generate more accurate and relevant content. It is not only a container for knowledge organization, but also an important supporting tool for improving the generation capability of large models. The present invention provides strong technical support for the generation of accurate and reliable content for large models by optimizing the construction method of the large model knowledge base.

[0023] The target text refers to the original document content used to build the knowledge base. It is the input data source of the entire knowledge base construction process, and is organized into different levels of knowledge representation after parsing and multi-granularity processing. It includes various types of text files, such as plain text files (txt), rich text files (word, pdf), and other document forms that need to be processed; it can be a document with a clear structure (such as titles and paragraphs) or an unstructured text (such as a scanned document recognized by OCR).

[0024] The document structure analysis of the target text can select a suitable analysis tool according to the format of the target text, using Python's python-docx library for word files and Python's fitz library for pdf files. The specific analysis method can be freely selected by those skilled in the art in the art, and the present invention does not make further limitations.

[0025] The reason why the present invention considers constructing three levels of knowledge, namely title, fragment and sentence, is that by organizing and representing knowledge in a hierarchical manner, it can capture the overall semantic structure of the document and refine it to local key points to meet diverse query requirements and improve the accuracy and efficiency of retrieval. The technical considerations for constructing each level of knowledge are explained below: 1. The construction of title-level knowledge is because, on the one hand, title-level knowledge usually contains the core theme of the document and is a general description of the document content. Through the title structure, one can quickly understand the hierarchy and main content of the document, so it provides high-level semantic information; on the other hand, title-level knowledge can quickly match the user's global query; and thirdly, because the granularity of title knowledge is large, it is easy to quickly filter and rearrange, providing a contextual framework for subsequent fine-grained retrieval.

[0026] 2. The construction of fragment-level knowledge is because a fragment is usually a logically complete paragraph that can carry more specific semantics and provide mid-level semantic information; on the other hand, the integration of fragment-level knowledge with tables, paragraphs and additional information helps to fully express the content of a certain topic.

[0027] 3. The construction of sentence-level knowledge provides fine-grained information matching, which can minimize context redundancy, ensure that the retrieved content is highly relevant to user needs, and reduce the risk of information redundancy and hallucination. On the other hand, it can also be used in subsequent steps to build generative indexes, providing a basis for large language models to generate high-quality question-level knowledge, further enhancing the practicality of the knowledge base.

[0028] The three-level knowledge construction reflects the core concept of the multi-granularity knowledge base construction of the present invention: focusing on the global structure of the document while taking into account local details to ensure the comprehensiveness and accuracy of knowledge expression. This hierarchical design not only improves the adaptability of the knowledge base, but also provides strong technical support for different types of queries.

[0029] Those skilled in the art should know that the construction of the above three levels of knowledge can be obtained by using the commonly used word segmentation methods in the art, such as Word2Vec / Doc2Vec representation method or BERT embedding representation method. In some preferred embodiments, a knowledge construction method based on an index structure is provided, which specifically includes: S101. Obtain the target text and parse the document structure, traverse and obtain the type and content of each paragraph. The types specifically include title and body, where the paragraph style of the body is divided into two types, namely, the common type representing the standard style and the unique type representing the extended style. The present invention does not make further distinction between the two and both are identified as body.

[0030] S102. For paragraphs of title type, the title content is used as the index key, and the text contained in the title, the sub-title, and the text contained in the sub-title are sequentially spliced ​​as the index value to construct title-level knowledge. Its json representation is as follows: ; Among them, index, value and type represent the index key, index value and index type of knowledge storage respectively. Among them, title and content represent the title name and all the text contents contained under the current title respectively, and "title" is the index type string.

[0031] If there is a table structure before or after the text under the current title, the table structure is parsed and concatenated in the order of row name, column name and cell content, and then added to the text of the current title.

[0032] S103. For paragraphs of type body, the body content is used as index key and index value respectively to construct fragment-level knowledge. Its json representation is as follows: ; S104. Segment the paragraphs of the text type by period to obtain sentences, and use the content of each sentence as the index key and index value to construct sentence-level knowledge. Its json representation is as follows: .

[0033] If there is a table structure in the current sentence, the row name, column name, and cell content are concatenated in order as index keys and index values, respectively, to construct sentence-level knowledge.

[0034] In some preferred embodiments, the target text may not have preset document title information, in which case the construction of title-level knowledge needs to be skipped. In other preferred embodiments, for text in a cross-platform portable document format (such as PDF), OCR layout recognition technology can be used to detect each text block. If a title block is identified before the text block, since layout recognition may not be able to obtain multi-level title information, the recognition results of the title block and the text block are spliced ​​in sequence and used as index keys and index values, respectively, to construct fragment-level knowledge. If no title block is identified before the text block, the text block recognition results are used as index keys and index values, respectively, to construct fragment-level knowledge. Furthermore, if there is a table structure in the recognition result, after parsing the table structure, the row name, column name, and cell content are spliced ​​in sequence and used as index keys and index values, respectively, to construct fragment-level knowledge.

[0035] S2. Use the large language model to construct a generative index based on sentence-level knowledge as question-level knowledge. Extract and generate question-level knowledge from sentence-level knowledge to enhance the retrieval capability of the knowledge base. Through the construction of a generative index, the user's query can be directly matched with the generated question, thereby improving the accuracy of the retrieval and the fine-grained information coverage capability. Specific steps are given in some preferred embodiments: S201. Use a large language model to extract each knowledge point in a given text sentence sample and generate several related questions.

[0036] S202. Filter the best question with the highest matching degree with the knowledge point, and add the given text sentence sample and the best question corresponding to each knowledge point as an example to the prompt word. Combine the given text sentence sample with the screened question to construct a further generation example to optimize the accuracy of the next round of question generation. The given text sentence sample refers to a specific text sentence provided in advance by a person skilled in the art as input data, which is used to guide the model to understand, analyze, generate or process, and can be a statement or other type of sentence.

[0037] S203. Use the prompt words to guide the large language model to output several generative indexes based on each sentence-level knowledge, use the generative index as the index key, and use the content of the sentence-level knowledge as the index value to construct question-level knowledge. Use the prompt words to guide the model to generate diversified questions, covering different semantic angles of sentence-level knowledge.

[0038] The following is a specific example to further illustrate the above steps: Given a sample text sentence: "Apples are rich in vitamin C, which can enhance immunity. Bananas are rich in dietary fiber, which helps promote intestinal motility and prevent constipation." The knowledge extracted in step S201 includes: "Apples are rich in vitamin C, which can enhance immunity" and "Bananas are rich in dietary fiber, which helps promote intestinal peristalsis and prevent constipation." Based on this, questions that can be generated include: "What is the effect of vitamin C content in apples on immunity?", "Why can apples enhance immunity?", "What are the benefits of eating bananas?", "Which fruits contain dietary fiber like bananas?".

[0039] After manual screening in step S202, the best questions obtained are: "Why can apples enhance immunity?" and "What are the benefits of eating bananas?", and the given text sentence sample and the best questions are concatenated and added to the prompt words as examples.

[0040] Taking the sentence-level knowledge content "Artificial intelligence is an important part of the intelligent discipline. It attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is a very broad science, including robots, language recognition, image recognition, natural language processing, expert systems, machine learning, computer vision, etc." as an example, the prompt words are as follows: You are a question-answering assistant. You need to generate multiple related questions based on the knowledge points in the input text and output them in the form of a list. The following is an example: [Example] Input text: Apples are rich in vitamin C, which can enhance immunity. Bananas are rich in dietary fiber, which helps promote intestinal peristalsis and prevent constipation.

[0041] Output questions: ["Why can apples enhance immunity?", "What are the benefits of eating bananas?"]

Practical Application

[0042] Output problem: The final output answer list is ["What are the components of artificial intelligence?", "What directions does artificial intelligence include?"] The example of the question-level knowledge json constructed by the generative index obtained in step S203 is as follows: { “query”: “What are the components of artificial intelligence?”, "value": "Artificial intelligence is an important part of the intelligent discipline. It attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is a very broad science, including robots, language recognition, image recognition, natural language processing, expert systems, machine learning, computer vision, etc.", "type":"query" }, { “query”: “What directions does artificial intelligence include?”, "value": "Artificial intelligence is an important part of the intelligent discipline. It attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is a very broad science, including robots, language recognition, image recognition, natural language processing, expert systems, machine learning, computer vision, etc.", "type":"query" }, S3. Generate a first vector representing an index key of segment-level knowledge and / or sentence-level knowledge, and a second vector representing an index key of title-level knowledge and / or question-level knowledge using a vector model.

[0043] When generating vector representations, it is necessary to fully consider the differences in the expressions of index keys of different knowledge types: 1. Fragment-level knowledge and sentence-level knowledge: Index keys and index values ​​are usually semantic representations of a piece of text, suitable for capturing global or contextual information. Their vector representation can directly match user queries during retrieval and quickly recall relevant knowledge fragments.

[0044] 2. Title-level knowledge and question-level knowledge: Index keys are shorter and often only reflect the core topics or key issues of knowledge, which are suitable for efficiently locating important entries in the knowledge base. In this way, user queries can be matched with phrased index keys and directly associated with specific knowledge content.

[0045] The first vector is usually a relatively complete text description with global information, which can directly match the user query to locate the text fragments that are highly relevant to the query content. The second vector is more refined, often contains key information, and has a strong correlation with the corresponding index value. By matching the user query with these index keys, the key knowledge units can be quickly located and their associated content can be obtained.

[0046] This step builds hierarchical vector representations for different types of knowledge, which enables vectorized index keys to support efficient retrieval based on semantic similarity, and quickly locates relevant content regardless of whether the user's query is a global description or a specific question. At the same time, fragment-level and sentence-level knowledge provide contextual information, and title-level and question-level knowledge supplement the core summary, thus building a hierarchical knowledge base that can adapt to a variety of query needs.

[0047] S4. Store the first vector and the second vector in a vector database respectively, and store the index values ​​and index types corresponding to the first vector and the second vector in a relational database to complete the construction of the target large model knowledge base.

[0048] In this step, a hierarchical storage strategy is used to process different types of data separately to complete the construction of the target large model knowledge base. Among them, the vector database is dedicated to storing the semantic vector representation of the knowledge index key, which is convenient for quickly finding the knowledge items most similar to the user query through vector retrieval technology (such as cosine similarity or Euclidean distance). The relational database stores index values ​​(such as text content, table information) and corresponding index types (such as fragments, sentences, titles or questions) in a structured manner as the basic data of the knowledge base. Through the mutual cooperation of the two, the storage efficiency and retrieval performance of the knowledge base are significantly improved, and the rapid semantic matching of user queries and knowledge index keys is achieved. It performs well especially when processing massive categorical data, and has both high efficiency and flexibility, providing a good architectural foundation for the expansion and optimization of the knowledge base.

[0049] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for constructing a large model knowledge base based on multi-granularity retrieval, characterized in that: Includes steps: S1. Obtain the target text and parse the document structure, split and reorganize the text data according to the three levels of title, fragment and sentence, and construct knowledge at the three levels of title, fragment and sentence; each level of knowledge includes index key, index value and index type; S2. Use a large language model to build a generative index based on sentence-level knowledge as question-level knowledge; S3. Using the vector model to generate a first vector of an index key representing segment-level knowledge and / or sentence-level knowledge, and a second vector of an index key representing title-level knowledge and / or question-level knowledge; S4. Store the first vector and the second vector in a vector database respectively, and store the index values ​​and index types corresponding to the first vector and the second vector in a relational database to complete the construction of the target large model knowledge base.

2. The method for constructing a large model knowledge base based on multi-granularity retrieval according to claim 1, characterized in that: Step S1 includes: S101. Obtain the target text and parse the document structure, traversing to obtain the type and content of each paragraph; S102. For paragraphs of the title type, the title content is used as the index key, and the text contained in the title, the sub-title, and the text contained in the sub-title are sequentially spliced ​​as the index value to construct title-level knowledge; S103. For paragraphs of type body, the body content is used as index key and index value respectively to construct fragment-level knowledge; S104. The paragraphs of the text type are segmented by periods to obtain sentences, and the content of each sentence is used as an index key and an index value to construct sentence-level knowledge.

3. The method for constructing a large model knowledge base based on multi-granularity retrieval as claimed in claim 2, characterized in that: Step S102 also includes: for the table structures existing before and after the text, after parsing, splicing them in the order of row name, column name and cell content, and adding them to the text of the current title.

4. The method for constructing a large model knowledge base based on multi-granularity retrieval as claimed in claim 2, characterized in that: Step S104 also includes: after parsing the table structure, concatenating the row name, column name and cell content in order as index keys and index values ​​respectively to construct sentence-level knowledge.

5. The method for constructing a large model knowledge base based on multi-granularity retrieval according to claim 1, characterized in that: Step S2 includes: S201. Use the large language model to extract each knowledge point in the given text sentence sample and generate several related questions; S202. Filter the best question with the highest matching degree with the knowledge point, and concatenate the given text sentence sample and the best question corresponding to each knowledge point and add them to the prompt word as an example; S203. Use the prompt words to guide the large language model to output a number of generative indexes based on each sentence-level knowledge, use the generative index as the index key, and use the content of the sentence-level knowledge as the index value to construct question-level knowledge.

6. The method for constructing a large model knowledge base based on multi-granularity retrieval according to claim 1, characterized in that: For target texts that do not contain title information, the construction of title level knowledge is skipped.

7. The method for constructing a large model knowledge base based on multi-granularity retrieval according to claim 1, characterized in that: For text in the cross-platform portable document format, if a title block is recognized before the text block, the recognition results of the title block and the text block are concatenated in sequence and used as index keys and index values ​​respectively to construct fragment-level knowledge.

8. The method for constructing a large model knowledge base based on multi-granularity retrieval according to claim 1, characterized in that: For text in the cross-platform portable document format, if there is no title block before the text block recognition, the text block recognition results are used as index keys and index values ​​respectively to construct fragment-level knowledge.

9. The method for constructing a large model knowledge base based on multi-granularity retrieval according to claim 7 or 8, characterized in that: If there is a table structure in the recognition result, after parsing the table structure, the row name, column name, and cell content are concatenated in order and used as index keys and index values ​​respectively to construct fragment-level knowledge.

Images