Knowledge retrieval method based on multilayer index

Abstract

The invention discloses a knowledge retrieval method based on a multilayer index, which comprises the following steps of: obtaining a knowledge base of corresponding classification according to a knowledge retrieval statement in combination with a knowledge classification label; obtaining a plurality of retrieval target documents through vector similarity analysis according to the knowledge retrieval statements in combination with the document abstracts in the knowledge base of the corresponding classification; and obtaining a plurality of similar text blocks through vector similarity analysis according to the knowledge retrieval statement in combination with the text blocks of the plurality of retrieval target documents, and outputting the similar text blocks as knowledge retrieval. According to the method, a three-layer index structure containing knowledge classification tags, document abstracts and text blocks is constructed, the classification tags are dynamically matched through a large language model to reduce the retrieval range, high-correlation documents are rapidly positioned in combination with document abstract vector similarity analysis, and context logic relations are reserved based on text block hierarchical vector matching. The problems of fuzzy classification, low efficiency of document recall and context breakage in the traditional technology are solved, and accurate retrieval from macroscopic to microscopic is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of knowledge retrieval, and in particular relates to a knowledge retrieval method based on multi-layer indexing. Background Art

[0002] With the rapid development of information technology, knowledge management and intelligent retrieval systems are playing an increasingly important role across various industries. Especially in specialized fields like equipment maintenance and asset management, efficient knowledge retrieval not only improves work efficiency but also reduces operating costs and ensures business continuity. However, traditional knowledge bases and retrieval methods often struggle to meet these growing specialized needs.

[0003] Large language models have been widely used in information retrieval due to their powerful natural language understanding and generation capabilities. However, while these models excel in general domains, they still face challenges when handling complex problems in specific industries or professional fields. For example, when it comes to specialized knowledge in the field of equipment maintenance, basic large language models may not be able to provide accurate answers, as this type of knowledge often requires analysis in the context of specific context and historical data.

[0004] Therefore, RAG technology combines retrieval with generation and uses external knowledge base to enhance the ability of large language models, enabling them to rely on actual data support when answering questions. This method improves the accuracy of answers to a certain extent.

[0005] However, existing knowledge bases lack an effective classification system, contain a large number of documents, and lack fine-grained classification. This results in RAG having to search multiple knowledge bases during the recall phase, increasing retrieval complexity and time costs. For example, in the field of asset equipment maintenance, when a user queries for solutions to "excessive temperature rise on a conveyor reducer," without specifically selecting the "equipment knowledge" category, the system may simultaneously search other unrelated categories, including "management regulations," reducing retrieval efficiency. Summary of the Invention

[0006] The present invention provides a knowledge retrieval method based on multi-layer indexing. By constructing a three-layer index based on knowledge classification tags, document summaries and text blocks, it achieves comprehensive coverage from macro to micro, enhances the accuracy of retrieval, and effectively solves the problem of low retrieval efficiency in the existing technology.

[0007] The technical solution adopted in the present invention is:

[0008] A knowledge retrieval method based on multi-layer indexing, comprising:

[0009] According to the knowledge retrieval statement and the knowledge classification label, the knowledge base of the corresponding classification is obtained;

[0010] According to the knowledge retrieval statement, combined with the document summaries in the knowledge base of the corresponding classification, a plurality of retrieval target documents are obtained through vector similarity analysis;

[0011] According to the knowledge retrieval statement, the text blocks of multiple retrieval target documents are combined and vector similarity analysis is performed to obtain multiple similar text blocks as knowledge retrieval output.

[0012] The multi-layer indexing-based knowledge retrieval method disclosed in the present invention also has the following additional technical features:

[0013] According to the knowledge search statement and the knowledge classification label, the knowledge base of the corresponding classification is obtained, which is as follows:

[0014] Based on the knowledge retrieval statement and the label description of the knowledge classification label, the label prompt word is obtained. Based on the semantic understanding and context relevance of the large model, the classification probability corresponding to the knowledge retrieval statement and the label prompt word is obtained;

[0015] Priority sorting is performed based on the classification probabilities to obtain a knowledge base of the classification corresponding to the knowledge retrieval sentence.

[0016] The knowledge base is specifically:

[0017] A knowledge base is set up according to the knowledge classification tags, and a classification description is set up, wherein the knowledge classification tags include at least one of equipment knowledge and maintenance experience;

[0018] According to the classification description, the document is set in the knowledge base corresponding to the knowledge classification label.

[0019] According to the knowledge retrieval statement, combined with the document abstracts in the knowledge base of the corresponding category, through vector similarity analysis, specifically:

[0020] The document similarity is obtained by combining the sentence vector obtained from the knowledge retrieval sentence with the document vector obtained from the document summary in the knowledge base of the corresponding category through vector similarity analysis;

[0021] According to the document similarity, a plurality of retrieval target documents are obtained.

[0022] The document vector obtained by the document summary is specifically:

[0023] Based on the document, a large language model is used to summarize and generate a summary; based on the summary, a document vector is obtained through vectorization processing.

[0024] According to the knowledge retrieval statement, combined with the text blocks of multiple retrieval target documents, multiple similar text blocks are obtained through vector similarity analysis, specifically:

[0025] A first similarity is obtained by combining a sentence vector obtained from a knowledge retrieval sentence with a text block vector obtained from text blocks of a plurality of retrieval target documents through vector similarity analysis;

[0026] Obtaining a first similar text block of each of the retrieval target documents using the first similarity, and adjusting the first similarity based on the distance similarity between the text block and the first similar text block in the corresponding retrieval target document to obtain a second similarity;

[0027] A plurality of similar text blocks are obtained according to the second similarity.

[0028] The first similarity is adjusted according to the distance similarity between the text block and the first similar text block in the corresponding search target document, specifically:

[0029]

[0030] Among them, score new Indicates the second similarity, score old Indicates the first similarity; C max represents the coefficient of the first similar text block; k represents the attenuation coefficient; index represents the position index of the current text block; index0 represents the position index of the first similar text block.

[0031] The text block is specifically:

[0032] The document is split into multiple text blocks, and overlapping segments are set between adjacent text blocks; and a position index is set for each text block.

[0033] The present invention also discloses a storage medium.

[0034] The storage medium stores a computer program, which implements the steps of the multi-layer index-based knowledge retrieval method when executed.

[0035] The present invention again discloses a processing device, comprising:

[0036] memory for storing computer programs;

[0037] A processor is used to implement the steps of the multi-layer index-based knowledge retrieval method when executing the computer program.

[0038] Due to the adoption of the above technical solution, the beneficial effects achieved by the present invention are as follows:

[0039] 1. In the present invention, based on the knowledge retrieval statement and combined with the knowledge classification label, a knowledge base of the corresponding classification is obtained. A large language model is used to perform semantic analysis on the query statement, automatically matching the most relevant knowledge classification label (such as "equipment knowledge", "maintenance experience"), and narrowing the search scope to the knowledge base in a specific field. Traditional RAG technology lacks fine-grained classification and needs to traverse multiple irrelevant knowledge bases, resulting in low retrieval efficiency. This method directly locates the highly relevant knowledge base through dynamic classification adaptation, significantly reducing redundant calculations, improving classification accuracy, and shortening retrieval response time.

[0040] Based on the knowledge search statement and the corresponding document summaries within the knowledge base, vector similarity analysis is performed to identify multiple target documents. Within the filtered knowledge base, vector representations of the document summaries are extracted and similarity is calculated with the query statement vector to quickly locate candidate documents. Traditional RAG relies on full-text search and cannot quickly filter out irrelevant documents. This method, through efficient comparison of summary vectors, retains only highly relevant documents, avoiding inefficient scanning and improving document recall speed.

[0041] Based on the knowledge retrieval statement, text blocks from multiple target documents are combined and vector similarity analysis is performed to obtain multiple similar text blocks as the knowledge retrieval output. Further refinement is performed at the text block level, using vector similarity analysis to obtain text blocks that are highly relevant to the query statement while implicitly preserving their contextual logical relationships. Traditional block segmentation methods can easily lead to contextual fragmentation and key information fragmentation. This method uses vector matching at the text block level to ensure that the output results contain complete operational steps or logical chains, improving the contextual completeness of the retrieval text blocks and enhancing the quality score of answer generation.

[0042] The present invention realizes accurate retrieval from macro to micro by constructing a three-layer index structure (knowledge classification label → document summary → text block), solving the problems of classification ambiguity, document recall efficiency, and context fragmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0044] Figure 1 The figure is a flowchart of the multi-layer indexing-based knowledge retrieval method according to one embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to more clearly illustrate the overall concept of the present invention, a detailed description is given below in an exemplary manner in conjunction with the accompanying drawings.

[0046] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0047] like Figure 1 As shown, a knowledge retrieval method based on multi-layer indexing includes:

[0048] S100: Based on the knowledge search statement and the knowledge classification label, a knowledge base of the corresponding classification is obtained.

[0049] The core goal of this step is to address the existing issues of ambiguous knowledge base classification and overly broad search scope. By dynamically matching knowledge search statements with pre-set knowledge classification tags, the search scope is narrowed to the most relevant knowledge base, thereby improving search efficiency and accuracy.

[0050] Knowledge search statements are user-entered queries, such as "How to deal with excessive temperature rise in a conveyor reducer?" Knowledge classification tags are pre-defined classification systems, such as "Equipment Knowledge," "Maintenance Experience," and "Management Regulations." Classification descriptions are detailed descriptions of each tag, defining its scope of application (e.g., "Equipment Knowledge" is described as "covering equipment operating principles and fault diagnosis methods").

[0051] The knowledge retrieval statement and each label are input into the large language model (LLM), and the classification probability of the statement and label is calculated through semantic understanding and contextual relevance analysis. If the semantic match between the statement and the "device knowledge" label is the highest (such as a probability of 85%), the knowledge base corresponding to this label is given priority. If the probabilities of multiple labels are close (such as "device knowledge" 85% vs. "maintenance experience" 78%), further screening is performed through priority sorting rules (such as label weights and user historical preferences). Among them, the classification probability is the semantic match value between the statement and the label output by the large model (such as a probability value between 0 and 1).

[0052] The results are sorted according to the classification probability, and the most relevant knowledge base (such as the "device knowledge" base) is output.

[0053] In general, in traditional RAG (Retrieval-Augmented Generation) technology, knowledge bases often lack fine-grained classification, requiring the system to traverse multiple unrelated knowledge bases during the recall phase (for example, mixed searches for "equipment knowledge" and "management regulations"). This significantly increases computational overhead and reduces the relevance of the results. This step introduces a semantic matching mechanism for knowledge classification labels to directly locate highly relevant knowledge bases, avoiding redundant computation.

[0054] S200: According to the knowledge retrieval statement, combined with the document summaries in the knowledge base of the corresponding classification, a plurality of retrieval target documents are obtained through vector similarity analysis.

[0055] The core goal of this step is to quickly locate target documents that are highly relevant to the query within the filtered knowledge base by analyzing the vector similarity of document summaries. This step addresses the inefficient retrieval problem caused by full document scanning in existing technologies while also improving the accuracy of document relevance.

[0056] It is understandable that the knowledge retrieval statement is a query request input by the user (such as "How to deal with excessive temperature rise of the conveyor reducer"). The document summary is a collection of document summaries in the classified knowledge base, each of which is generated by a large language model and stored in vector form.

[0057] A large language model (such as LLM) is used to summarize the original document and generate a concise summary. The document summary, generated by the large language model, is a brief description of the document, condensing the core content of the document (e.g., "The scraper conveyor equipment documentation includes usage precautions, component principles, repair and maintenance information, and other information"). A vectorization model (such as Sentence-BERT) is used to convert the summary into a fixed-dimensional vector representation to facilitate subsequent similarity calculations. This vectorization significantly reduces computational overhead and conserves computing resources compared to directly processing the full document.

[0058] After the knowledge retrieval statement is vectorized, it is compared with the vector of each document summary for similarity (such as cosine similarity or Euclidean distance). Vector similarity analysis evaluates the degree of match between the query statement and the document summary by calculating a similarity metric (such as cosine similarity) between the two vectors.

[0059] Documents are sorted by similarity score, and the top-K (e.g., top 5-10) documents with the highest similarity are selected as search targets. If multiple documents have similar scores, they are further filtered based on priority criteria (e.g., document update time, authority score). The top-K documents are the first K documents selected after sorting by similarity score and are used for subsequent refined searches (e.g., K = 5).

[0060] Output the Top-K documents selected after vector similarity analysis for subsequent refined retrieval at the text block level.

[0061] In general, in traditional RAG technology, the knowledge base lacks an effective classification system, requiring the system to traverse all documents to find matching content, resulting in wasted computing resources and poorly-quality results. This step vectorizes document summaries, using a large language model to summarize the document content and store the summaries in vector form, reducing the complexity of directly processing the full text. Based on vector similarity analysis, the similarity between the query statement and the document summary vector is calculated to quickly identify the top-K candidate documents, avoiding inefficient word-by-word comparisons.

[0062] S300: According to the knowledge retrieval statement, multiple text blocks of the retrieval target documents are combined and vector similarity analysis is performed to obtain multiple similar text blocks as knowledge retrieval output.

[0063] The core goal of this step is to solve the context break problem caused by document segmentation in traditional RAG technology, and to ensure that the retrieval results are both relevant and logically coherent through vector similarity analysis and position proximity adjustment.

[0064] In this step, the knowledge retrieval statement is the query request entered by the user (e.g., "How to deal with excessive temperature rise in a conveyor reducer?"). The target documents are the top-K documents filtered through the previous step (e.g., five documents related to "equipment knowledge"). The text blocks are segments of a document divided into fixed sizes (e.g., 300 words).

[0065] Use a large language model (such as the Sentence-BERT model) to vectorize each text block. Vectorization reduces the complexity of directly processing the entire text and saves computing resources compared to traditional methods.

[0066] After vectorizing the knowledge search statement, perform a similarity comparison (e.g., cosine similarity) with all text block vectors. Sorting is performed based on similarity, and the top-N (e.g., top-5) text blocks are selected as output. The sorted top-N text blocks are output for subsequent answer generation.

[0067] In general, existing technologies often fragment key information by segmenting documents into different blocks. For example, the operating procedures in a device maintenance manual are divided into different blocks, making the search results unable to cover the entire operation process. This step directly compares the vectors of text blocks through vector similarity analysis at the text block level, avoiding the inefficiency of full document scanning and enhancing the integrity of text block associations.

[0068] As a preferred embodiment of the present invention, based on the knowledge search statement and the knowledge classification label, a knowledge base of the corresponding classification is obtained, specifically:

[0069] Based on the knowledge retrieval statement and the label description of the knowledge classification label, the label prompt word is obtained. Based on the semantic understanding and context relevance of the large model, the classification probability corresponding to the knowledge retrieval statement and the label prompt word is obtained;

[0070] Priority sorting is performed based on the classification probabilities to obtain a knowledge base of the classification corresponding to the knowledge retrieval sentence.

[0071] The core goal of this implementation is to solve the problems of fuzzy knowledge base classification and excessive search scope in the existing technology. By dynamically matching knowledge search statements with preset knowledge classification tags, the search scope is narrowed to the most relevant knowledge base, thereby improving search efficiency and accuracy.

[0072] The knowledge search statement is a query request entered by the user (e.g., "How to deal with excessive temperature rise of the conveyor reducer"). The knowledge classification tag is a preset classification system, such as "equipment knowledge," "maintenance experience," and "management regulations." Each tag contains a classification description (e.g., "equipment knowledge covers equipment working principles and fault diagnosis methods").

[0073] Generate corresponding tag prompts based on the classification descriptions of knowledge classification tags. For example, prompts for the "Equipment Knowledge" tag might be "Equipment Working Principle, Fault Diagnosis, Repair Procedures." Use natural language processing (NLP) tools to extract keywords from classification descriptions, or leverage large models to generate more precise prompts.

[0074] The knowledge retrieval sentence and each label prompt word are input into the large language model (LLM), and the classification probability of the sentence and label prompt word is calculated through semantic understanding and context relevance analysis.

[0075] If the sentence has the highest semantic match with the "Device Knowledge" tag (e.g., 85% probability), the knowledge base corresponding to that tag is prioritized. If the probabilities of multiple tags are similar (e.g., 85% for "Device Knowledge" vs. 78% for "Maintenance Experience"), further screening is performed using priority sorting rules (e.g., tag weights and user historical preferences).

[0076] The results are sorted according to the classification probability, and the most relevant knowledge base (such as the "device knowledge" base) is output.

[0077] In summary, traditional RAG technology, due to its unclassified or coarse-grained classification, requires searching multiple irrelevant knowledge bases, resulting in a mixture of low-relevance content in the search results. This invention directly locates high-relevance knowledge bases through semantic matching of large models, improving classification accuracy.

[0078] Traditional methods require traversing all knowledge bases, which is time-consuming. However, this step filters through classification tags and searches only the target knowledge base, shortening response time. For the query "Conveyor reducer temperature rise is too high," traditional RAG might simultaneously search multiple libraries, such as "Equipment Knowledge" and "Management Regulations." However, this method only searches the "Equipment Knowledge" library, reducing unnecessary computational overhead.

[0079] This embodiment solves the problems of fuzzy classification and excessive retrieval scope in traditional RAG technology through a dynamic classification adaptation mechanism.

[0080] As an example of this implementation, the knowledge base is specifically:

[0081] A knowledge base is set up according to the knowledge classification tags, and a classification description is set up, wherein the knowledge classification tags include at least one of equipment knowledge and maintenance experience;

[0082] According to the classification description, the document is set in the knowledge base corresponding to the knowledge classification label.

[0083] The core goal of this embodiment is to solve the problems of fuzzy knowledge base classification and low document retrieval efficiency in the prior art. Through fine-grained classification tags and classification descriptions, accurate document classification is achieved, providing an efficient indexing foundation for subsequent retrieval.

[0084] Among them, at least two classification tags are preset (such as "equipment knowledge" and "maintenance experience"), and each tag corresponds to a specific field (such as equipment knowledge covers equipment principles and fault diagnosis; maintenance experience covers maintenance cases and operating specifications). Write a detailed description for each classification tag to clarify its scope of application. For example, the classification description of "equipment knowledge" is "covering the systematic understanding of equipment working principles, structural characteristics, operating specifications, maintenance points and fault diagnosis methods." Through classification descriptions, ensure that users can clearly select the most relevant classification tag when uploading documents to avoid misclassification of documents.

[0085] When users upload documents, they must select a corresponding category tag (e.g., "Equipment Knowledge"). Based on the category description, the system verifies that the document content fits within the selected category tag. For example, if a user uploads a document about "lubrication system maintenance," the system must verify that it falls under "Equipment Knowledge" rather than "Maintenance Experience."

[0086] Store documents in the knowledge base under the corresponding category label, and establish an independent vector knowledge base for each category label (such as "device knowledge vector library"). By storing independent categories, the interference of cross-category searches is reduced and the search efficiency is improved.

[0087] It's important to note that users are allowed to add new classification tags (such as "Equipment Management Regulations") based on business needs and configure corresponding classification descriptions. Hierarchical relationships between classification tags are also supported (for example, "Equipment Knowledge" can be subdivided into "Conveyor Knowledge" and "Motor Knowledge"). This improves the system's flexibility and adaptability to meet the classification needs of different industries or scenarios.

[0088] In summary, traditional RAG technology, due to its unclassified or coarse-grained classification, requires searching multiple irrelevant knowledge bases, resulting in a mixture of low-relevance content in the search results. This invention directly locates the highly relevant knowledge base through the precise matching of classification labels and classification descriptions, thus improving classification accuracy.

[0089] As a preferred embodiment of the present invention, according to the knowledge retrieval statement, combined with the document abstracts in the knowledge base of the corresponding classification, vector similarity analysis is performed, specifically:

[0090] The document similarity is obtained by combining the sentence vector obtained from the knowledge retrieval sentence with the document vector obtained from the document summary in the knowledge base of the corresponding category through vector similarity analysis;

[0091] According to the document similarity, a plurality of retrieval target documents are obtained.

[0092] The core goal of this implementation is to quickly locate target documents highly relevant to the query within a filtered knowledge base through vector similarity analysis of document summaries. This solves the inefficient retrieval problem caused by full document scanning in existing technologies while also improving the accuracy of document relevance.

[0093] The knowledge retrieval statement is a query request entered by the user (e.g., “How to deal with excessive temperature rise of the conveyor reducer”). The document summary is a collection of document summaries in the classified knowledge base, each of which is generated by a large language model and stored in vector form.

[0094] Use a large language model (such as LLM) to summarize the original document and generate a concise summary (for example, "The scraper conveyor equipment document contains information on usage precautions, composition principles, repair and maintenance, etc."). Use a vectorization model (such as Sentence-BERT) to convert the summary into a fixed-dimensional vector representation to facilitate subsequent similarity calculations. This vectorization significantly reduces computational overhead and conserves computing resources compared to directly processing the full document.

[0095] After the knowledge retrieval sentence is vectorized, a similarity comparison (such as cosine similarity or Euclidean distance) is performed with the vector of each document summary.

[0096] In a specific embodiment, the similarity score formula is:

[0097]

[0098] Where Q is the query statement vector, D i is the summary vector of the i-th document.

[0099] Documents are sorted by similarity score, and the top-K (e.g., top 5-10) documents with the highest similarity are selected as retrieval targets. If multiple documents have similar scores, they are further filtered based on priority criteria (e.g., document update time, authority score). The top-K documents selected after vector similarity analysis are output for subsequent refined retrieval at the text block level.

[0100] In general, in traditional RAG technology, the knowledge base has not formed an effective classification system. The system needs to traverse all documents to find matching content, resulting in a waste of computing resources and difficulty in ensuring the quality of results. This implementation method vectorizes document summaries, uses a large language model to summarize the document content and generate a summary, and stores the summary in vector form, reducing the complexity of directly processing the full text. Through vector similarity analysis, by calculating the similarity between the query statement and the document summary vector, the Top-K candidate documents are quickly screened out, avoiding inefficient word-by-word comparison.

[0101] As a preferred embodiment of this implementation, the document vector obtained by document summary is specifically:

[0102] Based on the document, a large language model is used to summarize and generate a summary; based on the summary, a document vector is obtained through vectorization processing.

[0103] The core goal of this implementation is to generate summaries from original documents using a large language model (LLM) and vectorize them to build an efficient and accurate document vector database. This solves the inefficient retrieval problem caused by full document scanning in existing technologies, while also improving the accuracy of document relevance.

[0104] When users upload documents, they must select a corresponding classification tag (such as "Equipment Knowledge"). The system then verifies that the document's content meets the classification requirements based on the classification description. For example, if a user uploads a document about "lubrication system maintenance," the system must verify that it falls under "Equipment Knowledge" rather than "Maintenance Experience." Accurately matching classification tags prevents misclassification of documents and reduces interference with subsequent searches.

[0105] Use a large language model (such as LLM) to summarize document content and generate a concise summary. For example, for the document "Scraper Conveyor Equipment Documentation," the generated summary content is: "Scraper Conveyor Equipment Documentation Content, mainly including scraper conveyor usage precautions, component principles, repair and maintenance, including..." Summarization significantly reduces computational overhead and saves computing resources compared to directly processing the full document.

[0106] The summary is converted into a fixed-dimensional vector representation through a vectorization model (such as Sentence-BERT). Fixed-dimensional vectors facilitate efficient comparison and support fast retrieval of large-scale documents.

[0107] The generated document vector is stored in the document summary vector database and associated with the classification label and document ID as the index basis. By indexing the classification label and document ID, subsequent searches can quickly locate the target document, avoiding a full scan.

[0108] In general, in traditional RAG (Retrieval-Augmented Generation) technology, the knowledge base lacks an effective classification system, requiring the system to traverse all documents to find matching content, resulting in wasted computing resources and poorly-quality results. This step generates document summaries using a large language model to condense document content, extract core information, and reduce redundant data. Through vectorization, the summaries are converted into fixed-dimensional vector representations, facilitating subsequent vector similarity analysis and significantly reducing computational complexity.

[0109] As a preferred embodiment of the present invention, according to the knowledge retrieval statement, combined with the text blocks of multiple retrieval target documents, multiple similar text blocks are obtained through vector similarity analysis, specifically:

[0110] A first similarity is obtained by combining a sentence vector obtained from a knowledge retrieval sentence with a text block vector obtained from text blocks of a plurality of retrieval target documents through vector similarity analysis;

[0111] Obtaining a first similar text block of each of the retrieval target documents using the first similarity, and adjusting the first similarity based on the distance similarity between the text block and the first similar text block in the corresponding retrieval target document to obtain a second similarity;

[0112] A plurality of similar text blocks are obtained according to the second similarity.

[0113] The core goal of this implementation is to solve the context breakage problem caused by document segmentation in traditional RAG technology, and to ensure that the retrieval results are both relevant and logically coherent through vector similarity analysis and position proximity adjustment.

[0114] The knowledge retrieval statement is a user-entered query (e.g., "How do I handle excessive temperature rise on a conveyor reducer?"). The target documents are the top-K documents filtered through the previous steps (e.g., five documents related to "equipment knowledge"). Text chunking involves splitting a document into chunks with 30% overlap, and assigning a position index to each chunk (e.g., chunk 1, chunk 2, and so on, for document A).

[0115] It should be noted that the text block is specifically split into multiple text blocks, with overlapping segments set between adjacent text blocks, and a position index set for each text block. The core goal of this step is to solve the contextual disconnection problem caused by document segmentation in traditional RAG technology. By setting overlapping segments and position indexes, the search results are both relevant and logically coherent.

[0116] In existing technologies, document segmentation often leads to fragmentation of key information. For example, the operating procedures in a device maintenance manual are divided into different blocks, making the search results unable to cover the entire process. This step uses overlapping fragment design to set overlapping fragments between adjacent text blocks, preserving contextual continuity. Position index association assigns a unique position identifier to each text block, facilitating subsequent secondary sorting and contextual integrity analysis.

[0117] Among them, the document is divided into multiple text blocks of a fixed size (such as 300 words), and 30% overlapping segments are set between adjacent blocks (such as the last 90 words of the previous block overlap with the first 90 words of the next block). Overlapping segments ensure that key information is not separated. For example, the "lubrication check → load adjustment → heat dissipation detection" steps in the maintenance manual may be fully retained in adjacent blocks. The proportion of overlapping segments is dynamically adjusted according to the length of the document (such as 30% for long documents and 20% for short documents) to avoid excessive redundancy. In addition, natural language processing (NLP) tools are used to detect whether the segmented text contains complete semantic units (such as sentences or paragraphs).

[0118] Each text block is assigned a unique position index in the format of "document ID-block number" (e.g., "doc_001-block_3"). The text block vector and position index are stored together in the text block vector database to facilitate rapid location during subsequent retrieval. The position index provides the basis for subsequent secondary similarity adjustments, ensuring that high-scoring blocks and their adjacent blocks are prioritized.

[0119] Use a large language model (such as the Sentence-BERT model) to vectorize each text block. Vectorization reduces the complexity of directly processing the entire text, saving 70% of computing resources compared to traditional methods.

[0120] After the knowledge retrieval statement is vectorized, it is compared with all text block vectors for similarity (e.g., cosine similarity). The text blocks are preliminarily sorted based on the first similarity, and the top N candidate text blocks are retained (e.g., Top-10).

[0121] For each text block in the document, the position distance between the text block and the block with the highest score (ie, the block with the first highest similarity) in the document is calculated.

[0122] The first similarity is adjusted according to the distance similarity between the text block and the first similar text block in the corresponding search target document, specifically:

[0123]

[0124] Among them, score new Indicates the second similarity, score old Indicates the first similarity; C max represents the coefficient of the first similar text block; k represents the attenuation coefficient; index represents the position index of the current text block; index0 represents the position index of the first similar text block.

[0125] The core goal of this step is to solve the problem of key information fragmentation caused by text block segmentation in traditional retrieval, and to improve the contextual integrity and relevance of retrieval results by introducing position proximity adjustment.

[0126] The knowledge retrieval sentence and text block are vectorized separately using the Large Language Model (LLM), and the cosine similarity, i.e. the first similarity, is calculated.

[0127] For each text block within a document, we calculate its distance to the highest-scoring block within the document (i.e., the block with the highest first similarity), and introduce an exponential decay function to adjust the first similarity. This exponential decay function ensures that the similarity between the high-scoring block and its adjacent blocks is preserved, while the similarity of distant blocks is attenuated.

[0128] In the search field, existing technologies often split documents into independent text blocks for vectorized retrieval. However, this block segmentation can result in key technological innovations being dispersed across different blocks (e.g., "motor heat dissipation structure" and "electromagnetic winding optimization" are separated into different blocks). This step uses positional proximity adjustment to calculate the positional distance between the text block and the highest-scoring block, exponentially decaying the original similarity to preserve contextual continuity. Combining the original similarity with positional proximity, a secondary similarity is generated to ensure that highly relevant blocks and their adjacent blocks are prioritized.

[0129] The text blocks are finally sorted based on the second similarity, and the top-N (such as top-5) are selected as output. The top-N text blocks after the second sorting are output for subsequent answer generation.

[0130] In general, in existing technologies, document segmentation often leads to the fragmentation of key information. For example, the operating steps in a device maintenance manual are divided into different blocks, making the search results unable to cover the complete operating process. This embodiment directly compares the vectors of text blocks through vector similarity analysis at the text block level, avoiding the inefficiency of full document scanning. Through positional proximity adjustment, the positional distance between the text block and the highest-scoring block is introduced as a secondary sorting criterion to enhance contextual integrity.

[0131] The present invention also provides a storage medium,

[0132] The storage medium stores a computer program, which implements the steps of the multi-layer index-based knowledge retrieval method when executed.

[0133] Therefore, any effect of the knowledge retrieval method based on multi-layer indexing can be achieved, which will not be elaborated here.

[0134] The present invention again provides a processing device, comprising:

[0135] memory for storing computer programs;

[0136] A processor is used to implement the steps of the multi-layer index-based knowledge retrieval method when executing the computer program.

[0137] Therefore, any effect of the knowledge retrieval method based on multi-layer indexing can be achieved, which will not be elaborated here.

[0138] Anything not described in the present invention can be achieved by adopting or drawing on existing technologies.

[0139] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0140] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A knowledge retrieval method based on multi-layer indexing, characterized in that: include: According to the knowledge retrieval statement and the knowledge classification label, the knowledge base of the corresponding classification is obtained; According to the knowledge retrieval statement, combined with the document summaries in the knowledge base of the corresponding classification, a plurality of retrieval target documents are obtained through vector similarity analysis; According to the knowledge retrieval statement, the text blocks of multiple retrieval target documents are combined and vector similarity analysis is performed to obtain multiple similar text blocks as knowledge retrieval output.

2. The knowledge retrieval method based on multi-layer indexing according to claim 1, characterized in that: According to the knowledge search statement and the knowledge classification label, the knowledge base of the corresponding classification is obtained, which is as follows: Based on the knowledge retrieval statement and the label description of the knowledge classification label, the label prompt word is obtained. Based on the semantic understanding and context relevance of the large model, the classification probability corresponding to the knowledge retrieval statement and the label prompt word is obtained; Priority sorting is performed based on the classification probabilities to obtain a knowledge base of the classification corresponding to the knowledge retrieval sentence.

3. The knowledge retrieval method based on multi-layer indexing according to claim 2, characterized in that: The knowledge base is specifically: A knowledge base is set up according to the knowledge classification tags, and a classification description is set up, wherein the knowledge classification tags include at least one of equipment knowledge and maintenance experience; According to the classification description, the document is set in the knowledge base corresponding to the knowledge classification label.

4. The knowledge retrieval method based on multi-layer indexing according to claim 1, characterized in that: According to the knowledge retrieval statement, combined with the document abstracts in the knowledge base of the corresponding category, through vector similarity analysis, specifically: The document similarity is obtained by combining the sentence vector obtained from the knowledge retrieval sentence with the document vector obtained from the document summary in the knowledge base of the corresponding category through vector similarity analysis; According to the document similarity, a plurality of retrieval target documents are obtained.

5. The knowledge retrieval method based on multi-layer indexing according to claim 4 is characterized in that: The document vector obtained by the document summary is specifically: Based on the document, a large language model is used to summarize and generate a summary; based on the summary, a document vector is obtained through vectorization processing.

6. The knowledge retrieval method based on multi-layer indexing according to claim 1, characterized in that: According to the knowledge retrieval statement, combined with the text blocks of multiple retrieval target documents, multiple similar text blocks are obtained through vector similarity analysis, specifically: A first similarity is obtained by combining a sentence vector obtained from a knowledge retrieval sentence with a text block vector obtained from text blocks of a plurality of retrieval target documents through vector similarity analysis; Obtaining a first similar text block of each of the retrieval target documents using the first similarity, and adjusting the first similarity based on the distance similarity between the text block and the first similar text block in the corresponding retrieval target document to obtain a second similarity; A plurality of similar text blocks are obtained according to the second similarity.

7. The knowledge retrieval method based on multi-layer indexing according to claim 6, characterized in that: The first similarity is adjusted according to the distance similarity between the text block and the first similar text block in the corresponding search target document, specifically: Among them, score new Indicates the second similarity, score old Indicates the first similarity; C max represents the coefficient of the first similar text block; k represents the attenuation coefficient; index represents the position index of the current text block; index0 represents the position index of the first similar text block.

8. The knowledge retrieval method based on multi-layer indexing according to claim 7, characterized in that: The text block is specifically: The document is split into multiple text blocks, and overlapping segments are set between adjacent text blocks; and a position index is set for each text block.

9. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed, implements the steps of the multi-layer index-based knowledge retrieval method according to any one of claims 1 to 8.

10. A processing device, characterized in that: include: memory for storing computer programs; A processor is configured to implement the steps of the multi-layer indexing-based knowledge retrieval method according to any one of claims 1 to 8 when executing the computer program.

Images