A hierarchical search method, storage medium and computer device

Abstract

This invention discloses a hierarchical retrieval method, storage medium, and computer device, belonging to the field of document processing technology. It includes: S100, obtaining the hierarchical paragraph parsing results of the input document; S200, retrieving the title paragraphs in the parsing results according to the user's query, generating a set of title paragraphs based on the retrieval results, determining whether the title paragraph set is empty, and if not, outputting a semantically complete paragraph; if so, proceeding to step S300; S300, retrieving the body text paragraphs in the parsing results according to the user's query, and outputting a semantically complete paragraph or summary based on the retrieval results. This invention, by combining the hierarchical parsing results of the document, can improve the completeness of text retrieval results in large-scale model retrieval and enhance document question answering, thereby improving the correctness and completeness of the generated answers in document question answering.

Description

Technical Field

[0001] This invention relates to the field of document processing technology, specifically to a hierarchical retrieval method, storage medium, and computer device. Background Technology

[0002] Large-Model Retrieval Augmented (RAG) document question answering systems are widely used in intelligent question answering scenarios. Document-related fragment retrieval is an essential stage in current RAG systems, aiming to find document fragments relevant to the user's input question. The process generally involves:

[0003] 1. Document parsing stage

[0004] In this stage, user-uploaded documents (such as Word documents and PDF documents) are parsed into plain text format and divided into multiple fragments of fixed length. Then, the document fragments are converted into semantic vectors using the Embedding model and saved to the vector database.

[0005] 2. Vector retrieval stage

[0006] In this stage, the user-input query is first converted into a vector using the Embedding model. Then, the similarity between the query vector and the document fragment vector is calculated in the vector database, and document fragments with similarity greater than the threshold are returned.

[0007] 3. Answer generation stage

[0008] In this stage, prompts for the large model are constructed using the retrieved document fragments, and the prompts are input into the large model via the large model interface to generate answers, which are then returned to the user.

[0009] The main problem with the above-mentioned large model-based document question answering system is that the document fragments retrieved using vectors can generally only return the most relevant paragraphs, and cannot return all semantically complete fragments. This results in insufficient contextual information when the large model generates answers, leading to unreliable and incomplete answer quality. Summary of the Invention

[0010] This invention provides a hierarchical retrieval method, storage medium, and computer device to address the problem that existing technologies using vector retrieval generally only return the most relevant paragraphs of document fragments, failing to return all semantically complete fragments. This results in insufficient contextual information, unreliable and incomplete answer quality when large models generate answers.

[0011] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0012] This invention provides a hierarchical retrieval method, comprising the following steps:

[0013] S100. Obtain the hierarchical paragraph parsing results of the input document;

[0014] S200: Retrieve the title paragraphs in the parsed results according to the user's query statement, generate a set of title paragraphs according to the search results, determine whether the set of title paragraphs is empty, if not, output the semantically complete paragraphs, if yes, proceed to step S300.

[0015] S300. Based on the user's query, retrieve the main text paragraphs in the parsed results and output semantically complete paragraphs or summaries based on the retrieval results.

[0016] Based on this, the present invention can be further improved as follows: In step S100, the method for obtaining the hierarchical paragraph parsing results of documents with different formats includes:

[0017] S110, Read the input document;

[0018] S120. Parse the document into tree-structured text fragments according to paragraphs, and divide the paragraphs into heading paragraphs and body paragraphs;

[0019] S130. Using the Embedding model, the parsed tree-structured text fragments are converted into vectors and stored in a vector database.

[0020] Based on this, the present invention can be further improved as follows: In step S200, the method for retrieving the title paragraphs in the parsed results according to the user query statement includes:

[0021] S210. Convert the user query statement into a Q-vector using the Embedding model;

[0022] S220. Calculate the similarity between the Q vector and all title paragraph vectors in the vector database, and determine whether the similarity is greater than the preset threshold. If so, save the title paragraph.

[0023] S230. Create a title paragraph set and put all the saved title paragraphs into the title paragraph set.

[0024] Based on this, the present invention can be further improved as follows: In step S300, the method for retrieving the main text paragraphs in the parsing results according to the user query includes:

[0025] S310. Establish a text set, calculate the similarity between the Q vector and all text paragraph vectors in the vector database, and put all text paragraph vectors with similarity greater than a preset threshold into the text set.

[0026] S320. Determine whether the text set is empty. If not, proceed to step S330. If yes, proceed to step S350.

[0027] S330: Obtain all main text paragraphs and context text with a similarity greater than a preset threshold;

[0028] S340. Based on the obtained main text paragraph and context text, output a semantically complete paragraph;

[0029] S350. Output a summary of the document based on the input document content.

[0030] Based on this, the present invention can be further improved as follows: In step S330, the method for obtaining all main text paragraphs and context text content with a similarity greater than a preset threshold includes:

[0031] S331. Obtain the set of heading levels where the main text paragraphs are located in the main text paragraph set;

[0032] S332. Determine the hierarchical inclusion relationship C in the title level set, and obtain the next level T above the level of the main text paragraph;

[0033] S333. Based on the level T above the main text paragraph, obtain the text content contained in the two headings above and below the heading containing the main text paragraph in level T.

[0034] Based on this, the present invention can be further improved as follows: In step S350, the method for outputting a summary of the document based on the input document content includes:

[0035] The document content is input into a large language model, and a summary of the document is generated based on the large language model.

[0036] The present invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of the method described above.

[0037] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described above.

[0038] The beneficial effects provided by this invention are:

[0039] This invention provides a hierarchical retrieval method for large-model retrieval enhanced document question answering scenarios. By combining the hierarchical parsing results of documents, it can improve the completeness of text retrieval results in large-model retrieval enhanced document question answering, thereby improving the correctness and completeness of the generated answers in document question answering.

[0040] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of a hierarchical retrieval method provided in an embodiment of the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0043] like Figure 1 As shown, the present invention provides a hierarchical retrieval method, characterized by comprising the following steps:

[0044] S100. Obtain the hierarchical paragraph parsing results of the input document;

[0045] S200: Retrieve the title paragraphs in the parsed results according to the user's query statement, generate a set of title paragraphs according to the search results, determine whether the set of title paragraphs is empty, if not, output the semantically complete paragraphs, if yes, proceed to step S300.

[0046] S300. Based on the user's query, retrieve the main text paragraphs in the parsed results and output semantically complete paragraphs or summaries based on the retrieval results.

[0047] The solution provided by this invention has the following effects:

[0048] This invention can accurately return semantically complete document fragments that are relevant to the user's query, thereby effectively improving the quality of answer generation for large models.

[0049] Optionally, in some possible implementations, the method for obtaining the hierarchical paragraph parsing results of documents with different formats in step S100 includes:

[0050] S110, Read the input document;

[0051] S120. Parse the document into tree-structured text fragments according to paragraphs, and divide the paragraphs into heading paragraphs and body paragraphs;

[0052] S130. Using the Embedding model, the parsed tree-structured text fragments are converted into vectors and stored in a vector database.

[0053] Optionally, in some possible implementations, the method for retrieving the title paragraphs in the parsed results based on the user query statement in step S200 includes:

[0054] S210. Convert the user query statement into a Q-vector using the Embedding model;

[0055] S220. Calculate the similarity between the Q vector and all title paragraph vectors in the vector database, and determine whether the similarity is greater than the preset threshold. If so, save the title paragraph.

[0056] S230. Create a title paragraph set and put all the saved title paragraphs into the title paragraph set.

[0057] Optionally, in some possible implementations, the method for retrieving text paragraphs from the parsed results based on the user query in step S300 includes:

[0058] S310. Establish a text set, calculate the similarity between the Q vector and all text paragraph vectors in the vector database, and put all text paragraph vectors with similarity greater than a preset threshold into the text set.

[0059] S320. Determine whether the text set is empty. If not, proceed to step S330. If yes, proceed to step S350.

[0060] S330: Obtain all main text paragraphs and context text with a similarity greater than a preset threshold;

[0061] S340. Based on the obtained main text paragraph and context text, output a semantically complete paragraph;

[0062] S350. Output a summary of the document based on the input document content.

[0063] Optionally, in some possible implementations, the method for obtaining all text paragraphs and contextual text content with a similarity greater than a preset threshold in step S330 includes:

[0064] S331. Obtain the set of heading levels where the main text paragraphs are located in the main text paragraph set;

[0065] S332. Determine the hierarchical inclusion relationship C in the title level set, and obtain the next level T above the level of the main text paragraph;

[0066] S333. Based on the level T above the main text paragraph, obtain the text content contained in the two headings above and below the heading containing the main text paragraph in level T.

[0067] Optionally, in some possible implementations, the method for outputting a summary of the document based on the input document content in step S350 includes:

[0068] The document content is input into a large language model, and a summary of the document is generated based on the large language model.

[0069] The present invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of the method described above.

[0070] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of the method described above.

[0071] It should be understood that in the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples" refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this description, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate different embodiments or examples described in this specification, as well as some features of different embodiments or examples.

[0072] Of course, those skilled in the art can make various corresponding changes and modifications based on the present invention without departing from its spirit and essence, but such changes and modifications should all fall within the protection scope of the claims of the present invention.

Claims

1. A hierarchical search method, characterized by, Includes the following steps: S100. Obtain the hierarchical paragraph parsing results of the input document; S200: Retrieve the title paragraphs in the parsed results according to the user's query statement, generate a set of title paragraphs according to the search results, determine whether the set of title paragraphs is empty, if not, output the semantically complete paragraphs, if yes, proceed to step S300. S300. Based on the user's query, retrieve the main text paragraphs in the parsed results and output semantically complete paragraphs or summaries based on the retrieval results.

2. The hierarchical search method of claim 1, wherein, In step S100, the method for obtaining the hierarchical paragraph parsing results of documents with different formats includes: S110, Read the input document; S120. Parse the document into tree-structured text fragments according to paragraphs, and divide the paragraphs into heading paragraphs and body paragraphs; S130. Using the Embedding model, the parsed tree-structured text fragments are converted into vectors and stored in a vector database.

3. The hierarchical search method of claim 1, wherein, In step S200, the method for retrieving the title paragraphs from the parsed results based on the user's query statement includes: S210. Convert the user query statement into a Q-vector using the Embedding model; S220. Calculate the similarity between the Q vector and all title paragraph vectors in the vector database, and determine whether the similarity is greater than the preset threshold. If so, save the title paragraph. S230. Create a title paragraph set and put all the saved title paragraphs into the title paragraph set.

4. The hierarchical search method of claim 1, wherein, In step S300, the method for retrieving text paragraphs from the parsed results based on the user query includes: S310. Establish a text set, calculate the similarity between the Q vector and all text paragraph vectors in the vector database, and put all text paragraph vectors with similarity greater than a preset threshold into the text set. S320. Determine whether the text set is empty. If not, proceed to step S330. If yes, proceed to step S350. S330: Obtain all main text paragraphs and context text with a similarity greater than a preset threshold; S340. Based on the obtained main text paragraph and context text, output a semantically complete paragraph; S350. Output a summary of the document based on the input document content.

5. The hierarchical search method of claim 1, wherein, In step S330, the method for obtaining all main text paragraphs and context text content with a similarity greater than a preset threshold includes: S331. Obtain the set of heading levels where the main text paragraphs are located in the main text paragraph set; S332. Determine the hierarchical inclusion relationship C in the title level set, and obtain the next level T above the level of the main text paragraph; S333. Based on the level T above the main text paragraph, obtain the text content contained in the two headings above and below the heading containing the main text paragraph in level T.

6. The hierarchical search method of claim 1, wherein, In step S350, the method for outputting a summary of the document based on the input document content includes: The document content is input into a large language model, and a summary of the document is generated based on the large language model.

7. A computer apparatus comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program, when executed by the processor, causes the processor to perform the method of any one of claims 1 to 6. The processor executes the computer program to implement the steps of the method according to claims 1-6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in claims 1-6.

Images