Document question and answer method, system and storage medium

Abstract

The present application discloses a document question-answering method, system and storage medium, the method comprising: dividing a document into blocks, obtaining a plurality of language blocks, and constructing a tree index between each language block; determining the relevance of the language block with the question, obtaining the relevance ranking position of each language block, defining a preset number of language blocks at the head of the relevance ranking as the first language block, and locating the first language block according to the question; determining the level to which the first language block belongs according to the tree index; determining the optional recall content extension type of the question according to the level to which the first language block belongs; determining the search scope of the second language block according to the optional recall content extension type; matching the question with the language blocks within the search scope to obtain the second language block; summarizing the first language block and the second language block to obtain a recall enhanced knowledge base about the question, and answering the question according to the recall enhanced knowledge base. The present application can improve the comprehensiveness and accuracy of the answer based on the document content.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a document question-answering method, system and storage medium. Background Art

[0002] As an important branch of natural language processing, document question answering technology aims to extract accurate and complete information from documents to answer users' questions. However, existing document question answering methods have many problems when dealing with complex and diverse documents.

[0003] Although the fixed-size chunking method of existing document question-answering methods is simple and intuitive, there are often problems of semantic incompleteness and information loss between chunks, which makes it difficult for document question-answering methods to accurately understand and answer questions related to the boundaries of these semantic chunks. It is also difficult to include the structural information of the entire document, resulting in the segmented semantic chunks being unable to accurately reflect the true intent of the document.

[0004] Existing document question answering methods use a fixed context window step recall strategy for content recall chunks, that is, to retrieve relevant information within a preset text range. The fixed preset text retrieval range is often difficult to adapt to the complexity of the text, ignoring the continuity of sentence semantics, resulting in the omission of key information when processing long sentences, complex contexts, or when deep understanding of the context is required, affecting the comprehensiveness and accuracy of the question answering.

[0005] In summary, existing document question answering methods have many shortcomings when dealing with complex and diverse documents, and further research and improvement are needed to improve the accuracy and comprehensiveness of question answering. Summary of the invention

[0006] In order to address the deficiencies of the prior art, the purpose of this application is to provide a document question-and-answer method and system to improve the efficiency of document information processing and enhance the comprehensiveness and accuracy of answers based on document content.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0008] In a first aspect, the present application provides a document question-answering method, wherein the document question-answering method answers questions based on documents, and the method comprises the following steps:

[0009] Divide the document into blocks to obtain a plurality of language blocks, and construct a tree index between the language blocks, wherein the tree index includes a hierarchical relationship between the language blocks;

[0010] Determine the relevance of the language blocks with the question, obtain the relevance ranking position of each language block, define a preset number of language blocks at the head of the relevance ranking as the first language block, and locate the first language block according to the question;

[0011] Determining, according to the tree index, the level to which the first language block belongs;

[0012] Determining, according to the level to which the first language block belongs, an optional recall content extension type of the question;

[0013] Determining a search scope of the second language block according to the optional recalled content extension type;

[0014] Matching the question with the language blocks within the search range to obtain the second language block;

[0015] The first language block and the second language block are aggregated to obtain a recall-enhanced knowledge base about the question, and the question is answered according to the recall-enhanced knowledge base.

[0016] Furthermore, the optional recall content extension types include:

[0017] A first extension type, under which the search range includes a first type of language block, and in the tree index, a node corresponding to the first type of language block is a ancestor node of a node corresponding to the first language block.

[0018] Furthermore, the optional recall content extension types include:

[0019] A second extended type, under which the search range includes a second type of language block, and in the tree index, a node corresponding to the second type of language block is a sibling node of a node corresponding to the first language block.

[0020] Furthermore, the optional recall content extension types include:

[0021] A third extended type, under which the search range includes a third type of language block, and in the tree index, a node corresponding to the third type of language block is a descendant node of a node corresponding to the first language block.

[0022] Furthermore, the method further comprises:

[0023] Based on the tree index, text segmentation is performed on each language block to obtain segmented sentences, and any segmented sentence is accompanied by position information of the segmented sentence in the tree index.

[0024] Furthermore, the method further comprises:

[0025] The segmentation statement is vectorized and expressed using an embedding model, and the vectorized expression of the segmentation statement is stored in a vector database according to the tree index.

[0026] Furthermore, the method further comprises:

[0027] The question is expressed in vector form; cosine similarity is calculated between the vector expression of the question and the vectors in the vector database to obtain the cosine similarity of each segmented sentence; the cosine similarities of the segmented sentences are sorted, and the language blocks to which a number of segmented sentences at the head of the cosine similarity sorting belong are determined in combination with the tree index, and a preset number of the first language blocks are output.

[0028] Furthermore, the method further comprises:

[0029] According to the search range and in combination with the tree index, a matching range of vectors in the vector database is obtained;

[0030] A vector whose cosine similarity with the vector expression of the question is higher than the preset threshold is selected within the matching range, and the second language block is determined based on the vector and the tree index.

[0031] In a second aspect, the present application also provides a document question and answer system, which adopts the above-mentioned document question and answer method.

[0032] In a third aspect, the present application further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned document question and answer method are executed.

[0033] In summary, the document question-answering method provided by the present application divides the target document into multiple language blocks, obtains a tree index with a language block hierarchical relationship, obtains a first language block by judging the relevance of the language block to the question content, and determines the level to which the first language block belongs based on the tree index, determines the optional recall content expansion type of the question, thereby obtaining a second language block related to the first language block, and finally, based on the first language block and the second language block, constructs a recall enhanced knowledge base about the question content, answers the question based on the recall enhanced knowledge base, enhances the understanding of the language block context, improves the accuracy of the answer, and avoids interference of irrelevant information on the answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flowchart of a document question-and-answer method provided by an embodiment of the present application;

[0035] Figure 2 It is a structural diagram of a tree index in a document question-and-answer method provided by an embodiment of the present application;

[0036] Figure 3 This is a flowchart of the steps of determining the first language block in the document question-and-answer method provided by an embodiment of the present application;

[0037] Figure 4 This is a schematic diagram of the types of extended types of recalled content in a document question-and-answer method provided by an embodiment of the present application;

[0038] Figure 5 It is a schematic diagram of the distribution of language chunks of any document in the document question-answering method provided by an embodiment of the present application;

[0039] Figure 6 It is a flowchart of the steps of determining the second language block in the document question and answer method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the specific implementation of the present application will be clearly and completely described below in conjunction with the drawings in the implementation of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention.

[0041] First, as Figure 1 As shown, the present application provides a document question-answering method, comprising the following steps:

[0042] Step S1, dividing the document into blocks to obtain a plurality of language blocks, and constructing a tree index between the language blocks, wherein the tree index includes a hierarchical relationship between the language blocks;

[0043] Step S2, determining the relevance of the language block and the question, obtaining the relevance ranking position of each language block, defining a preset number of language blocks at the head of the relevance ranking as the first language block, and locating the first language block according to the question;

[0044] Step S3, determining the level to which each first language block belongs according to the tree index;

[0045] Step S4, determining the optional recall content extension type of the question according to the level to which the first language block belongs;

[0046] Step S5, determining the search scope of the second language block according to the optional recall content extension type;

[0047] Step S6, matching the question with the language blocks within the search range to obtain a second language block;

[0048] Step S7: Summarize the first language block and the second language block to obtain a recall-enhanced knowledge base about the question, and answer the question according to the recall-enhanced knowledge base.

[0049] Specifically, in step S1, the target document can be processed into blocks based on the chapter structure of the target document, and each block is a complete or relatively independent information unit in the document. By subdividing the content of the target document into multiple blocks, the processing speed and understanding of the target document information are improved, and the accuracy and efficiency of information retrieval are also improved, laying a data foundation for the question content matching and information integration in the subsequent document question-and-answer method. After completing the document block processing, a tree index between each block can be constructed based on the chapter structure or title structure of the target document. The tree index contains the hierarchical relationship between each block. Each block has a position in the tree index that conforms to the logical framework of the target document, which is convenient for computer understanding and retrieval. The tree index clearly reflects the hierarchical position information of the block in the target document and the logical relationship between each block. According to the tree index, the position of each block in the document can be quickly determined, which is convenient for quickly locating the block.

[0050] like Figure 2 As shown in FIG. , the tree-like index relationship between language blocks is shown as an example. Figure 2 As shown, the language block corresponding to the root node (for the sake of convenience, "the language block corresponding to the node" is referred to as the node below) represents the top level of the entire document or document collection (such as the article title, abstract, etc., which are located at the top level of the document), and nodes A, B, and C are direct child nodes of the root node. They are first-level language blocks (analogous to the first-level directory of the document, the chapter forewords, chapter summaries, etc. under the first-level directory), and are sibling language blocks; nodes A1, A2, B1, B2, C1, and C2 are second-level language blocks (analogous to the contents under the second-level directory of the document), and they are respectively located under three different parent nodes A, B, and C; nodes A11, A12, A21, A22, B11, B12, B21, B22, C11, C12, C21, and C22 are third-level language blocks, and they are respectively located under the second-level language blocks.

[0051] It should be noted that in the embodiment of the present application, after the document is divided into language blocks, the contents of the generated language blocks do not overlap. Taking the language block corresponding to node A as an example, the content of the language block corresponding to node A does not include the content of the language blocks corresponding to nodes A1 and A2. Similarly, the content of the language block corresponding to node A1 does not include the content of the language blocks corresponding to nodes A11 and A12. In general, according to Figure 2In the tree index shown, the language blocks of the document are arranged in the following order: the root node corresponds to the language block; A language block, A1 language block, A11 language block, A12 language block, A2 language block, A21 language block, A22 language block, B language block, B1 language block, B11 language block, B12 language block, B2 language block, B21 language block, B22 language block; C language block, C1 language block, C11 language block, C12 language block, C2 language block, C21 language block, C22 language block.

[0052] In step S2, after the target document is subdivided into multiple language blocks, the language blocks are relevance-determined based on the question content, and the language blocks are sorted from high to low according to the relevance. A preset number of language blocks at the head of the relevance sorting are defined as the first language block, so as to locate the first language block according to the question content. In the embodiment of the present application, the preset number is set to 5, and the language blocks ranked in the top five in the relevance sorting are determined as the first language block.

[0053] In step S3, the level to which the first language block belongs is determined based on the tree index. Specifically, in the document question-answering method of the embodiment of the present application, the specific position of the first language block in the chapter structure of the target document is quickly identified based on the tree index between each language block. Determining the hierarchical position of the first language block provides a basis for the subsequent selection of the recall content extension type.

[0054] In step S4, the optional recall content extension type of the question is determined according to the level to which the first language block belongs. Specifically, in the document question-answering method of the embodiment of the present application, based on the hierarchical relationship of the tree index, the hierarchical position of the first language block determined in step S3 is used to identify the predecessor, sibling, and descendant language blocks of the first language block. The predecessor, sibling, and descendant language blocks of the first language block can be used as candidate language blocks for the recall content extension type. It should be noted that, for example, when Figure 2 When A11 in is the first chunk, the first chunk has no descendant chunks; when Figure 2 When A in is used as the first block, there is no predecessor block for the first block.

[0055] The predecessor blocks, sibling blocks, and descendant blocks described in the present application are based on the node positions of the blocks in the tree index. If node X is the predecessor node of node Y, the block corresponding to node X is called the predecessor block of the block corresponding to node Y. Similarly, if node X is the sibling node of node Y, the block corresponding to node X is called the sibling block of the block corresponding to node Y. If node X is the descendant node of node Y, the block corresponding to node X is called the descendant block of the block corresponding to node Y.

[0056] In step S5, the search scope of the second language block is determined according to the optional recall content extension type. Specifically, in the document question-answering method of the embodiment of the present application, based on the optional recall content extension type selected in step S4, it is determined whether the first language block has one or more of the following types of language blocks: ancestor language block, sibling language block, descendant language block, so as to define the search scope of the second language block.

[0057] In step S6, the question is matched with the language blocks in the search range to obtain the second language block. Specifically, in the document question-answering method of the embodiment of the present application, based on the second language block search range determined in step S5, the question content is matched with the language blocks in the search range to accurately locate the second language block.

[0058] In step S7, after locating the first language block and matching the second language block, a recall-enhanced knowledge base is constructed by summarizing the first language block and the second language block. The recall-enhanced knowledge base may include the first language block that is most relevant to the question and the second language block whose relevance to the question is higher than a preset threshold, forming a comprehensive and detailed information set. Answering questions based on the content of the recall-enhanced knowledge base realizes efficient and accurate extraction and integration of relevant information from massive documents, improves the efficiency and accuracy of document question-answering, and improves the comprehensiveness and detail of the answers.

[0059] According to the above description, in the document question and answer method provided in the embodiment of the present application, the first language block and the second language block related to the question are combined to obtain a recall enhanced knowledge base about the question, and the question is answered based on the recall enhanced knowledge base to improve the comprehensiveness and accuracy of the answer.

[0060] As a way to achieve this, Figure 3 As shown, in the embodiment of the present application, step S2 may specifically include the following steps:

[0061] Step S21: Based on the tree index, perform text segmentation on each language block to obtain segmented sentences. Any segmented sentence is accompanied by the position information of the segmented sentence in the tree index. Specifically, in the document question-answering method provided in the embodiment of the present application, based on the tree index containing the hierarchical relationship between each language block, the language block is further segmented and processed according to the symbols in the language block to obtain the segmented sentences. In this embodiment, the above symbols may include line breaks and periods. The above process is not limited to simple text information segmentation, but also includes each segmented sentence being accompanied by the hierarchical position information of the language block in the tree index. This step realizes the subdivision of language block information, ensures the independence and integrity of each segmented sentence, and retains the position information of the segmented sentence in the logical framework of the target document, providing an accurate positioning basis for subsequent information retrieval and information matching.

[0062] Step S22: Use the embedding model to vectorize the segmented sentences. Specifically, in the document question-answering method of the embodiment of the present application, the segmented sentences obtained in step S21 are converted into vectorized expressions through the embedding model. In this embodiment, the vectorized expressions may include continuous digital sequences. In the above process, deep learning technology is used to convert text information into vectors in a mathematical sense. The vectorized expression retains the semantic information of the sentence and the position information in the tree index. The correlation between the question content and the segmented sentences can be measured by comparing the vectors, which provides a basis for the intelligent matching of subsequent sentences with the question content.

[0063] Step S23: The vectorized expression of the segmentation statement is stored in the vector database according to the tree index. Specifically, in the document question-answering method of the embodiment of the present application, after the vectorized expression of the segmentation statement is completed, the vectorized expression of the segmentation statement is stored in the vector database in order according to the position information of the segmentation statement in the tree index. The storage process of the above-mentioned vector database ensures the structuring of the stored information, and the subsequent intelligent matching of the document content and the question content and the information retrieval of the document content can be performed based on the hierarchical relationship of the tree index, thereby improving the efficiency and accuracy of information retrieval.

[0064] Step S24: vectorize the question. Specifically, in the document question-answering method of the embodiment of the present application, the process of vectorizing the question content is similar to the process of vectorizing the segmentation statement in step S22, which uses deep learning technology to convert the semantic information of the question into a vector form in a mathematical sense, so that the question exists in the same form as the vectorized expression of the segmentation statement, providing a basis for the intelligent matching of subsequent statements with the question content.

[0065] Step S25: Calculate the cosine similarity between the vector expression of the question and the vector in the vector database to obtain the cosine similarity of each segmentation sentence. Specifically, in the document question-answering method of the embodiment of the present application, the vector expression of the question and the vector in the vector database may include two elements: size and direction. Calculate the cosine similarity between the vector expression of the question and the vector in the vector database. In this embodiment, the cosine similarity is calculated using the following formula: Z=(X·Y) / (||X||·||Y||). In the formula, Z represents the cosine similarity, X represents the vector expression of the question, and Y represents the vector in the vector database.

[0066] Step S26: Sort the cosine similarities of the segmented sentences. For example, sort the results of the cosine similarity calculation between the vector expression of the question and the vectors in the vector database from large to small, and combine the tree index to determine the language blocks to which a number of segmented sentences with higher cosine similarities belong, and output a preset number of language blocks (5 in the embodiment of the present application) with the top ranking as the first language block.

[0067] like Figure 4 As shown, in step S4, the optional recalled content extension type may include a first extension type, a second extension type, and a third extension type. It should be noted that in the embodiment of the present application, the selected recalled content extension type may include one or more of the above types.

[0068] The first extension type, under the first extension type, the search range includes the first type of language block, and in the tree index, the node corresponding to the first type of language block is the ancestor node of the node corresponding to the first language block.

[0069] The second extension type, under the second extension type, the search range includes the second type of language block, and in the tree index, the node corresponding to the second type of language block is a sibling node of the node corresponding to the first language block.

[0070] The third extension type: under the third extension type, the search range includes the third type of language block. In the tree index, the node corresponding to the third type of language block is a descendant node of the node corresponding to the first language block.

[0071] Specifically, the following definitions are provided. In this application, the direct previous node of any node is called the parent node of the node; the direct next node of any node is called the child node of the node; nodes with the same parent node are called sibling nodes; the parent node of any node, the parent node of the parent node, and so on are all ancestor nodes of the node; the child node of any node, the child node of the child node, and so on are all descendant nodes of the node.

[0072] by Figure 5 For example, if the corresponding node of the first language block is A12, the search range of the first extended type of the first language block includes nodes A1 and node A, the search range of the second extended type of the first language block includes nodes A11, A13 and A14, and the search range of the third extended type of the first language block includes nodes A121 and A122.

[0073] The search scope of the above three optional recall content extension types covers the predecessors, siblings, and descendants of the first language block in the tree index. By expanding the predecessors, siblings, and descendants of the first language block, the contextual content of the first language block can be deeply understood, the preceding content of the first language block can be elaborated, information different from the perspective of the first language block can be provided, a multi-dimensional answer system can be established, and the subsequent content of the first language block can be explored to supplement the subsequent evolution of the question and answer or the detailed expansion of related fields.

[0074] As a way to achieve this, Figure 6 As shown, in the embodiment of the present application, step S6 may specifically include the following steps:

[0075] Step S61: According to the search range and in combination with the tree index, the matching range of the vector in the vector database is obtained.

[0076] Specifically, in the document question-answering method of the embodiment of the present application, according to the second language block search range determined in step S5, the language block hierarchical relationship of the tree index and the position information attached to the segmentation sentence are combined to obtain the matching range of the vectors of the segmentation sentence in the vector database. By combining the language block hierarchical relationship of the tree index and the position information attached to the segmentation sentence, the vectors in the vector database that meet the search range are intelligently located, thereby improving the efficiency and accuracy of information retrieval.

[0077] Step S62: Screening vectors within the matching range whose cosine similarity with the vector expression of the question is higher than a preset threshold, and determining the second language block based on the vector and in combination with the tree index.

[0078] Specifically, in the document question-and-answer method of the embodiment of the present application, based on the vector matching range in the vector database obtained in step S61, the vector expression of the question is subjected to cosine similarity calculation with the vectors in the matching range in the vector database, and the vectors whose cosine similarity calculation results with the vector expression of the question and the vectors in the matching range in the vector database are screened out are higher than a preset threshold. Based on the vectors obtained by the above screening, combined with the position information retained by the vectorized expression of the segmented sentence and the language block hierarchical relationship of the tree index, the second language block to which the above vector belongs is accurately determined. It should be noted that in the embodiment of the present application, the second language block may include one language block or multiple language blocks.

[0079] According to the above description, the present application provides a document question-answering method, which divides the target document into multiple language blocks, obtains a tree index with a language block hierarchical relationship, obtains a first language block by judging the relevance of the language block to the question content, and determines the level to which the first language block belongs based on the tree index, determines the optional recall content expansion type of the question, and thus obtains a second language block related to the first language block. Finally, based on the first language block and the second language block, a recall-enhanced knowledge base about the question content is constructed, and the question is answered based on the recall-enhanced knowledge base, thereby enhancing the understanding of the language block context, improving the accuracy of the answer, and avoiding interference of irrelevant information on the answer.

[0080] On the second aspect, the present application also provides a document question and answer system, which can be applied to any of the document question and answer methods provided in the present application above, to understand the content of the question and give accurate answers to the question based on the target document.

[0081] On the third aspect, the present application also provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, it runs the steps in the above-mentioned document question and answer method to understand the content of the question and make accurate answers to the question based on the target document.

[0082] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims. It should be understood that for those of ordinary skill in the art, improvements or changes can be made according to the above description, and all these improvements and changes shall fall within the protection scope of the claims attached to the present application.

Claims

1. A document question-answering method, wherein the document question-answering method answers questions based on documents, and is characterized in that: The method comprises the following steps: Divide the document into chunks to obtain a plurality of chunks, and construct a tree index between the chunks, wherein the tree index includes a hierarchical relationship between the chunks, and after dividing the document into chunks, the contents of the chunks generated do not overlap; Determine the relevance of the language blocks with the question, obtain the relevance ranking position of each language block, define a preset number of language blocks at the head of the relevance ranking as the first language block, and locate the first language block according to the question; Determining, according to the tree index, a level to which each of the first language blocks belongs; Determining, according to the level to which the first language block belongs, an optional recall content extension type of the question; Determining a search scope of the second language block according to the optional recalled content extension type; Matching the question with the language blocks within the search range to obtain the second language block; Aggregating the first language block and the second language block to obtain a recall-enhanced knowledge base about the question, and answering the question according to the recall-enhanced knowledge base; Perform text segmentation on each language block to obtain segmented sentences, and use the embedding model to vectorize the segmented sentences. Storing the vectorized expression of the segmentation statement in a vector database according to the tree index; According to the search range and in combination with the tree index, a matching range of vectors in the vector database is obtained; Screening a vector within the matching range whose cosine similarity with the vector expression of the question is higher than a preset threshold, and determining the second language block based on the vector and the tree index; The optional recall content extension types include: A first extension type, under which the search range includes a first type of language block, and in the tree index, a node corresponding to the first type of language block is a ancestor node of a node corresponding to the first language block.

2. The document question-answering method according to claim 1, characterized in that: The optional recall content extension types include: A second extended type, under which the search range includes a second type of language block, and in the tree index, a node corresponding to the second type of language block is a sibling node of a node corresponding to the first language block.

3. The document question-answering method according to claim 1, characterized in that: The optional recall content extension types include: A third extended type, under which the search range includes a third type of language block, and in the tree index, a node corresponding to the third type of language block is a descendant node of a node corresponding to the first language block.

4. The document question-answering method according to claim 1, characterized in that: The method further comprises: Based on the tree index, any of the segmentation statements is accompanied by the position information of the segmentation statement in the tree index.

5. The document question-answering method according to claim 1, characterized in that: The method further comprises: Expressing the question in vector form; Calculate the cosine similarity between the vector expression of the question and the vectors in the vector database to obtain the cosine similarity of each segmented sentence; The cosine similarities of the segmented sentences are sorted, and the language blocks to which a number of segmented sentences located at the head of the cosine similarity sorting belong are determined in combination with the tree index, and a preset number of the first language blocks are output.

6. A document question-answering system, characterized in that: The document question and answer system adopts the document question and answer method as described in any one of claims 1 to 5.

7. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are executed.

Images