Water document multi-modal retrieval enhanced question and answer method based on hierarchical index and modal perception sorting

Abstract

The application discloses a water conservancy document multimodal retrieval enhanced question and answer method based on layered index and modal perception sorting, relates to the technical field of water conservancy information processing and retrieval enhancement generation, and comprises the following steps: S1, a water conservancy engineering PDF document is acquired; page analysis, optical character recognition, table recognition and image extraction processing are performed on the water conservancy engineering PDF document; and a structured analysis result containing chapter titles, text paragraphs, table names, table headers, row data, original images and figure caption information is obtained; S2, according to the structured analysis result, a text parent-child retrieval unit, a table row-level retrieval unit and an image abstract retrieval unit are respectively constructed, and an original object mapping relationship between each retrieval unit and original evidence is established. The application solves the problems of insufficient multimodal evidence organization capability and the fact that retrieval sorting cannot adapt to query modal requirements in the existing water conservancy document retrieval enhanced question and answer process.

Description

Technical Field

[0001] This invention relates to the field of water conservancy information processing and retrieval enhancement technology, and in particular to a multimodal retrieval enhancement question-answering method for water conservancy documents based on hierarchical indexing and modality-aware sorting. Background Technology

[0002] With the development of digital construction in water conservancy projects, a large number of water conservancy project specifications, design documents, operation and maintenance data, etc., are stored and managed in PDF format. These documents typically contain various information formats such as text, tables, engineering images, and monitoring curves. Traditional keyword-based retrieval methods are insufficient to meet the needs of acquiring complex engineering knowledge. In recent years, with the development of natural language processing, vector retrieval, and large language model technologies, retrieval enhancement generation technology has been gradually applied to the field of intelligent question answering for professional documents. Existing technologies typically parse and segment document content, construct semantic retrieval indexes using text embedding models, and combine keyword retrieval methods to achieve multi-path recall. The retrieval results are then used as context input to a large language model, thereby achieving intelligent question answering for professional documents and improving the efficiency of engineering data utilization.

[0003] Existing enhanced question-answering methods for water conservancy document retrieval still have certain shortcomings when processing complex engineering documents containing multimodal information such as text, tables, and images. On the one hand, traditional document segmentation methods typically process text content, making it difficult to maintain the contextual relationships between chapters, the relationships between table fields, and the semantic information of images, resulting in some important engineering evidence being difficult to retrieve accurately. On the other hand, existing multi-path retrieval fusion methods usually perform unified fusion based on retrieval ranking results, failing to fully consider the different needs of users for different modal information in their queries, thus preventing the priority return of table or image evidence that is more relevant to the current query. Therefore, there is a need for an enhanced question-answering method that can uniformly organize, accurately recall, and dynamically rank multiple types of evidence, tailored to the characteristics of multimodal documents in water conservancy engineering. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides an enhanced question-answering method for multimodal retrieval of water conservancy documents based on hierarchical indexing and modality-aware sorting to solve the problems of insufficient multimodal evidence organization and inability of retrieval sorting to adapt to query modality requirements in water conservancy documents.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a multimodal retrieval and question-answering method for water conservancy documents based on hierarchical indexing and modality-aware sorting. The method includes: S1, obtaining a water conservancy project PDF document, performing layout parsing, optical character recognition, table recognition and image extraction processing on the water conservancy project PDF document, and obtaining a structured parsing result containing chapter titles, main text paragraphs, table names, table headers, row data, original images and figure caption information; S2. Based on the structured parsing results, construct text parent-child retrieval units, table row-level retrieval units, and image summary retrieval units respectively, and establish original object mapping relationships between each retrieval unit and the original evidence; S3. Generate unified search text according to the modal type corresponding to different search units, associate evidence unit identifier, modal tag and original object mapping information with the unified search text, and write them into the semantic search index and BM25 search index respectively; S4. Receive user query, perform parallel candidate evidence recall based on the semantic retrieval index and BM25 retrieval index, merge and deduplicate the two recall results according to the evidence unit identifier, and obtain a candidate evidence set. S5. The user query is segmented and keywords are extracted. The obtained query term set is matched with the pre-constructed text modality domain feature term set, table modality domain feature term set and image modality domain feature term set respectively. The matching score of each modality is calculated and the matching score of each modality is normalized to obtain the corresponding modality confidence weight. S6. Calculate the standard reciprocal fusion ranking score based on the ranking of candidate evidence in the semantic retrieval ranking list and the BM25 retrieval ranking list. Using uniform modal weight as a benchmark, calculate the modal calibration factor based on the modal confidence weight. Combine the modal calibration factor corresponding to the modality to which the candidate evidence belongs with the standard reciprocal fusion ranking score to obtain the modality-aware reciprocal fusion ranking score. Select the top K retrieval units based on the modality-aware reciprocal fusion ranking score. S7. Based on the original object mapping relationship corresponding to the first K retrieval units, backtrack the corresponding text parent block, complete original table or original image respectively, organize the backtracked original evidence into a multimodal context, and input the user query and the multimodal context into the large language model to generate water conservancy document question and answer results.

[0007] As a preferred embodiment of the multimodal retrieval and question-answering method for water conservancy documents based on hierarchical indexing and modality-aware sorting described in this invention, in step S1, the layout of the water conservancy engineering PDF document is parsed, and the main text content extraction, optical character recognition, table structure recognition, original image extraction, and figure caption extraction are performed according to the parsed layout area to form the structured parsing result, wherein the structured parsing result includes at least chapter titles, main text paragraphs, table names, table headers, row data, original images, and figure captions.

[0008] As a preferred embodiment of the enhanced question-answering method for multimodal retrieval of water conservancy documents based on hierarchical indexing and modality-aware sorting as described in this invention, the construction process of the text parent-child retrieval unit includes: Based on the chapter title numbers and paragraph boundaries in the structured parsing results, a text parent block is constructed to store chapter context information; Within each parent text block, the text is segmented according to sentence integrity to form sub-text blocks for retrieval and matching. Establish a parent-child mapping relationship between text sub-block identifiers and text parent block identifiers. In the retrieval stage, fine-grained positioning is performed through text sub-blocks, and the corresponding text parent block is traced back according to the parent-child mapping relationship.

[0009] As a preferred embodiment of the multimodal retrieval and enhanced question-answering method for water conservancy documents based on hierarchical indexing and modality-aware sorting described in this invention, the construction process of the table row-level retrieval unit includes: Extract table names, complete table headers, and row data from the structured parsing results; Each row in the table is used as a basic retrieval unit, and the table name, complete table header, and current row data are concatenated according to a preset template; Record the table object identifier and row number corresponding to each table row-level retrieval unit, establish the mapping relationship between the table row-level retrieval unit and the complete original table, so that the retrieval stage can trace back to the complete original table based on the table object identifier.

[0010] As a preferred embodiment of the multimodal retrieval and enhanced question-answering method for water conservancy documents based on hierarchical indexing and modality-aware ranking described in this invention, the construction process of the image summarization retrieval unit includes: Extract the original image and corresponding captions from the structured analysis results; The original image and captions are input into a multimodal large language model to generate an image summary; The image summary includes at least one of the following information: image type, main object, key annotations, structural relationships, and the meaning of coordinate axes or curves; Image retrieval text is composed of image summaries and captions, and the mapping relationship between image object identifiers and original images is recorded.

[0011] As a preferred embodiment of the multimodal retrieval and enhanced question-answering method for water conservancy documents based on hierarchical indexing and modality-aware sorting described in this invention, the unified retrieval text generation process includes: For text modalities, the main text content of the text sub-blocks is used as the unified search text; For table-based modalities, the table name, complete table header, and current row data are concatenated according to a preset template to form a unified search text; For image modalities, image summaries and captions are concatenated according to a preset template to form a unified search text; The unified search text for the three modalities is associated with the corresponding evidence unit identifier, modal tag, and original object mapping information.

[0012] As a preferred embodiment of the multimodal retrieval and enhanced question-answering method for water conservancy documents based on hierarchical indexing and modality-aware ranking described in this invention, wherein: in step S4: Semantic retrieval maps user queries and various unified search texts into dense vectors through an embedding model, and sorts them according to cosine similarity to form a semantic retrieval ranking list; BM25 search calculates term matching scores based on the term frequency, inverse document frequency, and text length of the query term in the unified search text, and then forms a BM25 search ranking list based on the term matching scores.

[0013] As a preferred embodiment of the enhanced question-answering method for multimodal retrieval of water conservancy documents based on hierarchical indexing and modality-aware sorting described in this invention, in step S5, the modality matching score is the sum of the matching weights of each query term in the intersection of the query term set and the corresponding modality domain feature term set; The Softmax function is used to normalize the matching scores of each modality to obtain the modality confidence weights corresponding to the text modality, table modality, and image modality. When a user's query does not match any modality domain feature words, the confidence weights of each modality are set to be equal.

[0014] As a preferred embodiment of the multimodal retrieval and enhanced question-answering method for water conservancy documents based on hierarchical indexing and modality-aware ranking described in this invention, wherein: in step S6: The standard reciprocal fusion ranking score is determined based on the ranking of candidate evidence in the semantic retrieval ranking list and the BM25 retrieval ranking list, as well as the smoothing constant. The modal calibration factor is obtained by superimposing the modal calibration intensity and the deviation of the modal confidence weight relative to the uniform modal weight on a base value of 1; The modality-aware reciprocal fusion ranking score is obtained by multiplying the modality calibration factor corresponding to the modality to which the candidate evidence belongs by the standard reciprocal fusion ranking score of the candidate evidence.

[0015] As a preferred embodiment of the multimodal retrieval and question answering method for water conservancy documents based on hierarchical indexing and modality-aware sorting described in this invention, wherein: the semantic retrieval and BM25 retrieval respectively return the top N candidate evidences, where N is 50; The smoothing constant in the standard reciprocal fusion ranking score is set to 60, the temperature parameter of the Softmax function is set to 1, and the modal calibration intensity is set to 0.3. The top 6 retrieval units are selected based on the modality-aware reciprocal fusion ranking score. After completing the backtracking of original evidence and the merging of homologous evidence, the multimodal context is input into the large language model to generate water conservancy document question-and-answer results.

[0016] The beneficial effects of this invention are as follows: By performing multimodal parsing on PDF documents of water conservancy projects and constructing hierarchical retrieval units for text, tables, and images respectively, the structured organization and associated storage of different types of engineering evidence are realized, improving the expressive ability and retrieval foundation of key information in complex water conservancy documents; through the construction of text parent-child retrieval units, table row-level retrieval units, and image summary retrieval units, the text content can retain contextual relationships, table information can maintain field associations, and image content can be converted into searchable semantics, achieving accurate positioning and complete backtracking of multimodal evidence, and improving the effectiveness and interpretability of retrieval results; through unified retrieval text generation and semantic retrieval index construction with BM25 retrieval index, the invention achieves... The combined retrieval of semantic association information and professional keyword information improves the matching ability of complex knowledge in water conservancy professional documents. By matching query terms with feature word sets of different modal domains and calculating modal confidence weights, and adjusting the reciprocal fusion ranking results in conjunction with modal calibration factors, a ranking mechanism that dynamically perceives the need for different modal evidence based on user query content is realized. This allows text, table, or image evidence that better meets the query requirements to participate in question-answering generation first. Evidence backtracking is performed on the retrieval results through the original object mapping relationship, and the complete text parent block, original table, and original image are organized into a multimodal context and input into the large language model. This achieves an effective combination between retrieval evidence and generation model, improving the accuracy and completeness of water conservancy document question-answering results. This solves the problems of insufficient multimodal evidence organization capability and inability of retrieval ranking to adapt to query modal requirements in the existing water conservancy document retrieval enhanced question-answering process. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the modal perception reciprocal fusion ranking and evidence backtracking of the present invention.

[0019] Figure 2 This is a schematic diagram illustrating the construction of the multimodal hierarchical index of the present invention.

[0020] Figure 3 This is a flowchart of the enhanced question-answering method for multimodal retrieval of water conservancy documents based on hierarchical indexing and modality-aware sorting, as presented in this invention. Detailed Implementation

[0021] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0022] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0023] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0024] Reference Figures 1-3 This is one embodiment of the present invention, which provides an enhanced question-answering method for multimodal retrieval of water conservancy documents based on hierarchical indexing and modality-aware ranking, including the following steps: In step S1, a PDF document of a water conservancy project is acquired, and a document parsing tool is used to parse the layout. Based on the parsed layout area, text content extraction, optical character recognition (OCR), table structure recognition, original image extraction, and figure caption extraction are performed to form a structured parsing result. The structured parsing result records at least the chapter titles, text paragraphs, table names, table headers, row data, original images, and figure captions. In this embodiment, MinerU is used to perform the parsing; however, document parsing tools with corresponding layout parsing, OCR, table recognition, and image extraction capabilities can also be used to achieve the same parsing output.

[0025] In step S2, according to Figure 2 The three branches shown construct a multimodal hierarchical index. For text content, a parent text block is constructed based on chapter title numbers and paragraph boundaries. The parent text block is used to store the context within the chapter. Within each parent text block, sub-text blocks are segmented according to sentence integrity. These sub-text blocks are responsible for retrieval and matching tasks. Identifiers are assigned to both the sub-text blocks and the parent text block, and a parent-child mapping is established. In the retrieval phase, relevant segments are first located using the sub-text blocks, and then the parent text blocks are traced back based on the parent-child mapping.

[0026] For the table content, the table name, complete table header, and data for each row are extracted from the structured parsing results. Each row is used as a basic retrieval unit, and the table name, complete table header, and current row data are concatenated according to a preset template. A table object identifier and row number are recorded for each row-level retrieval unit, establishing a mapping between the row-level retrieval unit and the complete original table. During the retrieval phase, the relevant rows are located first, and then the complete original table is traced back based on the table object identifier, thus simultaneously preserving fine-grained parameter location capabilities and the relationships between all table fields.

[0027] For image content, the original image and its caption are extracted from the structured parsing results. The original image and caption are then input into a multimodal large language model to generate a retrieval-oriented image summary. The image summary may include at least one of the following: image type, main object, key annotations, structural relationships, and the meaning of coordinate axes or curves. The image summary and caption are combined to form the image retrieval text, and the mapping between image object identifiers and the original image is recorded. During the retrieval phase, the image summary retrieval unit is located based on the caption or image summary, and then the original image is retrieved based on the image object identifier.

[0028] In step S3, a unified search text is generated according to formula (1). The text modality uses the main text content of the text sub-blocks; the table modality uses the concatenation result of the table name, complete table header, and current row data; and the image modality uses the concatenation result of the image summary and caption. The unified search text is associated with the evidence unit identifier, modality label, and original object mapping, and is converted into dense vector representation and BM25 term representation, respectively, and written into the semantic search index and BM25 search index.

[0029] (1) In the formula, Indicates the first One unit of evidence, Indicates evidence unit Modal type, This represents a modal set consisting of text, tables, and images. Indicates a unified search text; This represents the main text content of a text sub-block; , and These represent the table name, table header, and current row data, respectively. and These represent image summaries and captions, respectively. This expression concatenates the fields according to a preset template. While preserving the relationships between table fields and the original text of figure captions, it supplements visual semantics with image summaries.

[0030] In step S4, a user query is received. The embedding model maps the user query and each unified search text into dense vectors, calculates the cosine semantic similarity according to formula (2), and obtains the top N candidate evidences from high to low scores to form a semantic retrieval ranking list. At the same time, the BM25 term matching score is calculated according to formula (3), and the top N candidate evidences from high to low scores are obtained to form a BM25 retrieval ranking list. The two candidate evidences are merged and deduplicated according to the evidence unit identifier to obtain a candidate evidence set.

[0031] (2) In the formula, This indicates a user query. and These represent the user query and the unified search text, respectively. Dense vectors, Describing the vector norm, Represents user query and evidence unit The semantic similarity is used to rank the results from highest to lowest. The candidate evidence is used to form a semantic retrieval ranking list.

[0032] (3) In the formula, Represents the set of query terms. Indicates terms Inverse document frequency, Indicates terms In unified search text word frequency in Indicates the uniform search text length. This represents the average length of the retrieved text in the corpus. Used to control the impact of word frequency on the score. Used to control the degree of length normalization. The top scores are ranked from highest to lowest according to the BM25 score. One candidate piece of evidence.

[0033] (4) In the formula, Represents the set of retrieval channels. Indicates channel The returned sorted list, Indicates evidence unit In the list The rankings in the middle, This indicates the indicator function, which takes the value 0 if the function is not recalled by the channel. This represents the smoothing constant. The higher the ranking of the evidence in any list, the greater its contribution.

[0034] In step S5, the user query is segmented and keywords are extracted to obtain a query term set. For text, table, and image modalities, the pre-constructed corresponding domain feature term sets are read respectively, and the matching weights of the query terms and the corresponding sets are accumulated according to formula (5) to obtain the matching scores for the three modalities. In this embodiment, the matching weight of each matching feature term is set to 1. The domain feature term set is a pre-constructed domain feature term set, and the present invention does not limit the construction or extraction method of the set.

[0035] (5) In the formula, Represents the set of query terms. Represents a pre-built mode The set of domain feature words, Indicating characteristic words Matching weights This represents the matching score. This embodiment uses an equal-weighting system, where the matching weight for each matched feature word is set to 1. This invention does not limit the specific construction method of the domain feature word set.

[0036] The Softmax function shown in formula (6) is used to normalize the matching scores of each modality to obtain the modality confidence weights, which satisfy the normalization constraints shown in formula (7). When a user's query does not match any modality's domain feature words, the matching scores of the three modalities are all 0, and the confidence weights of the three modalities are equal after normalization. When the query matches a domain feature word of a certain modality, the confidence weight of that modality increases accordingly. When the query matches multiple modality domain feature words at the same time, multiple modalities can obtain higher weights simultaneously.

[0037] (6) In the formula, Indicates user query Corresponding to mode Modal confidence weights, Represents the modal set Any mode participating in the normalization calculation, A temperature parameter that controls the smoothness of the weight distribution. In one embodiment, Take 1.

[0038] (7) In the formula, This represents a modal set consisting of text, tables, and images. Representing modes The modality confidence weights are calculated as follows: When the query fails to match any domain feature word, the matching score for each modality is 0, and after Softmax normalization, the confidence weight for each modality is 0. .

[0039] In step S6, the standard reciprocal fusion ranking score of the candidate evidence is first calculated according to formula (4). To avoid the scaling problem caused by directly adding modal confidence weights as additional scores, uniform modal weights are used. Centered on the modality, the modality calibration factor is calculated according to formula (8). Then, according to formula (9), the modality calibration factor of each candidate piece of evidence is multiplied by its standard reciprocal fusion ranking score to obtain the modality-aware reciprocal fusion ranking score. The processing flow is as follows: Figure 1 As shown.

[0040] (8) In the formula, Indicates user query Modal calibration factor for mode m, In this embodiment, the number of modal types is indicated. It is 3. This indicates the modal calibration intensity. When the query does not show a clear modal tendency... =1; when When the value is 0, the calibration degenerates into a standard reciprocal fusion ranking.

[0041] (9) In the formula, Indicates evidence unit Modal type Indicates query Evidence modality The calibration factor, and the meanings of the other symbols are the same as in formula (4). Candidate evidence is sorted from highest to lowest and the top K retrieval units are selected.

[0042] Modal calibration intensity The confidence weights are between 0 and 1. When the query has no obvious modal bias, the confidence weights for each modality are all equal. Each modality's calibration factor is 1, and the standard reciprocal fusion ranking score of candidate evidence remains unchanged. When a query has a high confidence weight for a particular modality, the calibration factor for that modality is greater than 1, while the calibration factors for other modalities decrease accordingly. For all three modalities, the modality calibration factor remains positive, thus not completely excluding evidence from any modality. When the value is 0, this method degenerates into the standard reciprocal fusion ranking.

[0043] Candidate evidence is sorted from highest to lowest according to the modality-aware reciprocal fusion ranking score, and the top K retrieval units are selected. In step S7, evidence backtracking is performed based on the original object mapping recorded in each retrieval unit: text sub-blocks are backtracked to their corresponding text parent blocks, table row-level retrieval units are backtracked to the complete original table, and image summary retrieval units are backtracked to the original image. After merging the evidence from the same source, the backtracked text parent blocks, complete original tables, or original images are organized into a multimodal context and input into the large language model along with the user query to generate water conservancy document question-and-answer results.

[0044] This embodiment illustrates a parameter setting and verification method. Semantic retrieval and BM25 retrieval each return before... One candidate piece of evidence, Take 50; after ranking by modal perception reciprocal fusion, select the top 50. One retrieval unit, Take 6; Smoothing constant for standard reciprocal merging ranking Set the temperature to 60; Softmax temperature parameter Set to 1; Modal calibration intensity Take 0.3. After completing the tracing of original evidence and merging of evidence from the same source, the previous... Complete evidence is provided to the large language model. The parameters above are example values ​​used in this embodiment and can be adjusted according to the document size and question-answering scenario without changing the step dependencies.

[0045] This embodiment uses 1135 question-and-answer samples constructed from water conservancy engineering specification documents for verification, including 609 text entries, 296 table entries, and 230 images. In the index ablation experiment, text parent-child indexes, table row-level indexes, and image summary indexes are gradually introduced; in the ranking method comparison, the multimodal index and two initial candidate sets remain unchanged, only the standard reciprocal fusion ranking is replaced with modality-aware reciprocal fusion ranking.

[0046] Validation results show that text parent-child indexes can preserve chapter context, table row-level indexes can improve the accuracy of parameter field location, and image summary indexes can supplement the searchable semantics of image content. Based on this, modality-aware reciprocal fusion ranking utilizes modality confidence weights and their calibration factors to adjust the order of candidate evidence according to the modality requirements of the user query. After adopting this method, the context precision and context recall reached 0.8320 and 0.8780, respectively; compared with the baseline using fixed-length blocks and standard reciprocal fusion ranking, these represent improvements of 8.02 and 7.87 percentage points, respectively. These results indicate that hierarchical indexing mainly functions to organize and fully backtrack multimodal evidence, while modality-aware ranking mainly functions to sort candidate evidence on demand; both synergistically improve context precision and context recall.

[0047] In summary, this invention, through multimodal parsing of water conservancy engineering PDF documents and the construction of hierarchical retrieval units for text, tables, and images, achieves structured organization and associated storage of different types of engineering evidence, improving the expressive power and retrieval foundation of key information in complex water conservancy documents. By constructing text parent-child retrieval units, table row-level retrieval units, and image summary retrieval units, it enables text content to retain contextual relationships, table information to maintain field associations, and image content to be converted into searchable semantics, achieving precise location and complete backtracking of multimodal evidence, and improving the effectiveness and interpretability of retrieval results. Furthermore, by unifying retrieval text generation and semantic retrieval index construction with the BM25 retrieval index, it achieves semantic relationship... The combined retrieval of linked information and professional keyword information improves the matching ability of complex knowledge in water conservancy professional documents. By matching query terms with feature word sets of different modal domains and calculating modal confidence weights, and adjusting the reciprocal fusion ranking results in conjunction with modal calibration factors, a ranking mechanism that dynamically perceives the need for different modal evidence based on user query content is realized. This allows text, table, or image evidence that better meets the query requirements to participate in question-answering generation first. Evidence backtracking is performed on the retrieval results through the original object mapping relationship, and the complete text parent block, original table, and original image are organized into a multimodal context and input into the large language model. This achieves an effective combination between retrieval evidence and generation model, improving the accuracy and completeness of water conservancy document question-answering results. It solves the problems of insufficient multimodal evidence organization capability and inability of retrieval ranking to adapt to query modal requirements in the existing water conservancy document retrieval enhanced question-answering process.

[0048] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multimodal retrieval and enhanced question-answering method for water conservancy documents based on hierarchical indexing and modality-aware ranking, characterized in that: include, S1. Obtain the PDF document of the water conservancy project, and perform layout parsing, optical character recognition, table recognition and image extraction processing on the PDF document of the water conservancy project to obtain a structured parsing result containing chapter titles, text paragraphs, table names, table headers, row data, original images and figure caption information; S2. Based on the structured parsing results, construct text parent-child retrieval units, table row-level retrieval units, and image summary retrieval units respectively, and establish original object mapping relationships between each retrieval unit and the original evidence; S3. Generate unified search text according to the modal type corresponding to different search units, associate evidence unit identifier, modal tag and original object mapping information with the unified search text, and write them into the semantic search index and BM25 search index respectively; S4. Receive user query, perform parallel candidate evidence recall based on the semantic retrieval index and BM25 retrieval index, merge and deduplicate the two recall results according to the evidence unit identifier, and obtain a candidate evidence set. S5. The user query is segmented and keywords are extracted. The obtained query term set is matched with the pre-constructed text modality domain feature term set, table modality domain feature term set and image modality domain feature term set respectively. The matching score of each modality is calculated and the matching score of each modality is normalized to obtain the corresponding modality confidence weight. S6. Calculate the standard reciprocal fusion ranking score based on the ranking of candidate evidence in the semantic retrieval ranking list and the BM25 retrieval ranking list. Using uniform modal weight as a benchmark, calculate the modal calibration factor based on the modal confidence weight. Combine the modal calibration factor corresponding to the modality to which the candidate evidence belongs with the standard reciprocal fusion ranking score to obtain the modality-aware reciprocal fusion ranking score. Select the top K retrieval units based on the modality-aware reciprocal fusion ranking score. S7. Based on the original object mapping relationship corresponding to the first K retrieval units, backtrack the corresponding text parent block, complete original table or original image respectively, organize the backtracked original evidence into a multimodal context, and input the user query and the multimodal context into the large language model to generate water conservancy document question and answer results.

2. The enhanced question-answering method for multimodal retrieval of water conservancy documents based on hierarchical indexing and modality-aware ranking as described in claim 1, characterized in that: In step S1, the layout of the water conservancy project PDF document is parsed. Based on the parsed layout area, the main text content extraction, optical character recognition, table structure recognition, original image extraction, and figure caption extraction are performed to form the structured parsing result. The structured parsing result includes at least chapter titles, main text paragraphs, table names, table headers, row data, original images, and figure captions.

3. The enhanced question-answering method for multimodal retrieval of water conservancy documents based on hierarchical indexing and modality-aware ranking as described in claim 2, characterized in that: The construction process of the text parent-child retrieval unit includes: Based on the chapter title numbers and paragraph boundaries in the structured parsing results, a text parent block is constructed to store chapter context information; Within each parent text block, the text is segmented according to sentence integrity to form sub-text blocks for retrieval and matching. Establish a parent-child mapping relationship between text sub-block identifiers and text parent block identifiers. In the retrieval stage, fine-grained positioning is performed through text sub-blocks, and the corresponding text parent block is traced back according to the parent-child mapping relationship.

4. The enhanced question-answering method for multimodal retrieval of water conservancy documents based on hierarchical indexing and modality-aware ranking as described in claim 3, characterized in that: The process of constructing the table row-level retrieval unit includes: Extract table names, complete table headers, and row data from the structured parsing results; Each row in the table is used as a basic retrieval unit, and the table name, complete table header, and current row data are concatenated according to a preset template; Record the table object identifier and row number corresponding to each table row-level retrieval unit, establish the mapping relationship between the table row-level retrieval unit and the complete original table, so that the retrieval stage can trace back to the complete original table based on the table object identifier.

5. The enhanced question-answering method for multimodal retrieval of water conservancy documents based on hierarchical indexing and modality-aware ranking as described in claim 4, characterized in that: The construction process of the image summary retrieval unit includes: Extract the original image and corresponding captions from the structured analysis results; The original image and captions are input into a multimodal large language model to generate an image summary; The image summary includes at least one of the following information: image type, main object, key annotations, structural relationships, and the meaning of coordinate axes or curves; Image retrieval text is composed of image summaries and captions, and the mapping relationship between image object identifiers and original images is recorded.

6. The enhanced question-answering method for multimodal retrieval of water conservancy documents based on hierarchical indexing and modality-aware ranking as described in claim 5, characterized in that: The unified search text generation process includes: For text modalities, the main text content of the text sub-blocks is used as the unified search text; For table-based modalities, the table name, complete table header, and current row data are concatenated according to a preset template to form a unified search text; For image modalities, image summaries and captions are concatenated according to a preset template to form a unified search text; The unified search text for the three modalities is associated with the corresponding evidence unit identifier, modal tag, and original object mapping information.

7. The enhanced question-answering method for multimodal retrieval of water conservancy documents based on hierarchical indexing and modality-aware ranking as described in claim 6, characterized in that: In step S4: Semantic retrieval maps user queries and various unified search texts into dense vectors through an embedding model, and sorts them according to cosine similarity to form a semantic retrieval ranking list; BM25 search calculates term matching scores based on the term frequency, inverse document frequency, and text length of the query term in the unified search text, and then forms a BM25 search ranking list based on the term matching scores.

8. The enhanced question-answering method for multimodal retrieval of water conservancy documents based on hierarchical indexing and modality-aware ranking as described in claim 7, characterized in that: In step S5, the matching score for each modality is the sum of the matching weights of each query term in the intersection of the query term set and the corresponding modality domain feature term set; The Softmax function is used to normalize the matching scores of each modality to obtain the modality confidence weights corresponding to the text modality, table modality, and image modality. When a user's query does not match any modality domain feature words, the confidence weights of each modality are set to be equal.

9. The enhanced question-answering method for multimodal retrieval of water conservancy documents based on hierarchical indexing and modality-aware ranking as described in claim 8, characterized in that: In step S6: The standard reciprocal fusion ranking score is determined based on the ranking of candidate evidence in the semantic retrieval ranking list and the BM25 retrieval ranking list, as well as the smoothing constant. The modal calibration factor is obtained by superimposing the modal calibration intensity and the deviation of the modal confidence weight relative to the uniform modal weight on a base value of 1; The modality-aware reciprocal fusion ranking score is obtained by multiplying the modality calibration factor corresponding to the modality to which the candidate evidence belongs by the standard reciprocal fusion ranking score of the candidate evidence.

10. The enhanced question-answering method for multimodal retrieval of water conservancy documents based on hierarchical indexing and modality-aware ranking as described in claim 9, characterized in that: The semantic retrieval and BM25 retrieval respectively return the top N candidate evidence, where N is 50; The smoothing constant in the standard reciprocal fusion ranking score is set to 60, the temperature parameter of the Softmax function is set to 1, and the modal calibration intensity is set to 0.

3. The top 6 retrieval units are selected based on the modality-aware reciprocal fusion ranking score. After completing the backtracking of original evidence and the merging of homologous evidence, the multimodal context is input into the large language model to generate water conservancy document question-and-answer results.

Images