Method and equipment for enhancing generation of questions and answers in water conservancy literature retrieval

Abstract

The invention relates to the technical field of natural language processing and intelligent information retrieval, and discloses a water conservancy document retrieval enhancement question and answer generation method and equipment. Extracting formula variables, positioning definitions, and constructing a variable formula dependency map; carrying out constraint semantic partitioning on the text based on a graph topological structure, and preventing logic truncation by utilizing a boundary extension strategy; generating an enhanced vector based on graph neighbor aggregation, and constructing a mixed index database containing inverted, vector and graph relationships; in response to user query, executing double-path mixed retrieval, executing recursive completion based on the graph relation index, and forcibly recalling associated nodes with a dependency relationship to generate a final context; and finally, injecting the context into the large language model to generate an answer. According to the method, the problem of retrieval information missing caused by formula and definition separation is effectively solved, a logic closed loop is realized through map enhancement, and the numerical reasoning accuracy of water conservancy professional questions and answers is improved.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and intelligent information retrieval technology, specifically to a method and device for enhancing question-and-answer generation in water conservancy literature retrieval. Background Technology

[0002] With the deepening of water conservancy informatization, the water conservancy industry has accumulated a massive amount of unstructured technical documents, including various engineering design specifications, hydrological dispatch manuals, geological exploration reports, and scientific research papers. These documents are not only numerous but also contain a large number of professional terms, data tables, and complex mathematical formulas, constituting the core knowledge assets of the water conservancy field. In recent years, the rise of Large Language Models (LLM) has brought a new paradigm to knowledge management, especially Retrieval-Augmented Generation (RAG) technology. By injecting external knowledge base retrieval results as context into the model, it effectively alleviates the knowledge cutoff and illusion problems of large models, becoming the mainstream solution for knowledge question answering in current vertical domains.

[0003] However, existing general-purpose RAG technologies face significant technical bottlenecks when processing water conservancy industry documents, primarily due to a mismatch between semantic document segmentation and logical retrieval. Mainstream RAG systems typically employ a sliding window mechanism based on fixed character lengths or natural paragraphs to segment documents, converting long documents into independent text chunks for vectorized storage. This mechanical segmentation method ignores the strong logical dependencies inherent in water conservancy documents. In water conservancy technical documents, calculation formulas and their natural language definitions are often spatially discontinuous. For example, a formula might be located in the current paragraph, while the physical meaning, value range, or empirical coefficient definitions of variables are distributed across subsequent paragraphs or even tables spanning multiple pages.

[0004] In the aforementioned scenarios, traditional segmentation strategies easily separate formulas and their variable definitions into different semantic blocks, leading to semantic fragmentation. When users initiate queries involving numerical calculations or principle deductions, retrieval models based on vector similarity often only recall fragments containing formula source code or only fragments containing variable keywords, failing to recall both simultaneously. Because the context of the input large language model lacks a complete logical chain of formula definitions, the model, lacking specific meanings or numerical evidence for variables, cannot perform correct reasoning and calculations, and can only generate seemingly plausible but incorrect answers based on probability, resulting in severe machine illusion. Furthermore, existing multimodal document parsing technologies mostly focus on layout restoration or simple OCR recognition, lacking in-depth modeling and utilization of the document's internal micro-logical structure (especially the referencing relationships between mathematical symbols and text definitions), making it difficult to meet the stringent requirements of high accuracy and traceability for question-answering systems in the field of water conservancy engineering. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method and device for enhancing the generation of questions and answers for water conservancy literature retrieval. This solves the problem that existing retrieval enhancement generation technologies, when processing water conservancy industry literature, suffer from incomplete retrieval context and low accuracy of numerical reasoning in large models due to the separation of calculation formulas from variable definitions caused by mechanical document segmentation.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The first aspect of the present invention provides a method for enhancing the generation of questions and answers through water conservancy literature retrieval.

[0007] This method first performs structured parsing of multi-source heterogeneous documents. Considering the diverse document formats in the water conservancy industry, the method distinguishes between scanned documents and native electronic documents by recognizing the file header format, and then uses optical character recognition (OCR) engines or document object modeling (DOM) techniques to extract plain text and tables, respectively. Specifically, for the computational formulas abundant in the documents, the method utilizes a Transformer-based encoder-decoder model to perform visual encoding and sequence decoding of the page, converting the image-based mathematical formulas into LaTeX string sequences, and encapsulating them into structured objects based on their positional information, thereby achieving fine-grained semantic extraction from unstructured data.

[0008] Building upon structured parsing, this method constructs a variable-formula dependency graph and optimizes semantic chunking accordingly. The method decomposes formula strings using an abstract syntax tree parser, extracts independent variable symbols, and uses these symbols as anchors to perform definition searches within the text based on rule templates and named entity recognition. This creates directed edges from formula nodes to definition nodes, forming the variable-formula dependency graph. When chunking the document, the method does not simply rely on a fixed character length but instead checks whether the initial segmentation boundary truncates dependency edges or strongly connected components in the graph. If a logical break is detected, the method executes a boundary extension strategy, automatically adjusting the end position of the chunk to ensure that each generated semantic block logically contains a complete formula and its corresponding variable definitions, thus preventing semantic fragmentation at the source.

[0009] This method further constructs a hybrid index library containing graph contextual information. In the vector encoding stage, the method utilizes the variable formula dependency graph to retrieve neighbor nodes of the current semantic block, concatenates the associated variable definition fragments into the original text through semantic fusion, and inputs them into a pre-trained deep language model to generate augmented vectors. This process compresses scattered contextual information into a single vector, improving the semantic density of the vector representation. Subsequently, the method builds an inverted index to record keyword mappings, a vector index to store augmented vectors, and a graph relation index to serialize and store the adjacency list structure of the graph, thereby constructing an underlying data structure that supports multi-dimensional retrieval.

[0010] In the retrieval and question-answering phases, this method performs hybrid retrieval and graph recursive completion. Responding to user queries, the method calculates vector semantic similarity scores and keyword matching scores in parallel, obtaining initial retrieval results through weighted fusion. Subsequently, using the semantic block identifiers in the initial results as keys, the method retrieves outgoing edges with variable definition dependencies in the graph relation index, forcibly adding associated target neighbor nodes to the result set. This step ensures that even if the variable definition paragraph has low relevance to the query terms, it can still be recalled based on logical dependencies, forming a final context containing a complete logical chain. Finally, the method injects this context into a large language model, combines it with thought chain guide words to generate answers, verifies the consistency between citation tags and context formulas, and outputs an answer containing traceable links.

[0011] A second aspect of the present invention provides a computer device.

[0012] The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the above-described method for enhancing the retrieval of documents for the water conservancy industry and generating question answers.

[0013] This invention provides a method and device for enhancing question-and-answer generation in water conservancy literature retrieval. It has the following beneficial effects: 1. This invention constructs a variable formula dependency graph and executes a semantic block segmentation and boundary extension strategy constrained by the graph topology during the document parsing stage. This mechanism enables the division of semantic blocks to no longer rely solely on character length or physical paragraphs, but can automatically identify and retain the complete dependency path of formula variable definitions, effectively preventing key calculation formulas and their variable definitions from being cut into different fragments, thereby ensuring the logical self-containment of data particles in the underlying knowledge base.

[0014] 2. This invention introduces a recursive completion mechanism based on graph relation indexing during the retrieval stage. By forcibly recalling associated nodes that have variable definition dependencies with the initial retrieval results, this invention ensures that the context of the input large language model simultaneously contains the formula prototypes required for calculation and specific variable value information. This logically closed-loop context construction method makes up for the shortcomings of traditional retrieval methods that only recall highly similar texts and miss logically related texts, enhances the performance of large models in water conservancy numerical calculation and logical deduction scenarios, and reduces the generation of illusions.

[0015] 3. This invention adopts a hybrid index architecture that integrates inverted indexes and vector indexes, and combines it with citation verification technology. This architecture not only utilizes keyword matching to ensure the retrieval accuracy of water conservancy industry terminology, but also utilizes vector retrieval to improve the breadth of semantic understanding. Combined with the mapping verification between the generated citation tags and the original documents, it ensures that the final generated natural language answer is not only accurate and reliable, but also has clickable evidence sources, meeting the high requirements of the engineering field for the rigor of technical data. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a detailed flowchart illustrating the search completion and answer generation process of the present invention. Detailed Implementation

[0017] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] See attached document Figure 1 This invention includes a dynamic knowledge question-answering system for the water conservancy industry based on variable formula dependency graph enhancement. The system may include terminal devices, a communication network, and a server. The terminal devices establish a communication connection with the server through the communication network. The terminal devices are configured to receive natural language query requests input by users, send the query requests to the server, and receive and display the answer data returned by the server. The server is configured to execute core processing logic such as document parsing, graph construction, index creation, retrieval calculation, and answer generation. The server can be a single physical server, a server cluster consisting of multiple physical servers, or a distributed system.

[0019] The server includes a processor and memory. The memory stores computer programs, and the processor is configured to execute the computer programs to implement the method steps described in this specification. At the logical functional level, the server includes a document acquisition module, a multimodal parsing module, a graph construction and segmentation module, a graph enhancement indexing module, a hybrid retrieval module, and a generative question answering module. The document acquisition module is configured to access and aggregate multi-source heterogeneous water conservancy industry document data. The multimodal parsing module is configured to perform type recognition, text recognition, table extraction, and formula recognition on documents, transforming unstructured documents into structured data objects. The graph construction and segmentation module is configured to parse variable symbols in formulas, search for variable definitions, construct a variable formula dependency graph, and perform semantic segmentation of text based on graph constraints. The graph enhancement indexing module is configured to generate text vectors that incorporate graph neighbor information and construct vector indexes and graph indexes. The hybrid retrieval module is configured to receive query vectors, calculate similarity scores by combining keyword weights, and perform context completion based on the graph. The generative question answering module is configured to combine the completed context with the user query input into a large language model to generate the final answer.

[0020] See attached document Figure 1 This invention provides a dynamic knowledge question-answering method for the water conservancy industry based on variable formula dependency graph enhancement, which may include the following steps: First, multi-source heterogeneous document collection and initial multi-modal parsing are performed through the document acquisition module and the multimodal parsing module. For multi-source documents in the water conservancy industry, including bulletins, monitoring reports, dispatch instructions, and technical specifications, a unified data access layer is established. The water conservancy industry document set is defined as follows: The set contains A separate document, A positive integer. Document collection. Represented as: ; in, Represents the first element in the set. This document, The value range is 1 to .

[0021] Retrieving document collections Then, for each input document Type determination is performed to determine the technical path to be used in subsequent parsing. A type detection function, `DetectType`, is defined, and its output indicates the document's file format attributes. The type determination process is represented as follows: ; in, This indicates a scanned portable document format. Indicates a natively parseable portable document format. This indicates the format of a text processing document. This represents the Hypertext Markup Language format. For For documents of this type, the optical character recognition engine is invoked for processing; for , and For documents of a certain type, the corresponding Document Object Model parser or text extraction tool is invoked for processing.

[0022] After determining the document type, a multimodal recognition model is invoked based on the mathematical formulas and special symbols contained within the document. Extraction is performed. Multimodal recognition model. Configured as a Transformer-based encoder-decoder network, it is used to convert formula image regions or character sequences in a document into standard LaTeX expressions or MathML representations. For the... document The extracted set of formulas is denoted as The calculation process is expressed as follows: ; in, Includes documents All identified mathematical formulas and their coordinates within the document.

[0023] After independently extracting text, tables, and formulas, the above information is integrated into a unified structured object. This is done for each document. Parsed as containing plain text content Table objects and formula objects Structured representation The process of generating structured representations is as follows: ; in, A sequence of paragraph text in a document. This refers to the table data identified in the document, along with its corresponding titles and annotations. This structured representation... This serves as the foundational input data for subsequent graph construction and semantic segmentation.

[0024] In obtaining structured documents Subsequently, data cleaning and standardization are performed through the graph construction and segmentation module. Structured documents undergo unit unification, noise reduction, and full-width / half-width character conversion to obtain a standard text stream. The cleaning process is represented as follows: ; in, These are pre-defined text cleaning functions used to remove garbled characters, standardize unit formats, and correct recognition errors. Standard text stream. The logical order of the original document was maintained, and a mapping relationship was established between the text content and the page numbers and coordinates of the original document.

[0025] Next, a variable formula dependency graph is constructed using graph construction and block decomposition modules. The dependency graph is defined as follows: ,in Represents the set of nodes in the graph. Represents the set of edges in the graph. Node set. This corresponds to logical content units in the document, including formula block nodes and definition text block nodes. For formula sets... Any formula in Parse its mathematical expression to extract the set of variable symbols it contains. Specifically, abstract syntax tree analysis or regular expression matching techniques are used to separate all independent variable symbols from LaTeX expressions.

[0026] After extracting the variable symbols, in the standard text stream of the entire document Perform a defined search within the specified range. Utilize pre-defined natural language pattern matching templates and named entity recognition technology to locate... The text paragraph containing the natural language definition of each variable. If determining the text block... Formulas and text block Formulas If the variables are defined and described, then a path is established in the spectrum from... point to directed edges Directed edge Belongs to the edge set , is used to characterize the dependency of a formula on the definition of a variable.

[0027] After constructing the dependency graph, semantic chunking under graph constraints is performed. This combines the document's natural paragraph structure with the dependency graph. The topological structure divides the document into a set of semantic blocks. . The set is represented as: ; in, Document The first in A semantic block, This represents the total number of semantic blocks into which the document is segmented. Indicates from 1 to Each semantic block Each node is associated with a unique identifier and maintains an index mapping with the corresponding node in the dependency graph.

[0028] In step S1, which is the stage of acquiring and structuring heterogeneous documents from multiple sources, the system executes specific document processing logic to ensure the integrity and parsability of the data.

[0029] In the document acquisition phase, the document acquisition module accesses data from different sources by configuring various adapter interfaces. For publicly available data such as bulletins and monitoring reports, distributed web crawler technology is used, employing breadth-first traversal based on a pre-defined Uniform Resource Locator (URL) seed list to obtain web page content or file streams via Hypertext Transfer Protocol (HTTP / HTTPS). For internal scheduling instructions and technical specifications, data is directly synchronized from the business system's file storage service or relational database via database connectors (such as JDBC / ODBC). During data access, the system calculates the file's hash value (such as MD5 or SHA-256 checksum) and compares it with the hash values ​​of historical documents. If the hash values ​​match, the document is identified as a duplicate and filtered out, thus ensuring that every document in the document collection is unique. Each one is unique.

[0030] In the document type determination stage, to implement the specific logic of the DetectType function, the system first reads the binary header information (MagicNumber) of the input file to identify the file's true format, preventing parsing errors caused by file extension tampering. For files identified as Portable Document Format (PDF), the system further performs content-level detection: using a PDF parsing library, it reads the first page to a preset number of pages (e.g., the first 5 pages) and calculates the ratio of the number of extractable characters on each page to the total page area, i.e., text density. If the text density is lower than a preset threshold (e.g., 5%), the document is determined to be a scanned PDF (scanPDF), whose content is mainly in bitmap form; if the text density is higher than or equal to the threshold, it is determined to be a natively parsable PDF (parsePDF), whose content contains directly extractable character encoding layers.

[0031] For different document types, the multimodal parsing module calls differentiated processing subroutines. For natively parsable PDF, Word, and HTML documents, the system adopts a parsing strategy based on the Document Object Model (DOM) tree, traversing the document tree nodes to extract text content, font attributes (such as bold and font size), paragraph spacing, and image objects, while preserving the original layout information. For scanned PDFs, the system initiates an Optical Character Recognition (OCR) process: first, the image is preprocessed, including grayscale conversion, binarization, and skew correction; then, a deep learning-based text detection network (such as the Connected Text Proposal Network (CTPN) or the Differentiable Binarization Network (DBNet)) is used to locate text line regions; finally, the text line images are input into a sequence recognition network (such as CRNN or a Transformer-based recognition network) to convert them into computer-encoded text.

[0032] In the specific implementation of formula extraction and recognition, targeting The system employs a three-stage processing strategy for model construction and operation: detection, cropping, and generation. First, an object detection network (e.g., YOLO series or Faster R-CNN) scans the entire document page to locate the bounding boxes containing mathematical formulas, distinguishing between inline formulas and formulas on separate lines. After obtaining image slices of the formula regions, these slices are input into an image-to-sequence (Image-to-Sequence) generation model. This model includes a visual encoder and a text decoder. The visual encoder (e.g., ResNet-101 or ViT) transforms the formula image into a high-dimensional feature map, capturing the geometric features and spatial relationships of mathematical symbols; the text decoder (e.g., TransformerDecoder based on an attention mechanism) receives the feature map and predicts the corresponding LaTeX character sequence through autoregression. During this process, the model is trained to distinguish similar-looking characters (e.g., variables). With Greek letters It accurately restores subscripts, superscripts, fractions, and matrix structures.

[0033] For tables in documents, the system employs computer vision-based table structure recognition technology. It reconstructs the table's cell adjacency matrix by detecting horizontal and vertical borders or using semantic segmentation networks to identify row and column separators. The system converts the recognized tables into standard HTML table tags or Markdown format, and extracts the caption above the table as metadata for the table object, ensuring semantic indexing of the table data in subsequent processing.

[0034] After completing the above parsing, the system reassembles all extracted elements according to their reading order in the original document, generating a structured document object. This object is stored in a hierarchical JSON format, including: text objects Includes paragraph content and its corresponding page coordinates; table object Includes structured table data and its headers; formula objects It includes the LaTeX source code of the formulas, the screenshot coordinates in the document, and the paragraph context ID to which they belong. This refined structured representation not only preserves the semantic information of the document but also the visual layout information, providing a precise data foundation for subsequently constructing a variable-formula dependency graph.

[0035] In step S2, the system performs the transformation from unstructured text streams to a logical semantic graph, and performs document chunking based on the graph topology. In the text cleaning and standardization stage, the graph construction and chunking module first loads a pre-defined industry unit mapping table and a noise filtering rule base. For text streams within structured objects, the system uses regular expression matching to remove non-printable control characters and garbled symbols generated during optical character recognition. Addressing the unique differences in measurement unit representations specific to the water conservancy field, the system uses a mapping table to uniformly convert them to standard International System of Units (SI) symbols (e.g., unified to...). This eliminates ambiguity in subsequent semantic matching. Simultaneously, the system performs standardized conversion from full-width characters to half-width characters to ensure consistency in all punctuation and numeric encodings.

[0036] In the variable symbol extraction stage, for the formula set For each LaTeX-formatted formula string, the system calls the symbolic mathematics parsing engine to parse it into an Abstract Syntax Tree (AST). The system traverses the leaf nodes of the AST, identifies node objects of type identifier, and filters out those belonging to the set of mathematical reserved words (such as...). , , , ) and universal constant symbols (such as , The remaining identifiers constitute the set of variable symbols for this formula. For example, for the formula The system parses the set of variables. .

[0037] In the cross-document definition search phase, the system uses the extracted variable symbols as anchor points within the standard text stream. The system performs context scanning. The search strategy employs a two-layer probing mechanism combining rule matching and sequence labeling. The first layer is rule matching; the system preloads a set of definitional description templates, including but not limited to sentence structures such as "[VAR] represents…" and "[VAR] is…". The system then searches for the paragraphs in which the formula appears and the adjacent text before and after it. These templates are applied within a sliding window consisting of paragraphs (e.g., three paragraphs before and after) to extract candidate definition text. The second layer is sequence labeling. For implicit definitions that fail to be recalled through rule matching, the system calls a pre-trained Named Entity Recognition (NER) model (e.g., based on the BERT-BiLSTM-CRF architecture). This model has been fine-tuned with water conservancy corpus and can identify physical quantity entities such as water level, flow rate, and reservoir capacity. When the model detects that a physical quantity entity and a variable symbol have a subject-verb relationship or appositive relationship in the same syntactic dependency tree, the entity is marked as a candidate definition.

[0038] In the dependency graph generation stage, the system instantiates a directed graph structure. The system instantiates each logical paragraph containing a formula as a formula node, and each logical paragraph containing valid variable definitions as a definition node. For formula nodes... variables in If at node Found in The exact definition is that the system creates a directed edge between the two nodes. The direction points from the formula node to the definition node, and the variable symbol is recorded in the edge's attributes. And confidence scores. If a variable has multiple definitions in a document (for example, H is defined as the design head in Chapter 1 and the actual head in Chapter 2), the system calculates the weight based on the distance between the formula node and the definition node in the document tree structure, retaining the definition relationship that is closest and located in the same chapter scope, thereby eliminating reference errors caused by polysemy.

[0039] In the semantic segmentation stage under graph constraints, the system executes a graph topology-based text segmentation algorithm. Initially, the system performs sliding window segmentation according to a preset character length (e.g., 500 characters) or semantic integrity (e.g., ending with a period). When generating each segmentation boundary, the system checks whether the boundary cuts any edge in the dependency graph or splits any graph node. Specifically, if the current segmentation point is located between the start and end positions of a graph node, or if the segmentation point separates a formula node within a strongly connected component from its core definition node into two different blocks, the system automatically executes a boundary extension strategy. This strategy moves the end position of the current segment backward until it contains a complete graph node or a complete strongly dependent subgraph, or completely copies the relevant definition text and appends it to the metadata field of the current segment. Through this mechanism, each generated semantic block... Logically, they are all self-contained, avoiding the separation of the formula from its definition in physical storage.

[0040] In step S3, the graph augmentation indexing module performs semantic augmentation vectorization based on the dependency graph and constructs a multi-layer index structure for subsequent hybrid retrieval.

[0041] In the graph context aggregation and embedding stage, the system does not vectorize individual semantic blocks in isolation, but rather uses the variable formula dependency graph constructed in step S2. Execute context-aware embeddings. For chunked collections. Any text block in The system first retrieves the set of its directly adjacent neighbor nodes in the graph. .like For a formula block node, its neighbor set typically contains the definition block nodes of the variables that the formula depends on.

[0042] The system defines an aggregation function `Aggregate` to extract key semantic information from neighboring nodes. This function does not directly copy the entire text of a neighboring node; instead, it extracts specific syntactic segments from the neighboring node that are marked as variable definitions. For example, for a neighboring node... Represents the water level in front of the dam; the function only extracts... This represents the core semantic unit: the water level in front of the dam. Subsequently, the system performs a semantic fusion operation. , will the current block The original text content is concatenated with the aggregated neighbor semantic fragments, and a specific delimiter (Token) is inserted in the middle to form an enhanced semantic representation string.

[0043] Next, the system calls a pre-trained deep language model (such as BERT, RoBERTa, or a dedicated embedding model) as the embedding function. The enhanced semantic representation string is then input into the model, which uses a multi-layer self-attention mechanism to capture the semantic association between formula symbols and variable definitions, outputting a dense vector of fixed dimensions. The calculation formula is as follows: ; in, , The vector dimension is specified (e.g., 768 or 1024 dimensions). Through this process, even if the user only enters a query about the formula, the generated vector implicitly contains the semantic features of the variable definitions in the formula, thereby improving the matching accuracy of the vector space.

[0044] In the hybrid index construction phase, the system constructs a three-layer hybrid storage structure that includes inverted indexes, vector indexes, and graph relation indexes to support multi-way retrieval and context completion in step S4.

[0045] The first level is an inverted index, used to support exact keyword-based matching. The system matches all semantic blocks. The system performs word segmentation, removes stop words, and extracts a keyword set. For specialized terms in the water conservancy industry (such as flood control limits and guarantee rates), the system maintains a dedicated dictionary and assigns higher weights to these terms during indexing. An inverted index records the mapping relationship between keywords and semantic block IDs, used for quickly calculating keyword matching scores.

[0046] The second layer is the VectorIndex, used to support fuzzy retrieval based on semantic similarity. The system will calculate the enhanced vector... The data is stored in a high-dimensional vector retrieval engine (such as FAISS or Milvus). To improve retrieval efficiency, the system employs a quantization-based index structure, such as an inverted file system (NF) or a hierarchical navigation small-world graph (HNSW). When building the index, the system configures the distance metric to be cosine similarity or Euclidean distance (L2Distance) to quickly find the Top-K vectors that are spatially closest to the query vector.

[0047] The third layer is the graph index, used to support context completion after retrieval. The system will use the dependency graph generated in step S2. Serialization is stored as an adjacency list or in a graph database (such as Neo4j). The index explicitly records each semantic block ID, its associated parent node ID, child node ID, and edge attributes. Especially for formula block IDs, the graph index can... The time complexity is sufficient to retrieve the IDs of all variable definition blocks that the index depends on, thus providing physical data support for subsequent context autocomplete. These three layers of indexes are logically linked through unique semantic block IDs, achieving end-to-end data support from keyword matching and semantic retrieval to logical completion.

[0048] In step S4, the hybrid retrieval module executes high-precision retrieval logic, which combines the semantic generalization capability of dense vector retrieval with the precise matching capability of sparse keyword retrieval, and uses graph relationships to force recall dependent context.

[0049] In the query vectorization stage, the hybrid retrieval module first receives the natural language query text input by the user. To ensure that the query vector and the text block vectors in the document library are in the same semantic space, the system calls the pre-trained deep language model (EmbeddingModel) exactly the same as in step S3 as the encoder. During inference, the system truncates or pads the query text to meet the model's maximum input length limit, and then extracts the pooling results of the last hidden state (such as the output of CLStoken or MeanPooling) through forward propagation to generate the query vector. This vector With document vector Same dimensions .

[0050] In the keyword-enhanced similarity scoring stage, the system aims to address the semantic drift problem that may occur when pure vector retrieval processes specific water conservancy industry terms. The system pre-loads a core set of industry terms. This collection was constructed through TF-IDF analysis of a large number of water conservancy standard documents or based on an expert knowledge base.

[0051] The retrieval process employs a dual-path parallel computing strategy. The first path calculates semantic similarity: the system uses a vector index (such as the HNSW index) to calculate the query user's semantic similarity. eigenvectors Compared with all candidate text block vectors in the index The cosine similarity between them. The second approach calculates the keyword matching score: the system uses an efficient multi-pattern matching algorithm (such as the Aho-Corasick automaton) to scan the query user. With candidate text blocks Determine the set of terms Do the words in the text appear in both texts simultaneously?

[0052] To merge the two results, the system executes a weighted scoring logic. For any candidate text block... Its basic correlation score The calculation implementation logic is as follows: ; in, Representing the querying user eigenvectors, The feature vectors representing the cross-section selection, and They are vectors and The Euclidean norm, It is a Boolean indicator logic if and only if the feature Also exists in the query user and text blocks Set to 1 if the condition is met, otherwise set to 0; Terminology collection It is a feature The weights; This is a hyperparameter used to adjust the contribution ratio of semantic matching to keyword matching (e.g., set to a floating-point number between 0.3 and 0.7). Using this formula, text blocks containing key entities in the user query (such as specific dam names or water level terms) will receive additional points, thus ranking higher in the search results.

[0053] In the Top-K retrieval and initial set generation stages, the system sorts all candidate text blocks in descending order based on the calculated basic relevance scores. To balance retrieval performance and response speed, the system employs a min-heap algorithm to select the highest-scoring text. A text block (e.g.) Or 20), forming the initial search result set. The set at this time Although it is highly relevant to the query semantically, there may be logical gaps. That is, it may only retrieve the paragraph containing the formula, while missing the paragraph in which the variable is defined (because the variable definition paragraph may not contain the query term, resulting in a lower score).

[0054] In the recursive completion stage of the graph, this is the core innovative step of this embodiment, which aims to utilize the variable formula dependency graph constructed in step S2. Repair the aforementioned logical inconsistencies. The system iterates through the initial set. Each text block in Read the semantic block ID from its metadata.

[0055] The system accesses the graph index (GraphIndex) with the current text block. The ID is used as the query key to retrieve its outgoing edge connections in the dependency graph. Specifically, the system performs a graph traversal query: if the node... There are pointers to other nodes. directed edges If the edge's attribute is identified as a variable definition dependency, then the target node will be... The context is marked as requiring recall. The system will retrieve all neighbor nodes found through graph association. Add to the initial set, perform a union operation on the sets, and remove duplicate semantic blocks. The final search result set. The construction logic is as follows: ; This step ensures the logical completeness of the search results. For example, when the initial search results contain formulas... Even at that time Even if the paragraph containing the data represents the flow rate and has extremely low similarity to the user's query, the system will still force it to be included in the final results. middle.

[0056] See attached document Figure 2 In step S5, the generative question answering module is responsible for converting the structured search results into natural language answers that users can understand, and ensuring that the logical chain of the answer is complete and traceable.

[0057] In the context reorganization and serialization stage, the generative question answering module first receives the final search result set output from step S4. .because The module includes high-scoring text blocks retrieved through keyword matching and related text blocks forcibly retrieved through graph completion. The generative question-answering module performs deduplication and ranking operations. The ranking strategy does not solely rely on similarity scores but prioritizes preserving the logical adjacency of formula definition groups. The module then processes each semantic block in the set... The text is formatted into text fragments with metadata, such as [Data ID - Paragraph ID] content text, so that the large language model can identify the source of information when generating answers. The generative question answering module calculates the total number of tokens for all formatted text fragments. If it exceeds the maximum context window limit of the large language model, it is truncated according to relevance scores from low to high, but it is configured to forcibly retain the definition text block recalled through dependency graph completion to prevent missing computational logic. In the prompt construction stage, the generative question answering module calls a preset structured instruction template.

[0058] The generative question-answering template consists of three parts: System Prompt, Context data area, and Query. The System Prompt contains constraints such as strictly adhering to the context, prohibiting the fabrication of unmentioned facts, and requiring specific formulas and variable values ​​to be listed when calculations are involved. The Context data area is filled with restructured... Text sequence. Enter the original user query in the user intent area. For queries involving numerical calculations, the module dynamically inserts chain-of-thought prompts into the prompt, such as asking you to extract the formula step by step, confirm the variable values, and perform the calculation, in order to activate the logical reasoning ability of the large model.

[0059] In the inference and generation phases, the module deploys a large language model (LLM) based on the pre-built complete Prompt input. This model is configured to perform autoregressive generation tasks. During inference, the model leverages its pre-trained semantic understanding capabilities, combined with formula text blocks (derived from formula nodes in the graph) and variable definition text blocks (derived from definition nodes in the graph) provided in the Prompt, to establish logical mappings within its internal state. For example, when the problem involves calculating the current rainfall carrying capacity of a reservoir, the model first locates the calculation formula from the context, then extracts the specific values ​​or definitions of the variables required for the formula (such as the current water level and flood control limit water level) from the completed context, and finally performs numerical calculations or logical deductions.

[0060] Final Answer The generation process follows mathematical expression: ; in, Represents the inference function of a large language model. This is the generated natural language text sequence.

[0061] In the answer tracing and output stage, the generative question answering module provides the model's output. Post-processing and parsing are performed. The module detects document ID tags referenced in the output text and replaces them with the original document's title and page number links, generating clickable reference footnotes. If the generated answer contains calculation steps, the module verifies whether the formulas referenced by the model are consistent with the formulas provided in the context. If key parameters are detected as missing (i.e., the necessary variable definitions are not included in the Prompt), the module is configured to output variables that cannot be calculated based on the current data and are missing. Instead of outputting incorrect calculation results, it provides a definition prompt.

[0062] Through this mechanism, the system utilizes the context of graph completion to ensure a logical closed loop in the generated results, thus solving the technical defect of traditional models that are prone to producing illusions when dealing with complex industry formulas.

[0063] This invention provides a dynamic knowledge question-answering system for the water conservancy industry based on variable formula dependency graph enhancement. The system runs on a server or distributed computing cluster at the hardware level and includes the following modules at the logical level: document acquisition module, multimodal parsing module, graph construction and segmentation module, graph enhancement indexing module, hybrid retrieval and completion module, and generative question-answering module.

[0064] The document acquisition module, configured as the system's input interface, is used to access and deduplicate multi-source data. This module includes a web crawler unit and a database connection unit. The web crawler unit is configured with a Hypertext Transfer Protocol (HTTP) interface to crawl publicly available web page document streams according to a preset Uniform Resource Locator (URI) list. The database connection unit is configured with a Java database connection or open database connection interface to read technical specifications and scheduling instruction files from internal business systems. The document acquisition module integrates a hash calculation unit, configured to calculate an MD5 or SHA-256 digest value for each accessed document binary stream, compare this digest value with the digest values ​​of documents already stored in memory, and mark the document as a new document only if the digest value does not exist. And transmit it to the multimodal parsing module.

[0065] The multimodal parsing module, connected to the document acquisition module, is configured to convert unstructured document streams into structured data objects. This module includes a type recognition unit, an optical character recognition unit, and a formula extraction unit. The type recognition unit reads the binary code of the file header to determine the file format. The optical character recognition unit is equipped with a deep neural network (such as CRNN) to process documents identified as scanned PDFs, mapping image pixels to text character encodings. The formula extraction unit internally deploys an encoder-decoder model based on the Transformer architecture. The model is configured to identify mathematical formula regions in a document and decode the image features of these regions into a LaTeX format string sequence. The output of the multimodal parsing module is configured to output plain text content. Table objects and formula objects Structured Document Object .

[0066] The graph construction and segmentation module, connected to the multimodal parsing module, is configured to establish logical dependencies in document content and perform semantic segmentation. Internally, this module includes a variable parsing submodule, a definition mining submodule, and a graph generation submodule. The variable parsing submodule is configured to call an abstract syntax tree parser to decompose the LaTeX string output by the formula extraction unit and extract independent variable symbols. The definition mining submodule is configured to perform regular expression matching and named entity recognition in the text stream to locate the natural language definition paragraphs corresponding to the variable symbols. The graph generation submodule is configured to instantiate a directed graph structure in memory. This module creates directed edges from the paragraph node containing the formula to the paragraph node containing the definition. It also includes a chunking controller configured to read the graph topology and, during document segmentation, detect whether the split points truncate directed edges in the graph. If truncated, it automatically adjusts the split boundaries and outputs a semantically complete set of chunks. .

[0067] The graph enhancement indexing module, connected to the graph construction and chunking module, is configured to convert semantic chunks into computer-computable vector representations and store them. This module includes a graph context aggregation unit and a vector encoding unit. The graph context aggregation unit is configured to access dependent graphs. The system retrieves the content of neighboring nodes in the current block and concatenates the definition text of the neighboring nodes with the text of the current block. The vector encoding unit is loaded with a pre-trained deep language model, configured to receive the concatenated text and output a dense vector of fixed dimensions. The graph-enhanced indexing module is also connected to a hybrid database cluster, which includes: a search engine for storing keyword inverted indexes, a vector database for storing dense vectors, and a graph database for storing graph topological relationships.

[0068] The hybrid retrieval and completion module connects to the hybrid database cluster of the graph augmentation indexing module and the generative question answering module, respectively. This module is configured to execute retrieval and recall logic. Internally, it includes a query encoder, a multi-way scorer, and a graph walk controller. The query encoder is configured to process user queries... Convert to query vector The multi-path scorer is configured to calculate vector cosine similarity and keyword matching scores in parallel, and then calculates the scores based on preset weights. Calculate the overall score and select the initial candidate set. The graph walk controller is configured to start with the semantic block IDs in the initial candidate set, perform a breadth-first search or a traversal of a specific depth in the graph database, find all associated nodes connected by dependency edges defined by variables, and merge these associated nodes into the final result set. This allows for logical completion of the context.

[0069] The generative question-answering module, connected to the hybrid retrieval and completion module, is configured to generate natural language answers. This module internally includes a prompt word assembler and a large language model inference engine. The prompt word assembler is configured to combine system commands and the completed context output from the hybrid retrieval and completion module according to a preset template. and user queries The sequences are concatenated into a Prompt sequence that the model can recognize. The large language model inference engine loads a generative pre-trained model that has been fine-tuned according to instructions, configured to receive the Prompt sequence, perform autoregressive decoding calculations, and output the predicted answer text. The generative question-answering module also includes a citation validator, configured to detect citation markers in the output answer and map them back to the metadata information of the original document (such as document title and page number), forming the final structured response data to be sent to the terminal device.

[0070] The modules communicate with each other via an internal bus or a remote procedure call protocol. Specifically, the document acquisition module, multimodal parsing module, graph construction and segmentation module, and graph enhancement indexing module are configured to run asynchronously offline for building the knowledge base; the hybrid retrieval and completion module and the generative question answering module are configured to run synchronously online for real-time responses to user requests.

Claims

1. A method for enhancing question-and-answer generation in water conservancy literature retrieval, characterized in that, Includes the following steps: Step S1: Collect multi-source heterogeneous documents from the water conservancy industry, perform multimodal parsing on the multi-source heterogeneous documents, and transform unstructured documents into structured document objects containing plain text, table objects, and formula sets; Step S2: Parse the set of formulas in the structured document object to extract variable symbols, search for the natural language definitions of the variable symbols in the plain text, and construct a variable formula dependency graph; Based on the topological structure of the variable formula dependency graph, the plain text is semantically segmented under graph constraints to obtain a set of semantic blocks; Step S3: Based on the variable formula dependency graph, perform graph context aggregation and vector encoding on each semantic block in the semantic block set to generate enhanced vectors, and construct a hybrid index library containing inverted index, vector index and graph relation index; Step S4: In response to the user's query request, perform a dual-path hybrid retrieval using keywords and vectors through the hybrid index library to obtain an initial retrieval result set; perform graph recursive completion on the initial retrieval result set based on the graph relationship index, forcibly add related nodes with dependencies to the initial retrieval result set, and generate a final context set; Step S5: Assemble the final context set and the user-input query request into a prompt word sequence according to a preset template, input it into the large language model, and generate a natural language answer containing citation tracing.

2. The enhanced question-and-answer generation method for water conservancy literature retrieval according to claim 1, characterized in that, The steps for performing multimodal parsing on the multi-source heterogeneous documents include: The document header format is identified. If it is a scanned document, the optical character recognition engine is used to extract the text. If it is a native electronic document, the text and tables are parsed based on the document object model tree. The encoder-decoder model based on the Transformer architecture is used to identify mathematical formula regions in a document page. The image features of the mathematical formula regions are decoded into LaTeX format string sequences, and the mathematical formula regions are combined with the position information of the mathematical formula regions in the document to encapsulate them into the formula set.

3. The enhanced question-and-answer generation method for water conservancy literature retrieval according to claim 1, characterized in that, The steps for constructing the variable formula dependency graph include: The abstract syntax tree parser is invoked to decompose the LaTeX format strings in the formula set, extract the identifiers in the leaf nodes and filter out mathematical reserved words to obtain the set of independent variable symbols; Using the variable symbols in the set of independent variable symbols as anchor points, a definition search based on rule template matching and named entity recognition is performed within a preset sliding window of the plain text and table objects to locate the definition text block containing the variable definition. Instantiate a directed graph structure by instantiating paragraphs containing formulas as formula nodes and paragraphs containing the definition text blocks as definition nodes, and create directed edges representing definition dependencies between the formula nodes and the definition nodes to form the variable formula dependency graph.

4. The enhanced question-and-answer generation method for water conservancy literature retrieval according to claim 3, characterized in that, In step S2, the semantic segmentation of the plain text under graph constraints based on the topological structure of the variable formula dependency graph includes: The plain text is initially segmented according to a preset character length or natural paragraph, generating segmentation boundaries; Detect whether the dividing boundary truncates the directed edges in the variable formula dependency graph, or whether it separates formula nodes and definition nodes that are in the same strongly connected component into two different regions; If truncation or segmentation is detected, a boundary extension strategy or a semantic splicing strategy is executed according to the preset block length constraint: the end position of the current block is moved backward to include the complete node, or the text of the segmented associated node is retrieved and copied and spliced ​​into the current block, and the logically self-contained set of semantic blocks is output.

5. The enhanced question-and-answer generation method for water conservancy literature retrieval according to claim 1, characterized in that, In step S3, performing graph context aggregation and vector encoding on each semantic block in the semantic block set includes: For the current semantic block in the set of semantic blocks, retrieve the set of directly adjacent neighbor nodes in the variable formula dependency graph; Extract semantic segments marked as variable definitions from the set of neighboring nodes, and concatenate them into the original text of the current semantic block through a semantic fusion operation to form an enhanced semantic representation string; The enhanced semantic representation string is input into a pre-trained deep language model, and a dense vector of fixed dimensions is calculated and output through a multi-layer self-attention mechanism as the enhanced vector.

6. The enhanced question-and-answer generation method for water conservancy literature retrieval according to claim 1, characterized in that, In step S3, the step of constructing a hybrid index library containing inverted indexes, vector indexes, and graph relation indexes includes: The semantic block set is segmented and stop word removed, keywords are extracted, and an inverted index is constructed by combining industry term weights to record the mapping relationship between keywords and semantic block identifiers; The enhanced vectors are stored using a quantization-based index structure to construct the vector index for supporting similarity measurement; The variable formula depends on the graph serialization and storage as an adjacency list. A graph relationship index is constructed to record semantic block identifiers, parent-child node associations, and edge attributes to support neighbor queries with constant time complexity.

7. The enhanced question-and-answer generation method for water conservancy literature retrieval according to claim 6, characterized in that, In step S4, the process of performing a dual-path hybrid retrieval using keywords and vectors using the hybrid index library to obtain an initial set of retrieval results includes: The user-input query request is transformed into a query vector, and the semantic similarity score between the query vector and the candidate text block vectors in the vector index, as well as the keyword matching score between the keywords in the query request and the inverted index, are calculated in parallel. The semantic similarity score and the keyword matching score are weighted and summed based on preset adjustment parameters to obtain the basic relevance score; Candidate semantic blocks are sorted in descending order based on the aforementioned basic relevance scores, and the top [number] blocks are selected. The initial search result set consists of several semantic blocks.

8. The water conservancy literature retrieval enhancement question-and-answer generation method according to claim 6, characterized in that, In step S4, the step of performing graph recursive completion on the initial retrieval result set based on the graph relation index includes: Traverse each semantic block in the initial retrieval result set and read the corresponding semantic block identifier; use the semantic block identifier as the query key to retrieve outgoing edge connections with variable definition dependencies in the graph relation index and locate the target neighbor node; The retrieved target neighbor nodes are forcibly added to the initial search result set, and a deduplication operation is performed to obtain the final context set containing the complete formula definition logic chain.

9. The enhanced question-and-answer generation method for water conservancy literature retrieval according to claim 1, characterized in that, In step S5, the step of inputting a large language model and generating a natural language answer containing citation tracing includes: Inject thought chain guiding words into the prompt word sequence to instruct the large language model to extract formulas step by step and confirm variable values; Receive the predicted text output by the large language model and parse the citation tags therein; Verify whether the formula corresponding to the reference mark is consistent with the formula in the final context set. If they are consistent, map the reference mark to the title and page number link of the original document to generate the natural language answer.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

Images