A multi-modal tender document key element extraction method and system

Abstract

The application provides a kind of multimodal tender document key element extraction method and system, it is related to document analysis technical field.The method includes: tender document multimodal layout analysis, identifies text, table and drawing area and chapter metadata;Text blocks are merged based on semantic position correlation degree;Table and drawing area are cut into pieces with chapter metadata;Text fragments are extracted by large language model index, key-value pairs are obtained by visual table model, and triplets are extracted from drawings by multimodal large model;With chapter information as anchor point, semantic alignment and intent clustering are carried out, and knowledge graph with directed edges containing constraints, mutual exclusion, time dependence and equivalence relationship is constructed;Cross-modal conflicts are detected based on the graph, main verification and supporting information anchor points are compared to form confidence, correlation strength is calculated to construct verification chain, errors are corrected along the chain, and conflict-free results are output.The problems of difficult correlation and integration of multimodal information, lack of accurate traceability, inability to automatically correct cross-modal conflicts and insufficient description of logical relationships are solved.

Description

Technical Field

[0001] This invention relates to the field of document parsing technology, and more specifically, to a method and system for extracting key elements from multimodal tender documents. Background Technology

[0002] Tender documents are comprehensive legal texts that contain key information such as project requirements, technical specifications, commercial terms, and evaluation criteria in bidding and procurement activities. Their content often exists in multiple modalities, including text paragraphs, structured tables, and design drawings. Quickly and accurately extracting various key elements from tender documents is crucial for bidding decisions, compliance reviews, and knowledge management.

[0003] Currently, the extraction of key elements from tender documents mainly relies on manual reading and transcription, or single-modal text extraction methods based on rules and traditional natural language processing techniques. Some solutions introduce visual recognition models to structure tables or utilize optical character recognition (OCR) technology to extract text information from drawings. However, these methods are mostly designed for single-modal independent applications and have significant shortcomings in practical applications. First, cross-modal information is difficult to effectively correlate and integrate. The same business entity in the tender documents may appear repeatedly and complement each other in text descriptions, table parameters, and drawing annotations. However, existing technologies usually treat text, tables, and drawings as isolated information sources and process them separately. They lack a unified anchor point to align and merge the extraction results of different modalities at the semantic level, and thus cannot form a consistent and complete expression of elements.

[0004] Second, the results extracted by large language models lack controllability and traceability. In recent years, although attempts have been made to use large language models for information extraction, the models are often required to directly generate the values ​​of the target fields, which can easily lead to "illusions." The generated content may deviate from the original text, and it is impossible to accurately trace the exact location of the extracted fragment in the original document, which makes subsequent review and proofreading difficult.

[0005] Third, cross-modal data conflicts are difficult to detect and correct automatically. When text descriptions, table values, and drawing annotations provide inconsistent information about the same attribute, existing solutions cannot systematically detect such conflicts. Furthermore, they lack mechanisms for assessing the confidence of conflicting nodes and the ability to verify and correct untrusted sources based on trusted sources, resulting in insufficient reliability of the extraction results.

[0006] Fourth, the logical relationships between elements are not adequately expressed. Key elements in tender documents often have complex relationships such as constraints, mutual exclusion, time dependence, and equivalence. Simply outputting flat key-value pairs of fields cannot depict the deep semantic structure between elements, making it difficult to support subsequent automated review and reasoning.

[0007] In summary, there is an urgent need for a method for extracting key elements from multimodal tender documents that can deeply integrate multimodal information, accurately locate extracted segments, and automatically verify and correct cross-modal inconsistencies. Summary of the Invention

[0008] The purpose of this invention is to provide a method and system for extracting key elements of multimodal tender documents, which aims to solve the problems in the existing extraction of key elements of tender documents, such as difficulty in linking and integrating multimodal information, lack of accurate traceability of extraction results, inability to automatically detect and correct cross-modal data conflicts, and insufficient characterization of logical relationships between elements.

[0009] This invention is achieved through the following technical solution: A method for extracting key elements from multimodal tender documents includes the following steps: Multimodal layout analysis is performed on the tender documents to identify text areas, table areas, and drawing / image areas, as well as their physical coordinates and chapter / page number metadata. For the text areas, the text blocks are merged based on the weighted fusion association strength of the semantic association degree and the positional distance association degree of adjacent basic text blocks to generate data text blocks. The table areas and drawing / image areas are respectively cropped into table image fragments and drawing image fragments with chapter / page number metadata. After segmenting the data text into sentences and assigning indices, the large language model is constrained by instructions to output only the start and end indices of the target key fields in the sentence list. Based on the start and end indices, the original text fragments are extracted. The key-value pair structured data is extracted from the table image fragments through the visual table recognition model. The parameter-value-unit triplet structured data is extracted from the drawing image fragments through the multimodal large language model. Using the chapter information carried by the output data of each model as physical anchors, the output data of each model is classified into the corresponding candidate data clusters for semantic alignment and intent clustering, and a global knowledge graph of bidding containing nodes, attributes and directed edges is constructed. The directed edges are used to represent the constraints, mutual exclusion, time subordination and equivalence relationships between nodes. Based on the global knowledge graph of the bidding process, data conflicts between cross-modal nodes are detected. For node pairs that conflict and point to the same business entity, the confidence levels of the two nodes in the pair are compared. The node with the higher confidence level is set as the primary verification information node, and the node with the lower confidence level is set as the supporting information node. The two form an information anchor pair. The association strength between the nodes in the information anchor pair is calculated, and anchor pairs with association strength greater than the strength threshold are linked into an information anchor pair verification chain. Along the verification chain, the supporting information node is verified and corrected by the primary verification information node, and the key element extraction results without conflicts are output.

[0010] Optionally, the specific process of performing multimodal layout analysis on the tender documents is as follows: After obtaining the original tender documents and converting them into page images, the content modality of the page is identified page by page using a multimodal visual model. Plain text paragraphs are marked as text modal areas, areas containing row and column structures are marked as table modal areas, and areas containing graphics or engineering symbols are marked as image modal areas. During the recognition process, sub-images corresponding to each modal region are extracted, the physical coordinate frame of each modal region on the original page is recorded, and according to the pre-extracted directory structure or page number mapping rules, chapter page number metadata containing the chapter title and the page number is added to each modal region. The identifiers, modality types, physical coordinate frames, screenshot save paths, and chapter / page number metadata of each modal region are encapsulated into a standardized multimodal fragmentation information set and stored in a structured manner.

[0011] Optionally, the weighted fusion of association strength based on the semantic relevance and positional distance relevance of adjacent basic text blocks includes: The text to be processed is divided into multiple basic text blocks according to a preset physical length; For adjacent basic text blocks, pre-trained semantic coding models are used to extract contextual semantic features, generate corresponding high-dimensional semantic vectors, and calculate the cosine similarity between the two high-dimensional semantic vectors as the semantic relevance. Based on the physical coordinates of adjacent basic text blocks on the tender document page, determine the physical distance between the end of the previous basic text block and the beginning of the next basic text block, using a natural constant. The positional distance correlation is calculated using the product of a preset distance attenuation coefficient and the physical distance to the page as the base and the exponent as the exponent. The semantic correlation degree and the location distance correlation degree are weighted and summed according to preset semantic weights and spatial weights to obtain the weighted fusion correlation strength; when the weighted fusion correlation strength is greater than the preset segmentation fusion threshold, adjacent basic text blocks are merged into the same data text block; otherwise, a segmentation breakpoint is formed at this point. The distance decay coefficient, semantic weight, spatial weight, and segmentation fusion threshold are determined by cross-validation and joint parameter tuning, with the semantic integrity and granularity balance of the segmentation results as the optimization objectives, based on the manually annotated text block merging and segmentation boundaries.

[0012] Optionally, after segmenting the data text block into sentences and assigning indices, the large language model is constrained by instructions to output only the start and end indices of the target key field in the sentence list, and the original text fragments are extracted from the original text based on the start and end indices, including: The data text block is segmented into sentences according to punctuation to obtain a list of sentences carrying physical coordinate frames, and each sentence is assigned a unique index number to generate a sentence sequence with index tags. Construct an instruction template by filling the instruction template with a list of indexed sentence sequences and target key fields. The instruction template constrains the output format of the large language model to be a preset structured data containing only target key field identifiers, start sentence index, and end sentence index, and sets a hard stop sequence to prevent the model from generating additional explanatory text. The instruction template is input into the large language model, and the preset structured data returned by the large language model is parsed to obtain the starting sentence index and ending sentence index corresponding to each target key field. Determine the corresponding target sentence based on the starting sentence index and the ending sentence index, obtain the physical coordinate frame of all target sentences, and calculate the minimum bounding rectangle as the physical cropping area of ​​the target key field on the page image; Extract a sub-image from the page image of the tender document that corresponds to the physical cropping area, and perform text recognition on the sub-image to obtain the original text cropping segment corresponding to the target key field.

[0013] Optionally, the step of extracting key-value pair structured data from table image slices using a visual table recognition model includes: The table image is segmented and input into a pre-trained table recognition model based on the visual Transformer architecture. The table's row and column separators are detected, the bounding boxes of all cells are determined, and text recognition is performed on the image regions within each bounding box to obtain a set of cell text containing row and column coordinates. Based on the row and column coordinates, the header area and data area are identified. The text of the header cell is used as the key, and the text of the corresponding data cell in the same row or column is used as the value. The data is then assembled into key-value pairs row by row or column by column. The chapter and page number metadata carried by the table image segments is appended as metadata to the key-value pairs to obtain key-value pair structured data with chapter and page number metadata.

[0014] Optionally, the extraction of parameter-value-unit triplet structured data from drawing image segments using a multimodal large language model includes: The drawing images are segmented and input into a multimodal large language model to recognize the technical parameter annotations and engineering symbol text in the drawings, and output the drawing text recognition results; Based on a preset data cleaning template that matches the technical parameter annotation format of the bidding drawings, the text recognition results of the drawings are parsed to extract the parameter item names, values ​​and units of measurement; The extracted parameter item name, value, and unit of measurement are constructed into a parameter item-value-unit triple, and the chapter and page number metadata carried by the drawing image fragments are appended to the triple to obtain structured drawing data with chapter and page number metadata.

[0015] Optionally, the specific process of constructing a global knowledge graph for bidding that includes nodes, attributes, and directed edges is as follows: Based on the chapter and page metadata attached to the original text clippings, the key-value pair structured data, and the parameter-value-unit triplet structured data, the extraction results of different modalities belonging to the same chapter are aggregated into the same candidate data cluster; Semantic vector encoding is performed on each data item within the candidate data cluster, and the semantic similarity between data items is calculated. Different data items with semantic similarity higher than the preset entity alignment threshold are identified as pointing to the same business entity and merged into the same entity node. Assign the field values ​​corresponding to each merged data item to the attribute of the entity node; Connecting words representing logical relationships between entities are identified from the original text context of each data item. According to the preset logical relationship mapping rules, the identified connecting words are transformed into constraint, mutual exclusion, time subordination or equivalence relationship types. Directed edges are established between the corresponding entity nodes along the transformed relationship direction to form the global knowledge graph of the bidding.

[0016] Optionally, comparing the confidence levels of the two nodes in the node pair includes: Obtain the original extracted data corresponding to each node in the conflict node pair in the global knowledge graph of the bidding, and determine the model type used to generate the original extracted data; the original extracted data is the original text fragment with chapter and page number metadata, key-value pair structured data, or parameter-value-unit triplet structured data; the model type includes large language model, visual table recognition model, and multimodal large language model; When the model type is a large language model, the information of the node is derived from the original text fragment. Therefore, the sequence generation probability is extracted from the process of generating the start and end index of the large language model as the basic confidence score. When the model type is a visual table recognition model, it means that the information of the node comes from key-value pair structured data. Then, the character recognition confidence is extracted from the cell text recognition results of the visual table recognition model as the base confidence. When the model type is a multimodal large language model, the information representing the node comes from the parameter-value-unit triplet structured data. Therefore, the token generation probability is extracted from the process of outputting the parameter-value-unit triplet structured data from the multimodal large language model as the basic confidence level. Based on the information source of the node, a corresponding modal weight factor is assigned. Among them, the weight factor of the parameter-value-unit triplet structured data is greater than the modal weight factor of the key-value pair structured data, and the modal weight factor of the key-value pair structured data is greater than the modal weight factor of the original text fragment. The confidence level of a node is obtained by multiplying the base confidence level of each node by the corresponding modal weight factor. Compare the confidence levels of the two nodes in a node pair to determine the relative confidence levels of the two nodes.

[0017] Optionally, the calculation of the association strength between nodes in the information anchor pair, and linking anchor pairs with association strength greater than a strength threshold into an information anchor pair verification chain, includes: Extract the attribute name sets of the two nodes in the information anchor pair in the global knowledge graph of the bidding, and calculate the Jaccard similarity coefficient of the two attribute name sets; Using a pre-trained semantic coding model, semantic vectors are encoded for the entity names of the two nodes, and the cosine similarity between the two semantic vectors is calculated. The calculated Jaccard similarity coefficient and cosine similarity are weighted and summed according to preset weights to obtain the association strength between nodes; Based on manually labeled associated node pairs, the strength threshold is determined by grid search with the goal of maximizing the F1 score of the association strength judgment. Information anchor pairs with a correlation strength greater than the strength threshold are sequentially linked according to their shared node relationship to form an information anchor pair verification chain.

[0018] Based on the same inventive concept, this invention also provides a multimodal tender document key element extraction system, used to implement the aforementioned multimodal tender document key element extraction method, including: The multimodal layout analysis module is used to perform multimodal layout analysis on tender documents, identify text areas, table areas, and drawing / image areas, as well as their physical coordinates and chapter / page number metadata, and merge text areas into data text blocks based on the weighted fusion association strength of the semantic association degree and positional distance association degree of adjacent basic text blocks, and crop table areas and drawing / image areas into table image fragments and drawing image fragments with chapter / page number metadata, respectively; The multimodal key element extraction module, connected to the multimodal layout analysis module, is used to segment the data text block into sentences and assign indexes. Then, through instruction constraints, the large language model outputs only the start and end indices of the target key fields in the sentence list and extracts the original text fragments accordingly. The visual table recognition model extracts key-value pair structured data from the table image segments, and the multimodal large language model extracts parameter-value-unit triplet structured data from the drawing image segments. The knowledge graph construction module, connected to the multimodal key element extraction module, is used to use the chapter information carried by the output data of each model as physical anchors to classify the original text fragments, key-value pair structured data and parameter item-value-unit triplet structured data into corresponding candidate data clusters for semantic alignment and intent clustering, and to construct a global knowledge graph for bidding that includes nodes, attributes and directed edges. The directed edges are used to represent the constraints, mutual exclusion, temporal dependence and equivalence relationships between nodes. The conflict detection and correction module, connected to the knowledge graph construction module, is used to detect data conflicts between cross-modal nodes based on the global knowledge graph of the bidding process. It compares the confidence levels of node pairs that conflict and point to the same business entity, designating nodes with relatively higher confidence levels as primary verification information nodes and nodes with relatively lower confidence levels as supporting information nodes to form information anchor pairs. It calculates the association strength between nodes in the information anchor pairs and links anchor pairs with association strength greater than a strength threshold into an information anchor pair verification chain. Along the verification chain, the supporting information nodes are verified and corrected using the primary verification information node, and the result of extracting conflict-free key elements is output.

[0019] The technical solution of the present invention has at least the following advantages and beneficial effects: By using multimodal layout analysis to obtain the physical coordinates and chapter / page number metadata of text, table, and drawing areas, and using chapter information as a unified physical anchor point, the extraction results of each modality are grouped into candidate data clusters for semantic alignment and intent clustering to construct a global knowledge graph for bidding. This overcomes the shortcomings of existing technologies that treat text, tables, and drawings as isolated information sources and process them separately. It can automatically merge the scattered descriptions of the same business entity in text descriptions, table parameters, and drawing annotations to form a consistent and complete element expression.

[0020] By constraining the large language model to output only the start and end indices of the target key fields in the sentence list, and then accurately cutting and extracting segments from the original text based on the index, the model avoids the "illusion" and content bias that may be introduced by directly generating field values. At the same time, each extracted segment can be accurately traced back to the exact sentence in the original tender document through the index, providing a reliable basis for subsequent manual review, proofreading and dispute tracing.

[0021] Based on the global knowledge graph of bidding, data conflicts between cross-modal nodes are automatically discovered. By comparing confidence levels, an information anchor verification chain consisting of a main verification information node and supporting information nodes is constructed. The verification and correction of untrusted source information is completed along this chain, and finally, conflict-free extraction results are output. This effectively solves the problem that existing solutions cannot systematically discover and correct inconsistencies in multi-source data, and improves the reliability of key element extraction results.

[0022] The constructed global knowledge graph for bidding contains directed edges representing relationships such as constraints, mutual exclusion, temporal subordination, and equivalence. It can model and characterize the complex business logic between elements. Compared with existing methods that only output flat key-value pairs, the structured knowledge output by this invention can directly support automated compliance review and logical reasoning, thereby improving the intelligence level of subsequent applications.

[0023] By deeply integrating multimodal layout analysis, controllable indexed text extraction, visual structured recognition, and knowledge graph reasoning, this approach effectively solves bottleneck problems such as information fragmentation, lack of result traceability, difficulty in discovering conflicts, and difficulty in expressing relationships in the extraction of key elements from multimodal bidding documents. It improves the accuracy, automation, and engineering applicability of key element extraction, and provides high-quality structured data support for bidding decisions, compliance review, and knowledge management. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the method for extracting key elements from multimodal tender documents according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of the multimodal tender document key element extraction system according to an embodiment of the present invention. Detailed Implementation

[0025] The following is a detailed description of the embodiments, in conjunction with the accompanying drawings.

[0026] Reference Figure 1 A method for extracting key elements from multimodal tender documents, comprising the following steps: Step 1: Perform multimodal layout analysis on the tender documents to identify text areas, table areas, and drawing / image areas, as well as their physical coordinates and chapter / page number metadata. For the text areas, merge them based on the weighted fusion association strength of the semantic relevance and positional distance relevance of adjacent basic text blocks to generate data text blocks. The table areas and drawing / image areas are respectively cropped into table image fragments and drawing image fragments with chapter / page number metadata.

[0027] In some embodiments, the specific process of performing multimodal layout analysis on the tender documents is as follows: After obtaining the original tender documents and converting them into page images, a multimodal visual model is used to identify the content modality of each page. Plain text paragraphs are marked as text modal regions, regions containing row and column structures are marked as table modal regions, and regions containing graphics or engineering symbols are marked as image modal regions. During the identification process, sub-images corresponding to each modal region are extracted, and the physical coordinate frame of each modal region on the original page is recorded. Based on the pre-extracted directory structure or page number mapping rules, chapter and page number metadata containing the chapter title and the page number is added to each modal region. The identifier, modality type, physical coordinate frame, screenshot save path, and chapter and page number metadata of each modal region are encapsulated into a standardized multimodal fragment information set and stored in a structured manner.

[0028] Furthermore, firstly, the original tender document to be processed is obtained, which can be in PDF (Portable Document Format), Word document, or scanned copy. To unify the subsequent processing interface and preserve the original layout's visual characteristics, a file conversion engine (such as a PDF rendering library or Office document conversion service) is invoked to render the tender document page by page into high-resolution page images. For scanned copies, if they are already a collection of images, they are used directly; if they are multi-page image-based PDFs, they are broken down into independent image frames per page. Each converted page image is assigned a unique page number, forming a page image sequence. This unified conversion ensures that all modalities, including text, tables, and graphics, are analyzed under the same visual channel, avoiding modal fragmentation caused by differences in file formats.

[0029] Subsequently, the sequence of page images is sequentially input into a pre-trained multimodal vision model. This model can be a vision-based Transformer architecture (such as the Swing Window Transformer or the Vision Transformer (ViT)) combined with a layout analysis model that integrates textual semantics (such as LayoutLMv3 or the Document Image Transformer (DiT)). After fine-tuning, it is equipped with region detection and modality classification capabilities. For each page image, the model performs modality region detection and classification, as well as bounding box refinement and physical coordinate recording operations. The modality region detection and classification operations include: The model generates several candidate bounding boxes on the page image using an object detection head, and simultaneously predicts the modality category of each bounding box. Specifically, regions containing continuous text paragraphs without regular row and column lines are labeled as "text modal regions"; regions with obvious table lines or matrix arrangements are labeled as "table modal regions"; and regions containing non-text-based elements such as line drawings, engineering symbols, header lines, and product illustrations are labeled as "figure modal regions". The classification criteria include not only visual features but also preliminary optical character recognition (OCR) text density and semantics within the bounding box. For example, a combination of high-density short text and separator lines tends to indicate a table modality.

[0030] Bounding box refinement and physical coordinate recording operations include: For each detected modal region, the model outputs its corresponding physical bounding box. This physical bounding box is represented using normalized or absolute coordinates in the pixel coordinate system of the page image. These coordinates are recorded as "physical bounding boxes" and used to characterize the position of each modal region on the original page, providing direct evidence for subsequent text segmentation, source tracing, and positional association.

[0031] After identifying each modal region, sub-image cropping and chapter / page number metadata appending are performed. Sub-image cropping includes: Based on the physical coordinate frame of each modal region, corresponding sub-images are cropped from the original page image. These sub-images contain only the pure content of a single modality, removing distractions such as headers, footers, and page numbers, while still preserving the original visual details (such as fonts, lines, and annotations). Each sub-image is assigned a unique region identifier and saved to temporary or persistent storage via a preset path, while also recording the screenshot save path.

[0032] Chapter page number metadata appendices include: To achieve semantic alignment of cross-modal information, each modal region is assigned a corresponding chapter affiliation and page number information. This process relies on pre-extracted directory structure or page number mapping rules. Specifically, the directory information of the tender document is obtained in advance through OCR recognition or document structure parsing, and a mapping table of "chapter title → starting page number" is established. During page image processing, the page number of the current page is read through OCR or PDF metadata. Then, based on the page number of each modal region, the directory mapping table is queried to determine its corresponding chapter title. Finally, chapter page number metadata is generated for each modal region, including the chapter title and the page number (e.g., "Chapter 3 Technical Specifications, Page 23"). In this way, even if text areas, table areas, and drawing areas are physically scattered, as long as they belong to the same chapter, they carry the same chapter title metadata, providing a natural physical anchor for the candidate data cluster division based on chapter information in step three.

[0033] Finally, all the above information is standardized and encapsulated to form a structured multimodal fragmented information set.

[0034] In some embodiments, the weighted fusion of association strength based on the semantic relevance and positional distance relevance of adjacent basic text blocks includes: The text to be processed is divided into multiple basic text blocks according to a preset physical length. For adjacent basic text blocks, contextual semantic features are extracted using a pre-trained semantic coding model to generate corresponding high-dimensional semantic vectors. The cosine similarity between the two high-dimensional semantic vectors is calculated as the semantic relevance. Based on the physical coordinates of adjacent basic text blocks on the tender document page, the physical distance between the end of the previous basic text block and the beginning of the next basic text block is determined, using a natural constant. Using the product of a preset distance attenuation coefficient and the physical distance of the page as the base, the positional distance correlation is calculated; the semantic correlation and the positional distance correlation are weighted and summed according to preset semantic weights and spatial weights to obtain the weighted fusion correlation strength; when the weighted fusion correlation strength is greater than the preset segmentation fusion threshold, adjacent basic text blocks are merged into the same data text block; otherwise, a segmentation breakpoint is formed at this point; wherein, the distance attenuation coefficient, semantic weight, spatial weight, and segmentation fusion threshold are based on the manually annotated text block merging and segmentation boundaries, and the semantic integrity and granularity balance of the segmentation results are optimized as the optimization objectives, and are determined through cross-validation and joint parameter tuning.

[0035] Furthermore, firstly, for the identified text modal regions (i.e., plain text paragraph regions), the continuous text flow is divided into multiple basic text blocks according to the text line layout within its physical coordinate frame and a preset physical length. This "preset physical length" can be defined as a fixed number of lines (e.g., one block every 5 lines) or a fixed physical height (e.g., every 2 centimeters), or it can be combined with natural paragraph markers detected in the layout analysis (e.g., first-line indentation). In practical implementation, the preferred approach is to use the line-level coordinates output by OCR to accumulate line height from top to bottom. When the accumulated height exceeds a threshold (e.g., 1 / 8 of the page height), a basic text block is formed, and an initial segmentation candidate point is set at that location. Finally, a sequence of basic text blocks is obtained. , For the first in the sequence A basic text block Each basic text block contains the following attributes: text content, the smallest bounding physical coordinate box that encloses all text lines in the block (recording the coordinates of the top left and bottom right corners), and the page number information.

[0036] For each pair of adjacent basic text blocks, the semantic continuity of their text content needs to be evaluated. Pre-trained semantic encoding models (such as Sentence-BERT or a Siamese encoder fine-tuned based on bidding domain corpora) are used to extract the contextual semantic features of the two basic text blocks respectively. To obtain more robust semantic representations, the context can be appropriately augmented during extraction, i.e., the basic text blocks... The text and its immediate context (such as the previous base text block) and the next basic text block The text is concatenated to form the input sequence. Similarly, construct Then and The data are fed into the semantic encoder to obtain the corresponding semantic vectors. and The cosine similarity is used to measure the degree of association between two semantic vectors, and the semantic association degree is obtained as shown in the following formula:

[0037] in, For adjacent basic text blocks and The semantic relevance between them, the value range is The larger the value, the more semantically coherent the two basic text blocks are, and the more they should be merged into a single whole.

[0038] Physical distance on the page is another strong signal for determining whether basic text blocks belong to the same logical paragraph: text lines within the same paragraph are usually spaced closely and are fixed in spacing, while the vertical spacing between paragraphs is larger. This physical distance can be mapped using an exponential decay function. The correlation between location and distance within a range can reflect the cognitive pattern that "the closer the distance, the stronger the correlation, and the farther the distance, the faster the correlation weakens."

[0039] Basic text blocks of adjacent blocks and Take the bottom ordinate of the previous block's tail. The top y-coordinate of the next block header Calculate the physical distance between the vertical page and the physical page. (Units can be pixels or points), as shown in the following formula:

[0040] If the horizontal coordinate ranges of the two blocks overlap, use the direct vertical difference; if there is no overlap and it is a multi-column layout, the distance can be calculated within the same column first, and this distance is always measured within the same reading flow. When overlap or boundary calculation errors occur, take... .

[0041] The location distance correlation is defined as a natural constant. The exponential decay function with base is as follows:

[0042] in, Indicates two adjacent basic text blocks and The degree of association between items on the page due to their distance from each other, with a value range of [value missing]. A larger value indicates a closer distance and a stronger correlation. This is the distance attenuation coefficient, which controls the rate at which the correlation decreases with increasing distance. ; The larger the value, the more sensitive it is to distance; only very close blocks can achieve high correlation. The smaller the value, the more lenient the merging allowed; when hour, , indicating a perfect fit; when tending towards infinity It approaches 0.

[0043] Semantic relevance and location distance relevance are weighted according to a preset semantic weight. and spatial weights The weighted summation yields the fusion correlation strength between adjacent blocks, as shown in the following formula:

[0044] in, Indicates two adjacent basic text blocks and The degree of fusion correlation; , And let To maintain normalization.

[0045] The calculated fusion correlation strength is compared with the preset segmentation fusion threshold. Comparison: like Then it is believed and They should be merged, and no split point should be set between them; like This forms a dividing point. This will serve as the starting point for a new data text block.

[0046] Traverse the entire sequence of basic text blocks, concatenate all the basic blocks corresponding to consecutive "merge" states to obtain a data text block. Each data text block inherits the text of the basic blocks it contains and the physical coordinate frame of the merged block (the smallest bounding rectangle of the union of all basic text blocks), which serves as the direct input for subsequent sentence segmentation and index extraction.

[0047] For the four adjustable parameters: distance attenuation coefficient Semantic weight Spatial weights and segmentation fusion threshold To ensure the merged results align with the layout of the tender documents, a supervised parameter adjustment mechanism based on manual annotation can be adopted, including: A sample set of annotations is constructed by collecting a large number of adjacent basic text block pairs from historical bidding documents. Professional annotators determine whether these pairs should be merged or split based on semantic integrity and reasonable granularity, thus forming binary labels. The annotation follows the goal of "granularity balance" to avoid data text blocks that are too long (e.g., spanning multiple independent topics) or too short (containing only scattered phrases). The optimization objective is defined as the accuracy, recall, and harmonic mean F1 score of "merging / splitting" on the labeled set as evaluation indicators, with an emphasis on maximizing the F1 score to balance the accuracy of merging and splitting. Parameter search and cross-validation divide the labeled sample set into training and validation sets in the parameter space. A grid search is performed. For each parameter combination, the F1 score on the validation set is calculated, and the parameter combination that yields the highest F1 score is selected as the final configuration. The cross-validation fold count can be set to 5 folds to ensure generalization ability.

[0048] Step 2: After segmenting the data text blocks into sentences and assigning indices, the large language model is constrained by instructions to output only the start and end indices of the target key fields in the sentence list. Based on the start and end indices, the original text fragments are extracted. The key-value pair structured data is extracted from the table image fragments using a visual table recognition model. The parameter-value-unit triplet structured data is extracted from the drawing image fragments using a multimodal large language model.

[0049] In some embodiments, after segmenting the data text block into sentences and assigning indexes, the large language model is constrained by instructions to output only the start and end indices of the target key field in the sentence list, and the original text fragments are extracted from the original text based on the start and end indices, including: The data text block is segmented into sentences according to punctuation to obtain a list of sentences carrying physical coordinate frames, and each sentence is assigned a unique index number to generate a sentence sequence with index tags. Construct an instruction template by filling the instruction template with a list of indexed sentence sequences and target key fields. The instruction template constrains the output format of the large language model to be a preset structured data containing only target key field identifiers, start sentence index, and end sentence index, and sets a hard stop sequence to prevent the model from generating additional explanatory text. The instruction template is input into the large language model, and the preset structured data returned by the large language model is parsed to obtain the starting sentence index and ending sentence index corresponding to each target key field. Determine the corresponding target sentence based on the starting sentence index and the ending sentence index, obtain the physical coordinate frame of all target sentences, and calculate the minimum bounding rectangle as the physical cropping area of ​​the target key field on the page image; Extract a sub-image from the page image of the tender document that corresponds to the physical cropping area, and perform text recognition on the sub-image to obtain the original text cropping segment corresponding to the target key field.

[0050] Furthermore, in the specific implementation process, the detailed method for constructing and using the instruction template is as follows: First, the extraction context is dynamically populated based on a pre-defined instruction template framework. This instruction template employs a structured prompt design and comprises three parts: system role settings, indexed input text blocks, and output instructions. During implementation, the program iterates through all data text blocks generated in step one, segments each data text block into sentences according to punctuation marks such as periods and semicolons, assigns a globally unique numerical index to each sentence, and explicitly marks each sentence with a specific marker, generating a sequence of indexed sentences. Simultaneously, a list containing all target fields to be extracted is defined, such as ["Project Name", "Bid Deadline", "Bid Bond Amount"].

[0051] Next, output format constraints and anti-illusion design are implemented. The indexed sentence sequence and target key field list are filled into the reserved placeholders in the template. The instruction template explicitly constrains the output format of the large language model to be plain text containing only pre-defined structured data, requiring the model to "must and can only return a standard JSON (JavaScript Object Notation) array," where each element corresponds to a target field and contains three keys: "field identifier," "starting sentence index," and "ending sentence index." The instruction emphasizes "do not generate any explanations, greetings, or code block markers." To completely prevent the model from generating additional explanatory text from a technical perspective, a "hard stop sequence" is set using the specific generation control parameters provided by the large language model API (Application Programming Interface) when calling it. Specifically, the closing symbol "]" of the JSON array can be set as the stop sequence. Once the model has generated the index data of the last field and output "]", the generation process will be forcibly truncated, physically preventing the generation of subsequent free text.

[0052] Finally, extraction and parsing are performed. The completed instruction prompts are input into the large language model. The model follows the instructions, performing semantic matching between sentences. When it determines that several consecutive sentences jointly express "project name," it outputs the start and end indices of that sentence string. The program intercepts the string returned by the model. Since its output follows a JSON array format and has no extra characters, it can be directly deserialized using a JSON parser. After parsing, the array elements are traversed to obtain the identifier of each target key field and its corresponding start and end sentence indices, completing the entire process from instruction construction to structured index output.

[0053] In some embodiments, the extraction of key-value pair structured data from table image slices using a visual table recognition model includes: The table image is segmented and input into a pre-trained table recognition model based on the visual Transformer architecture. The table's row and column separators are detected, the bounding boxes of all cells are determined, and text recognition is performed on the image regions within each bounding box to obtain a set of cell text containing row and column coordinates. Based on the row and column coordinates, the header area and data area are identified. The text of the header cell is used as the key, and the text of the corresponding data cell in the same row or column is used as the value. The data is then assembled into key-value pairs row by row or column by column. The chapter and page number metadata carried by the table image segments is appended as metadata to the key-value pairs to obtain key-value pair structured data with chapter and page number metadata.

[0054] In some embodiments, the extraction of parameter-value-unit triplet structured data from drawing image segments using a multimodal large language model includes: The drawing images are segmented and input into a multimodal large language model to recognize the technical parameter annotations and engineering symbol text in the drawings, and output the drawing text recognition results; Based on a preset data cleaning template that matches the technical parameter annotation format of the bidding drawings, the text recognition results of the drawings are parsed to extract the parameter item names, values ​​and units of measurement; The extracted parameter item name, value, and unit of measurement are constructed into a parameter item-value-unit triple, and the chapter and page number metadata carried by the drawing image fragments are appended to the triple to obtain structured drawing data with chapter and page number metadata.

[0055] Step 3: Using the chapter information carried by the output data of each model as physical anchors, classify the output data of each model into the corresponding candidate data clusters for semantic alignment and intent clustering, and construct a global knowledge graph of bidding that includes nodes, attributes and directed edges. The directed edges are used to represent the constraints, mutual exclusion, time subordination and equivalence relationships between nodes.

[0056] In some embodiments, the specific process of constructing a global knowledge graph for bidding that includes nodes, attributes, and directed edges is as follows: Based on the chapter and page metadata attached to the original text clippings, the key-value pair structured data, and the parameter-value-unit triplet structured data, the extraction results of different modalities belonging to the same chapter are aggregated into the same candidate data cluster; Semantic vector encoding is performed on each data item within the candidate data cluster. The semantic similarity between data items is calculated. Different data items with semantic similarity higher than a preset entity alignment threshold are identified as pointing to the same business entity and merged into the same entity node. The entity alignment threshold can be determined in the following way: A large number of data item pairs, consisting of text extraction fragments, table key-value pairs, and drawing triples, were collected from the candidate data clusters of historical bidding documents. Domain experts then performed binary classification labeling ("should be merged / should not be merged") based on the identity of business entities, constructing an aligned label set. Next, for each data item, a semantic encoding model fine-tuned with bidding domain corpus (such as Sentence-BERT) was used to extract the semantic vector of its entity name and core attribute concatenated text, and the cosine similarity of the two vectors was calculated as the semantic similarity. Based on the label, an alignment threshold was set... A grid search is performed within the value space (e.g., [0.5, 0.95]) to calculate the entity alignment precision, recall, and F1 score at each threshold. Five-fold cross-validation is used to select the threshold that maximizes the average F1 score as the preset entity alignment threshold. In actual alignment determination, if the semantic similarity between any two data items from different modal sources within a candidate data cluster is higher than the threshold, they are identified as pointing to the same business entity and merged into the same entity node; otherwise, they remain as independent entities. This achieves a balance between high recall and high precision, ensuring the reliability and consistency of multimodal information alignment.

[0057] The field values ​​corresponding to each merged data item are assigned as attributes of the entity node; connectors representing logical relationships between entities are identified from the original text context of each data item; according to preset logical relationship mapping rules, the identified connectors are transformed into constraint, mutual exclusion, time dependency, or equivalence relationship types, and directed edges are established between the corresponding entity nodes along the transformed relationship direction to form the global knowledge graph of the bidding process. The specific implementation of the preset logical relationship mapping rules is as follows: A pre-built dictionary mapping logical connectors to relation types is constructed. This dictionary is compiled through analysis of typical logical sentences in a large number of bidding documents and manual summarization based on the experience of domain experts. It specifically includes four categories of mapping entries: The first category is constraint relation connectors, such as "should satisfy," "must comply with," "must not be lower than," and "refer to…execute." When such connectors are identified from the context of an entity, the entity before the connector is taken as the constraint subject, and the entity after the connector is taken as the constrained object, establishing a directed edge from the subject to the object; the second category is mutual exclusion relation connectors, such as "cannot be simultaneously," "any…", etc. The first category consists of conjunctions such as "choose one," "cannot be used together with other conjunctions based on," and "mutually exclusive." After identification, a bidirectional mutually exclusive edge is established between the two entities associated with the conjunctions. The second category consists of conjunctions related to time dependence, such as "before," "after," "must be completed before," and "effective from," etc. The time reference entity is used as the parent node, and the time dependent entity is used as the child node. A directed edge is established from the parent node to the child node to represent the time order dependency. The third category consists of conjunctions related to equivalence, such as "equivalent to," "that is," "also called," "consistent with," and "see the same," etc. After identification, a bidirectional equivalence edge is established between the two entities. When applying this mapping rule, for each entity node's corresponding original text context (i.e., the source sentence of the original text fragment, table key-value pair, or drawing triple), firstly, dependency parsing or shallow semantic analysis is used to extract verb phrases, prepositional structures, and conjunctions near the entity mentions. Then, a positive maximum matching is performed with the connectors in the mapping dictionary. When a match is successful, the starting and ending points of the directed edges are determined according to the direction determination rules of the dictionary entries (such as the "nsubj" (nominal subject) and "dobj" (direct object) relationship based on dependency arcs or fixed preceding and following entity positions). Finally, the relationship type and direction are assigned to the corresponding edges, completing the transformation from natural language logical expression to structured relationships in a knowledge graph. This mapping rule library supports continuous iteration and can be supplemented with new connectors and relationship types based on application feedback to cover more complex logical expressions in bidding documents.

[0058] Step 4: Based on the global knowledge graph of the bidding process, detect data conflicts between cross-modal nodes. For node pairs that conflict and point to the same business entity, compare the confidence levels of the two nodes in the pair. Set the node with the higher confidence level as the primary verification information node and the node with the lower confidence level as the supporting information node. The two form an information anchor pair. Calculate the association strength between the nodes in the information anchor pair. Link anchor pairs with association strength greater than the strength threshold into an information anchor pair verification chain. Along the verification chain, use the primary verification information node to verify and correct the supporting information nodes, and output the key element extraction results without conflicts.

[0059] In some embodiments, comparing the confidence levels of the two nodes in the node pair includes: Obtain the original extracted data corresponding to each node in the bidding global knowledge graph in the conflict node pair, and determine the model type used to generate the original extracted data; the original extracted data is the original text fragment with chapter and page number metadata, key-value pair structured data, or parameter-value-unit triplet structured data; the model type includes large language model, visual table recognition model, and multimodal large language model; when the model type is large language model, it means that the node information comes from the original text fragment, so the sequence generation probability is extracted from the start and end index generation process of the large language model as the basic confidence; when the model type is visual table recognition model, it means that the node information comes from key-value pair structured data, so the character recognition confidence is extracted from the cell text recognition result of the visual table recognition model as the basic confidence. The base confidence score is calculated as follows: When the model type is a multimodal large language model, it indicates that the information of the node comes from the parameter-value-unit triplet structured data. The token generation probability is extracted during the output of the parameter-value-unit triplet structured data from the multimodal large language model and used as the base confidence score. Based on the information source of the node, a corresponding modal weight factor is assigned. The weight factor from the parameter-value-unit triplet structured data is greater than the modal weight factor from the key-value pair structured data, and the modal weight factor from the key-value pair structured data is greater than the modal weight factor from the original text fragment. The base confidence score of each node is multiplied by its corresponding modal weight factor to obtain the node's confidence score. The confidence scores of two nodes in a node pair are compared to determine the relative confidence levels of the two nodes.

[0060] Furthermore, in the specific implementation, firstly, for each node in a pair of conflicting nodes, the original extracted data corresponding to it in the knowledge graph is traced back to determine the model type that generated the data, and the basic confidence level is extracted from it. .

[0061] To unify the expression of the basic confidence scores of different model extraction results, the three cases can be merged into a general piecewise function formula. By introducing a model type indicator variable, a unified description of generative models (large language models, multimodal large language models) and recognition models (visual table recognition) can be achieved, as shown in the following formula:

[0062] in, For the first in the output sequence The probability of generating each token (given by the softmax probability or log probability output by the model); This represents the total number of tokens in the output sequence. For the first character in the recognition result string The confidence score for character recognition (output by the OCR engine, with a value range of [value missing]). ); The total number of characters that make up the text of a complete key-value pair cell; To indicate the judgment value, When the data comes from a large language model or a multimodal large language model, take... When taking from the visual table recognition model For large language models (text source), the output sequence is the start and end index sequence of the target field (e.g., ...). The input is an indexed sentence sequence; for a multimodal large language model (drawing source), the output sequence is a text sequence of parameter-value-unit triples (e.g., "rated voltage, 220, V"), and the input is a drawing image slice.

[0063] Based on experience with the reliability of modal information in tender documents, the following criteria are established: technical parameters in drawings, being precise values ​​approved by the design team, have the highest reliability; tabular data, being structured quantitative indicators, have the next highest reliability; and plain text descriptions, potentially introducing biases due to generalizations or typos, have the lowest reliability. Therefore, preset modal weighting factors are assigned to these three sources. ,satisfy: The specific values ​​can be set based on the experience of domain experts, for example, taking: , , and These are modal weighting factors corresponding to original text fragments, key-value pair structured data, and parameter-value-unit triplet structured data, respectively. Each weighting factor is stored in a configuration file or as a global constant for easy adjustment.

[0064] For any node in the conflict node pair The final confidence score of a node is obtained by multiplying its base confidence score by the corresponding modality weight factor, as shown in the following formula:

[0065] in, This represents the final confidence level of the node; a higher value indicates that the attribute value carried by the node is more reliable. This represents the baseline confidence level of the original extracted data corresponding to this node. The modal weighting factor is assigned based on the source of node information. .

[0066] Calculate the middle node of the conflict node and nodes After determining the confidence level, compare the numerical values: like , then it represents a node The confidence level is relatively high, and the nodes As the primary verification information node, the node As supporting information nodes; If they are equal, further determination is made based on the modal weight factor. If they are still equal, both are marked for manual adjudication.

[0067] By implementing the above methods, the original probability output by the model is effectively combined with modal prior knowledge, so that the confidence level can reflect both the uncertainty of the model itself and the inherent credibility differences of different information carriers, providing a quantitative basis for the primary and secondary determination in subsequent conflict resolution.

[0068] In some embodiments, calculating the association strength between nodes in an information anchor pair and linking anchor pairs with association strength greater than a strength threshold into an information anchor pair verification chain includes: Extract the attribute name sets of the two nodes in the information anchor pair in the global knowledge graph of the bidding, and calculate the Jaccard similarity coefficient of the two attribute name sets; Using a pre-trained semantic coding model, semantic vectors are encoded for the entity names of the two nodes, and the cosine similarity between the two semantic vectors is calculated. The calculated Jaccard similarity coefficient and cosine similarity are weighted and summed according to preset weights to obtain the association strength between nodes; Based on manually labeled associated node pairs, the strength threshold is determined by grid search with the goal of maximizing the F1 (F1 score) value of the association strength judgment. Information anchor pairs with a correlation strength greater than the strength threshold are sequentially linked according to their shared node relationship to form an information anchor pair verification chain.

[0069] Based on the same inventive concept, and corresponding to any of the above embodiments, refer to... Figure 2 This invention provides a multimodal tender document key element extraction system for implementing the aforementioned multimodal tender document key element extraction method, comprising: The multimodal layout analysis module is used to perform multimodal layout analysis on tender documents, identify text areas, table areas, and drawing / image areas, as well as their physical coordinates and chapter / page number metadata, and merge text areas into data text blocks based on the weighted fusion association strength of the semantic association degree and positional distance association degree of adjacent basic text blocks, and crop table areas and drawing / image areas into table image fragments and drawing image fragments with chapter / page number metadata, respectively; The multimodal key element extraction module, connected to the multimodal layout analysis module, is used to segment the data text block into sentences and assign indexes. Then, through instruction constraints, the large language model outputs only the start and end indices of the target key fields in the sentence list and extracts the original text fragments accordingly. The visual table recognition model extracts key-value pair structured data from the table image segments, and the multimodal large language model extracts parameter-value-unit triplet structured data from the drawing image segments. The knowledge graph construction module, connected to the multimodal key element extraction module, is used to use the chapter information carried by the output data of each model as physical anchors to classify the original text fragments, key-value pair structured data and parameter item-value-unit triplet structured data into corresponding candidate data clusters for semantic alignment and intent clustering, and to construct a global knowledge graph for bidding that includes nodes, attributes and directed edges. The directed edges are used to represent the constraints, mutual exclusion, temporal dependence and equivalence relationships between nodes. The conflict detection and correction module, connected to the knowledge graph construction module, is used to detect data conflicts between cross-modal nodes based on the global knowledge graph of the bidding process. It compares the confidence levels of node pairs that conflict and point to the same business entity, designating nodes with relatively higher confidence levels as primary verification information nodes and nodes with relatively lower confidence levels as supporting information nodes to form information anchor pairs. It calculates the association strength between nodes in the information anchor pairs and links anchor pairs with association strength greater than a strength threshold into an information anchor pair verification chain. Along the verification chain, the supporting information nodes are verified and corrected using the primary verification information node, and the result of extracting conflict-free key elements is output.

[0070] Based on the same inventive concept, corresponding to any of the above embodiments, the present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform a method for extracting key elements of a multimodal tender document according to an embodiment.

[0071] Alternatively, the aforementioned electronic device may be a server.

[0072] In addition, this embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method for extracting key elements of a multimodal tender document according to the embodiment.

[0073] It is understood that the processor in the embodiments of the present invention may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0074] The method steps in the embodiments of the present invention can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0075] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a storage medium or transmitted through a storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive (SSD)).

Claims

1. A method for extracting key elements from multimodal tender documents, characterized in that, Includes the following steps: Multimodal layout analysis is performed on the tender documents to identify text areas, table areas, and drawing / image areas, as well as their physical coordinates and chapter / page number metadata. For the text areas, the text blocks are merged based on the weighted fusion association strength of the semantic association degree and the positional distance association degree of adjacent basic text blocks to generate data text blocks. The table areas and drawing / image areas are respectively cropped into table image fragments and drawing image fragments with chapter / page number metadata. After segmenting the data text into sentences and assigning indices, the large language model is constrained by instructions to output only the start and end indices of the target key fields in the sentence list. Based on the start and end indices, the original text fragments are extracted. The key-value pair structured data is extracted from the table image fragments through the visual table recognition model. The parameter-value-unit triplet structured data is extracted from the drawing image fragments through the multimodal large language model. Using the chapter information carried in the output data of each model as physical anchors, the output data of each model is categorized into corresponding candidate data clusters for semantic alignment and intent clustering, constructing a global knowledge graph of bidding that includes nodes, attributes, and directed edges. The directed edges are used to represent constraints, mutual exclusion, temporal dependence, and equivalence relationships between nodes. The specific process of constructing the global knowledge graph of bidding that includes nodes, attributes, and directed edges is as follows: Based on the chapter and page metadata attached to the original text clippings, the key-value pair structured data, and the parameter-value-unit triplet structured data, the extraction results of different modalities belonging to the same chapter are aggregated into the same candidate data cluster; semantic vector encoding is performed on each data item in the candidate data cluster, the semantic similarity between data items is calculated, and different data items with semantic similarity higher than the preset entity alignment threshold are identified as pointing to the same business entity and merged into the same entity node; the field values ​​corresponding to each merged data item are assigned as attributes of the entity node; the connecting words representing the logical relationship between entities are identified from the original text context of each data item, and according to the preset logical relationship mapping rules, the identified connecting words are transformed into constraint, mutual exclusion, time subordination, or equivalence relationship types, and directed edges are established between the corresponding entity nodes along the transformed relationship direction to form the bidding global knowledge graph; Based on the global knowledge graph of the bidding process, data conflicts between cross-modal nodes are detected. For node pairs that conflict and point to the same business entity, the confidence levels of the two nodes in the pair are compared. The node with the higher confidence level is set as the primary verification information node, and the node with the lower confidence level is set as the supporting information node. The two form an information anchor pair. The association strength between the nodes in the information anchor pair is calculated, and anchor pairs with association strength greater than the strength threshold are linked into an information anchor pair verification chain. Along the verification chain, the supporting information node is verified and corrected by the primary verification information node, and the key element extraction results without conflicts are output. The calculation of the association strength between nodes in the information anchor pair, and linking anchor pairs with association strength greater than a strength threshold into an information anchor pair verification chain, includes: Extract the attribute name sets of the two nodes in the information anchor pair in the global knowledge graph of the bidding, and calculate the Jaccard similarity coefficient of the two attribute name sets; Using a pre-trained semantic coding model, the entity names of two nodes are semantically encoded, and the cosine similarity of the two semantic vectors is calculated. The calculated Jaccard similarity coefficient and cosine similarity are weighted and summed according to preset weights to obtain the association strength between nodes. Based on manually labeled associated node pairs, the strength threshold is determined by grid search with the goal of maximizing the F1 value of association strength judgment. Information anchor pairs with association strength greater than the strength threshold are sequentially concatenated according to shared node relationships to form an information anchor pair verification chain.

2. The method for extracting key elements of multimodal tender documents as described in claim 1, characterized in that, The specific process of performing multimodal layout analysis on the tender documents is as follows: After obtaining the original tender documents and converting them into page images, the content modality of the page is identified page by page using a multimodal visual model. Plain text paragraphs are marked as text modal areas, areas containing row and column structures are marked as table modal areas, and areas containing graphics or engineering symbols are marked as image modal areas. During the recognition process, sub-images corresponding to each modal region are extracted, the physical coordinate frame of each modal region on the original page is recorded, and according to the pre-extracted directory structure or page number mapping rules, chapter page number metadata containing the chapter title and the page number is added to each modal region. The identifiers, modality types, physical coordinate frames, screenshot save paths, and chapter / page number metadata of each modal region are encapsulated into a standardized multimodal fragmentation information set and stored in a structured manner.

3. The method for extracting key elements of multimodal tender documents as described in claim 1, characterized in that, The weighted fusion of association strength based on the semantic relevance and positional distance relevance of adjacent basic text blocks includes: The text to be processed is divided into multiple basic text blocks according to a preset physical length; For adjacent basic text blocks, pre-trained semantic coding models are used to extract contextual semantic features, generate corresponding high-dimensional semantic vectors, and calculate the cosine similarity between the two high-dimensional semantic vectors as the semantic relevance. Based on the physical coordinates of adjacent basic text blocks on the tender document page, determine the physical distance between the end of the previous basic text block and the beginning of the next basic text block, using a natural constant. The positional distance correlation is calculated using the product of a preset distance attenuation coefficient and the physical distance to the page as the base and the exponent as the exponent. The semantic correlation degree and the location distance correlation degree are weighted and summed according to preset semantic weights and spatial weights to obtain the weighted fusion correlation strength; when the weighted fusion correlation strength is greater than the preset segmentation fusion threshold, adjacent basic text blocks are merged into the same data text block; otherwise, a segmentation breakpoint is formed at this point. The distance decay coefficient, semantic weight, spatial weight, and segmentation fusion threshold are determined by cross-validation and joint parameter tuning, with the semantic integrity and granularity balance of the segmentation results as the optimization objectives, based on the manually annotated text block merging and segmentation boundaries.

4. The method for extracting key elements of multimodal tender documents as described in claim 1, characterized in that, After segmenting the data text block into sentences and assigning indices, the large language model is constrained by instructions to output only the start and end indices of the target key fields in the sentence list. Based on these start and end indices, fragments of the original text are extracted, including: The data text block is segmented into sentences according to punctuation to obtain a list of sentences carrying physical coordinate frames, and each sentence is assigned a unique index number to generate a sentence sequence with index tags. Construct an instruction template by filling the instruction template with a list of indexed sentence sequences and target key fields. The instruction template constrains the output format of the large language model to be a preset structured data containing only target key field identifiers, start sentence index, and end sentence index, and sets a hard stop sequence to prevent the model from generating additional explanatory text. The instruction template is input into the large language model, and the preset structured data returned by the large language model is parsed to obtain the starting sentence index and ending sentence index corresponding to each target key field. Determine the corresponding target sentence based on the starting sentence index and the ending sentence index, obtain the physical coordinate frame of all target sentences, and calculate the minimum bounding rectangle as the physical cropping area of ​​the target key field on the page image; Extract a sub-image from the page image of the tender document that corresponds to the physical cropping area, and perform text recognition on the sub-image to obtain the original text cropping segment corresponding to the target key field.

5. The method for extracting key elements of multimodal tender documents as described in claim 1, characterized in that, The extraction of key-value pair structured data from table image slices using a visual table recognition model includes: The table image is segmented and input into a pre-trained table recognition model based on the visual Transformer architecture. The table's row and column separators are detected, the bounding boxes of all cells are determined, and text recognition is performed on the image regions within each bounding box to obtain a set of cell text containing row and column coordinates. Based on the row and column coordinates, the header area and data area are identified. The text of the header cell is used as the key, and the text of the corresponding data cell in the same row or column is used as the value. The data is then assembled into key-value pairs row by row or column by column. The chapter and page number metadata carried by the table image segments is appended as metadata to the key-value pairs to obtain key-value pair structured data with chapter and page number metadata.

6. The method for extracting key elements of multimodal tender documents as described in claim 1, characterized in that, The extraction of parameter-value-unit triplet structured data from drawing image segments using a multimodal large language model includes: The drawing images are segmented and input into a multimodal large language model to recognize the technical parameter annotations and engineering symbol text in the drawings, and output the drawing text recognition results; Based on a preset data cleaning template that matches the technical parameter annotation format of the bidding drawings, the text recognition results of the drawings are parsed to extract the parameter item names, values ​​and units of measurement; The extracted parameter item name, value, and unit of measurement are constructed into a parameter item-value-unit triple, and the chapter and page number metadata carried by the drawing image fragments are appended to the triple to obtain structured drawing data with chapter and page number metadata.

7. The method for extracting key elements of multimodal tender documents as described in claim 1, characterized in that, The comparison of the confidence levels of the two nodes in the node pair includes: Obtain the original extracted data corresponding to each node in the conflict node pair in the global knowledge graph of the bidding, and determine the model type used to generate the original extracted data; the original extracted data is the original text fragment with chapter and page number metadata, key-value pair structured data, or parameter-value-unit triplet structured data; the model type includes large language model, visual table recognition model, and multimodal large language model; When the model type is a large language model, the information of the node comes from the original text segment. Therefore, the sequence generation probability is extracted from the process of generating the start and end index of the large language model as the basic confidence. When the model type is a visual table recognition model, it means that the information of the node comes from key-value pair structured data. Then, the character recognition confidence is extracted from the cell text recognition results of the visual table recognition model as the base confidence. When the model type is a multimodal large language model, the information representing the node comes from the parameter-value-unit triplet structured data. Therefore, the token generation probability is extracted from the process of outputting the parameter-value-unit triplet structured data from the multimodal large language model as the basic confidence level. Based on the information source of the node, a corresponding modal weight factor is assigned. Among them, the weight factor of the parameter-value-unit triplet structured data is greater than the modal weight factor of the key-value pair structured data, and the modal weight factor of the key-value pair structured data is greater than the modal weight factor of the original text fragment. The confidence level of a node is obtained by multiplying the base confidence level of each node by the corresponding modal weight factor. Compare the confidence levels of the two nodes in a node pair to determine the relative confidence levels of the two nodes.

8. A system for extracting key elements of a multimodal tender document, used to implement the method for extracting key elements of a multimodal tender document as described in any one of claims 1-7, characterized in that, include: The multimodal layout analysis module is used to perform multimodal layout analysis on tender documents, identify text areas, table areas, and drawing / image areas, as well as their physical coordinates and chapter / page number metadata, and merge text areas into data text blocks based on the weighted fusion association strength of the semantic association degree and positional distance association degree of adjacent basic text blocks, and crop table areas and drawing / image areas into table image fragments and drawing image fragments with chapter / page number metadata, respectively; The multimodal key element extraction module, connected to the multimodal layout analysis module, is used to segment the data text block into sentences and assign indexes. Then, through instruction constraints, the large language model outputs only the start and end indices of the target key fields in the sentence list and extracts the original text fragments accordingly. The visual table recognition model extracts key-value pair structured data from the table image segments, and the multimodal large language model extracts parameter-value-unit triplet structured data from the drawing image segments. The knowledge graph construction module, connected to the multimodal key element extraction module, is used to use the chapter information carried by the output data of each model as physical anchors to classify the original text fragments, key-value pair structured data and parameter item-value-unit triplet structured data into corresponding candidate data clusters for semantic alignment and intent clustering, and to construct a global knowledge graph for bidding that includes nodes, attributes and directed edges. The directed edges are used to represent the constraints, mutual exclusion, temporal dependence and equivalence relationships between nodes. The conflict detection and correction module, connected to the knowledge graph construction module, is used to detect data conflicts between cross-modal nodes based on the global knowledge graph of the bidding process. It compares the confidence levels of node pairs that conflict and point to the same business entity, designating nodes with relatively higher confidence levels as primary verification information nodes and nodes with relatively lower confidence levels as supporting information nodes to form information anchor pairs. It calculates the association strength between nodes in the information anchor pairs and links anchor pairs with association strength greater than a strength threshold into an information anchor pair verification chain. Along the verification chain, the supporting information nodes are verified and corrected using the primary verification information node, and the result of extracting conflict-free key elements is output.

Images