A cross-modal collaborative analysis method based on a multi-modal bidding document

Abstract

A kind of cross-modal collaborative analysis method based on multi-modal tender document, including the following stages: data analysis stage, the multi-modal tender document is converted into picture format, then the content type is identified and is cut to obtain fragment information;The table fragment information is handled to obtain structured table data, the picture fragment information is handled to obtain structured picture data;Then the text fragment information, the table fragment information and the picture fragment information are associated, to obtain integrated multi-modal data;Target item extraction stage, extract target item in the tender document;Consistency check stage, generate structured analysis result.The present application realizes the automatic analysis, association of multi-modal information of text, table and drawing in tender document, and consistency check conflict detection work, solves the problem of traditional manual analysis of text, table and picture multi-modal information fragmentation, low efficiency, easy to make mistake.

Description

Technical Field

[0001] This invention relates to the field of multimodal information processing technology at the intersection of natural language processing (NLP) and computer vision (CV), and particularly to a cross-modal collaborative parsing method based on multimodal tender documents. Background Technology

[0002] Currently, the parsing of tender documents mainly relies on two methods: one is purely manual page-by-page verification, which requires processing text descriptions, table parameters, and drawing technical requirements separately. This is time-consuming and prone to parsing errors due to "conflicts between text and table data" and "omissions of drawing details." The other method relies on single-modal parsing tools, which can only extract text keywords or table structured data, ignoring information such as technical parameters and structural components in the drawings. Technical parameters include equipment dimensions and cable cross-section requirements. Furthermore, it cannot verify the consistency of multimodal information, such as the inability to identify conflicts between the text "construction period of 90 days" and the table "total duration of construction nodes of 120 days."

[0003] Existing technologies have three major pain points: 1. Multimodal data is scattered throughout the document, including text, tables, and drawings. The tables and drawings corresponding to the text are not linked together, resulting in fragmented information; 2. There is no collaborative mechanism between the Natural Language Model (LLM) and the visual model, making it impossible to use visual data to supplement text parsing; 3. There is a lack of multimodal information consistency verification rules, making it impossible to automatically determine data conflicts, resulting in low reliability of verification results.

[0004] Therefore, it is essential to provide a cross-modal collaborative parsing method based on multimodal bidding documents to address the shortcomings of existing technologies. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a cross-modal collaborative parsing method for multimodal bidding documents. This method automatically parses and associates multimodal information (text, tables, and drawings) in bidding documents, and performs consistency checks and conflict detection. It solves the problems of fragmented, inefficient, and error-prone multimodal information in traditional manual parsing, and is suitable for intelligent review and information extraction of bidding documents in scenarios such as bidding agencies, corporate bidding departments, and government procurement platforms.

[0006] The above-mentioned objectives of the present invention are achieved through the following technical measures: A cross-modal collaborative parsing method based on multimodal bidding documents is provided, comprising the following stages: The data analysis phase involves converting the multimodal tender documents into image format, identifying content types, and cropping them to obtain segmented information, including text segmentation, table segmentation, and image segmentation. The table segmentation is processed to obtain structured table data, and the image segmentation is processed to obtain structured image data. Finally, the text segmentation, table segmentation, and image segmentation are correlated to obtain integrated multimodal data. Target item extraction stage—Extracting target items from the tender documents; Consistency verification phase – Based on the pre-built bidding conflict verification rule base, the target item and the integrated multimodal data are analyzed through LLM fusion to perform consistency verification and generate structured parsing results.

[0007] Preferably, the above data analysis phase is performed by the following steps: S1. Convert the tender document into an image format, then identify and crop it to obtain segmentation information. The segmentation information includes text segmentation information, table segmentation information, and image segmentation information. Proceed to S2. S2. Input the table segmentation information into the table processing channel to obtain structured table data, and write the structured table data back to the content of the table segmentation information in S1 for later use; input the image segmentation information into the image processing channel to obtain structured image data, and write the structured image data back to the content of the image segmentation information in S1 for later use. S3. The text fragment information obtained in S1, the structured table data and the structured image data obtained in S2 are associated through LLM to obtain integrated multimodal data.

[0008] Preferably, S1 is performed by the following steps: S1.1 When the tender document is a Word document, first use the docx2pdf library to convert the Word document to PDF format, and then use the pdf2image library to convert the PDF format to image format; when the tender document is a PDF document, directly use the pdf2image library to convert the PDF format to image format. S1.2. Traverse all pages of the image format from top to bottom. For each page of the image format in S1.1, use the PaddleOCR-VL multimodal model to identify the document content type. Mark documents with plain text paragraphs as text data, documents with row and column structures as table data, and documents with at least one of graphics or engineering symbols as image data. Then, use an image cropping tool to extract the text data, table data, and image data from the document to obtain fragment information that records corresponding metadata, where the metadata is chapter and page number. S1.3 Save the sharding information obtained in S1.2 in JSON format to info.json and proceed to S2.

[0009] Preferably, the above JSON format includes an identifier (Id), a type (Type), a content field (Content), an image screenshot field (Img_path), a chapter field (chapter), and an image page number (Page).

[0010] Preferably, the above-mentioned table processing channel processes the table fragmentation information through the following steps: A1. Detect the table row and column boundaries in the table segmentation information using the improved visual table recognition model Table Transformer, and proceed to A2; A2. Extract the contents of cell A2 and move them to cell A3; A3. Convert the cell content into key-value structured data, while retaining the page number (Page) and chapter field of the image containing the table, to obtain structured table data.

[0011] Preferably, the above-mentioned image processing channel processes the image segmentation information through the following steps: B1. Use Deepseek-OCR multimodal large model to identify the technical parameter annotations in the image segmentation information, then extract the text content in the technical parameter annotations, and proceed to B2; B2. Convert the data into structured data according to the format of "Technical Parameter Item-Value-Unit", and associate it with the corresponding technical section of the image to obtain structured image data.

[0012] Preferably, S3 above includes the following steps: S3.1. Using the chapter field of all segmented information as the anchor point, group the text data, image data and table data under the same chapter field into the same group, and then perform semantic analysis, proceeding to S3.2. S3.2. Use the Qwen3-32B large model to perform text clustering on the same group, obtain the classification results, and proceed to S3.3; S3.3 Based on the classification results of S3.2, the text data, image data, and table data are associated to obtain integrated multimodal data.

[0013] Preferably, the above consistency verification phase is performed by the following steps: C1. Input the bidding conflict verification rule base, the target item, and the integrated multimodal data into the LLM; C2. Based on the bidding conflict verification rule base and in conjunction with the target item, review the integrated multimodal data to generate a structured parsing result, which includes the conflict content, conflict location, and judgment criteria.

[0014] Preferably, the target item extraction stage above uses the Qwen3-32B large model to extract the target items from the tender document.

[0015] Preferably, the above-mentioned target items are at least one of the following: evaluation criteria, construction period requirements, and technical parameter requirements.

[0016] Preferably, the aforementioned bidding conflict verification rule base stores multiple conflict rules, and the conflict rules are at least one of the following: text data and table data conflict rules, text data and image data conflict rules, or table data and image data conflict rules.

[0017] This invention discloses a cross-modal collaborative parsing method based on multimodal tender documents, comprising the following stages: a data analysis stage, converting the multimodal tender documents into image format, then identifying content types and cropping them to obtain segmented information, wherein the segmented information includes text segmented information, table segmented information, and image segmented information; processing the table segmented information to obtain structured table data, and processing the image segmented information to obtain structured image data; then associating the text segmented information, table segmented information, and image segmented information to obtain integrated multimodal data; a target item extraction stage, extracting target items from the tender documents; and a consistency verification stage, performing consistency verification based on a pre-built tender conflict verification rule base, the target items, and the integrated multimodal data through LLM fusion analysis, thereby generating structured parsing results. This cross-modal collaborative parsing method based on multimodal tender documents establishes for the first time a collaborative mechanism between LLM and visual models, associating text with corresponding table and drawing content to achieve cross-modal data association and binding; moreover, this invention automatically verifies the consistency of multimodal information, outputs verification results, and locates conflict positions. This invention enables automatic parsing and association of multimodal information in tender documents, including text, tables, and drawings, as well as consistency verification and conflict detection. It solves the problems of fragmented, inefficient, and error-prone multimodal information in traditional manual parsing, and is suitable for intelligent review and information extraction of tender documents in scenarios such as tendering agencies, corporate bidding departments, and government procurement platforms. Attached Figure Description

[0018] The invention will be further described with reference to the accompanying drawings, but the contents of the drawings do not constitute any limitation on the invention.

[0019] Figure 1 This is a flowchart of S1 in the cross-modal collaborative parsing method for multimodal bidding documents based on the present invention.

[0020] Figure 2 This is a flowchart for processing channels in table-type processing.

[0021] Figure 3 This is a flowchart for image processing channel throughput.

[0022] Figure 4 This is a flowchart of S3 in the cross-modal collaborative parsing method for multimodal bidding documents based on the present invention.

[0023] Figure 5 This is a flowchart of the consistency verification stage in the cross-modal collaborative parsing method for multimodal bidding documents based on the present invention.

[0024] Figure 6 This is a screenshot of the bidding documents for power equipment in Example 2.

[0025] Figure 7 This is a screenshot of the power equipment tender document converted into image format in Example 2.

[0026] Figure 8 This is a screenshot of the fragmentation information in Example 2.

[0027] Figure 9 This is a screenshot of the table input in the table-type processing channel in Example 2.

[0028] Figure 10 This is a screenshot of the image input to the image processing channel in Example 2. Detailed Implementation

[0029] The technical solution of the present invention will be further described in conjunction with the following embodiments.

[0030] Example 1 A cross-modal collaborative parsing method based on multimodal bidding documents includes the following stages: The data analysis phase involves converting the multimodal tender documents into image format, identifying content types, and cropping them to obtain segmented information, including text segmentation, table segmentation, and image segmentation. The table segmentation is processed to obtain structured table data, and the image segmentation is processed to obtain structured image data. Finally, the text segmentation, table segmentation, and image segmentation are correlated to obtain integrated multimodal data. Target item extraction stage—Extracting target items from the tender documents; Consistency verification phase – Based on the pre-built bidding conflict verification rule base, the target item and the integrated multimodal data are analyzed through LLM fusion to perform consistency verification and generate structured parsing results.

[0031] In the target item extraction stage of this invention, the Qwen3-32B large model is used to extract the target items from the bidding documents. The target items are at least one of the following: evaluation criteria, schedule requirements, and technical parameter requirements.

[0032] The specific code for the target item extraction stage is as follows: Input: Document content Output: Target Item Large model prompt: "You are an excellent information extraction expert. Help me extract data on 'evaluation criteria,' 'construction period requirements,' and 'technical parameter requirements' from the input document content. Note that each criterion or requirement is presented in a list format."

[0033] #Enter content Document content #Output Format { "Evaluation Criteria":[], "Schedule Requirements":[], Technical specifications: [] } " The bidding conflict verification rule base stores multiple conflict rules, which are at least one of the following: text data and table data conflict rules, text data and image data conflict rules, or table data and image data conflict rules. For example, a text data and table data conflict rule would be determined as follows: the text "construction period 90 days" is determined to be conflicted with the table "total construction period 120 days"; a text data and image data conflict rule would be determined as follows: the text "cable cross-section ≥ 100mm²" is determined to be conflicted with the image label "YJV-4×95+1×50"; a table data and image data conflict rule would be determined as follows: the table "equipment power 5kW" is determined to be conflicted with the image "equipment parameters 4.5kW".

[0034] The data analysis phase consists of the following steps: S1. Convert the tender document into an image format, then identify and crop it to obtain segmentation information. The segmentation information includes text segmentation information, table segmentation information, and image segmentation information. Proceed to S2. S2. Input the table segmentation information into the table processing channel to obtain structured table data, and write the structured table data back to the content of the table segmentation information in S1 for later use; input the image segmentation information into the image processing channel to obtain structured image data, and write the structured image data back to the content of the image segmentation information in S1 for later use. S3. The text fragment information obtained in S1, the structured table data and the structured image data obtained in S2 are associated through LLM to obtain integrated multimodal data.

[0035] S1 is performed by the following steps, as follows: Figure 1 : S1.1 When the tender document is a Word document, first use the docx2pdf library to convert the Word document to PDF format, and then use the pdf2image library to convert the PDF format to image format; when the tender document is a PDF document, directly use the pdf2image library to convert the PDF format to image format. S1.2. Traverse all pages of the image format from top to bottom. For each page of the image format in S1.1, use the PaddleOCR-VL multimodal model to identify the document content type. Mark documents with plain text paragraphs as text data, documents with row and column structures as table data, and documents with at least one of graphics or engineering symbols as image data. Then, use an image cropping tool to extract the text data, table data, and image data from the document to obtain fragment information that records corresponding metadata, where the metadata is chapter and page number. S1.3 Save the sharding information obtained in S1.2 in JSON format to info.json and proceed to S2.

[0036] The JSON format storage directory is as follows: Xxx document |——info.json |——images |——xxx.jpg The JSON format is as follows: [ { "id":"1", "type":"title", "content":"xxx", "img_path":"xxxx.jpg", "chapter":"1xxxx", "page":"1" } , ... ] It should be noted that common formats for tender documents are PDF (which may contain scanned copies and vector graphics), Word, etc. This invention uniformly converts them to PDF format and processes them as images.

[0037] The JSON format includes an identifier (Id), a type (Type), a content field (Content), an image screenshot field (Img_path), a chapter field (chapter), and an image page number (Page).

[0038] It should be noted that the identifier `Id` is a unique identifier. The `Type` field contains titles, chapters, text, tables, and images. The `Content` field stores plain text content; for tables, it stores the content of key-value structured data (A3 recognition); for images, it stores the content of structured data (B2 recognition). The `Img_path` field contains screenshots of each section. The `chapter` field stores the chapter information, recording the content of the most recent chapter above the current section. The `Page` field stores the page numbers of the PDF images.

[0039] The table processing channel processes the table fragmentation information through the following steps, such as... Figure 2 : A1. Detect the table row and column boundaries in the table segmentation information using the improved visual table recognition model Table Transformer, and proceed to A2; A2. Extract the contents of cell A2 and move them to cell A3; A3. Convert the cell content into key-value structured data, while retaining the page number (Page) and chapter field of the image containing the table, to obtain structured table data.

[0040] It should be noted that when extracting the cell content "Equipment unit price scoring standard: ≥500,000 yuan gets 3 points" from A2, the key of the "key-value" structured data is the equipment unit price scoring standard, and the value is ≥500,000 yuan getting 3 points. The improved visual table recognition model, Table Transformer, is a ViT-based Table Transformer.

[0041] The image processing channel processes the image segmentation information through the following steps: Figure 3 : B1. Use Deepseek-OCR multimodal large model to identify the technical parameter annotations in the image segmentation information, such as technical parameter annotations as "Cable cross-section: YJV-4×120+1×70" and "Equipment size: 1200×800×600mm"; then extract the text content in the technical parameter annotations and proceed to B2. B2. Convert the data into structured data according to the format of "technical parameter item-value-unit" and associate it with the corresponding technical chapter of the image, such as "Chapter 3 Equipment Technical Requirements", to obtain structured image data.

[0042] S3 includes the following steps, such as Figure 4 : S3.1. Using the chapter field "chapter" as the anchor point for all segmented information, such as "Chapter 2 Evaluation Method" and "Chapter 4 Schedule Requirements", text data, image data, and table data under the same chapter field "chapter" are grouped together, and then semantic analysis is performed, such as identifying the core theme and parameter correlation, and then proceeding to S3.2. S3.2. Use the Qwen3-32B large model to perform text clustering on the same group to obtain classification results, such as "evaluation method unit" and "engineering unit", then proceed to S3.3; S3.3 Based on the classification results of S3.2, the text data, image data, and table data are associated to obtain integrated multimodal data.

[0043] The tables, images, and text content are linked together: Input: The shard information (id) and content of the same group.

[0044] Output: Grouping results.

[0045] Large model prompt: "You are an excellent information classification expert. Please help me group the related IDs in the input content."

[0046] #Enter content {"id":"1",content":"xxx"} {"id":"3",content":"xxx"} ... #Output Format [ { "group_id":"1", "reason":"Reason for grouping", "ids":["xx","xx"] }, { "group_id":"2", "reason":"Reason for grouping", "ids":["xx"] }, ... ] " The consistency verification phase consists of the following steps: Figure 5 : C1. Input the bidding conflict verification rule base, the target item, and the integrated multimodal data into the LLM; C2. Based on the bidding conflict verification rule base and in conjunction with the target item, review the integrated multimodal data to generate a structured parsing result, which includes the conflict content, conflict location, and judgment criteria.

[0047] It should be noted that generating structured parsing results facilitates subsequent adjustments to the tender documents.

[0048] During the consistency verification phase, the LLM is a fine-tuned Llama 3 or Qwen. It should be noted that the LLM in this invention is an existing model; it is an end-to-end model. The working principle of the LLM is to input the corresponding text into the model, which then directly outputs the corresponding text. The working principle of the LLM is common knowledge in the field and will not be elaborated upon here.

[0049] The structured analysis results specifically include the conflict location, conflict content, and preliminary judgment criteria. For example, the conflict location is "the text in Chapter 2 conflicts with the table in Chapter 3 regarding the project duration", the conflict content is "the text has 90 days vs. the table has 120 days", and the preliminary judgment criteria is "according to the priority of the tender documents, the table data has higher validity than the text description".

[0050] Large model prompt: "You are an excellent information analysis expert. Please refer to the conflict rules and determine whether there is a conflict between the input content and the standards and requirements."

[0051] #Conflict Rules Xxx #Standards and Requirements Xxx #Enter content Document content #Output Format Thought: Your thought process Answer: [{ "Conflict content":"", "Conflict Location":"", "Judgment basis:" }, ...] " This cross-modal collaborative parsing method based on multimodal bidding documents establishes a collaborative mechanism between LLM and visual models for the first time, linking text with corresponding tables and drawings to achieve cross-modal data association and binding. Furthermore, this invention automatically verifies the consistency of multimodal information, outputs verification results, and locates conflict points. This invention achieves automatic parsing and association of multimodal information (text, tables, and drawings) in bidding documents, as well as consistency verification and conflict detection, solving the problems of fragmented, inefficient, and error-prone multimodal information in traditional manual parsing. It is suitable for intelligent review and information extraction of bidding documents in scenarios such as bidding agencies, corporate bidding departments, and government procurement platforms.

[0052] Example 2 An application of a cross-modal collaborative parsing method based on multimodal tender documents, as described in Example 1, wherein the tender document is a power equipment tender document and is in Word format, such as... Figure 6 During the data analysis phase, it is converted into image format using S1.1, such as... Figure 7 Then, S1.2 trims to obtain the fragment information as follows: Figure 8 .

[0053] The sharding information is saved in JSON format to info.json using S1.3. The specific code is as follows: [ { "id":"1", "type":"image", "content":"", "img_path":"images / image_1.jpg", "chapter":"3.1 Hat Body", page: "22" }, { "id":"2", "type":"table", "content":"", "img_path":"images / table_1.jpg", "chapter":"3.1 Hat Body", page: "23" }, { "id":"14", "type":"text", "content":"3.4.2 Technical Requirements for Heat Sinks" Appearance and material requirements: 1) Material: 6061 aluminum alloy 2) Burrs and rough edges: There should be no rough edges that can fall off or affect assembly. There should be no rough edges in the visible area, and there should be no rough feel. 3) Scratches: Length <10mm, depth <0.2mm, width <0.1mm, and no more than 2 scratches on the same visible plane. 4) Color difference: The color inside and outside the same batch must be consistent. 5) Dirt: Allowable area within the same visible plane <15mm 2 No more than two erasable stains are allowed: For non-exterior surfaces on the same plane, the area of ​​erasable stains must be less than 20mm. Document content type identification is required, and the number of erasable stains must not exceed five. 6) Material debris: No material debris or metal residue is allowed on the surface, in gaps, or inside threaded holes. 7) Sand holes, bubbles, and spots: Both diameter and height must be less than 0.5mm, and they must not affect assembly; no more than two spots should be present on the same visible plane. "img_path":"", Chapter 3.4.2 Technical Requirements for Heat Sinks page: "26" }, { "id":"15", "type":"image", "content":"", "img_path":"images / image_3.jpg", Chapter 3.4.2 Technical Requirements for Heat Sinks page: "27" }, ... ] Among them, the table input processed by the table processing channel is as follows: Figure 9 Its output is as follows: "Component": "Cap cover, wire cover, battery cover, lever", "Material": "ABS", "Density (g / cm³): "1.05±0.2", "Tensile strength (MPa): "≥21", "Elongation at break (%)": "≥40", "Flexural strength (MPa): "≥40", "Heat distortion temperature (°C): ">100", "Notched impact strength (kJ / m): ">15", "Rockwell hardness (R): "75" "Component":"Cap Liner", "Material":"PE", "Density (g / cm³):"0.96", "Tensile Strength (MPa):"≥24.3", "Elongation at Break (%)":">600", "Flexural Strength (MPa):"≥23", "Heat Deflection Temperature (°C):">71", "Notched Impact Strength (kJ / m):">4", "Rockwell Hardness (R):"64" After the table-based processing channel completes its processing, the recognition result is written back to the content field of the corresponding segment information. The specific code is as follows: { "id":"2", "type":"table", "content":"\"Components\":\"Cap Cover, Battery Cover, Toggle Lever\":{\"Material\":\"ABS\",\"Density (g / cm³)\":\"1.05±0.2\",\"Tensile Strength (MPa)\":\"≥21\",\"Elongation at Break (%)\":\"≥40\",\"Flexural Strength (MPa)\":\"≥40\",\"Heat Deflection Temperature (°C)\":\">100\",\"Notched Impact Strength (kJ / m)\":\">15\",\"Rockwell Hardness (R)\":\"75\" "Component": "Cap Liner", "Material": "PE", "Density (g / cm³)": "0.96", "Tensile Strength (MPa)": "≥24.3", "Elongation at Break (%)": "600", "Flexural Strength (MPa)": "≥23", "Heat Deflection Temperature (°C)": "71", "Notched Impact Strength (kJ / m)": "4", "Rockwell Hardness (R)": "64" ", "img_path":"images / table_1.jpg", "chapter":"3.1 Hat Body", page: "23" } Among them, the image input processed by the image processing channel is as follows: Figure 10 Its output is as follows: The image is a grayscale diagram showing five plastic parts of a safety helmet, neatly arranged in top-to-bottom order: 1. Top curved shell (PE material) 2. Battery cover (small rectangle with two round holes) 3. Lever (long strip with a lever protrusion at one end) 4. Cable cover (flat with grooves) 5. Bottom ring-shaped liner (ABS material). All parts are filled with the same grayscale, without color or labeling, against a pure white background, and arranged in an exploded view layout for easy identification of the names and shapes of each part.

[0054] After processing the image data, the recognition result is written back to the content field of the corresponding segment information. The specific code is as follows: "id":"1", "type":"image", "content":"The image is a grayscale diagram showing five plastic parts of a safety helmet, neatly arranged in top-to-bottom order: 1. Top curved shell (PE material) 2. Battery cover (small rectangle with two round holes) 3. Lever (long strip with a lever protrusion at one end) 4. Cable cover (flat with grooves) 5. Bottom ring-shaped liner (ABS material). All parts are filled with the same grayscale, without color or labeling, against a pure white background, and arranged in an exploded view layout for easy identification of the names and shapes of each part."

[0055] ", "img_path":"images / image_1.jpg", "chapter":"3.1 Hat Body", page: "22" }, The specific relationships between text fragmentation, table fragmentation, and image fragmentation in S3 are as follows: Large model prompt: "You are an excellent information classification expert. Please help me group the related IDs in the input content."

[0056] #Enter content {"id":"1", content":"The image is a grayscale diagram showing five plastic parts of a safety helmet, neatly arranged in top-to-bottom order: 1. Top curved shell (PE material) 2. Battery cover (small rectangle with two round holes) 3. Lever (long strip with a lever protrusion at one end) 4. Cable cover (flat with grooves) 5. Bottom ring-shaped liner (ABS material). All parts are filled with the same grayscale, without color or labeling, against a pure white background, presented in an exploded view layout for easy identification of the parts' names and shapes."} {"id":"2",content":"\"Component\":\"Cap Cover, Battery Cover, Toggle Switch\",\"Material\":\"ABS\",\"Density (g / cm³)\":\"1.05±0.2\",\"Tensile Strength (MPa)\":\"≥21\",\"Elongation at Break (%)\":\"≥40\",\"Flexural Strength (MPa)\":\"≥40\",\"Heat Deflection Temperature (°C)\":\">100\",\"Notched Impact Strength (kJ / m)\":\">15\",\"Rockwell Hardness (R)\":\"75\"} "Component": "Cap Liner", "Material": "PE", "Density (g / cm³)": "0.96", "Tensile Strength (MPa)": "≥24.3", "Elongation at Break (%)": "600", "Flexural Strength (MPa)": "≥23", "Heat Deflection Temperature (°C)": "71", "Notched Impact Strength (kJ / m)": "4", "Rockwell Hardness (R)": "64" "}, {"id":"14","content":"3.4.2 Technical Requirements for Heat Sinks"} Appearance and material requirements: 1) Material: 6061 aluminum alloy 2) Burrs and rough edges: There should be no rough edges that can fall off or affect assembly. There should be no rough edges in the visible area, and there should be no rough feel. 3) Scratches: Length <10mm, depth <0.2mm, width <0.1mm, and no more than 2 scratches on the same visible plane. 4) Color difference: The color inside and outside the same batch must be consistent. 5) Dirt: Allowable area within the same visible plane <15mm 2 No more than two areas of dirt can be wiped clean: The area of ​​dirt that cannot be wiped clean on the same plane (excluding the exterior surface) must be less than 20mm. 2 and no more than 5 places 6) Material debris: No material debris or metal residue is allowed on the surface, in gaps, or inside threaded holes. 7) Sand holes, bubbles, and spots: Both diameter and height must be less than 0.5mm, and they must not affect assembly; no more than two spots should be present on the same visible plane. ... #Output Format [ { "group_id":"1", "reason":"The contents of id1 and 2 are both descriptions of the helmet shell." "ids":["1","2"] }, ... ] " In this embodiment, the specific code for the target item extraction stage is as follows: Large model prompt: "You are an excellent information extraction expert. Help me extract data on 'evaluation criteria,' 'construction period requirements,' and 'technical parameter requirements' from the input document content. Note that each criterion or requirement is presented in a list format."

[0057] #Enter content Document content #Output Format { "Evaluation Criteria":[], "Schedule Requirements":[], Technical Specifications Requirements: [] } " Output: {"Evaluation Criteria":["Bid Price (60 points)", "Technical Response (25 points)", "Supply Performance and Operational Proof (10 points)", "After-Sales Service and Warranty Commitment (5 points)"], "Construction Period Requirements": ["All equipment must be delivered within 90 days of the contract taking effect", "Commissioning must be completed within 30 days of the site being ready for installation", "The overall commissioning must not affect the annual power grid maintenance plan"] Technical Specifications: ["Cap shell made of ABS material, bending strength ≥ 40MPa", "Cap liner made of PE material, tensile strength ≥ 20MPa",...]} The following are some of the conflict rules in the bidding conflict verification rule base of this embodiment: The text "cable cross-section ≥100mm²" is determined to be in conflict with the image label "YJV-4×95+1×50"; The table "Equipment power 5kW" and the image "Equipment parameters 4.5kW" are considered to be in conflict; The text contains 5 parameters for the device, while the image only contains 4, which is considered a conflict.

[0058] The specific code for the consistency verification phase in this embodiment is as follows: Large model prompt: "You are an excellent information analysis expert. Please refer to the conflict rules and determine whether there is a conflict between the input content and the standards and requirements."

[0059] #Conflict Rules The text "cable cross-section ≥ 100mm²" is determined to conflict with the image label "YJV-4×95+1×50". The table showing "Equipment power 5kW" and the image showing "Equipment parameters 4.5kW" are considered to be in conflict. The text contains 5 parts of the device parameters, while the image only contains 4 parts, which is considered a conflict. ... #Standards and Requirements Evaluation Criteria: ["Bid Price (60 points)", "Technical Response (25 points)", "Supply Performance and Operational Proof (10 points)", "After-Sales Service and Warranty Commitment (5 points)"] "Construction Period Requirements": ["All equipment must be delivered within 90 days of the contract taking effect", "Commissioning must be completed within 30 days of the site being ready for installation", "The overall commissioning must not affect the annual power grid maintenance plan"] Technical Specifications: ["Cap shell made of ABS material, bending strength ≥ 40MPa", "Cap liner made of PE material, tensile strength ≥ 20MPa",...] #Enter content {"id":"1","type":"image", "content":"The image is a grayscale diagram showing five plastic parts of a safety helmet, neatly arranged in top-to-bottom order: 1. Top curved shell (PE material) 2. Battery cover (small rectangle with two round holes) 3. Lever (long strip with a lever protrusion at one end) 4. Cable cover (flat with grooves) 5. Bottom ring-shaped liner (ABS material). All parts are filled with the same grayscale, without color or labeling, against a pure white background, and arranged in an exploded view layout for easy identification of the names and shapes of each part.}

[0060] ","chapter":"3.1 Hat body","page":"22" } {"id":"2","type":"table","content":"\"Component\":\"Cap Cover, Battery Cover, Toggle Lever\",\"Material\":\"ABS\",\"Density (g / cm³)\":\"1.05±0.2\",\"Tensile Strength (MPa)\":\"≥21\",\"Elongation at Break (%)\":\"≥40\",\"Flexural Strength (MPa)\":\"≥40\",\"Heat Deflection Temperature (°C)\":\">100\",\"Notched Impact Strength (kJ / m)\":\">15\","Rockwell Hardness (R)\":\"75\"} "Component": "Cap Liner", "Material": "PE", "Density (g / cm³)": "0.96", "Tensile Strength (MPa)": "≥24.3", "Elongation at Break (%)": "600", "Flexural Strength (MPa)": "≥23", "Heat Deflection Temperature (°C)": "71", "Notched Impact Strength (kJ / m)": "4", "Rockwell Hardness (R)": "64" ","chapter":"3.1 Hat body","page":"22"} #Output Format Thought: Your thought process Answer: [{ "Conflict content":"", "Conflict Location":"Page x, Chapter xx", "Judgment basis:" }, ...] Output: Thought: The images and tables above describe the five parts of the cap shell: the cap shell, cable cover, battery cover, lever, and cap liner. The table data for the cap shell, cable cover, battery cover, and lever, specifically the parameters: "Material": ABS, "Density (g / cm³): 1.05±0.2", "Tensile Strength (MPa): ≥21", "Elongation at Break (%)": ≥40", "Flexural Strength (MPa): ≥40", "Heat Deflection Temperature (°C): ">100", "Notched Impact Strength (kJ / m): ">15", and "Rockwell Hardness (R): 75," meet the tender requirement of "the cap shell is made of ABS material with a flexural strength ≥40MPa."

[0061] The cap liner parameters are as follows: Material: PE, Density (g / cm³): 0.96, Tensile Strength (MPa): ≥24.3, Elongation at Break (%): ">600", Flexural Strength (MPa): ≥23, Heat Deflection Temperature (°C): ">71", Notched Impact Strength (kJ / m): ">4", Rockwell Hardness (R): 64. This is consistent with the tender standard requirement that "the cap liner uses PE material with a tensile strength ≥20MPa". There are no conflicts. However, the image data shows that the cap shell is made of PE material, while the cap liner is made of ABS material, which may be an error in the drawing.

[0062] Answer: [{ "Conflict Content": "There is a conflict in the image labeling; the cap shell should be made of ABS material, and the cap liner should be made of PE material." "Conflict Location": "Page 22, Chapter 3.1 Hat Body", The judgment criteria state: "The technical parameters require that the helmet shell be made of ABS material with a bending strength ≥40MPa, and the helmet liner be made of PE material with a tensile strength ≥20MPa. In the table data, the helmet shell, cable cover, battery cover, and lever are all made of ABS material, while the helmet liner is made of PE material. However, the image data indicates that the helmet shell is made of PE material, and the helmet liner is made of ABS material." }] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A cross-modal collaborative parsing method based on multimodal tender documents, characterized in that, It includes the following stages: The data analysis phase involves converting the multimodal tender documents into image format, identifying the content type, and cropping them to obtain segmentation information, which includes text segmentation information, table segmentation information, and image segmentation information. The table segmentation information is then processed to obtain structured table data, and the image segmentation information is processed to obtain structured image data. Then, the text segmentation information, the table segmentation information, and the image segmentation information are aligned to obtain integrated multimodal data; Target item extraction stage—Extracting the target items from the tender documents; Consistency verification phase – Based on the pre-built bidding conflict verification rule base, the target item and the integrated multimodal data are analyzed through LLM fusion to perform consistency verification and generate structured parsing results; The data analysis phase is carried out through the following steps: S1. Convert the tender document into an image format, then identify and crop it to obtain segmentation information, and proceed to S2; S2. Input the table segmentation information into the table processing channel to obtain structured table data, and write the structured table data back to the content of the table segmentation information in S1 for later use; input the image segmentation information into the image processing channel to obtain structured image data, and write the structured image data back to the content of the image segmentation information in S1 for later use. S3. Align the text fragment information obtained in S1, the structured table data and the structured image data obtained in S2 using LLM to obtain integrated multimodal data; S3 specifically includes the following steps: S3.

1. Using the chapter field of all segmented information as the anchor point, group the text data, image data and table data under the same chapter field into the same group, and then perform semantic analysis, proceeding to S3.

2. S3.

2. Use the Qwen3-32B large model to perform text clustering on the same group, obtain the classification results, and proceed to S3.3; S3.3 Based on the classification results of S3.2, align the text data, image data, and table data to obtain integrated multimodal data; The consistency verification phase is performed by the following steps: C1. Input the bidding conflict verification rule base, the target item, and the integrated multimodal data into the LLM; C2. Based on the bidding conflict verification rule base and in conjunction with the target item, review the integrated multimodal data to generate a structured parsing result, which includes the conflict content, conflict location, and judgment criteria.

2. The cross-modal collaborative parsing method based on multimodal bidding documents according to claim 1, characterized in that: S1 is performed by the following steps: S1.1 When the tender document is a Word document, first use the docx2pdf library to convert the Word document to PDF format, and then use the pdf2image library to convert the PDF format to image format; when the tender document is a PDF document, directly use the pdf2image library to convert the PDF format to image format. S1.

2. Traverse all pages of the image format from top to bottom. For each page of the image format in S1.1, use the PaddleOCR-VL multimodal model to identify the document content type. Mark documents with plain text paragraphs as text data, documents with row and column structures as table data, and documents with at least one of graphics or engineering symbols as image data. Then, use an image cropping tool to extract the text data, table data, and image data from the document to obtain fragment information that records corresponding metadata, where the metadata is chapter and page number. S1.3 Save the sharding information obtained in S1.2 to info.json in JSON format, and then proceed to S2; The JSON format includes an identifier (Id), a type (Type), a content field (Content), an image screenshot field (Img_path), a chapter field (chapter), and an image page number (Page).

3. The cross-modal collaborative parsing method based on multimodal bidding documents according to claim 2, characterized in that: The table processing channel processes the table fragmentation information through the following steps: A1. Detect the table row and column boundaries in the table segmentation information using the improved visual table recognition model Table Transformer, and proceed to A2; A2. Extract the contents of cell A2 and move them to cell A3; A3. Convert the cell content into key-value structured data, while retaining the page number (Page) and chapter field (chapter) of the image containing the table, to obtain structured table data.

4. The cross-modal collaborative parsing method based on multimodal bidding documents according to claim 3, characterized in that, The image processing channel processes the image segmentation information through the following steps: B1. Use Deepseek-OCR multimodal large model to identify the technical parameter annotations in the image segmentation information, then extract the text content in the technical parameter annotations, and proceed to B2; B2. Convert the data into structured data according to the format of "Technical Parameter Item - Value - Unit", and associate it with the corresponding technical section of the image to obtain structured image data.

5. The cross-modal collaborative parsing method based on multimodal tender documents according to any one of claims 1 to 4, characterized in that: The target item extraction stage uses the Qwen3-32B large model to extract the target items from the tender document; The target item is at least one of the following: evaluation criteria, construction period requirements, and technical parameter requirements.

6. The cross-modal collaborative parsing method based on multimodal tender documents according to claim 3 or 4, characterized in that: The improved visual table recognition model, Table Transformer, is a ViT-based Table Transformer.

7. The cross-modal collaborative parsing method based on multimodal tender documents according to any one of claims 1 to 4, characterized in that: The bidding conflict verification rule base stores multiple conflict rules, which are at least one of the following: text data and table data conflict rules, text data and image data conflict rules, or table data and image data conflict rules.

Images