Highway bridge drawing multi-modal information extraction and semantic understanding method and system

Abstract

The invention belongs to the technical field of engineering information intelligent processing, and relates to a highway bridge drawing multi-modal information extraction and semantic understanding method and system.The method comprises the steps that firstly, self-adaptive judgment is conducted on a vector drawing and a scanning drawing, geometric distortion correction, drawing frame and title bar positioning and layout segmentation are completed, and then the vector drawing and the scanning drawing are obtained; constructing a hierarchical document structure comprising texts, tables, images and two-dimensional drawing objects; then, a front-end target detection network and a rear-end cross-modal document understanding model are fused, and detection and relation reasoning of elements such as view blocks, labels, symbols and tables are achieved; and image-text feature alignment is further performed by using a visual coding network and a text coding network, structural description conforming to engineering semantics is generated by means of a multi-modal language model, an engineering parameter database is established, and parameter query and multi-modal question and answer output are supported. According to the method, the accuracy and efficiency of automatic acquisition and semantic understanding of the key information of the highway bridge drawing are improved.

Description

Technical Field

[0001] This invention belongs to the field of intelligent processing technology of engineering information, specifically relating to a method and system for multimodal information extraction and semantic understanding of highway bridge drawings. Background Technology

[0002] In highway bridge construction, design documents are typically archived in PDF format, containing various types of drawings and textual materials, including general layout drawings, standard cross-sections, structural details, and general design specifications. These documents not only contain graphic information such as component layout and structural nodes, but also a large number of dimensional parameters, symbols, tables, and textual annotations, serving as the fundamental basis for design, review, and construction. In practice, designers and reviewers need to repeatedly compare dimensions, materials, and component relationships across multiple drawings to verify the consistency and safety of the overall design, resulting in high labor costs and a high risk of errors.

[0003] Existing document parsing technologies include OCR-based character recognition, library function-based layout parsing, multi-module pipeline solutions, and end-to-end multimodal models. Among them, the multi-module solution of "layout detection—feature partitioning—calling a dedicated recognizer—result reconstruction" has achieved good results in general documents such as academic papers and reports. However, bridge engineering drawings are characterized by diverse views, dense annotations, complex professional symbols, and strong cross-page references. General methods have obvious shortcomings in engineering scenarios: First, the recognition accuracy of professional elements such as dimension annotations, component symbols, section lines, and legends is insufficient; second, it is difficult to restore the reading order and component reference relationships across pages and views; third, there is a lack of joint modeling of geometric elements and engineering semantics, making it difficult to convert the drawing content into computable structured data; and fourth, the industry adaptability for the professional scenario of highway bridges is insufficient, making it difficult to support applications such as parameter verification, engineering retrieval, and engineering Q&A based on drawings.

[0004] Therefore, it is necessary to propose a multimodal element detection, information extraction and semantic understanding technology for highway engineering bridge drawings. This technology can automatically extract key parameters and core component information from complex engineering drawings, realize geometric-semantic coupled modeling, and output the data in a consistent structured form, providing basic support for the digital and intelligent processing of design documents. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a method and system for multimodal element detection, information extraction and semantic understanding of bridge drawings for highway engineering. It can automatically extract key parameters and core component information from complex engineering drawings, realize geometric-semantic coupled modeling, and output in a consistent structured form, providing basic support for the digital and intelligent processing of design documents.

[0006] The technical solution of this invention is as follows: A method for multimodal information extraction and semantic understanding of highway bridge drawings includes the following steps: Step 1: Document Adaptive Judgment and Preprocessing: Acquire the complete set of design drawings for the highway engineering bridge project. For each page of PDF drawings, parse the page objects and metadata, and determine whether the drawing is a vector drawing or a scanned drawing based on the type of objects contained therein. For vector drawings, extract the drawing frame, title bar, and content area based on the layout parsing algorithm, and establish the mapping relationship between the drawing frame coordinates and the page coordinates. For scanned drawings, perform geometric distortion correction and noise suppression to obtain a distortion-free drawing image, and extract the drawing frame area and title bar area based on page area detection and drawing frame positioning. Step 2, Multi-dimensional Heterogeneous Data Structured Modeling: The pre-processed drawing content is organized into a hierarchical document structure by page. Each node in the document structure includes at least a page number attribute and an object set attribute. The object set includes at least one of text objects, table objects, ordinary image objects, and two-dimensional drawing objects. Through the layout analysis model and the optical character recognition model, the content of each object and its geometric position on the page are jointly extracted and aligned to obtain a structured document representation carrying content description and coordinate information. Step 3: Engineering layout and element detection based on multi-model fusion: Based on the structured document representation, a front-end object detection network is used to perform multi-scale candidate region detection on the entire page drawing to obtain candidate boxes, category priors, confidence scores, and preliminary structural relationships. The image slices, layout geometric features, and associated text corresponding to the candidate regions are fed into the back-end cross-modal document understanding model for semantic verification and fine-grained reclassification, outputting the layout element categories and their layout relationships. By constructing a page element relationship graph, overall reasoning is performed on relationships such as alignment, containment, adjacency, and cross-page continuity to obtain the detection results of view blocks, annotation elements, symbol elements, and table elements. Step 4: Semanticization of Multimodal Information Integrating Context: The 2D drawing image is segmented and visual features are extracted based on a visual encoding network. The annotation text in the drawing is segmented and embedded based on a text encoding network. A cross-modal attention mechanism is used to align visual features with text features. The aligned multimodal features are input into a multimodal language model. A constrained decoding strategy is used to generate a structured description that conforms to engineering semantics. The semantic parsing of the drawing content is completed in a hierarchical manner of page level - local unit level - overall structure level. Step 5: Output and Integration of Structured Engineering Information: Based on the structured description, an engineering parameter database is constructed, and key design parameters spanning multiple pages and views are uniformly represented using a hierarchical JSON structure. When a user query is received, the query statement is semantically parsed and intent is identified, and a multi-level search is performed in the engineering parameter database in the order of exact matching, similarity retrieval, and association expansion. The search results, along with the drawing context, are input into a multimodal generation model to generate an engineering answer containing parameter values, units, and applicable conditions. At the same time, the corresponding JSON-formatted structured data and a question-and-answer report carrying the search path are output.

[0007] Furthermore, the geometric distortion correction includes at least: global rotation angle estimation based on line detection, perspective or affine transformation correction, mesh displacement field estimation and thin plate spline transformation for local rolling distortion, and background noise removal based on adaptive threshold.

[0008] Furthermore, in step 2: the document structure is a tree-like hierarchical structure, with upper-level nodes corresponding to engineering projects, atlases, or chapters, and lower-level nodes corresponding to specific pages or drawing units; the text object includes at least one of text content, bounding box coordinates, font style, and reading order index; the table object includes table header information, cell content, and row and column indexes in the document; the two-dimensional drawing object includes drawing image data, title bar area, detail bar area, and position index of the view area.

[0009] Furthermore, in step 3: the front-end object detection network is a YOLO series document layout detection model finely tuned for engineering document scenarios; the back-end cross-modal document understanding model is a multimodal Transformer model that co-encodes text content, image features, and layout information, which calculates the relationship scores between layout elements by constructing a fully connected graph, and clusters and divides the elements into regions based on the relationship scores; during the training phase, asymmetric collaborative training and knowledge distillation are adopted to introduce the soft labels and element relationship constraints generated by the back-end into the front-end training process.

[0010] Furthermore, the construction of the page element relationship graph includes: using text blocks, image blocks, table blocks, and two-dimensional drawing view blocks as graph nodes; encoding the alignment, containment, adjacency, and cross-page continuity relationships between nodes as graph edges; and using a graph structure reasoning algorithm to adjust the category and belonging region of the page elements based on the weight of the graph edges.

[0011] Furthermore, in step 4: the visual encoding network uses a hierarchical Vision Transformer to dynamically segment the drawing image and extract multi-layer spatial features; the text encoding network uses a sub-word segmenter to encode text such as dimension annotations, technical specifications, and material specifications, and adds positional encoding to maintain the text sequence order; the cross-modal attention mechanism uses a dual-stream attention structure, enabling visual features and text features to query each other through cross-attention, establishing semantic correspondence between image regions and text fragments, and introducing relative positional bias to enhance the model's ability to model the spatial layout of the drawing.

[0012] Furthermore, in step 5: each record in the engineering parameter database includes at least the parameter name, parameter value, unit, source drawing identifier, source page number, and source coordinate area; the question and answer report is used to generate the retrieval path for the engineering answer, the parameter items involved in the calculation and their weights, and the corresponding drawing evidence screenshots or location information, so as to achieve the traceability of the parameter query process.

[0013] A system for multimodal information extraction and semantic understanding of highway bridge drawings using the above method includes: The document preprocessing module is used to perform document type recognition, geometric distortion correction, noise removal, and positioning of the drawing frame and title block on PDF drawings of highway engineering bridge designs, and output the mapping relationship between the drawing frame coordinates and the page coordinates. The data structuring modeling module is used to convert pre-processed drawing content into a hierarchical document structure with page number attributes and object set attributes, and to perform unified modeling of text, tables, ordinary images and two-dimensional drawings; The layout and feature detection module is used to perform multi-model fusion engineering layout analysis and feature detection on document pages based on a front-end object detection network and a back-end cross-modal document understanding model, and to construct a page element relationship diagram. The multimodal semantic parsing module is used to extract multimodal features based on visual encoding networks and text encoding networks, perform image-text alignment through cross-modal attention mechanisms, and call multimodal language models to generate engineering semantic structured descriptions. The structured information output and integration module is used to convert the structured description into a hierarchical JSON structured engineering parameter database, respond to user query requests, and complete parameter retrieval and engineering Q&A result output.

[0014] Furthermore, the layout and element detection module includes: The high-speed sensing front-end sub-module is used to perform multi-scale candidate region detection on the entire page of drawings and output candidate boxes, category priors, confidence scores, and preliminary structural relationships. The semantic discrimination post-module is used to receive image slices, layout geometry information and associated text of the candidate region, and to perform semantic verification and fine-grained category correction on the candidate region; The routing control submodule is used to directly adopt the front-end detection results for easy-to-determine samples based on front-end confidence and context consistency indicators, and to route difficult samples to the semantic discrimination back-end submodule for review.

[0015] Furthermore, the structured information output and integration module is also configured to: perform semantic parsing and intent recognition based on the query statement input by the user, perform exact matching, similarity retrieval and association expansion from the engineering parameter database, and input the retrieval results and drawing context into the multimodal generation model; output including engineering answer text, corresponding JSON format structured data, and a question-and-answer report recording the retrieval path and evidence weights.

[0016] The beneficial effects of this invention are: 1. Adaptive multi-source unified modeling for bridge drawings, improving the automation of information acquisition: By adaptively distinguishing between vector drawings and scanned drawings, and correcting geometric distortion, as well as positioning the drawing frame and title block, this invention can automatically construct a unified hierarchical document structure on highway bridge design drawings from different sources and of different quality. It standardizes the representation of text, tables, images, and two-dimensional engineering drawings, and achieves automatic structuring across formats and layouts, significantly reducing the workload of manual browsing and manual data entry.

[0017] 2. Multi-model fusion of layout and element detection improves the accuracy and robustness of complex engineering layout recognition: By combining a high-speed front-end object detection network with a back-end cross-modal document understanding model and introducing force-based element relationship reasoning, this invention has stronger robustness and fine classification capabilities in the detection of multiple types of elements such as view blocks, annotations, symbols, and tables. It can adapt to complex scenarios such as multiple views, multiple scales, dense annotations, and uneven scanning quality in bridge drawings, thereby improving the overall accuracy and stability of layout analysis and engineering element recognition.

[0018] 3. Integrating geometric and semantic multimodal understanding to solve the problem of disconnect between geometric and semantic information: This invention utilizes a combination of visual encoding networks and text encoding networks with a cross-modal attention mechanism to explicitly align the geometric features of drawings with annotation text and explanatory text. Then, it generates engineering semantic descriptions through a multimodal language model, which can simultaneously maintain the consistency between the spatial positional relationship of components and engineering semantics, solving the problem in traditional methods where "text is correctly recognized but cannot correspond to specific components or views".

[0019] 4. Construct an engineering parameter database and support traceable Q&A to improve parameter retrieval and verification efficiency: By organizing the parsed key design parameters into a hierarchical JSON structure engineering parameter database, and combining semantic parsing, multi-level retrieval, and multimodal generation, this invention can directly respond to engineering parameter query requests such as "span, width, elevation, and material strength," and output answers and Q&A reports with numerical values, units, applicable conditions, and drawing coordinate evidence. This enables traceability of parameter sources and verification of the query process, which is beneficial for rapid retrieval during design review, verification, and operation and maintenance phases.

[0020] 5. Modular architecture and good scalability, easy to integrate into existing engineering digitization platforms: The method and system of this invention are designed in a modular manner according to the "preprocessing - structured modeling - layout detection - semantic parsing - structured output" process. It can be deployed independently according to project needs or connected with existing design review systems, BIM platforms, and engineering archive systems. It has good scalability and engineering implementation, and is also easy to migrate to other engineering drawing scenarios such as tunnels and municipal engineering, and has high promotion value. Attached Figure Description

[0021] Figure 1 This is a flowchart of the steps of the present invention.

[0022] Figure 2 This is a schematic diagram of the document structure of the present invention.

[0023] Figure 3 This is a schematic diagram of the structured data of the present invention.

[0024] Figure 4 This is a schematic block diagram of the multimodal fusion model system of the present invention.

[0025] Figure 5 This is a schematic diagram illustrating a design specification text layout recognition example of the present invention.

[0026] Figure 6 This is a schematic diagram illustrating a case of two-dimensional constructed image and annotation text layout recognition according to the present invention.

[0027] Figure 7 This is a schematic diagram illustrating a table layout recognition example of the present invention.

[0028] Figure 8 This is a schematic diagram of the engineering structured information output and integration of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described below with reference to the accompanying drawings and specific embodiments. Those skilled in the art should understand that various equivalent substitutions or combinations can be made to the following embodiments without departing from the spirit and essence of this invention, and all such substitutions or combinations should fall within the protection scope of this invention.

[0030] Example 1: See Figure 1 This embodiment provides a multimodal structured information extraction and semantic understanding method for highway engineering bridge drawings, applicable to a complete set of PDF design drawings for highway engineering bridge projects. The overall process includes the following steps: S01: By performing adaptive document discrimination and preprocessing on different types of bridge engineering drawings, a foundation is laid for subsequent feature extraction and recognition; S02: Using the data processed by S01, realize the structured modeling of multi-dimensional heterogeneous data; S03: Utilize the structured information of the two-dimensional drawings defined in S02 to perform engineering layout analysis and element detection based on multi-model fusion; S04: Based on the engineering multimodal analysis results obtained in S03, realize the semanticization of multimodal information fused with context; S05: Based on context-based engineering structured information, construct standardized data structures for output and integration.

[0031] The following is a detailed explanation of each step: Step 1: Document Adaptive Judgment and Preprocessing: Obtain the complete set of PDF design drawings for highway engineering bridge projects. These files are typically multi-page documents containing different types of structural drawings and text descriptions. The system reads the page objects and metadata page by page using a PDF parsing library, and performs the following processing on each page of the PDF drawing: Step 1.1, Document Type Identification: Based on the types of objects contained on the page, the drawing is identified as either a vector drawing or a scanned drawing. Specifically: if the page contains a large number of vector primitives (such as line segments, paths, vector text objects, etc.), it is identified as a vector drawing; if the page only contains one or more embedded bitmap image objects and lacks vector primitives, it is identified as a scanned drawing. For vector drawings, the system directly parses the vector primitives and text objects on the page to extract the geometric elements that constitute the drawing frame, title bar, and content area; for scanned drawings, the system extracts the embedded bitmaps into engineering drawing images and proceeds to the image-level geometric correction and denoising process.

[0032] Step 1.2, Page Area Detection and Title Frame Location: For vector graphics or scanned drawings that have undergone geometric correction, the system performs page area detection and title frame location based on edge detection, rectangle detection, and projection analysis. Specifically, this includes: searching for closed rectangular borders in the page edge region; if a closed rectangle conforming to engineering drawing specifications is detected, it is used as the title frame boundary; if no complete closed rectangle is detected, the binarized page image is horizontally and vertically projected, and an approximate page area rectangle is determined based on the edge density distribution; identifying the rectangular area where the title bar is located, and obtaining the title bar area boundary; the system outputs a set of rectangular information, including the entire page boundary, the title frame boundary, and the title bar boundary, and establishes a mapping relationship between the title frame coordinates and the page coordinates for subsequent unified representation of the geometric positions of each object.

[0033] Step 1.3, Geometric Distortion Correction and Noise Suppression: Preferably, for scanned drawings, the geometric distortion correction includes at least the following processing steps: Based on line detection (such as probabilistic Hough transform), the angles of dominant lines such as the edge of the drawing frame and the baseline of the text are estimated to obtain the overall rotation angle, and rotation correction is performed on the drawing. With the four corners of the title bar clearly visible, detect the coordinates of the four corner points of the title bar, construct homography transformation or affine transformation matrices, and perform perspective or affine distortion correction on the drawing. For scanned drawings with local rolling distortion, a regular grid is constructed on the image, the displacement field of the grid nodes is estimated, and local nonlinear geometric correction is performed on the entire image based on the thin plate spline model. After geometric correction, an adaptive thresholding algorithm (such as Sauvola / Niblack adaptive thresholding) is used to binarize the grayscale drawing to suppress background noise, scan texture and blemishes, and improve the quality of subsequent layout analysis and OCR recognition.

[0034] Step 1.4 Project Metadata Extraction: For vector drawings, extract project metadata such as drawing name, drawing number, scale, design unit, and date by parsing the text objects in the title bar area; for scanned drawings, call the OCR model to perform text recognition in areas such as the title bar, drawing name bar, and scale bar to extract the above-mentioned project metadata. This metadata can be used as auxiliary fields for subsequent project parameter databases and retrieval results.

[0035] Step 1 above enables adaptive discrimination and unified preprocessing of PDF drawings for different types of bridge projects, and provides standardized geometric and text input for subsequent structured modeling and feature detection.

[0036] Step 2: Structured Modeling of Multi-variable Heterogeneous Data: Highway engineering bridge and other transportation engineering drawings commonly contain complex data types with multiple modalities and levels. Their elements exhibit both high heterogeneity and dense semantic relationships, as shown in Table 1. The drawings not only include textual and structured information such as technical specifications and parameter tables, but also image data such as geological profiles and cross-sectional diagrams, as well as two-dimensional geometric drawings representing spatial structures. Table 1; like Figure 2 As shown, after completing the drawing preprocessing, this embodiment constructs a tree-like hierarchical document structure model, unifying multimodal content such as text, tables, ordinary images, and two-dimensional engineering drawings into a structured representation. Step 2 specifically includes: Step 2.1 Document Tree Structure Definition: Preferably, a tree-like hierarchical structure is used to organize the entire set of engineering drawings. Document tree nodes can be defined as follows: ,in, As a unique identifier for the node, For node names, such as "General Layout Drawing", "Standard Cross Section", "Construction Details" and "Design Specifications"; This refers to the page number or page set identifier corresponding to the node. This is the set of objects for that node.

[0037] Step 2.2 Object Type and Attributes: Preferably, the object set is divided into at least one of the following types: text object, table object, image object, and two-dimensional drawing object. A text object includes at least the following attributes: text content (text_content), bounding box coordinates (bbox) in the page or drawing frame coordinate system, font information such as font size, font type, bold, italic, etc., and reading order index (reading_order) used to restore the reading order of paragraphs and sentences. A table object includes at least: the overall table bounding box (table_bbox), row count, column count (col_count), and a collection of cells, each cell containing at least a row and column index (r, c) and cell content (cell_text). The table includes a cell bounding box (cell_bbox) and a header area identifier to distinguish between indicator names and data areas; image objects must include at least: an image bounding box (image_bbox), an image type (e.g., a geological profile or schematic diagram), and an image data index or reference; two-dimensional drawing objects must include at least: a drawing body area (bbox_2d), a title block area (title_block_bbox), a details block or material list area (detail_block_bbox), and a view region list (view_regions) to distinguish between plan views, elevation views, section views, construction details, etc.

[0038] Step 2.3 Layout Analysis and OCR Alignment: The system calls the layout analysis model for each page of drawing to perform preliminary page segmentation, attaching the page segments to the corresponding document tree nodes. Then, it calls the OCR engine (PaddleOCR is used for the OCR recognition model) to obtain the text string and coordinate information. By aligning the page segment coordinates with the OCR coordinates, the recognized text is attached to the corresponding text objects. For two-dimensional drawing objects, the view area, title bar area, and detail bar area are associated with their attached text and table objects to form a unified structured representation of "geometry + semantics".

[0039] Step 3: Engineering layout and feature detection based on multi-model fusion: such as Figure 3 As shown in Figure 4, after the document tree and object set are constructed, this embodiment performs multi-model fusion engineering layout and feature detection on the two-dimensional engineering drawings to obtain view blocks, annotation features, symbol features, and table features, etc.; specifically, it includes the following steps: Step 3.1, Front-end Object Detection Network: In a preferred embodiment, the front-end object detection network is a YOLO series document layout detection model finely tuned for engineering document scenarios, such as DocLayout-YOLO. This network takes a full-page drawing image as input and outputs multi-scale candidate regions, including the position, size, category prior, and confidence score of the candidate boxes. Candidate categories may include titles, body paragraphs, view blocks, tables, legends, dimension annotations, etc.

[0040] Step 3.2, Backend Cross-Modal Document Understanding Model: Preferably, the backend cross-modal document understanding model is a multimodal Transformer model that encodes text content, image features, and layout information, such as one based on the LayoutLMv3 architecture. The input to the backend model includes: image slices of candidate regions detected by the frontend, geometric information of candidate boxes (coordinates, size, relative positional relationships), and text content of the candidate regions and their neighborhoods. The model encodes the above information in an integrated manner, calculates the category score of each candidate region and the relationship score between them, thereby performing semantic verification and fine-grained reclassification of the candidate regions, and outputting the corrected layout element categories.

[0041] Step 3.3, Front-end and Back-end Asymmetric Collaborative Training: During the training phase, asymmetric collaborative training and knowledge distillation strategies are preferably adopted: The back-end model is fully trained on labeled samples and outputs soft labels for candidate regions of the front-end as teacher signals; the front-end model introduces a distillation loss term on the basis of its standard detection loss, so that the category probability distribution of the front-end output is close to the back-end soft labels, and at the same time, element relationship constraint loss is introduced to encourage elements in the same semantic region to have similar feature representations; in this way, the discrimination ability and structure perception ability of the front-end model are improved.

[0042] Step 3.4, Page Element Relationship Graph Construction and Inference: Preferably, after completing the front-end and back-end detection, this embodiment constructs a page element relationship graph to further resolve issues of element overlap, cross-page splitting, and category ambiguity. The page element relationship graph includes: using text blocks, image blocks, table blocks, and two-dimensional drawing view blocks as graph nodes; encoding the alignment, inclusion, adjacency, and cross-page continuity relationships between nodes as graph edges; using a graph structure inference algorithm (e.g., edge weight-based clustering or graph neural network) to process the graph, merging or grouping elements according to the relationship edge weights, and adjusting the page element categories and their respective regions; through the above processing, the final detection result containing view blocks, annotation elements, symbol elements, and table elements is obtained.

[0043] Step 3.5, Parameter Candidate Set Construction: For the identified key annotations, specifications, tabular data, etc., the system constructs a parameter candidate set C. Each candidate parameter element includes at least text content, parameter category, drawing location coordinates, and confidence level, for subsequent semantic parsing and engineering parameter database construction. The expression for the parameter candidate set C is: , in, For text content, For the predicted values ​​of the parameter categories, Here are the coordinates of the position on the drawing. This represents the confidence score.

[0044] Step 4: Semantize multimodal information by fusing context: such as Figure 5-7 As shown, to achieve cross-modal semantic parsing of drawings, this embodiment proposes a multimodal semantic parsing method for two-dimensional engineering drawings based on the Qwen-VL architecture. This method achieves a deep understanding of the drawing content through joint vision-language modeling. Specifically, it includes the following steps: Step 4.1, Visual Encoding Network: Preferably, the visual encoding network adopts a hierarchical Vision Transformer to perform block division and multi-layer self-attention calculation on the two-dimensional engineering drawings to extract multi-scale spatial features; in order to preserve the geometric and scale information of the bridge structure, a larger patch size can be used in the large-scale view and a smaller patch size can be used in the detailed structural view to enhance the ability to express details.

[0045] Step 4.2, Text Encoding Network: Preferably, the text encoding network performs word-level segmentation (e.g., a BPE-based segmenter) on text such as dimension annotations, technical specifications, and material specifications, then maps it into continuous vectors through an embedding layer, and adds positional encoding to maintain the order information of the text sequence.

[0046] Step 4.3, Cross-modal attention mechanism: The cross-modal attention mechanism adopts a two-stream or cross-attention structure, enabling visual features and text features to query each other through cross-attention. Using text features as queries, we search for image regions in visual features that are semantically related to them, and establish a mapping between "text and image". Using visual features as queries, we search for text features that describe the region and establish an "image → text" mapping. By introducing relative position offsets, the spatial relationships between elements in the drawings are encoded into attention calculations, thereby improving the ability to model the spatial layout of engineering drawings.

[0047] Step 4.4, Multimodal Language Model and Constraint Decoding: The aligned multimodal features are fed into the multimodal language model. The model can be any pre-trained multimodal language model that supports text and image input, such as Qwen-VL. The model uses an autoregressive decoding method to generate structured engineering semantic descriptions. To ensure that the output meets the expression habits and standards of the engineering field, a constraint decoding strategy is adopted during the decoding process to constrain the use of terminology, unit format, and parameter representation, so that the generated results are standardized and consistent.

[0048] Step 4.5, Layered Progressive Parsing: In a preferred embodiment, semantic parsing is performed in a layered manner of "page level → local unit level → global structure level": First, the text and image content and view blocks are identified at the page level; then, local engineering semantic units are constructed within the view blocks, such as "superstructure layout", "bridge deck system construction", "substructure and foundation", etc.; finally, at the project level, each local unit is integrated into an overall engineering structure description.

[0049] Step 5, Output and Integration of Structured Engineering Information: such as Figure 8 As shown, this embodiment constructs a context-based engineering question-answering and verifiable output system based on multimodal semantics. Through standardized structured engineering information, it achieves accurate retrieval and visual verification of design parameters; specifically, it includes the following steps: Step 5.1, Construction of Engineering Parameter Database: Preferably, an engineering parameter database is constructed based on the aforementioned structured description. Key design parameters spanning multiple pages and views are uniformly represented using a hierarchical JSON structure. Each record in the engineering parameter database includes at least: parameter name, parameter value, unit, source drawing identifier, source page number, and coordinate area in the drawing. Additional information such as the component to which it belongs and the structural unit to which it belongs may also be included if necessary.

[0050] Step 5.2, Query Parsing and Multi-level Retrieval: When the system receives a user query (such as "main beam span specification" or "bridge deck width limit"), it performs semantic parsing and intent recognition on the query, extracts keywords and constraints related to engineering parameters, and then performs multi-level retrieval in the engineering parameter database in the order of exact matching, similarity retrieval, and association expansion to obtain a set of candidate parameters.

[0051] Step 5.3, Multimodal Question Answering Result Output and Report: Preferably, the search results, along with the corresponding drawing context, are input into a multimodal generation model (such as Qwen-VL). The model generates an engineering answer containing parameter values, units, applicable conditions, etc. Simultaneously, the system generates a question answering report to record: the parameter records involved in generating the engineering answer and their search paths, the weight or importance index of each parameter in the multimodal generation process, and screenshots or location information of the corresponding drawing evidence. Through the question answering report, the verifiability and traceability of the engineering parameter query process are realized.

[0052] Example 2: This example provides a multimodal information extraction and semantic understanding system for highway bridge drawings. This system is suitable for deployment on servers, workstations, or local terminals, and its functions implement the method described in Example 1. The system includes the following functional modules: The document preprocessing module is used to identify document type, correct geometric distortion, remove noise, locate the drawing frame and title block of PDF drawings for highway engineering bridge design, and output the mapping relationship between drawing frame coordinates and page coordinates to realize the document adaptive discrimination and preprocessing functions. The data structuring modeling module is used to convert pre-processed drawing content into a hierarchical document structure with page number attributes and object set attributes, and to perform unified modeling of text, tables, images and two-dimensional drawings; The layout and feature detection module is used to perform multi-model fusion engineering layout analysis and feature detection on document pages based on a front-end object detection network and a back-end cross-modal document understanding model, and to construct a page element relationship diagram. The multimodal semantic parsing module is used to extract multimodal features based on visual encoding networks and text encoding networks, perform image-text alignment through cross-modal attention mechanisms, and call multimodal language models to generate engineering semantic structured descriptions. The structured information output and integration module is used to convert the structured description into a hierarchical JSON structured engineering parameter database, respond to user query requests, and complete parameter retrieval and engineering Q&A result output.

[0053] Preferably, the layout and element detection module includes: The high-speed sensing front-end sub-module is used to perform multi-scale candidate region detection on the entire page of drawings, and output candidate boxes, category priors, confidence scores and preliminary structural relationships; The semantic discrimination post-module is used to receive image slices, layout geometry information and associated text of candidate regions, and to perform semantic verification and fine-grained category correction on candidate regions; The routing control submodule is used to directly use the front-end detection results for easily judged samples based on front-end confidence and context consistency indicators, and to route difficult samples to the semantic discrimination back-end submodule for review.

[0054] Preferably, the structured information output and integration module is further configured as follows: Based on the user's query, semantic parsing and intent recognition are performed, and precise matching, similarity retrieval, and association expansion are executed from the engineering parameter database; The search results and the drawing context are input into the multimodal generation model, and the output includes the engineering answer text, the corresponding JSON format structured data, and a question-and-answer report that records the search path and evidence weights.

[0055] The above modules can be implemented by software programs and / or hardware circuits, and can be deployed in the same computing device or distributed across multiple devices and communicate through a network. Those skilled in the art can choose a suitable deployment scheme according to the actual project scale and performance requirements.

[0056] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art. The general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for highway bridge drawing multi-modal information extraction and semantic understanding, characterized in that, Comprise the following steps: Step 1, document adaptive discrimination and pretreatment: obtaining a complete set of design drawings of highway engineering bridge projects, analyzing page objects and metadata for each page of PDF drawings, and discriminating drawings into vector drawings or scanned drawings according to the types of objects contained; wherein, for vector drawings, the layout analysis algorithm is used to extract the frame, title bar and content area, and the mapping relationship between the frame coordinates and the page coordinates is established; for scanned drawings, geometric distortion correction and noise suppression are performed to obtain distortion-free drawing images, and the frame area and title bar area are extracted based on the center of the layout detection and frame positioning; Step 2, structured modeling of multi-element heterogeneous data: the contents of the pretreated drawings are organized into hierarchical document structures in units of pages, each node in the document structure includes at least a page number attribute and an object set attribute, and the object set contains at least one of text objects, table objects, ordinary image objects and two-dimensional drawing objects; the contents and geometric positions of each object are extracted and aligned by a layout analysis model and an optical character recognition model to obtain a structured document representation carrying content description and coordinate information; Step 3, engineering layout and element detection based on multi-model fusion: based on the structured document representation, a front-end object detection network is used to detect multi-scale candidate regions of the whole page drawing to obtain candidate boxes, class priors, confidence and preliminary structure relationship; the image slices corresponding to the candidate regions, layout geometric features and associated texts are input into a back-end cross-modal document understanding model for semantic verification and fine-grained reclassification, and the layout element class and layout relationship are output; by constructing a page element relationship graph, the alignment, inclusion, adjacency and cross-page continuity relationships are inferred as a whole to obtain the detection results of view blocks, labeled class elements, symbolic class elements and table class elements; Step 4, multi-modal information semanticization of fused context: the two-dimensional drawing image is processed by block and the visual features are extracted based on a visual coding network, the labeled text in the drawing is segmented and embedded based on a text coding network, and a cross-modal attention mechanism is used to align the visual features and text features; the aligned multi-modal features are input into a multi-modal language model, a constraint decoding strategy is used to generate structured descriptions conforming to engineering semantics, and the semantic analysis of drawing content is completed in a hierarchical manner of page level-local unit level-global structure level; Step 5, engineering structured information output and integration: based on the structured description, an engineering parameter database is constructed, and key design parameters across pages and multiple views are uniformly represented in a hierarchical JSON structure; when a user query is received, the query statement is semantically analyzed and intent recognized, and multi-level retrieval is performed in the engineering parameter database in the order of exact matching, similarity retrieval and associated expansion; the retrieval results are input into a multi-modal generation model together with the context of the drawing to generate engineering answers containing parameter values, units and applicable conditions, and output corresponding JSON format structured data and question and answer reports carrying retrieval paths.

2. The method of claim 1, wherein, The geometric distortion correction at least includes: overall rotation angle estimation based on straight line detection, perspective or affine transformation correction, mesh displacement field estimation for local rolling distortion and thin plate spline transformation, and background noise removal based on adaptive threshold.

3. The method of claim 1, wherein, In the step 2, the document structure is a tree hierarchy, the upper layer node corresponds to an engineering project, a drawing set or a chapter, and the lower layer node corresponds to a specific page or a drawing unit; the text object includes at least one of text content, a bounding box coordinate, a font style and a reading order index; the table object includes table header information, cell content and row and column indexes in the document; and the two-dimensional drawing object includes drawing image data, a title bar region, a detail bar region and a position index of a view region.

4. The method of claim 1, wherein, In the step 3, the front-end target detection network is a YOLO series document layout detection model fine-tuned for an engineering document scene; the back-end cross-modal document understanding model is a multi-modal Transformer model that jointly encodes text content, image features and layout information, calculates relationship scores between layout elements by constructing a full connection graph, and clusters and divides the elements according to the relationship scores; in the training stage, asymmetric collaborative training and knowledge distillation are used to introduce the soft labels and element relationship constraints generated by the back-end into the front-end training process.

5. The method of claim 1, wherein, The step of constructing a page element relationship graph comprises: taking a text block, an image block, a table block and a two-dimensional drawing view block as a graph node; encoding the alignment relationship, the containing relationship, the adjacency relationship and the cross-page continuous relationship between the nodes as a graph edge; and adjusting the category and the attribution area of the layout element based on the weight of the graph edge by using a graph structure reasoning algorithm.

6. The method of claim 1, wherein, In the step 4, the visual coding network adopts a hierarchical Vision Transformer to dynamically divide the drawing image into blocks and extract multi-layer spatial features; The text coding network adopts a sub-word level tokenizer to encode the text such as size marking, technical description and material specification, and adds position encoding to maintain the sequence order of the text sequence; and the cross-modal attention mechanism adopts a double-flow attention structure, so that the visual features and the text features query each other through cross-attention, establish a semantic corresponding relationship between the image regions and the text segments, and introduce a relative position bias to enhance the modeling capability of the drawing space layout.

7. The method of claim 1, wherein, In the step 5, each record in the engineering parameter database at least includes a parameter name, a parameter value, a unit, a source drawing identifier, a source page number and a source coordinate area; and the question and answer report is used to generate a retrieval path of an engineering answer, a parameter item participating in calculation and a weight thereof, and corresponding drawing evidence screenshots or position information, so as to realize traceability of a parameter query process.

8. A highway bridge drawing multi-modal information extraction and semantic understanding system applying the method of claim 1, characterized in that, The method comprises the following steps: a document preprocessing module is configured to perform document type identification, geometric distortion correction, noise removal, frame and title bar positioning on the PDF drawing of the highway engineering bridge design, and output a mapping relationship between frame coordinates and page coordinates; A data structured modeling module is configured to convert the preprocessed drawing content into a hierarchical document structure with page number attributes and object set attributes, and uniformly model text, tables, common images and two-dimensional drawings; A layout and element detection module is configured to perform engineering layout analysis and element detection on a document page based on a front-end target detection network and a back-end cross-modal document understanding model, and construct a page element relationship graph; A multi-modal semantic analysis module is configured to extract multi-modal features based on a visual coding network and a text coding network, perform image-text alignment through a cross-modal attention mechanism, and call a multi-modal language model to generate an engineering semantic structured description; A structured information output and integration module is configured to convert the structured description into an engineering parameter database in a hierarchical JSON structure, respond to user query requests, and complete parameter retrieval and engineering question and answer result output.

9. The system of claim 8, wherein, The layout and element detection module includes: A high-speed perception front-end sub-module is configured to perform multi-scale candidate region detection on an entire page of drawings and output candidate boxes, class priors, confidence levels and preliminary structure relationships; A semantic discrimination back-end sub-module is configured to receive image slices of the candidate regions, layout geometric information and associated text, and perform semantic verification and fine-grained class correction on the candidate regions; A routing control sub-module is configured to directly use front-end detection results for easy-to-determine samples based on front-end confidence levels and context consistency indicators, and route difficult samples to the semantic discrimination back-end sub-module for review.

10. The system of claim 8, wherein, The structured information output and integration module is further configured to perform semantic analysis and intent recognition based on a user input query statement, perform accurate matching, similarity retrieval and associated expansion on an engineering parameter database, and input the retrieval results and drawing context into a multi-modal generation model; and output an engineering answer text, corresponding JSON format structured data and a question and answer report recording a retrieval path and evidence weight.

Images