A building drawing multi-modal recognition and structured processing method and system

Abstract

This invention discloses a method and system for multimodal recognition and structured processing of architectural drawings, belonging to the field of engineering digitization technology. The method includes: firstly, standardizing and preprocessing the engineering drawings by using a visual model with a rotated bounding box to fully scan and correct the coordinate system; then extracting and isolating four functional areas—tables, details, annotations, and drawing frames—through deep semantic segmentation; secondly, after OCR recognition of the table area, coordinate correction is performed, and the data is input into a large language model to complete topological reasoning and semantic error correction, outputting standardized table data without templates; thirdly, extracting multimodal features from the detail area, achieving cross-modal alignment through cross-attention, and verifying component material attributes through multi-point sampling; finally, completing end-to-end error correction through three-level closed-loop verification, outputting a highly reliable structured dataset that can be directly integrated with engineering systems such as BIM and cost estimation. This invention can significantly improve the efficiency of engineering drawing digitization and is applicable to intelligent processing of drawings in multiple fields such as architecture, municipal engineering, and railways.

Description

Technical Field

[0001] This invention relates to the field of engineering digitization technology, and in particular to a method and system for multimodal recognition and structured processing of architectural drawings. Background Technology

[0002] In the fields of architectural, structural, and mechanical and electrical engineering design and digital delivery, engineering drawings are the core carrier for conveying design intent and guiding construction. With the popularization of Building Information Modeling (BIM) technology, the industry has placed extremely high demands on the rapid parsing, key information extraction, and standardized structuring of massive amounts of PDF and CAD format drawings.

[0003] Existing intelligent processing solutions for engineering drawings generally adopt a pipeline model of "traditional image processing + rule matching + basic OCR". Some solutions introduce deep learning object detection for component localization, but the following core defects still exist:

[0004] 1. The table structure relies on preset templates. When faced with merged cells across rows, heterogeneous non-standard table headers, and borderless tables, the topology recognition is prone to errors, resulting in extremely poor versatility.

[0005] 2. Lack of engineering semantic understanding capabilities: Existing OCR can only extract spatial coordinates and text strings, and cannot parse the engineering logical constraints between components, dimensions, material specifications and numbers. The recognition results are only scattered data.

[0006] 3. Without a cross-modal verification mechanism, visual recognition is prone to "AI illusion". Detailed drawings and material lists, as well as individual drawings and summary lists, cannot be cross-verified, resulting in frequent errors such as omissions and data conflicts. The recognition results cannot be directly connected to downstream cost and construction systems.

[0007] Therefore, there is an urgent need for a method and system for multimodal recognition and structured processing of architectural drawings to solve the existing technical problems. Summary of the Invention

[0008] The present invention aims to solve at least one of the technical problems existing in the prior art, and proposes a method and system for multimodal recognition and structured processing of architectural drawings.

[0009] In a first aspect, embodiments of the present invention provide a method for multimodal recognition and structured processing of architectural drawings, including:

[0010] S100. Standardize and preprocess the input engineering drawings, perform full-domain scanning using a visual model with rotation bounding box perception capability, correct the drawing space coordinate system, and then extract and isolate four types of regions of interest: table area, detail area, annotation area, and standard drawing frame area through a deep semantic segmentation network.

[0011] S200. Perform optical character recognition on the table area to obtain the text set and corresponding spatial coordinates. Correct the text through a coordinate correction algorithm. Input the corrected text, standardized coordinates and spatial adjacency matrix into the large language model to complete row and column topology relationship reasoning and engineering semantic error correction, and output standardized table structured data.

[0012] S300. Visual geometric features and textual semantic features are extracted from the detailed drawing area respectively. The spatial dimension and semantic dimension are aligned in both directions through a multimodal cross-attention mechanism. The association mapping between the component graphics and the annotation text is established. The confidence level of the component material properties is confirmed through a multi-point sampling verification mechanism.

[0013] S400 sequentially performs three levels of verification: single-image closed-loop self-check, multi-image association and integration, and total quantity closed-loop approval and offsetting. It performs full-link logical consistency verification and error correction on the recognition results and outputs a full-scale structured dataset of verified architectural drawings.

[0014] Furthermore, the specific steps for implementing S100 include:

[0015] S101. Receive engineering drawings in PDF, DWG or image format, perform Gaussian filtering for noise reduction, bilinear interpolation for scale normalization, and convert them into a standard resolution grayscale tensor image;

[0016] S102. Input the grayscale tensor image into the visual model based on the feature pyramid network and the rotating bounding box detection mechanism, extract the topological anchor points of the drawing, and complete the absolute space coordinate system correction;

[0017] S103. Perform pixel-by-pixel semantic labeling on the corrected image using a Transformer-based semantic segmentation network, and output region masks for the table area, detail area, annotation area, and standard frame area.

[0018] Furthermore, the coordinate correction algorithm in S200 is specifically as follows: a linear correction algorithm based on maximizing the variance of projection density is used to calculate the global rotation angle θ, and to perform rotation transformation correction on all text coordinate points. The transformation formula is:

[0019]

[0020] Where (xi,yi) are the original text center coordinates, and (xi′,yi′) are the corrected coordinates. The rotation transformation makes the variance of the projected coordinates of the text in the same row / column converge to a local minimum.

[0021] Furthermore, the specific methods for topological relation reasoning in the large language model in S200 include:

[0022] Construct an input sequence containing semantic hints for engineering tables. The sequence includes corrected text, standardized coordinates, and spatial relative adjacency matrices. The large language model performs engineering common sense error correction and probabilistic reasoning of row and column topological relationships in the vector space. It automatically identifies merged cells and multi-level headers, outputs a symbolic description of the table topological relationships, and then generates standardized JSON-formatted table structured data through the symbolic rendering module.

[0023] Furthermore, the specific steps for implementing S300 include:

[0024] S301. Extracting multimodal features: Visual branch extraction of the geometry, size boundary lines, and spatial geometric tensors pointing to the leader lines of the detailed image. The text branch extracts semantic feature tensors from detail annotations, numbers, and material lists. ;

[0025] S302. Perform cross-modal alignment: This is achieved through a multimodal cross-attention mechanism. and Perform spatial and semantic bidirectional alignment to establish a high-confidence association mapping between component graphic features and corresponding labeled text;

[0026] S303. Perform multi-point sampling verification: Perform feature sampling on multiple spatial points of the detailed drawing leader line, the main body of the large detail drawing, and the annotation text. When the matching degree of the sampling results meets the majority voting threshold or the engineering logic verification rules, confirm the component material attributes and write them into the structured database.

[0027] Furthermore, the specific methods for single-map closed-loop self-testing in S400 include:

[0028] The physical attributes of the components extracted from the detailed drawing area are semantically aligned and logically compared with the material list data reconstructed from the table area on the same page. The verification indicators include the matching degree of the number, the equivalence of geometric specifications, and the rationality of the physical tolerance of the quantity calculation. When a conflict is detected, the abnormal context and image slices are input into the large language model for engineering common sense review. High-confidence misidentification is automatically corrected, conflict markers are designed and recorded in the single drawing verification list.

[0029] Furthermore, the specific methods for multi-drawing association and integration in S400 include: summarizing the structured data of all distributed drawings in the entire project, establishing a directed graph topology network among the drawing data based on the correspondence between drawing index symbols and detail drawing numbers, and constructing a unified digital foundation for the project.

[0030] Furthermore, the specific methods for total closed-loop approval hedging in S400 include:

[0031] The total sum of components and categories at the bottom of the statistical project's digital base. Nominal total amount compared to the project's top-level master material summary table Perform hedging verification and calculate the difference value. ,in = - When ΔQ≠0 or component parameter level conflicts occur, the coordinates of the corresponding sub-graphs, cells or details are accurately located, and a visual full-project verification report with highlighted marks and jump links is generated.

[0032] Secondly, this invention also discloses a multimodal recognition and structured processing system for architectural drawings, comprising: a drawing preprocessing and multi-scale semantic segmentation perception module, a neural symbol structured reconstruction module for heterogeneous tables, a multimodal detail drawing deconstruction and cross-modal alignment module, and a multi-dimensional consistency collaborative verification and data output module; wherein:

[0033] The drawing preprocessing and multi-scale semantic block perception module is used to perform standardized preprocessing on the input engineering drawings. It performs full-domain scanning through a visual model with the ability to perceive rotating bounding boxes, corrects the spatial coordinate system of the drawings, and then extracts and isolates four types of regions of interest: table area, detail area, annotation area, and standard drawing frame area through a deep semantic segmentation network.

[0034] The neural symbol structure reconstruction module for heterogeneous tables is used to perform optical character recognition on the table area, obtain the text set and corresponding spatial coordinates, correct the text through a coordinate correction algorithm, input the corrected text, standardized coordinates and spatial adjacency matrix into a large language model, complete row and column topological relationship reasoning and engineering semantic error correction, and output standardized table structured data.

[0035] The multimodal detail map deconstruction and cross-modal alignment module is used to extract visual geometric features and textual semantic features from the detail map area respectively. It achieves bidirectional alignment of spatial and semantic dimensions through a multimodal cross-attention mechanism, establishes an association mapping between component graphics and annotation text, and confirms the confidence level of component material properties through a multi-point sampling verification mechanism.

[0036] The multi-dimensional consistency collaborative verification and data output module is used to sequentially perform three levels of verification: single-image closed-loop self-check, multi-image association integration, and total quantity closed-loop approval and offsetting. It performs full-link logical consistency verification and error correction on the identification results and outputs a full-scale structured dataset of verified architectural drawings.

[0037] Thirdly, the present invention also discloses an electronic device, comprising:

[0038] One or more processors;

[0039] Memory, used to store one or more programs;

[0040] When the one or more programs are executed by the one or more processors, the one or more processors implement the method.

[0041] This invention discloses a method and system for multimodal recognition and structured processing of architectural drawings. The method includes: firstly, standardizing and preprocessing the engineering drawings by using a visual model with a rotated bounding box to fully scan and correct the coordinate system; then extracting and isolating four functional areas—tables, details, annotations, and drawing frames—through deep semantic segmentation; secondly, after OCR recognition of the table area, coordinate correction is performed, and the data is input into a large language model to complete topological reasoning and semantic error correction, outputting standardized table data without templates; thirdly, extracting multimodal features from the detail area, achieving cross-modal alignment through cross-attention, and verifying component material attributes through multi-point sampling; finally, completing end-to-end error correction through three-level closed-loop verification, outputting a highly reliable structured dataset that can be directly integrated with engineering systems such as BIM and cost estimation. This invention can significantly improve the efficiency of digitizing engineering drawings and is applicable to intelligent processing of drawings in multiple fields such as architecture, municipal engineering, and railway engineering. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating a method for multimodal recognition and structured processing of architectural drawings provided in an embodiment of the present invention.

[0043] Figure 2 A structural block diagram of a multimodal recognition and structuring system for architectural drawings provided in an embodiment of the present invention;

[0044] Figure 3 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0045] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0046] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0047] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0048] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0049] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0050] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0051] To address at least one of the technical problems existing in the aforementioned related technologies, the present invention provides a method and system for multimodal recognition and structured processing of architectural drawings.

[0052] This implementation discloses a method for multimodal recognition and structured processing of architectural drawings, such as... Figure 1 ,include:

[0053] S100. Standardize and preprocess the input engineering drawings, perform full-domain scanning using a visual model with rotation bounding box perception capability, correct the drawing space coordinate system, and then extract and isolate four types of regions of interest: table area, detail area, annotation area, and standard drawing frame area through a deep semantic segmentation network.

[0054] In this embodiment, the specific steps of S100 include:

[0055] S101. Receive engineering drawings in PDF, DWG or image format, perform Gaussian filtering for noise reduction, bilinear interpolation for scale normalization, and convert them into a standard resolution grayscale tensor image;

[0056] S102. Input the grayscale tensor image into the visual model based on the feature pyramid network and the rotating bounding box detection mechanism, extract the topological anchor points of the drawing, and complete the absolute space coordinate system correction;

[0057] S103. Perform pixel-by-pixel semantic labeling on the corrected image using a Transformer-based semantic segmentation network, and output region masks for the table area, detail area, annotation area, and standard frame area.

[0058] Specifically, the process begins with inputting and standardizing the drawings. This involves receiving engineering drawings in PDF, DWG, or image formats, performing Gaussian filtering for noise reduction, bilinear interpolation for scale normalization, and converting multiple pages of drawings into a standard-resolution grayscale tensor image (I). Next, a full-domain multi-scale scan is performed. This involves inputting the tensor image (I) into a visual model based on a Feature Pyramid Network (FPN) and a rotation box perception mechanism (such as an improved detection network with OBB). Topological anchor points (such as the corner points of the outer frame and the intersection of the inner frame baseline) are extracted from the drawings, and the absolute spatial coordinate system of the drawings is corrected accordingly. Finally, ROI semantic segmentation is performed. This involves using a deep semantic segmentation network (such as a Transformer-based semantic segmentation architecture) to perform pixel-by-pixel semantic category labeling on the corrected image, outputting four types of ROI region masks: standard frame area, table area, detail area, and annotation area. The formula is expressed as: Where I is the corrected grayscale tensor image, and Mroi is the set of region of interest masks. Region masks are used to physically isolate different functional regions, avoiding cross-regional interference.

[0059] S200. Perform optical character recognition on the table area to obtain the text set and corresponding spatial coordinates. Correct the text through a coordinate correction algorithm. Input the corrected text, standardized coordinates and spatial adjacency matrix into the large language model to complete row and column topology relationship reasoning and engineering semantic error correction, and output standardized table structured data.

[0060] In this embodiment, the coordinate correction algorithm in S200 is specifically as follows: based on a linear correction algorithm that maximizes the variance of projection density, the global rotation angle θ is calculated, and rotation transformation correction is applied to all text coordinate points. The transformation formula is:

[0061]

[0062] Among them, (xi, yi) is the center coordinate of the original text, and (xi′, yi′) is the coordinate after rectification. Through rotation transformation, the projection coordinate variance of the text in the same row / column on the corresponding coordinate axis converges to a local minimum value.

[0063] Specifically, mask the table area Perform optical character recognition (OCR) on the mapped original image area, and output a set of text strings and their corresponding two-dimensional center coordinates of the bounding boxes , where . For the problems of axis tilt and line misalignment caused by scanning or format conversion, a linear rectification algorithm based on maximizing the projection density variance is adopted to calculate the global rotation angle θ, and all coordinate points are corrected as follows:

[0064]

[0065] Through correction, the text belonging to the same row or the same column is made to or The projection coordinate variance on the axis converges to a local minimum value, thus solving the problems of misalignment and overlap.

[0066] In this embodiment, the specific method of topological relationship reasoning of the large language model in S200 includes:

[0067] Construct an input sequence containing engineering table semantic prompts, which includes rectified text, standardized coordinates, and spatial relative adjacency matrix; the large language model completes engineering common sense error correction and row-column topological relationship probability reasoning in the vector space, automatically identifies merged cells and multi-level headers, outputs a symbolic description of the table topological relationship, and then generates standardized JSON format table structured data through the symbolic rendering module.

[0068] Specifically, take the rectified text sequence, standardized coordinates, and spatial relative adjacency matrix as inputs, construct Prompts with engineering table semantic prompt words, and input them into the large language model (LLM). The LLM not only aligns the text with engineering common sense in the vector space (for example, correcting the misrecognized "Φ20" to "Φ20" based on the structural context), but also performs probability reasoning on the row-column topological relationship based on spatial boundary symbols to determine whether there are merged cells or multi-level headers. According to the topological relationship output by the large language model, the symbolic rendering module reconstructs the table logic and directly outputs a standardized JSON structure.

[0069] S300. Extract visual geometric features and text semantic features from the detail area respectively, achieve two-way alignment of the spatial dimension and the semantic dimension through the multi-modal cross-attention mechanism, establish the association mapping between the component graphics and the annotation text, and confirm the confidence of the component material attributes through the multi-point sampling verification mechanism;

[0070] In this embodiment, the specific steps of S300 include:

[0071] S301. Extracting multimodal features: Visual branch extraction of the geometry, size boundary lines, and spatial geometric tensors pointing to the leader lines of the detailed image. The text branch extracts semantic feature tensors from detail annotations, numbers, and material lists. ;

[0072] Specifically, for the detailed area mask (e.g., detailed drawings of foundations, columns, walls, and towers) activate a multimodal feature fusion network. The visual branch uses convolution or self-attention mechanisms to extract the geometric features of the reinforcement details, including dimensional boundary lines and the direction of leader lines, into a spatial geometric tensor (Vgeo). The text branch extracts semantic feature tensors from the annotations, numbers, and material list text surrounding the detailed drawings. .

[0073] S302. Perform cross-modal alignment: This is achieved through a multimodal cross-attention mechanism. and Spatial and semantic bidirectional alignment is performed to establish a high-confidence association mapping between component graphic features and corresponding labeled text; specifically, a multimodal cross-attention mechanism is used to... and Perform bidirectional alignment of spatial and physical semantic dimensions. For example, when a visual branch captures a "circular cross-section rebar detail line," the cross-attention mechanism automatically associates this graphic feature with the text phrase that is spatially nearest to it or that the leader line points to. "Establish a high-confidence association mapping to bind material number, spatial geometry and physical properties."

[0074] S303. Multi-point sampling verification: Feature sampling is performed at multiple spatial points, including the detailed drawing leader line, the main body of the large-scale drawing, and the explanatory text. When the matching degree of the sampling results meets the majority voting threshold or the engineering logic verification rules, the component material attributes are confirmed and written into the structured database. Specifically, to address the technical defects of unstable identification of key parameters such as rebar diameter, spacing, and number of bars due to background noise and intersecting lines, the system establishes a multi-point sampling mechanism at multiple different spatial points, such as the detailed drawing leader line, the main body of the large-scale drawing, and the explanatory text. Only when the identification results of multiple different spatial sampling points are logically matched (e.g., total span / spacing + 1 = number of bars) or meet the majority voting threshold (e.g., >75% consistency rate), can the final component material attributes be determined and written into the detailed drawing structured database, ensuring robustness in complex backgrounds.

[0075] S400 sequentially performs three levels of verification: single-image closed-loop self-check, multi-image association and integration, and total quantity closed-loop approval and offsetting. It performs full-link logical consistency verification and error correction on the recognition results and outputs a full-scale structured dataset of verified architectural drawings.

[0076] In this embodiment, the specific method for single-graph closed-loop self-testing in S400 includes:

[0077] The physical attributes of the components extracted from the detailed drawing area are semantically aligned and logically compared with the material list data reconstructed from the table area on the same page. The verification indicators include the matching degree of the number, the equivalence of geometric specifications, and the rationality of the physical tolerance of the quantity calculation. When a conflict is detected, the abnormal context and image slices are input into the large language model for engineering common sense review. High-confidence misidentification is automatically corrected, conflict markers are designed and recorded in the single drawing verification list.

[0078] Specifically, the system automatically calls the single Figure 1 The consistency verification module performs semantic alignment and logical comparison between the "actual physical properties of detailed components" deconstructed in step three (e.g., a certain type of foundation has 5 steel bars drawn in the detailed drawing) and the "material list data on this page" reconstructed in step two (e.g., the number of reinforcement bars for the foundation with this number registered in the table). Verification indicators include: whether the numbering matches perfectly, whether the geometric specifications are equivalent, and whether the quantity calculation is within a reasonable physical tolerance range. The conflict handling mechanism includes: once a conflict between the drawing and text, omissions, or outliers are found (e.g., the quantity in the detailed drawing does not match the quantity in the list), the system first re-inputs the abnormal context and image slice features into the LLM, and then uses the LLM's common sense in architectural engineering for review (e.g., using standard drawing specifications to determine which value conforms to the common sense of engineering mechanics). If the large model has a high confidence level (>90%) and determines that it is a visual text misidentification, it is automatically corrected and recorded; if it is determined to be a conflict in the design itself, it is marked as "design drawing conflict" and recorded in the corresponding single drawing verification list.

[0079] In this embodiment, the specific method for multi-drawing association and integration in S400 includes: summarizing the structured data of all distributed drawings in the entire project, establishing a directed graph topology network among the drawing data based on the correspondence between drawing index symbols and detail drawing numbers, and constructing a unified digital foundation for the project. Specifically, after the single-drawing self-check is completed, the system globally integrates the distributed structured data of the entire project, which includes multiple sub-drawings (such as foundation drawings, column grid drawings, and beam reinforcement drawings), and establishes a directed graph topology network among the drawing data based on the index association between drawings (such as the correspondence between index symbols on the floor plan and detail drawing numbers), thus constructing a unified digital foundation for the project.

[0080] In this embodiment, the specific method for total quantity closed-loop approval hedging in S400 includes:

[0081] The total sum of components and categories at the bottom of the statistical project's digital base. Nominal total amount compared to the project's top-level master material summary table Perform hedging verification and calculate the difference value. ,in = - When ΔQ≠0 or component parameter level conflicts occur, the coordinates of the corresponding sub-graphs, cells or details are accurately located, and a visual full-project verification report with highlighted marks and jump links is generated.

[0082] Specifically, the "bottom-to-top cumulative total Qsum" calculated by statistically analyzing components and categories in the global digital base is compared end-to-end with the "nominal total Qtarget" extracted from the "total material summary table" at the top level of the project in step two. Through a difference analysis algorithm, ΔQ is precisely calculated. When ΔQ ≠ 0 or there is a conflict at the component parameter level, the system automatically and accurately locates the specific sub-drawing number, row / column cell, or detailed drawing coordinate area causing the difference. Conflicting, missing, and redundant items are highlighted in red, generating a visual full-project verification report with conflict jump links.

[0083] In this embodiment, after three-level closed-loop verification and confirmation, or after final manual correction of the visualized conflict list, the final full-volume reliable structured dataset is output, supporting JSON, Excel, or direct writing to a relational database. It is then integrated with the system. Specifically, this standardized data structure perfectly matches downstream business standards and can be seamlessly integrated via standard APIs into BIM automated modeling engines (automatically generating 3D components), cost estimation systems (achieving one-click automated full-volume quantity review and calculation), and the construction site's material supply chain management system, enabling seamless data reuse and refined management across different stages.

[0084] This embodiment discloses a method for multimodal recognition and structured processing of architectural drawings, including: firstly, standardizing and preprocessing the engineering drawings by using a visual model with a rotated bounding box to fully scan and correct the coordinate system; then extracting and isolating four functional areas—tables, details, annotations, and drawing frames—through deep semantic segmentation; secondly, after OCR recognition of the table area, coordinate correction is performed, and the data is input into a large language model to complete topological reasoning and semantic error correction, outputting standardized table data without templates; thirdly, multimodal features are extracted from the detail area, cross-modal alignment is achieved through cross-attention, and component material attributes are confirmed by multi-point sampling verification; finally, end-to-end error correction is completed through three-level closed-loop verification, outputting a highly reliable structured dataset that can be directly integrated with engineering systems such as BIM and cost estimation. This invention can significantly improve the efficiency of digitizing engineering drawings and is applicable to intelligent processing of drawings in multiple fields such as architecture, municipal engineering, and railway engineering.

[0085] Based on the same inventive concept, embodiments of the present invention also provide a system for multimodal recognition and structured processing of architectural drawings, such as... Figure 2 It includes: a drawing preprocessing and multi-scale semantic segmentation perception module, a neural symbol structured reconstruction module for heterogeneous tables, a multimodal detailed drawing deconstruction and cross-modal alignment module, and a multi-dimensional consistency collaborative verification and data output module; among which:

[0086] The drawing preprocessing and multi-scale semantic block perception module is used to perform standardized preprocessing on the input engineering drawings. It performs full-domain scanning through a visual model with the ability to perceive rotating bounding boxes, corrects the spatial coordinate system of the drawings, and then extracts and isolates four types of regions of interest: table area, detail area, annotation area, and standard drawing frame area through a deep semantic segmentation network.

[0087] The neural symbol structure reconstruction module for heterogeneous tables is used to perform optical character recognition on the table area, obtain the text set and corresponding spatial coordinates, correct the text through a coordinate correction algorithm, input the corrected text, standardized coordinates and spatial adjacency matrix into a large language model, complete row and column topological relationship reasoning and engineering semantic error correction, and output standardized table structured data.

[0088] The multimodal detail map deconstruction and cross-modal alignment module is used to extract visual geometric features and textual semantic features from the detail map area respectively. It achieves bidirectional alignment of spatial and semantic dimensions through a multimodal cross-attention mechanism, establishes an association mapping between component graphics and annotation text, and confirms the confidence level of component material properties through a multi-point sampling verification mechanism.

[0089] The multi-dimensional consistency collaborative verification and data output module is used to sequentially perform three levels of verification: single-image closed-loop self-check, multi-image association integration, and total quantity closed-loop approval and offsetting. It performs full-link logical consistency verification and error correction on the identification results and outputs a full-scale structured dataset of verified architectural drawings.

[0090] The specific working methods of the drawing preprocessing and multi-scale semantic block perception module, the neural symbol structured reconstruction module of heterogeneous tables, the multimodal detailed drawing deconstruction and cross-modal alignment module, and the multi-dimensional consistency collaborative verification and data output module have been described in detail in the above-mentioned method for multimodal recognition and structured processing of architectural drawings, and will not be repeated here in this embodiment.

[0091] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 3 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 3As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement any of the methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0092] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0093] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0094] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0095] This invention also provides a computer-readable medium. The computer-readable medium stores a computer program, which, when executed by a processor, implements the steps of any of the methods described in the above embodiments. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0096] This invention also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described method.

[0097] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0098] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0099] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0100] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0101] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0102] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0103] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0104] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0105] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0106] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A method for multimodal recognition and structured processing of architectural drawings, characterized in that, include: S100. Standardize and preprocess the input engineering drawings, perform full-domain scanning using a visual model with rotation bounding box perception capability, correct the drawing space coordinate system, and then extract and isolate four types of regions of interest: table area, detail area, annotation area, and standard drawing frame area through a deep semantic segmentation network. S200. Perform optical character recognition on the table area to obtain the text set and corresponding spatial coordinates. Correct the text through a coordinate correction algorithm. Input the corrected text, standardized coordinates and spatial adjacency matrix into the large language model to complete row and column topology relationship reasoning and engineering semantic error correction, and output standardized table structured data. S300. Visual geometric features and textual semantic features are extracted from the detailed drawing area respectively. The spatial dimension and semantic dimension are aligned in both directions through a multimodal cross-attention mechanism. The association mapping between the component graphics and the annotation text is established. The confidence level of the component material properties is confirmed through a multi-point sampling verification mechanism. S400 sequentially performs three levels of verification: single-image closed-loop self-check, multi-image association and integration, and total quantity closed-loop approval and offsetting. It performs full-link logical consistency verification and error correction on the recognition results and outputs a full-scale structured dataset of verified architectural drawings.

2. The method according to claim 1, characterized in that, The specific steps for implementing S100 include: S101. Receive engineering drawings in PDF, DWG or image format, perform Gaussian filtering for noise reduction, bilinear interpolation for scale normalization, and convert them into a standard resolution grayscale tensor image; S102. Input the grayscale tensor image into the visual model based on the feature pyramid network and the rotating bounding box detection mechanism, extract the topological anchor points of the drawing, and complete the absolute space coordinate system correction; S103. Perform pixel-by-pixel semantic labeling on the corrected image using a Transformer-based semantic segmentation network, and output region masks for the table area, detail area, annotation area, and standard frame area.

3. The method according to claim 1, characterized in that, The coordinate correction algorithm in S200 is specifically as follows: a linear correction algorithm based on maximizing the variance of projection density is used to calculate the global rotation angle θ, and to perform rotation transformation correction on all text coordinate points. The transformation formula is: Where (xi,yi) are the original text center coordinates, and (xi′,yi′) are the corrected coordinates. The rotation transformation makes the variance of the projected coordinates of the text in the same row / column converge to a local minimum.

4. The method according to claim 1, characterized in that, The specific methods for topological relation reasoning in the large language model of S200 include: Construct an input sequence containing semantic hints for engineering tables. The sequence includes corrected text, standardized coordinates, and spatial relative adjacency matrices. The large language model performs engineering common sense error correction and probabilistic reasoning of row and column topological relationships in the vector space. It automatically identifies merged cells and multi-level headers, outputs a symbolic description of the table topological relationships, and then generates standardized JSON-formatted table structured data through the symbolic rendering module.

5. The method according to claim 1, characterized in that, The specific steps for implementing S300 include: S301. Extracting multimodal features: Visual branch extraction of the geometry, size boundary lines, and spatial geometric tensors pointing to the leader lines of the detailed image. The text branch extracts semantic feature tensors from detail annotations, numbers, and material lists. ; S302. Perform cross-modal alignment: This is achieved through a multimodal cross-attention mechanism. and Perform spatial and semantic bidirectional alignment to establish a high-confidence association mapping between component graphic features and corresponding labeled text; S303. Perform multi-point sampling verification: Perform feature sampling on multiple spatial points of the detailed drawing leader line, the main body of the large detail drawing, and the annotation text. When the matching degree of the sampling results meets the majority voting threshold or the engineering logic verification rules, confirm the component material attributes and write them into the structured database.

6. The method according to claim 1, characterized in that, The specific methods for single-map closed-loop self-testing in S400 include: The physical attributes of the components extracted from the detailed drawing area are semantically aligned and logically compared with the material list data reconstructed from the table area on the same page. The verification indicators include the matching degree of the number, the equivalence of geometric specifications, and the rationality of the physical tolerance of the quantity calculation. When a conflict is detected, the abnormal context and image slices are input into the large language model for engineering common sense review. High-confidence misidentification is automatically corrected, conflict markers are designed and recorded in the single drawing verification list.

7. The method according to claim 1, characterized in that, The specific methods for multi-drawing association and integration in S400 include: summarizing the structured data of all distributed drawings in the entire project, establishing a directed graph topology network among the drawing data based on the correspondence between drawing index symbols and detail drawing numbers, and constructing a unified digital foundation for the project.

8. The method according to claim 1, characterized in that, The specific methods for total closed-loop approval hedging in S400 include: The total sum of components and categories at the bottom of the statistical project's digital base. Nominal total amount compared to the project's top-level master material summary table Perform hedging verification and calculate the difference value. ,in = - When ΔQ≠0 or component parameter level conflicts occur, the coordinates of the corresponding sub-graphs, cells or details are accurately located, and a visual full-project verification report with highlighted marks and jump links is generated.

9. A multimodal recognition and structured processing system for architectural drawings, employing any one of the methods in claims 1-8, characterized in that, include: The system includes a drawing preprocessing and multi-scale semantic segmentation perception module, a neural symbol structured reconstruction module for heterogeneous tables, a multimodal detail map deconstruction and cross-modal alignment module, and a multi-dimensional consistency collaborative verification and data output module; among which: The drawing preprocessing and multi-scale semantic block perception module is used to perform standardized preprocessing on the input engineering drawings. It performs full-domain scanning through a visual model with the ability to perceive rotating bounding boxes, corrects the spatial coordinate system of the drawings, and then extracts and isolates four types of regions of interest: table area, detail area, annotation area, and standard drawing frame area through a deep semantic segmentation network. The neural symbol structure reconstruction module for heterogeneous tables is used to perform optical character recognition on the table area, obtain the text set and corresponding spatial coordinates, correct the text through a coordinate correction algorithm, input the corrected text, standardized coordinates and spatial adjacency matrix into a large language model, complete row and column topological relationship reasoning and engineering semantic error correction, and output standardized table structured data. The multimodal detail map deconstruction and cross-modal alignment module is used to extract visual geometric features and textual semantic features from the detail map area respectively. It achieves bidirectional alignment of spatial and semantic dimensions through a multimodal cross-attention mechanism, establishes an association mapping between component graphics and annotation text, and confirms the confidence level of component material properties through a multi-point sampling verification mechanism. The multi-dimensional consistency collaborative verification and data output module is used to sequentially perform three levels of verification: single-image closed-loop self-check, multi-image association integration, and total quantity closed-loop approval and offsetting. It performs full-link logical consistency verification and error correction on the identification results and outputs a full-scale structured dataset of verified architectural drawings.

10. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 8.

Images