DWG drawing text information query method and system based on multi-modal large model

Abstract

This invention discloses a method and system for querying DWG drawing text information based on a multimodal large model. The method relates to the field of DWG drawing text information query technology and includes the following steps: data parsing and multimodal representation construction, multimodal large model semantic understanding and reasoning, structured output and post-processing, and backup output assurance. This invention parses DWG files, extracts text and its coordinates, generates a structured description, inputs this description and user query commands into a multimodal large model for semantic matching and reasoning, performs structured transformation, logical verification, and knowledge base association on the output results, and finally generates the query results. When text parsing fails, a backup reasoning mechanism of the large model is activated to ensure system availability, improving the adaptability of DWG drawing text information query and solving the problem of low adaptability in existing technologies.

Description

Technical Field

[0001] This invention relates to the field of DWG drawing text information query technology, and in particular to a method and system for querying DWG drawing text information based on multimodal large models. Background Technology

[0002] During the drawing loading and preprocessing stage, users open the DWG (Drawing) file using CAD (Computer-Aided Design) software or a lightweight viewing tool (such as Autodesk Viewer). First, the drawing is cleaned up, for example, by using the "PURGE" command to delete redundant data and ensuring that text styles are displayed correctly to avoid garbled characters due to missing fonts. Then, the software's query function is used to enter the core query stage. Users can select the query scope according to their needs: for a global search, "FIND" can be entered in the command line, and keywords (such as the device number "EQ-101") can be entered in the pop-up dialog box. The software will scan all single-line text, multi-line text, and block attributes in the entire drawing model space and layout space, and display the results in a list. Clicking on a result will locate the corresponding element. For targeted queries, the "Quick Selection" function can be used to quickly filter specific categories of text information by setting filter conditions (such as layer, color, and text height).

[0003] In DWG-based collaborative environments, textual information is often linked to external databases or project management platforms (such as Vault). For example, equipment labels in drawings may be associated with an SQL (Structured Query Language) database containing procurement status and maintenance records. During queries, dynamic data retrieval can be achieved through APIs (Application Programming Interfaces), transforming static drawing information into dynamic assets. Furthermore, for batch queries of large numbers of drawings, existing technologies such as AutoCAD scripts or .NET API programming can automate the process. By writing programs to traverse multiple DWG files, extract textual information, and export it to Excel or XML formats, it facilitates asset management and data analysis. In cloud collaboration platforms (such as BIM 360), drawing textual information can even be linked to issue tracking and task assignment, allowing query results to directly drive workflows.

[0004] For example, Chinese invention patent application CN118410196B discloses a method, system, and apparatus for identifying drawing labels, which includes: establishing a drawing label database, which includes the key field names and key field values ​​of different drawings; parsing the target drawing to obtain text information in the target drawing; determining the query field from the drawing label database based on the text information; and determining the target drawing label information based on the query field and the text information.

[0005] For example, Chinese invention patent application CN115269633A discloses a method for intelligently querying goods based on CAD drawings, which includes: classifying imported drawings, automatically identifying various components in the drawings using a convolutional neural network, and matching them to a product library according to the classification granularity; identifying the components and text information of the products separately; and matching each component with products in the product library based on its characteristics and corresponding details and annotation information to generate a search list.

[0006] However, in the process of implementing the inventive technical solution in the embodiments of this application, it was found that the above-mentioned technology has at least the following technical problems: In existing technologies, firstly, the operation threshold is high and the process is complex. Traditional methods require users to be proficient in operating professional CAD software and to understand its complex interactive interface and query commands (such as "FIND" or "Quick Select"). For non-professional users or collaborators who only need to query simple information, this requirement for professional knowledge and operational skills constitutes a significant barrier to use, limiting the flow of CAD drawing information and the efficiency of collaboration.

[0007] Secondly, existing solutions lack intelligence and adaptability. Traditional queries rely on precise keyword matching or pre-defined element attributes (such as layers and colors) filtering, failing to understand the vague or comprehensive query intent expressed by users in natural language. When drawing content is complex, text formats are inconsistent, or information is nested within complex block attributes, traditional methods often require users to perform multiple, tedious manual filtering and locating steps, resulting in low efficiency.

[0008] Furthermore, current technologies lack the ability to deeply integrate and understand the multimodal information in drawings. DWG files are essentially complex data packages containing multiple information modalities, such as geometric vectors, text annotations, and layer structures. Traditional text extraction methods typically process text content in isolation, severing the connection between the text and its spatial location (coordinates) and related graphics within the drawing. This results in "flattened" extracted information, losing its semantic relevance within the original design context, making it difficult to support in-depth queries and analyses that require combining spatial location and semantics.

[0009] Finally, existing solutions have limited generalization and scalability. Many existing information extraction tools or scripts are custom-developed for specific types of drawings (such as standard drawing frames for a particular profession). Their rule engines are prone to failure when faced with different industry standards, design institute drafting habits, or changes in drawing versions. They require a significant investment of manpower for rule maintenance and adaptation, and cannot serve as a unified framework to handle diverse input conditions and query needs. This results in low adaptability of DWG drawing text information queries. Summary of the Invention

[0010] This application provides a method and system for querying DWG drawing text information based on a multimodal large model. This solves the problem in the prior art where it is difficult to use a unified framework to handle diverse drawing inputs and flexible user query needs, resulting in low adaptability of DWG drawing text information query. This improves the adaptability of DWG drawing text information query.

[0011] On the one hand, a method for querying DWG drawing text information based on a multimodal large model is provided, including the following steps: Step 1, using a professional parsing library to parse the DWG file and extract the text information in the file, the text information includes descriptions, annotations, labels, and title block text, and simultaneously obtaining the insertion point coordinates corresponding to each text information, determining the position parameters of the text information through coordinate mapping calculation, and associating the text information with the corresponding position parameters to form a text dataset; Step 2, inputting the structured natural language description constructed in Step 1 as context information into the multimodal large model, and simultaneously receiving the query command input by the user, the multimodal large model utilizes its visual understanding, text understanding, and reasoning capabilities to perform a query on the structured natural language description corresponding to the text information. Step 1: Compare and analyze the quantitative information with the user's query command, and perform inference operations according to the preset structured output format requirements; Step 2: Parse the inference results output by the multimodal large model and convert them into structured data in the form of a dictionary or database record; Perform logical verification on the structured data based on CAD domain rules to check the rationality of the extracted information; Associate and match the verified structured data with relevant atlas specification knowledge bases to enrich the attribute information of the structured data, and output structured results that meet the user's query requirements; Step 3: If the parsing and extraction of DWG file text information in Step 1 fails, infer the text information to be queried and generate inferred results as backup output to ensure the overall availability of the query method.

[0012] On the other hand, a DWG drawing text information query system based on a multimodal large model is provided, including: a data parsing and multimodal representation construction module, a multimodal large model semantic understanding and reasoning module, a structured output and post-processing module, and a backup output guarantee module. The data parsing and multimodal representation construction module is used to parse DWG files using a professional parsing library, extracting text information from the files. This text information includes descriptions, annotations, labels, and title blocks, etc., and simultaneously obtains the coordinates of the insertion points corresponding to each piece of text information. The positional parameters of the text information are determined through coordinate mapping calculation, and the text information is associated with the corresponding positional parameters to form a text dataset. The multimodal large model semantic understanding and reasoning module is used to input the structured natural language description constructed in step one as contextual information into the multimodal large model, and simultaneously receives query commands input by the user. The system utilizes its visual understanding, text understanding, and reasoning capabilities to compare and analyze the vector information corresponding to the structured natural language description with the user's query instructions, and performs reasoning operations according to the preset structured output format requirements. The structured output and post-processing module parses the reasoning results output by the multimodal large model, converting them into structured data in dictionary or database record format. It performs logical verification on the structured data based on CAD domain rules to check the rationality of the extracted information. It then associates and matches the verified structured data with relevant atlas and specification knowledge bases to enrich the attribute information of the structured data and outputs structured results that meet the user's query requirements. The backup output guarantee module, if the DWG file text information parsing and extraction fails in step one, infers the text information to be queried and generates a predicted result as a backup output to ensure the overall availability of the query method.

[0013] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: 1. By transforming traditional manual queries relying on professional CAD software commands into intelligent queries based on natural language interaction, and by parsing drawing text and coordinate information into structured multimodal representations, and leveraging the semantic understanding capabilities of large-scale models, non-professional users can directly obtain target information through simple natural language commands (such as "find all descriptions of rebar usage") without needing to learn complex software operations. Simultaneously, the backup output mechanism of this invention can still provide reasonable inferences when partial information parsing fails, ensuring the system's availability and robustness. This effectively solves the problems of high professional knowledge requirements and cumbersome processes inherent in traditional methods, greatly improving the flow and collaboration efficiency of drawing information.

[0014] 2. Through deep semantic reasoning using a multimodal large-scale model, the system understands the user's fuzzy and comprehensive query intent. By calculating the semantic similarity between the query command and the drawing text vector, and combining this with a comprehensive judgment based on the coordinate context, it can accurately locate deeply related information. Furthermore, the post-processing stage incorporates CAD domain rule validation and association with the atlas and specification knowledge base. This not only performs logical error correction on the extracted results (such as unit and numerical range validation), but also automatically supplements extended attributes such as standard codes and material specifications. This makes the structured output not only accurate but also more complete and valuable for engineering applications, achieving a leap from "finding text" to "understanding and applying information."

[0015] 3. By leveraging the general understanding capabilities of a pre-trained large model and its structured multimodal representation of drawings (text + coordinates), the solution can adapt to DWG drawings from different industries and with different drafting standards. Only the post-processing domain rule base and knowledge base need to be adjusted to quickly adapt to new application scenarios. This decoupled design, where the parsing layer handles general data transformation and the understanding layer relies on the general capabilities of the large model, makes the system a unified framework capable of handling diverse inputs. This avoids the high costs of customizing and maintaining rule engines for each drawing type, laying a technological foundation for large-scale, cross-domain intelligent applications of engineering drawings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 Flowchart of the method for querying text information of DWG drawings based on multimodal large models provided in this application embodiment; Figure 2 A flowchart illustrating the backup reasoning and completion mechanism of the DWG drawing text information query method based on a multimodal large model provided in this application embodiment; Figure 3 This is a schematic diagram of the structure of the DWG drawing text information query system based on a multimodal large model provided in this application embodiment. Detailed Implementation

[0018] The technical solution provided in this application will now be described in conjunction with the accompanying drawings.

[0019] To facilitate understanding of the embodiments of this application, the following points will be explained first: First, in this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates an "or" relationship between the preceding and following related objects, but it does not exclude the possibility of indicating an "and" relationship; the specific meaning can be understood in context. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c; a and b; a and c; b and c; or a and b and c. Here, a, b, and c can be single or multiple.

[0020] Second, the use of prefixes such as "first" and "second" in this application is solely for the purpose of distinguishing and describing different things belonging to the same category, and does not constrain the order, size, or quantity of things. For example, "first message" and "second message" are simply different messages, and there is no chronological, size, or priority relationship between them.

[0021] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0022] like Figure 1The diagram shows a flowchart of a DWG drawing text information query method based on a multimodal large model provided in this application embodiment. The method includes the following steps: Step 1: Parsing the DWG file using a professional parsing library to extract text information from the file. The text information includes descriptions, annotations, labels, and title blocks, etc. Simultaneously, the insertion point coordinates corresponding to each piece of text information are obtained. Position parameters of the text information are determined through coordinate mapping calculation, and the text information is associated with its corresponding position parameters to form a text dataset. Step 2: The structured natural language description constructed in Step 1 is input as context information into the multimodal large model. Simultaneously, the user-input query command is received. The multimodal large model utilizes its visual understanding, text understanding, and reasoning capabilities to process the structured natural language description. The process involves four steps: First, comparing and analyzing the vector information corresponding to the query command with the user's query instructions, and performing inference operations according to the preset structured output format requirements. Second, parsing the inference results output by the multimodal large model and converting them into structured data in the form of a dictionary or database record. Third, logically verifying the structured data based on CAD domain rules to check the rationality of the extracted information. Fourth, associating and matching the verified structured data with relevant atlas specification knowledge bases to enrich the attribute information of the structured data and outputting structured results that meet the user's query requirements. Fifth, if the parsing and extraction of DWG file text information fails in step one, inferring the text information to be queried and generating inferred results as backup outputs to ensure the overall availability of the query method.

[0023] It should be noted that in step one, the DWG file is deeply parsed using a professional parsing library to extract all text information, including descriptions, annotations, labels, and title blocks, and simultaneously obtains the corresponding insertion point coordinates. Coordinate mapping calculations are then used to form a text dataset with clearly defined positional parameters and a clear connection between text and graphics. This step not only transforms unstructured drawing information into machine-readable standardized data but also provides crucial spatial context for subsequent semantic understanding through precise coordinate association, fundamentally improving the completeness and structure of information extraction. In step two, the structured description containing text content and coordinate information is used as context and input along with the user's natural language query command into a multimodal large model. The model utilizes its visual understanding and text reasoning capabilities to perform deep semantic comparison and analysis of the vectorized drawing information and user commands, achieving a leap from "keyword matching" to "intent understanding." This step directly addresses the pain points of traditional query methods' insufficient intelligence and inability to understand complex or ambiguous query intents, enabling non-professional users to conveniently obtain target information through natural language. In step three, the inference results output by the model are parsed and converted into structured data such as dictionaries or database records. Further, logical verification is performed based on CAD domain rules (such as numerical rationality and unit consistency), and the results are matched with the atlas and specification knowledge base to supplement attributes such as standard codes and material specifications. This series of post-processing operations not only ensures the accuracy and engineering practicality of the output results but also significantly enriches the added value of the information through knowledge base association, enabling the query results to be directly used for subsequent professional work such as quantity surveying and compliance review. In step four, if text parsing and extraction fail, the system will utilize the visual understanding and common-sense reasoning capabilities of the multimodal large model, combined with the drawing context, to make reasonable inferences about the missing information and generate backup results. This mechanism effectively enhances the system's robustness in cases of incomplete drawings or parsing anomalies, ensuring the continuous availability of the query service and improving the overall user experience.

[0024] In this embodiment, by deeply fusing the text and spatial information of the drawings through multimodal large models and introducing rigorous post-processing and backup mechanisms, a smart, highly available DWG drawing text information query solution with results that can be directly applied in engineering is finally realized.

[0025] Furthermore, determining the positional parameters of text information through coordinate mapping calculation specifically includes: extracting the geometric attribute information of the text entity, wherein the geometric attribute information includes at least the text insertion point coordinates and the text bounding box defined by the minimum point coordinates and the maximum point coordinates; performing mapping calculations in the global coordinate system of the DWG drawing based on the insertion point coordinates and the bounding box coordinates to accurately determine the position, occupied area, and layout relationship of each text information in the drawing space; after forming the text dataset, the step of vectorizing the text dataset is included: inputting the text content and its associated positional parameters in the text dataset into a vectorization model; the vectorization model converts the text content into numerical vectors and generates a mapping relationship table containing text ID, vector value, coordinate position, and text content, thereby transforming the unstructured DWG drawing text information into a structured, multimodal representation that can be understood by a multimodal large model.

[0026] In this embodiment, firstly, the geometric attribute information of text entities is extracted from the DWG drawing. This information includes at least the coordinates of the text insertion point and the text bounding box defined by its minimum and maximum point coordinates. Subsequently, based on the aforementioned insertion point coordinates and bounding box coordinates, a precise mapping calculation is performed in the global coordinate system of the DWG drawing to uniquely determine the absolute position of each text in the drawing space, its occupied area, and its layout relationship with surrounding elements. This step not only achieves precise quantification of the text's spatial position but, more importantly, establishes a strong association between the text content and its spatial context, laying the foundation for subsequent understanding of the semantic role and function of the text on the drawing surface. The text content and its associated positional parameters from the aforementioned text dataset are input into a pre-trained vectorization model (such as a text embedding model). This model converts the text content into high-dimensional numerical vectors (i.e., semantic vectors) and generates a structured mapping table that records the text ID, corresponding vector value, coordinate position, and original text content. This step fundamentally transforms the unstructured, scattered text information of the DWG drawing into a structured, machine-understandable, multimodal representation (integrating semantic vectors and spatial coordinates). This representation method enables multimodal large models to simultaneously "understand" the semantics of text and its spatial meaning in drawings, thereby responding to complex queries that combine spatial and semantic elements, such as "find all annotations in the upper left corner area" or "locate the descriptions adjacent to a certain component." This greatly expands the dimensions and intelligence of queries, breaking through the limitations of traditional methods that can only perform pure text matching.

[0027] Furthermore, the text dataset includes text content, coordinate data, font size, number of characters, and coordinates of text boundary extreme points; based on a vectorization model, the text dataset is converted into a structured natural language description, and the structured natural language description establishes a mapping relationship between vector values ​​and text content and coordinate information to construct multimodal representation data adapted to a multimodal large model.

[0028] It's important to explain that, firstly, the constructed text dataset is a structured collection containing multidimensional attributes. It not only includes the core text content but also precisely records coordinate data, font size, character count, and the coordinates of text boundary extrema defined by minimum and maximum points. This design ensures that the information extracted from the original drawings is complete and rich in metadata, providing a rich feature foundation for subsequent deep processing. Subsequently, the text dataset is converted into a structured natural language description adapted to large models based on a vectorization model. Specifically, the text content in the dataset is input into a vectorization model (such as the embedding layer of models like BERT or GPT) to generate high-dimensional numerical vectors (i.e., vector values) that represent its semantics. Simultaneously, metadata such as coordinate information and font size is transformed into standardized natural language description fragments (e.g., "a label located at coordinates (1024, 768) with a font size of 2.5"). Finally, by establishing a mapping table between vector values, text content, and coordinate information, a unified multimodal representation data is constructed. In this representation, semantic information is encoded as vectors, and spatial and stylistic information is expressed as structured text, together forming a "language" that the large model can parse.

[0029] In this embodiment, the scattered and heterogeneous graphic and textual information in DWG drawings was successfully "translated" into a format that a multimodal large model can natively understand and handle. The model can not only perform precise semantic matching through vectors, but also understand the visual context of the text (such as position and size) through structured natural language descriptions. This allows users to initiate complex queries that deeply integrate semantic, spatial, and visual attributes, such as "find all bolded descriptions in the lower right corner of the drawing" or "summarize all technical parameters related to components near a specific coordinate point." This achieves a significant leap in the intelligent and refined mining capabilities of drawing information, which is impossible with traditional keyword-based or simple attribute-based query methods.

[0030] Furthermore, in step two, the step of inputting the structured natural language description constructed in step one as context information into the multimodal large model specifically includes: vectorizing the structured description text to generate a prompt word template that meets the input interface requirements of the multimodal large model. The prompt word template includes at least three parts: instruction, context data, and user query. The instruction is used to define the model task as drawing information query and specify the output format. The context data consists of the structured description text, and the user query is the received natural language query instruction. The combined prompt word template is then input into the multimodal large model.

[0031] It should be noted that the specific process of inputting the structured description into the multimodal large model in step two is the key interface design for achieving efficient and accurate queries in this invention. This process first carefully vectorizes the structured natural language description generated in step one, assembling it into a prompt word template that conforms to the input specifications of the large model. This template is a structured text instruction block, explicitly containing at least three core parts: First, the "instruction" part clearly defines the model's task as "DWG drawing information query" and specifies that the output should be in a structured format such as JSON. This fundamentally guides the model's role and behavior, ensuring that its output meets engineering application requirements. Second, the "context data" part embeds complete structured drawing information containing text semantic vectors and coordinate descriptions, providing the model with all the necessary "factual" background for reasoning. Finally, the "user query" part directly fills in the natural language instructions input by the user.

[0032] In this embodiment, the complex problem of querying CAD drawing information is transformed into a context-based reading comprehension and instruction following task, which is well-suited to large-scale models. The model does not require retraining for the CAD domain; it can leverage its inherent powerful semantic understanding and logical reasoning capabilities to accurately locate and answer the questions posed in the "user query" section within the provided "contextual data" (i.e., formatted drawing information). This prompting engineering approach significantly lowers the barriers and costs of technology implementation, enabling rapid and efficient transfer of general-purpose large-scale models to specialized fields. This allows the system to flexibly and intelligently respond to diverse user query needs.

[0033] Furthermore, the vector information corresponding to the structured natural language description is compared and analyzed with the user's query command. Specifically, the multimodal large model, based on its pre-trained cross-modal embedding space, maps the semantic content of the user's input query command into a query vector; the semantic similarity between the query vector and the numerical vectors corresponding to each text content in the structured description text is calculated; the semantic similarity scores are ranked, and combined with contextual analysis of text coordinate information and type information, comprehensive reasoning is performed to locate and filter the target text information that is most semantically relevant to the user's query command.

[0034] It should be understood that this process begins with the multimodal large model utilizing the cross-modal embedding space obtained through its pre-training: the model first maps the semantic content of the user's input natural language query instruction (such as "find all descriptions about concrete strength") into a high-dimensional "query vector." This step essentially transforms the user's query intent into a computable mathematical representation. Next, the core semantic matching calculation is initiated: by calculating the cosine similarity and other metrics between the aforementioned "query vector" and the "numerical vector" (i.e., the semantic vector of the text content) generated for each text block within the drawing in step one, a semantic relevance score between each piece of text information and the user's query is obtained. However, this invention does not stop at simple vector similarity ranking. The key improvement lies in introducing comprehensive contextual reasoning: the system further combines the text's coordinate information (to determine spatial relevance) and type information (such as distinguishing between "annotation" and "title") to weight and correct the initial semantic similarity ranking results. For example, the final ranking of an annotation that is semantically highly relevant but located in an area unrelated to the drawing may be adjusted.

[0035] In this embodiment, the query process is upgraded from traditional "string matching" or "fixed attribute filtering" to a fusion reasoning based on deep "semantic understanding" and "contextual association." The system can not only find text that literally contains "concrete strength," but also locate semantically similar but differently expressed content (such as "C30 grade concrete requirements"), and combine spatial location to exclude irrelevant information, accurately filtering out the target text that the user truly cares about. This fundamentally solves the core pain points of traditional methods—the inability to understand user intent, handle fuzzy queries, and synonymous expressions—achieving a qualitative leap from "mechanical search" to "intelligent question answering," significantly improving query accuracy and user experience.

[0036] Furthermore, the data is converted into structured data in the form of a dictionary or database record. Specifically, this includes: parsing the inference results output by the multimodal large model, identifying and extracting the text content, associated coordinate information, and entity type labels inferred by the model contained in the inference results; according to the preset database table structure, using the extracted text content as values ​​and its associated coordinate information and entity type labels as key attributes, assembling them to generate standardized key-value pair structured data, thus completing the conversion to structured data.

[0037] It should be understood that, firstly, the inference results, typically natural language or semi-structured text, output by the multimodal large model are deeply analyzed. Using rule-based or lightweight models, three core elements are precisely identified and extracted: the target text content itself, the associated coordinate information (such as the X and Y values ​​of the insertion point), and the entity type labels inferred by the model from the context (such as "material specifications," "dimensions," "drawing number," etc.). Next, a standardized data assembly stage is entered: based on a pre-designed database table structure or JSON schema for the drawing information, the text content extracted in the previous step is used as the "value," while its coordinates and entity type information are used as the "key" or attribute fields. Through this mapping relationship, standard key-value pair structured data, such as a complete record, is assembled. This step completes the final transformation from non-standardized model output to structured data that can be seamlessly read, exchanged, and processed by machines.

[0038] In this embodiment, the results of intelligent queries are ensured to be no longer isolated, readable text fragments, but rather data assets that can be directly imported into databases, integrated with BIM systems, or used for batch statistical analysis. For example, all extracted "material annotations" can be automatically summarized into a bill of materials, and "dimension annotations" with coordinates can be used for automatic verification. This enhances the engineering practicality and automation potential of the query results, avoiding the tediousness and errors of manual data entry.

[0039] Furthermore, the specific steps for logically verifying the structured data based on CAD domain rules are as follows: Perform at least one verification on the structured data according to a preset CAD domain rule base: perform unit consistency verification and numerical range rationality verification on text content containing numerical values; perform matching degree verification on text content representing component types with standard component names in the preset CAD domain rule base; perform semantic relevance and logical conflict verification on text content from different sources but with similar spatial coordinates, and mark the data that passes the verification as valid results.

[0040] It should be understood that during the logical verification phase, intelligent "quality inspection" is performed on the initially extracted structured data. This step, based on a pre-set CAD domain rule base, performs multi-dimensional and targeted verification of the data, specifically executing at least three types of checks: First, for text content containing numerical values, unit consistency verification (ensuring correct unit symbols) and numerical range rationality verification are performed. This effectively intercepts abnormal data caused by identification errors or drawing errors. Second, for text content representing component types, a matching degree verification is performed with standard component names in the rule base. This automatically corrects colloquialisms or abbreviations and standardizes them to standard terms such as "frame beam-1". Finally, spatial-semantic association verification is implemented: for different texts with similar spatial coordinates, the system analyzes their semantic association and detects whether there are obvious logical conflicts.

[0041] Furthermore, the key entity text content in the structured data is used as the query key to perform a search and matching in the pre-built atlas specification knowledge base. If the match is successful, one or more extended attribute information from the corresponding standard code, material specifications, and construction process supplementary instructions are obtained from the pre-built atlas specification knowledge base and associated with and attached to the structured data to generate a more complete enhanced query result.

[0042] It's important to note that the knowledge base association enhancement phase represents a leap from "extracting information" to "enriching and empowering information." First, the key entity text content from the validated structured data is used as the precise query key for retrieval and matching within a pre-built domain knowledge base containing a vast amount of standard atlases, design specifications, and product catalogs. Upon successful matching, the system automatically triggers the information enhancement process: precisely retrieving one or more extended attribute information associated with the entity from the knowledge base. This information typically includes national or industry standard codes, detailed material specifications, recommended construction techniques and methods, and even supplier reference information. Subsequently, the system seamlessly associates these authoritative extended attributes as new fields and appends them to the original structured data record.

[0043] In this embodiment, the "information density" and "readiness" of the query results have been altered. Users receive not just raw text identified from drawings, but an "enhanced information package" containing standard references, technical parameters, and process requirements. For example, a query for "column" returns results including not only its drawing number and dimensions, but also potentially automatically linked concrete strength grade standards, steel reinforcement mechanical property tables, and even links to detailed node construction drawings. This allows the query results to directly serve downstream professional work such as material procurement, construction briefings, cost estimation, and compliance review, eliminating the need for engineers to manually consult voluminous specification manuals. This process significantly improves the efficiency of design data flow and knowledge reuse, activating static drawing information into dynamic, computable engineering knowledge, and significantly enhancing the digital management and collaborative efficiency of projects.

[0044] It should be noted that, as Figure 2 The diagram shows the flowchart of the backup reasoning and completion mechanism of the DWG drawing text information query method based on a multimodal large model provided in this application embodiment. Specifically, it is as follows: First, the "monitoring of text extraction process" is started, and the "extraction success" judgment node is used for verification. If the extraction is successful, the regular result is generated directly. If the extraction fails or partially fails, the backup reasoning process in step two is entered. Incomplete information such as coordinates and graphic features is collected first, and then the multimodal large model performs reasoning and completion based on this incomplete information to generate a labeled speculative backup result. Finally, the regular result or speculative backup result is summarized and output.

[0045] Furthermore, during the text information extraction process in step one, the extraction success rate is monitored in real time. If the text information extraction for an entity fails or the extracted content is empty, a backup processing mechanism is activated. The spatial coordinates of the failed entity, contextual graphic features, or adjacent successfully extracted text content are input as incomplete information into the multimodal large model. The multimodal large model receives the incomplete information and, based on its pre-trained visual understanding and common sense reasoning capabilities, combined with the general context of CAD drawings, makes reasonable inferences and completes the missing text information. The inferred and completed content generated by the multimodal large model is used as a backup result, distinguished and marked from the regular results from the successful extraction step, and then output together for user reference, thereby ensuring overall usability in the case of partial information loss.

[0046] In this embodiment, during the text information extraction process in step one, the success rate of extraction for each graphic entity is monitored in real time. When the monitoring system detects that the text information extraction for a certain entity fails (e.g., due to font corruption, excessively deep block nesting) or the extracted content is empty, a backup processing flow is immediately triggered. The core of the backup processing lies in creatively utilizing the reasoning capabilities of the multimodal large model to complete the information. The system does not simply report an error or return a null value, but instead inputs the available "clues" for the failed entity, including its spatial coordinate information, surrounding graphic features (such as connected lines and fill patterns), and adjacent successfully extracted text content, as a set of "incomplete information" into the multimodal large model. Subsequently, based on its powerful visual understanding and engineering common sense reasoning capabilities obtained through pre-training, combined with its grasp of the general context of CAD drawings (such as architectural design and mechanical drawing), the large model comprehensively analyzes these clues and makes logical inferences and completes the missing text information. For example, for a graphic element with missing text but located next to a series of door components, the model may infer that its text is "M-1" (door number). Ultimately, the system explicitly labels the model-generated inferences as "backup inferences," distinguishing them from the more reliable "regular results" successfully extracted directly from the drawings, and outputs them to the user for reference. This mechanism produces a key technical effect: it ensures that even under suboptimal conditions such as incomplete drawing files, damaged information, or limitations in the parsing database, the system can still provide valuable and continuous services, rather than becoming completely ineffective. This greatly enhances the practicality and reliability of the method in complex engineering environments, guarantees the consistency of the user experience, and enables the intelligent query tool to maintain high availability when faced with drawing data of varying quality in reality.

[0047] like Figure 3The diagram shows the structure of a DWG drawing text information query system based on a multimodal large model provided in this application embodiment. The system includes: a data parsing and multimodal representation construction module, a multimodal large model semantic understanding and reasoning module, a structured output and post-processing module, and a backup output guarantee module. Specifically, the data parsing and multimodal representation construction module is used to parse DWG files using a professional parsing library, extract text information from the files, including descriptions, annotations, labels, and title bar text. It also obtains the coordinates of the insertion points corresponding to each piece of text information, calculates the positional parameters of the text information through coordinate mapping, and associates the text information with the corresponding positional parameters to form a text dataset. The multimodal large model semantic understanding and reasoning module is used to input the structured natural language description constructed in step one as contextual information into the multimodal large model, while... Upon receiving a query command input by the user, the multimodal large model utilizes its visual understanding, text understanding, and reasoning capabilities to compare and analyze the vector information corresponding to the structured natural language description with the user's query command, and performs reasoning operations according to the preset structured output format requirements. The structured output and post-processing module parses the reasoning results output by the multimodal large model, converting them into structured data in dictionary or database record format. It performs logical verification on the structured data based on CAD domain rules to check the rationality of the extracted information. It then associates and matches the verified structured data with relevant atlas and specification knowledge bases to enrich the attribute information of the structured data and outputs structured results that meet the user's query requirements. The backup output guarantee module, if the DWG file text information parsing and extraction fails in step one, infers the text information to be queried and generates a predicted result as a backup output to ensure the overall availability of the query method.

[0048] In this embodiment, firstly, the data parsing and multimodal representation construction module uses a professional parsing library to deeply parse DWG files, accurately extracting all text information, including descriptions and annotations, and their insertion point coordinates. It then establishes a text dataset linking the text and graphics through coordinate mapping. This module fundamentally transforms unstructured graphic data into machine-readable information rich in spatial context, laying a structured foundation for subsequent intelligent understanding. Secondly, the multimodal large-scale model semantic understanding and reasoning module receives user natural language commands and inputs the structured description generated by the previous module as context into the large-scale model. The model utilizes its cross-modal capabilities to perform deep semantic comparison and reasoning between the vectorized drawing information and user commands. This module directly addresses the pain points of traditional queries relying on keyword matching and failing to understand complex intents, achieving a leap from "mechanical search" to "intelligent question answering," significantly lowering the barrier to entry for non-professional users. Subsequently, the structured output and post-processing module parses the model output, converting it into standard formats such as dictionaries or database records. It then performs logical verification based on CAD domain rules (such as numerical rationality and unit consistency), and associates the results with the atlas specification knowledge base to supplement extended attributes such as standard coding. This module not only ensures the accuracy of the results but also significantly enhances the engineering practical value of the information through knowledge fusion, enabling the query results to be directly applied to professional scenarios such as quantity surveying and compliance review. Finally, the backup output guarantee module activates in real time when text extraction fails. Utilizing the visual and common-sense reasoning capabilities of the large model, it makes reasonable inferences about missing information based on spatial coordinates, adjacent text, and other clues, and outputs the marked inference results along with the regular results. This design ensures the system's continued availability and robustness when drawings are incomplete or parsing anomalies, guaranteeing a consistent user experience. The entire system, through modular pipeline operations, transforms complex professional queries into fluent natural language interaction. While improving efficiency and ensuring reliability, it realizes a value leap for drawing information from static data to dynamic, computable knowledge, providing a powerful tool for digital engineering collaboration.

[0049] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0050] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0051] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0052] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0053] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0054] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for querying text information in DWG drawings based on multimodal large models, characterized in that, Includes the following steps: Step 1: Use a professional parsing library to parse the DWG file and extract the text information in the file. The text information includes text such as descriptions, annotations, labels and title bars. At the same time, obtain the coordinates of the insertion point corresponding to each text information. Calculate the position parameters of the text information through coordinate mapping and associate the text information with the corresponding position parameters to form a text dataset. Step 2: The structured natural language description constructed in Step 1 is input as context information into the multimodal large model. At the same time, the query command input by the user is received. The multimodal large model calls on its visual understanding, text understanding and reasoning capabilities to compare and analyze the vector information corresponding to the structured natural language description with the user query command, and performs reasoning operations according to the preset structured output format requirements. Step 3: Parse the inference results output by the multimodal large model and convert them into structured data in the form of a dictionary or database records; perform logical verification on the structured data based on CAD domain rules to check the rationality of the extracted information; associate and match the verified structured data with relevant atlas specification knowledge bases to enrich the attribute information of the structured data and output structured results that meet the user's query requirements. Step four: If the DWG file text information parsing and extraction fails in step one, the text information to be queried is inferred, and the inferred result is generated as a backup output to ensure the overall availability of the query method.

2. The method for querying DWG drawing text information based on a multimodal large model as described in claim 1, characterized in that, The parameters for determining text information through coordinate mapping specifically include: Extract the geometric attribute information of the text entity, which includes at least the coordinates of the text insertion point and the text bounding box defined by the minimum and maximum point coordinates; Based on the coordinates of the insertion point and the coordinates of the bounding box, a mapping calculation is performed in the global coordinate system of the DWG drawing to accurately determine the position, area occupied, and layout relationship of each text information in the drawing space. After forming the text dataset, the step includes vectorizing the text dataset: The text content and its associated location parameters in the text dataset are input into a vectorized model. The vectorized model converts the text content into numerical vectors and generates a mapping table containing text ID, vector value, coordinate position and text content, thereby transforming the unstructured DWG drawing text information into a structured, multimodal representation that can be understood by a multimodal large model.

3. The method for querying DWG drawing text information based on a multimodal large model as described in claim 1, characterized in that, The text dataset includes text content, coordinate data, font size, number of characters, and coordinates of text boundary extreme points; The text dataset is converted into a structured natural language description based on a vectorization model. The structured natural language description establishes a mapping relationship between vector values ​​and text content and coordinate information, thereby constructing multimodal representation data that is adapted to a multimodal large model.

4. The method for querying DWG drawing text information based on a multimodal large model as described in claim 1, characterized in that, In step two, the step of inputting the structured natural language description constructed in step one as contextual information into the multimodal large model specifically includes: The structured description text is vectorized to generate a prompt word template that meets the requirements of the multimodal large model input interface. The prompt word template includes at least three parts: instruction, context data, and user query. The instruction is used to define the model task as drawing information query and specify the output format. The context data consists of the structured description text. The user query is the received natural language query instruction. The combined prompt word template is input into the multimodal large model.

5. The method for querying DWG drawing text information based on a multimodal large model as described in claim 1, characterized in that, The comparison and analysis of the vector information corresponding to the structured natural language description with the user's query command specifically includes: The multimodal large model, based on its pre-trained cross-modal embedding space, maps the semantic content of the user-input query command into a query vector; and calculates the semantic similarity between the query vector and the numerical vectors corresponding to each text content in the structured description text. Based on the semantic similarity scores, and combined with contextual analysis of text coordinate and type information, the target text information most semantically relevant to the user's query instruction is located and filtered out.

6. The method for querying DWG drawing text information based on a multimodal large model as described in claim 1, characterized in that, The structured data converted into dictionary or database record format specifically includes: The inference results output by the multimodal large model are analyzed, and the text content, associated coordinate information, and entity type labels inferred by the model are identified and extracted from the inference results. Based on the preset database table structure, the extracted text content is used as the value, and its associated coordinate information and entity type label are used as key attributes to assemble and generate standardized key-value pair structured data, thus completing the transformation into structured data.

7. The method for querying DWG drawing text information based on a multimodal large model as described in claim 1, characterized in that, The specific steps for performing logical verification on the structured data based on CAD domain rules are as follows: The structured data is validated according to at least one of the preset CAD domain rule bases: The text content representing the component type is checked for matching degree with the standard component names in the preset CAD domain rule base; Semantic relevance and logical conflict checks are performed on text content from different sources but with similar spatial coordinates, and data that passes the checks are marked as valid results.

8. Perform unit consistency verification and numerical range reasonableness verification on text content containing numerical values; as described in claim 1, the method for querying DWG drawing text information based on multimodal large models is characterized in that... The key entity text content in the structured data is used as the query key to perform retrieval and matching in the pre-built graph atlas specification knowledge base; If a match is successful, one or more extended attribute information from the corresponding standard code, material specifications, and construction process supplementary instructions are obtained from the pre-built atlas specification knowledge base, and then associated and attached to the structured data to generate more complete enhanced query results.

9. The method for querying DWG drawing text information based on a multimodal large model as described in claim 1, characterized in that, During the text information extraction process in step one, the extraction success rate is monitored in real time. If it is detected that the text information extraction for an entity fails or the extracted content is empty, a backup processing mechanism is activated, and the spatial coordinate information, contextual graphic features, or adjacent successfully extracted text content of the failed entity are input as incomplete information into the multimodal large model. The multimodal large model receives incomplete information and, based on its pre-trained visual understanding and common sense reasoning abilities, combined with the general context of CAD drawings, makes reasonable inferences and completes the missing text information. The speculative completion content generated by the multimodal large model is used as a backup result. It is then distinguished and marked from the regular results from the successful extraction step, and output together for user reference, thereby ensuring overall usability in the event of partial information loss.

10. A DWG drawing text information query system based on a multimodal large model, employing the DWG drawing text information query method based on a multimodal large model as described in any one of claims 1-9, characterized in that, It includes a data parsing and multimodal representation construction module, a multimodal large model semantic understanding and reasoning module, a structured output and post-processing module, and a backup output guarantee module: The data parsing and multimodal representation construction module is used to parse DWG files using a professional parsing library, extract text information from the files, including descriptions, annotations, labels, and title bars, and obtain the coordinates of the insertion points corresponding to each text information. The position parameters of the text information are determined through coordinate mapping calculation, and the text information is associated with the corresponding position parameters to form a text dataset. The multimodal large model semantic understanding and reasoning module is used to input the structured natural language description constructed in step one as context information into the multimodal large model, and at the same time receive the query command input by the user. The multimodal large model calls its visual understanding, text understanding and reasoning capabilities to compare and analyze the vector information corresponding to the structured natural language description with the user query command, and perform reasoning operations according to the preset structured output format requirements. The structured output and post-processing module is used to parse the inference results output by the multimodal large model and convert them into structured data in the form of a dictionary or database record; to perform logical verification on the structured data based on CAD domain rules and check the rationality of the extracted information; to associate and match the verified structured data with relevant atlas specification knowledge bases to enrich the attribute information of the structured data and output structured results that meet the user's query requirements. The backup output guarantee module is used to infer the text information to be queried if the DWG file text information parsing and extraction fails in step one, and generate the inferred result as backup output to ensure the overall availability of the query method.

Images