Drawing content question and answer method and device, electronic equipment and storage medium

Abstract

The invention discloses a drawing content question and answer method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: extracting a drawing image file generated based on drawing source file rendering, and obtaining drawing image features; performing structured analysis on the drawing source file to obtain drawing entity features; performing feature alignment on the drawing image features and the drawing entity features; generating cue words according to a user question and answer request input by a user for the drawing content, the drawing image features after feature alignment and the drawing entity features; and inputting the cue word into a large language model, and generating a reply content corresponding to the question and answer request. According to the invention, the generalization ability and question answering accuracy of drawing content questions and answers can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and in particular to a drawing content question and answer method and device, an electronic device, and a storage medium. BACKGROUND

[0002] With the development of information technology, digital processing and intelligent analysis of drawing content become increasingly important. In particular, in nuclear power engineering, power, engineering construction and other vertical fields, the demand for accurate extraction and understanding of drawing content is increasing. In actual application, users usually ask questions about drawing content with the help of intelligent systems to efficiently obtain the required information and improve drawing interpretation efficiency.

[0003] In the prior art, drawing content question and answer usually directly inputs drawing images or simple features thereof into a large language model to generate question and answer replies relying on the large language model. However, most large language models have limited training data, which mainly comes from Internet text. Even if some models introduce specific field drawing data for fine-tuning in the research and development stage, their application scope is still strictly limited to specific fields (such as only applicable to building structure drawings) or specific drawing styles. Moreover, different fields use different drawing specifications, and drawings in the same field also present high diversity due to design software, drawing habits and other factors. If the above method is used for drawing content question and answer, it will be difficult to understand the content of drawings in different fields or different styles, and some content in the drawings will be missed or misrecognized, resulting in incomplete and inaccurate extraction of drawing content, and thus low generalization ability and question and answer accuracy of drawing content question and answer. SUMMARY

[0004] The present disclosure provides a drawing content question and answer method, device, electronic device and storage medium. The main purpose is to improve the generalization ability and question and answer accuracy of drawing content question and answer.

[0005] According to a first aspect of the present disclosure, a drawing content question and answer method is provided, which comprises: extracting a drawing image file rendered based on a drawing source file to obtain drawing image features, and structurally analyzing the drawing source file to obtain drawing entity features; aligning the drawing image features and the drawing entity features, and generating a prompt word according to a user question and answer request input by a user for drawing content, the aligned drawing image features and the drawing entity features; inputting the prompt word into a large language model to generate reply content corresponding to the question and answer request.

[0006] Optionally, the structurally analyzing the drawing source file to obtain drawing entity features comprises: convert the drawing source file into a drawing exchange file in a preset standard format; inserting a decomposition operation into each block in the drawing exchange file, and adding each entity contained in the drawing exchange file to a preset model space; extracting key attributes of each entity in the preset model space, and inputting the key attributes into a graph node embedding generator of a corresponding entity type to generate a graph node corresponding to each entity; generating the drawing entity features according to the graph nodes.

[0007] Optionally, the generating the drawing entity features according to the graph nodes comprises: determining the association relationship between each entity based on the key attributes of each entity; determining the edge connection between each graph node based on the association relationship to construct a graph structure; performing information aggregation and feature extraction on the graph structure to output a context-enhanced graph node embedding; arranging the graph node embeddings in a first preset order of the corresponding entities in the drawing source file to form a graph node embedding sequence, and taking the graph node embedding sequence as the drawing entity features.

[0008] Optionally, the determining the edge connection between each graph node based on the association relationship to construct a graph structure comprises: initializing the edge connection between each graph node based on the association relationship; determining the number of connection edges of any graph node; if the number of connection edges of any graph node exceeds a preset threshold, removing part of the connection edges from the corresponding graph node through random sampling; constructing a graph structure based on the connection edges after the removal processing.

[0009] Optionally, the extracting the drawing image file to obtain drawing image features comprises: scaling the drawing image file according to a preset size specification; performing block processing on the scaled image to obtain a plurality of image blocks, and adding a corresponding position embedding to each image block; based on the position embedding, performing global context information interaction between each image block to output an image embedding corresponding to each image block; arranging the image embeddings in a second preset order in the drawing image file to form an image embedding sequence, and taking the image embedding sequence as the drawing image features.

[0010] Optionally, the aligning the drawing image features and the drawing entity features comprises: map the image feature and the entity feature to a feature embedding space of the large language model respectively to obtain an image feature embedding and an entity feature embedding, wherein the embedding dimensions of the image feature embedding and the entity feature embedding are respectively the same as a text embedding dimension of the large language model; The generating of the prompt word according to the drawing question requested by the user, the aligned image feature and the entity feature of the drawing, comprises: The user request embedding is converted from a user question and answer request input by the user for the drawing content; The user request embedding, the image feature embedding and the entity feature embedding are spliced according to a preset prompt template to generate a prompt word.

[0011] Optionally, before the extracting of the drawing image file based on the drawing source file to obtain the drawing image feature, the method further comprises: obtaining a drawing source file; visualizing and rendering the drawing source file into a drawing image file.

[0012] Optionally, the extracting of the drawing image file based on the drawing source file to obtain the drawing image feature, and the structured analysis of the drawing source file to obtain the entity feature of the drawing, comprises: extracting the drawing image file based on the drawing source file to obtain the drawing image feature through a preset image feature extractor, and performing structured analysis on the drawing source file through a preset drawing entity feature extractor to obtain the entity feature of the drawing; The feature alignment of the drawing image feature and the entity feature of the drawing comprises: aligning the drawing image feature and the entity feature of the drawing through a preset feature alignment module.

[0013] Optionally, the preset drawing entity feature extractor is an entity feature extractor based on a graph neural network; and the information aggregation and feature extraction of the graph structure to output the context-enhanced graph node embedding, comprises: The information aggregation and feature extraction of the graph structure are performed through a graph neural network to output the context-enhanced graph node embedding.

[0014] Optionally, the preset drawing image feature extractor is an image feature extractor based on an attention mechanism; and the global context information interaction in each image block based on the position embedding to output the image embedding corresponding to each image block, comprises: The global context information interaction in each image block based on the position embedding is performed through an attention mechanism to output the image embedding corresponding to each image block.

[0015] Optionally, the method further comprises: obtaining a pre-trained large language model and freezing the model weight thereof; constructing a training data set, the training data set comprising: a training drawing source file, a training drawing image file rendered from the training drawing source file, and an artificially annotated question and answer pair for the training drawing; training an image feature extractor, an entity feature extractor, and a feature alignment module using the training data set with the question and answer accuracy rate as the optimization target to obtain a preset drawing image feature extractor, a preset drawing entity feature extractor, and a preset feature alignment module.

[0016] Optionally, the training of the image feature extractor, the entity feature extractor, and the feature alignment module using the training data set with the question and answer accuracy rate as the optimization target to obtain the preset drawing image feature extractor, the preset drawing entity feature extractor, and the preset feature alignment module comprises: generating training drawing image features through the image feature extractor to be trained based on the training drawing image file; generating training drawing entity features through the entity feature extractor to be trained based on the training drawing source file; mapping the training image features and the training drawing entity features to a feature embedding space of the large language model through the feature alignment module to be trained; generating training multi-modal prompt words from the question request in the artificially annotated question and answer pair and the training features after feature alignment, inputting the pre-trained large language model, and obtaining model output replies; calculating loss information between the reply model output and a standard reply; updating model parameters of the image feature extractor, the entity feature extractor, and the feature alignment module based on the loss information until a preset convergence condition is reached to obtain the training completed preset drawing image feature extractor, preset drawing entity feature extractor, and preset feature alignment module.

[0017] According to a second aspect of the present disclosure, a drawing content question and answer device is provided, comprising: an extraction module configured to extract a drawing image file generated based on a rendering of a drawing source file to obtain drawing image features.

[0018] an analysis module configured to perform structural analysis on the drawing source file to obtain drawing entity features.

[0019] an alignment module configured to perform feature alignment on the drawing image features and the drawing entity features.

[0020] The generating module is configured to generate a prompt word according to a user question and answer request input by a user for a drawing content, a feature of a drawing image after feature alignment, and a drawing entity feature.

[0021] The generating module is further configured to input the prompt word to a large language model to generate a reply content corresponding to the question and answer request.

[0022] Optionally, the analyzing module comprises: The converting submodule is configured to convert the drawing source file into a drawing exchange file in a preset standard format; The decomposing submodule is configured to perform a decomposing operation on each block in the drawing exchange file and add each entity contained therein to a preset model space; The extracting submodule is configured to extract key attributes of each entity in the preset model space; The generating submodule is configured to input the key attributes to a graph node embedding generator of a corresponding entity type to generate a graph node corresponding to each entity; The generating submodule is configured to generate the drawing entity feature according to the graph node.

[0023] Optionally, the generating submodule is specifically configured to determine an association relationship between each entity based on the key attributes of each entity, determine an edge connection between each graph node based on the association relationship to construct a graph structure, perform information aggregation and feature extraction on the graph structure to output a context-enhanced graph node embedding, arrange the graph node embedding in a first preset order of the corresponding entity in the drawing source file to form a graph node embedding sequence, and take the graph node embedding sequence as the drawing entity feature.

[0024] Optionally, the generating submodule is further configured to initialize the edge connection between each graph node based on the association relationship, determine the number of connection edges of any graph node, remove part of the connection edges from the corresponding graph node through random sampling if the number of connection edges of any graph node exceeds a preset threshold, and construct a graph structure based on each connection edge after the removal processing.

[0025] Optionally, the extracting module comprises: The scaling submodule is configured to scale the drawing image file according to a preset size specification; The processing submodule is configured to perform block processing on the scaled image to obtain a plurality of image blocks and add a corresponding position embedding to each image block; The output submodule is configured to perform global context information interaction between each image block based on the position embedding to output an image embedding corresponding to each image block; The determining sub-module is configured to arrange the image embeddings in a second preset order according to their positions in the drawing image file to form an image embedding sequence, and take the image embedding sequence as the drawing image feature.

[0026] Optionally, the alignment module is specifically configured to map the drawing image feature and the drawing entity feature to a feature embedding space of a large language model respectively to obtain an image feature embedding and an entity feature embedding, where the embedding sizes of the image feature embedding and the entity feature embedding are respectively the same as a text embedding size of the large language model. The generation module is specifically configured to convert a user question and answer request input by a user for drawing content into a user request embedding, splice the user request embedding, the image feature embedding and the entity feature embedding according to a preset prompt template to generate a prompt word.

[0027] Optionally, the device further comprises: The acquisition module is configured to acquire a drawing source file. The rendering module is configured to visually render the drawing source file into a drawing image file.

[0028] Optionally, the extraction module is specifically configured to extract the drawing image file generated based on the drawing source file by using a preset image feature extractor to obtain a drawing image feature. The analysis module is specifically configured to perform structural analysis on the drawing source file by using a preset drawing entity feature extractor to obtain a drawing entity feature. The alignment module is specifically configured to perform feature alignment on the drawing image feature and the drawing entity feature by using a preset feature alignment module.

[0029] Optionally, the preset drawing entity feature extractor is an entity feature extractor based on a graph neural network, and the generation sub-module is specifically configured to use the graph neural network to perform information aggregation and feature extraction on the graph structure to output a graph node embedding enhanced by context.

[0030] Optionally, the preset drawing image feature extractor is an image feature extractor based on an attention mechanism, and the output sub-module is specifically configured to perform global context information interaction among the image blocks by using the attention mechanism based on the position embedding to output an image embedding corresponding to each image block.

[0031] Optionally, the device further comprises a construction module, a freezing module and a training module.

[0032] The acquisition module is further configured to acquire a pre-trained large language model. The freezing module is configured to freeze model weights of a pre-trained large language model. The construction module is configured to construct a training data set, the training data set comprising: a training drawing source file, a training drawing image file rendered from the training drawing source file, and an artificially annotated question and answer pair for the training drawing. The training module is configured to train an image feature extractor, an entity feature extractor, and a feature alignment module using the training data set, with the question and answer accuracy as an optimization objective, to obtain a preset drawing image feature extractor, a preset drawing entity feature extractor, and a preset feature alignment module.

[0033] Optionally, the training module is specifically configured to generate training drawing image features by the image feature extractor to be trained based on the training drawing image file. generate training drawing entity features by the entity feature extractor to be trained based on the training drawing source file; map the training image features and the training drawing entity features to a feature embedding space of the large language model through the feature alignment module to be trained; generate training multi-modal prompt words from the question request in the artificially annotated question and answer pair and the training features after feature alignment, input the pre-trained large language model, and obtain model output replies; calculate loss information between the reply model output and a standard reply; update model parameters of the image feature extractor, the entity feature extractor, and the feature alignment module based on the loss information until a preset convergence condition is reached, to obtain the training completed preset drawing image feature extractor, the preset drawing entity feature extractor, and the preset feature alignment module.

[0034] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of the first aspect.

[0035] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method of the first aspect.

[0036] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method according to the first aspect described above.

[0037] The drawing content question and answer method, device, electronic equipment and storage medium provided by the present disclosure can not only extract drawing image features from the drawing image file rendered and generated from the drawing source file to capture the overall visual information of the drawing, but also directly analyze the structure of the drawing source file to obtain drawing entity features, thereby obtaining accurate structured entity information inside the drawing. Through the double-modal feature fusion mechanism, the present disclosure can effectively identify and understand the overall visual information of the drawing, the geometric relationship between specific entities, the layer semantics and the topological connection between entities, and other deep structured information, as the drawing context information of the subsequent drawing content question and answer model, thereby improving the compatibility and generalization ability of drawings in different fields and styles. In addition, by aligning the drawing image features and the drawing entity features, and combining the question request of the user input for the drawing content, the present disclosure can construct a multi-modal prompt word, so that the large language model can accurately understand the fine-grained question of the user for a specific entity or relationship based on the overall visual information and structured entity information in the multi-modal prompt word, and provide detailed and accurate question answers. Therefore, the present disclosure improves the generalization ability of the drawing content question and answer in multiple fields and styles, enhances the ability to understand the structural integrity and semantic accuracy of the drawing, thereby improving the accuracy of the drawing content question and answer, and effectively overcoming the problems of missing structure understanding and content missing or misidentification caused by relying on single image input in the prior art. At the same time, the drawing content question and answer of the present disclosure does not rely on a fixed question and answer task design process, which can ensure the diversity and flexibility of the question and answer task, and further improve the accuracy of the drawing content question and answer.

[0038] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0039] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them: Figure 1 A flowchart of a drawing content question and answer method provided by an embodiment of the present disclosure; Figure 2 A flowchart of another drawing content question and answer method provided by an embodiment of the present disclosure; Figure 3 A flowchart of another drawing content question and answer method provided by an embodiment of the present disclosure; Figure 4This is a schematic diagram of the structure of a drawing content question-and-answer device provided in an embodiment of the present disclosure; Figure 5 A schematic block diagram of an example electronic device 400 provided for embodiments of this disclosure. Detailed Implementation

[0040] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0041] With the development of information technology, the digital processing and intelligent analysis of drawing content have become increasingly important. This is especially true in various vertical fields such as nuclear power engineering, power, and construction engineering, where the demand for accurate extraction and understanding of drawing content is growing. In practical applications, users typically use intelligent systems to ask questions about the drawing content, thereby efficiently obtaining the necessary information and improving the efficiency of drawing interpretation.

[0042] In existing technologies, question answering based on drawing content typically involves directly inputting the drawing image or its simple features into a large language model, relying on the model to generate the question and answer. However, most large language models have limited training data, primarily derived from internet text. Even if some models incorporate domain-specific drawing data for fine-tuning during the development phase, their applicability remains strictly limited to specific domains (e.g., only applicable to architectural structural drawings) or specific drawing styles. Furthermore, different domains employ different drafting standards, and even within the same domain, drawings exhibit high diversity due to factors such as design software and drawing habits. If the above method is used for question answering based on drawing content, it will be difficult to understand the content of drawings from different domains or styles, leading to omissions or misidentifications of certain content in the drawings. This results in incomplete and inaccurate extraction of drawing content, consequently causing low generalization ability and accuracy in question answering based on drawing content.

[0043] In existing technologies, visual language multimodal large models are used to fuse drawings and related document data to verify the logical consistency of drawing information, thus solving the problems of complex drawing structures and insufficient recognition accuracy. For the content to be recognized, existing technologies focus on carriers such as engineering drawings, obtain geometric component information by parsing vector graphics, and extract key content such as component type and parameters after encoding and model learning, thereby improving the efficiency and accuracy of information extraction.

[0044] While these drawing processing solutions have demonstrated some effectiveness in their respective fields, they still possess certain limitations. On one hand, most solutions explicitly define the applicable target domain and data format during the development phase, such as adapting only to drawing specifications or fixed-format source files specific to a particular industry. This lacks the ability to model drawing data across different or even unknown domains in a unified way, and makes it difficult to efficiently construct high-quality standardized training datasets from drawing data with significant domain differences. Consequently, the models can only function within pre-defined scenarios, and their effectiveness drops drastically when applied across different domains. On the other hand, the solutions are highly task-specific, mostly designed only for pre-defined, fixed tasks, and cannot flexibly handle new tasks not covered in the training data.

[0045] Therefore, how to solve the problems of incomplete and inaccurate extraction of drawing content, and the low generalization ability and accuracy of drawing content question-and-answer, are urgent technical issues to be addressed in the field of drawing content question-and-answer. Similarly, how to ensure the versatility of feature extraction for drawings from multiple fields and styles, and how to guarantee the diversity and flexibility of drawing content question-and-answer tasks are also urgent technical issues to be addressed in the field of drawing content processing.

[0046] This disclosure provides a method, apparatus, electronic device, and storage medium for answering questions about drawing content, aiming to improve compatibility and generalization capabilities for drawings from different fields and styles, enhance the ability to understand the structural integrity and semantic accuracy of drawings, thereby improving the accuracy of question answering questions about drawing content. Simultaneously, it aims to ensure the diversity and flexibility of question answering questions about drawing content, further improving the accuracy of question answering questions about drawing content.

[0047] The following description, with reference to the accompanying drawings, outlines a drawing-content question-and-answer method, apparatus, electronic device, and storage medium according to embodiments of this disclosure.

[0048] Figure 1 This is a flowchart illustrating a question-and-answer method for drawing content provided in an embodiment of this disclosure.

[0049] like Figure 1 As shown, the method includes the following steps: Step 101: Extract the drawing image file generated based on the drawing source file to obtain the drawing image features.

[0050] In this embodiment, step 101 can essentially be the extraction of visual representation features from the drawing. Specifically, the original drawing source file is first converted into a standard image format using a graphics rendering engine to generate a drawing image file that is visually consistent with the original drawing. This image retains the layout, lines, annotations, legends, and other visual elements of the drawing. The graphics rendering engine can be a renderer in Autodesk Computer Aided Design (AutoCAD) software, or an open-source rendering library, etc. The standard image format of the drawing image file can be a lossless compressed image format (Portable Network Graphics, PNG) or a lossy compressed format (Joint Photographic Experts Group, JPEG).

[0051] It should be noted that the image features of the drawing can be used to represent the semantic content of different regions in the image, capturing visual information such as the overall composition, component appearance, text position, and line style of the drawing. Step 101 transforms the visual information of the drawing into a machine-understandable numerical representation, providing a foundation for subsequent fusion with structured drawing entity features and the construction of multimodal prompts. It is worth noting that this process does not rely on manual annotation, but rather achieves end-to-end feature extraction through a pre-trained or fine-tuned visual model.

[0052] Step 102: Perform structured parsing on the source drawing file to obtain the entity features of the drawing.

[0053] In this context, an entity refers to a basic graphic element or composite object in the drawing source file that has independent geometric or semantic meaning; it is the smallest identifiable unit constituting the content of the drawing. Entity types can include basic primitives or composite objects. Basic graphic elements can be lines, circles, arcs, text, etc., which are the basic elements constituting an image. Composite objects can be block references, such as door and window symbols, equipment symbols, and polylines, which are composed of multiple basic primitives. Each entity can include key attributes such as coordinates, dimensions, layers, colors, line types, block names, and attribute labels. Drawing entity features can be represented by a set of structured vector sequences, with each vector corresponding to an engineering entity in the drawing. These vectors not only contain the entity's own attributes but also encode its geometric and semantic relationships with surrounding entities. Drawing entity features can accurately reflect the inherent logical structure of the drawing, compensating for abstract relational information that pure image methods cannot obtain. Key attributes of an entity can refer to a set of structured fields that uniquely identify the entity's identity, type, location, or function. These are metadata extracted from the drawing source file that is crucial for understanding the semantics of the drawing; they are key features constituting graph nodes, supporting relational reasoning and question answering.

[0054] In this embodiment of the disclosure, step 102 can extract structured semantic information from the raw data of the drawing, rather than relying solely on visual appearance. Specifically, step 102 can directly read and parse the drawing source file, and use geometric and semantic rules to identify and organize the graphic elements contained therein. Specifically, the structured parsing of the drawing source file includes, but is not limited to, the following operations: converting the drawing source file into a drawing exchange file of a preset standard format; performing a decomposition operation on each block insertion in the drawing exchange file, adding each entity contained therein to a preset model space; extracting the key attributes of each entity in the preset model space, and inputting the key attributes into a graph node embedding generator of the corresponding entity type to generate graph nodes corresponding to each entity; generating the drawing entity features based on the graph nodes. Alternatively, it can identify entities in the drawing source file; extract the key attributes of each graphic element; perform a decomposition operation on each block-type composite object, recursively expand its internally nested sub-entities, and uniformly classify them into the model space; input each entity and its attributes into a graph node embedding generator of the corresponding type to generate initial graph nodes; and generate drawing entity features that integrate contextual semantics based on the graph structure corresponding to the initial graph nodes.

[0055] Step 103: Align the image features and entity features of the drawing.

[0056] In this embodiment, the drawing image features can be generated by a visual encoder, typically with a first dimension (e.g., 768), reflecting pixel-level or region-level visual semantics. The drawing entity features can be generated by a graph neural network, typically with a second dimension (e.g., 512), encoding component-level geometric and topological relationships. The text embedding space of the large language model has a third dimension (e.g., 4096), and its internal parameters can only effectively process vectors of this specific dimension. That is, the first, second, and third dimensions are usually inconsistent. If the original image features or entity features are directly concatenated and input into the large language model, the mismatch in dimensions and inconsistent distribution will lead to semantic confusion, failure of the attention mechanism, and ultimately reduced question-answering accuracy. To eliminate differences between different modalities and improve question-answering accuracy, step 103 can be performed. Specifically, drawing image features and drawing entity features from different sources, dimensions, and semantic spaces can be mapped to a unified semantic embedding space, enabling them to be collaboratively understood and processed by the subsequent large language model.

[0057] It should be noted that the specific process of executing step 103 may include: inputting the image features and entity features of the drawing into the corresponding trainable mapping network, and projecting them into the text embedding space of the large language model through the trainable mapping network to obtain aligned multimodal features. The aligned features can be seamlessly spliced ​​with the text embedding of the user's question to form a unified multimodal prompt word. The large language model can understand the visual content and structural semantics in the drawing, just like understanding ordinary text, to support fine-grained interaction; for example, when a user asks "What device does this circle represent?", the large language model can combine the aligned image region features and entity features to jointly infer the answer.

[0058] Step 104: Generate prompt words based on the user's question and answer request for the drawing content, the feature-aligned drawing image features, and the drawing entity features.

[0059] In this embodiment of the disclosure, based on a user's question-and-answer request regarding specific drawing content, and combining the already feature-aligned image features and entity features of the drawing, effective prompt words are generated for further processing or querying. Step 104 can be a crucial bridge connecting user interaction and system understanding, ensuring that the subsequent large language model accurately provides information according to the user's actual needs. The generated prompt words can be multimodal prompt words. By generating multimodal prompt words, image and entity features can be combined into the prompt words, enabling the model to not only understand textual descriptions but also comprehensively consider visual elements and structured information in the drawing, thereby providing a more comprehensive answer. This helps the large language model better focus on the essence of the question, avoid misunderstandings or off-topic responses, and thus improve the accuracy and relevance of the answer.

[0060] It should be noted that the specific process of generating prompts based on the user's question-and-answer request regarding the drawing content, the feature-aligned drawing image features, and the drawing entity features may include: performing semantic analysis on the user's input question-and-answer request to determine the user's intent and focus. Specifically, natural language processing techniques, such as text classification and named entity recognition, can be used to extract key question elements and parse the obtained user intent; Step 105: Input the prompt words into the large language model to generate the response content corresponding to the question-and-answer request.

[0061] Among them, the large language model can be a mainstream open source or commercial model such as a decoder-only architecture; after receiving the above multimodal prompt words, the large language model uses its internal self-attention mechanism to jointly model the semantics of the text and the semantics of the drawing; during the generation process, the large language model predicts the next most likely token word by word until a complete natural language sentence is generated.

[0062] It should be noted that the prompt words can be input into a large language model to generate responses to user questions. The large language model performs joint semantic reasoning based on the user's question, drawing image features, and drawing entity features from the prompt words, and generates responses in natural language. The entire process requires no additional fine-tuning of the large language model parameters; it relies solely on high-quality alignment features provided by the front-end module to achieve accurate reasoning and output responses that conform to human reading habits. The responses can be natural language text, and they incorporate not only surface-level text but also deeper structural information from the drawing (such as topological connections and layer affiliations), thus ensuring semantic accuracy.

[0063] The drawing content question-answering method provided in this disclosure not only extracts drawing image features from the drawing image file generated by rendering the drawing source file to capture the overall visual information of the drawing, but also directly performs structured parsing on the drawing source file to obtain drawing entity features, thereby acquiring precise structured entity information within the drawing. Through this dual-modal feature fusion mechanism, this disclosure can effectively identify and understand the overall visual information of the drawing and deep-level structured information such as geometric relationships between specific entities, layer semantics, and topological connections between entities, serving as the drawing context information for subsequent drawing content question-answering models. This improves compatibility and generalization ability for drawings from different fields and styles. Furthermore, by aligning drawing image features and drawing entity features, and combining user-inputted questions regarding drawing content to construct multimodal prompts, this disclosure enables a large language model to accurately understand the user's fine-grained questions about specific entities or relationships based on the overall visual information and structured entity information of the drawing in the multimodal prompts, providing detailed and accurate question answers. Therefore… This disclosure enhances the generalization ability of question-and-answer (Q&A) for drawing content across multiple domains and styles, strengthening the understanding of the structural integrity and semantic accuracy of drawings. This improves the accuracy of Q&A for drawing content, effectively overcoming the shortcomings of existing technologies that rely on single image inputs, leading to incomplete structural understanding and missed or misidentified content. Furthermore, this disclosure's Q&A method does not depend on a fixed Q&A task design process, ensuring the diversity and flexibility of Q&A tasks, thereby further improving the accuracy of Q&A for drawing content.

[0064] Figure 2 This is a flowchart illustrating another drawing content question-and-answer method provided in an embodiment of this disclosure.

[0065] like Figure 2 As shown, the method includes the following steps: Step 201: Obtain the source drawing file.

[0066] The drawing source file is a vector format file that can contain structured graphic entities and metadata.

[0067] Step 202: Visualize and render the source drawing file into a drawing image file.

[0068] Specifically, source drawing files can be visualized and rendered into drawing image files using a preset image rendering tool. This preset image rendering tool can be a self-developed rendering engine or an existing software rendering interface. The source file can be rasterized according to standard drafting specifications. The rendering process includes: resolving layer visibility, applying line types and color styles, defining expanded blocks, drawing text annotations, and finally outputting a standard bitmap image file as the drawing image file. The output drawing image file is visually identical to the drawing seen by the engineer in the drafting software, preserving key visual information such as layout, scale, and annotation positions, and is stored in pixel matrix format for easy subsequent processing.

[0069] Step 203: Extract the drawing image file to obtain the drawing image features.

[0070] In an optional embodiment of this disclosure, the step of extracting drawing image features from the drawing image file may specifically include: scaling the drawing image file according to a preset size specification; dividing the scaled image into blocks to obtain multiple image blocks, and adding corresponding position embeddings to each image block; based on the position embeddings, performing global context information interaction between the image blocks, and outputting the image embeddings corresponding to each image block; arranging the image embeddings according to their second preset order in the drawing image file to form an image embedding sequence, and using the image embedding sequence as the drawing image feature.

[0071] It should be noted that step 203 can be performed by a preset drawing image feature extractor. The preset drawing image feature extractor may include a visual encoder or a self-attention mechanism module. Since the original rendered drawing image may have too high a resolution or different sizes, in order to adapt to the input requirements of the visual encoder, the drawing image is first uniformly scaled to a preset size. This scaling operation can maintain the aspect ratio or adopt a filling strategy to avoid stretching and distortion of key graphic information.

[0072] Subsequently, the scaled image can be divided into several fixed-size, non-overlapping image blocks, allowing the image to be modeled. To overcome the lack of spatial awareness in subsequent processing, a corresponding location embedding can be added to each image block, preserving the spatial location information of each block in the original image. The location embedding is a learnable vector whose value is determined by the block's row and column coordinates in the image grid. The location embedding is added to the visual features of the image blocks to form a location-aware initial token.

[0073] Finally, all image blocks with location embeddings are input into a visual encoder. During this process, each block interacts with all other blocks through a self-attention mechanism, dynamically aggregating the global context. For example, a text annotation block can be associated with the beam line block it points to through attention weights, thus understanding "which structural member the text describes." After encoding by multiple layers of visual encoders, each image block outputs a high-dimensional vector, called an image embedding. The second preset order can be their spatial order in the original image; that is, these image embeddings can be arranged according to their spatial order in the original image, forming an ordered image embedding sequence. This sequence fully preserves the local details and global layout information of the drawing and exists in the form of a token sequence compatible with large language models. It is ultimately used as the image feature of the drawing for subsequent feature alignment and multimodal cue construction.

[0074] Step 204: Perform structured parsing on the source drawing file to obtain the entity features of the drawing.

[0075] In an optional embodiment of this disclosure, in order to accurately extract the entity features of the drawing, the step of performing structured parsing of the drawing source file to obtain the entity features of the drawing may specifically include: converting the drawing source file into a drawing exchange file of a preset standard format; performing a decomposition operation on each block insertion in the drawing exchange file, adding each entity contained therein to a preset model space; extracting the key attributes of each entity in the preset model space, and inputting the key attributes into a graph node embedding generator of the corresponding entity type to generate graph nodes corresponding to each entity; and generating the drawing entity features based on the graph nodes.

[0076] It should be noted that step 203 can be performed using a preset drawing entity feature extractor, which may include a graph neural network, a graph convolutional network, or a graph attention network. To support graph neural network processing, a dedicated graph node embedding generator can be configured for each type of entity.

[0077] First, since the source files generated by different drafting software have different formats, in order to unify the processing flow, the source drawing files can be converted into a pre-defined standard format drawing exchange file. Specifically, the original source drawing files are converted into an open, standardized intermediate format—the pre-defined standard format can be the Drawing Exchange Format (DXF) text-based vector graphics exchange format defined by Autodesk. The DXF format has good cross-platform compatibility and clear structure, making it easy for programs to parse.

[0078] Secondly, to unify entity management, decomposition operations can be performed on each block insertion in the drawing exchange file. In a DXF file, a block is a composite object mechanism used to encapsulate reusable graphic units; for example, a block can be a door / window, equipment symbol, or annotation frame. A block insertion can record its reference position and scaling parameters, and its internal graphic definition is stored in a separate block table. To avoid semantic information being "hidden" in nested structures, the system performs decomposition operations on all block insertion entities. Specifically, this can involve recursively expanding the block definition; copying and transforming the sub-entities within the block to the global coordinate system; and uniformly managing them within a preset model space.

[0079] Subsequently, to generate entity features for the drawing, key attributes of each entity in the pre-defined model space can be extracted, and these key attributes are input into the corresponding entity type's graph node embedding generator to generate graph nodes for each entity. That is, all entities in the pre-model space can be traversed, and based on the entity type, the corresponding key attributes are extracted and input into the corresponding entity type's graph node embedding generator. Entity types can be lines, circles, arcs, etc., and key attributes include, but are not limited to: geometric attributes, layer information, identification information, and semantic content. Geometric attributes can include: start / end coordinates, radius, angle, bounding box, etc.; identification information can include: entity handle, color, line type, etc.; semantic content can include: entity text content, labels, and values, etc. These attributes constitute the structured metadata of each entity. It should be noted that the key attributes of each entity can be encoded into an initial feature vector; different types of entities may use different encoding strategies. Finally, the graph node embedding generator generates graph nodes for each entity. Each graph node can represent the basic unit of any corresponding entity in the drawing knowledge graph.

[0080] Finally, based on the graph nodes, the drawing entity features are generated. That is, all graph nodes can ultimately form a graph structure, which can be a semantic graph structure for the drawing, where nodes represent entities and edges can be dynamically constructed based on geometric relationships or semantic associations. Through multi-round message passing and feature aggregation of this graph using a graph neural network, each graph node incorporates its neighbor information, ultimately outputting context-aware drawing entity features. Geometric relationships can include endpoint overlap, enclosure, proximity, etc.; semantic associations can include belonging to the same layer, sharing the same block name, etc. Drawing entity features not only contain the entity's own attributes but also encode its topological role and functional context within the drawing.

[0081] In an optional embodiment of this disclosure, the step of generating the drawing entity features based on the graph nodes may specifically include: determining the association relationships between entities based on the key attributes of each entity; determining the edge connections between each graph node based on the association relationships to construct a graph structure; performing information aggregation and feature extraction on the graph structure to output context-enhanced graph node embeddings; arranging the graph node embeddings according to a first preset order of their corresponding entities in the drawing source file to form a graph node embedding sequence, and using the graph node embedding sequence as the drawing entity features.

[0082] It should be noted that, based on the graph nodes, generating the drawing entity features is not simply a matter of directly splicing the initial embeddings of each entity. Instead, it involves constructing a semantic graph structure for the drawing and performing graph neural network inference, so that the representation of each entity can incorporate its contextual information, thereby generating drawing entity features that are more semantically rich and have greater engineering significance.

[0083] First, the key attributes of all entities in the model space (such as coordinates, layers, text content, geometric types, etc.) are analyzed to identify potential semantic or geometric relationships between them. Second, each entity's corresponding graph node is treated as a vertex in the graph. Based on the relationships identified in the previous step, directed or undirected edges are established between related nodes to construct a semantic graph structure for the drawing. Then, the constructed graph structure is input into graph neural networks, graph convolutional networks, or graph attention networks. Through multiple rounds of "information transmission" mechanisms, each graph node aggregates information from its neighboring graph nodes. After several layers of propagation, the embedding vector of each graph node not only contains its original attributes but also incorporates the semantic and structural information of surrounding components, forming context-enhanced graph node embeddings. Finally, to adapt to subsequent integration with large language models, the system arranges all enhanced graph node embeddings according to their first preset order in the original drawing source file. For example, they can be arranged according to the order in which entities appear in the file or according to the identification order of entity handles, forming an ordered graph node embedding sequence. Graph node embedding sequences can ensure the input stability of large language input models, are formally aligned with image embedding sequences, and can be used to splice into user questions to form multimodal prompt words for large language models.

[0084] In an optional embodiment of this disclosure, the step of determining the edge connections between each graph node based on the association relationship to construct the graph structure may specifically include: initializing the edge connections between each graph node based on the association relationship; and constructing the graph structure based on each connection edge.

[0085] In an optional embodiment of this disclosure, in order to simplify the computational cost of graph structure construction, as a preferred solution, the step of determining the edge connections between each graph node based on the association relationship to construct the graph structure may specifically include: initializing the edge connections between each graph node based on the association relationship; determining the number of connected edges of any graph node; if the number of connected edges of any graph node exceeds a preset threshold, removing some edges from the connected edges of the corresponding graph node by random sampling; and constructing the graph structure based on the removed connected edges.

[0086] In other words, based on the relationships between entities, such as geometric connections and labeling mentioned above, initial edge connections are established between corresponding graph nodes. This step creates a preliminary graph structure containing all potential relationships. Subsequent steps involve analyzing each graph node and counting its number of edges to identify which nodes have excessive edges, as a large number of edges can complicate or inefficient subsequent processing steps. Specifically, a preset threshold can be set to limit the maximum number of edges a single node can have. If a node's edge count exceeds this threshold, its edges need to be reduced. This reduction is done through random sampling to determine which edges to remove. This approach avoids excessive computational burden caused by individual nodes having too many connections. Random sampling can be used instead of simply removing the earliest or latest edges, which helps maintain the diversity and information integrity of the graph structure to some extent. Finally, the graph structure is reconstructed based on the edge set processed in the above steps. This new graph structure is more streamlined than the initial version, with each node's edge count not exceeding the set threshold, making the entire graph more suitable for specific analysis tasks or algorithm applications.

[0087] Step 205: Map the drawing image features and drawing entity features to the feature embedding space of the large language model, respectively, to obtain image feature embedding and entity feature embedding.

[0088] In an optional embodiment of this disclosure, the step of aligning the drawing image features and the drawing entity features may specifically include: mapping the drawing image features and the drawing entity features to the feature embedding space of a large language model, respectively, to obtain image feature embedding and entity feature embedding, wherein the embedding size corresponding to each of the image feature embedding and entity feature embedding is the same as the text embedding size of the large language model.

[0089] Step 206: Convert the user's Q&A request for the drawing content into a user request embedding.

[0090] In an optional embodiment of this disclosure, the user question-and-answer request can be processed by a large language model word segmenter to obtain the user request embedding.

[0091] Step 207: The user request embedding, image feature embedding, and entity feature embedding are concatenated according to the preset prompt template to generate prompt words.

[0092] Specifically, let the user-input natural language format drawing content question be the string "{User Question Request}", the system task prompt word obtained after subsequent prompt word engineering be the string "{System Prompt Word}", the feature-aligned drawing image features be a high-dimensional vector embedding "<Image Feature Embedding>", and the drawing entity features be a high-dimensional vector embedding "<Entity Feature Embedding>". The string template used to concatenate the above content is "{Prompt Word Template}". Then, for any input "{User Question Request}", the model first concatenates it with "{System Prompt Word}", "[Image Feature Placeholder]", and "[Entity Feature Placeholder]" based on the "{Prompt Word Template}" to obtain the "{Original Input Sequence}". Among them, "[Image Feature Placeholder]" and "[Entity Feature Placeholder]" have no practical meaning; their function is to indicate the positions for inserting "<Image Feature Embedding>" and "<Entity Feature Embedding>" after word segmentation.

[0093] The original input sequence obtained after the above concatenation is as follows: "System prompt: {System prompt word}. The image features of the drawing are: [Image feature placeholder], the features of each entity in the drawing are: [Entity feature placeholder], and the user's question is: {User question request}. Now, please answer the user's question based on the drawing information. Answer: ".

[0094] For the aforementioned {original input sequence}, the module calls the large language model's word segmenter to convert it into text embeddings. Subsequently, the embeddings of [image feature placeholders] and [entity feature placeholders] are replaced with their corresponding <image feature embeddings> and <entity feature embeddings>, respectively, achieving drawing-text feature concatenation at the embedding level. Since the feature-concatenated embedding conforms to the standard text embedding format, the large language model can directly process it and generate a natural language response through text completion.

[0095] Step 208: Input the prompt words into the large language model to generate the response content corresponding to the user's question and answer request.

[0096] The large language model module mainly parses user queries about drawing content in natural language format, and combines the results of the aforementioned drawing feature extraction modules to perform inference, and finally answers the user's questions using natural language.

[0097] When selecting a large language model, any mainstream decoder-only architecture that has been pre-trained and has good fact-reasoning capabilities can be chosen. To balance model performance and inference complexity, the parameter capacity of the selected model should be in the range of no less than 1B and no more than 75B.

[0098] Optionally, the steps of extracting drawing image features from the drawing image file rendered based on the drawing source file and performing structured parsing on the drawing source file to obtain drawing entity features may specifically include: extracting drawing image features from the drawing image file rendered based on the drawing source file using a preset image feature extractor; and performing structured parsing on the drawing source file using a preset drawing entity feature extractor to obtain drawing entity features.

[0099] The step of aligning the drawing image features and the drawing entity features may specifically include: aligning the drawing image features and the drawing entity features using a preset feature alignment module. The preset feature alignment module may include image domain feature aligners corresponding to the drawing image features and the drawing entity features; that is, the drawing image features and the drawing entity features can be aligned using the image domain feature aligners corresponding to the drawing image features and the drawing entity features.

[0100] Optionally, the preset drawing entity feature extractor is an entity feature extractor based on a graph neural network; the step of performing information aggregation and feature extraction on the graph structure and outputting context-enhanced graph node embeddings may specifically include: using a graph neural network to perform information aggregation and feature extraction on the graph structure and outputting context-enhanced graph node embeddings.

[0101] Optionally, the preset drawing image feature extractor is an image feature extractor based on an attention mechanism. The step of performing global context information interaction in each image block based on the position embedding and outputting the image embedding corresponding to each image block may specifically include: performing global context information interaction in each image block through an attention mechanism based on the position embedding and outputting the image embedding corresponding to each image block.

[0102] Optionally, the method further includes: acquiring a pre-trained large language model and freezing its model weights; constructing a training dataset, the training dataset including: training drawing source files, training drawing image files rendered from the training drawing source files, and manually annotated question-and-answer pairs for the training drawings; using the training dataset, with question-and-answer accuracy as the optimization objective, training an image feature extractor, an entity feature extractor, and a feature alignment module to obtain a preset drawing image feature extractor, a preset drawing entity feature extractor, and a preset feature alignment module.

[0103] Optionally, the step of using the training dataset to train an image feature extractor, an entity feature extractor, and a feature alignment module with question-answering accuracy as the optimization objective, to obtain a preset drawing image feature extractor, a preset drawing entity feature extractor, and a preset feature alignment module, includes: generating training drawing image features based on the training drawing image file using the image feature extractor to be trained; generating training drawing entity features based on the training drawing source file using the entity feature extractor to be trained; mapping the training image features and training drawing entity features to the feature embedding space of the large language model via the feature alignment module to be trained; generating training multimodal prompt words using the question requests in the manually annotated question-answer pairs and the feature-aligned training features, inputting them into the pre-trained large language model to obtain the model output response; calculating the loss information between the response model output and the standard response; and updating the model parameters of the image feature extractor, entity feature extractor, and feature alignment module based on the loss information until a preset convergence condition is reached, thereby obtaining the trained preset drawing image feature extractor, preset drawing entity feature extractor, and preset feature alignment module.

[0104] Figure 3 This is a flowchart illustrating yet another drawing content question-and-answer method provided in an embodiment of this disclosure. Figure 3 As shown, 1. Input the DXF format drawing source file into the drawing entity feature extractor to extract the drawing entity features, and then input it into the image domain feature aligner of the feature alignment module to perform feature alignment, thereby obtaining the drawing entity embedding; 2. Render the source drawing file to obtain a PNP format drawing image file; 3. Input the drawing image file into the drawing image feature extractor for drawing image feature extraction, and then input it into the image domain feature aligner of the feature alignment module for feature alignment to obtain the drawing image embedding; 4. Input the user's input request into the large language model word segmenter to obtain the user request embedding; 5. Based on the prompt word module, the embedded entities in the drawing, the embedded images in the drawing, and the embedded user requests are spliced ​​together to obtain system prompt words or multimodal prompt words; 6. Input the system prompt words or multimodal prompt words into the large language model to obtain the model output, that is, to obtain the response content corresponding to the question-and-answer request.

[0105] The drawing content question-answering method provided in this disclosure not only extracts drawing image features from the drawing image file generated by rendering the drawing source file to capture the overall visual information of the drawing, but also directly performs structured parsing on the drawing source file to obtain drawing entity features, thereby acquiring precise structured entity information within the drawing. Through this dual-modal feature fusion mechanism, this disclosure can effectively identify and understand the overall visual information of the drawing and deep-level structured information such as geometric relationships between specific entities, layer semantics, and topological connections between entities, serving as the drawing context information for subsequent drawing content question-answering models. This improves compatibility and generalization ability for drawings from different fields and styles. Furthermore, by aligning drawing image features and drawing entity features, and combining user-inputted questions regarding drawing content to construct multimodal prompts, this disclosure enables a large language model to accurately understand the user's fine-grained questions about specific entities or relationships based on the overall visual information and structured entity information of the drawing in the multimodal prompts, providing detailed and accurate question answers. Therefore… This disclosure enhances the generalization ability of question-and-answer (Q&A) for drawing content across multiple domains and styles, strengthening the understanding of the structural integrity and semantic accuracy of drawings. This improves the accuracy of Q&A for drawing content, effectively overcoming the shortcomings of existing technologies that rely on single image inputs, leading to incomplete structural understanding and missed or misidentified content. Furthermore, this disclosure's Q&A method does not depend on a fixed Q&A task design process, ensuring the diversity and flexibility of Q&A tasks, thereby further improving the accuracy of Q&A for drawing content.

[0106] Figure 4 This is a schematic diagram of the structure of a drawing content question-and-answer device provided in an embodiment of this disclosure, such as... Figure 4 As shown, it includes: Extraction module 31 is configured to extract drawing image features from drawing image files generated based on the drawing source file rendering.

[0107] The parsing module 32 is configured to perform structured parsing on the drawing source file to obtain the drawing entity features.

[0108] Alignment module 33 is configured to perform feature alignment between the drawing image features and the drawing entity features.

[0109] The generation module 34 is configured to generate prompt words based on the user's question-and-answer request for the content of the drawing, the feature-aligned drawing image features, and the drawing entity features.

[0110] The generation module 34 is also configured to generate the response content corresponding to the question-and-answer request by inputting the prompt words into the large language model.

[0111] Optionally, the parsing module 32 includes: The conversion submodule is configured to convert the drawing source file into a drawing exchange file in a preset standard format; The decomposition submodule is configured to perform a decomposition operation on each block insertion in the drawing exchange file, adding each entity contained therein to a preset model space. The extraction submodule is configured to extract key attributes of each entity in a preset model space. The generation submodule is configured to input the key attributes into the graph node embedding generator of the corresponding entity type to generate graph nodes corresponding to each entity; The generation submodule is configured to generate the drawing entity features based on the drawing nodes.

[0112] Optionally, the generation submodule is specifically used to determine the association relationship between entities based on the key attributes of each entity; determine the edge connection between each graph node based on the association relationship to construct a graph structure; perform information aggregation and feature extraction on the graph structure to output context-enhanced graph node embeddings; arrange the graph node embeddings according to the first preset order of their corresponding entities in the drawing source file to form a graph node embedding sequence, and use the graph node embedding sequence as the drawing entity feature.

[0113] Optionally, the generation submodule is further configured to initialize edge connections between graph nodes based on the association relationship; determine the number of connected edges of any graph node; if the number of connected edges of any graph node exceeds a preset threshold, remove some edges from the connected edges of the corresponding graph node through random sampling; and construct a graph structure based on the removed connected edges.

[0114] Optionally, the extraction module 31 includes: The scaling submodule is configured to scale the drawing image file according to a preset size specification; The processing submodule is configured to divide the scaled image into blocks, obtain multiple image blocks, and add corresponding position embeddings to each image block. The output submodule is configured to perform global context information interaction between various image blocks based on the location embedding, and output the image embedding corresponding to each image block. The determination submodule is configured to arrange the image embeddings in a second preset order in the drawing image file to form an image embedding sequence, and to use the image embedding sequence as the drawing image feature.

[0115] Optionally, the alignment module is specifically configured to map the drawing image features and the drawing entity features to the feature embedding space of the large language model, respectively, to obtain image feature embedding and entity feature embedding, wherein the embedding size corresponding to the image feature embedding and entity feature embedding is the same as the text embedding size of the large language model. The generation module is specifically configured to convert user question-and-answer requests input by the user regarding the content of the drawing into user request embeddings; and to concatenate the user request embeddings, image feature embeddings, and entity feature embeddings according to a preset prompt template to generate prompt words.

[0116] Optionally, the drawing content question-and-answer device further includes: The acquisition module is configured to acquire drawing source files; The rendering module is configured to visually render the source drawing file into a drawing image file.

[0117] Optionally, the extraction module 31 is specifically configured to extract drawing image features from the drawing image file generated by rendering based on the drawing source file using a preset image feature extractor; The parsing module 32 is specifically configured to perform structured parsing on the drawing source file using a preset drawing entity feature extractor to obtain drawing entity features; The alignment module 33 is specifically configured to perform feature alignment between the drawing image features and the drawing entity features through a preset feature alignment module.

[0118] Optionally, the preset drawing entity feature extractor is an entity feature extractor based on a graph neural network; the generation submodule is specifically configured to use a graph neural network to perform information aggregation and feature extraction on the graph structure, and output context-enhanced graph node embeddings.

[0119] Optionally, the preset drawing image feature extractor is an image feature extractor based on an attention mechanism. The output submodule is specifically configured to perform global context information interaction in each image block based on the position embedding through the attention mechanism, and output the image embedding corresponding to each image block.

[0120] Optionally, the drawing content question-and-answer device further includes: a construction module, a freezing module, and a training module.

[0121] The acquisition module is also configured to acquire a pre-trained large language model; The freeze module is configured to freeze the model weights of the pre-trained large language model. The construction module is configured to construct a training dataset, which includes: training drawing source files, training drawing image files rendered from the training drawing source files, and manually annotated question-and-answer pairs for the training drawings; The training module is configured to use the training dataset to train an image feature extractor, an entity feature extractor, and a feature alignment module with question-answering accuracy as the optimization objective, thereby obtaining a preset drawing image feature extractor, a preset drawing entity feature extractor, and a preset feature alignment module.

[0122] Optionally, the training module is specifically configured to generate training drawing image features based on the training drawing image file using an image feature extractor to be trained; Based on the source file of the training drawing, entity features of the training drawing are generated through the entity feature extractor to be trained; The training image features and training drawing entity features are mapped to the feature embedding space of the large language model through the feature alignment module to be trained. Using the question requests and feature-aligned training features from the manually annotated question-answer pairs, train multimodal prompt words are generated and input into the pre-trained large language model to obtain the model's output response. Calculate the loss information between the response model output and the standard response; Based on the loss information, the model parameters of the image feature extractor, entity feature extractor, and feature alignment module are updated until the preset convergence condition is met, thus obtaining the trained preset drawing image feature extractor, preset drawing entity feature extractor, and preset feature alignment module.

[0123] The drawing content question-answering device provided in this disclosure can not only extract drawing image features from the drawing image file generated by rendering the drawing source file to capture the overall visual information of the drawing, but also directly perform structured parsing on the drawing source file to obtain drawing entity features, thereby acquiring precise structured entity information within the drawing. Through this dual-modal feature fusion mechanism, this disclosure can effectively identify and understand the overall visual information of the drawing and deep-level structured information such as geometric relationships between specific entities, layer semantics, and topological connections between entities, serving as the drawing context information for subsequent drawing content question-answering models. This improves the compatibility and generalization ability for drawings from different fields and styles. Furthermore, by aligning drawing image features and drawing entity features, and combining user-inputted questions regarding drawing content to construct multimodal prompts, this disclosure enables a large language model to accurately understand the user's fine-grained questions about specific entities or relationships based on the overall visual information and structured entity information of the drawing in the multimodal prompts, providing detailed and accurate question answers. Therefore… This disclosure enhances the generalization ability of question-and-answer (Q&A) for drawing content across multiple domains and styles, strengthening the understanding of the structural integrity and semantic accuracy of drawings. This improves the accuracy of Q&A for drawing content, effectively overcoming the shortcomings of existing technologies that rely on single image inputs, leading to incomplete structural understanding and missed or misidentified content. Furthermore, this disclosure's Q&A method does not depend on a fixed Q&A task design process, ensuring the diversity and flexibility of Q&A tasks, thereby further improving the accuracy of Q&A for drawing content.

[0124] It should be noted that the foregoing explanation of the method embodiments also applies to the apparatus of the embodiments of this disclosure, and the principle is the same. Therefore, the embodiments of this disclosure are not limited thereto.

[0125] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0126] According to embodiments of this disclosure, an electronic device is provided, comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform the aforementioned... Figures 1-3 The method described.

[0127] According to embodiments of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to perform the aforementioned... Figures 1-3 The method described.

[0128] According to embodiments of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the aforementioned... Figures 1-3 The method described.

[0129] Figure 5 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0130] like Figure 5 As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 402 or a computer program loaded from storage unit 408 into random access memory (RAM) 403. The RAM 403 may also store various programs and data required for the operation of the electronic device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0131] Multiple components in electronic device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of displays, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows electronic device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0132] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), graphics processing units (GPUs), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as the drawing content question-and-answer method. For example, in some embodiments, the drawing content question-and-answer method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the aforementioned drawing content question-and-answer method by any other suitable means (e.g., by means of firmware).

[0133] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0134] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0135] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, electrically programmable read-only memory (EPROM) or flash memory, optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0136] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode-ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0137] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0138] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0139] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0140] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0141] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A question-and-answer method for drawing content, characterized in that, include: The drawing image file generated by rendering based on the drawing source file is extracted to obtain the drawing image features; and the drawing source file is subjected to structured parsing to obtain the drawing entity features. Align the features of the drawing image and the features of the drawing entity; And based on the user's question-and-answer request for the drawing content, the feature-aligned drawing image features, and the drawing entity features, prompt words are generated; The prompt words are input into the large language model to generate the response content corresponding to the question-and-answer request.

2. The question-and-answer method for drawing content according to claim 1, characterized in that, The process of performing structured parsing on the source drawing file to obtain the entity features of the drawing includes: Convert the source drawing file into a drawing exchange file of a preset standard format; The block insertion in the drawing exchange file is decomposed, and the entities contained therein are added to the preset model space. Extract the key attributes of each entity in the preset model space, and input the key attributes into the graph node embedding generator of the corresponding entity type to generate graph nodes corresponding to each entity. The drawing entity features are generated based on the graph nodes.

3. The question-and-answer method for drawing content according to claim 2, characterized in that, The step of generating the drawing entity features based on the drawing nodes includes: Based on the key attributes of each entity, determine the relationships between the entities; Based on the aforementioned relationships, the edge connections between each graph node are determined to construct the graph structure; Information aggregation and feature extraction are performed on the graph structure to output context-enhanced graph node embeddings; The graph nodes are embedded in a first preset order according to their corresponding entities in the drawing source file to form a graph node embedding sequence, and the graph node embedding sequence is used as the drawing entity feature.

4. The question-and-answer method for drawing content according to claim 3, characterized in that, The step of determining the edge connections between graph nodes based on the association relationship to construct the graph structure includes: Based on the aforementioned association, initialize the edge connections between each graph node; Determine the number of connected edges for any node in the graph. If the number of connected edges of any graph node exceeds a preset threshold, some edges are removed from the connected edges of the corresponding graph node through random sampling. Based on the removed connection edges, a graph structure is constructed.

5. The question-and-answer method for drawing content according to claim 1, characterized in that, The step of extracting the drawing image file to obtain drawing image features includes: The drawing image file is scaled according to the preset size specifications; The scaled image is divided into blocks to obtain multiple image blocks, and corresponding position embeddings are added to each image block. Based on the location embedding, global context information is exchanged in each image block, and the image embedding corresponding to each image block is output. The images are embedded in a second preset order in the drawing image file to form an image embedding sequence, and the image embedding sequence is used as the drawing image feature.

6. The question-and-answer method for drawing content according to any one of claims 1-5, characterized in that, The step of aligning the features of the drawing image and the features of the drawing entity includes: The image features and entity features of the drawing are mapped to the feature embedding space of the large language model to obtain image feature embedding and entity feature embedding, respectively. The embedding size of the image feature embedding and entity feature embedding is the same as the text embedding size of the large language model. The process of generating prompts based on user-inputted questions and answers regarding the drawing content, the feature-aligned drawing image features, and the drawing entity features includes: Convert user-input Q&A requests for drawing content into user-requested embeddings; The user request embedding, image feature embedding, and entity feature embedding are concatenated according to a preset prompt template to generate prompt words.

7. A drawing content question-and-answer device, characterized in that, include: The extraction module is configured to extract drawing image features from drawing image files generated based on the drawing source file. The parsing module is configured to perform structured parsing on the drawing source file to obtain the drawing entity features; The alignment module is configured to perform feature alignment between the drawing image features and the drawing entity features; The generation module is configured to generate prompt words based on user Q&A requests for the content of the drawing, the feature-aligned drawing image features, and the drawing entity features; The generation module is also configured to generate the response content corresponding to the question-and-answer request by inputting the prompt words into the large language model.

8. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images