Engineering drawing recognition method, device and equipment

Abstract

The application relates to an engineering drawing recognition method, device and equipment, wherein the method comprises the following steps: obtaining a vector drawing and a raster drawing corresponding to a target engineering drawing, and performing information extraction processing on the vector drawing to obtain vector drawing data; performing structural analysis processing on the vector drawing data through a preset large language model to obtain semantic structured information of each vector graph element in the vector drawing, and performing visual recognition processing on the raster drawing through a preset large visual language model to obtain visual semantic information of the raster drawing; inputting the visual semantic information and the semantic structured information into a preset multi-modal large model for fusion processing to obtain a fusion processing result; and determining a drawing recognition result of the target engineering drawing based on the fusion processing result. The application can improve the accuracy and reliability of the final drawing recognition result.

Description

Technical Field

[0001] This application relates to the technical field of drawing recognition, and in particular to a method, apparatus and equipment for recognizing engineering drawings. Background Technology

[0002] In the field of power engineering, the identification and understanding of electrical wiring diagrams, equipment layout diagrams, and other drawings are fundamental steps in infrastructure review and other related processes. Power engineering drawings mainly include vector graphics generated by CAD software and raster image formats such as photographs. Vector graphics suffer from format compatibility issues and are difficult to use directly for visual recognition, while raster drawings lose crucial information such as layer semantics and topological structure.

[0003] In related technologies, vector and raster versions of the same engineering drawing are fused to ensure the sufficiency of the final recognition result. During fusion, the vector drawing is typically rasterized directly into an image, which is then used as input along with the raster version. Visual features are spatially bound to textual features using geometric coordinates, resulting in structured drawing data containing symbol categories and location information. However, this processing method loses the structured information of the vector drawing, leading to lower accuracy in the final drawing recognition result. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, and equipment for recognizing engineering drawings to address the aforementioned technical problems.

[0005] In a first aspect, this application provides a method for recognizing engineering drawings, the method comprising:

[0006] Obtain the vector graphics and raster graphics corresponding to the target engineering drawings, and perform information extraction processing on the vector graphics to obtain vector graphics data;

[0007] By performing structured parsing on vector paper data using a pre-set large language model, the semantic structured information of each vector element in the vector paper is obtained. Then, by performing visual recognition processing on raster paper using a pre-set large visual language model, the visual semantic information of the raster paper is obtained.

[0008] Visual semantic information and various semantic structured information are input into a pre-set multimodal large model for fusion processing to obtain the fusion processing result;

[0009] Based on the fusion processing results, the drawing recognition results of the target engineering drawings are determined.

[0010] In one embodiment, the vector graphic paper data includes the geometric and attribute information of each vector graphic element; the vector graphic paper data is subjected to structured parsing processing using a preset large language model to obtain the semantic structured information of each vector graphic element in the vector graphic paper, including:

[0011] For each vector graphic primitive, its geometric and attribute information is converted into a structured text sequence according to preset rules;

[0012] The structured text sequence is input into a pre-defined large language model for semantic processing to obtain the semantic structured information of vector primitives.

[0013] In one embodiment, the raster paper is visually recognized using a preset large visual language model to obtain the visual semantic information of the raster paper, including:

[0014] The raster paper is input into the visual encoder of the preset large visual language model for encoding processing to obtain the global visual features of the raster paper.

[0015] By using a pre-defined large visual language model text decoder, global visual features are decoded to obtain the visual semantic information of the raster paper.

[0016] In one embodiment, visual semantic information and various semantic structured information are input into a preset multimodal large model for fusion processing to obtain the fusion processing result, including:

[0017] Spatial alignment is performed between the coordinate system of vector graphics paper and the pixel coordinate system of raster graphics paper to ensure that the vector graphics primitives and the image areas of the raster graphics paper have a spatial correspondence.

[0018] All semantic structured information is uniformly encoded into text features, and visual semantic information is encoded into visual features;

[0019] By processing text features and visual features through a pre-defined cross-attention layer of a multimodal large model, the semantic association weights between text features and visual features are obtained.

[0020] Based on semantic association weights, text features and visual features are fused to obtain the fusion result.

[0021] In one embodiment, spatial alignment of the coordinate system of the vector drawing paper with the pixel coordinate system of the raster drawing paper includes:

[0022] Extract the same-named primitives from vector and raster drawings as control points;

[0023] Calculate the perspective transformation matrix based on the control points;

[0024] The perspective transformation matrix is ​​used to spatially align the coordinate system of vector paper with the pixel coordinate system of raster paper.

[0025] In one embodiment, based on the fusion processing result, the drawing recognition result of the target engineering drawing is determined, including:

[0026] Obtain a preset engineering symbol knowledge base, encode the rules in the preset engineering symbol knowledge base into natural language to obtain the encoded rules, and encode the fusion processing result into natural language to obtain the encoded processing result;

[0027] Based on the encoded rules and the encoded processing results, construct verification prompt words;

[0028] The verification prompts are input into a preset large language model for verification processing to obtain the verification results. The verification processing flow includes symbol existence verification, attribute consistency verification, and topological relationship verification.

[0029] Based on the verification results, determine the drawing recognition result of the target engineering drawing.

[0030] In one embodiment, the fusion processing result includes multiple fusion processing result items; based on the verification result, the drawing recognition result of the target engineering drawing is determined, including:

[0031] If the verification result is successful, the fusion processing result will be output according to the preset structured format to obtain structured drawing recognition data, and the structured drawing recognition data will be identified as the drawing recognition result of the target engineering drawing.

[0032] If the verification result is that the verification fails, the failed fusion processing result item in the fusion processing result is marked and pushed. When the corrected fusion processing result is obtained, the corrected fusion processing result is output in a preset structured format as the drawing recognition result of the target engineering drawing.

[0033] In one embodiment, the method further includes:

[0034] Obtain the drawing recognition results of at least two historical versions corresponding to the target project drawings;

[0035] The recognition results of drawings from each historical version are processed by structured text sequence conversion to obtain the text sequence of the drawing recognition results;

[0036] The text sequences of the recognition results of each drawing are input into a preset large language model for comparison processing to obtain the correspondence between vector primitives of different versions;

[0037] Based on the correspondence, determine the change type of each vector character across different versions;

[0038] Generate a version change report for the target engineering drawing based on the change type of each vector graphic element.

[0039] Secondly, this application also provides an engineering drawing recognition device, which includes:

[0040] The engineering drawing acquisition module is used to acquire the vector graphics and raster graphics corresponding to the target engineering drawings, and to extract information from the vector graphics to obtain vector graphics data.

[0041] The parallel drawing processing module is used to perform structured parsing processing on vector drawing data through a preset large language model to obtain the semantic structured information of each vector element in the vector drawing, and to perform visual recognition processing on raster drawings through a preset large visual language model to obtain the visual semantic information of the raster drawings.

[0042] The drawing information fusion module is used to input visual semantic information and various semantic structured information into a preset multimodal large model for fusion processing to obtain the fusion processing result;

[0043] The recognition result determination module is used to determine the drawing recognition result of the target engineering drawing based on the fusion processing result.

[0044] Thirdly, this application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0045] Obtain the vector graphics and raster graphics corresponding to the target engineering drawings, and perform information extraction processing on the vector graphics to obtain vector graphics data;

[0046] By performing structured parsing on vector paper data using a pre-set large language model, the semantic structured information of each vector element in the vector paper is obtained. Then, by performing visual recognition processing on raster paper using a pre-set large visual language model, the visual semantic information of the raster paper is obtained.

[0047] Visual semantic information and various semantic structured information are input into a pre-set multimodal large model for fusion processing to obtain the fusion processing result;

[0048] Based on the fusion processing results, the drawing recognition results of the target engineering drawings are determined.

[0049] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0050] Obtain the vector graphics and raster graphics corresponding to the target engineering drawings, and perform information extraction processing on the vector graphics to obtain vector graphics data;

[0051] By performing structured parsing on vector paper data using a pre-set large language model, the semantic structured information of each vector element in the vector paper is obtained. Then, by performing visual recognition processing on raster paper using a pre-set large visual language model, the visual semantic information of the raster paper is obtained.

[0052] Visual semantic information and various semantic structured information are input into a pre-set multimodal large model for fusion processing to obtain the fusion processing result;

[0053] Based on the fusion processing results, the drawing recognition results of the target engineering drawings are determined.

[0054] Fifthly, this application also provides a computer program product comprising a computer program that, when executed by a processor, performs the following steps:

[0055] Obtain the vector graphics and raster graphics corresponding to the target engineering drawings, and perform information extraction processing on the vector graphics to obtain vector graphics data;

[0056] By performing structured parsing on vector paper data using a pre-set large language model, the semantic structured information of each vector element in the vector paper is obtained. Then, by performing visual recognition processing on raster paper using a pre-set large visual language model, the visual semantic information of the raster paper is obtained.

[0057] Visual semantic information and various semantic structured information are input into a pre-set multimodal large model for fusion processing to obtain the fusion processing result;

[0058] Based on the fusion processing results, the drawing recognition results of the target engineering drawings are determined.

[0059] The aforementioned engineering drawing recognition method, apparatus, and equipment first acquire the vector graphics and raster graphics corresponding to the target engineering drawing, and then perform information extraction processing on the vector graphics to obtain vector graphics data. Next, the vector graphics data is subjected to structured parsing processing through a preset large language model to obtain the semantic structured information of each vector graphic element in the vector graphics. Then, the raster graphics are subjected to visual recognition processing through a preset large visual language model to obtain the visual semantic information of the raster graphics. The visual semantic information and each semantic structured information are then input into a preset multimodal large model for fusion processing to obtain the fusion processing result. Finally, based on the fusion processing result, the drawing recognition result of the target engineering drawing is determined. This application employs the aforementioned method, which leverages the cross-modal semantic understanding capabilities of a pre-defined multimodal large model to fuse the semantic structured information of vector graphics primitives with the visual semantic information of raster drawings. This allows for deep alignment and fusion of the semantic structured information of vector graphics primitives with the visual morphological features and textual semantics of raster drawings without losing the precise geometric information, layer semantics, and topological structure of the vector graphics primitives. This achieves complementary enhancement of primitive-level structural information and pixel-level visual information, effectively reducing the probability of losing the structured information of vector graphics primitives due to rasterization of vector drawings or only shallow fusion in traditional solutions. Consequently, the final drawing recognition result retains both the complete structured information of the vector drawings and integrates the rich visual semantic information of the raster drawings, ultimately significantly improving the accuracy and reliability of the drawing recognition results. Attached Figure Description

[0060] Figure 1 Flowcharts of engineering drawing recognition methods provided in some embodiments of this application;

[0061] Figure 2 Flowcharts for determining semantic structured information provided in some embodiments of this application;

[0062] Figure 3 Flowcharts for determining visual semantic information provided in some embodiments of this application;

[0063] Figure 4 Flowcharts illustrating the fusion process results provided in some embodiments of this application;

[0064] Figure 5 A flowchart illustrating coordinate system space alignment provided for some embodiments of this application;

[0065] Figure 6 Flowcharts for obtaining drawing recognition results provided in some embodiments of this application;

[0066] Figure 7 Flowcharts for obtaining drawing recognition results provided for other embodiments of this application;

[0067] Figure 8 A flowchart illustrating the process of obtaining a version change report is provided for some embodiments of this application;

[0068] Figure 9 Structural block diagram of an engineering drawing recognition device provided in some embodiments of this application;

[0069] Figure 10 Structural block diagram of an engineering drawing recognition system provided in some embodiments of this application;

[0070] Figure 11 This is an internal structural diagram of a computer device provided in some embodiments of this application. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0072] The engineering drawing recognition method provided in this application embodiment can be applied to terminals in power systems. These terminals include, but are not limited to, personal computers, laptops, smartphones, and tablets. When the terminal executes the engineering drawing recognition method provided in this application embodiment, it first acquires the vector graphics and raster graphics corresponding to the target engineering drawing, and performs information extraction processing on the vector graphics to obtain vector graphics data. Then, it performs structured parsing processing on the vector graphics data using a preset large language model to obtain the semantic structured information of each vector graphic element in the vector graphics. Next, it performs visual recognition processing on the raster graphics using a preset large visual language model to obtain the visual semantic information of the raster graphics. Finally, it inputs the visual semantic information and each semantic structured information into a preset multimodal large model for fusion processing to obtain the fusion processing result. Finally, based on the fusion processing result, it determines the drawing recognition result of the target engineering drawing.

[0073] Thus, by utilizing the cross-modal semantic understanding capabilities of a pre-defined multimodal large model, the semantic structured information of vector graphics primitives and the visual semantic information of raster drawings are fused. This allows for deep alignment and fusion of the semantic structured information of vector graphics primitives with the visual morphological features and textual semantics of raster drawings without losing the precise geometric information, layer semantics, and topological structure of the vector graphics primitives. This achieves complementary enhancement of primitive-level structural information and pixel-level visual information, effectively reducing the probability of losing the structured information of vector graphics primitives due to rasterization of vector drawings or only shallow fusion in traditional solutions. As a result, the final drawing recognition result retains the complete structured information of the vector drawings while incorporating the rich visual semantic information of the raster drawings, ultimately significantly improving the accuracy and reliability of the drawing recognition result.

[0074] In one embodiment, such as Figure 1 As shown, the method is illustrated using the aforementioned terminal as an example. In this embodiment, the method includes the following steps:

[0075] Step 102: Obtain the vector drawing and raster drawing corresponding to the target engineering drawing, and perform information extraction processing on the vector drawing to obtain the vector drawing data.

[0076] The target engineering drawings are specific power engineering drawings that need to be identified and understood, such as electrical wiring diagrams and equipment layout diagrams. The corresponding vector graphics are vector format files, such as DWG or DXF, containing precise primitive structures, layers, geometry, and topology information. The corresponding raster graphics are raster format files, such as JPEG, TIFF, PDF scans, or on-site photographs, presenting visual content in a pixel matrix format. Vector graphics data is the raw data obtained after extracting information from the vector graphics, including at least the geometric information (such as type, coordinates, and dimensions) and attribute information (such as layers, colors, line types, and extended data) of each vector primitive.

[0077] Optionally, you can first obtain the vector format file and raster format file corresponding to the target engineering drawing through the file reading interface; then call the vector format parser to parse the vector format file, extract the original data such as the primitive type, layer information, geometric coordinates, dimension annotations and text annotations in the file, and obtain the vector drawing data.

[0078] Step 104: Perform structured parsing processing on vector paper data using a preset large language model to obtain the semantic structured information of each vector element in the vector paper, and perform visual recognition processing on the raster paper using a preset large visual language model to obtain the visual semantic information of the raster paper.

[0079] The pre-defined large language model is a pre-trained language model with natural language understanding and reasoning capabilities, used to process textual primitive information. Vector primitives are the smallest independent units constituting graphics in vector graphics paper, such as lines, circles, polylines, and block references. The semantic structured information of vector primitives is the information output after semantic processing by the large language model, including at least the engineering semantic category of the vector primitives (e.g., circuit breaker, disconnector) and the business meaning of the layers. The pre-defined large visual language model is a pre-trained multimodal model with the ability to process images and text simultaneously, used for multi-task visual recognition of raster drawings. The visual semantic information of raster drawings is the information output after recognition by the large visual language model, including at least the electrical symbol categories, text annotation content, and primitive region locations in the raster drawings.

[0080] Optionally, for each vector graphic element, the geometric and attribute information of the vector graphic element can be semantically processed by a preset large language model to identify the engineering semantic category and layer business meaning of each vector graphic element, and obtain the semantic structured information of the vector graphic element; at the same time, the raster drawing can be visually recognized by the visual encoder and text decoder in the preset large visual language model to identify electrical symbols, text annotations and graphic element areas in the raster drawing, and obtain the visual semantic information of the raster drawing.

[0081] Step 106: Input the visual semantic information and the semantic structured information into the preset multimodal large model for fusion processing to obtain the fusion processing result.

[0082] Among them, the pre-set multimodal large model is a large-scale pre-trained model with cross-modal semantic understanding capabilities, used to achieve deep alignment and fusion of vector primitive text features and raster image visual features. The fusion processing result is a comprehensive recognition result output after cross-modal fusion, which includes the engineering semantic category, topological relationship and corresponding raster image visual verification information of each vector primitive.

[0083] Optionally, the semantic structured information of each vector graphic element and the visual semantic information of the raster paper can be mapped to a unified semantic space first. Then, through a pre-set cross-modal semantic alignment mechanism of a multimodal large model, the correspondence between each vector graphic element and the image area of ​​the raster paper can be established. The features of the vector graphic elements with corresponding relationships are fused with the features of the raster image to obtain the fusion processing result.

[0084] Step 108: Based on the fusion processing results, determine the drawing recognition results of the target engineering drawing.

[0085] The drawing recognition result of the target engineering drawing is the final output structured drawing data, which includes at least the element identifier and category, the geometric position of the element, the layer affiliation, the topological relationship and the associated text annotation.

[0086] Optionally, the fusion processing result can be verified according to preset verification rules, and the fusion processing result that passes the verification can be output in a preset structured format to obtain the drawing recognition result of the target engineering drawing.

[0087] The aforementioned engineering drawing recognition method, by utilizing the cross-modal semantic understanding capability of a pre-defined multimodal large model, fuses the semantic structured information of vector graphics primitives with the visual semantic information of raster drawings. This allows for deep alignment and fusion of the semantic structured information of vector graphics primitives with the visual morphological features and textual semantics of raster drawings without losing the precise geometric information, layer semantics, and topological structure of the vector graphics primitives. This achieves complementary enhancement of primitive-level structural information and pixel-level visual information, effectively reducing the probability of losing the structured information of vector graphics primitives due to rasterization or only shallow fusion in traditional solutions. Therefore, the final drawing recognition result retains both the complete structured information of the vector graphics and the rich visual semantic information of the raster drawings, ultimately significantly improving the accuracy and reliability of the drawing recognition results.

[0088] In one embodiment, such as Figure 2 As shown, vector graphics data includes the geometric and attribute information of each vector graphic element; by performing structured parsing processing on the vector graphics data using a pre-defined large language model, the semantic structured information of each vector graphic element in the vector graphics is obtained, including:

[0089] Step 202: For each vector graphic primitive, convert the geometric information and attribute information of the vector graphic primitive into a structured text sequence according to preset rules.

[0090] The geometric information of vector primitives consists of mathematical data describing their shape and position. For example, this geometric information includes primitive type (e.g., line / circle / block reference), coordinate position (e.g., start point / end point / center / insertion point), dimensions (e.g., radius, length, width), and spatial transformation parameters (e.g., rotation angle, scaling factor). The attribute information of vector primitives consists of non-geometric data describing their hierarchical relationships and how they are displayed. For example, this attribute information includes layer affiliation (e.g., the name and attributes of the layer the primitive belongs to), color, line type, line width, and extended data (e.g., device number, rated parameters, and other custom attributes).

[0091] A structured text sequence converts the geometric and attribute information of each vector primitive into a structured natural language descriptive text according to a preset template. The descriptive text of all primitives, when concatenated in a preset order, forms a complete structured text sequence. For example, a structured text sequence could be: "Primitive A: Type is circle, center coordinates (3500, 4200), radius 15, layer is 'E-Device Layer', color is red."

[0092] Optionally, the geometric and attribute information of each vector graphic element can be converted into corresponding natural language description statements according to a preset text template, with each graphic element corresponding to one description statement; then, the description statements of all graphic elements can be concatenated in a preset order to form a structured text sequence.

[0093] Step 204: Input the structured text sequence into the preset large language model for semantic processing to obtain the semantic structured information of vector primitives.

[0094] Optionally, the structured text sequence can be input into a preset large language model. The preset large language model can then perform contextual understanding on the structured text sequence, identify the standard engineering symbol category corresponding to each vector graphic element, and perform semantic recognition on the layer information of each vector graphic element to obtain the semantic structured information of each vector graphic element.

[0095] In this embodiment, by converting the geometric and attribute information of vector graphics primitives into structured text sequences, the structured information of vector drawings can be directly understood and processed by a large language model, reducing the probability of structured information loss due to rasterization processing in traditional solutions. In addition, by performing semantic processing on the structured text sequences through a large language model, its semantic understanding capabilities can be used to accurately identify the engineering semantic categories and layer business meanings of vector graphics primitives. This can provide vector graphics primitive features rich in engineering semantics for subsequent cross-modal fusion processing, thereby ensuring the accuracy and reliability of the final drawing recognition results.

[0096] In one embodiment, the semantic structured information includes standard engineering symbol categories, layer business meanings, topological relationships, and association relationships. The structured text sequence is input into a preset large language model for semantic processing to obtain the semantic structured information of the vector graphics primitives. This includes: performing contextual interpretation processing on the structured text sequence using the preset large language model to map the vector graphics primitives to standard engineering symbol categories; performing semantic recognition processing on the layer information in the structured text sequence using the preset large language model and preset layer naming rules to obtain the layer business meanings of the vector graphics primitives; performing spatial relationship reasoning processing on the spatial location information and adjacency relationships of each vector graphics primitive in the structured text sequence using the preset large language model to obtain the topological relationships between the vector graphics primitives; and performing association processing on the dimension annotation text in the structured text sequence using the preset large language model to obtain the association relationship between the dimension annotations and the vector graphics primitives.

[0097] The standard engineering symbol category refers to the name of the power engineering standard symbol corresponding to the vector primitive, such as "circuit breaker," "disconnector," "current transformer," "busbar," and "transformer." Layer business meaning refers to the engineering semantics of the layer where the vector primitive resides, such as "equipment layer" (for equipment symbols), "conductor layer" (for connecting lines), "annotation layer" (for text annotations), and "border layer" (for drawing frames). Topological relationships are the spatial connections between vector primitives, including connection relationships (A connects to B), containment relationships (A contains B), and adjacency relationships (A and B are adjacent). Association relationships are the correspondence between dimension annotation text and the corresponding vector primitive, such as "a length annotation of 5000mm corresponds to a busbar primitive."

[0098] In this embodiment, a pre-set large language model is used to perform contextual understanding, layer semantic recognition, spatial relationship reasoning, and dimension annotation association processing on the structured text sequence. By leveraging the semantic understanding and reasoning capabilities of the large language model, the standard engineering symbol categories and layer business meanings of vector primitives are automatically identified. The topological relationships between primitives and the association between dimension annotations and primitives are accurately constructed, transforming the original vector primitive metadata into structured information rich in engineering semantics. This solves the problem that the structured information of vector drawings is not fully semantically understood in traditional solutions, thus providing complete and accurate semantic features of vector primitives for subsequent cross-modal fusion processing, thereby ensuring the accuracy and reliability of the final drawing recognition results.

[0099] In one embodiment, such as Figure 3 As shown, visual recognition processing of the raster paper is performed using a pre-defined large visual language model to obtain the visual semantic information of the raster paper, including:

[0100] Step 302: Input the raster paper into the visual encoder of the preset large visual language model for encoding processing to obtain the global visual features of the raster paper.

[0101] The visual encoder in the pre-defined large visual language model is a neural network module used to extract image features. For example, the Vision Transformer is responsible for converting the input raster image into a computer-understandable numerical feature representation. Global visual features are feature vectors or feature maps extracted by the visual encoder from the entire raster image, containing global spatial structure and visual semantic information.

[0102] Optionally, the raster drawing can be input into a visual encoder of a preset large visual language model. The visual encoder encodes the image information of the raster drawing and extracts the visual features related to electrical symbols, text labels, and graphic element areas in the raster drawing, thereby obtaining the global visual features of the raster drawing.

[0103] Step 304: The global visual features are decoded using a text decoder based on a pre-defined large visual language model to obtain the visual semantic information of the raster paper.

[0104] Among them, the text decoder of the pre-defined large visual language model is a neural network module in the large visual language model used to convert visual features into text output. It is responsible for decoding image features into structured text descriptions, such as symbol categories and annotation content.

[0105] Optionally, the global visual features can be input into the text decoder of the preset large visual language model. The text decoder can then decode the global visual features to generate recognition results such as the electrical symbol categories, text annotation content, and the location of graphic element areas in the raster drawing, thereby obtaining the visual semantic information of the raster drawing.

[0106] In this embodiment, the raster drawing is input into a visual encoder of a preset large visual language model for encoding processing to obtain global visual features containing global spatial structure and visual semantic information. Then, the global visual features are decoded by a text decoder to generate visual semantic information of the raster drawing. This allows multi-dimensional information such as electrical symbol categories, text annotation content, and primitive area positions in the raster drawing to be jointly encoded and understood within a unified model framework. This provides semantically complete and structurally unified visual features for subsequent cross-modal fusion processing, ensuring the accuracy and completeness of the final drawing recognition result.

[0107] In one embodiment, such as Figure 4 As shown, visual semantic information and various semantic structured information are input into a preset multimodal large model for fusion processing to obtain the fusion processing result, including:

[0108] Step 402: Spatial alignment processing is performed on the coordinate system of the vector paper and the pixel coordinate system of the raster paper to make the vector primitives and the image areas of the raster paper correspond in space.

[0109] Optionally, corresponding primitives in vector paper and raster image are extracted as control points, and a transformation matrix is ​​calculated based on the control points. The vector coordinate system is then spatially aligned with the pixel coordinate system of the raster image using the transformation matrix.

[0110] Step 404: Encode each semantic structured information into text features and encode the visual semantic information into visual features.

[0111] Optionally, the semantic structured information of each vector graphic element can be input into the text encoder of the preset multimodal large model and encoded as text features; the visual semantic information of the raster paper can be input into the visual encoder of the preset multimodal large model and encoded as visual features.

[0112] Step 406: Process the text features and visual features through the cross-attention layer of the preset multimodal large model to obtain the semantic association weights between the text features and the visual features.

[0113] Optionally, text features and visual features can be input into the cross-attention layer of a pre-defined multimodal large model, using text features as queries and visual features as keys and values, to calculate the semantic association weights between text features and visual features, so as to characterize the degree of semantic correlation between each vector primitive and each image region in the raster paper.

[0114] Step 408: Based on semantic association weights, the text features and visual features are fused to obtain the fusion result.

[0115] Optionally, textual and visual features can be fused in the semantic space of a pre-defined multimodal large model based on semantic association weights to obtain the fusion processing result.

[0116] In this embodiment, by spatially aligning the vector paper coordinate system with the raster paper pixel coordinate system, a reference position region can be established for the vector primitives in the raster image, providing a positional prior for subsequent semantic alignment. Furthermore, by encoding semantic structured information into text features and visual semantic information into visual features, information from two different modalities can be uniformly represented within the same semantic space. Then, by calculating the semantic association weights between text features and visual features through a pre-set cross-attention layer of a multimodal large model, and utilizing the cross-modal semantic understanding capability of the large model, a precise semantic correspondence between each vector primitive and the raster paper image region is automatically established under the constraint of positional priors. This solves the problem of traditional coordinate alignment schemes failing in paper deformation scenarios. Finally, deep fusion is performed in the semantic space based on the semantic association weights, ensuring that the fusion processing result retains both the complete structured information of the vector paper and the rich visual semantics of the raster paper, thereby guaranteeing the accuracy and reliability of the final paper recognition result.

[0117] In one embodiment, such as Figure 5 As shown, spatial alignment of the coordinate system of vector paper and the pixel coordinate system of raster paper includes:

[0118] Step 502: Extract the same-named primitives from the vector paper and the raster paper as control points.

[0119] Optionally, feature points with clear geometric positions can be identified in vector drawings, such as the corner points of the drawing border, the corner points of the title block, and the intersection points of the crosshairs of the drawing frame, and the corresponding pixel positions of the same feature point can be located in the raster drawing to obtain multiple sets of control point pairs.

[0120] Step 504: Calculate the perspective transformation matrix based on the control points.

[0121] The perspective transformation matrix describes the projection mapping relationship between two planes, mapping coordinate points in the vector paper coordinate system to corresponding pixel positions in the raster image pixel coordinate system. The perspective transformation matrix can compensate for perspective distortions, such as tilt and perspective deformation, that occur during scanning or photographing of drawings, ensuring that vector primitives are accurately positioned within their corresponding spatial regions of the raster image.

[0122] Optionally, the coordinate correspondence of multiple sets of control point pairs can be used to establish a system of equations and solve for the values ​​of each element in the perspective transformation matrix.

[0123] Step 506: Spatially align the coordinate system of the vector paper with the pixel coordinate system of the raster paper using a perspective transformation matrix.

[0124] Optionally, a perspective transformation matrix can be used to transform the coordinates of each vector element in the vector paper, mapping the vector coordinates to the pixel coordinates of the raster image, so that each vector element obtains its corresponding reference position in the raster paper.

[0125] In this embodiment, by extracting corresponding primitives from vector drawings and raster drawings as control points, and using the coordinate correspondence of multiple sets of control points to calculate the perspective transformation matrix, the coordinate system differences and perspective distortion caused by scanning, photographing, or drawing deformation can be effectively corrected. This allows each vector primitive to obtain an accurate reference position area in the raster drawing, so as to provide a reliable positional prior for subsequent cross-modal semantic alignment, thereby ensuring the accuracy and reliability of the final drawing recognition result.

[0126] In one embodiment, such as Figure 6 As shown, based on the fusion processing results, the drawing recognition results of the target engineering drawing are determined, including:

[0127] Step 602: Obtain a preset engineering symbol knowledge base, encode the rules in the preset engineering symbol knowledge base using natural language to obtain the encoded rules, and encode the fusion processing result using natural language to obtain the encoded processing result.

[0128] The preset engineering symbol knowledge base is a database that stores standard symbols and related rules in the field of power engineering. It may include: visual feature templates of standard symbols, standardized naming of symbols, topological constraint rules between symbols (such as "incoming line → disconnecting switch → circuit breaker"), and mapping relationships between equipment and symbols.

[0129] Optionally, a preset engineering symbol knowledge base can be obtained first, and the standardized naming of symbols and topological constraint rules in the preset engineering symbol knowledge base can be converted into natural language descriptions according to a preset template to obtain the encoded rules; then, each recognition result item in the fusion processing result can be converted into natural language descriptions according to a preset template to obtain the encoded processing result.

[0130] Step 604: Based on the encoded rules and the encoded processing results, construct the verification prompt words.

[0131] Among them, the verification prompt words are natural language text composed of knowledge base rules, fusion processing results and verification task descriptions according to a preset template, which are provided as input to the large language model to instruct the model to perform the verification task.

[0132] Optionally, the encoded rules, the encoded processing results, and the description used to instruct the large language model to perform the verification task can be combined in a preset order to construct verification prompt words.

[0133] Step 606: Input the verification prompt words into the preset large language model for verification processing to obtain the verification result.

[0134] The verification results are structured verification conclusions output by the large language model, which may include whether each recognition result item passed the verification, the reasons for failure, and correction suggestions. The verification process includes symbol existence verification, attribute consistency verification, and topological relationship verification.

[0135] Symbol existence verification checks whether each symbol category in the fusion recognition result has a corresponding standard entry in the preset engineering symbol knowledge base. If the symbol category has a corresponding entry in the preset engineering symbol knowledge base, the verification passes; if the symbol category does not have a corresponding entry in the preset engineering symbol knowledge base, it is marked as "unknown symbol," and the name of the symbol and its location information in the drawing are recorded.

[0136] Attribute consistency verification checks whether the representation of the same device is consistent across different layers, views, or pages. Specifically, this includes verifying whether the device type identified in different views is the same, whether the device number matches, and whether the device parameters (such as rated voltage and rated current) are consistent. If the verification finds inconsistencies in the information of the same device in different locations, such as the same device number corresponding to two different device types, it is marked as "attribute inconsistency," and the specific details of the inconsistency are recorded.

[0137] Topology verification checks whether the topological connections between identified symbols conform to the topological constraints in a pre-defined engineering symbol knowledge base. Specifically, it includes checking for connections that do not conform to the standard order (e.g., whether the order of disconnectors and circuit breakers in the incoming circuit is correct), the presence of isolated equipment (equipment not connected to any other equipment), and unreasonable connection paths (e.g., busbars directly connected to loads without protection). If a topology anomaly is found, the anomaly type and the involved symbols are marked, and the corresponding standard topology requirements from the pre-defined engineering symbol knowledge base are output as a correction reference.

[0138] Optionally, the verification prompts can be input into a preset large language model. The preset large language model performs semantic understanding on the verification prompts, verifies each recognition result item in the fusion processing result, and outputs the verification result.

[0139] Step 608: Based on the verification results, determine the drawing recognition result of the target engineering drawing.

[0140] Optionally, after obtaining the verification results, for the verification-passed recognition results, output them according to a preset structured format to obtain the drawing recognition results of the target engineering drawing.

[0141] In this embodiment, by encoding the knowledge base rules and fusion processing results into natural language and constructing verification prompt words, the knowledge base rules can be understood and applied by the large language model without being written as hard-coded logic, which can improve the coding efficiency of rule matching. At the same time, by utilizing the semantic understanding capabilities of the large language model, even if there are differences between the symbol naming or topological description and the rules, the model can still accurately verify them, thereby solving the problem that hard-coded verification is prone to failure in scenarios with differences in description, and thus ensuring the accuracy and standardization of the output results.

[0142] In one embodiment, such as Figure 7 As shown, the fusion processing result includes multiple fusion processing result items; based on the verification result, the drawing recognition result of the target engineering drawing is determined, including:

[0143] Step 702: If the verification result is successful, the fusion processing result is output according to the preset structured format to obtain structured drawing recognition data, and the structured drawing recognition data is determined as the drawing recognition result of the target engineering drawing.

[0144] Optionally, when the verification result is that the fusion processing result passes, the fusion processing result is organized according to a preset structured format to generate structured drawing recognition data containing the identifiers, categories, geometric positions, layer affiliations, topological relationships, and associated text annotations of each graphic element, which serves as the drawing recognition result of the target engineering drawing.

[0145] Step 704: If the verification result is that the verification fails, mark and push the failed fusion processing result item in the fusion processing result. When the corrected fusion processing result is obtained, output the corrected fusion processing result in a preset structured format as the drawing recognition result of the target engineering drawing.

[0146] The preset structured format is a predefined, unified data output format used to organize drawing recognition results into a machine-readable structured form, including at least fields such as element identifier and category, element geometric location, layer affiliation, topological relationships, and associated text annotations. The corrected fusion processing result is the correct recognition result obtained after manual review and correction of fusion processing result items that failed verification. Correction methods can include direct manual modification, automatic correction based on correction suggestions, and automatic system updates after manual review and approval.

[0147] Optionally, when the verification result shows that there are failed items in the fusion processing result, the failed fusion processing result items are marked with their corresponding problem types and pushed to the manual review queue; when the fusion processing result after manual review or correction is obtained, the corrected fusion processing result is output in a preset structured format as the drawing recognition result of the target engineering drawing. For example, the output format of the drawing recognition result can adopt JSON or Protocol Buffers structured data format to support direct calls from downstream engineering systems.

[0148] In this embodiment, the fusion processing results that pass the verification are directly output in a preset structured format, which can ensure the efficiency of automated drawing processing; while the fusion processing results that fail the verification are marked with problem types and pushed to manual review, which can correct the erroneous identification results in a timely manner and ensure the recognition efficiency of engineering drawings.

[0149] In one embodiment, such as Figure 8 As shown, the method also includes:

[0150] Step 802: Obtain the drawing recognition results of at least two historical versions corresponding to the target engineering drawings.

[0151] Among them, historical versions are past versions of the same target engineering drawings saved at different points in time or at different design stages.

[0152] Optionally, the structured drawing recognition result data corresponding to at least two historical versions of the target engineering drawing can be read from the power system's storage system.

[0153] Step 804: Perform structured text sequence conversion on the drawing recognition results of each historical version to obtain the text sequence of the drawing recognition results.

[0154] The text sequence of the drawing recognition result is a structured text sequence formed by converting information such as the category, geometric position, topological relationship, and associated text annotation of each element in the drawing recognition result into natural language description according to a preset template, and then splicing all the descriptions together.

[0155] Optionally, the categories, geometric locations, layer affiliations, topological relationships, and associated text annotations of each graphic element in the drawing recognition results of each historical version can be converted into natural language descriptions according to a preset template, and the descriptions of all graphic elements can be spliced ​​together to obtain the text sequence corresponding to that version.

[0156] Step 806: Input the text sequence of each drawing recognition result into the preset large language model for comparison processing to obtain the correspondence between vector primitives of different versions.

[0157] Among them, the correspondence between vector primitives in different versions is the matching relationship between primitives belonging to the same project object (such as a circuit breaker) in different versions, which is identified by semantic understanding of the large language model. This includes one-to-one matching, adding primitives (no corresponding in the old version), and deleting primitives (no corresponding in the new version).

[0158] Optionally, different versions of text sequences can be combined according to a preset format and input into a preset large language model. Through the semantic understanding capabilities of the preset large language model, similarity analysis is performed on the categories, positions, and associated texts of graphic elements in each version to identify graphic elements belonging to the same engineering object in different versions and establish correspondences between graphic elements. Graphic elements that cannot establish correspondences are determined to be newly added or deleted graphic elements.

[0159] Step 808: Determine the change type of each vector graphic element between different versions based on the correspondence.

[0160] Among them, the change types between different versions are the categories of changes that occur to graphic elements between different versions, which can include position offset (coordinate change), attribute modification (parameter value change), device replacement (one device is replaced with another), and topology change (connection relationship change), etc.

[0161] Optionally, for primitives with corresponding relationships, the changes in their geometric position, attribute information, and associated text in different versions can be compared using a preset large language model, and the change type can be determined by combining engineering knowledge.

[0162] Step 810: Generate a version change report for the target engineering drawing based on the change type of each vector graphic element.

[0163] The version change report is a document generated after summarizing all change information. It may include the unique identifier of the changed graphic element, the change type, a comparison before and after the change, the coordinates of the changed location, the semantic description of the change, and a risk impact assessment.

[0164] Optionally, you can summarize the change type and information before and after the change for each graphic element, and generate a version change report that includes the changed graphic element identifier, change type, comparison before and after the change, change location coordinates, change semantic description, and risk impact assessment.

[0165] In this embodiment, the identification results of drawings from different historical versions are obtained, converted into structured text sequences, and input into a preset large language model for comparison. The semantic understanding capability of the preset large language model is used to automatically establish the correspondence between graphic elements in different versions and determine the change type. This can solve the problem of low efficiency in traditional solutions that require manual comparison of drawings one by one. At the same time, the preset large language model can not only identify geometric changes, but also understand the engineering semantics and potential risks of changes by combining engineering knowledge. Based on this, a version change report containing a description of the change semantics and a risk impact assessment is generated, providing reviewers with an intuitive and traceable understanding of changes, thereby significantly improving the management efficiency of drawing versions.

[0166] In one embodiment, before performing information extraction processing on the vector drawing to obtain vector drawing data, the method further includes preprocessing the raster drawing. The preprocessing of the raster drawing includes at least: tilt detection and correction based on deep learning, binarization processing based on adaptive threshold or deep network (to distinguish graphic areas and background areas in the raster drawing), and resolution unification based on interpolation methods.

[0167] For example, the preprocessing of raster drawings may further include: deep learning-based tilt correction, using U-Net (U-shaped network) or HED (Holistically-Nested Edge Detection) network to detect the direction of text lines and border lines in the drawing, calculating the tilt angle and performing affine transformation correction; binarization processing based on adaptive threshold to distinguish between graphic areas and background areas in the drawing; and using a large model-based image super-resolution method to improve the clarity of low-quality scanned drawings.

[0168] For large-format drawings, a sliding window segmentation strategy is adopted to segment the image into image blocks that match the preset size. The resolution of each image block is adapted to the input requirements of the large visual language model. A preset number of overlapping pixels are retained between adjacent image blocks to ensure spatial continuity across image blocks and avoid loss of integrity due to the splitting and recognition of cross-block primitives caused by the cutting of image block boundaries.

[0169] For drawings with poor scanning quality or problems such as blurriness or uneven lighting, a large model-based image enhancement method is adopted. The multimodal large model is used to repair the quality and improve the clarity of the blurry or unevenly lit drawing images, so as to improve the subsequent visual recognition effect.

[0170] Among them, the tilt detection and correction based on deep learning can use U-Net or HED (Holistically-Nested Edge Detection) to detect the direction of text lines and border lines in the raster paper, calculate the tilt angle and perform affine transformation correction.

[0171] In one embodiment, for the same vector graphic element, during the fusion process, when the vector parsing result is inconsistent with the raster recognition result—that is, when the semantic structured information corresponding to the vector graphic element is inconsistent with the visual semantic information of the raster drawing—the inconsistency information is input into a preset multimodal large model. The preset multimodal large model performs comprehensive reasoning and judgment based on engineering logic and contextual information, and outputs the judgment result and the reasoning for the judgment. Furthermore, when the difference between the confidence level corresponding to the vector parsing result and the confidence level corresponding to the raster recognition result is not greater than a preset threshold, the vector parsing result and the raster recognition result are weighted and combined, and the weighted combined result is used as the fusion processing result of the vector graphic element. When the difference is greater than the preset threshold, the vector graphic element is marked as requiring manual confirmation, and the marking result is used as the fusion processing result of the vector graphic element.

[0172] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0173] Based on the same inventive concept, this application also provides an engineering drawing recognition device for implementing the above-mentioned engineering drawing recognition method. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more engineering drawing recognition device embodiments provided below can be found in the limitations of the engineering drawing recognition method described above, and will not be repeated here.

[0174] In one embodiment, such as Figure 9 As shown, an engineering drawing recognition device is provided, comprising: an engineering drawing acquisition module 902, a drawing parallel processing module 904, a drawing information fusion module 906, and a recognition result determination module 908, wherein:

[0175] The engineering drawing acquisition module 902 is used to acquire the vector drawings and raster drawings corresponding to the target engineering drawings, and to extract information from the vector drawings to obtain vector drawing data.

[0176] The parallel drawing processing module 904 is used to perform structured parsing processing on vector drawing data through a preset large language model to obtain the semantic structured information of each vector element in the vector drawing, and to perform visual recognition processing on raster drawing through a preset large visual language model to obtain the visual semantic information of the raster drawing.

[0177] The drawing information fusion module 906 is used to input visual semantic information and various semantic structured information into a preset multimodal large model for fusion processing to obtain the fusion processing result;

[0178] The recognition result determination module 908 is used to determine the drawing recognition result of the target engineering drawing based on the fusion processing result.

[0179] In one embodiment, the parallel drawing processing module 904 is further configured to convert the geometric information and attribute information of each vector graphic primitive into a structured text sequence according to preset rules; input the structured text sequence into a preset large language model for semantic processing to obtain the semantic structured information of the vector graphic primitive.

[0180] In one embodiment, the parallel drawing processing module 904 is further configured to input the raster drawing into the visual encoder of the preset large visual language model for encoding processing to obtain the global visual features of the raster drawing; and to decode the global visual features through the text decoder of the preset large visual language model to obtain the visual semantic information of the raster drawing.

[0181] In one embodiment, the drawing information fusion module 906 is further configured to perform spatial alignment processing on the coordinate system of vector drawing paper and the pixel coordinate system of raster drawing paper, so that the vector primitives and the image areas of raster drawing paper have a spatial correspondence; uniformly encode each semantic structured information into text features, and encode visual semantic information into visual features; process the text features and visual features through a preset multimodal large model cross-attention layer to obtain the semantic association weight between text features and visual features; and perform fusion processing on text features and visual features based on semantic association weight to obtain the fusion processing result.

[0182] In one embodiment, the drawing information fusion module 906 is further used to extract the same-name primitives in vector drawings and raster drawings as control points; calculate the perspective transformation matrix based on the control points; and spatially align the coordinate system of the vector drawings with the pixel coordinate system of the raster drawings using the perspective transformation matrix.

[0183] In one embodiment, the recognition result determination module 908 is further configured to acquire a preset engineering symbol knowledge base, encode the rules in the preset engineering symbol knowledge base into natural language to obtain encoded rules, and encode the fusion processing result into natural language to obtain encoded processing result; construct verification prompt words based on the encoded rules and encoded processing result; input the verification prompt words into a preset large language model for verification processing to obtain verification result; the verification processing flow includes symbol existence verification, attribute consistency verification and topological relationship verification; and determine the drawing recognition result of the target engineering drawing based on the verification result.

[0184] In one embodiment, the recognition result determination module 908 is further configured to, if the verification result is a pass, output the fusion processing result according to a preset structured format to obtain structured drawing recognition data, and determine the structured drawing recognition data as the drawing recognition result of the target engineering drawing; if the verification result is a fail, mark and push the failed fusion processing result item in the fusion processing result; when a corrected fusion processing result is obtained, output the corrected fusion processing result according to a preset structured format as the drawing recognition result of the target engineering drawing.

[0185] In one embodiment, the device further includes a version change report generation module, which is used to obtain the drawing recognition results of at least two historical versions corresponding to the target engineering drawing; perform structured text sequence conversion processing on the drawing recognition results of each historical version to obtain the text sequence of the drawing recognition results; input the text sequence of each drawing recognition result into a preset large language model for comparison processing to obtain the correspondence between vector graphics primitives in different versions; determine the change type of each vector graphics primitive in different versions based on the correspondence; and generate a version change report of the target engineering drawing based on the change type of each vector graphics primitive.

[0186] Each module in the aforementioned engineering drawing recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0187] In one embodiment, such as Figure 10As shown, an engineering drawing recognition system is provided, including a data input and preprocessing module 1002, a parallel recognition module 1004, a cross-modal fusion module 1006, a knowledge base verification and output module 1008, and a version management module 1010.

[0188] The data input and preprocessing module 1002 is used to acquire, parse, and preprocess vector and raster drawings, and extract drawing metadata. This module includes a vector format parser, a raster preprocessing engine, and a metadata extractor, providing standardized input for subsequent parallel recognition. The module also includes a drawing block processor, used to divide large-format drawings into image blocks adapted to the input size of large models.

[0189] The parallel recognition module 1004 is used to perform structured parsing and visual semantic recognition on vector graphics and raster graphics, respectively. This module includes a vector graphics parsing engine based on a large language model and a raster image recognition engine based on a large visual language model. The two engines work in parallel, outputting vector graphics metadata features and raster visual features, respectively.

[0190] The cross-modal fusion module 1006 is used to map vector primitive features and raster visual features to a unified semantic space, achieving alignment and complementary enhancement between the two. This module includes a coordinate registration engine, a large-model cross-modal attention layer, and a large-model-driven cross-validator. It utilizes the cross-modal understanding capabilities of the large visual language model to achieve semantic-level fusion of vector and raster features, outputting the fused recognition result.

[0191] The knowledge base verification and output module 1008 is used to verify the professionalism and consistency of the fusion recognition results based on the power engineering symbol knowledge base, and outputs the verified results in a structured format. This module includes a symbol knowledge base, a large language model verification engine, and a structure generator. Samples that fail verification are marked with problem types and then routed to a manual review queue. The verification engine uses natural language as an interface to convert knowledge base rules into prompt words, driving the large model to perform semantic-level verification without the need to write code logic for each rule.

[0192] Version Management Module 1010 is used for structured difference comparison and change tracking of different versions of the same drawing. This module includes a primitive alignment engine, a large language model change classifier, and a report generator. It achieves engineering semantic-level understanding of changes and report generation through the semantic understanding capabilities of the large model, and supports tracking the evolution history of drawings by timeline.

[0193] The above modules are connected in sequence. The output of the data input and preprocessing module 1002 is connected to the vector image parsing engine and the raster image recognition engine in the parallel recognition module 1004, respectively. The output of the parallel recognition module 1004 is connected to the cross-modal fusion module 1006. The output of the cross-modal fusion module 1006 is connected to the knowledge base verification and output module 1008. The structured data output by the knowledge base verification and output module 1008 can be selectively connected to the version management module 1010, thereby realizing the fully automated recognition process from drawing input to structured output to version comparison.

[0194] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements an engineering drawing recognition method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0195] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0196] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0197] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0198] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0199] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0200] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0201] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0202] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for recognizing engineering drawings, characterized in that, The method includes: Obtain the vector graphics and raster graphics corresponding to the target engineering drawings, and perform information extraction processing on the vector graphics to obtain vector graphics data; The vector paper data is processed by a pre-set large language model to obtain the semantic structured information of each vector element in the vector paper. The raster paper is then processed by a pre-set large visual language model to obtain the visual semantic information of the raster paper. The visual semantic information and each of the semantic structured information are input into a preset multimodal large model for fusion processing to obtain the fusion processing result; Based on the fusion processing results, the drawing recognition result of the target engineering drawing is determined.

2. The method according to claim 1, characterized in that, The vector graphic data includes the geometric and attribute information of each vector graphic element; the step of performing structured parsing processing on the vector graphic data using a preset large language model to obtain the semantic structured information of each vector graphic element in the vector graphic includes: For each vector graphic primitive, the geometric information and attribute information of the vector graphic primitive are converted into a structured text sequence according to preset rules; The structured text sequence is input into the preset large language model for semantic processing to obtain the semantic structured information of the vector primitives.

3. The method according to claim 1, characterized in that, The step of performing visual recognition processing on the raster paper using a preset large visual language model to obtain the visual semantic information of the raster paper includes: The raster paper is input into the visual encoder of the preset large visual language model for encoding processing to obtain the global visual features of the raster paper; The global visual features are decoded using the text decoder of the preset large visual language model to obtain the visual semantic information of the raster paper.

4. The method according to any one of claims 1-3, characterized in that, The step of inputting the visual semantic information and each of the semantic structured information into a preset multimodal large model for fusion processing to obtain the fusion processing result includes: The coordinate system of the vector paper is spatially aligned with the pixel coordinate system of the raster paper so that the vector primitives and the image areas of the raster paper have a spatial correspondence. The semantic structured information is uniformly encoded into text features, and the visual semantic information is encoded into visual features; The text features and the visual features are processed by the cross-attention layer of the preset multimodal large model to obtain the semantic association weights between the text features and the visual features; Based on the semantic association weights, the text features and the visual features are fused to obtain the fusion result.

5. The method according to claim 4, characterized in that, The spatial alignment process between the coordinate system of the vector drawing and the pixel coordinate system of the raster drawing includes: Extract the same-named primitives from the vector paper and the raster paper as control points; Calculate the perspective transformation matrix based on the control points; The perspective transformation matrix is ​​used to spatially align the coordinate system of the vector drawing paper with the pixel coordinate system of the raster drawing paper.

6. The method according to any one of claims 1-3, characterized in that, The determination of the drawing recognition result of the target engineering drawing based on the fusion processing result includes: A preset engineering symbol knowledge base is obtained; the rules in the preset engineering symbol knowledge base are encoded using natural language to obtain encoded rules; and the fusion processing result is encoded using natural language to obtain encoded processing result. Based on the encoded rules and the encoded processing results, a verification prompt word is constructed; The verification prompt words are input into the preset large language model for verification processing to obtain the verification result; the verification processing flow includes symbol existence verification, attribute consistency verification and topological relationship verification. Based on the verification results, the drawing recognition result of the target engineering drawing is determined.

7. The method according to claim 6, characterized in that, The fusion processing result includes multiple fusion processing result items; determining the drawing recognition result of the target engineering drawing based on the verification result includes: If the verification result is successful, the fusion processing result is output according to a preset structured format to obtain structured drawing recognition data, and the structured drawing recognition data is determined as the drawing recognition result of the target engineering drawing; If the verification result is that the verification fails, the failed fusion processing result item in the fusion processing result is marked and pushed. When the corrected fusion processing result is obtained, the corrected fusion processing result is output in a preset structured format as the drawing recognition result of the target engineering drawing.

8. The method according to claim 1, characterized in that, The method further includes: Obtain the drawing recognition results of at least two historical versions corresponding to the target engineering drawing; The recognition results of each historical version of the drawings are processed by structured text sequence conversion to obtain the text sequence of the drawing recognition results; The text sequences of the recognition results of each drawing are input into the preset large language model for comparison processing to obtain the correspondence between the vector primitives of different versions; Based on the aforementioned correspondence, determine the change type of each vector graphic element across different versions; Based on the change type of each vector graphic element, generate a version change report for the target engineering drawing.

9. An engineering drawing recognition device, characterized in that, The device includes: The engineering drawing acquisition module is used to acquire vector drawings and raster drawings corresponding to the target engineering drawings, and to perform information extraction processing on the vector drawings to obtain vector drawing data; The parallel drawing processing module is used to perform structured parsing processing on the vector drawing data through a preset large language model to obtain the semantic structured information of each vector element in the vector drawing, and to perform visual recognition processing on the raster drawing through a preset large visual language model to obtain the visual semantic information of the raster drawing. The drawing information fusion module is used to input the visual semantic information and each of the semantic structured information into a preset multimodal large model for fusion processing to obtain the fusion processing result; The identification result determination module is used to determine the drawing identification result of the target engineering drawing based on the fusion processing result.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images