Engineering drawing information extraction method, device, electronic equipment and storage medium
By directly parsing the DXF format geometric components of engineering drawings and combining vector learning and large language models, the problem of low computational efficiency in traditional methods is solved, achieving efficient and accurate extraction and intelligent processing of engineering drawing information.
Patent Information
- Application Number
- CN202411245233.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-09-06
AI Technical Summary
Existing technologies convert vector graphics data of engineering drawings into pixel images for information extraction, resulting in low computational efficiency, slow response speed, poor scalability, and significant information loss due to the difficulty of traditional visual models in accurately recognizing complex or heterogeneous images.
By acquiring DXF format geometric components from engineering drawings, embedding and encoding them, and then inputting them into a vector learning network, combined with a large language model, a hierarchical training strategy is adopted for training. The vector drawings are directly parsed, and feature extraction and learning are performed using graph neural networks and self-attention mechanisms.
It achieves efficient and lossless extraction of engineering drawing information, reduces storage space requirements, improves extraction accuracy and speed, enhances the model's ability to process complex geometric and topological information, and has high scalability and intelligence.
Smart Images

Figure CN118762381B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image recognition technology, and in particular to a method, device, electronic equipment and storage medium for extracting information from engineering drawings. Background Art
[0002] Engineering drawings often represent the valuable experience accumulated over a long period of time by a system or industry, which can be used as a good reference for future designs. It is of great social and practical significance to extract the information of these drawing images and manage them with computers. Existing engineering drawing information is not directly obtained based on vector graphics. For example, vector graphics data is converted into pixel graphics based on machine learning algorithms, and then computer vision models are used to extract engineering drawing information. Since pixel graphics lose a lot of details and visual models have limitations on the resolution of input images, it is difficult to balance accuracy and range. Since the storage space occupied by pixel graphics is much larger than that of engineering drawings themselves, calculations based on pixel graphics will result in high computing power consumption, low computing efficiency, and slow response speed. In addition, computer vision models that process pixel graphics rely on specific features of training data, and perform poorly in new fields or data with different training data distributions, resulting in poor scalability of this method. For complex or heterogeneous graphs commonly seen in engineering, converting them into pixel graphics will result in large information loss, and traditional visual models are difficult to accurately identify. Summary of the invention
[0003] The present invention provides a method, device, electronic device and storage medium for extracting information from engineering drawings, which are used to solve the problems of low computational efficiency, slow response speed, poor scalability, and inability to accurately identify complex or heterogeneous image defects in the traditional method of converting vector image data into pixel images and then using computer vision models to extract information.
[0004] The present invention provides a method for extracting information from engineering drawings, comprising:
[0005] Obtaining a vector diagram of an engineering drawing, parsing the vector diagram of the engineering drawing to obtain geometric components in a DXF format;
[0006] Embedding and encoding the DXF format geometric component, inputting the encoding result into a vector learning network, and outputting a high-dimensional feature vector corresponding to the global geometric feature through the global feature induction layer of the vector learning network;
[0007] Inputting the high-dimensional feature vector corresponding to the global geometric feature into a large language model to obtain the type and parameter information of each component in the engineering drawing;
[0008] The vector learning network is trained by a hierarchical training strategy, which includes pre-training the vector learning network on a large scale based on a multi-domain training data set, and fine-tuning the vector learning network based on an engineering drawing domain training data set based on the pre-training.
[0009] According to the method for extracting engineering drawing information provided by the present invention, obtaining an engineering drawing vector diagram and parsing the engineering drawing vector diagram to obtain a DXF format geometric component include:
[0010] Collect DWG format images of engineering drawings;
[0011] Segment the geometric elements of various components in the DWG format image to generate a set of geometric components in the DWG format;
[0012] A format conversion process is established based on a drawing design software development kit, and the geometric component set in the DWG format is processed according to the format conversion process and converted into the geometric component set in the DXF format in batches.
[0013] According to the method for extracting engineering drawing information provided by the present invention, after obtaining the geometric components in DXF format, the method further includes:
[0014] Performing data cleaning on the DXF format geometric components, and performing data normalization processing on the DXF format geometric components after data cleaning;
[0015] The DXF format geometric components after the data normalization are classified according to the geometric type and layer, the non-standard geometric primitive commands are converted into a unified representation form, and the geometric data of different layers are integrated into a unified layer framework;
[0016] Data enhancement is performed on the classified DXF format geometric components.
[0017] According to the method for extracting engineering drawing information provided by the present invention, embedding and encoding the DXF format geometric components includes:
[0018] The whole geometric information of the engineering drawing image data is described in the form of a two-dimensional matrix to obtain a DXF vector map of the engineering component. :
[0019] ;
[0020] in, For entities, For each entity, the command name corresponds to is a list of coordinates, is a list of parameters describing the geometry of the entity, is the visibility parameter, For layers, For other attributes;
[0021] for In the case of text, the Word2vec tool is used to embed words to obtain embedding codes;
[0022] for When it is not text, it is represented by multiple dimensions of embedding ,get Multi-dimensional vector representation of engineering components DXF vector map Reference The order of embedding coding is performed to generate an embedding coding matrix.
[0023] According to the method for extracting engineering drawing information provided by the present invention, the vector learning network further includes:
[0024] Feature extraction layer, used to extract geometric description features from the input sequence;
[0025] The type extraction layer is used to extract the command type in the input sequence;
[0026] The graph convolution layer is used to combine the geometric description features with the adjacency matrix of the graph to extract the geometric relationship features in the graph;
[0027] The fully connected layer is used to integrate the geometric relationship features and command types in the graph to obtain a feature vector;
[0028] The self-attention layer uses a multi-head self-attention mechanism to globally calculate the similarity between each feature vector and other feature vectors, dynamically adjusting the importance of features at different levels;
[0029] The output layer is used to output the calculation results of a specific task when the vector learning network is trained, and to verify whether the training of the vector learning network is completed according to the calculation results.
[0030] According to the method for extracting engineering drawing information provided by the present invention, the cross entropy loss function is used as the loss function during the training of the vector learning network:
[0031] ;
[0032] in: is the one-hot encoding of the true label, indicating whether category i is the true category of the training sample. is the probability that the model predicts category i, is the total number of categories.
[0033] The method for extracting engineering drawing information provided by the present invention further includes:
[0034] Optimizing the training process of the vector learning network using an optimization strategy, wherein the optimization strategy includes at least one of a sparse data optimization strategy, a training hyperparameter setting strategy, an overfitting optimization strategy, and a gradient clipping strategy;
[0035] The sparse data optimization strategy adopts a sparse matrix data format to perform sparse matrix processing, sparse coding and sparse convolution processing on the input sequence;
[0036] The training hyperparameter setting strategy includes setting an initial learning rate and gradually adjusting the initial learning rate through a cosine annealing learning rate scheduler;
[0037] The overfitting optimization strategy includes introducing a Dropout mechanism in the fully connected layer and the self-attention layer to randomly discard the outputs of some neurons during the training process;
[0038] The gradient clipping strategy includes setting the maximum gradient norm and introducing an early stopping strategy during the model training process. By monitoring the changes in model loss or model output accuracy, the training is terminated early when multiple output results do not show significant improvement.
[0039] The present invention also provides an engineering drawing information extraction device, comprising:
[0040] A parsing module, used for obtaining a vector diagram of an engineering drawing, and parsing the vector diagram of the engineering drawing to obtain geometric components in a DXF format;
[0041] An encoding module, used for embedding the DXF format geometric component, inputting the encoding result into a vector learning network, and outputting a high-dimensional feature vector corresponding to the global geometric feature through the global feature induction layer of the vector learning network;
[0042] An output module, used for inputting the high-dimensional feature vector corresponding to the global geometric feature into a large language model to obtain the type and parameter information of each component in the engineering drawing;
[0043] The vector learning network is trained by a hierarchical training strategy, which includes pre-training the vector learning network on a large scale based on a multi-domain training data set, and fine-tuning the vector learning network based on an engineering drawing domain training data set based on the pre-training.
[0044] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for extracting engineering drawing information as described in any one of the above items is implemented.
[0045] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the above-mentioned methods for extracting engineering drawing information.
[0046] The engineering drawing information extraction method, device, electronic device and storage medium provided by the present invention obtain engineering drawing vector graphics, parse the engineering drawing vector graphics to obtain DXF format geometric components; embed the DXF format geometric components, input the encoding results into a vector learning network, and output a high-dimensional feature vector corresponding to the global geometric feature through the global feature induction layer of the vector learning network; input the high-dimensional feature vector corresponding to the global geometric feature into a large language model to obtain the type and parameter information of each component in the engineering drawing; wherein the vector learning network is trained by a hierarchical training strategy, and the hierarchical training strategy includes a method based on multiple fields. The vector learning network is pre-trained on a large scale based on a domain training data set. Based on the pre-training, the vector learning network is fine-tuned based on the engineering drawing domain training data set. By directly parsing the CAD vector information corresponding to the engineering drawings, it is efficient and lossless. Through embedded coding, it can directly process drawings in CAD format, requires little disk storage space, and is completely unaffected by scaling, resolution, and image compression distortion, so it can achieve lossless parsing of engineering drawings. Combining the vector learning mechanism and the large language model, the output accuracy is improved. The hierarchical training strategy can effectively capture and learn the characteristic laws of vector graphics, thereby improving the intelligence level of the entire engineering drawing information extraction. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0048] Figure 1 It is a flowchart of a method for extracting engineering drawing information provided by an embodiment of the present invention;
[0049] Figure 2 It is a schematic diagram of a framework for establishing an engineering vector data set provided by an embodiment of the present invention;
[0050] Figure 3 It is a schematic diagram of a sample geometric vector diagram of some components in the engineering vector data set provided by an embodiment of the present invention;
[0051] Figure 4 It is a schematic diagram of eight types of DXF entities and their vector representations mainly involved in the engineering vector graphics provided by the embodiment of the present invention;
[0052] Figure 5 It is a schematic diagram of an end-to-end overall model framework provided by an embodiment of the present invention;
[0053] Figure 6 It is a schematic diagram of a vector learning network based on a graph neural network and a self-attention mechanism provided by an embodiment of the present invention;
[0054] Figure 7 It is a schematic diagram of the Accuracy and Loss change process of the vector learning network model classification task training process for the example data set provided by the embodiment of the present invention;
[0055] Figure 8 It is a functional structure diagram of an engineering drawing information extraction device provided by an embodiment of the present invention;
[0056] Fig. 9 It is a functional structure diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0057] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0058] Figure 1 A flowchart of a method for extracting engineering drawing information provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method for extracting engineering drawing information provided by the embodiment of the present invention includes:
[0059] Step 101, obtaining a vector diagram of an engineering drawing, parsing the vector diagram of the engineering drawing to obtain geometric components in a DXF format;
[0060] Step 102, embedding and encoding the DXF format geometric components, inputting the encoding results into a vector learning network, and outputting a high-dimensional feature vector corresponding to the global geometric features through the global feature induction layer of the vector learning network;
[0061] Step 103: input the high-dimensional feature vector corresponding to the global geometric feature into a large language model to obtain the type and parameter information of each component in the engineering drawing;
[0062] The vector learning network is trained by a hierarchical training strategy, which includes pre-training the vector learning network on a large scale based on a multi-domain training data set, and fine-tuning the vector learning network based on an engineering drawing domain training data set based on the pre-training.
[0063] Traditional methods for extracting information from engineering drawings use machine learning algorithms to convert vector graphics data into pixel graphics, and then use computer vision models to extract information. Since pixel graphics lose a lot of details and visual models have limitations on the resolution of input images, it is difficult to balance accuracy and range. Since pixel graphics occupy much more storage space than engineering drawings themselves, calculations based on pixel graphics will result in high computing power consumption, low computing efficiency, and slow response speed. In addition, computer vision models that process pixel graphics rely on specific features of training data, and perform poorly in new fields or data with different training data distributions, resulting in poor scalability of this method. For complex or heterogeneous graphs commonly seen in engineering, converting them into pixel graphics will result in large information loss, and traditional visual models are difficult to accurately identify.
[0064] The engineering drawing information extraction method provided by the embodiment of the present invention is used to obtain engineering drawing vector graphics through a parsing module, parse the engineering drawing vector graphics to obtain DXF format geometric components; embed the DXF format geometric components, input the encoding results into a vector learning network, and output high-dimensional feature vectors corresponding to global geometric features through the global feature induction layer of the vector learning network; input the high-dimensional feature vectors corresponding to the global geometric features into a large language model to obtain the types and parameter information of each component in the engineering drawing; wherein the vector learning network is trained by a hierarchical training strategy, and the hierarchical training strategy includes pre-training the vector learning network on a scale based on a multi-domain training data set, and fine-tuning the vector learning network based on a training data set in the engineering drawing field on the basis of the pre-training, and directly parsing the vector graphics is efficient and lossless, and the vector graphics can be directly encoded by embedded coding, and the required disk storage space is small, and it is completely unaffected by scaling, resolution, and image compression distortion, so that lossless parsing of engineering drawings can be achieved; combining the vector learning mechanism and the large language model, the output accuracy is improved, and the hierarchical training strategy can effectively capture and learn the characteristic laws of vector graphics primitives, thereby improving the intelligence level of the entire engineering drawing information extraction.
[0065] Based on any of the above embodiments, the step of obtaining an engineering drawing vector diagram and parsing the engineering drawing vector diagram to obtain a DXF format geometric component includes:
[0066] Step 201, collecting DWG format images of engineering drawings;
[0067] In an embodiment of the present invention, examples of engineering drawings in DWG format, such as commonly used architectural drawings and construction drawings, are used as a data basis, which are classified, collected, summarized, and organized into an engineering drawing database.
[0068] Step 202: Segment various components in the DWG format image by geometric elements to generate a set of geometric components in the DWG format;
[0069] In the embodiment of the present invention, various components in the drawing are segmented into geometric elements to form a geometric component set in DWG format.
[0070] Step 203: Establish a format conversion process based on the drawing design software development kit, process the geometric component set in the DWG format according to the format conversion process, and convert them into the geometric component set in the DXF format in batches.
[0071] In an embodiment of the present invention, a format conversion process is established based on the ODA Drawings SDK to batch convert geometric component sets in the DWG format into geometric component sets in the DXF format that are convenient for subsequent data processing and use, so as to facilitate subsequent processing.
[0072] In some embodiments of the present invention, after obtaining the geometric component in DXF format, the method further includes:
[0073] Performing data cleaning on the DXF format geometric components, and performing data normalization processing on the DXF format geometric components after data cleaning;
[0074] In an embodiment of the present invention, data cleaning includes:
[0075] Delete invalid data: Filter out invalid or unnecessary entities, such as blank layers, duplicate data, etc.
[0076] Repair data errors: Correct geometric errors such as self-intersections, extra vertices, etc.
[0077] Data normalization includes:
[0078] Unit conversion: Convert geometric data in different units into a unified unit.
[0079] Coordinate system conversion: convert geometric data in different coordinate systems into a unified coordinate system.
[0080] Coordinate origin alignment: Align the coordinate origins of all geometric data to a unified position.
[0081] Data Scaling: Scale all geometric data so that they can be compared and analyzed at the same scale.
[0082] Rotation Alignment: Aligns the orientation of all geometric data to a uniform direction.
[0083] Centralization: Centralize all geometric data so that they are based on a unified center point.
[0084] The DXF format geometric components after the data normalization are classified according to the geometric type and layer, the non-standard geometric primitive commands are converted into a unified representation form, and the geometric data of different layers are integrated into a unified layer framework;
[0085] Data enhancement is performed on the classified DXF format geometric components.
[0086] In an embodiment of the present invention, data enhancement includes:
[0087] Data augmentation: Generate more data samples through rotation, translation, mirroring, etc.
[0088] Noise addition: Introduce an appropriate amount of noise into the data to enhance the model's adaptability to actual conditions.
[0089] In terms of the primitive vector representation method and data set, the DXF data format is selected for data representation and processing of engineering drawings, and the open source format DXF format corresponding to DWG is selected as the data format basis. The DXF format is designed to facilitate data exchange between different CAD software. Figure 2 As shown in the figure, a training data set is established based on the DXF format. The data set construction includes label addition: adding labels to the geometric component data, marking its category, attribute and other information. Data segmentation: dividing the data into training set, validation set and test set to facilitate model training and evaluation.
[0090] In an embodiment of the present invention, some typical engineering plan component elements are selected for in-depth research and data extraction. These elements include but are not limited to doors, windows, columns, straight walls, cross walls, T-walls, etc., representing the most common component forms in architectural plan design. The selection of these components not only takes into account their wide application in actual engineering, but also their representation in CAD drawings and the parsability of their data structure. Based on these typical engineering component elements, an engineering vector data set covering a variety of engineering graphic elements is established by using format conversion, data processing and enhancement, as well as label addition and data segmentation methods. In an embodiment of the present invention, the engineering vector data set is established in combination with the collection of standard atlases published in various places. Four parts, namely columns, doors, windows and walls, are selected as the most typical key components, and 984 component CAD drawing instances are generated and processed into DXF data format to form a data set. Component geometry vector patterns, for example Figure 3As shown. This dataset covers 984 engineering component geometric vector elements in 6 major categories, all in DXF format, which can significantly represent various common components in building floor plans, and provide rich and diverse training data for subsequent engineering graphic perception and representation learning tasks. The construction of this dataset not only provides a solid foundation for the end-to-end vector perception method of engineering drawings in the present invention, but also lays an important data support for future engineering automation design and intelligent construction research. The following is a detailed introduction to the dataset establishment process of the four parts of columns, doors, windows and walls. The proportion of each type of data is shown in Table 1. At the same time, the training set and the test set are divided according to the ratio of 7.2:1.
[0091] Table 1 Proportion of each component category in the proposed engineering vector dataset
[0092]
[0093] The size and shape of the column include square and rectangular sections with side lengths between 300mm and 1200mm. The relative relationship between the column and the wall includes the following five methods: the center of the column is aligned with the center of the wall, the column is aligned below the wall edge, the center of the column is slightly above the center of the wall, the column is aligned above the wall edge, and the center of the column is slightly below the center of the wall. Each alignment method is combined with the size and shape of the column to organize and create 160 examples. All columns are saved separately in a CAD file, and the file name is "column + number", for example: "column 1".
[0094] There are three types of walls: straight wall, T-shaped wall where two walls intersect, and cross wall:
[0095] The single-piece wall, that is, the conventional single-piece wall, includes two forms: vertical and horizontal, with lengths ranging from 200mm to 4100mm, and a total of 168 single-piece wall instances.
[0096] The length of the T-wall where the two walls intersect ranges from 600mm to 4100mm. Five alignment modes are set at the intersection of the vertical wall and the horizontal wall: left alignment, center left, center, center right, and right alignment. A total of 168 T-wall instances are included.
[0097] The length of the cross wall ranges from 300mm to 9200mm. The intersection of the two walls is combined according to the left, center, and right of the horizontal wall and the top, center, and bottom of the vertical wall, forming a total of nine alignment methods, including a total of 168 cross wall instances.
[0098] All walls are saved separately in a CAD file, and the file name is "wall + number", for example: "cross wall 1".
[0099] The door category uses a typical door expression in the construction drawing. The door width ranges from 700mm to 1100mm, and is inserted into a wall with a width of 7500mm. The opening methods include inward-opening left door, inward-opening right door, outward-opening left door, and outward-opening right door. The positions of insertion into the wall include left end, center left, center, center right, and right end. Each combination of opening method and insertion position is included, including a total of 160 door instances. Each door is numbered according to the construction drawing standard, and the number is entered in the nearest position using the text tool. All doors are saved separately in a CAD file, and the file is named "door+number", for example: "door 1".
[0100] The window class uses a typical window expression in the construction drawing. The width of the window ranges from 600mm to 1380mm, and it is inserted into a wall with a width of 7500mm. The insertion positions include the left end, center left, center, center right, and right end. Each insertion position is included, and a total of 160 window instances are generated. Each window is numbered according to the construction drawing standard, and the number is entered in the nearest position using the text tool. All windows are saved separately in a CAD file, and the file name is "window + number", for example: "window 1".
[0101] Based on any of the above embodiments, the step of embedding the DXF format geometric component includes:
[0102] The whole geometric information of the engineering drawing image data is described in the form of a two-dimensional matrix to obtain a DXF vector map of the engineering component. :
[0103] ;
[0104] in, For entities, For each entity, the command name corresponds to is a list of coordinates, is a list of parameters describing the geometry of the entity, is the visibility parameter, For layers, For other attributes;
[0105] for In the case of text, the Word2vec tool is used to embed words to obtain embedding codes;
[0106] for When it is not text, it is represented by multiple dimensions of embedding ,get Multi-dimensional vector representation of engineering components DXF vector map Reference The order of embedding coding is performed to generate an embedding coding matrix.
[0107] In an embodiment of the present invention, in order to directly learn vector information, a set of vector primitive embedding encoding methods based on the relative order of vector parameter types, coordinate values and commands is established based on the vector description of the DXF data format, so that deep learning of vector data based on DXF is possible. DXF is a vector graphics format based on ASCII text encoding, which is stored in the form of plain text and contains a large number of labels and data pairs. Each entity (such as line segments, circles, text, etc.) is described by a specific label and value. Each data pair consists of a code and a value associated with the code, and the code is used to identify the next data type. The structure of a DXF file usually includes multiple parts, such as HEADER (containing global settings of the file), CLASSES (custom class definitions), TABLES (layers, line types, etc.), BLOCKS (definition blocks), ENTITIES (basic geometric objects of drawing) and OBJECTS (non-graphic data, such as dictionaries, etc.). Among them, the ENTITIES entity is the basic geometric description in the graphics, and it is also the key to the engineering vector representation and primitive feature learning.
[0108] The main ENTITIES entity types involved in engineering drawing vector graphics are the following eight categories: TEXT (text), LINE (straight line), CIRCLE (circle), ARC (arc), ELLIPSE (ellipse or elliptical arc), POLYLINE (polyline), LWPOLYLINE (lightweight polyline), SPLINE (spline), etc. The complete engineering component metafile can be decomposed into the above entities, and these entities themselves are defined as a series of specific drawing commands. Based on commands and parameters, closed or open curves are drawn. The main geometric entity drawing commands and their vector representation methods in the DXF data structure are as follows: Figure 4 shown.
[0109] In the embodiment of the present invention, G is used to represent vector graphics, and the first letter E of the ELLIPSE entity is used to refer to each command used to draw the entity. The DXF vector diagram of the engineering component after data processing can be represented by vector using formula (1). Usually, the number of entities n involved in a single component will not exceed 100. At the same time, as shown in formula (2), for the drawing of a single entity, its drawing command is the geometric representation of the entity, which consists of an entity type command and drawing parameters, which will be explained in detail in the next paragraph. Visibility is often used to hide unnecessary entity graphics. The layer directly specifies the line shape, color and other information of the line. The same layer usually has the same line shape, color and other attributes, which can be described as It is a set of all layer names, as shown in formula (2). The number of layers in each drawing is usually m≤100. In special cases, special line shapes can be represented by other attributes. Specified individually, attributes A list consisting of three common attributes: Color, LineType, and LineWeight, as well as uncommon attributes such as material, shadow, and style. Usually the total number of other attributes is l≤32.
[0110] (1)
[0111] (2)
[0112] Considering that the method of specifying various attributes through layers is commonly used in actual engineering drawing, The three types of information constitute the entire description of the actual engineering drawing information, so these three types of information are the main objects for vector embedding of DXF primitives. can be described as formula (3), ignoring uncommon command names, Then it belongs to the set of all 8 types of command names, as shown in formula (4); is a list of j coordinates describing the geometric position of the entity, as shown in formula (5). The length of the list j is determined by the command name Decision, usually 1 to 3, even in a polyline entity, the number of coordinates in the engineering scene will not exceed 8; It is a list of k parameters describing the geometric shape of the entity, as shown in formula (6). The entity parameters include direction vector, ratio of major and minor axes of ellipse, start and end parameters, etc., which are also determined by the command name. The most common 8 types of entities all satisfy k≤8.
[0113] (3)
[0114] (4)
[0115] (5)
[0116] (6)
[0117] Any entity can be abstracted into a vector list described by the following formula (7), that is, the entire geometric information of the engineering component vector map G can be described in the form of a two-dimensional matrix. The order is the order of parsing the entities within the primitives. This order has a clear physical meaning: it is the drawing order of engineering drawings.
[0118] (7)
[0119] According to the statistical characteristics of actual engineering data, the number of entities involved in the engineering vector diagram G will not exceed 128 dimensions. , In the list of various elements in , the command types do not exceed 8, the number of coordinates j usually does not exceed 8, the number of parameters k≤8, the number of layers m≤100, and the total number of other attributes l≤32. Therefore, the FP32 fixed-length vector method can be used to Embedded encoding.
[0120] In terms of the embedded coding method of each component, In the case of , the entity type is text, and no geometric graphics are involved in the embedding code, then refer to the natural language processing method and directly use Word2vec combined with the language model for embedding. In the case of , the corresponding entity types are all pure geometric figures, and Ei can be directly embedded as a 128-dimensional vector. The specific embedding method is as follows:
[0121] 1) Embedding with 8 dimensions ,The valid values of different dimensions represent different command types, and 8 dimensions are sufficient to embed all entity command types in the vector graph;
[0122] 2) Embed with 48 dimensions , which is enough to embed 16 items of coordinate information. However, the number of coordinates in actual scenes usually does not exceed 8 items. The reserved 16 coordinate embedding items are sufficient to fully meet the needs of geometric description of entity drawing.
[0123] 3) Embedding with 24 dimensions , usually the number of command parameters is 1~3, and no more than 8 items, among which a small part of the parameters are three-dimensional lists. Using 24 dimensions is sufficient to simultaneously meet the embedding of 8 three-dimensional vector parameters in extreme cases;
[0124] 4) Embedding with 8 dimensions ,Considering the memory alignment of deep learning, 8 is used as the minimum embedding dimension here;
[0125] 5) Embedding with 8 dimensions , similarly, it is convenient to meet the memory alignment of deep learning models;
[0126] 6) Embedding with 32 dimensions , which is sufficient to meet the needs of embedding other attributes of entities in various exceptional cases;
[0127] Referring to the above method, any unused vector values are directly set to -1 for completion, and the embedding can be made into a 128-dimensional vector , as shown in formula (8), the value of each sub-item in the vector is recorded as q. In the same way, G is referenced to The order of embedding coding can be processed into a two-dimensional array G, that is, the form of matrix G. In the embedding coding process, all sub-items q of unused sub-vectors are directly set to -1, which can complete the matrix, so that the engineering component vector map G can be embedded into a complete 128×128-dimensional two-dimensional array. The data type of each sub-item is FP32 floating point number, as shown in formula (9), which can realize the overall embedding coding of the engineering component DXF primitive. In summary, the engineering component vector map G can be embedded and encoded into a two-dimensional array G, which has all the geometric information and drawing order information of the engineering component, and can be conveniently used for subsequent vector learning and end-to-end perception training of drawing features.
[0128] (9)
[0129] (10)
[0130] It should be noted that the present invention does not limit the number of matrix dimensions. For more complex vector data, similar methods can be used to embed it into a higher-dimensional array or a higher-order tensor form for vector learning.
[0131] Based on any of the above embodiments, a complete end-to-end perception method for engineering drawing information is established through a vector learning network. The overall framework is as follows: Figure 5 The framework may involve multiple processes and steps, aiming to achieve end-to-end vectorized information-aware learning of engineering artifacts.
[0132] The framework uses the CAD vector graphics of engineering components as input data in the DXF data format. First, the CAD vector graphics data of engineering components is split into two types: text entities and geometric entities. The information in the text entity is divided into two parts: the text content of the entity and the geometric attributes. The text content is processed using natural language processing technology and a text segmentation method enhanced by a special vocabulary to facilitate subsequent input into the language model; the geometric entity is in the same encoding format as the geometric attribute information of the entity, and the unified engineering vector representation and DXF primitive embedding encoding method mentioned above are used to embed it into a two-dimensional array of 128×128 dimensions for subsequent vector feature learning.
[0133] The core part of the framework is the vector learning network, which establishes a deep learning model based on graph neural network and Transformer self-attention mechanism, can learn the vector features of a large number of engineering components, and establish a model that can accurately map vector features to vector space, which can be used to identify, classify, cluster, etc. different components. The results of model reasoning can be output through two parts. The first is the output layer, which is used to output the result annotation of each sub-item for specific classification, clustering and other tasks separately. As a description of the geometric information in the engineering vector map, its output is a lossy description of some geometric features, which can quickly meet various tasks; the second is the global feature induction layer, which is used to completely abstract the global geometric features obtained by vector learning into the form of high-dimensional feature vectors, which can be further embedded in various large models for further learning and reasoning. Its output is a lossless description of all geometric features, which can be used for more complete model training.
[0134] The framework also includes end-side output based on the domain-specific LLM. End-side output refers to the result after model processing, which is directly applied to the final output of a specific scene or device. The text content is embedded in NLP word segmentation, and the geometric entity is vector-encoded based on GNN and self-attention mechanism. Appropriate learning. For the learning results of the above information, a domain-specific large language model (LLM) is used for unified integrated output, which can describe semantic knowledge, global geometric features, and geometric information. It can accurately understand and generate content related to the field, so that the overall model can achieve end-to-end perception of drawing information.
[0135] In the embodiment of the present invention, the feature array of the engineering component CAD drawing is input into the vector learning network and the tensor operation and the various steps of the flow are as follows: Figure 6 shown.
[0136] The vector learning network structure includes basic modules such as input feature extraction layer, type embedding layer, graph convolution layer, self-attention layer, global feature induction layer, and output layer in sequence. The introduction of graph neural network and self-attention layer significantly enhances the vector relationship extraction and understanding ability of the model. Among them, the feature extraction layer is used to extract geometric description features in the input sequence; in an embodiment of the present invention, the input data is processed by a one-dimensional convolution operation to convert the input feature dimension into the output feature dimension. The recommended number of CNN layers is ≥6 layers, and stride layers and pooling layers are used after each convolution layer. Recommended data dimension parameters include: input data shape: (batch_size, 128, 128), output data shape: (batch_size, 256,32, 32), batch_size≥32.
[0137] The type extraction layer is used to extract the command type in the input sequence; only the first 8 items of the entity vector need to be extracted. The recommended number of MLP layers is ≥ 1, and each layer includes a fully connected layer and an activation function (GELU) and a dropout operation, and then reshaped to the output dimension. The recommended data dimension parameters include input data shape: (batch_size, 128, 8), output data shape: (batch_size, 32, 32), batch_size ≥ 32.
[0138] The graph convolution layer is used to combine the geometric description features with the adjacency matrix of the graph to extract the relational features of the geometry in the graph; through the graph convolution operation, the input node features are combined with the adjacency matrix of the graph to extract the relational features of the geometry in the graph. GCN (graph convolutional network) updates the representation of each vector feature point by aggregating the features of neighboring nodes, thereby capturing the dependency and structural information of the geometry. The recommended number of GCN layers is ≥ 2 layers, and at least two layers of GCN are usually used to allow for deeper feature aggregation between nodes. The first layer can aggregate the information of the first-order neighbors, and the second layer further aggregates the information of the second-order neighbors, thereby enhancing the model's understanding of the graph structure. Each GCN layer includes graph convolution, GELU activation function, normalization and Dropout. The recommended data dimension parameters include input data shape: (batch_size, 256, 32, 32), output data shape: (batch_size, 256, 32, 32), batch_size ≥ 32.
[0139] The fully connected layer is used to integrate the geometric relationship features and command types in the graph to obtain a feature vector;
[0140] The recommended number of MLP layers is ≥ 1, which is achieved through MLP and Concat concatenation and data shape transformation. The (batch_size, 32, 32) data output by the type output layer is transformed into (batch_size, 1, 32, 32), and then concatenated with the geometric features into (batch_size, 257, 32, 32), and then retracted into a tensor shape of (batch_size, 256, 32, 32) through a 1×1 convolution layer for output, which is convenient for subsequent self-attention learning. The recommended data parameter dimensions include input data shapes: (batch_size, 32) and (batch_size, 256, 32, 32), and output data shapes: (batch_size, 256, 32, 32) batch_size ≥ 32.
[0141] The self-attention layer calculates the similarity between each feature vector and other feature vectors globally through a multi-head self-attention mechanism, and dynamically adjusts the importance of different hierarchical features; the embodiment of the present invention can enhance the model's understanding of global contextual information. In particular, for high-dimensional vector features, the self-attention layer can effectively aggregate global information and improve the model's ability to learn complex feature patterns. The recommended number of Transformer structure layers is ≥6, and each layer calculates the attention weight through the inner product between the query, key, and value, and weighted sums them to obtain the output features. The recommended data dimension parameters include input data shape: (batch_size, 256, 32, 32), output data shape: (batch_size, 256, 32, 32), batch_size≥32.
[0142] The output layer is used to output the calculation results of specific tasks when the vector learning network is trained, and to verify whether the vector learning network is trained based on the calculation results. This layer needs to be specifically designed and implemented for specific tasks in order to meet the requirements of various tasks such as classification, regression, and clustering. Classification and regression tasks are usually implemented through global pooling and FC fully connected layers. Clustering tasks also need to superimpose the K-means algorithm on the above basis, and generation tasks are more complex, and multiple Transformer upsampling layers need to be superimposed to achieve it. The following is an example only for classification tasks. The recommended number of FC layers is ≥4 layers, and the output dimension of the last layer is equal to the number of categories of classification. Recommended data dimension parameters include input data shape: (batch_size, 256, 32, 32), output data shape: (batch_size, number of categories), batch_size≥32.
[0143] Global feature induction layer (pooling layer): The purpose of this layer is to output global geometric features, flattened into a one-dimensional vector for large language model learning and use against geometric information, which is implemented by multiple adaptive average pooling (AAP) layers. The recommended number of AAP layers is ≥ 2, and the recommended data dimension parameters include input data shape: (batch_size, 256, 32, 32), output data shape: (batch_size, m≥2048), batch_size≥32.
[0144] In an embodiment of the present invention, the cross entropy loss function is used as the loss function during the vector learning network training:
[0145] ;
[0146] in: is the one-hot encoding of the true label, indicating whether category i is the true category of the training sample. is the probability that the model predicts category i, is the total number of categories.
[0147] The model training strategy needs to be adjusted according to the different goals of end-to-end perception of engineering data. The training strategy of the vector learning network determines the model's effective learning ability for vector primitive features. During the training process, large-scale pre-training is first performed to obtain a wide range of vector feature expression capabilities, and then fine-tuning is performed for the field of engineering drawings to ensure that the model has a high degree of accuracy and robustness in identifying and understanding complex vector components. Through a hierarchical training strategy, the performance of the model in specific field tasks is gradually improved, so that it can effectively capture and learn the characteristic laws of vector primitives, thereby improving the overall intelligence level of the system. Specifically, in most cases, the output of the vector learning network is a two-dimensional tensor with a data shape of (batch_size, m), where m can represent the number of categories in the classification task (or the dimension of the output feature vector, etc.), and each value in the output tensor is the predicted probability that each sample belongs to each category. Through the softmax function, these probability values will be between [0, 1], and the sum of the probabilities of all categories is 1. During the training process of the model, the true label label of the dataset provides the target output of the model, that is, the correct category corresponding to each input sample, which is used to calculate the difference between the model prediction and the true label. Therefore, the vector learning network model uses the cross-entropy loss function as the loss function used for model training, which can measure the difference between the model prediction and the true label and guide the model to optimize.
[0148] The above loss function is used to measure the difference between the feature probability distribution of the component vector map sample predicted by the vector learning network and the true distribution. The smaller the loss, the closer the model's prediction is to the actual situation. In multi-classification tasks, cross entropy loss can effectively guide model learning, thereby improving classification accuracy. Specifically, the feature extraction layer and graph convolution layer of the vector learning network are used to extract features, and the self-attention layer based on the Transformer structure is used to capture global information. The final prediction result has a shape of (batch_size, num_classes). The cross entropy loss function compares this prediction result with the true label label, calculates the loss value, and updates the model parameters through back propagation to optimize the model.
[0149] In an embodiment of the present invention, the engineering drawing information extraction method further includes:
[0150] Optimizing the training process of the vector learning network using an optimization strategy, wherein the optimization strategy includes at least one of a sparse data optimization strategy, a training hyperparameter setting strategy, an overfitting optimization strategy, and a gradient clipping strategy;
[0151] The sparse data optimization strategy adopts a sparse matrix data format to perform sparse matrix processing, sparse coding and sparse convolution processing on the input sequence;
[0152] Compared with traditional grid diagrams, vector diagrams can express richer geometric features with very little effective information, and the data is much sparser after encoding, so the primary training strategy is to optimize the characteristics of sparse data. Specifically, it includes: using sparse matrix data format, reducing unnecessary calculations through sparse matrix processing (such as arranging the data in the sparse matrix to the diagonal), sparse coding (converting the data in the sparse matrix into tensors), and sparse convolution, and improving training results through L1 regularization and model pruning. These strategies can not only improve computing efficiency, but also effectively reduce overfitting, improve the generalization ability of the model and training stability.
[0153] The training hyperparameter setting strategy includes setting an initial learning rate and gradually adjusting the initial learning rate through a cosine annealing learning rate scheduler;
[0154] In the gradient descent process of training, the Adam optimizer was selected. This optimizer combines the advantages of momentum gradient descent and RMSProp and can perform well when processing high-dimensional data and sparse gradients. The initial learning rate is set to 0.0001 and is gradually adjusted by the cosine annealing learning rate scheduler. During the entire training process, the learning rate decreases according to the cosine curve. This strategy can prevent the model from converging too quickly in the early stage of training, and maintain a low learning rate in the later stage of training to finely optimize the model parameters.
[0155] The overfitting optimization strategy includes introducing a Dropout mechanism in the fully connected layer and the self-attention layer to randomly discard the outputs of some neurons during the training process;
[0156] To prevent the model from overfitting, the Dropout mechanism is introduced in the fully connected layer and the Transformer layer. Dropout randomly discards the output of some neurons during training to simulate the effect of model integration. The recommended Dropout rate is 0.2 to 0.3 to ensure that the model does not lose too much information when extracting complex features. In addition, in the convolutional layer, BatchNormalization is introduced to normalize the data distribution of each batch and eliminate the internal covariate shift phenomenon. This not only speeds up the convergence of the model, but also makes the training process more stable.
[0157] The gradient clipping strategy includes setting the maximum gradient norm and introducing an early stopping strategy during the model training process. By monitoring the changes in model loss or model output accuracy, the training is terminated early when multiple output results do not show significant improvement.
[0158] The maximum gradient norm is set to 1.0 to prevent abnormal parameter updates that may be caused by gradient explosion. During model training, an early stopping strategy is also introduced. By monitoring the changes in the validation set loss or accuracy, if no significant improvement is seen within a predetermined number of epochs, the training will be terminated early to avoid unnecessary overfitting.
[0159] Model checkpointing is performed regularly during the training process, and the optimal model parameters are saved based on the performance of the validation set (such as the lowest loss or the highest accuracy), ensuring that the best model can be restored and used when the training is interrupted or reaches the optimal level. The application of these comprehensive strategies not only improves the learning ability of the model in the multi-dimensional feature space, but also ensures the efficiency of the training process and the stability of the model in practical applications.
[0160] In the embodiment of the present invention, a vector learning network and a domain-specific large language model for outputting end-side vector learning results are included. The latter is implemented based on an open source large language model skeleton, and after fine-tuning the special corpus in the field of engineering drawings, the parameter scale reaches 2 billion. Since the model has completed comprehensive pre-training and fine-tuning, under the task of the present invention, good regularity learning and end-to-end perception effect output can be achieved by adjusting the prompt.
[0161] Based on any of the above embodiments, model training is performed under the Ubuntu 20.04 operating system, using a high-performance computing configuration. In terms of hardware, an Intel Xeon E5-2686V4 processor and 64GB of memory are used to ensure the efficiency of data processing and model calculation. In addition, the training process relies on the NVIDIA RTX 3090 graphics card (24GB of video memory) for deep learning calculations to accelerate the training and reasoning of large-scale models. The software environment is based on Python 3.8 and uses the Pytorch2.3 deep learning framework to ensure compatibility with higher versions, providing a stable foundation for the development and deployment of the model. Based on the proposed vector learning method based on GNN and self-attention mechanism, its deep learning model instance is implemented and trained in the Pytorch framework. In order to improve the generalization ability of the model, floating-point operations with a precision of fp32 are used during training, and a dropout mechanism is introduced in the model, with a dropout rate set to 0.2. In addition, the learning rate is dynamically adjusted using the cosine annealing strategy. The initial learning rate is 0.0001, and the final learning rate is 10% of the initial value, that is, 0.1 times. In the configuration of the self-attention mechanism, the model adopts a multi-head attention structure with the following parameters: depth is 8 and the number of heads is 8. These parameter settings ensure that the model can fully capture the complex associations and subtle features between graphics when learning vector graphics features, thereby improving the performance of the model in practical applications.
[0162] The entire model contains 1,605,378 parameters, and uses a moderate batch size of 32. The memory usage during training is 416 MiB. The dataset is randomly divided into training and validation sets according to a ratio of 7.2:1. The training set is used to train and update the vector learning model, and the validation dataset is used to test performance. The model training was performed for 30 epochs, and the total time consumed was 1744.254 seconds. The accuracy and loss function curves of the training process are shown in Figure 2. Figure 7 As shown. At the 28th epoch, the model achieved the best performance with a validation set accuracy of 95.8% and a loss function value of 0.372. At the same time, in order to verify the results, the instance model prediction output is compared with the gold standard, and the precision (P), recall (R) and F1 score (F1) are calculated for each semantic label:
[0163] (11)
[0164] (12)
[0165] (13)
[0166] Among them, N{correct classification, classification, true} represents the number of elements of {model correct classification, model classification, true} for each label. Finally, the weighted average (micro-average) F1 score of each label is used to measure the overall performance. After experimentation, the test results of the model on the validation set are shown in Table 2. It can be seen that the model achieved a comprehensive accuracy, recall rate and F1 score of 97.06%, 95.70%, and 95.71%, respectively. This result shows that the proposed vector learning method based on GNN and self-attention mechanism can be applied to the feature learning of various engineering component vector data, and can obtain quite accurate results.
[0167] Table 2 Six-category training results of the vector learning model
[0168]
[0169] At the same time, its total video memory occupies only 416MiB of video memory, and can run smoothly on any consumer-grade GPU. On the RTX3090 GPU, when the inference batch size is set to 32, the above example model can achieve a running speed of more than 1000FPS per second, far exceeding traditional image feature learning methods based on grid maps such as CNN or ViT.
[0170] After completing the training of the vector learning model, the vector learning method can extract the components in the drawings and classify and extract attribute information. On this basis, the features of the vector learning model need to be input into a domain-specific large language model to complete the end-to-end perception of engineering drawing information and generate a description statement of the geometric information of the engineering drawing.
[0171] In the implementation of the domain-specific large language model: through the fine-tuning of special sentences, the RAG knowledge base framework, and the prompt project, the effect of accepting the output features of the vector model and outputting the text description of the vector map is achieved. Specifically, the open source Gemma 2 2b model is selected as the basic framework of the large model. On this basis, it is fine-tuned using text data containing hundreds of engineering component description information to make it better adapt to the proposed end-to-end perception method task. Specifically, the large language model is designed to receive the component geometric features extracted by the vector learning model and generate the corresponding engineering component description sentences. In order to further improve the generation ability of the large language model, a Retrieval-Augmented Generation (RAG) system is built for the large language model. In this system, the model can access and utilize a knowledge base embedded with professional architectural construction drawing data. This knowledge base contains rich background information in the engineering field, so that the large language model can combine more industry-specific professional knowledge when generating description sentences, thereby significantly enhancing the accuracy and professionalism of the text generated by the vector map. With the support of this RAG system, the large language model can not only generate descriptions that conform to geometric features, but also be closer to actual engineering application scenarios in expression, and achieve efficient docking between vector graphics and text descriptions. Finally, a dedicated descriptive graphic text prompt is used to establish a calling method for the large model. After the above process, the tuned large model is deployed on the software and hardware platform, and quantized using the int4 method to further improve the model generation speed. The above-mentioned dedicated large model quantized by int4 can achieve a high-efficiency output of 70 to 100 tokens per second in the text description generation, while occupying only 4GB of video memory. When deployed together with the vector learning model in the previous article, the total video memory occupied does not exceed 5GB. In addition to the training platform with RTX3090 GPU, it can also run smoothly on mobile notebooks with RTX4060 and above GPUs. The end-to-end perception methods of engineering drawing information can well describe the characteristics of engineering vector graphics.
[0172] The engineering drawing information extraction method provided by the embodiment of the present invention has a strong vector map learning ability, can realize complete geometric vector information learning in the plane map sample, and complete the end-to-end perception of the geometric information of the engineering drawing. The proposed method lays a complete theoretical and practical foundation for the future larger-scale drawing vector learning and application. It shows significant advantages in processing vector drawing information. First, by directly processing vector data, the processing speed and accuracy are greatly improved, ensuring the integrity and accuracy of the design information. Secondly, the technical integration of the graph neural network and the self-attention mechanism not only improves the model's ability to process complex geometric and topological information in the drawings, but also significantly enhances the model's learning efficiency and application potential. After testing, the method achieved an accuracy of 95.8% on a typical data set, and at the same time, it can achieve an inference speed of more than 1000FPS, reaching a leading level in the accuracy and efficiency of drawing information perception. Direct extraction of engineering drawing vector features: Compared with the traditional processing method of engineering drawings, that is, converting vector maps into pixel images and then learning, directly using vector CAD drawings as input can more accurately retain all the details and accuracy of the design, which is a major innovation. This method avoids the information loss caused by image resolution limitations and can better utilize the rich geometric and structural information in CAD drawings. High-efficiency and high-precision understanding of geometric meaning: An end-to-end deep learning model that can learn and understand geometric meaning directly from engineering drawing vectors is established, which can accurately parse the geometric structure of the drawings with high efficiency, greatly improving the efficiency and accuracy of information extraction, and ensuring the integrity and accuracy of information. High scalability and generalization ability: The present invention adopts graph neural network and Transformer architecture, which can process complex geometric and topological information while naturally having high scalability and generalization ability, and can easily expand the model scale. Through training and fine-tuning, it can be applied to engineering drawings of different types, larger scales, and higher complexity. The introduction of graph neural network and self-attention mechanism achieves an accuracy rate that exceeds existing methods. In the implementation case, the accuracy rate reached 100% in the 2-classification and 3-classification tasks, and the accuracy rate was as high as 95.7% in the 6-classification task. Based on the vector learning mechanism, the input scale of data and the parameter requirements of the geometric feature learning model are greatly reduced, which is significantly lower than the grid diagram methods such as ResNet and ViT, and convergence can be achieved after dozens of training. At the same time, the computing efficiency is significantly improved. The vector learning network part of the model can achieve an inference speed of more than 1000FPS on a consumer-grade GPU. Strong scalability and high application potential. The vector learning model is connected to a dedicated large language model through a global feature layer and a custom-adjusted output layer to form a complete end-to-end vector perception architecture. It has strong scalability and can be used for traditional applications such as classification and clustering. It also has the technical potential to be used for various large models such as vector graphics to expand application scenarios.
[0173] The engineering drawing information extraction device provided by the present invention is described below. The engineering drawing information extraction device described below and the engineering drawing information extraction method described above can be referred to each other.
[0174] Figure 8 A schematic diagram of the structure of an engineering drawing information extraction device provided by an embodiment of the present invention, such as Figure 8 As shown, the engineering drawing information extraction device provided by the embodiment of the present invention includes:
[0175] Parsing module 801, used to obtain engineering drawing vector diagrams, and parse the engineering drawing vector diagrams to obtain geometric components in DXF format;
[0176] The encoding module 802 is used to embed the DXF format geometric component, input the encoding result into the vector learning network, and output the high-dimensional feature vector corresponding to the global geometric feature through the global feature induction layer of the vector learning network;
[0177] An output module 803 is used to input the high-dimensional feature vector corresponding to the global geometric feature into a large language model to obtain the type and parameter information of each component in the engineering drawing;
[0178] The vector learning network is trained by a hierarchical training strategy, which includes pre-training the vector learning network on a large scale based on a multi-domain training data set, and fine-tuning the vector learning network based on an engineering drawing domain training data set based on the pre-training.
[0179] The engineering drawing information extraction device provided by the embodiment of the present invention obtains the engineering drawing vector map, parses the engineering drawing vector map to obtain the DXF format geometric components; embeds the DXF format geometric components, inputs the encoding results into the vector learning network, and outputs the high-dimensional feature vector corresponding to the global geometric features through the global feature induction layer of the vector learning network; inputs the high-dimensional feature vector corresponding to the global geometric features into the large language model to obtain the type and parameter information of each component in the engineering drawing; wherein the vector learning network is trained by a hierarchical training strategy, the hierarchical training strategy includes pre-training the vector learning network on a scale based on a multi-domain training data set, and fine-tuning the vector learning network based on the engineering drawing field training data set on the basis of the pre-training, and directly parses the CAD vector information corresponding to the engineering drawing, which is efficient and lossless, can directly process the CAD format drawings through embedded coding, requires small disk storage space, and is completely unaffected by scaling, resolution, and image compression distortion, and can realize lossless analysis of engineering drawings; combines the vector learning mechanism and the large language model to improve the output accuracy, and uses the hierarchical training strategy to effectively capture and learn the characteristic laws of vector primitives, thereby improving the intelligence level of the entire engineering drawing information extraction.
[0180] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device may include: a processor (processor) 910, a communication interface (Communications Interface) 920, a memory (memory) 930 and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other through the communication bus 940. The memory 930 includes a computer program, an operating system, and acquired data. The processor 910 can call the logic instructions in the memory 930 to execute the engineering drawing information extraction method, which includes: collecting engineering drawing image data, converting the engineering drawing image data into a format, and acquiring DXF format geometric components; embedding and encoding the DXF format geometric components, inputting the encoding results into a vector learning network, and outputting high-dimensional feature vectors corresponding to global geometric features through the global feature induction layer of the vector learning network; inputting the high-dimensional feature vectors corresponding to the global geometric features into a large language model to obtain the types and parameter information of each component in the engineering drawing; wherein the vector learning network is trained by a hierarchical training strategy, and the hierarchical training strategy includes pre-training the vector learning network on a scale based on a multi-domain training data set, and on the basis of the pre-training on a scale, fine-tuning the vector learning network based on an engineering drawing domain training data set.
[0181] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0182] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the engineering drawing information extraction method provided by the above-mentioned methods, the method comprising: collecting engineering drawing image data, converting the format of the engineering drawing image data, and obtaining DXF format geometric components; embedding and encoding the DXF format geometric components, inputting the encoding results into a vector learning network, and outputting high-dimensional feature vectors corresponding to global geometric features through the global feature induction layer of the vector learning network; inputting the high-dimensional feature vectors corresponding to the global geometric features into a large language model to obtain the types and parameter information of each component in the engineering drawing; wherein the vector learning network is trained by a hierarchical training strategy, and the hierarchical training strategy comprises pre-training the vector learning network on a scale based on a multi-domain training data set, and on the basis of the pre-training on a scale, fine-tuning the vector learning network based on an engineering drawing domain training data set.
[0183] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0184] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.
[0185] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for extracting information from engineering drawings, characterized in that: include: Obtaining a vector diagram of an engineering drawing, parsing the vector diagram of the engineering drawing to obtain geometric components in a DXF format; Embedding the DXF format geometric component, inputting the encoding result into a vector learning network, and outputting a high-dimensional feature vector corresponding to the global geometric feature through the global feature induction layer of the vector learning network; embedding the DXF format geometric component includes: The entire geometric information of the engineering drawing vector diagram is described in the form of a two-dimensional matrix to obtain a DXF vector diagram of the engineering component. : ; in, For entities, For each entity, the command name corresponds to is a list of coordinates, is a list of parameters describing the geometry of the entity, is the visibility parameter, For layers, For other attributes; for In the case of text, the Word2vec tool is used to embed words to obtain embedding codes; for When it is not text, it is represented by multiple dimensions of embedding , , , , , ,get Multi-dimensional vector representation of engineering components DXF vector map Reference The order of embedding coding is generated to generate an embedding coding matrix; Inputting the high-dimensional feature vector corresponding to the global geometric feature into a large language model to obtain the type and parameter information of each component in the engineering drawing; The vector learning network is trained by a hierarchical training strategy, which includes pre-training the vector learning network on a large scale based on a multi-domain training data set, and fine-tuning the vector learning network based on an engineering drawing domain training data set based on the pre-training.
2. The method for extracting engineering drawing information according to claim 1, characterized in that: The step of parsing the engineering drawing vector diagram to obtain the DXF format geometric components includes: Collect DWG format images of engineering drawings; Segment the geometric elements of various components in the DWG format image to generate a set of geometric components in the DWG format; A format conversion process is established based on a drawing design software development kit, and the geometric component set in the DWG format is processed according to the format conversion process and converted into the geometric component set in the DXF format in batches.
3. The method for extracting engineering drawing information according to claim 1 or 2, characterized in that: After obtaining the DXF format geometric component, the method further includes: Performing data cleaning on the DXF format geometric components, and performing data normalization processing on the DXF format geometric components after data cleaning; The DXF format geometric components after the data normalization are classified according to the geometric type and layer, the non-standard geometric primitive commands are converted into a unified representation form, and the geometric data of different layers are integrated into a unified layer framework; Data enhancement is performed on the classified DXF format geometric components.
4. The method for extracting engineering drawing information according to claim 1, characterized in that: The vector learning network further includes: Feature extraction layer, used to extract geometric description features from the input sequence; Type extraction layer, used to extract command types from input sequences; The graph convolution layer is used to combine the geometric description features with the adjacency matrix of the graph to extract the geometric relationship features in the graph; The fully connected layer is used to integrate the geometric relationship features and command types in the graph to obtain a feature vector; The self-attention layer is used to globally calculate the similarity between each feature vector and other feature vectors through a multi-head self-attention mechanism, and dynamically adjust the importance of different hierarchical features; The output layer is used to output the calculation results of a specific task when the vector learning network is trained, and verify whether the training of the vector learning network is completed according to the calculation results.
5. The method for extracting engineering drawing information according to claim 4, characterized in that: The vector learning network uses the cross entropy loss function as the loss function during training: ; in: is the one-hot encoding of the true label, indicating whether category i is the true category of the training sample. is the probability that the model predicts category i, is the total number of categories.
6. The method for extracting engineering drawing information according to claim 1, characterized in that: Also includes: Optimizing the training process of the vector learning network using an optimization strategy, wherein the optimization strategy includes at least one of a sparse data optimization strategy, a training hyperparameter setting strategy, an overfitting optimization strategy, and a gradient clipping strategy; The sparse data optimization strategy adopts a sparse matrix data format to perform sparse matrix processing, sparse coding and sparse convolution processing on the input sequence; The training hyperparameter setting strategy includes setting an initial learning rate and gradually adjusting the initial learning rate through a cosine annealing learning rate scheduler; The overfitting optimization strategy includes introducing a Dropout mechanism in the fully connected layer and the self-attention layer to randomly discard the outputs of some neurons during the training process; The gradient clipping strategy includes setting the maximum gradient norm and introducing an early stopping strategy during the model training process. By monitoring the changes in model loss or model output accuracy, the training is terminated early when multiple output results do not show significant improvement.
7. A device for extracting information from engineering drawings, characterized in that: include: A parsing module, used for obtaining a vector diagram of an engineering drawing, and parsing the vector diagram of the engineering drawing to obtain geometric components in a DXF format; The encoding module is used to embed the DXF format geometric component, input the encoding result into the vector learning network, and output the high-dimensional feature vector corresponding to the global geometric feature through the global feature induction layer of the vector learning network; the embedding encoding of the DXF format geometric component includes: The entire geometric information of the engineering drawing vector diagram is described in the form of a two-dimensional matrix to obtain a DXF vector diagram of the engineering component. : ; in, For entities, For each entity, the command name corresponds to is a list of coordinates, is a list of parameters describing the geometry of the entity, is the visibility parameter, For layers, For other attributes; for In the case of text, the Word2vec tool is used to embed words to obtain embedding codes; for When it is not text, it is represented by multiple dimensions of embedding , , , , , ,get Multi-dimensional vector representation of engineering components DXF vector map Reference The order of embedding coding is generated to generate an embedding coding matrix; An output module, used for inputting the high-dimensional feature vector corresponding to the global geometric feature into a large language model to obtain the type and parameter information of each component in the engineering drawing; The vector learning network is trained by a hierarchical training strategy, which includes pre-training the vector learning network on a large scale based on a multi-domain training data set, and fine-tuning the vector learning network based on an engineering drawing domain training data set based on the pre-training.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for extracting engineering drawing information as described in any one of claims 1 to 6 is implemented.
9. A non-transitory readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for extracting engineering drawing information as described in any one of claims 1 to 6 is implemented.
