A method and system for semantic recognition of key symbols in industrial drawings

Abstract

The invention relates to the field of image recognition, and discloses a method and system for semantic recognition of key symbols in industrial drawings. The method comprises the following steps: designing a general drawing segmentation model DrawSAM based on a basic model SAM, using an efficient fine-tuning method DoRA to adaptively fine-tune SAM, and combining a Box Generator prompt generator to perform image segmentation, so as to adapt to various drawing types and complex structure layouts, and improve the detection and recognition accuracy and efficiency of text and symbols in drawings; and through an optimized text detection algorithm DrawDETR, reducing the misrecognition rate when recognizing engineering drawings, improving the recognition accuracy, effectively solving the misrecognition and symbol unrecognition problems of general OCR models in this field, significantly improving the retrieval and application of electronic drawing data, and meeting industrial needs.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method and system for semantic recognition of key symbols in industrial drawings. Background Art

[0002] In the industrial field, especially in engineering projects involving complex structures and precision manufacturing, industrial drawings are crucial tools. Drawings not only record in detail the design parameters, specification requirements, installation location and connection methods of the equipment, but also contain various material codes, component numbers, and specific technical symbols and annotations to guide on-site construction and subsequent maintenance.

[0003] At present, OCR models are commonly used for recognition, and their accuracy in conventional document recognition has reached a fairly high level. However, these general models have significant deficiencies in the recognition of text and symbols such as material codes and component numbers in engineering drawings. This is mainly due to the following technical defects: the image size of engineering drawings is usually large, while the text and symbols in them are relatively small. This size difference limits the recognition ability of OCR models when detecting small-sized text, which easily leads to inaccurate recognition results. Secondly, the close fit between text and symbol boxes also complicates the recognition process. The OCR model may mistakenly merge adjacent characters or symbols, causing misrecognition. In addition, engineering drawings usually contain multiple devices, and the symbols of some devices are easily misrecognized as text, which increases the difficulty of model processing; at the same time, the distribution direction of symbols on drawings is often very random, which affects the comprehensiveness and accuracy of recognition. On the other hand, the advancement of target detection technology has made the automatic positioning of drawing symbols more efficient. Many existing studies use deep learning algorithms such as convolutional neural networks (CNNs) to accurately locate and classify symbols in drawings, but these methods usually require a large amount of labeled data for training.

[0004] Therefore, a method is needed that can improve the recognition accuracy of industrial drawing text and symbols while achieving more efficient information extraction and utilization. Summary of the invention

[0005] In view of the above existing problems, the present invention is proposed.

[0006] Therefore, the present invention provides a method and system for semantic recognition of key symbols in industrial drawings to solve the current problems of recognition, missed recognition and misrecognition of small-sized text in industrial drawings, and insufficient efficiency and accuracy in information extraction.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] In a first aspect, the present invention provides a method for semantic recognition of key symbols in industrial drawings, comprising: obtaining an industrial drawing;

[0009] Perform image segmentation on the industrial drawings, encode them using an image encoder with a fine-tuning method to obtain an image embedding vector, generate bounding box prompt information according to the device image in the drawings, obtain prompt coding based on the bounding box prompt information, and jointly input the image embedding vector and the prompt coding to generate a final device mask map to obtain a final device map;

[0010] Perform text detection on each device image obtained by segmentation through a text detection algorithm, including extracting features from the device image and generating several candidate detection frame information. A reference point is obtained through the center point and proportion information of the candidate detection frame. A composite query is formed based on the reference point to generate and optimize the coordinates of the text detection frame to locate the text area in the device image.

[0011] The text area is subjected to text recognition by a text recognition algorithm to obtain semantic information of characters and symbols.

[0012] As a preferred solution of the method for semantic recognition of key symbols in industrial drawings described in the present invention, the method comprises: encoding using an image encoder with a fine-tuning method to obtain an image embedding vector, including:

[0013] Introducing the fine-tuning method DoRA in each Transformer block of SAM’s image encoder;

[0014] Through DoRA, the weight parameters of the image encoder are decomposed into two parts: amplitude and direction for fine-tuning;

[0015] The low-rank structure LoRA is used to update the parameters of the direction part, and the parameters of the amplitude part do not participate in the gradient update to obtain the image embedding vector.

[0016] As a preferred solution of the method for semantic recognition of key symbols in industrial drawings of the present invention, the method of generating boundary box prompt information according to the device image in the drawing includes:

[0017] ResNet-50 is used as the backbone network. Combined with the multi-level feature fusion strategy, the multi-layer convolution of the backbone network is fused through upsampling and splicing to obtain multi-layer fusion features.

[0018] The multi-layer fusion features are used to generate first bounding box prompt information through two first convolutional layers with different void rates and one second convolutional layer;

[0019] Set the loss-box to perform continuous iterative training so that the first bounding box prompt information is closer to the true value, and update the second bounding box prompt information.

[0020] As a preferred solution of the method for semantic recognition of key symbols of industrial drawings described in the present invention, text detection is performed on each device diagram obtained by segmentation through a text detection algorithm, including feature extraction of the device diagram and generation of information of a number of candidate detection frames, and reference points are obtained through the center point and proportion information of the candidate detection frames, specifically including:

[0021] The extracted device image is input into the ResNet-50 backbone network and Transformer encoder for feature extraction;

[0022] Generate several candidate detection box information in the final encoding layer based on the extracted features;

[0023] Through the center point and scale information of the candidate detection box, a preset number of reference point coordinates are explicitly sampled at the top and bottom using the Bezier center curve, which are used as reference points for the deformable cross attention module.

[0024] As a preferred solution of the method for semantic recognition of key symbols in industrial drawings of the present invention, a composite query is formed based on reference points, including:

[0025] In the decoder, the coordinates of the reference points generated by the Bezier center curve are encoded for object query;

[0026] The object query is added to the corresponding reference point content query to form a composite query, which is fed back to the deformable criss-cross attention module.

[0027] As a preferred solution of the method for semantic recognition of key symbols in industrial drawings described in the present invention, the coordinates of the text detection frame are generated and optimized, including:

[0028] After the composite query is processed by the deformable cross attention module, the coordinates of the text detection box are generated using the prediction head;

[0029] A detection box loss function is set, and in each iterative training, the loss function is minimized to extract the coordinates of each text in each device image, thereby locating the text area in the device image.

[0030] As a preferred solution of the method for semantic recognition of key symbols in industrial drawings of the present invention, the text area is subjected to text recognition by a text recognition algorithm to obtain semantic information of text and symbols, including:

[0031] The text area is divided into several small text blocks through the SVTR model and represented as character components;

[0032] Performing feature mixing, merging and concatenation operations on the character components in stages to perceive feature patterns within and between characters;

[0033] Based on the feature pattern, character recognition is performed through linear prediction.

[0034] In a second aspect, the present invention provides a system for recognizing the semantics of key symbols in industrial drawings, comprising:

[0035] Acquisition module, used to acquire industrial drawings;

[0036] A segmentation module is used to perform image segmentation on the industrial drawing, encode the image using an image encoder with a fine-tuning method to obtain an image embedding vector, generate bounding box prompt information according to the device image in the drawing, obtain a prompt code based on the bounding box prompt information, and jointly input the image embedding vector and the prompt code to generate a final device mask map to obtain a final device map;

[0037] The text detection module is used to perform text detection on each device image obtained by segmentation through a text detection algorithm, including feature extraction of the device image and generation of information of several candidate detection frames, obtaining reference points through the center points and proportion information of the candidate detection frames; forming a composite query based on the reference points to generate and optimize the coordinates of the text detection frame to locate the text area in the device image;

[0038] The text recognition module is used to perform text recognition on the text area through a text recognition algorithm to obtain semantic information of characters and symbols.

[0039] In a third aspect, the present invention provides an electronic device, comprising:

[0040] Memory and processor;

[0041] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the method for semantic recognition of key symbols of industrial drawings are implemented.

[0042] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the method for semantic recognition of key symbols in industrial drawings.

[0043] Compared with the prior art, the present invention has the following beneficial effects: the present invention designs a general drawing segmentation model DrawSAM based on the basic model SAM, uses the efficient fine-tuning method DoRA to adaptively fine-tune SAM, and combines the Box Generator prompt generator for image segmentation. It can adapt to various drawing types and complex structure layouts, and improve the detection and recognition accuracy and efficiency of text and symbols in drawings; and through the optimized text detection algorithm DrawDETR, it reduces the misrecognition rate when identifying engineering drawings, improves the recognition accuracy, effectively solves the problems of misrecognition and unrecognizable symbols of general OCR models in this field, significantly improves the retrieval and application of electronic drawing data, and meets industrial needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0045] Figure 1 A schematic diagram of the overall process of a method for semantic recognition of key symbols in industrial drawings according to an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of a single device in an industrial drawing in a method for semantic recognition of key symbols in industrial drawings according to an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of an image segmentation process in a method for semantic recognition of key symbols in industrial drawings according to an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of a module of DrawSAM in a method for semantic recognition of key symbols in industrial drawings according to an embodiment of the present invention;

[0049] Figure 5 A schematic diagram of a module of BoxGenerator in a method for semantic recognition of key symbols in industrial drawings according to an embodiment of the present invention;

[0050] Figure 6 The present invention is a schematic diagram of a module of DrawDETR in a method for semantic recognition of key symbols in industrial drawings according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0052] Reference Figure 1-Figure 6 , as an embodiment of the present invention, provides a method for semantic recognition of key symbols in industrial drawings, comprising:

[0053] S100: Obtain industrial drawings;

[0054] S200: performing image segmentation on the industrial drawing, encoding the image using an image encoder with a fine-tuning method to obtain an image embedding vector, generating bounding box prompt information according to the device image in the drawing, obtaining a prompt code based on the bounding box prompt information, and jointly inputting the image embedding vector and the prompt code to generate a final device mask map to obtain a final device map;

[0055] S300: Performing text detection on each device image obtained by segmentation through a text detection algorithm, including extracting features of the device image and generating a number of candidate detection frame information, obtaining a reference point through the center point and proportion information of the candidate detection frame; forming a composite query based on the reference point to generate and optimize the coordinates of the text detection frame to locate the text area in the device image;

[0056] S400: Performing text recognition on the text area by using a text recognition algorithm to obtain semantic information of characters and symbols.

[0057] It should be noted that in the recognition of text and symbols such as material codes and component numbers in engineering drawings, general OCR models often face the problem of misrecognition and missed recognition when processing engineering drawings, resulting in unsatisfactory recognition accuracy. Figure 2 In the case shown, some device symbols may be confused with the surrounding text, and the text instances are highly skewed, resulting in recognition errors.

[0058] like Figure 3 As shown, the industrial drawing on the left is segmented by the image segmentation algorithm in the above step S200 to obtain an intermediate mask image, so that each device in the drawing is extracted separately to obtain a single device image below; for each segmented device, the text detection algorithm designed in step S300 is used to locate the text and symbol area inside it; finally, the text recognition algorithm is used to identify the content of the located area, which can solve the problems of recognition, missed recognition and misrecognition of small-sized text in industrial drawings.

[0059] In the implementation manner of the present application, the industrial drawing obtained in step S100 may be in the format of pdf, png or jpg, etc.

[0060] In the implementation manner of the present application, in step S200, encoding is performed using an image encoder with a fine-tuning method to obtain an image embedding vector, including the following steps A1-A3:

[0061] A1: Introducing the fine-tuning method DoRA in each Transformer block of SAM’s image encoder;

[0062] A2: Decompose the weight parameters of the image encoder into two parts, amplitude and direction, through DoRA for fine-tuning;

[0063] A3: Use the low-rank structure LoRA to update the parameters of the direction part, and the parameters of the amplitude part do not participate in the gradient update to obtain the image embedding vector.

[0064] It should be noted that the above is a general drawing segmentation model DrawSAM optimized based on the basic model SAM. Among them, SAM is a model based on visual transformer, which consists of three parts: an image encoder, a flexible prompt encoder and a lightweight mask decoder. The image encoder uses ViT pre-trained by MAE to process high-resolution input images; the prompt encoder requires two sets of prompts: sparse prompts (points, boxes, text) and dense prompts (masks). The prompt encoder decodes points and boxes through a combination of position embedding and learnable embedding. Text and dense prompts are represented by the text encoder of CLIP respectively; the mask decoder combines the information extracted by the image and prompt encoders to obtain the predicted mask.

[0065] The present invention is based on the SAM optimized segmentation model DrawSAM, such as Figure 4As shown in the figure, it uses the efficient fine-tuning method DoRA to adaptively fine-tune SAM and designs a border hint generator. For a given drawing image, DrawSAM uses the image encoder inserted into DoRA to encode it into an image embedding vector. At the same time, the hint generator generates a bounding box hint based on the drawing image. Finally, the bounding box hint information is input into the hint encoder to obtain the hint encoding, and the image embedding vector and the hint encoding are jointly input into the mask decoder to generate the segmentation result. The entire segmentation process is end-to-end and does not require any additional manual hints. The fine-tuning method DoRA reduces the computational cost and improves the adaptability of the model by decomposing the weight parameters into two parts, the amplitude and the direction, and only fine-tuning the direction part. At the same time, since the parameters of the image encoder are fixed, and the low-rank structure parameters inserted into the direction part of the image encoder are trainable, the parameters for training are relatively small, and high GPU memory consumption is not required. Moreover, since the existing drawing data set is limited, fewer parameters can effectively avoid overfitting.

[0066] In another optional implementation, the fine-tuning method may also be full fine-tuning or layer-wise freezing fine-tuning, which can also effectively improve the performance of the model in processing industrial drawings.

[0067] In the implementation manner of the present application, generating boundary box prompt information according to the device image in the drawing in step S200 includes the following steps B1-B3:

[0068] B1: ResNet-50 is used as the backbone network, combined with a multi-level feature fusion strategy, and the multi-layer convolution of the backbone network is upsampled and concatenated to obtain multi-layer fusion features;

[0069] B2: Through two first convolutional layers with different void rates and one second convolutional layer, the multi-layer fusion features are generated into the first bounding box prompt information;

[0070] Specifically, two 3 x 3 convolutional layers with different void rates and one 1 x 1 convolutional layer can be used to generate a rough first bounding box prompt information (X, Y, W, H) by fusing multiple layers of features.

[0071] B3: Set the loss-box to perform continuous iterative training so that the first bounding box prompt information is closer to the true value, and update the second bounding box prompt information.

[0072] It should be noted that the Box Generator in DrawSAM can generate box prompts that are very similar to manual prompts. Figure 5As shown, compared with SAM which requires precise manual guidance, it is more efficient and improves the user experience.

[0073] In another optional embodiment, the generation of the bounding box prompt information in step S200 can also be generated by a Faster R-CNN model, that is, by using RPN (Region Proposal Network) to generate a candidate box, and extracting features corresponding to the candidate box from the feature map through ROI Pooling, and then further optimizing the position and category of the bounding box through a classification and regression network.

[0074] In another optional implementation, the generation of bounding box prompt information in step S200 can also be performed by an SSD model, which is also a single-stage detector, which simultaneously predicts bounding boxes on feature maps of multiple scales, and uses default boxes and offsets to predict the final bounding box position. A model that can better adapt to these characteristics can be selected according to the actual application requirements of industrial drawings, such as real-time requirements, model size restrictions, dense lines, complex graphics, accuracy requirements, etc.

[0075] In the implementation mode of this application, Figure 6 As shown, in step S300, text detection is performed on each device image obtained by segmentation through the text detection algorithm DrawDETR, including feature extraction of the device image and generation of a number of candidate detection frame information, and a reference point is obtained through the center point and proportion information of the candidate detection frame, which specifically includes the following steps C1-C3:

[0076] C1: Extract the device image and input it into the ResNet-50 backbone network and Transformer encoder for feature extraction;

[0077] C2: Generate several candidate detection box information in the final encoding layer based on the extracted features;

[0078] C3: Through the center point and scale information of the candidate detection box, a preset number of reference point coordinates are explicitly sampled at the top and bottom using the Bezier center curve, which are used as reference points for the deformable cross attention module.

[0079] It should be noted that the reference points are mainly used to more flexibly adapt to text regions of different shapes and features.

[0080] In the implementation mode of this application, Figure 6 As shown, in step S300, the text detection algorithm DrawDETR forms a compound query based on the reference point, including the following steps D1-D2:

[0081] D1: In the decoder, the coordinates of the reference points generated by the Bezier center curve are encoded for object query;

[0082] D2: Add the object query to the corresponding reference point content query to form a composite query, which is fed back to the deformable criss-cross attention module.

[0083] It should be noted that compound queries are intended to enhance the understanding and association of text features.

[0084] In the implementation mode of this application, Figure 6 As shown, in step S300, the text detection algorithm DrawDETR generates and optimizes the coordinates of the text detection box, including the following steps E1-E2:

[0085] E1: After the composite query is processed by the deformable cross attention module, the coordinates of the text detection box are generated using the prediction head;

[0086] E2: Set the detection box loss function and minimize the loss function in each iterative training to extract the coordinates of each text in each device image, thereby locating the text area in the device image.

[0087] It should be noted that in order to continuously optimize the detection results, a detection box loss is set, and this loss function is minimized in each iterative training. This process can ensure that the coordinates of the generated text detection box gradually approach the real text box, thereby improving the accuracy and robustness of the model in text detection tasks.

[0088] In the implementation manner of the present application, in step S400, text recognition is performed on the text area by using a text recognition algorithm to obtain semantic information of characters and symbols, including the following steps F1-F3:

[0089] F1: The text area is divided into several small text blocks through the SVTR model and represented as character components PatchEmbedding;

[0090] F2: Perform feature mixing, merging and concatenation operations on character components in stages;

[0091] It should be noted that by designing the global and local hybrid module in this way, the module can perceive the characteristic patterns within and between characters, forming a multi-granularity character component perception component to perceive the characteristic patterns within and between characters;

[0092] F3: Character recognition through linear prediction based on feature patterns.

[0093] In another optional embodiment, the text recognition algorithm in step S400 can also be a CRNN algorithm, which combines a convolutional neural network (CNN) and a recurrent neural network (RNN). CNN is first used to extract the features of the text line, and then these feature sequences are modeled through RNN. Finally, the CTC (Connectionist Temporal Classification) loss function is used for training and decoding, which is suitable for the recognition of long text lines.

[0094] In another optional implementation, the text recognition algorithm in step S400 may also be an EAST model, which is particularly suitable for detecting text regions of arbitrary shapes through end-to-end text detection and recognition, and can directly output geometric information and recognition results of the text region.

[0095] In summary, the advantage of the present invention lies in the accurate semantic recognition of small-sized text and symbols in engineering drawings, which effectively solves the problem of misrecognition and unrecognizable symbols of general OCR models in this field. Through the optimized algorithm and model, the misrecognition rate in identifying engineering drawings is reduced and the recognition accuracy is improved; in addition, the existing technology is limited by fixed rules and feature extraction methods, while the solution of the present invention can adapt to a variety of drawing types and complex structural layouts, improve recognition accuracy and efficiency, significantly improve the retrieval and application of electronic drawing data, improve the work efficiency of engineers in the design and analysis process, and meet industrial needs.

[0096] The above is a schematic scheme of a method for semantic recognition of key symbols in industrial drawings of this embodiment. It should be noted that the technical scheme of the system for semantic recognition of key symbols in industrial drawings and the technical scheme of the method for semantic recognition of key symbols in industrial drawings described above belong to the same concept. For details not described in detail in the technical scheme of the system for semantic recognition of key symbols in industrial drawings of this embodiment, please refer to the description of the technical scheme of the method for semantic recognition of key symbols in industrial drawings described above.

[0097] This embodiment also provides a system for a method for semantic recognition of key symbols in industrial drawings, including:

[0098] Acquisition module, used to acquire industrial drawings;

[0099] A segmentation module is used to perform image segmentation on the industrial drawing, encode the image using an image encoder with a fine-tuning method to obtain an image embedding vector, generate bounding box prompt information according to the device image in the drawing, obtain a prompt code based on the bounding box prompt information, and jointly input the image embedding vector and the prompt code to generate a final device mask map to obtain a final device map;

[0100] The text detection module is used to perform text detection on each device image obtained by segmentation through a text detection algorithm, including feature extraction of the device image and generation of information of several candidate detection frames, obtaining reference points through the center points and proportion information of the candidate detection frames; forming a composite query based on the reference points to generate and optimize the coordinates of the text detection frame to locate the text area in the device image;

[0101] The text recognition module is used to perform text recognition on the text area through a text recognition algorithm to obtain the semantic information of words and symbols.

[0102] This embodiment also provides a computing device suitable for the semantic recognition of key symbols in industrial drawings, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the method for semantic recognition of key symbols in industrial drawings as proposed in the above embodiment.

[0103] This embodiment further provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for realizing the semantic recognition of key symbols of industrial drawings as proposed in the above embodiment is implemented.

[0104] The storage medium proposed in this embodiment and the method for realizing semantic recognition of key symbols in industrial drawings proposed in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0105] Through the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware. Based on such an understanding, the technical solution of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer's floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk or an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention.

[0106] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for semantic recognition of key symbols in industrial drawings, characterized in that: include: Obtain industrial drawings; Perform image segmentation on the industrial drawings, encode them using an image encoder with a fine-tuning method to obtain an image embedding vector, generate bounding box prompt information according to the device image in the drawings, obtain prompt coding based on the bounding box prompt information, and jointly input the image embedding vector and the prompt coding to generate a final device mask map to obtain a final device map; The text detection algorithm is used to perform text detection on each device image obtained by segmentation, including feature extraction of the device image and generation of several candidate detection frame information. The reference point is obtained through the center point and proportion information of the candidate detection frame, including: The extracted device image is input into the ResNet-50 backbone network and Transformer encoder for feature extraction; Generate several candidate detection box information in the final encoding layer based on the extracted features; Through the center point and scale information of the candidate detection box, the number and coordinates of reference points are preset by using explicit sampling of Bezier center curves at the top and bottom, which are used as reference points for the deformable cross attention module; Forming a composite query based on the reference points to generate and optimize the coordinates of the text detection box to locate the text area in the device diagram; Wherein, a compound query is formed based on the reference point, including: In the decoder, the coordinates of the reference points generated by the Bezier center curve are encoded for object query; The object query is added to the corresponding reference point content query to form a composite query, which is fed back to the deformable criss-cross attention module; The text area is subjected to text recognition by using a text recognition algorithm to obtain semantic information of characters and symbols.

2. The method for semantic recognition of key symbols in industrial drawings according to claim 1, characterized in that: The image is encoded using an image encoder with a fine-tuning method to obtain an image embedding vector, including: Introducing the fine-tuning method DoRA in each Transformer block of SAM's image encoder; Through DoRA, the weight parameters of the image encoder are decomposed into two parts: amplitude and direction for fine-tuning; The low-rank structure LoRA is used to update the parameters of the direction part, and the parameters of the amplitude part do not participate in the gradient update to obtain the image embedding vector.

3. The method for semantic recognition of key symbols in industrial drawings according to claim 2, characterized in that: Generate bounding box prompt information based on the device image in the drawing, including: ResNet-50 is used as the backbone network. Combined with the multi-level feature fusion strategy, the multi-layer convolution of the backbone network is fused through upsampling and splicing to obtain multi-layer fusion features. The multi-layer fusion features are used to generate first bounding box prompt information through two first convolutional layers with different void rates and one second convolutional layer; Set the loss-box to perform continuous iterative training so that the first bounding box prompt information is closer to the true value, and update the second bounding box prompt information.

4. The method for semantic recognition of key symbols in industrial drawings according to claim 3, characterized in that: Generate the coordinates of the text detection box and optimize it, including: After the composite query is processed by the deformable cross attention module, the coordinates of the text detection box are generated using the prediction head; A detection box loss function is set, and in each iterative training, the loss function is minimized to extract the coordinates of each text in each device image, thereby locating the text area in the device image.

5. The method for semantic recognition of key symbols in industrial drawings according to claim 4, characterized in that: Performing text recognition on the text area by using a text recognition algorithm to obtain semantic information of characters and symbols, including: The text area is divided into several small text blocks through the SVTR model and represented as character components; Performing feature mixing, merging and concatenation operations on the character components in stages to perceive feature patterns within and between characters; Based on the feature pattern, character recognition is performed through linear prediction.

6. A system for the method for semantic recognition of key symbols in industrial drawings according to any one of claims 1 to 5, characterized in that: include: Acquisition module, used to acquire industrial drawings; A segmentation module is used to perform image segmentation on the industrial drawing, encode the image using an image encoder with a fine-tuning method to obtain an image embedding vector, generate bounding box prompt information according to the device image in the drawing, obtain a prompt code based on the bounding box prompt information, and jointly input the image embedding vector and the prompt code to generate a final device mask map to obtain a final device map; The text detection module is used to perform text detection on each device image obtained by segmentation through a text detection algorithm, including feature extraction of the device image and generation of information of several candidate detection frames, obtaining reference points through the center points and proportion information of the candidate detection frames; forming a composite query based on the reference points to generate and optimize the coordinates of the text detection frame to locate the text area in the device image; The text recognition module is used to perform text recognition on the text area through a text recognition algorithm to obtain semantic information of characters and symbols.

7. An electronic device comprising: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the method for semantic recognition of key symbols in industrial drawings described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the method for semantic recognition of key symbols in industrial drawings as described in any one of claims 1 to 5.

Images