A recognition method for two-dimensional drawings of substations

Abstract

The present invention relates to a method for recognizing two-dimensional drawings of a substation, which belongs to the field of image recognition technology and solves the problems of low accuracy and missing content in existing two-dimensional drawings. It includes segmenting the image of the substation two-dimensional drawings after preprocessing to obtain each sub-image, obtaining each image to be detected and its corresponding auxiliary image based on each sub-image; transferring each image to be detected and its corresponding auxiliary image to a component detection model in sequence, and fusing them to obtain a component frame and a component category; intercepting a component frame image according to the component frame and the component category, and identifying a first text box and a first text content in the component frame image; covering the first text box and the component frame respectively to obtain two text detection images, and transferring them to a text box detection model to obtain a second text box outside the component frame; intercepting the second text box image and transferring it to a text recognition model to recognize the second text content; associating the first text box and the second text box with the component frame to obtain a recognition result. Efficient and accurate drawing recognition is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method for recognizing a two-dimensional drawing of a substation. Background Art

[0002] For a long time, both mechanical and electrical information has been recorded in two-dimensional drawings. With the increase in the scale and complexity of two-dimensional drawings and the iterative update of topological relationships, there is an urgent need to realize automatic and intelligent recognition of two-dimensional drawings.

[0003] With the development of artificial intelligence technology, target detection based on deep learning has become a commonly used method. However, the electrical components in the two-dimensional drawings of substations are small in size, large in number, and densely distributed. The general field target detection model for image processing is difficult to accurately and efficiently identify the components in the drawings.

[0004] Moreover, existing methods are prone to miss the text in two-dimensional drawings, resulting in missing recognition results. Summary of the invention

[0005] In view of the above analysis, an embodiment of the present invention aims to provide a method for recognizing a two-dimensional drawing of a substation, so as to solve the problems of low recognition accuracy and missing content of existing two-dimensional drawings.

[0006] The embodiment of the present invention provides a method for identifying a two-dimensional drawing of a substation, comprising the following steps:

[0007] After preprocessing the image of the substation two-dimensional drawing, each wiring diagram in the image is segmented to obtain each sub-image; each image to be detected is obtained by scaling each sub-image to a preset size, and each sub-image is rotated and scaled to a preset size to obtain an auxiliary image corresponding to the image to be detected;

[0008] Each image to be inspected and its corresponding auxiliary image are sequentially input into the component inspection model, and the two inspection results output are fused to obtain the component frame and component category in each image to be inspected;

[0009] According to the component frame and component category, the component frame image is intercepted, and the first text frame and the first text content in the component frame image are identified; the first text frame and the component frame are covered respectively to obtain two text detection images for each image to be detected, and the images are simultaneously transmitted to the text frame detection model to obtain the second text frame outside the component frame; the second text frame image is intercepted from each image to be detected and transmitted to the text recognition model to identify the second text content;

[0010] The first text box and the second text box are associated with the component box, and the text content corresponding to each component in the two-dimensional drawing of the substation is obtained as the recognition result.

[0011] Based on further improvement of the above method, the recognition method also includes: when there is no component box in the detection result output by the component detection model, marking the input image to be detected as an information image; identifying the third text box and the third text content in the information image; parsing the third text content to obtain the label, component category, name, model and quantity as standard data; and verifying the recognition result based on the standard data to obtain a verification result.

[0012] Based on the further improvement of the above method, the component detection model is obtained by training the improved YOLOv5 model; the improved YOLOv5 model includes input layer, backbone layer, channel attention layer, bottleneck layer and prediction layer in sequence; the input layer is single-channel input, the backbone layer is serially constructed by Focus module, three groups of CBL+CSP modules, and CBL+spatial pyramid pooling module, the feature maps extracted by the second group of CBL+CSP modules, the third group of CBL+CSP modules and the spatial pyramid pooling module in the backbone layer are respectively passed to the channel attention layer, the feature maps of three scales extracted by the backbone layer are compressed and stimulated respectively and then passed to the bottleneck layer; the bottleneck layer is composed of feature pyramid network and path aggregation network, and the feature maps of different scales output by the channel attention mechanism are fused; the prediction layer predicts component frames of different scales according to the feature maps of different scales output by the bottleneck layer.

[0013] Based on the further improvement of the above method, the training samples of the component detection model include original samples constructed according to the historical wiring diagram and auxiliary samples obtained by rotating the historical wiring diagram; the loss function of the component detection model is obtained by calculating the first classification loss, confidence loss, positioning loss and inter-class difference loss and weighted summing them; the first classification loss, confidence loss and positioning loss are respectively obtained by calculating the losses corresponding to each scale, and then weighted summing them according to the weight factors of the prediction boxes of each scale; the inter-class difference loss is obtained by calculating the difference between the prediction boxes of the original sample and the corresponding auxiliary sample.

[0014] Based on the further improvement of the above method, the inter-class difference loss is calculated by the following formula:

[0015] ,

[0016] in, Indicates the number of prediction boxes, , and Represents the original sample The center point coordinates, size width and size height of the prediction box, and Indicates auxiliary sample The width and height of the prediction box, Indicates auxiliary sample The center point coordinates of the predicted box The value obtained after the inverse rotation transformation.

[0017] Based on the further improvement of the above method, the two detection results output by fusion are used to obtain the component frame and component category in each image to be detected, including: according to the transformation matrix between the auxiliary image and the image to be detected, the coordinates of each component frame in the second detection result corresponding to the auxiliary image are transformed to obtain the third detection result; with the error between the component frames being minimized and less than the boundary error threshold as the matching condition, the component frame in the third detection result and the component frame in the first detection result corresponding to the image to be detected are matched, and the final component frame is obtained by calculating the intersection and union ratio for each pair of matched component frames, and the component category with the largest probability is taken as the final component category.

[0018] Based on the further improvement of the above method, the text box detection model is obtained by training the improved Faster R-CNN model; the improved Faster R-CNN model receives two images at the same time, and uses two ResNet networks as dual-branch feature extraction networks to extract features from the two input images respectively to obtain dual-branch feature maps, and adds a feature pyramid network to fuse the dual-branch feature maps before passing them to the region proposal network and the region recognition network; the region proposal network predicts the candidate region based on the fused feature map and passes it to the region recognition network, the region recognition network first uses the ROI pooling layer to extract the feature sub-map of the candidate region from the candidate region and the fused feature map, and then uses multiple fully connected layers to perform category prediction and bounding box regression on the candidate region based on the feature sub-map to obtain the coordinates of the text box.

[0019] Based on the further improvement of the above method, the loss function of the text box detection model is obtained by calculating the second classification loss, boundary regression loss and dual-branch feature map difference and weighted summing them up, where the dual-branch feature map difference is obtained by calculating the cosine distance of the dual-branch feature map.

[0020] Based on the further improvement of the above method, the text recognition model is trained by introducing a multi-head self-attention mechanism into the CNN-LSTM combination model; the loss function of the text recognition model is The attention weight regularization loss is calculated by the following formula and character sequence loss And weighted summation gives:

[0021] ,

[0022] in, Indicates the maximum length of the output text sequence, represents the regularization coefficient, Indicates The attention weight of the head, represents the total number of attention heads, Indicates taking the L2 norm; Indicates Characters predicted as character set Middle The probability of a character, Indicates the real sequence One-hot encoding of character labels, Represents the weight of the character sequence loss.

[0023] Based on the further improvement of the above method, the second text content includes a label and a terminal number; the first text box and the second text box are associated with the component box to obtain the text content corresponding to each component in the two-dimensional drawing of the substation, including: the first text box is associated with the component box where it is located, and the first text content is used as the terminal number of the corresponding component; the distance between the component box and the center point coordinates of each second text box is calculated, and the second text box with a distance less than a distance threshold is associated with the corresponding component box. If there is only one associated second text box, the second text content in the second text box is the label of the component in the current component box; otherwise, the second text content in multiple second text boxes is identified. When the second text content is pure numbers, it is the terminal number corresponding to the component in the current component box; when the second text content contains characters, it is the label corresponding to the component in the current component box.

[0024] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0025] 1. Considering that the placement direction of components in two-dimensional drawings will affect the recognition effect, the auxiliary image is obtained by rotating the image to be detected, and the accuracy of component recognition is improved by fusing the detection results of the two images; considering the existence of terminal numbers inside the components, different methods are used to identify the text inside and outside the components to ensure the integrity of the text content; considering that some components are close to the text, the image for text content recognition is constructed by covering the component frame and the text inside the component frame respectively, and recognition is performed simultaneously, and the accuracy of text recognition is improved by the complementary fusion of the two image features.

[0026] 2. Fully explore the various parts of information in the two-dimensional drawings, identify the design drawing information and wiring diagram component information in the two-dimensional drawings, on the one hand improve the recognition results, on the other hand extract standard data from them to verify the recognition results, judge whether the recognition results are accurate, or whether there are design problems in the wiring diagram itself, thereby improving data quality and work efficiency.

[0027] 3. In the loss function of the component detection model, the difference between the detection results of the two images is increased to reduce the impact of different placement directions of the components. By introducing the channel attention mechanism in the component detection model, the model can better focus on important information and improve the accuracy of the model. Different weight factors are assigned to prediction boxes of different scales according to the contour area range of each wiring diagram, which enhances the robustness of the model and improves the detection performance of targets of different scales.

[0028] 4. The text box detection model supports the simultaneous input of two images, and uses the feature pyramid network to fuse the feature maps of the two images, so that the model can use complementary feature information to improve the richness of feature expression and reduce the difference between the two feature maps in the loss function, prompting the model to learn more consistent feature representations, thereby improving the stability and generalization ability of the model.

[0029] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;

[0031] Figure 1 A flow chart of a method for identifying a two-dimensional drawing of a substation in an embodiment of the present invention;

[0032] Figure 2 It is a wiring diagram included in the two-dimensional drawing of the substation in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0034] A specific embodiment of the present invention discloses a method for recognizing a two-dimensional drawing of a substation, such as Figure 1 As shown, the following steps are included:

[0035] S1. After preprocessing the image of the two-dimensional drawing of the substation, each wiring diagram in the image is segmented to obtain each sub-image; each image to be detected is obtained by scaling each sub-image to a preset size, and each sub-image is rotated and scaled to a preset size to obtain an auxiliary image corresponding to the image to be detected.

[0036] It should be noted that the received substation 2D drawings are first converted into JPG format images. Considering that there are some small noise points in the converted original images and there are some blurred areas around the outlines of some components in the drawings, in order to improve the visual effect and clarity of the images, the preprocessing of the original images includes: using a 3×3 Gaussian convolution kernel for Gaussian filtering calculation; graying and binarizing the smoothed original images to enhance the line outline and make it easier for subsequent segmentation and recognition.

[0037] Furthermore, it is considered that a substation drawing usually contains multiple independent wiring diagrams. For example, the secondary circuit drawing of a 10KV switch cabinet includes wiring diagrams such as control circuit, energy storage circuit, AC current circuit, AC power circuit, heater circuit, lighting circuit, and protection current circuit. This embodiment segments each wiring diagram in the image to obtain each sub-image, including: using an edge detection algorithm to obtain the edge of each wiring diagram. For example, the Canny edge detection algorithm is used, and then the contour tracking algorithm of OpenCV is used to perform connectivity analysis on the edge of each wiring diagram to obtain the contour of each wiring diagram; the respective rectangular bounding boxes are calculated in turn according to the contour of each wiring diagram; and each sub-image is obtained by cutting out according to each rectangular bounding box in the preprocessed image.

[0038] Each sub-image is scaled to a preset size to obtain an image to be detected, which is to fix the height-to-width ratio of each sub-image unchanged, calculate the height and width scaling ratios according to the preset size, and take the minimum scaling ratio as the ratio to be processed. If the ratio to be processed is greater than 1 and less than the maximum enlargement threshold, the sub-image is enlarged according to the ratio to be processed; if the ratio to be processed is greater than the maximum enlargement threshold, the sub-image is enlarged according to the maximum enlargement threshold; if the ratio to be processed is less than 1, the sub-image is reduced according to the ratio to be processed; if the scaled size is inconsistent with the preset size, the blank area is filled with white pixels to obtain the image to be detected.

[0039] Exemplarily, the maximum magnification threshold is 3, and the processing ratio of a relatively small wiring diagram is magnified 4 times, which exceeds the maximum magnification threshold. In order to avoid image blurring caused by excessive magnification, after magnification according to the maximum magnification threshold, the remaining part of the preset size is filled with white pixels to obtain the corresponding image to be detected.

[0040] At the same time, an auxiliary image corresponding to the image to be detected is obtained by rotating and scaling each sub-image to a preset size. Each sub-image is rotated 90 degrees clockwise and then scaled to a preset size according to the above method to obtain the auxiliary image. The image to be detected and the auxiliary image obtained based on the same sub-image correspond one to one.

[0041] Calculate and save the transformation matrix that converts the pixel coordinates in each auxiliary image to each image to be detected.

[0042] S2. Pass each pair of the to-be-detected image and the auxiliary image into the component detection model in sequence, and fuse the two output detection results to obtain the component frame and component category in each to-be-detected image.

[0043] It should be noted that the component detection model is obtained by training the improved YOLOv5 model. The improvements include:

[0044] 1) Construct paired samples and adjust the model input channels.

[0045] It should be noted that, considering that some components in the wiring diagram are placed horizontally and some are placed vertically, the same component may have different recognition effects when placed in different directions. In this embodiment, after obtaining the original samples corresponding to each historical wiring diagram according to the method of step S1, in order to improve the recognition accuracy of the components, each historical wiring diagram is rotated 90 degrees clockwise to obtain the auxiliary samples corresponding to the original samples and put them into the sample set. Use labelimg software to mark the components in the sample, organize them into .txt format, and put them into the sample set folder.

[0046] Since the image to be detected and its auxiliary image in this embodiment are both black and white images, the input channels of the existing YOLOv5 model are changed from three color channels of RGB (red, green, and blue) to single channel input.

[0047] 2) Adjust the model structure and introduce the channel attention mechanism.

[0048] It should be noted that the improved YOLOv5 model includes five parts in sequence: input layer, backbone layer, channel attention layer, bottleneck layer and prediction layer. After receiving the image, the input layer passes it to the backbone layer. The backbone layer is constructed in series by the Focus module, three groups of CBL+CSP modules, and CBL+spatial pyramid pooling module SPP to extract feature maps of different scales. Among them, the Focus module is used to slice and then splice the image; CBL is composed of convolution Conv, batch normalization BN and activation function Leaky Relu; CSP module is composed of CBL module, residual module and convolution layer.

[0049] The feature maps extracted by the second group of CBL+CSP modules, the third group of CBL+CSP modules and the spatial pyramid pooling module SPP in the backbone layer are respectively passed to the channel attention layer, that is, the compression module and the excitation module embedded in SENet (Squeeze-and-Excitation Networks), and the feature maps of the three scales extracted by the backbone layer are compressed and excited respectively and then passed to the bottleneck layer. The compression is to compress the information of each channel of the feature map through global average pooling, and the excitation module learns the importance weight of each channel through two fully connected layers, ignoring irrelevant features, thereby increasing the learning efficiency of the entire model. In addition, the number of parameters of the compression module and the excitation module is very small, which will not significantly increase the number of parameters of the entire model.

[0050] The bottleneck layer is composed of the feature pyramid network FPN and the path aggregation network PAN, which fuses the feature maps of different scales output by the channel attention layer; for example, the large-scale feature map has a lot of detailed information, which is upsampled and fused into the feature map of smaller size in the bottleneck layer to assist in the detection of small targets. The bottleneck layer passes the fused feature maps of different sizes to the prediction layer.

[0051] The prediction layer predicts objects of different scales in the image based on the feature maps of different scales output by the bottleneck layer.

[0052] 3) Improve the model loss function for model parameter optimization.

[0053] It should be noted that the existing YOLOv5 model loss function consists of three parts: the first classification loss , confidence loss and positioning loss The classification loss uses the cross entropy loss function to measure the difference between the model's prediction of the target category and the actual label; the confidence loss uses the binary cross entropy loss function to measure the confidence level of whether the target exists in the predicted box; the positioning loss uses CIoU (Complete Intersection over Union) as the loss function of bounding box regression to measure the difference between the predicted target bounding box and the true bounding box. The loss of each part is calculated by calculating the respective losses at three scales (large, medium, and small) and then merging them.

[0054] Considering the different sizes of each wiring diagram, for example, the control circuit wiring diagram is much larger than the lighting circuit wiring diagram, the probability of small components in the small wiring diagram is relatively high, and the probability of large components in the large wiring diagram is relatively high. Therefore, when calculating the loss function, this embodiment dynamically adjusts the weight factor of the prediction box of different scales according to the ratio of the contour area of ​​the historical wiring diagram to the area of ​​the sample image. The smaller the contour area, the greater the weight factor of the small-scale prediction box.

[0055] Exemplarily, when the ratio of the contour area of ​​the historical wiring diagram in the original sample to the sample image area is less than 0.4, the weight factors of the small, medium and large scale prediction boxes are 6, 5 and 3 respectively; when the ratio is greater than or equal to 0.4 and less than 0.7, the weight factors of the small, medium and large scale prediction boxes are 5, 6 and 3 respectively; when the ratio is greater than or equal to 0.7, the weight factors of the small, medium and large scale prediction boxes are 3, 5 and 6 respectively.

[0056] Added inter-class difference loss to the loss function , which is used to measure the difference in the center point coordinates, size width and size height of the prediction box between the original sample and the corresponding auxiliary sample. The formula is as follows:

[0057] ,

[0058] in, Indicates the number of prediction boxes, , and Represents the original sample The center point coordinates, size width and size height of the prediction box, and Indicates auxiliary sample The size width and size height of the prediction box. Since the auxiliary sample is obtained by rotating 90 degrees clockwise, the size width and size height of the prediction box in the auxiliary sample correspond to the size height and size width of the prediction box in the original sample after reverse rotation. Indicates auxiliary sample The center point coordinates of the predicted box The value obtained after the inverse rotation transformation.

[0059] Finally, the loss function of this scheme is Includes: First classification loss , confidence loss , positioning loss and inter-class difference loss , each part is given a certain weight coefficient, and the formula is as follows:

[0060] ,

[0061] in, , , and Represent the weight coefficients of the first classification loss, confidence loss, positioning loss and inter-class difference loss respectively; , and Represent the weight factors of large, medium and small scales respectively; , and Represents the classification loss of large, medium and small scales respectively; , and Represents the confidence loss of large, medium and small scales respectively; , and They represent the localization losses at large, medium and small scales respectively.

[0062] It should be noted that the choice of weight coefficients depends on the requirements of the specific task and the performance of the model. For example, if the model performs poorly in positioning, increase the weight of positioning loss; if the model has problems in classification, increase the weight of classification loss.

[0063] During the training process, 500 samples are randomly selected every 10 rounds for model verification, and the model with the best verification effect is saved, that is, the parameters with the smallest overall loss are saved. The batch size is set to 10, and the initial learning rate is set to 0.0001. The training is performed until the learning rate reaches 0.000001 and no longer decays, or the maximum number of iterations is reached, and the trained YOLOv5 model is obtained as the component detection model.

[0064] The original image to be detected and its rotated auxiliary image form an image pair which are input into the trained component detection model for detection, and the first detection result of the image to be detected and the second detection result of the auxiliary image are output. Both the first detection result and the second detection result include the coordinates of each component frame (2 coordinate values ​​of the upper left corner, 2 coordinate values ​​of the lower right corner, and the coordinates of the center point of the component frame), the height and width of the component frame, the confidence level, and the component category probability.

[0065] Finally, the two detection results are fused to obtain the component frames and component categories in each image to be detected, including:

[0066] According to the transformation matrix between each image to be detected and the corresponding auxiliary image, the coordinates of each component frame in the second detection result of the auxiliary image are converted to the coordinates in the image to be detected to obtain the third detection result; the component frames in the first detection result and the third detection result are matched with the minimum error between the component frames and less than the boundary error threshold as the matching condition, and the final component frame is obtained by calculating the intersection and union ratio for each pair of matched component frames, and the component category with the highest probability is taken as the final component category. The component frame coordinates and component category in the current wiring diagram are recorded.

[0067] It should be noted that the final component frame is obtained by calculating the intersection and union ratio for each pair of matching component frames, including:

[0068] According to the coordinates of the upper left corner and the lower right corner of the two matching component frames, the coordinates of the upper left corner and the lower right corner of the intersection are obtained; according to the coordinates of the upper left corner and the lower right corner of the intersection, the intersection area is calculated, and according to the intersection area and the areas of the two component frames, the union area is obtained;

[0069] The intersection over union (IoU) is calculated based on the intersection area and the union area. If the IoU is higher than the preset IoU threshold, the two bounding boxes are merged. Otherwise, the component box with higher confidence is taken from the two component boxes as the final component box.

[0070] S3. Capture the component frame image according to the component frame and the component category, identify the first text box and the first text content in the component frame image, obtain two text detection images for each image to be detected according to the first text box and the component frame, and simultaneously pass them into the text box detection model to obtain the second text box outside the component frame; capture the second text box image from each image to be detected and pass it into the text recognition model to identify the second text content.

[0071] It should be noted that the text content in the wiring diagram includes: labels and pin numbers; the labels represent the component numbers, and the pin numbers represent the positions of the component connecting the circuit lines.

[0072] Specifically, if Figure 2 As shown, there are many ways to indicate the number, including: directly using the component code, for example, YK is used to indicate the remote control switch, and KK is used to indicate the control switch; using the component code + number, for example, S1, S2 and S3 are used to indicate three switches; using the component code + symbol + pin number, for example, CZ:A2 and CZ:A3 are used to indicate the number of aviation plugs CZ with different pins. If the component is placed horizontally, the number is above the component; if the component is placed vertically, the number is on the right side of the component.

[0073] The pin number is represented by different numbers. If the component uses a circle to represent its terminal, the pin number is written inside the circle. Otherwise, the pin number is written at the connection point between the two ends of the component and the circuit. For example, the transfer switch uses two circles to represent the switch terminals, and the pin number is written inside the circle; the switch status integrated indicator uses a rectangle to represent its shell, which includes many terminals represented by circles, and the pin number is also written inside the circle.

[0074] This embodiment sequentially performs the following steps S31 and S32 to respectively identify the text content inside the component frame and the text content outside the component frame:

[0075] S31 . Capture a component frame image according to the component frame and the component category, and identify a first text frame and a first text content in the component frame image.

[0076] Since the pin numbers inside components are usually in digital form, they are relatively easy to identify. To intercept the component frame image based on the component frame and component category is to obtain the components with the pin numbers written inside the components according to the component category, intercept a single component frame image based on the component frame coordinates of these components, and then use Tesseract-OCR to identify the first text box and the first text content in the component frame image, that is, to obtain the text box coordinates and pin number of the pin number inside the component.

[0077] S32, respectively covering the first text box and the component box to obtain two text detection images for each image to be detected, and simultaneously passing them into the text box detection model to obtain the second text box outside the component box; intercepting the second text box image from each image to be detected and passing it into the text recognition model to recognize the second text content.

[0078] It should be noted that the text outside the component is in the wiring diagram, with different lengths, variable positions, and uncertain directions. The text includes letters, numbers, and symbols. In addition, due to the different complexity of the wiring diagram, some places may have dense lines and text close to lines. It is difficult to accurately recognize all of them using the general optical character recognition method, and the effect is not good. Therefore, this embodiment first uses the text box detection model to obtain an accurate text box, and then cuts out the text image according to the text box and transmits it to the text recognition model to obtain the text in the text box.

[0079] It should be noted that the text box detection model is obtained by training the improved Faster R-CNN model. Considering that the components in the wiring diagram may affect the recognition of the text, directly removing the recognized component box may also remove part of the text and cause recognition errors. Therefore, the improved Faster R-CNN model in this embodiment receives two images at the same time, namely, the text content inside the component in the original image is covered, and the image obtained by covering each component in the original image, and uses two ResNet networks as a dual-branch feature extraction network to extract features from the two input images to obtain feature maps, and adds a feature pyramid network to fuse the feature maps extracted by the dual branches and then pass them to the region proposal network (RPN) and the region recognition network; the region proposal network predicts the candidate region based on the fused feature map and then passes it to the region recognition network, the region recognition network first uses the ROI pooling layer to extract the feature sub-map of the candidate region from the candidate region and the fused feature map, and then uses multiple fully connected layers to perform category prediction and bounding box regression on the candidate region based on the feature sub-map, that is: further extract features from the feature sub-map to predict the bounding box coordinates of the candidate region, and screen the bounding box of the candidate region through non-maximum suppression to obtain the precise coordinates of the text box.

[0080] When training the improved Faster R-CNN model, the loss function is calculated by calculating the second classification loss , boundary regression loss Difference between the two-branch feature map The second classification loss uses cross entropy loss, including the binary classification loss of the region proposal network for the foreground and background, and the classification loss of the region recognition network for the candidate region category; the boundary regression loss uses smooth L1 loss, including the offset loss of the predicted border and the real border of the candidate region by the region proposal network and the region recognition network; the difference of the two-branch feature map is obtained by calculating the cosine distance of the two-branch feature map, and the formula is as follows:

[0081] ,

[0082] in, , and Represents the classification loss , boundary regression loss and dual-branch feature map difference loss The weight of; P and Q represent the dual-branch feature map, Indicates taking the L2 norm.

[0083] It should be noted that the text recognition model is trained by introducing a multi-head self-attention mechanism into the CNN-LSTM combination model. The multi-head self-attention mechanism helps the model learn more subtle and accurate text information. The text recognition model includes: Convolutional Neural Network (CNN), sliding window module and Long Short-Term Memory (LSTM) with multi-head self-attention mechanism. Among them, the convolutional neural network CNN is used to extract the feature map of the text box image, and the sliding window module is used to select the feature vector in the feature map by sliding the window, and then pass it to the long short-term memory network module with multi-head attention mechanism after forming the feature matrix, to encode and decode the feature sequence, and output the text sequence in each text box image.

[0084] Total loss function when training a text recognition model The attention weight regularization loss is calculated by the following formula and character sequence loss The attention weight regularization loss is used to ensure that the model's attention distribution is concentrated in the text area and improve recognition accuracy; the character sequence loss measures the difference between the character sequence output by the model and the true character sequence:

[0085] ,

[0086] Among them, represents the maximum length of the output text sequence, represents the regularization coefficient, represents the attention weight of the th head, represents taking the L2 norm; represents the th character is predicted as the th character in the character set probability, represents the one-hot encoding of the th character label in the true sequence, represents the weight of the character sequence loss.

[0087] After the improved Faster R-CNN model is trained, a text box detection model is obtained. After the CNN-LSTM model introducing the multi-head self-attention mechanism is trained, a text recognition model is obtained.

[0088] During implementation, for each image to be detected obtained in step S1, the first text box is covered with white to obtain the first text detection image; the component box is covered with white to obtain the second text detection image. The first text detection image and the second text detection image are simultaneously transmitted to the text box detection model to obtain the coordinates of the second text box. According to the coordinates of the second text box, the text box image is intercepted and scaled into a second text box image of the same size and then transmitted to the text recognition model to output the character sequence in each second text box image and recognize the second text content.

[0089] Preferably, considering that there will be no special symbols at the head and tail of the recognized character sequence, such as "?", "[", "]", "|", "丿", etc., by defining a regular expression, the illegal characters at the head and tail of the text are removed to obtain the second text content corresponding to each second text box.

[0090] S4. Associate the first text box and the second box with the component box, and obtain the text content corresponding to each component in the substation two-dimensional drawing as the recognition result.

[0091] The second text content includes a label and a terminal number; the first text content and the second text content are associated with the component frame to obtain the text content corresponding to each component in the two-dimensional drawing of the substation: including: the first text box is associated with the component frame where it is located, and the first text content is used as the terminal number of the corresponding component; the distance between the component frame and the center point coordinates of each second text box is calculated, and the second text box with a distance less than the distance threshold is associated with the corresponding component frame. If there is only one associated second text box, the second text content in the second text box is the label of the component in the current component frame; otherwise, the second text content in multiple second text boxes is identified. When the second text content is pure numbers, it is the terminal number corresponding to the component in the current component frame; when the second text content contains characters, it is the label corresponding to the component in the current component frame.

[0092] After the above steps S1-S4, the recognition result is obtained, including: all the components in each wiring diagram in the two-dimensional drawing, and the corresponding labels and terminal numbers of the components.

[0093] Furthermore, the number of wiring diagrams in the two-dimensional drawing is counted; for each wiring diagram, the component categories, the total number of components, and the number of components in each component category are counted; for each component, the component number, component category, and the terminal number used are displayed.

[0094] It should be noted that the design drawing information and wiring diagram component information are generally written in the form of a table in the lower right corner of the two-dimensional drawing. The design drawing information includes: project name, design unit name, date, drawing name and drawing number; the wiring diagram component information includes: serial number, label, name, model, quantity and remarks.

[0095] In this embodiment, the wiring diagram information in the lower right corner is used to verify the recognition result and to improve the component data, so as to maximize the data value in the two-dimensional drawing.

[0096] Specifically, in step S2, when the component detection model is used to identify the component frame in each image to be detected, when there is no component frame in the detection result output by the component detection model, it means that the input image to be detected contains design unit information and wiring diagram component information, and it is marked as an information image; Tesseract-OCR is used to identify the third text box and the third text content in the information image; the third text content is parsed to obtain the label, component category, name, model and quantity as standard data; and the recognition result is verified based on the standard data to obtain a verification result.

[0097] It should be noted that parsing the third text content includes: obtaining wiring diagram information from the third text content according to a table template, identifying whether the label in the wiring diagram information contains preset characters, if so, truncating the label according to the preset characters to obtain the component code, obtaining the component category according to the component code, and obtaining the label of a single component according to the component code and quantity; if not, the label in the wiring diagram is the label of a single component, and the component category is obtained according to the label.

[0098] For example, the label in the wiring diagram information is "LP1~7", the name is "connecting piece", the model is "JL1-2.5 / 2", and the quantity is "7", that is, 7 connecting pieces are represented by one piece of information; when parsing, the component code "LP" is obtained according to the preset character "~", and the corresponding component category can be obtained according to the component code, and then according to the number of components, a digital number is added after the component code to get the label of a single component.

[0099] Furthermore, verification results are obtained based on verification of the recognition results with standard data, including: obtaining a list of components that exist in the recognition results but do not exist in the standard data, and a list of components that do not exist in the recognition results but exist in the standard data according to component categories; obtaining a list of components that exist in the recognition results but do not exist in the standard data, and a list of components that do not exist in the recognition results but exist in the standard data according to component numbers; obtaining component categories with inconsistent quantities according to component categories and quantities.

[0100] If the verification result is not empty, it means that there is a problem with the recognition result or the wiring diagram design itself; if the verification result is empty, it means that the recognition result is consistent with the wiring diagram information, and the basic information of the corresponding component can be improved according to the component model in the wiring diagram information.

[0101] Preferably, the design drawing information is parsed from the third text content and supplemented into the recognition result.

[0102] Compared with the prior art, the present embodiment provides a method for recognizing two-dimensional drawings of substations. Considering that the placement direction of components in the two-dimensional drawings will affect the recognition effect, an auxiliary image is obtained by rotating the image to be detected, and the accuracy of component recognition is improved by fusing the detection results of the two images; considering the existence of terminal numbers inside the components, different methods are used to recognize the text inside and outside the components to ensure the integrity of the text content; considering that some components are close to the text, an image for text content recognition is constructed by covering the component frame and the text inside the component frame respectively, and recognition is performed simultaneously, and the accuracy of text recognition is improved by complementary fusion of the two image features; fully explore the information of each part in the two-dimensional drawings, identify the design drawing information and wiring diagram component information in the two-dimensional drawings, on the one hand improve the recognition results, and on the other hand extract standard data from them to verify the recognition results and judge Whether the recognition result is accurate, or whether there are design problems in the wiring diagram itself, improves data quality and work efficiency; in the loss function of the component detection model, the difference between the detection results of the two images is increased to reduce the impact of different placement directions of the components; by introducing the channel attention mechanism in the component detection model, the model is helped to better focus on important information and improve the accuracy of the model; different weight factors are given to prediction boxes of different scales according to the range of the contour area of ​​each wiring diagram, which enhances the robustness of the model and improves the detection performance of targets of different scales; the text box detection model supports the simultaneous input of two images, and uses the feature pyramid network to fuse the feature maps of the two images, so that the model uses complementary feature information to improve the richness of feature expression, and reduces the difference between the two feature maps in the loss function, prompting the model to learn more consistent feature representations, thereby improving the stability and generalization ability of the model.

[0103] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0104] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for identifying a two-dimensional drawing of a substation, characterized in that: The following steps are involved: After preprocessing the image of the substation two-dimensional drawing, each wiring diagram in the image is segmented to obtain each sub-image; each image to be detected is obtained by scaling each sub-image to a preset size, and each sub-image is rotated and scaled to a preset size to obtain an auxiliary image corresponding to the image to be detected; Each image to be inspected and its corresponding auxiliary image are sequentially input into the component inspection model, and the two inspection results output are fused to obtain the component frame and component category in each image to be inspected; According to the component frame and the component category, a component frame image is intercepted, and a first text frame and a first text content in the component frame image are identified; the first text frame and the component frame are respectively covered to obtain two text detection images for each image to be detected, and the images are simultaneously transmitted to a text frame detection model to obtain a second text frame outside the component frame; a second text frame image is intercepted from each image to be detected and transmitted to a text recognition model to identify a second text content; The first text box and the second text box are associated with the component box, and the text content corresponding to each component in the two-dimensional drawing of the substation is obtained as the recognition result.

2. The method for recognizing a two-dimensional drawing of a substation according to claim 1, characterized in that: The recognition method also includes: when there is no component box in the detection result output by the component detection model, marking the input image to be detected as an information image; identifying a third text box and a third text content in the information image; parsing the third text content to obtain a label, component category, name, model and quantity as standard data; and verifying the recognition result based on the standard data to obtain a verification result.

3. The method for recognizing a two-dimensional drawing of a substation according to claim 1, characterized in that: The component detection model is obtained by training an improved YOLOv5 model; the improved YOLOv5 model includes an input layer, a backbone layer, a channel attention layer, a bottleneck layer and a prediction layer in sequence; the input layer is a single-channel input, the backbone layer is serially constructed by a Focus module, three groups of CBL+CSP modules, and a CBL+spatial pyramid pooling module, the feature maps extracted by the second group of CBL+CSP modules, the third group of CBL+CSP modules and the spatial pyramid pooling module in the backbone layer are respectively transmitted to the channel attention layer, the feature maps of three scales extracted by the backbone layer are respectively compressed and stimulated and then transmitted to the bottleneck layer; the bottleneck layer is composed of a feature pyramid network and a path aggregation network, and the feature maps of different scales output by the channel attention mechanism are fused; the prediction layer predicts component frames of different scales according to the feature maps of different scales output by the bottleneck layer.

4. The method for recognizing a two-dimensional drawing of a substation according to claim 1, characterized in that: The training samples of the component detection model include original samples constructed according to the historical wiring diagram and auxiliary samples obtained by rotating the historical wiring diagram; the loss function of the component detection model is obtained by calculating the first classification loss, confidence loss, positioning loss and inter-class difference loss and weighted summing them; the first classification loss, confidence loss and positioning loss are respectively obtained by calculating the losses corresponding to each scale, and then weighted summing them according to the weight factors of the prediction boxes of each scale; the inter-class difference loss is obtained by calculating the difference between the prediction boxes of the original sample and the corresponding auxiliary sample.

5. The method for recognizing a two-dimensional drawing of a substation according to claim 4, characterized in that: The inter-class difference loss is calculated by the following formula: ， in, Indicates the number of prediction boxes, , and Represents the original sample The center point coordinates, size width and size height of the prediction box, and Indicates auxiliary sample The width and height of the prediction box, Indicates auxiliary sample The center point coordinates of the predicted box The value obtained after the inverse rotation transformation.

6. The method for recognizing a two-dimensional drawing of a substation according to claim 1, characterized in that: The two detection results of the fusion output are used to obtain component frames and component categories in each image to be detected, including: according to the transformation matrix between the auxiliary image and the image to be detected, the coordinates of each component frame in the second detection result corresponding to the auxiliary image are transformed to obtain a third detection result; with the error between the component frames being minimized and less than a boundary error threshold as a matching condition, the component frame in the third detection result and the component frame in the first detection result corresponding to the image to be detected are matched, and the final component frame is obtained by calculating the intersection and union ratio for each pair of matched component frames, and the component category with a large probability is taken as the final component category.

7. The method for recognizing a two-dimensional drawing of a substation according to claim 1, characterized in that: The text box detection model is obtained by training an improved Faster R-CNN model; the improved Faster R-CNN model receives two images at the same time, uses two ResNet networks as dual-branch feature extraction networks to extract features from the two input images respectively to obtain dual-branch feature maps, and adds a feature pyramid network to fuse the dual-branch feature maps before passing them to a region proposal network and a region recognition network; the region proposal network predicts a candidate region based on the fused feature map and passes it to the region recognition network, the region recognition network first uses a ROI pooling layer to extract a feature sub-map of the candidate region from the candidate region and the fused feature map, and then uses multiple fully connected layers to perform category prediction and bounding box regression on the candidate region based on the feature sub-map to obtain the coordinates of the text box.

8. The method for recognizing a two-dimensional drawing of a substation according to claim 7, characterized in that: The loss function of the text box detection model is obtained by calculating the second classification loss, the boundary regression loss and the dual-branch feature map difference and weighted summing them up, wherein the dual-branch feature map difference is obtained by calculating the cosine distance of the dual-branch feature map.

9. The method for recognizing a two-dimensional drawing of a substation according to claim 1, characterized in that: The text recognition model is trained by introducing a multi-head self-attention mechanism into the CNN-LSTM combined model; the loss function of the text recognition model is The attention weight regularization loss is calculated by the following formula and character sequence loss And weighted summation gives: ， in, Indicates the maximum length of the output text sequence, represents the regularization coefficient, Indicates The head's attention weight, represents the total number of attention heads, Indicates taking the L2 norm; Indicates Characters predicted as character set Middle The probability of a character, Indicates the real sequence One-hot encoding of character labels, Represents the weight of the character sequence loss.

10. The method for recognizing a two-dimensional drawing of a substation according to claim 1, characterized in that: The second text content includes a label and a terminal number; the first text box and the second text box are associated with the component box to obtain the text content corresponding to each component in the two-dimensional drawing of the substation, including: the first text box is associated with the component box where it is located, and the first text content is used as the terminal number of the corresponding component; the distance between the component box and the center point coordinates of each second text box is calculated, and the second text box with a distance less than the distance threshold is associated with the corresponding component box. If there is only one associated second text box, the second text content in the second text box is the label of the component in the current component box; otherwise, the second text content in multiple second text boxes is identified. When the second text content is pure numbers, it is the terminal number corresponding to the component in the current component box; when the second text content contains characters, it is the label corresponding to the component in the current component box.

Images