A method for identifying and structuring terminal block wiring

Abstract

A method for identifying and structuring terminal block wiring, collecting a substation terminal block content image, using OCR technology to perform text recognition, then performing number type recognition on the OCR-recognized text according to the position coordinates of the recognized text box and image features, and finally identifying the correspondence between the terminal group, terminal port and wiring according to the position of the text box, and deriving a structured result. The present invention solves the problem that the identification of substation terminal block wiring currently mainly relies on manual identification with low efficiency, and realizes rapid and accurate identification of substation terminal block wiring.

Description

Technical Field

[0001] The invention belongs to the technical field of image recognition, relates to the recognition of characters printed on equipment, is used for intelligent recognition of secondary circuit wiring of a transformer substation, and is a method for terminal block wiring recognition and structured export. Technical Background

[0002] In substation construction, with the development of computer technology, IoT-based recognition technology can intelligently identify the on-site wiring of the substation secondary circuit. By intelligently comparing it with the CAD design drawings, it can quickly discover potential risks caused by inconsistencies between the substation site and the completed drawings.

[0003] Optical Character Recognition (OCR) is a technology that recognizes and extracts the target text in a complex image background, thereby converting the text in the image into editable text information. In today's trend of electronic information, OCR technology, as an important link in the process of text electronicization, plays a huge role and is widely used in finance, medical care, e-commerce, government affairs and other fields.

[0004] Text location and text recognition are two basic algorithms of OCR, namely finding the location of text in an image and recognizing and understanding the content of text. With the development of applications, users have also put forward requirements for understanding text content, and it is necessary to output the results required by users based on the spatial relationship and semantics of text. For example, in document recognition and some form-based document recognition applications, OCR recognition can support exporting structured results in the form of KV (key-value).

[0005] For the intelligent digital design and construction of substations, the main problem at present is that there is no OCR recognition and text understanding technology that can meet the application requirements in the specific scenario of substation terminal blocks. Therefore, it is of great significance to study text recognition and structured export suitable for terminal blocks.

[0006] The text areas in existing text understanding application scenarios, such as certificates and form documents, have text information with characteristics such as concentration and prominence, and the recognition scenes are mostly flat, making it easy to achieve a recognition environment with good lighting. Compared with these image text understanding applications, the terminal block at the substation site is a spatial three-dimensional recognition scene, which is also affected by the installation position. The terminal block text information has the characteristics of discrete distribution, small text, and easy occlusion or deformation, which creates difficulties for the terminal block, a special text understanding scenario. In addition, the results of conventional text recognition still require further manual identification of which part of the terminal block the recognized text belongs to, which cannot meet the recognition requirements. Summary of the invention

[0007] The problem to be solved by the present invention is to solve the problem that the current identification of the terminal block wiring of a substation mainly relies on manual identification with low efficiency, and the purpose is to quickly and accurately automatically identify the terminal block wiring of the substation.

[0008] The technical solution of the present invention is: a method for identifying and structuring terminal block wiring, which collects a substation terminal block image, first performs text recognition on the image through OCR, and then identifies the terminal block wiring number type based on the text box position and image features recognized by OCR, and finally derives the structured result of the terminal block wiring number based on constraints, wherein the structured result refers to the identified terminal block structure corresponding to the identified number and number type.

[0009] Furthermore, the OCR text recognition is specifically as follows:

[0010] 1) Preprocessing of substation terminal row images, including geometric distortion, deformity correction, blur removal and image enhancement;

[0011] 2) Using a bottom-up approach, fine-grained character text is detected and then connected into coarse-grained text regions;

[0012] 3) Training the text recognition network: input the text recognition information of the coarse-grained text area recognition into the ResNet recognition network for training. The loss function uses CTC loss. When the loss value reaches the minimum, the recognition is considered successful.

[0013] 4) Greedy encoding is performed on the recognized text information. The greedy strategy is: the selected word has the highest probability;

[0014] 5) Generate and output the final text result.

[0015] Furthermore, the number type identification is specifically:

[0016] 1) Starting from the position of the text box, filter out the text boxes whose recognition results are single-digit type to obtain the terminal port number;

[0017] 2) Based on the image features, the main line of the terminal row is identified, and then the terminal group number is obtained according to the position of the main line and the text box corresponding to the main line;

[0018] 3) Identify the terminal block outline in the terminal block image. Except for the terminal port number and terminal group number, the content of other text boxes inside the outline is the wiring number.

[0019] Furthermore, the steps of deriving structured results according to the constraints are as follows:

[0020] 1) Sort the terminal group text box, terminal port text box and wiring text box from top to bottom respectively;

[0021] 2) For each terminal port, the wiring frame with the smallest longitudinal distance from it on the left and right sides is considered to correspond to it; if the terminal port with corresponding wiring on the left and right sides is not found, the left and right wiring is set to empty;

[0022] 3) Match each terminal port and its left and right wiring to the corresponding terminal group.

[0023] The method of the present invention provides a practical application scenario for substation terminal blocks, and studies the technology of substation terminal block wiring identification and structured export. It not only identifies the text on the terminal block, but also corresponds it to the terminal block structure. The exported result does not need to manually distinguish the text recognition result from the terminal block structure to determine the number corresponding to the terminal port, and the recognition result can be directly and intuitively matched with the terminal block design drawing. The present invention uses OCR technology to overall identify the text on the terminal block content image, and then combines the position of the identified text box and image features to perform number classification, and finally analyzes the relationship between the text boxes according to the position to export the structured result. The present invention realizes end-to-end processing, greatly reduces the computing overhead, and almost reaches real-time speed, which can realize the rapid recognition and structuring of terminal block text.

[0024] The effective benefits of the present invention are: a terminal block wiring recognition technology is proposed, which helps to solve the problem that the current identification of substation terminal block wiring mainly relies on manual recognition with low efficiency, and improves the accuracy of terminal block text relationship recognition from the two aspects of text position and image features, and realizes the rapid and accurate identification of substation terminal block wiring, and the identification output result corresponds to the terminal block structure, which can be convenient for intuitive use. The present invention has good extensiveness and practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is an implementation flow chart of the present invention.

[0026] Figure 2 These are the relevant results in the terminal block wiring identification process of the present invention, (a) is the original image, and (b) is the number type identification result. DETAILED DESCRIPTION

[0027] The present invention proposes a method for identifying and deriving terminal block wiring structure, such as Figure 1 As shown, the following steps are included:

[0028] 1)OCR recognizes the text content of the image.

[0029] 1.1) Perform image preprocessing on the input content image, including: geometric transformation (perspective, distortion, rotation, etc.), deformity correction, blur removal, and image enhancement.

[0030] 1.2) Perform text detection on the input content image. Through a bottom-up approach, fine-grained text is detected and then connected into coarse-grained text regions.

[0031] Get the coordinate box of each character and get fine-grained text. Manually annotate the data set, use the classic convolutional neural network VGG16 for model training, and use the Gaussian heat map CAM visualization network to guide whether the high response area of ​​the classification falls on the core part of the target. Use the trained model to predict the coordinate box of the character.

[0032] After obtaining the character coordinate frame information, it is necessary to consider the closeness between the two characters. The present invention adopts the following strategy to determine whether the two characters are in the same text area: suppose the coordinate values ​​of the upper left, upper right, lower right, and lower left of the coordinate frame of character m are [(x1, y1), (x2, y2), (x3, y3), (x4, y4)], then the coordinate value of the center point of character m (x m ,y m )for:

[0033]

[0034] Similarly, we get the coordinate value of the center point of character n (x n ,y n ), the Euclidean distance d between two characters and the angle θ between two characters can be calculated.

[0035]

[0036] If the Euclidean distance d between two characters does not exceed k times the length of any character, k=1.5~2, and the angle θ of the relative position is within thirty degrees, then the two characters are judged to be in the same text area, and the length of the characters is obtained from the character coordinate frame.

[0037] The position features, size features, distance features, etc. of the text are extracted from the sequence formed by fine-grained text, and multiple characters are merged into a text line region. At the same time, a bidirectional long short-term memory network, i.e., bidirectional LSTM (Long short-term memory), is added to the detection network to retain the contextual features of the extracted text. The coordinate box of each text line region is output, i.e., the coarse-grained text region.

[0038] 1.3) Perform text recognition on the coarse-grained text area. Input the character coordinate information, text line area coordinate information and text context features obtained in step 1.2 into the text recognition program; train a ResNet recognition network, using the character information of the terminal strip as the positive sample, and the terminal strip parts and body after covering the character information as the negative sample. The positive sample focuses on the recognition of Chinese and English letters, and a large number of positive and negative samples are used to improve the recognition effect. Fine-tune is used to improve the recall rate of the network. The loss function of the recognition model uses CTC loss. When the loss value reaches the minimum, the recognition is considered successful.

[0039] 1.4) Greedy encode the recognized text. The input is the text character recognition result of step 1.3. The greedy strategy is: the word with the highest probability is selected.

[0040] 1.5) Generate and output the final text result.

[0041] 2) Using the text box position coordinates and text content information obtained in step 1), combined with the terminal row image, the type of number is identified.

[0042] 2.1) Pre-process the terminal strip image, including conversion to grayscale image, Gaussian blur, and edge sharpening.

[0043] 2.2) Identify the terminal group text box. Based on the terminal block image features, first identify the circular area where the terminal block body is arranged vertically, and use the RANSAC algorithm to identify the straight lines arranged in the circular area. Compared with the ordinary linear regression method, this algorithm can better resist the interference of outliers. The specific steps of the algorithm are as follows:

[0044] a. Select the smallest data set that can estimate the straight line

[0045] b. Use this minimum data set to obtain the straight line parameter a and slope b

[0046] c. Substitute all data into the straight line expression to get the estimated value of the ordinate. If the difference between the estimated value and the true value is less than the set threshold sigma, it is considered that the data point can be represented by this straight line, that is, it is an "inlier". Calculate the number of "inliers"

[0047] d. Compare the number of "inner points" between the current line and the best line previously derived. The larger the number, the better the line.

[0048] e. Repeat a to d until the iteration ends or the number of "inner points" exceeds a certain number

[0049] The resulting straight line is the straight line where the terminal block body is located, so the text box intercepted by the straight line is considered to be the terminal group text box, and its text content is the terminal group number.

[0050] 2.3) According to the recognition result of step 1), filter the text box of the terminal port.

[0051] First, the terminals with numbers as recognition results are extracted as candidate terminals. Then, the density-based clustering algorithm DBSCAN is used to cluster all text boxes recognized as numbers. Because terminal ports are generally clustered together, clustering can exclude text boxes that are not terminal ports but recognized as numbers. The cluster with the most coordinate points in the clustering results is considered to be the cluster where the terminal port is located, thereby excluding interference from other text boxes recognized as numbers. The text content of the terminal port text box is the terminal port number.

[0052] 2.4) Identify the text box of the wiring number. Threshold processing is performed on the grayscale image of the terminal block, and the cluttered area is eroded and then expanded. The contour is then identified to obtain the approximate area of ​​the terminal block and the wiring. The text box that is not marked as the terminal group number or the terminal port number is considered to be the text box of the wiring number if its coordinates are within the contour, and its text content is the wiring number.

[0053] 3) Export structured results.

[0054] Preprocessing includes sorting the terminal port number text boxes from top to bottom for easy output in sequence. The top and bottom here refer to the terminal port numbers from small to large, with the smallest terminal port being 'top' and the largest terminal port being 'bottom'. Generally, the terminal row is arranged in a positive vertical direction, which is described as from top to bottom.

[0055] Calculate the center coordinates of each text box. For each text box [(x1, y1), (x2, y2), (x3, y3), (x4, y4)], the center point coordinate value (x m ,y m )for:

[0056]

[0057] Identify each terminal port and the wiring on its left and right sides in the terminal row image, calculate the vertical distance between the terminal port and the wiring on the left and right sides and compare it with the set threshold. If it is less than the set threshold, it is the wiring on the left and right sides of the terminal port. If there are wiring on the left and right sides, the wiring with the smallest vertical distance from the terminal port is used as the wiring corresponding to the terminal port, and a pair of terminal port-wiring pairs with the closest vertical distance is obtained. According to the horizontal coordinate of the center of the text box of the corresponding terminal port number and wiring number, it is determined whether it is a left wiring or a right wiring, and the wiring is assigned to the terminal port to obtain a terminal port number-wiring number pair. For terminal ports that are not assigned to wiring on both the left and right sides, both their left and right wirings are set to empty.

[0058] Assign the terminal openings to the corresponding terminal groups: each terminal opening belongs to the terminal group that follows it vertically.

[0059] Finally, the recognized structured object is output in JSON format.

[0060] The following is an example to illustrate the effect of the present invention. Figure 2 , which are the related results of the number type identification of the present invention: Figure 2 (a) is the original image, and (b) is the result of number type recognition. The results of number type recognition are as follows:

[0061] [[[[295,30],[396,37],[394,67],[293,60]],'3CD:',0.9010699391365051,'LINE'],

[0062] [[[378,34],[590,39],[588,75],[377,69]],'He Junyi Line Unit',0.9939853549003601,'OTHER'],

[0063] [[[166,189],[285,220],[279,239],[161,208]],'743(0) / B140E',0.8160243630409241,'LINE'],

[0064] [[[164,240],[282,266],[277,287],[159,262]],'747(7) / B149E',0.8516485095024109,'LINE'],

[0065] [[[175,326],[299,341],[295,362],[172,347]],'201(7) / B1480',0.836646556854248,'LINE'],

[0066] [[[157,455],[308,465],[307,486],[156,476]],'33R11(8) / B148',0.8301181793212891,'LINE'],

[0067] [[[379,548],[572,550],[572,576],[379,574]],'4CD:He Liangyi Unit',0.9376855492591858,'OTHER'],

[0068] [[[144,611],[302,623],[300,645],[142,632]],'741 / 22XL11-117B',0.9820869565010071,'LINE'],

[0069] [[[139,713],[287,702],[288,720],[140,731]],'743 / 22X111-117',0.7982786297798157,'LINE'],

[0070] [[[656,807],[717,801],[721,825],[660,832]],'1B17',0.9045082926750183,'LINE'],

[0071] [[[663,860],[726,857],[728,880],[664,883]],'15',0.7972480058670044,'LINE'],

[0072] [[[178,872],[317,846],[322,868],[182,894]],'201 / XL2-B137A',0.964455246925354,'LINE'],

[0073] [[[157,1011],[322,982],[326,1005],[162,1030]],'29R11 / XL2-B137A',0.8134316205978394,'LINE']]

[0074] The above recognition results include: text box coordinate information, text recognition content, corresponding loss value and number type. Figure 2 In (b), the type recognition result is in the upper left corner of each detection box, such as LINE, PORT, etc. The text box with text recognition can match its type and perform structured output. The present invention not only recognizes the text printed on the terminal block, but also further recognizes the terminal block number type corresponding to the text, which is beneficial to the actual application needs of the substation site.

[0075] for Figure 2 The json format of the structured result finally output by the present invention is expressed as:

[0076]

[0077]

[0078]

[0079] According to the Json format, it can be converted into a picture or even directly into a design drawing, and can be further converted and used as needed.

[0080] The present invention proposes a method for outputting structured terminal block wiring recognition results. For text obtained by a text recognition method, the structured results of the corresponding terminal block are output, which is beneficial to the use of on-site construction and improves the efficiency of on-site construction verification.

Claims

1. A method for identifying and deriving terminal strip wiring structure, characterized in that The substation terminal block image is collected, and the text is first recognized by OCR. The terminal block wiring number type is identified according to the text box position and image features recognized by OCR. Finally, the structured result of the terminal block wiring number is derived according to the constraints. The structured result refers to the recognized terminal block structure corresponding to the recognized number and number type. The number type identification includes: 1) Starting from the position of the text box, filter out the text boxes whose recognition results are single-digit type, and the type corresponding to the text box is the terminal port number; 2) Based on the image features, the main line of the terminal row is identified, and then according to the position of the main line, the type corresponding to the text box on the main line is the terminal group number; 3) Identify the terminal block outline in the terminal block image. Except for the terminal port number and terminal group number, the types of other text boxes inside the outline are wiring numbers; The structured export is as follows: 1) Sort the text boxes of the terminal port numbers from top to bottom. The smallest terminal port is "up" and the largest terminal port is "down". 2) For each terminal port, calculate the longitudinal distance between the terminal port and the left and right wirings and compare it with the set threshold. If it is less than the set threshold, it is the wiring on the left and right sides of the terminal port. If there are wirings on the left and right sides, the wiring with the smallest longitudinal distance from the terminal port is used as the wiring corresponding to the terminal port; if there are no corresponding wirings on the left and right sides of the terminal port, the left and right wirings are set to empty; 3) Allocate the terminal ports to the corresponding terminal groups: each terminal port belongs to the terminal group that follows it in the vertical direction; 4) Match each terminal port and its left and right wiring to the corresponding terminal group to obtain the structured recognition result of the terminal row.

2. A method for identifying and structuring terminal block wiring according to claim 1, characterized in that OCR The text recognition is as follows: 1) Preprocessing of substation terminal row images, including geometric distortion, deformity correction, blur removal and image enhancement; 2) Using a bottom-up approach, fine-grained character text is detected and connected into coarse-grained text regions, specifically: Perform text detection on the input image, obtain the coordinate frame of each character, and obtain fine-grained text: Based on the manually annotated data set, use the convolutional neural network VGG16 for model training, and use the Gaussian heat map CAM visualization network to guide whether the high response area of ​​the classification falls on the core part of the target, and use the trained model to predict the coordinate frame of the character; After obtaining the character coordinate frame information, determine whether the two characters are in the same text area based on the closeness between the two characters; Extract the position, size and distance features of the text in the sequence formed by fine-grained text, merge multiple characters into a text line region, add a bidirectional long short-term memory network to the detection network, retain the contextual features of the extracted text, and output the coordinate box of each text line region, that is, the coarse-grained text region; 3) Training the text recognition network: input the text recognition information of the coarse-grained text area recognition into the ResNet recognition network for training. The loss function uses CTC loss. When the loss value reaches the minimum, the recognition is considered successful. 4) Greedy encoding of the recognized text information is performed GREEDY DECODER. The greedy strategy is: the selected word has the highest probability; 5) Generate and output the final text result.

3. A method for identifying and deriving terminal strip wiring according to claim 1, characterized in that the numbering The specific type identification is: 2.1) Pre-process the terminal strip image, including conversion to grayscale image, Gaussian blur, and edge sharpening; 2.2) Identify the terminal group text box. Based on the terminal block image features, first identify the circular area where the terminal block body is arranged vertically. Use the RANSAC algorithm to identify the straight line arranged in the circular area. The straight line obtained is the straight line where the terminal block body is located. The text box intercepted by the straight line is the terminal group text box, and its text content is the terminal group number. 2.3) Identify the terminal port text box. First, extract the text boxes with the recognition result of numbers as candidate terminal ports. Then use the density-based clustering algorithm DBSCAN to cluster all the text boxes recognized as numbers to obtain the clustering results. The cluster with the most coordinate points is considered to be the cluster where the terminal port is located. The terminal port text box is determined, and its text content is the terminal port number. 2.4) Identify the text box of the wiring number, perform threshold processing on the grayscale image of the terminal block, erode the cluttered area and then expand it, and then identify the contour to obtain the regional contour of the terminal block and the wiring. The text box that is not marked as the terminal group number or terminal port number is considered to be the text box of the wiring number if its position is inside the contour, and its text content is the wiring number.

Images