A method and device for identifying multi-pointer instruments in substations based on multi-task operation

Abstract

This invention relates to the field of machine recognition technology, and particularly to a multi-task-based method and device for recognizing multi-pointer instruments in substations. This invention proposes a highly efficient multi-task instrument recognition network that can simultaneously handle three key tasks: target detection, instrument centering, and pointer segmentation of substation instruments. This significantly improves the detection speed while greatly enhancing the accuracy of instrument pointer detection. The algorithm proposed in this invention has higher universality, enabling the recognition of multiple instruments and multiple pointers, effectively solving the problem of using a different recognition algorithm for each instrument. The dataset for this invention is entirely derived from actual substations. Even in outdoor scenes and under conditions of poor image quality, it achieves accurate recognition of single and double-needle instruments in substations, including instruments in switchgear, discharge counters, main transformer temperature instruments, current transformers, voltage transformers, and surge arresters, with an accuracy approaching 100%. The algorithm is flexible in configuration, easy to set up, and reliable in operation.

Description

Technical Field

[0001] This invention relates to the field of machine recognition technology, and in particular to a method and apparatus for recognizing multi-pointer instruments in substations based on multi-task operation. Background Technology

[0002] During the inspection of substations, pointer instruments are one of the main inspection equipment. Due to the influence of shooting angle, lighting, aging or damage of the instrument itself, and external weather, the automatic reading of pointer instruments will have a large error.

[0003] Traditional methods for instrument recognition mainly include: (1) detecting the disc and scale lines through image binarization, edge detection, Hough transform, connected component analysis, etc., or detecting the instrument and pointer through SIFT features, and finally calculating the instrument reading by calculating the angle between the pointer and the scale. These methods are relatively fast, but their recognition effect cannot meet the actual requirements for different instruments and complex scenarios.

[0004] Traditional pointer-type meters primarily involve preprocessing the acquired image (color space conversion, noise reduction, and enhanced binarization), then using Hough circle detection to locate the dial, finally finding the pointer position through connected components, calculating the angle, and finally obtaining the numerical value. However, this method has the following main problems:

[0005] It lacks versatility; the same algorithm cannot adapt to different pointer instruments.

[0006] It has high requirements for environmental conditions and cannot achieve good results in natural environments;

[0007] It cannot adapt to the recognition of dynamic cameras, therefore it is not suitable for inspection robots or mobile cameras to detect pointer instruments.

[0008] With the continuous development of deep learning, more and more deep learning methods have been applied to pointer instrument reading recognition in recent years. This mainly involves using deep learning network target detection algorithms to find the instrument area, and then using traditional methods or image segmentation methods to locate and calculate the pointer; or directly using two-stage algorithms such as image segmentation algorithms Faster R-CNN, SSD, or YOLO to extract the pointer.

[0009] However, these methods generally suffer from problems such as complex network structures, slow detection speeds, and difficulty in deployment at the edge. Summary of the Invention

[0010] To address the aforementioned issues, this invention provides a multi-task-based substation multi-pointer instrument identification method and device, which solves problems such as complex network structure, slow detection speed, and difficulty in deployment at the edge in the prior art.

[0011] A multi-task-based substation multi-pointer instrument identification method, the identification method comprising:

[0012] The acquired images are preprocessed to enhance them, resulting in preprocessed images.

[0013] Based on the preprocessed image, a multi-task instrument recognition network is used to identify key points of the dial, center, and pointer.

[0014] The reading is calculated based on key points on the dial, center, and pointer, as well as the range set in the instrument parameters.

[0015] Furthermore, the preprocessing of the acquired images includes: performing image enhancement, noise reduction, and dehazing operations on the acquired images.

[0016] Furthermore, the image enhancement performed on the acquired images specifically includes:

[0017] A multi-scale fusion algorithm based on the Retinex algorithm is used to decompose the acquired image, obtain the illumination intensity, perform gamma correction on the illumination, perform bilateral filtering to suppress noise, introduce Laplacian pyramid to enhance the details of the image, and finally linearly weight the result with the reflection image.

[0018] Furthermore, the key points of the dial, center, and pointer identified by the multi-tasking instrument recognition network include:

[0019] The designed multi-task instrument recognition network is used to detect and segment the dial, center, and pointer. Based on the segmented center mask image and pointer mask image, the connected regions are calculated. Based on the different aspect ratios and areas of the center and pointer, it is further determined whether the center and pointer have been correctly extracted. At the same time, the coordinates and centroids of the dial, instrument center, and pointer are obtained.

[0020] Furthermore, the key points for identifying the dial, center, and pointer using a multi-tasking instrument recognition network specifically include:

[0021] The connected regions are calculated based on the circle center mask image and pointer mask image segmented by the MTMR-NET network. Based on the different aspect ratios and areas of the circle center and pointer, it is further determined whether the circle center and pointer have been correctly extracted. At the same time, the center point coordinates of the dial, instrument circle center and pointer are obtained.

[0022] Res2Net-APP is used as the backbone network of the Feature Pyramid Network (FPN) of the MTMR-NET network to extract multi-scale feature maps of the instrument images. At the same time, the APP (Adaptive Average Pooling Networks) attention module is added to Res2Net-APP.

[0023] The loss function of the MTMR-NET network consists of three parts: object detection loss (bounding box loss, object loss, and class loss), semantic segmentation loss, and overall loss;

[0024] The formula for the target detection loss is as follows:

[0025]

[0026] Among them, L class and L obj Using Focal Loss, L box CIOU Loss is used;

[0027] The semantic segmentation loss uses cross-entropy loss;

[0028] The formula for the total loss is as follows:

[0029]

[0030] Where α1, α2, α3, γ1, and γ2 are the weights of each part, and the parameters are adjusted during training to optimize the results.

[0031] Furthermore, the method of using a multi-tasking instrument recognition network to identify key points of the dial, center, and pointers also includes:

[0032] The correction of key points specifically includes:

[0033] First, the dashboard, center, and pointer are corrected based on the key points of the segmentation or by calculating the transformation matrix. For fixed cameras, parameter setting is used for correction. Then, the spatial coordinate relationship between the dashboard, center, and pointer determines which part of the instrument the center and pointer belong to.

[0034] Furthermore, the calculation of the reading based on key points of the dial, center, and pointer, as well as the range set in the instrument parameters, specifically includes:

[0035] Linear fitting is performed on the key points of the pointer to calculate the current pointer position, and then the reading is calculated according to the range set in the instrument parameters.

[0036] Furthermore, the step of calculating the reading based on the range set in the instrument parameters specifically includes:

[0037] For instruments with uniform scales, the angle α formed by the straight line fitted to the current pointer position and the straight line where the instrument zero value is located is calculated by combining the detected center of the instrument pointer. The instrument range is b.

[0038] The reading is determined using the angle a*b / 360.

[0039] Furthermore, the step of calculating the reading based on the range set in the instrument parameters specifically includes:

[0040] For instruments with uneven scales, the reading is determined by looking up a table.

[0041] Furthermore, after calculating the reading based on the range set in the instrument parameters, the process further includes:

[0042] The system calculates the error between the calculated instrument reading and the two previous readings stored in memory. If the error is less than a specified threshold, the reading is considered correct. If the readings are inconsistent, monitoring continues, thereby reducing false alarms in the system.

[0043] A multi-task-based substation multi-pointer instrument identification device includes: a preprocessing unit, an identification unit, and a reading unit;

[0044] The preprocessing unit is used to preprocess the acquired image to enhance the image and obtain a preprocessed image;

[0045] The recognition unit is used to identify key points of the dial, center point, center of circle and pointer based on the pre-processed image using a designed multi-task instrument recognition network.

[0046] The reading unit is used to calculate the reading based on key points such as the dial, center point, center of the circle, and pointer, as well as the range set in the instrument parameters.

[0047] This invention proposes a high-efficiency multi-task instrument identification network that can simultaneously handle three key tasks: target detection, instrument centering, and pointer segmentation of substation instruments. This significantly improves the detection accuracy of instrument pointers while increasing the detection speed.

[0048] The deep learning model proposed in this invention uses Res2Net-APP as the backbone network. Users can choose network models of different depths according to the specific hardware configuration. At the same time, the trained model can be converted into ONNX format, which effectively solves the problem of deployment on multiple platforms.

[0049] The multi-task instrument recognition network proposed in this invention can effectively extract the center of the circle and the pointer in various complex scenarios. Combined with the spatial position relationship of the dial obtained by target detection, it can accurately and effectively identify multiple instruments and pointers.

[0050] The algorithm proposed in this invention has higher universality, enabling the identification of various instruments and pointers, effectively solving the problem of requiring a separate identification algorithm for each instrument. The dataset for this invention is entirely derived from actual substations. Even in outdoor scenes and with poor image quality, it accurately identifies single and double needle instruments (with and without standardized scales), including SF6 instruments, switchgear instruments, discharge counters, main transformer temperature instruments, current transformers, voltage transformers, and surge arresters, in substations, achieving an accuracy rate close to 100%. Furthermore, the algorithm is flexible in configuration, easy to set up, and reliable in operation.

[0051] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures pointed out in the description and the drawings. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a general block diagram of the identification method according to an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of the identification model in an embodiment of the present invention;

[0055] Figure 3 This is a flowchart illustrating the specific implementation of the identification method in this invention.

[0056] Figure 4 This is a diagram of the MTMR-NET network structure according to an embodiment of the present invention;

[0057] Figure 5 This is a structural diagram of the Res2Net-APP module according to an embodiment of the present invention;

[0058] Figure 6 This is a structural diagram of the APP module according to an embodiment of the present invention;

[0059] Figure 7 This is a schematic diagram of network segmentation results according to an embodiment of the present invention;

[0060] Figure 8 This is a schematic diagram of coordinate transformation in an embodiment of the present invention;

[0061] Figure 9 This is a schematic diagram of the dial tilt in an embodiment of the present invention;

[0062] Figure 10 This is a schematic diagram of dial correction according to an embodiment of the present invention;

[0063] Figure 11 This is a schematic diagram of the instrument segmentation result according to an embodiment of the present invention;

[0064] Figure 12 This is a schematic diagram illustrating angle calculation in an embodiment of the present invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] In recent years, an increasing number of deep learning methods have been applied to pointer instrument reading recognition. These methods mainly employ deep learning networks to extract the instrument readings, followed by traditional methods to extract the pointer, or a two-stage algorithm that uses object detection algorithms to detect the instrument and then image segmentation algorithms to extract the pointer. However, these methods generally suffer from problems such as complex network structures, slow detection speeds, and difficulty in deployment at edge computing environments.

[0067] For example, Beijing Fuwei Image Technology Co., Ltd. has developed a pointer instrument recognition algorithm that can identify some instruments. Chengdu Nengtuo Technology Co., Ltd. provides an instrument recognition algorithm library that can identify some instruments. Baidu provides a complete deep learning framework (PaddlePaddle), which includes industrial application solutions for instrument pointers and can identify some instruments.

[0068] Currently, some scholars at home and abroad have conducted research on the recognition algorithm of instrument pointers and published many academic papers, but most of them focus on the design and improvement of algorithms for a single phenotype.

[0069] To address this, the present invention proposes a multi-task-based substation multi-pointer instrument identification method and apparatus, including a multi-task-based substation multi-pointer instrument identification method and a multi-task-based substation multi-pointer instrument identification apparatus.

[0070] This invention proposes a highly efficient multi-task instrument recognition network that can simultaneously handle three key tasks: target detection, instrument center segmentation, and pointer segmentation for substation instruments. This significantly improves both detection speed and pointer accuracy. The proposed multi-task instrument recognition network can effectively extract center segments and pointers in various complex scenarios, enabling the recognition of multiple instruments and pointer types.

[0071] Firstly, such as Figure 1 As shown, this invention proposes a multi-task-based substation multi-pointer instrument identification method, the identification method comprising:

[0072] The acquired images are preprocessed to enhance them, resulting in preprocessed images.

[0073] Based on the preprocessed image, a multi-task instrument recognition network is used to identify key points of the dial, center, and pointer.

[0074] The reading is calculated based on key points on the dial, center, and pointer, as well as the range set in the instrument parameters.

[0075] In this embodiment, the preprocessing of the acquired image includes: performing image enhancement, noise reduction, and dehazing operations on the acquired image.

[0076] In this embodiment, the image enhancement of the acquired image specifically includes:

[0077] A multi-scale fusion algorithm based on the Retinex algorithm is used to decompose the acquired image, obtain the illumination intensity, perform gamma correction on the illumination, perform bilateral filtering to suppress noise, introduce Laplacian pyramid to enhance the details of the image, and finally linearly weight the result with the reflection image.

[0078] In this embodiment, the key points of the dial, center, and pointer identified by the multi-task instrument recognition network include:

[0079] The designed multi-task instrument recognition network is used to detect and segment the dial, center, and pointer. Based on the segmented center mask image and pointer mask image, the connected regions are calculated. Based on the different aspect ratios and areas of the center and pointer, it is further determined whether the center and pointer have been correctly extracted. At the same time, the coordinates and centroids of the dial, instrument center, and pointer are obtained.

[0080] In this embodiment, the key points of the dial, center, and pointer identified by the multi-task instrument recognition network specifically include:

[0081] The connected regions are calculated based on the circle center mask image and pointer mask image segmented by the MTMR-NET network. Based on the different aspect ratios and areas of the circle center and pointer, it is further determined whether the circle center and pointer have been correctly extracted. At the same time, the center point coordinates of the dial, instrument circle center and pointer are obtained.

[0082] Res2Net-APP is used as the backbone network of the Feature Pyramid Network (FPN) of the MTMR-NET network to extract multi-scale feature maps of the instrument images. At the same time, the APP (Adaptive Average Pooling Networks) attention module is added to Res2Net-APP.

[0083] The loss function of the MTMR-NET network consists of three parts: object detection loss (bounding box loss, object loss, and class loss), semantic segmentation loss, and overall loss;

[0084] The formula for the target detection loss is as follows:

[0085]

[0086] Among them, Lclass and Lobj use Focal Loss, while Lbox uses CIOU Loss;

[0087] The semantic segmentation loss uses cross-entropy loss;

[0088] The formula for the total loss is as follows:

[0089]

[0090] Where α and γ are the weights of each part, and the parameters are adjusted during training to optimize the results.

[0091] In this embodiment, the key points of the dial, center, and pointer are identified using a multi-tasking instrument recognition network, followed by:

[0092] The correction of key points specifically includes:

[0093] First, the dashboard, center, and pointer are corrected based on the key points of the segmentation or by calculating the transformation matrix. For fixed cameras, parameter setting is used for correction. Then, the spatial coordinate relationship between the dashboard, center, and pointer determines which part of the instrument the center and pointer belong to.

[0094] In this embodiment, the calculation of the reading based on key points of the dial, center, and pointer, as well as the range set in the instrument parameters, specifically includes:

[0095] Linear fitting is performed on the key points of the pointer to calculate the current pointer position, and then the reading is calculated according to the range set in the instrument parameters.

[0096] In practice, the center point of the detected instrument pointer is the center of the circle. The straight line fitted by the current pointer position forms an angle α with the straight line where the instrument zero value is located, and the instrument reading is calculated based on this angle.

[0097] In this embodiment, calculating the reading based on the range set in the instrument parameters specifically includes:

[0098] For instruments with uniform scales, the angle α formed by the straight line fitted to the current pointer position and the straight line where the instrument zero value is located is calculated by combining the detected center of the instrument pointer. The instrument range is b.

[0099] The reading is determined using the angle a*b / 360.

[0100] In this embodiment, calculating the reading based on the range set in the instrument parameters specifically includes:

[0101] For instruments with uneven scales, the reading is determined by looking up a table.

[0102] In this embodiment, after calculating the reading based on the range set in the instrument parameters, the method further includes:

[0103] The system calculates the error between the calculated instrument reading and the two previous readings stored in memory. If the error is less than a specified threshold, the reading is considered correct. If the readings are inconsistent, monitoring continues, thereby reducing false alarms in the system.

[0104] In practice, the system reduces false alarms and increases accuracy by comparing the data with previous records.

[0105] Secondly, such as Figure 2 As shown, the present invention proposes a multi-task-based substation multi-pointer instrument identification device, comprising: a preprocessing unit, an identification unit, and a reading unit;

[0106] The preprocessing unit is used to preprocess the acquired image to enhance the image and obtain a preprocessed image;

[0107] The recognition unit is used to identify key points of the dial, center point, center of circle and pointer based on the pre-processed image using a designed multi-task instrument recognition network.

[0108] The reading unit is used to calculate the reading based on key points such as the dial, center point, center of the circle, and pointer, as well as the range set in the instrument parameters.

[0109] In specific implementation, the implementation process of the multi-task-based substation multi-pointer instrument identification device and the multi-task-based substation multi-pointer instrument identification method of the present invention correspond one-to-one, and will not be elaborated here.

[0110] To enable those skilled in the art to better understand the present invention, the principles of the present invention are explained below in conjunction with the accompanying drawings:

[0111] With the continuous advancement of deep learning, many pointer instruments now employ deep learning methods for recognition: first, object detection algorithms are used to identify the instrument area, and then traditional methods or image segmentation methods are used to locate and calculate the pointer; or networks such as Faster R-CNN, SSD, and YOLO are directly used to extract the pointer. These two methods offer some improvement over traditional methods, but in practical substation applications, the accuracy remains unsatisfactory. If a two-stage method is adopted—first using deep learning to detect the instrument and then segmenting the center and pointer—the actual recognition effect is improved, but the speed cannot meet real-time requirements.

[0112] This invention is based on deep learning methods and designs a multi-task learning network that includes target detection and segmentation. This network can perform target detection and segmentation simultaneously, and has high accuracy, fast running speed, and is easy to deploy in instrument recognition.

[0113] The existing technical solutions, particularly the deep learning-based instrument detection methods, mainly employ the following approaches:

[0114] (1) Use FCN network to extract the dial, then perform dial calibration, and finally use center projection method to detect the pointer;

[0115] (2) Use one-stage (SSD, YOLO) or two-stage (Faster R-CNN) methods or improved algorithms to detect dashboards and pointers.

[0116] (3) Using the image segmentation (Mask R-CNN, Unet) model, by adding the direction prediction branch of the object detection box to Mask R-CNN, the algorithm of instrument detection and arbitrary direction instrument scale value detection is used to detect and identify instruments, center, and pointer.

[0117] like Figure 3 As shown, the main technical solution of the present invention includes:

[0118] (1) Image acquisition and preprocessing: Most of the instruments in the substation are located outdoors. During the image acquisition process, due to the influence of light, angle and weather, the acquired images are prone to blurring. Therefore, the acquired images are first preprocessed using algorithms such as image enhancement, noise reduction and defogging.

[0119] (2) Instrument detection and segmentation algorithm: The multi-task instrument recognition network is designed to detect and segment the dial, center, and pointer. Based on the segmented center mask image and pointer mask image, the connected region is calculated and the center and pointer are further judged according to their different aspect ratios and areas. At the same time, the coordinates and centroids of the dial, instrument center and pointer are obtained.

[0120] (3) Algorithm for instrument calibration, pointer extraction and verification: First, the instrument panel, center and pointer are calibrated based on the key points of the segmentation or by calculating the transformation matrix. For fixed cameras, parameter setting can be used for calibration. The spatial coordinate relationship between the instrument panel, center and pointer is used to determine which instrument the center and pointer belong to.

[0121] (4) Pointer Angle and Value Calculation: Based on the pointer obtained in the above steps, further linear fitting is performed, and the angle is calculated by combining the center point of the instrument pointer detected in the above steps. Then, the reading is calculated according to the range set in the instrument parameters. For instruments with uneven scales, the table lookup method is used to determine the reading. The calculated instrument reading and the previous two readings stored in memory are compared to calculate the error. If the error is less than the specified threshold, it indicates that the reading is correct. If the readings are inconsistent, monitoring continues to reduce false alarms in the system.

[0122] The specific steps of this invention include:

[0123] 1. Image acquisition and image enhancement

[0124] Factors such as the natural conditions of the target area at the acquisition point and vibrations of the acquisition equipment can negatively impact subsequent image detection. Therefore, image enhancement processing is performed before instrument recognition. This invention employs a multi-scale fusion algorithm based on the Retinex algorithm for image enhancement.

[0125] Based on the principles of the human visual system, foggy images Decomposed into incident light components and reflected light components The main purpose of this algorithm is to decompose foggy and low-light images, eliminate or reduce the incident light component, thereby enhancing the image.

[0126] The single-scale Retinex algorithm (SSR) is prone to problems such as image color distortion and poor contrast. This algorithm uses multi-scale (MSR) to decompose the image, obtain the illumination intensity, perform gamma correction on the illumination, perform bilateral filtering to suppress noise, introduce Laplacian pyramid to enhance the details of the image, and finally linearly weight the result with the reflection image.

[0127] Retinex can transform raw, foggy images Decompose it, and then extract from it Decompose to eliminate the effects of uneven lighting.

[0128]

[0129] in (x,y) For the pixels of the image, I(x,y) The image to be processed. This is the reflection component of the object's texture; N The number of scales selected can be set in the instrument recognition settings; the default value is N=3. w i The weight assigned to the corresponding scale; , is the Gaussian filter function, and * represents the convolution operation.

[0130]

[0131] Furthermore, the Gaussian function satisfies the normalization principle:

[0132]

[0133] σ is the scaling parameter of the Gaussian function, and its value directly determines the effect of image enhancement. σ = 15, 80, 250.

[0134] 2. Instrument, center, and pointer detection and segmentation

[0135] like Figure 4 As shown, the circle center mask image and pointer mask image segmented by the MTMR-NET network are used to calculate the connected regions and further determine whether the circle center and pointer have been correctly extracted based on their different aspect ratios and areas. At the same time, the center point coordinates of the dial, instrument center, and pointer are obtained.

[0136] (1) Backbone: This invention uses Res2Net-APP as the backbone network of the Feature Pyramid Network (FPN) to extract multi-scale feature maps of the instrument image. Figure 4 (C3~C5), Res2Net-APP structure as follows Figure 5 As shown, the module uses 3×3 group convolutional layers instead of the 3×3 convolutional layers in the ResNet module. At the same time, an APP (Adaptive Average Pooling Networks) attention module is added to the module, which effectively improves the performance of the network in instrumentation, center (key point) detection, pointer detection and segmentation.

[0137] The Res2Net-APP module divides the feature map F(H×W×d) output from a 1×1 convolutional layer into n groups according to channels. Each group has a shape of H×W×d / n, where i∈{1, 2, ..., n}. The first group of features, X1, directly outputs Y1=X1. X2 passes through a 3x3 convolutional layer to output Y2=K(X2). X3 and Y2 are concatenated by a residual and passed through another convolutional layer to output Y3=K(X3+Y2). The final output, Yn, is calculated using the following formula:

[0138]

[0139] Where K(•) represents the convolution operation.

[0140] The structure of the APP module is as follows Figure 6 As shown, the output y of the Res2Net module is fed into the APP module. This module first uses multiple (4) adaptive average pooling methods to compress the features y (w×h×b×c) into y1', y2', y3', and y4', then concatenates them, and uses two fully connected layers to fit the correlation between channels. Finally, a sigmoid activation function is used for normalization. (Channels) The calculation formula is:

[0141]

[0142] Where FC represents a fully connected layer; δ represents the ReLU function; and σ represents the Sigmoid function. y'=concat(y1', y2', y3', y4') (concatenation operation) outputs f' = fc.y.

[0143] (2) Neck Module: This module adopts a multi-level (FPN) feature information fusion method. This module receives three feature maps C3, C4, and C5. After feature fusion, it outputs three feature maps P3, P4, and P5. The number of channels is 256 for each feature map. The small feature map is used to detect large objects, and the large feature map is used to detect small objects.

[0144] (3) Head Prediction Module: This network consists of an object detection head and an image segmentation head. The object detection head has three branches (inputs P3-P5), generating three anchors for each pixel of each feature map in ratios of 1:1, 1:2, and 2:1. The image segmentation head takes P3 as input, which is then convolved and upsampled to restore the image to (W, H, C) to generate a mask image. The actual segmentation results of this network are as follows: Figure 7 As shown.

[0145] (4) Loss function: This network function consists of three parts: target detection loss (Bounding box loss) Target loss and category loss (Semantic segmentation loss, overall loss)

[0146] The formula for the target detection loss is as follows:

[0147]

[0148] Among them, Lclass and Lobj use FocalLoss, while Lbox uses CIOU Loss.

[0149] Cross-entropy loss is used for semantic segmentation.

[0150] Total loss The formula is as follows:

[0151]

[0152] Where α1, α2, α3, γ1, and γ2 are the weights of each part, and the parameters can be adjusted during training to achieve the optimal result.

[0153] 3. Instrument calibration

[0154] Due to the influence of shooting angle and improper instrument installation, the instrument may exhibit phenomena such as tilting, twisting, and rotation. This invention uses the following algorithm to correct the detected instrument.

[0155] (1) Image correction principle

[0156] Define two coordinate systems P1(x,y,z) and P2(X,Y,Z). Points in coordinate system P1 are connected by a matrix. (mij is an element of the transformation matrix) Transform to the P2 coordinate system.

[0157]

[0158]

[0159]

[0160]

[0161] x,y are the image coordinates before correction, and x',y' are the image coordinates after correction.

[0162] Therefore, the transformation matrix can be calculated using the coordinates of four pairs of points in the two view planes, and then the instrument can be corrected using the transformation matrix.

[0163] (2) Method for obtaining the coordinates of four pairs of points

[0164] Using the dashboard image segmented through the above steps (Metermask), the minimum circumcircle parameter of the dashboard image can be obtained through the corresponding OpenCV function. Cx, Cy, r ] ( Cx, Cy: The coordinates of the center point of the circumcircle; r: radius of the circumcircle) and the smallest circumcircle ellipse [Ex, Ey, 2a, 2b, θ] (Ex, Ey: coordinates of the circumcircle ellipse; 2a, 2b: major and minor axes of the ellipse; θ: angle of inclination of the ellipse), coordinate transformations are as follows: Figure 8 As shown.

[0165] The four points [E1-E4] can be solved using the following system of equations.

[0166]

[0167]

[0168]

[0169] i=1,2,3,4

[0170]

[0171] Where: Calculate the major axis k = tanθ, f = a; When calculating the intersection of the minor axis k = -1 / tanθ, f = b .

[0172] The coordinates of [C1-C4] are: ]

[0173] The coordinates of [E1-E4] are: ] i=1,2,3,4

[0174] The transformation matrix can be calculated based on the obtained 4 pairs (8 points), and the instrument can be calibrated accordingly.

[0175] Currently, most substations use fixed cameras to take pictures (or the camera tracks to a designated location and then takes pictures). The changes in the images are not significant. Therefore, the correction data points can be calibrated manually and saved to the parameter file, or key points can be extracted via the network and then the transformation matrix can be calculated.

[0176] The specific algorithm is as follows:

[0177] (1) Read parameters: transMode / / : transMode=0 is for calibration parameter correction; transMode=1 is for key point correction

[0178] (2) If it is calibration parameter correction:

[0179] Read the calibrated matrix parameters

[0180] otherwise:

[0181] Get Metermask

[0182] Calculate the minimum circumcircle parameter and minimum circumcircle ellipse of the dashboard image.

[0183] Calculate [C1-C4], [E1-E4]

[0184] (3) Calculate the transformation matrix

[0185] (4) Traverse the detected instrument frames

[0186] Adjust image size to template size

[0187] Image correction.

[0188] Image before correction of tilt Figure 9 As shown, the corrected image result is as follows: Figure 10 As shown.

[0189] 4. Pointer positioning and preliminary angle calculation

[0190] The center and pointer mask images obtained after the above image segmentation and correction steps are as follows: Figure 11 As shown (this example divides the instrument into three categories: dial, center, and pointer).

[0191] As can be seen from the image, the Mask image obtained from image segmentation is noisy, so further processing is required.

[0192] The algorithm for calculating the pointer angle based on the obtained center and pointer Mask is as follows:

[0193] Input: pointer_num / / Number of instrument pointers: 1 for a single pointer; 2 for two pointers

[0194] pointer_threshold / / Minimum pointer area

[0195] dot__threshold / / Minimum area of ​​the center of the circle

[0196] Output: Angle array of the instrument

[0197] (1) Calculate the connected components for the center Mask image.

[0198] (2) Traverse all connected regions obtained by the center Mask image.

[0199] 1) Calculate the area of ​​a connected region: area_dot;

[0200] 2) Calculate the center point, width, and height of the connected region: x_dot, y_dot, w_dot, h_dot

[0201] 3) IF (area_dot>dot_threshold) and (w_dot / h_dot)>0.6 / / If the area area_dot is greater than the specified threshold dot_threshold, and the ratios of the width w_dot and height h_dot are close, it indicates that it is the center of the instrument.

[0202] find_dot=1 / / Find the center of the circle

[0203] Exit the loop

[0204] ELSE

[0205] Jump (2) to find the next connected component

[0206] (3) IF(find_dot==0) / / The center of the circle was not found

[0207] x_dot=W / S,y_dot=h / 2 / / Use the center of the instrument as the center of the circle

[0208] (4) Calculate the connected components for the pointer mask image.

[0209] (5) Traverse all connected regions obtained from the pointer mask image.

[0210] 1) Calculate the area of ​​a connected region: area_pointer;

[0211] 2) Obtain the center point x_pointer, y_pointer, angle_pointer, width w_pointer, and height h_pointer of the smallest bounding rectangle;

[0212] 3) IF (area_dot>pointer_threshold) and (w_pointer / h_pointer)>2 / / If the area is greater than the specified threshold and the width-to-height ratio is greater than 2, it indicates that it is a pointer of an instrument.

[0213] find_pointer++ / / Find the pointer

[0214] / / Put the data into the array data_pointer:

[0215] data_ pointer[find_ pointer].angle_ pointer=angle_ pointer

[0216] data_ pointer[find_ pointer].area_pointer= area_pointer

[0217] ELSE

[0218] Jump (5) to find the next connected component

[0219] (6) IF (data_pointer!=null && data_pointer.size()==pointer_num) / / This means that a pointer has been found and the number of pointers is the same as the number of pointers on the instrument.

[0220] IF (find_pointer == 1) / / If it is a single pointer

[0221] angle[1]=computer_angle(data_ pointer [1] )

[0222] IF (find_pointer == 2) / / If it's a two-pointer problem

[0223] angle[2] = computer_angle (data_pointer[2]) is the angle with the largest area of ​​the Unicom region.

[0224] |

[0225] Returns: an array of angles.

[0226] 5. Calculation of the included angle

[0227] like Figure 12 As shown, to ensure the accuracy of the included angle, this invention will perform least-squares linear fitting on the connected regions of the pointer in the pointer Mask image obtained in the above steps, thereby obtaining the slope K and angle q of the pointer. Due to the periodicity of the instrument angle (e.g., the slope is 0 when the pointer angle is 0° and 180°), it is necessary to further determine the quadrant in which the pointer is located. The specific algorithm is as follows:

[0228] (1) The center coordinates x_dot and y_dot can be obtained from the above steps;

[0229] (2) The center coordinates x_pointer and y_pointer of the smallest bounding rectangle of the pointer can be obtained according to the above steps;

[0230] (3) Calculate the distance between the x and y coordinates of two points:

[0231] dx = x_pointer - x_dot;

[0232] dy = y_pointer - y_dot;

[0233] (4) Calculate the angle:

[0234] The rules for determining angles are shown in Table 1:

[0235] Table 1 Angle Judgment Rules

[0236]

[0237] dx: The distance between the x-coordinate of the center of the smallest bounding rectangle of the pointer and the x-coordinate of the center of the circle (see...). Figure 12 ); dy: is the distance between the center coordinate of the smallest bounding rectangle of the pointer and the center coordinate of the circle (see...). Figure 12 );

[0238] K: The slope of the obtained pointer;

[0239] atan: Arctangent function.

[0240] For example, when dx > 0 and dy = 0, that is, when the pointer coincides with the positive X-axis, the angle is 0°;

[0241] When dx=0 and dy>0, that is, when the pointer coincides with the positive Y-axis, the angle is 90°.

[0242] 6. Calculate the measured values

[0243] (1) For instruments with uniform scale:

[0244] unit angle value of pointer instrument for:

[0245]

[0246] Actual measured value:

[0247] Where: v max v is the maximum value of the instrument's measuring range; min θ is the minimum value of the instrument's measuring range. max The pointer angle at maximum range; θ min The pointer angle when the range is at its smallest.

[0248] Note: The angle is defined as 0° in the negative Y-axis direction.

[0249] (2) Instruments with uneven scale

[0250] 1) Read instrument setting parameters

[0251] 2) The instrument value is obtained by looking up the table based on the angle.

[0252] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-task based substation multi-needle instrument identification method, characterized in that, The recognition method comprises: Preprocessing the collected image to enhance the image to obtain a pretreated image; According to the pretreated image, a multi-task instrument recognition network based on an MTMR-NET network is used to recognize the key points of the dial, the center and the pointer; According to the key points of the dial, the center and the pointer, and the range set in the instrument parameter, a reading is calculated; The multi-task instrument recognition network comprises: The designed multi-task instrument recognition network is used to realize the detection and segmentation of the dial, the center and the pointer, the connected regions are calculated according to the segmented center mask image and the pointer mask image, and according to the different length-width ratios and areas of the center and the pointer, it is further judged whether the center and the pointer are correctly extracted, and the coordinates and the center of mass of the dial, the instrument center and the pointer are obtained; The multi-task instrument recognition network comprises: According to the center mask image and the pointer mask image segmented by the MTMR-NET network, the connected regions are calculated, and according to the different length-width ratios and areas of the center and the pointer, it is further judged whether the center and the pointer are correctly extracted, and the center point coordinates of the dial, the instrument center and the pointer are obtained; Res2Net-APP is used as the backbone network of the feature pyramid network of the MTMR-NET network to extract multi-scale feature maps of the instrument image, and an APP attention module is added in the Res2Net-APP; The loss function of the MTMR-NET network includes three parts: target detection loss , semantic segmentation loss and overall loss; bounding box loss , target loss and class loss The formula of the target detection loss is as follows: wherein L class and L obj Focal Loss is adopted, L box CIOU Loss is adopted; The semantic segmentation loss adopts a cross-entropy loss. Total loss The formula is as follows: where: a1, a2, a3, g1, g2 are the weights of each part, the parameters are adjusted in training to optimize the results, is the semantic segmentation loss.

2. The multi-pointer instrument recognition method for a substation based on multiple tasks according to claim 1, wherein the preprocessing of the collected image comprises image enhancement, denoising and defogging operations on the collected image.

3. The multi-pointer instrument recognition method for a substation based on multiple tasks according to claim 2, wherein the image enhancement on the collected image comprises: A multi-scale fusion algorithm based on the Retinex algorithm is used to decompose the collected image to obtain the light intensity, gamma correction is performed on the light, bilateral filtering is performed to suppress noise, the details of the image are enhanced by introducing a Laplacian pyramid, and finally the result is linearly weighted with the reflection image.

4. The multi-pointer instrument recognition method for a substation based on multiple tasks according to claim 1, wherein after the multi-task instrument recognition network is used to recognize the key points of the dial, the center and the pointer, the method further comprises: The correction of the key points, specifically comprising: First, the dial, the center and the pointer are corrected according to the segmented key points or by calculating a transformation matrix, and for a fixed camera, a parameter correction is used; and according to the spatial coordinate relationship of the dial, the center and the pointer, it is judged which block the center and the pointer belong to.

5. The multi-pointer instrument recognition method for a substation based on multiple tasks according to claim 1, wherein the calculation of the reading according to the key points of the dial, the center and the pointer, and the range set in the instrument parameter comprises: ​ ​ ​ ​ The key points of the pointer are straight line fitted, the current pointer position is calculated, and then the reading is calculated according to the range set in the instrument parameter.

6. The multi-task-based substation multi-pointer instrument identification method of claim 5, wherein the reading is calculated according to the range set in the instrument parameter, and the calculation specifically comprises: For the instrument with uniform scale, the angle a is formed between the straight line fitted by the current pointer position and the straight line where the zero value of the instrument is located, and the range of the instrument is b; The reading is determined by a*b / 360.

7. The multi-task-based substation multi-pointer instrument identification method of claim 5, wherein the reading is calculated according to the range set in the instrument parameter, and the calculation specifically comprises: For the instrument with non-uniform scale, the reading is determined by the table lookup method.

8. The multi-task-based substation multi-pointer instrument identification method of any one of claims 1-7, wherein the reading is calculated according to the range set in the instrument parameter, and the calculation further comprises: Error calculation is performed on the calculated instrument reading and the previous two readings saved in the memory, and if the error is less than a specified threshold, it is indicated that the reading is correct, and if the readings are inconsistent, the monitoring is continued, so as to reduce the false positives of the system. It comprises: a preprocessing unit, an identification unit and a reading unit; The preprocessing unit is used for pre-processing the collected image to enhance the image and obtain a pre-processed image; 9. A multi-task based substation multi -pointer meter identification apparatus, characterized by, The identification unit is used for identifying the key points of the dial, the center point, the center and the pointer according to the pre-processed image by using the designed multi-task instrument identification network; The reading unit is used for calculating the reading according to the key points of the dial, the center point, the center and the pointer, and the range set in the instrument parameter; The multi-task instrument identification network is used to identify the key points of the dial, the center and the pointer, and comprises: The multi-task instrument identification network is used to identify the key points of the dial, the center and the pointer, and comprises: According to the different length-width ratio and area of the center and the pointer, it is further judged whether the center and the pointer are correctly extracted, and the coordinates and the center of mass of the dial, the instrument center and the pointer are obtained; The multi-task instrument identification network is used to identify the key points of the dial, the center and the pointer, and comprises: According to the different length-width ratio and area of the center and the pointer, it is further judged whether the center and the pointer are correctly extracted, and the coordinates and the center of mass of the dial, the instrument center and the pointer are obtained; Res2Net-APP is used as the backbone network of the feature pyramid network of the MTMR-NET network to extract multi-scale feature maps of the instrument image, and an APP attention module is added in the Res2Net-APP; The cross-entropy loss is used as the semantic segmentation loss. ​ The loss function of the MTMR-NET network includes three parts: target detection loss , semantic segmentation loss and overall loss; Boundary box loss , target loss and class loss The formula of the target detection loss is as follows: Wherein, L class and L obj Focal Loss is adopted, L box CIOU Loss is adopted; ​ Total loss The formula is as follows: where: a1, a2, a3, g1, g2 are the weights of each part, the parameters are adjusted in training to optimize the results, is the semantic segmentation loss.

Images