An Automatic Recognition Method for Industrial Pointer Instrument Readings Based on Deep Learning

Abstract

The present invention belongs to the technical field of pointer instrument reading recognition, and specifically discloses an automatic recognition method for industrial pointer instrument readings based on deep learning, including the following steps: collecting pointer instrument images using an industrial camera; detecting the instrument panel in the instrument image using a dashboard detection model based on YOLOX-DC; using an image segmentation model based on PM-SwinUnet to segment the pointer, key scales, and scale readings of the instrument panel to obtain a segmented image of the instrument; based on the segmented image of the instrument, using a skeleton thinning and center point extraction algorithm to remove edge pixels, obtaining the position of the pointer, the direction of the pointer, and the position of the scale, and then using an improved angle method to obtain the final reading of the pointer instrument. The present invention solves the problems of existing pointer instrument reading recognition, such as relying on prior information, poor generalization ability, and incomplete segmentation, and realizes the accurate recognition of pointer instrument readings.

Description

Technical Field

[0001] The present invention belongs to the technical field of pointer instrument reading recognition, and particularly relates to an automatic recognition method for industrial pointer instrument readings based on deep learning. Background Art

[0002] As an indispensable basic measuring instrument in industrial production, pointer instruments are widely used in fields such as power systems, manufacturing systems, remote sensing, aerospace, etc. to measure and record key production data. Regular verification and calibration of pointer instruments are crucial for maintaining their optimal performance and quality, ensuring accuracy and credibility, guaranteeing safety and reliability during use, and improving industrial production efficiency. Currently, the verification and calibration of pointer instruments mostly rely on metrology personnel to manually copy and record data in laboratories or industrial sites. On the one hand, it takes a long time, has low efficiency, and the reading accuracy is limited by light and pollution. On the other hand, in some special industrial sites such as substations, the working environment of the instruments is relatively harsh, and the long-term manual recording work also poses challenges to the physical and mental health of metrology personnel. With the development of artificial intelligence, image processing, and machine vision technologies, automatic reading recognition of pointer instruments based on machine vision is gradually replacing manual meter reading, becoming a key way to improve production automation, production efficiency, and quality, promoting the digital transformation of the instrument metrology and detection industry, and driving the process of new industrialization, which is of great significance.

[0003] Although existing automatic recognition methods for pointer readings have been successively proposed, when facing industrial scenarios with fuzzy, uneven illumination, and inclined instrument placement, there are problems such as low detection accuracy and insufficient feature extraction ability for small-scale and long targets, which restrict the generalization and expansion of pointer reading automatic recognition technology in complex industrial fields.

[0004] The automatic recognition method for pointer instrument readings consists of two key technologies: target detection and image segmentation.

[0005] Existing target detection methods for pointer instrument reading recognition can be divided into two categories: rule-based methods and statistics-based methods. Rule-based methods rely on prior information such as dial range, dial perimeter, and pointer features, and manually extracted features. In a complex and changing industrial environment, it is difficult to flexibly migrate and apply this method. Statistics-based methods use deep learning technology to automatically learn the image features of pointer instruments, such as YOLOv5, YOLOX, etc., which can get rid of the dependence on prior knowledge and manual features, but are sensitive to illumination changes and image quality, and have poor generalization ability and robustness in cases of uneven illumination and blurred images.

[0006] The existing image segmentation methods for pointer instrument reading recognition can be divided into two categories: traditional machine vision methods and deep learning methods. The methods based on traditional machine vision use methods such as threshold segmentation, edge detection, and line detection to extract the dial and the pointer, and different parameters need to be manually set for different pointer dials. For the image segmentation methods based on deep learning, manual setting of dial parameters is not required. Based on the convolutional neural network (CNN), different network structures are constructed, such as Mask R-CNN, U-Net, etc. Through autonomous learning of data labels, the segmentation of the dial and the pointer can be achieved for different instruments. However, due to the perception limitations of global features and long-distance features, the pointer segmentation is incomplete, affecting the accuracy and robustness of reading recognition.

[0007] In summary, the existing methods for pointer meter reading recognition have problems such as relying on prior information, poor generalization ability, and incomplete segmentation. When facing complex industrial environments, the accuracy and robustness of detection and segmentation need to be further improved. Summary of the Invention

[0008] The purpose of the present invention is to solve the defects of the existing pointer instrument reading recognition methods, such as relying on prior information, poor generalization ability, and incomplete segmentation, which lead to poor accuracy and robustness of detection and segmentation when facing complex industrial environments. A method for automatically recognizing industrial pointer instrument readings based on deep learning is proposed.

[0009] The technical solution of the present invention is as follows: A method for automatically recognizing industrial pointer instrument readings based on deep learning includes the following steps:

[0010] S1. Use an industrial camera to collect image data of the pointer-type instrument;

[0011] S2. Use a dashboard detection model based on YOLOX-DC to detect the dashboard in the image data of the pointer-type instrument;

[0012] S3. Use an image segmentation model based on PM-SwinUnet to segment the pointer, key scales, and scale readings of the dashboard to obtain a pointer mask image and a scale mask image;

[0013] S4. Based on the pointer mask image and the scale mask image, use a skeleton thinning and center point extraction algorithm to remove edge pixels, and analyze to obtain the position of the pointer, the direction of the pointer, and the position of the scale;

[0014] S5. According to the position of the pointer, the direction of the pointer, and the position of the scale, use an improved angle method to obtain the final reading of the pointer-type instrument, and complete the automatic recognition of the pointer instrument reading.

[0015] The beneficial effects of the present invention are:

[0016] 1. The present invention proposes a YOLOX-DC pointer instrument target detection model to realize pointer dial detection, which can better adapt to the appearance characteristics of circular pointer dials and has more efficient reasoning speed and better robustness when facing complex industrial scenarios.

[0017] 2. The present invention simultaneously segments the pointer, scale lines and scale values through the PM-SwinUnet pointer instrument image segmentation model, achieves better performance in the segmentation of dial elements, and solves the problems of missing small target detection and incomplete long target segmentation.

[0018] Preferably, the YOLOX-DC-based instrument panel detection model in step S2 comprises a backbone network, a neck network and a detection head connected in sequence;

[0019] The backbone network is used to receive the pointer instrument image data, perform feature extraction, and output features of the pointer instrument image data;

[0020] The neck network is used to receive the features of the pointer instrument image data, perform feature fusion, and output fused features;

[0021] The detection head is used to receive the fused features, perform classification and regression according to the fused features, and output the instrument panel in the pointer-type instrument image data.

[0022] Preferably, the backbone network adopts a CSPDarkNet structure, comprising: a Focus module, a first CBS layer, a first CSP layer, a second CBS layer, a second CSP layer, a third CBS layer, an SPP layer and a third CSP layer connected in sequence;

[0023] The output ends of the first CSP layer, the second CSP layer and the third CSP layer are also connected to the neck network;

[0024] The Focus module includes four Slice layers connected in parallel, the output ends of the four Slice layers are all connected to the input end of the first Concat layer; the output end of the first Concat layer is connected to the input end of the fourth CBS layer; the output end of the fourth CBS layer is the output end of the entire Focus module.

[0025] Preferably, the neck network comprises a sixth CBS layer, a first upsampling layer, a second Concat layer, a CBS&CSP layer, a second upsampling layer, a third Concat layer, a fifth CSP layer, a first downsampling layer, a fourth Concat layer, a sixth CSP layer, a second downsampling layer and a Concat&CSP layer connected in sequence;

[0026] Another input end of the second Concat layer is connected to the third CSP layer; another input ends of the third Concat layer are respectively connected to the first CSP layer and the second CSP layer; another output end of the fifth CSP layer is connected to the detection head; another input end of the fourth Concat layer is connected to the CBS&CSP layer; another output end of the sixth CSP layer is connected to the detection head; another input end of the Concat&CSP layer is connected to the sixth CBS layer, and the output end of the Concat&CSP layer is connected to the detection head.

[0027] Preferably, the detection head is a decoupled circular detection head, including: a first decoupled circular detection head, a second decoupled circular detection head, a third decoupled circular detection head, a first Reshape layer, a second Reshape layer and a third Reshape layer;

[0028] The input end of the first decoupled circular detection head is connected to the output end of the fourth CSP layer, and the output end of the first decoupled circular detection head is connected to the input end of the first Reshape layer;

[0029] The input end of the second decoupled circular detection head is connected to the output end of the fifth CSP layer, and the output end of the second decoupled circular detection head is connected to the input end of the second Reshape layer;

[0030] The input end of the third decoupled circular detection head is connected to the output end of the first Concat&CSP layer, and the output end of the third decoupled circular detection head is connected to the input end of the third Reshape layer;

[0031] The output ends of the first Reshape layer, the second Reshape layer and the third Reshape layer together serve as the output end of the entire detection head;

[0032] The detection head uses polar coordinates for the positioning of the pointer-type instrument panel. The calculation formula of the polar coordinates is:

[0033]

[0034] where represents the polar radius of the pointer-type instrument panel in the polar coordinate system, represents the polar angle of the pointer-type instrument panel in the polar coordinate system, represents the abscissa of the target position in the Cartesian coordinate system, represents the ordinate of the target position in the Cartesian coordinate system; arctan is the arctangent function.

[0035] The beneficial effects of the above preferred solution are:

[0036] The pointer dial detection is realized by constructing a decoupled circular detection head. Compared with the traditional rectangular detection frame, the method provided by the present invention can better adapt to the shape characteristics of the circular pointer dial. At the same time, since the method uses polar coordinates to locate the detection target, only two parameters ( and ) are required for positioning, which is simpler and more efficient than the four parameters of the rectangular frame, making the method have a more efficient inference speed and better robustness when facing complex industrial scenarios.

[0037] Preferably, the analysis of obtaining the position and direction of the pointer in step S4 specifically includes the following sub-steps:

[0038] S41. Using the center point extraction algorithm based on the mask graph, the center point of the pointer-type instrument image is extracted according to the scale mask image;

[0039] S42. According to the skeleton thinning algorithm, the pointer skeleton is extracted from the pointer mask image, and the pointer mask is thinned into a unit pixel-level straight line;

[0040] S43. Using the Hough transform to detect the straight lines in the pointer mask image. If a straight line is detected, traverse each straight line, extract the two endpoint coordinates of each straight line, and draw the straight line on the original image, and execute step S44; if no straight line is detected, output "the pointer cannot be detected" and end the process;

[0041] S44. Store the two endpoint coordinates in endpoint list 1 and endpoint list 2 respectively, and calculate the Euclidean distances from the two endpoints to the center point of the pointer-type instrument image respectively;

[0042] S45. Determine the endpoint closer to the center point of the pointer-type instrument image according to the Euclidean distance, obtain the starting point and ending point of the pointer line, and determine the position and direction of the pointer according to the starting point and ending point of the pointer line.

[0043] Preferably, the step S41 specifically includes the following steps:

[0044] S411. Perform connected component analysis on the scale mask image to identify the connected regions in the scale mask image;

[0045] S412. Traverse each connected region, filter out the connected regions with the number of pixels higher than the preset threshold, and generate a binary filtered image;

[0046] S413. Use the contour detection algorithm to extract the scale contours from the binary filtered image, and store the scale contours in the contour list;

[0047] S414. Traverse each scale contour in the contour list to obtain the minimum bounding rectangle of each contour;

[0048] S415. Extract the center point coordinates of each minimum bounding rectangle, convert them to integer type, and store them in the "center point coordinate list" to complete the extraction of the center point of the pointer-type instrument image.

[0049] Preferably, the calculation formula for the Euclidean distance in step S44 is:

[0050]

[0051] where, represents the Euclidean distance, represents the abscissa of the center point of the pointer-type instrument image, represents the abscissa of the endpoint of the line, represents the ordinate of the center point of the pointer-type instrument image, represents the ordinate of the endpoint of the line.

[0052] Preferably, the improved angle method described in step S5 is specifically:

[0053] A1. Establish a Cartesian coordinate system with the center point O of the instrument panel as the origin;

[0054] A2. In the Cartesian coordinate system, calculate the angle O between the ray OA from point OP to the 0 scale and the ray where the pointer is located, and the angle O between the ray OB from point OP to the second scale and the ray where the pointer is located;

[0055] The calculation formula for the angle is:

[0056]

[0057] where, , and respectively represent the vectors corresponding to the rays OA , ray OP and ray OB , represents the length of the vector; arccos represents the inverse cosine;

[0058] A3. Convert the angle and the angle from radians to degrees to obtain the reading of the pointer-type instrument , and its calculation formula is:

[0059]

[0060] Among them, represents the reading of the second scale.

[0061] The beneficial effects of the above preferred solution are:

[0062] This preferred solution proposes an improved angle method, which makes full use of the geometric relationship between the pointer and the scale line. Compared with the traditional angle method, the improved angle method minimizes the calculation load and error, can better adapt to the segmentation results of different image segmentation networks, and shows high robustness and accuracy.

[0063] Preferably, the reading of the second scale is calculated by a scale value recognition module; the scale value recognition module consists of multiple convolutional layers and a connection temporal classification layer, and realizes scale value recognition through parallel linear prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 Shown is a flowchart of an automatic recognition method for industrial pointer instrument readings based on deep learning.

[0065] Figure 2 Shown is a schematic structural diagram of the dashboard detection model based on YOLOX-DC provided in Embodiment 1 of the present invention.

[0066] Figure 3 Shown is a schematic structural diagram of the image segmentation model of PM-SwinUnet provided in Embodiment 1 of the present invention.

[0067] Figure 4 Shown is a schematic diagram of the recognition result of a low-quality instrument image provided in Embodiment 2 of the present invention.

[0068] Figure 5 Shown is an experimental result graph of the experiment conducted on the "Paddle Meter" dataset and the "MeterChallenge (MC1296)" dataset provided in Embodiment 2 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0069] Now, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be understood that the embodiments shown and described in the drawings are merely exemplary, intended to explain the principles and spirit of the present invention, and not to limit the scope of the present invention.

[0070] Embodiment 1, as Figure 1As shown in the figure, an automatic recognition method for industrial pointer instrument readings based on deep learning includes the following steps:

[0071] S1. Use an industrial camera to collect image data of the pointer-type instrument;

[0072] S2. Use the dashboard detection model based on YOLOX-DC to detect the dashboard in the image data of the pointer-type instrument;

[0073] S3. Use the image segmentation model based on PM-SwinUnet to segment the pointer, key scales, and scale readings of the dashboard to obtain the pointer mask image and the scale mask image;

[0074] S4. Based on the pointer mask image and the scale mask image, use the skeleton thinning and center point extraction algorithm to remove edge pixels, and analyze to obtain the position of the pointer, the direction of the pointer, and the position of the scale;

[0075] S5. According to the position of the pointer, the direction of the pointer, and the position of the scale, use the improved angle method to obtain the final reading of the pointer-type instrument, completing the automatic recognition of the pointer instrument reading.

[0076] In this embodiment, as Figure 2 shown, the dashboard detection model based on YOLOX-DC in step S2 includes a backbone network, a neck network, and a detection head connected in sequence;

[0077] The backbone network is used to receive the image data of the pointer-type instrument, perform feature extraction, and output the features of the image data of the pointer-type instrument;

[0078] The neck network is used to receive the features of the image data of the pointer-type instrument, perform feature fusion, and output the fused features;

[0079] The detection head is used to receive the fused features, perform classification and regression according to the fused features, and output the dashboard in the image data of the pointer-type instrument.

[0080] In this embodiment, the backbone network adopts the CSPDarkNet structure, including: a Focus module, a first CBS layer, a first CSP layer, a second CBS layer, a second CSP layer, a third CBS layer, an SPP layer, and a third CSP layer connected in sequence; among them, the Focus module downsamples the feature map, the SPP (Spatial Pyramid Pooling) module expands the receptive field, and multiple stacked CBS modules and CSP layers transfer and extract features between these two modules;

[0081] The output ends of the first CSP layer, the second CSP layer, and the third CSP layer are also connected to the neck network;

[0082] The Focus module includes four Slice layers connected in parallel, and the output ends of the four Slice layers are all connected to the input end of the first Concat layer; the output end of the first Concat layer is connected to the input end of the fourth CBS layer; the output end of the fourth CBS layer is the output end of the entire Focus module.

[0083] In this embodiment, the neck network includes a sixth CBS layer, a first upsampling layer, a second Concat layer, a CBS&CSP layer, a second upsampling layer, a third Concat layer, a fifth CSP layer, a first downsampling layer, a fourth Concat layer, a sixth CSP layer, a second downsampling layer, and a Concat&CSP layer connected in sequence;

[0084] Another input end of the second Concat layer is connected to the third CSP layer; another input ends of the third Concat layer are respectively connected to the first CSP layer and the second CSP layer; another output end of the fifth CSP layer is connected to the detection head; another input end of the fourth Concat layer is connected to the CBS&CSP layer; another output end of the sixth CSP layer is connected to the detection head; another input end of the Concat&CSP layer is connected to the sixth CBS layer, and the output end of the Concat&CSP layer is connected to the detection head.

[0085] In this embodiment, the detection head is a Decoupled Circle Head, including: a first decoupled circle head, a second decoupled circle head, a third decoupled circle head, a first Reshape layer, a second Reshape layer, and a third Reshape layer;

[0086] The input end of the first decoupled circle head is connected to the output end of the fourth CSP layer, and the output end of the first decoupled circle head is connected to the input end of the first Reshape layer;

[0087] The input end of the second decoupled circle head is connected to the output end of the fifth CSP layer, and the output end of the second decoupled circle head is connected to the input end of the second Reshape layer;

[0088] The input end of the third decoupled circle head is connected to the output end of the first Concat&CSP layer, and the output end of the third decoupled circle head is connected to the input end of the third Reshape layer;

[0089] The output ends of the first Reshape layer, the second Reshape layer, and the third Reshape layer together serve as the output end of the entire detection head;

[0090] The most intuitive difference between the decoupled circular detection head and the rectangular detection head lies in the representation of the bounding box. The traditional method uses a rectangular bounding box and needs to learn four parameters to locate the detection target. The decoupled circular detection head uses a polar coordinate method to decouple the pointer dial contour and uses polar coordinates for the positioning of the pointer-type instrument panel. Since the center of the circle is the origin of the polar coordinate system, the circle in the Cartesian coordinate system is converted into a straight line in the polar coordinate system. Therefore, the concentric circle scales in the Cartesian coordinate system can be converted into linear scales in the polar coordinate system. Then the polar coordinate The calculation formula is:

[0091]

[0092] where represents the radial distance of the pointer-type instrument panel in the polar coordinate system, represents the polar angle of the pointer-type instrument panel in the polar coordinate system, represents the abscissa of the target position in the Cartesian coordinate system, represents the ordinate of the target position in the Cartesian coordinate system; arctan is the arctangent function.

[0093] In this embodiment, the image segmentation model based on PM-SwinUnet combines the multi-scale feature information extraction and fusion representation ability of the U-Net network with the global modeling perception ability of the Swin Transformer; as Figure 3 shown, the PM-SwinUnet consists of an encoder, a decoder, a bottleneck connection, and a skip connection, and its basic unit is the SwinTransformer block. The encoder divides the pointer-type instrument image into non-overlapping patches of 4×4, and the feature dimension of each patch is 4×4×3 = 48. Each patch is converted into a patch token through a linear embedding layer. Through the combination of the Swin Transformer block and patch merging, the feature dimension 48 is mapped to an arbitrary dimension C. When calculating the instrument image features, the Swin Transformer adds the window features of the previous layer to the window of the next layer to calculate the global features of the image, solving the problem that the pointer pixel area is distributed in a relatively long area in the instrument image.

[0094] In this embodiment, a symmetric decoder based on Swin Transformer is designed. The decoder consists of Swin Transformer blocks and patch expansion layers. The expansion layer upsamples adjacent feature maps to twice the resolution, and then fuses the multi-scale features extracted by the encoder with the context features through skip connections to compensate for the information loss caused by downsampling. Finally, the expansion layer undergoes four rounds of upsampling until the resolution of the instrument image features is restored to the input resolution. The upsampled features after linear projection are output as the segmentation results of the instrument image elements. To determine the positions of the pointer and the scale, a pointer mask image and a scale mask image are obtained through the PM-SwinUnet segmentation network.

[0095] In this embodiment, in the pointer-type instrument image, the position of the scale is on the circumference where the pointer rotates. Therefore, the relationship between the pointer and the scale provides a basis for identifying the instrument reading. The specific steps for analyzing and obtaining the position and direction of the pointer in step S4 are as follows:

[0096] S41. Using the center point extraction algorithm based on the mask image, extract the center point of the pointer-type instrument image according to the scale mask image;

[0097] S42. According to the skeleton refinement algorithm, extract the pointer skeleton from the pointer mask image, and refine the pointer mask into a unit pixel-level straight line to assist in detecting the position and direction of the pointer; since the pointer pixel area is continuous and has the largest area, select this area as the pointer mask to eliminate the noise of non-pointer pixels;

[0098] S43. Use the Hough transform to detect the straight lines in the pointer mask image, and the parameter settings are: the distance resolution is 1, the threshold is 10, the minimum line segment length is 10, and the maximum gap is 400; if a straight line is detected, traverse each straight line, extract the coordinates of the two endpoints of each straight line, and draw the straight line on the original image, and execute step S44; if no straight line is detected, output "The pointer cannot be detected" and end the process;

[0099] S44. Store the coordinates of the two endpoints in endpoint list 1 and endpoint list 2 respectively, and calculate the Euclidean distances from the two endpoints to the center point of the pointer-type instrument image respectively;

[0100] S45. Determine the endpoint that is closer to the center point of the pointer-type instrument image according to the Euclidean distance, obtain the starting point and ending point of the pointer line, and determine the position and direction of the pointer according to the starting point and ending point of the pointer line.

[0101] In this embodiment, the specific steps of step S41 are as follows:

[0102] S411. Perform connected component analysis on the scale mask image to identify the connected regions in the scale mask image;

[0103] S412. Traverse each connected region, filter out the connected regions with the number of pixels higher than the preset threshold, and generate a binary filtered image;

[0104] S413. Use the contour detection algorithm to extract the scale contours from the binary filtered image and store the scale contours in the contour list;

[0105] S414. Traverse each scale contour in the contour list to obtain the minimum bounding rectangle of each contour;

[0106] S415. Extract the center point coordinates of each minimum bounding rectangle, convert them to integer type, and store them in the "center point coordinate list" to complete the extraction of the center point of the pointer-type instrument image.

[0107] In this embodiment, the calculation formula of the Euclidean distance in step S44 is:

[0108]

[0109] Where, represents the Euclidean distance, represents the abscissa of the center point of the pointer-type instrument image, represents the abscissa of the endpoint of the straight line, represents the ordinate of the center point of the pointer-type instrument image, represents the ordinate of the endpoint of the straight line.

[0110] In this embodiment, the improved angle method in step S5 is specifically:

[0111] A1. Establish a Cartesian coordinate system with the center point O of the instrument panel as the origin;

[0112] A2. In the Cartesian coordinate system, calculate the angle O between the ray OA from the point OP to the 0 scale and the ray where the pointer is located, and the angle O between the ray OB from the point OP to the second scale and the ray where the pointer is located;

[0113] The calculation formula of the angle is:

[0114]

[0115] Where, , and respectively represent the vectors corresponding to ray OA , ray OP and ray OB ; represents the length of the vector; arccos represents the inverse cosine;

[0116] A3. Convert the included angles and from radians to degrees to obtain the reading of the pointer-type instrument, and its calculation formula is:

[0117]

[0118] wherein, represents the reading of the second scale.

[0119] In this embodiment, the reading of the second scale is calculated by a scale value recognition module; the scale value recognition module is composed of multiple convolutional layers and a connectionist temporal classification layer, and realizes scale value recognition through parallel linear prediction.

[0120] Embodiment 2. On the basis of Embodiment 1, a comparative experiment is conducted by comparing the industrial pointer instrument reading automatic recognition method based on deep learning proposed by the present invention with the existing method to fully illustrate the technical effect of the present invention.

[0121] In order to accurately detect and recognize industrial instruments in complex scenarios, 400 images each under conditions such as blur, tilt, overexposure, and low light are collected by an industrial camera in a real scenario, for a total of 2000 instrument images. This dataset is named "Real Pointer-Type Instrument Dataset (RPMeter Dataset)". All images are manually annotated using LabelImg and Labelme. In addition, to verify the generalization ability of the method of the present invention, experiments are also conducted on the publicly available dataset "MeterChallenge (MC1296)" containing 1296 pointer instrument pictures and the "Paddle Meter" dataset containing 1197 instrument pictures.

[0122] The input image size of YOLOX-DC is 640×640. During training, the Stochastic Gradient Descent (SGD) method is used to optimize the YOLOX-DC parameters, with a momentum of 0.9 and a weight decay of 0.0005. The initial learning rate is set to 0.001, the batch size is 2, and the training time is 30 epochs. The scale range of data augmentation is (0.1, 2). The input image size and patch size of PM-SwinUnet are set to 224×224 and 4 respectively. The pre-trained weights on the ImageNet dataset are used to initialize the model parameters. During training, the initial learning rate of the model is 0.001, the batch size is 1, and the training time is 50 epochs. The SGD optimizer is used for backpropagation, with a momentum of 0.9 and a weight decay of 0.0001.

[0123] In this embodiment, in order to verify that the decoupled round head can detect the pointer dial more accurately than the rectangular detection head, a pointer dial detection comparison experiment is carried out.

[0124] YOLOX-DC with a decoupled round head is compared with four algorithms using rectangular detection heads: Retinanet, Faster R-CNN, YOLOv5, and YOLOX. Under low-light conditions, Retinanet and YOLOv5 cannot detect the dial, resulting in missed detections. In addition, Retinanet and Faster R-CNN mislabel other regions in the image as the dial, resulting in false detections. Although YOLOX can detect the dial in all images, its confidence is significantly lower compared to YOLOX-DC. The results show that the proposed YOLOX-DC algorithm can not only accurately detect the dial but also has a high detection accuracy. It is worth noting that for circular pointer dials, the YOLOX-DC detection results contain less background noise than the rectangular detection head algorithms.

[0125] Secondly, this experiment evaluates and compares the YOLOX-DC algorithm with the other four algorithms in terms of precision, recall, and inference time. The detection results on the RPMeter test set are shown in Table 1. It can be seen that at IoU50:95, the accuracy of YOLOX-DC is 0.996 and the recall is 0.997, higher than the other four algorithms. Although the inference time of YOLOX-DC is 0.04 ms slower than that of YOLOX, its accuracy and recall are increased by 3.2% and 2.4% respectively. The experimental results further verify that the detection accuracy of the decoupled circular detection head for the pointer dial is higher than that of the rectangular detection head.

[0126] Table 1 Detection results of the YOLOX-DC algorithm and the other four algorithms on the RPMeter test set

[0127]

[0128] Subsequently, to verify the robustness and generalization of the proposed method, experiments were conducted on the "Meter Challenge (MC1296)" dataset and the "Paddle Meter" dataset. The experiments show that YOLOX-DC can process meter images in various complex situations. Therefore, the above experiments have all proved the effectiveness of the YOLOX-DC algorithm in pointer dial detection.

[0129] In this embodiment, through the dial element segmentation experiment, the performance of the image segmentation model based on PM-SwinUnet is verified.

[0130] The input of the PM-SwinUnet segmentation network is 224*224*3. Using the YOLOX-DC dial detection method, 1966 dial images were obtained using the minimum bounding rectangle. Among them, 1600 images were used for training and 366 images were used for evaluation. To verify the effectiveness of the proposed PM-SwinUnet image segmentation network, in the same experimental environment, the PM-SwinUnet image segmentation network was compared with the VGG-Unet and U-Net models. For turntables with different lighting, overexposure, and image blur, the PM-SwinUnet network obtained better segmentation results. Although the VGG-Unet segmentation network can correctly fit the scale region and the pointer region, there are small serrated edges in the segmentation results, and some segmentation regions are not separated. The U-Net segmentation results not only have problems with regional connectivity but also have problems with scale detection missing and incomplete pointer segmentation in an overexposed environment. Compared with the other two networks, the results of the pointer, scale, and scale value of PM-SwinUnet have no problems with regional connectivity, the edges are smooth, and there are no missing scale detections or incomplete pointer segmentations. The experiments show that the PM-SwinUnet image segmentation network proposed in the present invention solves the problems of missed detection of small-scale targets and incomplete segmentation of long-scale targets.

[0131] In this embodiment, through the pointer reading recognition comparison experiment, the reading recognition method proposed in the present invention is applied to different image segmentation networks, and the results are compared with manual readings to verify the robustness of the proposed reading recognition method.

[0132] When reading manually, according to the specifications of pointer-type instruments, the reading should be controlled within 1 / 5 of the minimum scale. The detailed experimental results are shown in Table 1. It can be seen from Table 1 that the recognition method proposed in this paper has good adaptability to the segmentation results of different image segmentation networks, with an average relative error of 0.0092 and an average citation error of 0.0026. Compared with the VGG-Unet and U-Net networks, there are obvious improvements. The maximum error between the recognition reading and the manual reading of this method is 0.793 °C. Given that the error precision of the thermometer used in this experiment is ±1 °C, this error level is within a reasonable range.

[0133] Compare the recognition results of four types of low-quality instrument images: blurred, tilted, overexposed, and low-light, to further verify the performance of the reading recognition method for processing low-quality instrument images. The recognition results of low-quality instrument images are as Figure 4 shown. From Figure 4 it can be obtained that the scale and pointer of the instrument fit well, and the recognized reading is within the error range. It should be noted that this method can accurately recognize the reading even in overexposed images. The specific recognition results of low-quality instrument images are shown in Table 2. The maximum relative errors of blurred images, tilted images, overexposed images, and low-light images are 0.0093, 0.0523, 0.0039, and 0.0094 respectively. The maximum reference errors are 0.0027, 0.0133, 0.0009, and 0.0057 respectively. Compared with the allowable error of ±0.01, the recognition errors of other low-quality images are within the acceptable range except for tilted images. Although the performance of this method decreases on tilted images, it can be applied to most normal instrument images and has a high recognition accuracy.

[0134] Table 2 Recognition Results of Low-Quality Instrument Images

[0135]

[0136] To further verify the generalization of the reading recognition method, experiments were conducted on the "Paddle Meter" dataset and the "MeterChallenge (MC1296)" dataset. The experimental results are as Figure 5As shown, this method can accurately fit the pointers and scales of various instruments and correctly identify the instrument readings. The specific recognition results are shown in Table 3. Among them, the maximum reference errors of the three instruments in the "Paddle Meter" dataset are 0.0022, 0.0030, and 0.0087 respectively. The allowable errors of the three pointer instruments in the "Paddle Meter" dataset are 0.02, 0.02, and 0.03 respectively. The cited error of the proposed method is significantly smaller than the allowable error. The MC1296 dataset contains 7 types of pointer instrument types, and the reference error of the 5th image reaches 0.025. Compared with its allowable error of 0.05, this reference error is within a reasonable range. The experimental results show that this method has strong generalization performance.

[0137] Table 3 Verification Results on the "Paddle Meter" Dataset and the "Meter Challenge (MC1296)" Dataset

[0138]

[0139] Those of ordinary skill in the art will realize that the embodiments described herein are for helping the reader understand the principles of the present invention, and it should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those of ordinary skill in the art can make various other specific deformations and combinations that do not depart from the essence of the present invention based on the technical revelations disclosed in the present invention, and these deformations and combinations are still within the protection scope of the present invention.

Claims

1. An automatic recognition method for industrial pointer instrument readings based on deep learning, characterized in that, The method includes the following steps: S1. Collect image data of the pointer-type instrument using an industrial camera; S2. Use a dashboard detection model based on YOLOX-DC to detect the dashboard in the image data of the pointer-type instrument; the dashboard detection model based on YOLOX-DC replaces the rectangular detection box of the traditional YOLOX with a decoupled circular detection head; The decoupled circular detection head uses polar coordinates to position the pointer-type instrument panel, and the polar coordinates have the following calculation formula: Among them, represents the polar radius of the pointer-type dashboard in the polar coordinate system, represents the polar angle of the pointer-type dashboard in the polar coordinate system, represents the abscissa of the target location in the Cartesian coordinate system, represents the ordinate of the target location in the Cartesian coordinate system; arctan is the arctangent function; S3. Use an image segmentation model based on PM-SwinUnet to segment the pointer, key scales, and scale readings of the dashboard to obtain a pointer mask image and a scale mask image; the network structure of the image segmentation model based on PM-SwinUnet is the same as that of the traditional SwinUnet model; S4. Based on the pointer mask image and the scale mask image, use a skeleton thinning and center point extraction algorithm to remove edge pixels, and analyze to obtain the position of the pointer, the direction of the pointer, and the position of the scale; S5. According to the position of the pointer, the direction of the pointer, and the position of the scale, use an improved angle method to obtain the final reading of the pointer-type instrument, and complete the automatic recognition of the pointer instrument reading; The improved angle method is: calculate the angle between the ray from the center point of the Cartesian coordinate system to the 0 scale and the ray where the pointer is located, and the angle between the ray from the center point to the second scale and the ray where the pointer is located; calculate the ratio of the two angles and multiply it by the reading of the second scale to obtain the final reading of the pointer-type instrument.

2. The automatic recognition method for industrial pointer instrument readings based on deep learning according to claim 1, wherein In step S2, the dashboard detection model based on YOLOX-DC includes a backbone network, a neck network, and a detection head connected in sequence; The backbone network is used to receive the image data of the pointer-type instrument, perform feature extraction, and output the features of the image data of the pointer-type instrument; The neck network is used to receive the features of the image data of the pointer-type instrument, perform feature fusion, and output the fused features; The detection head is used to receive the fused features, perform classification and regression according to the fused features, and output the dashboard in the image data of the pointer-type instrument.

3. The automatic recognition method for industrial pointer instrument readings based on deep learning according to claim 2, characterized in that, The backbone network adopts a CSPDarkNet structure, including: a Focus module, a first CBS layer, a first CSP layer, a second CBS layer, a second CSP layer, a third CBS layer, a third CSP layer, a fourth CBS layer, an SPP layer, and a fourth CSP layer connected in sequence; The output ends of the first CSP layer, the second CSP layer, the third CSP layer, and the fourth CSP layer are also connected to the neck network; The Focus module includes four Slice layers connected in parallel, and the output ends of the four Slice layers are all connected to the input end of the first Concat layer; the output end of the first Concat layer is connected to the input end of the fifth CBS layer; the output end of the fifth CBS layer is the output end of the entire Focus module.

4. The automatic recognition method for industrial pointer instrument readings based on deep learning according to claim 3, characterized in that The neck network includes a sixth CBS layer, a first upsampling layer, a second Concat layer, a CBS&CSP layer, a second upsampling layer, a third Concat layer, a fifth CSP layer, a first downsampling layer, a fourth Concat layer, a sixth CSP layer, a second downsampling layer, and a Concat&CSP layer connected in sequence; Another input end of the second Concat layer is connected to a third CSP layer; another input ends of the third Concat layer are respectively connected to a first CSP layer and a second CSP layer; another output end of the fifth CSP layer is connected to a detection head; another input end of the fourth Concat layer is connected to the CBS&CSP layer; another output end of the sixth CSP layer is connected to the detection head; another input end of the Concat&CSP layer is connected to the sixth CBS layer, and the output end of the Concat&CSP layer is connected to the detection head.

5. The automatic recognition method for industrial pointer instrument readings based on deep learning according to claim 4, characterized in that The detection head includes: a first decoupled circular detection head, a second decoupled circular detection head, a third decoupled circular detection head, a first Reshape layer, a second Reshape layer, and a third Reshape layer; The input end of the first decoupled circular detection head is connected to the output end of the fourth CSP layer, and the output end of the first decoupled circular detection head is connected to the input end of the first Reshape layer; The input end of the second decoupled circular detection head is connected to the output end of the fifth CSP layer, and the output end of the second decoupled circular detection head is connected to the input end of the second Reshape layer; The input end of the third decoupled circular detection head is connected to the output end of the first Concat&CSP layer, and the output end of the third decoupled circular detection head is connected to the input end of the third Reshape layer; The output ends of the first Reshape layer, the second Reshape layer, and the third Reshape layer together serve as the output end of the entire detection head.

6. The automatic recognition method for industrial pointer instrument readings based on deep learning according to claim 1, characterized in that The specific steps of analyzing to obtain the position and direction of the pointer in step S4 include the following sub-steps: S41. Using a center point extraction algorithm based on a mask graph, extract the center point of the pointer-type instrument image according to the scale mask image; S42. According to the skeleton thinning algorithm, extract the pointer skeleton from the pointer mask image and thin the pointer mask into a straight line of unit pixel level; S43. Use the Hough transform to detect the straight lines in the pointer mask image. If a straight line is detected, traverse each straight line, extract the coordinates of the two endpoints of each straight line, and draw a straight line on the original image, and execute step S44; if no straight line is detected, output "The pointer cannot be detected" and end the process; S44. Store the coordinates of the two endpoints in endpoint list 1 and endpoint list 2 respectively, and calculate the Euclidean distances from the two endpoints to the center point of the pointer-type instrument image respectively; S45. Determine the endpoint closer to the center point of the pointer-type instrument image according to the Euclidean distance to obtain the starting point and ending point of the pointer line, and determine the position and direction of the pointer according to the starting point and ending point of the pointer line.

7. The automatic recognition method for industrial pointer instrument readings based on deep learning according to claim 6, characterized in that The specific steps of step S41 include the following steps: S411. Perform connected component analysis on the scale mask image to identify the connected regions in the scale mask image; S412. Traverse each connected region, filter out the connected regions with the number of pixels higher than the preset threshold, and generate a binary filtered image; S413. Use a contour detection algorithm to extract the scale contours from the binary filtered image and store the scale contours in a contour list; S414. Traverse each scale contour in the contour list to obtain the minimum bounding rectangle of each contour; S415. Extract the center point coordinates of each minimum bounding rectangle, convert them to integer type, and store them in the "center point coordinate list" to complete the extraction of the center point of the pointer-type instrument image.

8. The automatic recognition method for industrial pointer instrument readings based on deep learning according to claim 6, characterized in that The calculation formula for the Euclidean distance described in step S44 is: Among them, represents the Euclidean distance, represents the abscissa of the center point of the pointer-type instrument image, represents the abscissa of the end point of the straight line, represents the ordinate of the center point of the pointer-type instrument image, represents the ordinate of the end point of the straight line.

9. The automatic recognition method for industrial pointer instrument readings based on deep learning according to claim 1, characterized in that The improved angle method described in step S5 is specifically: A1. With the center point of the dashboard O as the origin, establish a Cartesian coordinate system; A2. In the Cartesian coordinate system, calculate the angle between the ray from the point O to the 0 scale OA and the ray where the pointer is located OP , and the angle between the ray from the point to the second scale O and the ray where the pointer is located OB ; OP and ; The calculation formula for the included angle is: Among them, , and respectively represent the vectors corresponding to the rays OA , ray OP and ray OB ; represents the length of the vector; arccos represents the inverse cosine;​ A3. Convert the included angles and from radians to degrees to obtain the reading of the pointer-type instrument , and its calculation formula is: Among them, represents the reading of the second scale.

10. The automatic recognition method for industrial pointer instrument readings based on deep learning according to claim 9, characterized in that, The reading of the second scale is recognized by the scale value recognition module; the scale value recognition module consists of multiple convolutional layers and a connectionist temporal classification layer, and realizes scale value recognition through parallel linear prediction.

Images