Industrial pointer instrument reading automatic identification method based on deep learning

Abstract

The invention belongs to the technical field of pointer instrument reading identification, and particularly discloses an industrial pointer instrument reading automatic identification method based on deep learning, which comprises the following steps of: acquiring an image of a pointer instrument by using an industrial camera; detecting an instrument panel in the instrument image by using an instrument panel detection model based on YOLOX-DC; segmenting a pointer, a key scale and a scale reading of the instrument panel by using an image segmentation model based on PM-SwinUnet to obtain a segmented image of the instrument; and based on the segmented image of the instrument, removing edge pixels by using a skeleton refinement and central point extraction algorithm to obtain the position of the pointer, the direction of the pointer and the position of the scale, and further obtaining a final pointer type instrument reading by using an improved angle method. The problems that existing pointer instrument reading recognition depends on prior information, generalization ability is poor and segmentation is incomplete are solved, and accurate recognition of pointer instrument reading is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of pointer instrument reading recognition, and in particular relates to an automatic recognition method for industrial pointer instrument readings based on deep learning. Background Art

[0002] Pointer instruments, as an indispensable basic measuring instrument in industrial production, are widely used in power systems, manufacturing systems, remote sensing, aerospace and other fields to measure and record key production data. Regular verification and calibration of pointer instruments are essential to maintain their optimal performance and quality, ensure accuracy and credibility, ensure safety and reliability during use, and improve industrial production efficiency. At present, the verification and calibration of pointer instruments mostly rely on manual meter reading and data recording by metrologists in laboratories or industrial sites. On the one hand, it takes a long time, is inefficient, and the reading accuracy is limited due to light and pollution. On the other hand, in some special industrial sites such as substations, the working environment of the instrument is relatively harsh, and long-term manual recording work will also bring challenges to the physical and mental health of metrologists. With the development of artificial intelligence, image processing and machine vision technology, automatic reading recognition of pointer instruments based on machine vision is gradually replacing manual meter reading, becoming a key way to improve production automation and improve production efficiency and quality, promoting the digital transformation of the instrument measurement and testing industry, and promoting the process of new industrialization, which is of great significance.

[0003] Although some methods for automatic recognition of pointer readings have been proposed, they still have problems such as low detection accuracy and insufficient feature extraction capability for small-scale and long targets when facing blurry, uneven lighting and tilted instrument placement industrial scenes. This restricts the generalization and expansion of automatic recognition of pointer readings technology in complex industrial fields.

[0004] The automatic recognition method of pointer instrument readings consists of two key technologies: target detection and image segmentation.

[0005] Existing target detection methods for pointer instrument reading recognition can be divided into two categories: rule-based methods and statistical-based methods. Rule-based methods rely on prior information such as dial range, dial circumference, and pointer features, as well as manually extracted features. In complex and changeable industrial environments, this method is difficult to flexibly migrate and apply. Statistical information-based methods use deep learning technology to automatically learn the image features of pointer instruments, such as YOLOv5 and YOLOX, which can get rid of the dependence on prior knowledge and artificial features, but are more sensitive to illumination changes and image quality, and have poor generalization and robustness in conditions such as uneven illumination and image blur.

[0006] Existing image segmentation methods for pointer instrument reading recognition can be divided into two categories: traditional machine vision methods and deep learning methods. The traditional machine vision-based method uses threshold segmentation, edge detection, and line detection to extract the dial and pointer, and different parameters need to be manually set for different pointer dials. The image segmentation method based on deep learning does not require manual setting of dial parameters. It builds different network structures based on convolutional neural networks (CNN), such as Mask R-CNN and U-Net. Through autonomous learning of data labels, it can realize the segmentation of dials and pointers for different instruments. However, it is limited by the perception of global features and long-distance features, resulting in incomplete pointer segmentation, which affects the accuracy and robustness of reading recognition.

[0007] In summary, the existing pointer meter reading recognition methods have problems such as reliance on prior information, poor generalization ability, and incomplete segmentation. When facing complex industrial environments, the accuracy and robustness of detection and segmentation need to be further improved. Summary of the invention

[0008] The purpose of the present invention is to solve the defects of existing pointer instrument reading recognition methods, such as reliance on prior information, poor generalization ability and incomplete segmentation, which lead to poor detection and segmentation accuracy and robustness when facing complex industrial environments. A method for automatic recognition of industrial pointer instrument readings based on deep learning is proposed.

[0009] The technical solution of the present invention is: a method for automatically identifying readings of industrial pointer instruments based on deep learning, comprising the following steps: S1. Collect pointer instrument image data using industrial cameras; S2. Using the instrument panel detection model based on YOLOX-DC, the instrument panel in the pointer instrument image data is detected; S3. Using the PM-SwinUnet-based image segmentation model, the pointer, key scales, and scale readings of the instrument panel are segmented to obtain the pointer mask image and the scale mask image; S4. Based on the pointer mask image and the scale mask image, use the skeleton refinement and center point extraction algorithm to remove edge pixels, and analyze the position of the pointer, the direction of the pointer and the position of the scale; S5. According to the position of the pointer, the direction of the pointer and the position of the scale, the final pointer instrument reading is obtained using the improved angle method to complete the automatic recognition of the pointer instrument reading.

[0010] The beneficial effects of the present invention are: 1. The present invention proposes a YOLOX-DC pointer instrument target detection model to realize pointer dial detection, which can better adapt to the appearance characteristics of circular pointer dials and has more efficient reasoning speed and better robustness when facing complex industrial scenarios.

[0011] 2. The present invention simultaneously segments the pointer, scale lines and scale values ​​through the PM-SwinUnet pointer instrument image segmentation model, achieves better performance in the segmentation of dial elements, and solves the problems of missing small target detection and incomplete long target segmentation.

[0012] Preferably, the YOLOX-DC-based instrument panel detection model in step S2 comprises a backbone network, a neck network and a detection head connected in sequence; The backbone network is used to receive the pointer instrument image data, perform feature extraction, and output features of the pointer instrument image data; The neck network is used to receive the features of the pointer instrument image data, perform feature fusion, and output fused features; The detection head is used to receive the fused features, perform classification and regression according to the fused features, and output the instrument panel in the pointer-type instrument image data.

[0013] Preferably, the backbone network adopts a CSPDarkNet structure, comprising: a Focus module, a first CBS layer, a first CSP layer, a second CBS layer, a second CSP layer, a third CBS layer, an SPP layer and a third CSP layer connected in sequence; The output ends of the first CSP layer, the second CSP layer and the third CSP layer are also connected to the neck network; The Focus module includes four Slice layers connected in parallel, the output ends of the four Slice layers are all connected to the input end of the first Concat layer; the output end of the first Concat layer is connected to the input end of the fourth CBS layer; the output end of the fourth CBS layer is the output end of the entire Focus module.

[0014] Preferably, the neck network comprises a sixth CBS layer, a first upsampling layer, a second Concat layer, a CBS&CSP layer, a second upsampling layer, a third Concat layer, a fifth CSP layer, a first downsampling layer, a fourth Concat layer, a sixth CSP layer, a second downsampling layer and a Concat&CSP layer connected in sequence; Another input end of the second Concat layer is connected to the third CSP layer; another input end of the third Concat layer is respectively connected to the first CSP layer and the second CSP layer; another output end of the fifth CSP layer is connected to the detection head; another input end of the fourth Concat layer is connected to the CBS&CSP layer; another output end of the sixth CSP layer is connected to the detection head; another input end of the Concat&CSP layer is connected to the sixth CBS layer, and the output end of the Concat&CSP layer is connected to the detection head.

[0015] Preferably, the detection head is a decoupled circular detection head, comprising: a first decoupled circular detection head, a second decoupled circular detection head, a third decoupled circular detection head, a first Reshape layer, a second Reshape layer and a third Reshape layer; The input end of the first decoupling circular detection head is connected to the output end of the fourth CSP layer, and the output end of the first decoupling circular detection head is connected to the input end of the first Reshape layer; The input end of the second decoupling circular detection head is connected to the output end of the fifth CSP layer, and the output end of the second decoupling circular detection head is connected to the input end of the second Reshape layer; The input end of the third decoupled circular detection head is connected to the output end of the first Concat&CSP layer, and the output end of the third decoupled circular detection head is connected to the input end of the third Reshape layer; The output end of the first Reshape layer, the output end of the second Reshape layer and the output end of the third Reshape layer together serve as the output end of the entire detection head; The detection head uses polar coordinates Positioning of the pointer instrument panel, the polar coordinates The calculation formula is:

[0016] in, Indicates the polar diameter of the pointer instrument panel in the polar coordinate system. Indicates the polar angle of the pointer instrument panel in the polar coordinate system. Indicates the horizontal coordinate of the target location in the Cartesian coordinate system. Indicates the vertical coordinate of the target location in the Cartesian coordinate system; arctan is the inverse tangent function.

[0017] The beneficial effects of the above preferred solution are: The detection of the pointer dial is realized by constructing a decoupled circular detection head. Compared with the traditional rectangular detection frame, the method provided by the present invention can better adapt to the appearance characteristics of the circular pointer dial. At the same time, since the method uses polar coordinates to locate the detection target, only two parameters ( and ) can be positioned, which is simpler and more efficient than the four parameters of the rectangular box, making this method have more efficient reasoning speed and better robustness when facing complex industrial scenarios.

[0018] Preferably, the step S4 of analyzing and obtaining the position and direction of the pointer specifically includes the following sub-steps: S41. Using a center point extraction algorithm based on a mask image, the center point of the pointer instrument image is extracted according to the scale mask image; S42. Extracting the pointer skeleton from the pointer mask image according to the skeleton refinement algorithm, and refining the pointer mask into straight lines at the unit pixel level; S43. Use Hough transform to detect straight lines in the pointer mask image. If a straight line is detected, traverse each straight line, extract the coordinates of the two endpoints of each straight line, and draw the straight line on the original image, and execute step S44; if no straight line is detected, output "pointer cannot be detected" and end the process; S44. The coordinates of the two endpoints are stored in endpoint list 1 and endpoint list 2, respectively, and the Euclidean distances from the two endpoints to the center point of the pointer instrument image are calculated; S45. Determine the endpoint closer to the center point of the pointer instrument image based on the Euclidean distance, obtain the starting point and the end point of the pointer line, and determine the position and direction of the pointer based on the starting point and the end point of the pointer line.

[0019] Preferably, the step S41 specifically includes the following steps: S411. Performing a connected domain analysis on the scale mask image to identify connected regions in the scale mask image; S412. Traverse each connected area, filter out the connected areas with a number of pixels higher than a preset threshold, and generate a binary filtered image; S413. Extracting the scale contour from the binary filtered image using a contour detection algorithm, and storing the scale contour in a contour list; S414. Traverse each scale contour in the contour list to obtain the minimum circumscribed rectangle of each contour; S415. Extract the center point coordinates of each minimum circumscribed rectangle, convert them into integer type, and store them in the "center point coordinate list", thereby completing the extraction of the center point of the pointer instrument image.

[0020] Preferably, the calculation formula of the Euclidean distance in step S44 is:

[0021] in, represents the Euclidean distance, Indicates the horizontal coordinate of the center point of the pointer instrument image, represents the horizontal coordinates of the endpoints of the line, Indicates the ordinate of the center point of the pointer instrument image. Indicates the ordinate of the endpoint of the line.

[0022] Preferably, the improved angle method in step S5 is specifically: A1. Take the center point of the instrument panel O Establish a Cartesian coordinate system for the origin; A2. In the Cartesian coordinate system, calculate the O Ray to 0 scale OA The ray where the pointer is located OP The angle between , and point O Ray to the second scale OB The ray where the pointer is located OP The angle between ; The calculation formula of the angle is:

[0023] in, , and Respectively represent rays OA ,ray OP and rays OB The corresponding vector, Represents the length of a vector; arccos represents inverse cosine; A3. and angle Convert from radians to degrees to get the reading of pointer instruments , and its calculation formula is:

[0024] in, Indicates the reading of the second scale.

[0025] The beneficial effects of the above preferred solution are: This preferred solution proposes an improved angle method that fully utilizes the geometric relationship between the pointer and the scale line. Compared with the traditional angle method, the improved angle method minimizes the computational load and error, can better adapt to the segmentation results of different image segmentation networks, and exhibits higher robustness and accuracy.

[0026] Preferably, the second scale reading is It is calculated by a scale value recognition module; the scale value recognition module is composed of multiple convolutional layers and a connection time series classification layer, and scale value recognition is achieved through parallel linear prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Shown is a flow chart of a method for automatic identification of industrial pointer instrument readings based on deep learning.

[0028] Figure 2It is a schematic diagram of the structure of the instrument panel detection model based on YOLOX-DC provided in Example 1 of the present invention.

[0029] Figure 3 The figure is a schematic diagram of the structure of the image segmentation model of PM-SwinUnet provided in Example 1 of the present invention.

[0030] Figure 4 Shown is a schematic diagram of the recognition results of the low-quality instrument image provided in Example 2 of the present invention.

[0031] Figure 5 The figure shows the experimental results of the experiment conducted on the “Paddle Meter” dataset and the “MeterChallenge (MC1296)” dataset provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0032] Now, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be understood that the embodiments shown and described in the accompanying drawings are only exemplary and are intended to explain the principles and spirit of the present invention, rather than to limit the scope of the present invention.

[0033] Embodiment 1, as Figure 1 As shown, a method for automatic identification of industrial pointer instrument readings based on deep learning includes the following steps: S1. Collect pointer instrument image data using industrial cameras; S2. Using the instrument panel detection model based on YOLOX-DC, the instrument panel in the pointer instrument image data is detected; S3. Using the PM-SwinUnet-based image segmentation model, the pointer, key scales, and scale readings of the instrument panel are segmented to obtain the pointer mask image and the scale mask image; S4. Based on the pointer mask image and the scale mask image, use the skeleton refinement and center point extraction algorithm to remove edge pixels, and analyze and obtain the pointer position, pointer direction and scale position; S5. According to the position of the pointer, the direction of the pointer and the position of the scale, the final pointer instrument reading is obtained using the improved angle method to complete the automatic recognition of the pointer instrument reading.

[0034] In this embodiment, if Figure 2 As shown, the YOLOX-DC-based instrument panel detection model described in step S2 includes a backbone network, a neck network, and a detection head connected in sequence; The backbone network is used to receive the pointer instrument image data, perform feature extraction, and output features of the pointer instrument image data; The neck network is used to receive the features of the pointer instrument image data, perform feature fusion, and output fused features; The detection head is used to receive the fused features, perform classification and regression according to the fused features, and output the instrument panel in the pointer-type instrument image data.

[0035] In this embodiment, the backbone network adopts a CSPDarkNet structure, including: a Focus module, a first CBS layer, a first CSP layer, a second CBS layer, a second CSP layer, a third CBS layer, an SPP layer, and a third CSP layer connected in sequence; wherein the Focus module downsamples the feature map, the SPP (spatial pyramid pooling) module expands the receptive field, and multiple stacked CBS modules and CSP layers transfer and extract features between the two modules; The output ends of the first CSP layer, the second CSP layer and the third CSP layer are also connected to the neck network; The Focus module includes four Slice layers connected in parallel, the output ends of the four Slice layers are all connected to the input end of the first concatenation (Concat) layer; the output end of the first Concat layer is connected to the input end of the fourth CBS layer; the output end of the fourth CBS layer is the output end of the entire Focus module.

[0036] In this embodiment, the neck network includes a sixth CBS layer, a first upsampling layer, a second Concat layer, a CBS&CSP layer, a second upsampling layer, a third Concat layer, a fifth CSP layer, a first downsampling layer, a fourth Concat layer, a sixth CSP layer, a second downsampling layer and a Concat&CSP layer connected in sequence; Another input end of the second Concat layer is connected to the third CSP layer; another input end of the third Concat layer is respectively connected to the first CSP layer and the second CSP layer; another output end of the fifth CSP layer is connected to the detection head; another input end of the fourth Concat layer is connected to the CBS&CSP layer; another output end of the sixth CSP layer is connected to the detection head; another input end of the Concat&CSP layer is connected to the sixth CBS layer, and the output end of the Concat&CSP layer is connected to the detection head.

[0037] In this embodiment, the detection head is a decoupled circle detection head (Decoupled Circle Head), including: a first decoupled circle detection head, a second decoupled circle detection head, a third decoupled circle detection head, a first reshape operation (Reshape) layer, a second Reshape layer and a third Reshape layer; The input end of the first decoupling circular detection head is connected to the output end of the fourth CSP layer, and the output end of the first decoupling circular detection head is connected to the input end of the first Reshape layer; The input end of the second decoupling circular detection head is connected to the output end of the fifth CSP layer, and the output end of the second decoupling circular detection head is connected to the input end of the second Reshape layer; The input end of the third decoupled circular detection head is connected to the output end of the first Concat&CSP layer, and the output end of the third decoupled circular detection head is connected to the input end of the third Reshape layer; The output end of the first Reshape layer, the output end of the second Reshape layer and the output end of the third Reshape layer together serve as the output end of the entire detection head; The most intuitive difference between the decoupled circular detection head and the rectangular detection head is the representation of the bounding box. The traditional method uses a rectangular bounding box and needs to learn four parameters to locate the detection target. The decoupled circular detection head uses the polar coordinate method to decouple the pointer disk contour. To locate the pointer instrument panel, since the center of the circle is the origin of the polar coordinate system, the circle in the Cartesian coordinate system is converted into a straight line in the polar coordinate system. Therefore, the concentric circle scale in the Cartesian coordinate system can be converted into a linear scale in the polar coordinate system. The calculation formula is:

[0038] in, Indicates the polar diameter of the pointer instrument panel in the polar coordinate system. Indicates the polar angle of the pointer instrument panel in the polar coordinate system. Indicates the horizontal coordinate of the target location in the Cartesian coordinate system. Indicates the vertical coordinate of the target location in the Cartesian coordinate system; arctan is the inverse tangent function.

[0039] In this embodiment, the PM-SwinUnet-based image segmentation model combines the multi-scale feature information extraction and fusion representation capabilities of the U-Net network with the global modeling perception capabilities of the Swin Transformer; Figure 3As shown, the PM-SwinUnet consists of an encoder, a decoder, a bottleneck connection and a jump connection, and its basic unit is a SwinTransformer block. The encoder divides the pointer instrument image into 4×4 non-overlapping patches, and the feature dimension of each patch is 4×4×3=48. Each patch is converted to a patch token through a linear embedding layer. Through the combination of Swin Transformer blocks and patch merging, the feature dimension 48 is mapped to an arbitrary dimension C. When calculating the instrument image features, the Swin Transformer adds the window features of the previous layer to the windows of the next layer to calculate the global features of the image, which solves the problem that the pointer pixel area is distributed in a longer area in the instrument image.

[0040] In this embodiment, a symmetric decoder based on Swin Transformer is designed, and the decoder consists of a Swin Transformer block and a patch expansion layer. The expansion layer upsamples the adjacent feature maps to twice the resolution, and then fuses the multi-scale features extracted by the encoder with the context features through jump connections to compensate for the information loss caused by downsampling. Finally, the expansion layer undergoes four rounds of upsampling until the resolution of the instrument image features is restored to the input resolution. The upsampled features after linear projection are output as the segmentation results of the instrument image elements. In order to determine the position of the pointer and scale, the pointer mask image and the scale mask image are obtained through the PM-SwinUnet segmentation network.

[0041] In this embodiment, in the pointer instrument image, the position of the scale is located on the circumference of the pointer rotation. Therefore, the relationship between the pointer and the scale provides a basis for identifying the instrument reading. The analysis of obtaining the position and direction of the pointer in step S4 specifically includes the following sub-steps: S41. Using a center point extraction algorithm based on a mask image, the center point of the pointer instrument image is extracted according to the scale mask image; S42. According to the skeleton thinning algorithm, the pointer skeleton is extracted from the pointer mask image, and the pointer mask is thinned into straight lines at the unit pixel level to assist in detecting the position and direction of the pointer; since the pointer pixel region is continuous and has the largest area, this region is selected as the pointer mask to eliminate noise of non-pointer pixels; S43. Use Hough transform to detect straight lines in the pointer mask image, and set the parameters as follows: distance resolution is 1, threshold is 10, minimum line segment length is 10, and maximum gap is 400; if a straight line is detected, traverse each straight line, extract the coordinates of the two endpoints of each straight line, and draw the straight line on the original image, and execute step S44; if no straight line is detected, output "pointer cannot be detected" and end the process; S44. The coordinates of the two endpoints are stored in endpoint list 1 and endpoint list 2, respectively, and the Euclidean distances from the two endpoints to the center point of the pointer instrument image are calculated; S45. Determine the endpoint closer to the center point of the pointer instrument image based on the Euclidean distance, obtain the starting point and the end point of the pointer line, and determine the position and direction of the pointer based on the starting point and the end point of the pointer line.

[0042] In this embodiment, step S41 specifically includes the following steps: S411. Performing a connected domain analysis on the scale mask image to identify connected regions in the scale mask image; S412. Traverse each connected area, filter out the connected areas with a number of pixels higher than a preset threshold, and generate a binary filtered image; S413. Extracting the scale contour from the binary filtered image using a contour detection algorithm, and storing the scale contour in a contour list; S414. Traverse each scale contour in the contour list to obtain the minimum circumscribed rectangle of each contour; S415. Extract the center point coordinates of each minimum circumscribed rectangle, convert them into integer type, and store them in the "center point coordinate list", thereby completing the extraction of the center point of the pointer instrument image.

[0043] In this embodiment, the calculation formula of the Euclidean distance in step S44 is:

[0044] in, represents the Euclidean distance, Indicates the horizontal coordinate of the center point of the pointer instrument image, represents the horizontal coordinates of the endpoints of the line, Indicates the ordinate of the center point of the pointer instrument image. Indicates the ordinate of the endpoint of the line.

[0045] In this embodiment, the improved angle method in step S5 is specifically: A1. Take the center point of the instrument panel O Establish a Cartesian coordinate system for the origin; A2. In the Cartesian coordinate system, calculate the O Ray to 0 scale OA The ray where the pointer is located OP The angle between , and point O Ray to the second scale OB The ray where the pointer is located OP The angle between ; The calculation formula of the angle is:

[0046] in, , and Respectively represent rays OA ,ray OP and rays OB The corresponding vector, Represents the length of a vector; arccos represents inverse cosine; A3. and angle Convert from radians to degrees to get the reading of pointer instruments , and its calculation formula is:

[0047] in, Indicates the reading of the second scale.

[0048] In this embodiment, the reading of the second scale is It is calculated by a scale value recognition module; the scale value recognition module is composed of multiple convolutional layers and a connection time series classification layer, and scale value recognition is achieved through parallel linear prediction.

[0049] Example 2, based on Example 1, compares the industrial pointer instrument reading automatic recognition method based on deep learning proposed by the present invention with the existing method to fully illustrate the technical effect of the present invention.

[0050] In order to accurately detect and identify industrial instruments in complex scenes, 400 images of blur, tilt, overexposure and dark light conditions were collected using industrial cameras in real scenes, totaling 2000 instrument images. This dataset is named "Real Pointer Meter Dataset (RPMeter Dataset)". All images are manually annotated using LabelImg and Labelme. In addition, to verify the generalization ability of the proposed method, experiments were conducted on the "MeterChallenge (MC1296)" public dataset containing 1296 pointer instrument images and the "Paddle Meter" dataset containing 1197 instrument images.

[0051] The input image size of YOLOX-DC is 640×640. During the training process, the stochastic gradient descent (SGD) method is used to optimize the YOLOX-DC parameters with a momentum of 0.9 and a weight decay of 0.0005. The initial learning rate is set to 0.001, the batch size is 2, and the training time is 30 epochs. The scale range of mosaic data augmentation is (0.1, 2). The input image size and patch size of PM-SwinUnet are set to 224×224 and 4, respectively. The pre-trained weights on the ImageNet dataset are used to initialize the model parameters. During the training process, the initial learning rate of the model is 0.001, the batch size is 1, and the training time is 50 epochs. The SGD optimizer is used for back propagation with a momentum of 0.9 and a weight decay of 0.0001.

[0052] In this embodiment, in order to verify that the decoupling round head can detect the pointer dial more accurately than the rectangular detection head, a pointer dial detection comparison experiment is performed.

[0053] YOLOX-DC with a decoupled round head is compared with four algorithms using rectangular detection heads: Retinanet, Faster R-CNN, YOLOv5, and YOLOX. In low-light conditions, Retinanet and YOLOv5 fail to detect the dial, resulting in missed detections. In addition, Retinanet and Faster R-CNN incorrectly mark other areas in the image as dials, resulting in false detections. Although YOLOX can detect the dial in all images, its confidence is significantly lower compared to YOLOX-DC. The results show that the proposed YOLOX-DC algorithm can not only accurately detect the dial, but also has high detection accuracy. It is worth noting that for the circular pointer dial, the YOLOX-DC detection results contain less background noise than the rectangular detection head algorithm.

[0054] Secondly, this experiment evaluates and compares the YOLOX-DC algorithm with the other four algorithms in terms of precision, recall, and reasoning time. The detection results on the RPMeter test set are shown in Table 1. It can be seen that at IoU50:95, the accuracy of YOLOX-DC is 0.996 and the recall is 0.997, which are higher than the other four algorithms. Although the reasoning time of YOLOX-DC is 0.04ms slower than that of YOLOX, its accuracy and recall are improved by 3.2% and 2.4% respectively. The experimental results further verify that the detection accuracy of the decoupled circular detection head for the pointer dial is higher than that of the rectangular detection head.

[0055] Table 1 Detection results of YOLOX-DC algorithm and other four algorithms on RPMeter test set

[0056] Subsequently, in order to verify the robustness and generalization of the proposed method, experiments were conducted on the "Meter Challenge (MC1296)" dataset and the "Paddle Meter" dataset. The experiments show that YOLOX-DC can process instrument images in various complex situations. Therefore, the above experiments have proved the effectiveness of the YOLOX-DC algorithm in pointer dial detection.

[0057] In this embodiment, a dial element segmentation experiment is performed to verify the performance of the PM-SwinUnet-based image segmentation model.

[0058] The input of PM-SwinUnet segmentation network is 224*224*3. YOLOX-DC dial detection method was used to obtain 1966 dial images using the minimum bounding rectangle. Among them, 1600 images were used for training and 366 images were used for evaluation. In order to verify the effectiveness of the proposed PM-SwinUnet image segmentation network. Under the same experimental environment, the PM-SwinUnet image segmentation network was compared with the VGG-Unet and U-Net models. For turntables with different illumination, overexposure and image blur, the PM-SwinUnet network obtained better segmentation results. Although the VGG-Unet segmentation network can correctly fit the scale region and pointer region, the segmentation results have small jagged edges and some segmented regions are not separated. The U-Net segmentation results not only have regional connectivity problems, but also have the problems of missing scale detection and incomplete pointer segmentation in overexposed environments. Compared with the other two networks, the results of PM-SwinUnet's pointer, scale and scale value have no regional connectivity problems, smooth edges, and no missed scale detection or incomplete pointer segmentation. Experiments show that the PM-SwinUnet image segmentation network proposed in the present invention solves the problems of missed detection of small-scale targets and incomplete segmentation of long-scale targets.

[0059] In this embodiment, through a pointer reading recognition comparison experiment, the reading recognition method proposed in the present invention is applied to different image segmentation networks, and the results are compared with manual readings to verify the robustness of the proposed reading recognition method.

[0060] When reading manually, according to the specification of pointer instruments, the reading should be controlled at 1 / 5 of the minimum scale. The detailed experimental results are shown in Table 1. As can be seen from Table 1, the recognition method proposed in this paper has good adaptability to the segmentation results of different image segmentation networks, with an average relative error of 0.0092 and an average reference error of 0.0026. Compared with the VGG-Unet and U-Net networks, there is a significant improvement. The maximum error between the recognition reading and the manual reading of this method is 0.793℃. Given that the error accuracy of the thermometer used in this experiment is ±1℃, this error level is within a reasonable range.

[0061] The recognition results of four low-quality instrument images, blur, tilt, overexposure and dim light, are compared to further verify the performance of the reading recognition method in processing low-quality instrument images. Figure 4 As shown. Figure 4 It can be seen that the scale and pointer of the instrument fit well, and the recognized readings are within the error range. It is worth noting that the method can accurately recognize the readings even in overexposed images. The specific recognition results of low-quality instrument images are shown in Table 2. The maximum relative errors of blurred images, tilted images, overexposed images, and low-light images are 0.0093, 0.0523, 0.0039, and 0.0094, respectively. The maximum reference errors are 0.0027, 0.0133, 0.0009, and 0.0057, respectively. Compared with the allowable error of ±0.01, the recognition errors of other low-quality images, except for tilted images, are within the acceptable range. Although the performance of this method on tilted images is reduced, it can be used for most normal instrument images with high recognition accuracy.

[0062] Table 2 Recognition results of low-quality instrument images

[0063] In order to further verify the generalization of the reading recognition method, experiments were conducted on the "Paddle Meter" dataset and the "MeterChallenge (MC1296)" dataset. The experimental results are shown in the figure. Figure 5As shown in the figure, this method can accurately fit the pointers and scales of various instruments and correctly identify the instrument readings. The specific recognition results are shown in Table 3, where the maximum reference errors of the three instruments in the "Paddle Meter" dataset are 0.0022, 0.0030, and 0.0087, respectively. The allowable errors of the three pointer instruments in the "Paddle Meter" dataset are 0.02, 0.02, and 0.03, respectively. The reference error of the proposed method is significantly smaller than the allowable error. The MC1296 dataset contains 7 types of pointer instruments, among which the reference error of the 5th image reaches 0.025. Compared with its allowable error of 0.05, the reference error is within a reasonable range. The experimental results show that the method has strong generalization performance.

[0064] Table 3 Verification results on the “Paddle Meter” dataset and the “Meter Challenge (MC1296)” dataset

[0065] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A method for automatic identification of industrial pointer instrument readings based on deep learning, characterized in that: The method comprises the following steps: S1. Collect pointer instrument image data using industrial cameras; S2. Using the instrument panel detection model based on YOLOX-DC, the instrument panel in the pointer instrument image data is detected; S3. Using the PM-SwinUnet-based image segmentation model, the pointer, key scales, and scale readings of the instrument panel are segmented to obtain the pointer mask image and the scale mask image; S4. Based on the pointer mask image and the scale mask image, use the skeleton refinement and center point extraction algorithm to remove edge pixels, and analyze the position of the pointer, the direction of the pointer and the position of the scale; S5. According to the position of the pointer, the direction of the pointer and the position of the scale, the final pointer instrument reading is obtained using the improved angle method to complete the automatic recognition of the pointer instrument reading.

2. The method for automatic identification of industrial pointer instrument readings based on deep learning according to claim 1 is characterized in that: The YOLOX-DC-based instrument panel detection model described in step S2 includes a backbone network, a neck network, and a detection head connected in sequence; The backbone network is used to receive the pointer instrument image data, perform feature extraction, and output features of the pointer instrument image data; The neck network is used to receive the features of the pointer instrument image data, perform feature fusion, and output fused features; The detection head is used to receive the fused features, perform classification and regression according to the fused features, and output the instrument panel in the pointer-type instrument image data.

3. The method for automatic identification of industrial pointer instrument readings based on deep learning according to claim 2 is characterized in that: The backbone network adopts a CSPDarkNet structure, including: a Focus module, a first CBS layer, a first CSP layer, a second CBS layer, a second CSP layer, a third CBS layer, a third CSP layer, a fourth CBS layer, an SPP layer and a fourth CSP layer connected in sequence; The output ends of the first CSP layer, the second CSP layer, the third CSP layer and the fourth CSP layer are also connected to the neck network; The Focus module includes four Slice layers connected in parallel, the output ends of the four Slice layers are all connected to the input end of the first Concat layer; the output end of the first Concat layer is connected to the input end of the fifth CBS layer; the output end of the fifth CBS layer is the output end of the entire Focus module.

4. The method for automatic identification of industrial pointer instrument readings based on deep learning according to claim 3 is characterized in that: The neck network includes a sixth CBS layer, a first upsampling layer, a second Concat layer, a CBS&CSP layer, a second upsampling layer, a third Concat layer, a fifth CSP layer, a first downsampling layer, a fourth Concat layer, a sixth CSP layer, a second downsampling layer and a Concat&CSP layer connected in sequence; Another input end of the second Concat layer is connected to the third CSP layer; another input end of the third Concat layer is respectively connected to the first CSP layer and the second CSP layer; another output end of the fifth CSP layer is connected to the detection head; another input end of the fourth Concat layer is connected to the CBS&CSP layer; another output end of the sixth CSP layer is connected to the detection head; another input end of the Concat&CSP layer is connected to the sixth CBS layer, and the output end of the Concat&CSP layer is connected to the detection head.

5. The method for automatic identification of industrial pointer instrument readings based on deep learning according to claim 4 is characterized in that: The detection head is a decoupled circular detection head, comprising: a first decoupled circular detection head, a second decoupled circular detection head, a third decoupled circular detection head, a first Reshape layer, a second Reshape layer and a third Reshape layer; The input end of the first decoupling circular detection head is connected to the output end of the fourth CSP layer, and the output end of the first decoupling circular detection head is connected to the input end of the first Reshape layer; The input end of the second decoupling circular detection head is connected to the output end of the fifth CSP layer, and the output end of the second decoupling circular detection head is connected to the input end of the second Reshape layer; The input end of the third decoupling circular detection head is connected to the output end of the first Concat&CSP layer, and the output end of the third decoupling circular detection head is connected to the input end of the third Reshape layer; The output end of the first Reshape layer, the output end of the second Reshape layer and the output end of the third Reshape layer together serve as the output end of the entire detection head; The detection head uses polar coordinates Positioning of the pointer instrument panel, the polar coordinates The calculation formula is: in, Indicates the polar diameter of the pointer instrument panel in the polar coordinate system. Indicates the polar angle of the pointer instrument panel in the polar coordinate system. Indicates the horizontal coordinate of the target location in the Cartesian coordinate system. Indicates the vertical coordinate of the target location in the Cartesian coordinate system; arctan is the inverse tangent function.

6. The method for automatic identification of industrial pointer instrument readings based on deep learning according to claim 1 is characterized in that: The step S4 of analyzing and obtaining the position and direction of the pointer specifically includes the following sub-steps: S41. Using a center point extraction algorithm based on a mask image, the center point of the pointer instrument image is extracted according to the scale mask image; S42. Extracting the pointer skeleton from the pointer mask image according to the skeleton refinement algorithm, and refining the pointer mask into straight lines at the unit pixel level; S43. Use Hough transform to detect straight lines in the pointer mask image. If a straight line is detected, traverse each straight line, extract the coordinates of the two endpoints of each straight line, and draw the straight line on the original image, and execute step S44; if no straight line is detected, output "pointer cannot be detected", and end the process; S44. The coordinates of the two endpoints are stored in endpoint list 1 and endpoint list 2, respectively, and the Euclidean distances from the two endpoints to the center point of the pointer instrument image are calculated; S45. Determine the endpoint closer to the center point of the pointer instrument image based on the Euclidean distance, obtain the starting point and the end point of the pointer line, and determine the position and direction of the pointer based on the starting point and the end point of the pointer line.

7. The method for automatic identification of industrial pointer instrument readings based on deep learning according to claim 6 is characterized in that: The step S41 specifically includes the following steps: S411. Performing a connected domain analysis on the scale mask image to identify connected regions in the scale mask image; S412. Traverse each connected area, filter out the connected areas with a number of pixels higher than a preset threshold, and generate a binary filtered image; S413. Extracting the scale contour from the binary filtered image using a contour detection algorithm, and storing the scale contour in a contour list; S414. Traverse each scale contour in the contour list to obtain the minimum circumscribed rectangle of each contour; S415. Extract the center point coordinates of each minimum circumscribed rectangle, convert them into integer type, and store them in the "center point coordinate list", thereby completing the extraction of the center point of the pointer instrument image.

8. The method for automatic identification of industrial pointer instrument readings based on deep learning according to claim 6 is characterized in that: The calculation formula of the Euclidean distance in step S44 is: in, represents the Euclidean distance, Indicates the horizontal coordinate of the center point of the pointer instrument image, represents the horizontal coordinates of the endpoints of the line, Indicates the ordinate of the center point of the pointer instrument image. Indicates the ordinate of the endpoint of the line.

9. The method for automatic identification of industrial pointer instrument readings based on deep learning according to claim 1 is characterized in that: The improved angle method in step S5 is specifically: A1. Take the center point of the instrument panel O Establish a Cartesian coordinate system for the origin; A2. In the Cartesian coordinate system, calculate the O Ray to 0 scale OA The ray where the pointer is located OP The angle between , and point O Ray to the second scale OB The ray where the pointer is located OP The angle between ; The calculation formula of the angle is: in, , and Respectively represent rays OA ,ray OP and rays OB The corresponding vector, Represents the length of a vector; arccos represents inverse cosine; A3. and angle Convert from radians to degrees to get the reading of pointer instruments , and its calculation formula is: in, Indicates the reading of the second scale.

10. The method for automatic identification of industrial pointer instrument readings based on deep learning according to claim 9 is characterized in that: The second scale reading It is identified by a scale value recognition module; the scale value recognition module is composed of multiple convolutional layers and a connection time series classification layer, and scale value recognition is achieved through parallel linear prediction.

Images