An automatic reading method for pointer instruments based on improved semantic segmentation network
Through improved semantic segmentation network and image correction technology, the accuracy and repair problems of scale lines and range digital extraction on pointer instruments are solved, and efficient and accurate instrument reading calculations are achieved on embedded devices.
Patent Information
- Application Number
- CN202111365897.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-18
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-11-18
AI Technical Summary
The prior art is difficult to accurately extract scale lines and range numbers on pointer instruments, and traditional methods lack scale repair capabilities, resulting in large reading errors and low working efficiency, and large model sizes and difficult to apply on embedded systems.
Using an improved semantic segmentation network, the binary graphs of scale lines, pointers and range numbers are obtained through semantic segmentation, image correction and reading calculation are performed, and the scale lines are repaired using ellipse fitting and perspective transformation, and combined with OCR and polar coordinate transformation to improve reading accuracy. Spend.
A small semantic segmentation network deployed on embedded devices is realized, which improves the accuracy and anti-interference ability of instrument readings, reduces operational complexity, and enhances the reliability of readings.
Smart Images

Figure CN114266881B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer processing, and more specifically, to an automatic reading method of a pointer-type instrument based on an improved semantic segmentation network. Background Art
[0002] Pointer instruments are widely used in power, metallurgy, chemical industry and other industries due to their simple structure, low cost and strong anti-interference performance. Faced with a large number of pointer instruments, manual readings will result in large reading errors, low work efficiency and possible endangerment to personal safety. Therefore, a method for automatic identification and accurate reading of pointer instruments based on computer vision technology has certain practical value.
[0003] Existing meter recognition methods mainly use traditional pattern recognition to achieve meter readings through a series of steps such as image preprocessing, target area detection, pointer recognition, and reading calculation. Although the specific implementation steps of the existing methods are different, the overall process and the technology used are similar. For example, in the document "Pointer-type meter reading recognition system in inspection robots", Xu Li extracted the pointer area through the target segmentation method, then located the pointer through hough transformation, and finally binarized the image, using the relationship between the angle of the pointer center line relative to the initial scale of the meter range and the range to identify the meter reading.
[0004] However, the existing traditional methods of meter recognition are difficult to accurately extract useful information from the dial due to interference from imaging vision, lighting, complex background and noise. Therefore, it is very necessary to perform accurate semantic extraction through a trained neural network model before calculating the meter readings.
[0005] For example, the Chinese patent application number CN202010114874.5 records a neural network-based instrument dial correction method, which locates the dial scale numbers through a neural network, fits an ellipse according to the center of the number, and then determines the feature points according to the ellipse parameters, and then corrects the instrument dial through perspective transformation. However, due to some contamination in actual work, which may cause the extracted scale lines to be missing or added, this type of method lacks the operation of repairing the scale lines, which seriously affects the final effect. At the same time, the network model used in this type of method is large in size, and it is difficult to apply it to various embedded systems on a large scale.
[0006] Therefore, a method that can accurately obtain the semantic information of the instrument pointer, scale lines, and range numbers, and can repair the scale lines during automatic reading calculation to improve reading accuracy becomes very necessary. Summary of the invention
[0007] In view of this, in order to solve the above problems, the present invention provides a method for automatically identifying pointer-type instrument readings, which can accurately obtain semantic information of instrument pointers, scale lines, and range numbers, and can repair the scale lines during automatic reading calculation to improve reading accuracy.
[0008] The technical scheme of the present invention comprises: S1: semantic segmentation, obtaining the original image of the target instrument and inputting it into the target neural network model to obtain the semantic prediction image, and respectively obtaining the binary images of the scale lines, pointers and range numbers from the target semantic prediction image; S2: image correction, fitting an ellipse through the coordinates of the center points of each scale contour in the binary image of the target scale line, establishing the perspective transformation of the ellipse and the standard circle, projecting the target instrument image and the binary images of the target scale lines, pointers and range numbers onto the plane where the target standard circle is located, to obtain the corrected image; S3: reading calculation, identifying the corrected range number binary image and the instrument image to obtain the start and end scale readings, obtaining the polar coordinate expansion image of the corrected scale line and pointer binary image, locating the positions of each scale line and pointer in the expansion image, repairing the target scale line, and calculating the relative position of the pointer and the scale line to obtain the final reading of the instrument.
[0009] Further, step S2 specifically includes: S21, performing a morphological corrosion operation on the target scale line binary image, filtering out external noise points using the size and position characteristics of the contour, and fitting an ellipse using the least squares method using the center point coordinates of the remaining contour; S22, obtaining the center point coordinates of the ellipse and the lengths of the major and minor axes and the endpoint coordinates, and establishing a standard circle covering the target ellipse with the major axis of the target ellipse as the diameter; S23, establishing a perspective transformation of the target ellipse and the standard circle, and projecting the target instrument image and the binary image of the target scale line, pointer, and range number onto the plane where the target standard circle is located to obtain the corresponding correction image.
[0010] Furthermore, in step S3, the corrected digital binary image of the measuring range and the original image of the instrument are identified, specifically: the corrected digital binary image of the target measuring range and the original image of the instrument are identified by an OCR algorithm to obtain the start and end scale readings of the target image.
[0011] Further, in step S3, a polar coordinate expansion diagram of the corrected scale line and pointer binary image is obtained, which specifically includes: denoising the corrected images of the scale lines and the pointer, and merging the denoised scale line and pointer binary images into the same binary image; using polar coordinate transformation to expand the target binary image into a rectangle to obtain the corresponding expansion diagram.
[0012] Further, in step S3, each scale line and pointer position is located in the expanded image, and the target scale line is repaired, which specifically includes: extracting the center line of the pointer and the scale line in the expanded image through an image thinning algorithm; obtaining a reference value of the distance between the real scale lines by statistically analyzing the set of adjacent scale line distances, and filling in the missing scale lines in the expanded image or deleting the redundant scale lines according to the target reference value.
[0013] Furthermore, before step S1, the method further includes: collecting multiple pictures of instruments, annotating multiple semantic labels in the instrument pictures, obtaining corresponding Gaussian heat map label pictures, and constructing an instrument image dataset; using the instrument image dataset, using an error back propagation algorithm to train an image semantic segmentation network; wherein the semantic labels in the instrument pictures at least include the 0 scale position point of the instrument, the range digital marking point in the instrument, the pointer fixed point in the instrument, and the pointer top position point in the instrument.
[0014] Furthermore, the annotation of multiple semantic labels in the instrument image in the previous step specifically includes: extracting the semantic labels of the scale lines, pointers and range numbers in the instrument image, annotating and filling the range number area in the form of a rectangle; binarizing the semantic labels of the instrument image and detecting the outline of each semantic label, and respectively calculating the corresponding minimum enclosing rectangle; respectively establishing the perspective transformation of the enclosing rectangle and the target square, and performing a standard Gaussian heat map projection on the semantic labels of the scale lines, pointers and range numbers through the perspective transformation to obtain the annotated Gaussian heat map label image.
[0015] Furthermore, the neural network model in step S1 includes: a first-stage subnetwork Stage1, which is used to obtain the original Figure 1 / 2 downsampling size feature map; the second stage sub-network Stage2 and the third stage sub-network Stage3 are used to obtain the original Figure 1 / 4, 1 / 8 down-sampling size feature map, wherein the Stage1, Stage2, and Stage3 are sequentially connected; a Gaussian heat map regression network is used to up-sample the feature map output by Stage1, match the number of channels with Stage3, map it to the corresponding Gaussian heat map value range [0,1] interval, and finally perform weighted fusion with the output feature map of Stage3 to obtain the feature map to be predicted; a semantic classification network is used to perform semantic prediction and classification on the target feature map to be predicted, and obtain a semantic prediction map containing semantic information such as scale lines, pointers, and range numbers.
[0016] Beneficial effects of the present invention:
[0017] 1. The improved semantic segmentation network model used in the present invention is small in size and easy to deploy in embedded terminal devices. 2. The present invention adds an attention module and a Gaussian heat map on the basis of the semantic segmentation network model, uses the Gaussian heat map label to train the attention module to learn the spatial weights of the pixels corresponding to the semantic elements of the pointer-type instrument image, and integrates it with the feature map of the original semantic segmentation network to improve the segmentation accuracy of the model, thereby improving the accuracy of the semantic information extracted by the improved network from the original image. 3. The present invention uses an image refinement algorithm to extract the center line of the pointer and the scale line in the correction image. Compared with the traditional method of locating the pointer by hough transform, it reduces the complexity of the operation while ensuring accuracy. In addition, the center line of the refined scale line is also easy to repair. It is then convenient to obtain a reference value of the distance between the real scale lines by statistical analysis of the set of adjacent scale line distances, thereby filling in the missing scale lines in the correction image or deleting the redundant scale lines, thereby improving the anti-interference ability of the present method and further improving the accuracy of the instrument reading calculation. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0019] Figure 1 A flowchart of the main steps of the method provided in this application;
[0020] Figure 2 The overall flow chart of the method provided in this application;
[0021] Figure 3 This is a diagram of the process of generating Gaussian heat map labels for instrument images in this application;
[0022] Figure 4 A structural reference diagram of a neural network model of the method provided in this application;
[0023] Figure 5 This is a flow chart of step S2 of the method provided in this application;
[0024] Figure 6 -a is a diagram showing the dial ellipse fitting result in the embodiment;
[0025] Figure 6 -b is a schematic diagram of perspective transformation principle in the embodiment;
[0026] Figure 7 Schematic diagram of the results of image correction and denoising in the embodiment;
[0027] Figure 8 This is a polar coordinate transformation effect diagram of the target binary image in step S24 of this embodiment;
[0028] Fig. 9 This is a diagram showing the positioning and repairing effects of the scale lines and pointer in step S3 of this embodiment;
[0029] Fig.10 This is a graph showing the ablation comparison experiment results of the image semantic segmentation model of the embodiment;
[0030] Fig.11 The result display table of the pointer meter reading experiment of the embodiment is shown. DETAILED DESCRIPTION
[0031] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
[0032] In the following introduction, the terms "first" and "second" are only used for descriptive purposes and are not to be understood as indicating or implying relative importance. The following description provides examples and does not limit the scope, applicability or examples set forth in the claims. Changes may be made to the functions and arrangements of the elements described without departing from the scope of the present application. Various processes or components may be appropriately omitted, substituted or added to each example. For example, the described method may be performed in an order different from the order described, and various steps may be added, omitted or combined. In addition, the features described in some examples may be combined in other examples.
[0033] The embodiment of the present application provides a method for automatic reading of a pointer-type instrument, which can be applied to an image processing device, which can be an independent device, or can be integrated into an electronic terminal or other device with image processing capabilities. Optionally, the operating system carried by the device or electronic terminal integrating the image processing device can be a Windows system, an Android system, or other operating systems, which are not limited here. In addition, the semantic segmentation network used in the present invention is not limited to the improved network provided in the embodiment of the present method, and other networks with the same functions can also be used.
[0034] Embodiment 1
[0035] See also Figure 1 and Figure 2 In an embodiment of the present application, the method includes steps S1 to S3.
[0036] Step S1, semantic segmentation, obtain the original image of the target instrument and input it into the target neural network model, and then obtain the semantic prediction map, so as to obtain the binary images of the scale line, pointer, and range number from the target semantic prediction map.
[0037] In addition, before executing step S1, it also includes: collecting multiple pictures of instruments, annotating multiple semantic labels in the instrument pictures, obtaining corresponding Gaussian heat map label pictures, and then constructing an instrument image data set; using the instrument image data set, using the error back propagation algorithm to train the image semantic segmentation network; wherein the semantic labels in the instrument pictures at least include the 0 scale position point of the instrument, the range digital marking point in the instrument, the pointer fixed point in the instrument, and the pointer top position point in the instrument.
[0038] like Figure 3 As shown, labeling multiple semantic labels in the instrument image specifically includes: extracting the semantic labels of the scale lines, pointers and range numbers in the instrument image, obtaining the corresponding pixel-level semantic label map, labeling and filling the range number area in the form of a closed quadrilateral. Figure 2 The contour of each semantic label is quantized and detected, and the corresponding minimum enclosing rectangle is calculated respectively; the perspective transformation of the enclosing rectangle and the target square is established respectively, and then the two-dimensional isotropic standard Gaussian heat map is projected through the perspective transformation to obtain the Gaussian heat map label, so as to obtain the annotated Gaussian heat map label image.
[0039] In this embodiment, the semantic components in the instrument image are annotated at the pixel level using Labelme software, and a pixel-level semantic label map is preliminarily obtained. In addition, in the generated pixel-level semantic label map, the value of each pixel corresponds to its semantic category. The semantic categories used in the present invention include three categories: scale lines, pointers, and range numbers. Other pixels are background categories, so there are a total of four types of pixels. In addition, in the embodiment of the present invention, the pixel-level semantic label only marks the position of the pixel corresponding to the semantic element of the instrument, while the Gaussian heat map label can indicate the importance of the pixel corresponding to the semantic element, which decreases from the center to the periphery. Therefore, the present invention uses Gaussian heat map labels to train the attention module to learn the spatial weights of the pixels corresponding to the semantic elements of the pointer-type instrument image, and fuses it with the feature map of the original semantic segmentation network to improve the segmentation accuracy of the model.
[0040] In addition, in this embodiment, when the error back propagation algorithm is used to train the image semantic segmentation network, the total loss function used is as follows:
[0041] L=L cls +λL mse
[0042] Among them, L represents the total loss of the model, L cls Represents the loss function of pixel classification results, L mse represents the loss function of the Gaussian heat map regression result, and λ represents the balance coefficient of the two types of loss functions.
[0043] Pixel classification uses cross entropy loss to evaluate the difference between the model prediction result and the semantic label. The calculation formula is:
[0044]
[0045] Among them, N is the total number of samples, M is the number of categories, and y c is a 1-dimensional vector whose elements can take only two values: 0 and 1. If the sample and the label have the same category, it takes 1, otherwise it takes 0. c Represents the probability that the predicted sample belongs to category c. L cls The smaller it is, the more accurate the pixel classification result is.
[0046] Gaussian heat map regression uses mean square error loss to evaluate the model's fit to the Gaussian heat map label. The calculation formula is:
[0047]
[0048] Where N is the total number of samples, y i represents the model prediction value, Indicates the label value. mse The smaller it is, the more accurate the Gaussian heat map position regression is.
[0049] like Figure 4 As shown, in this embodiment, the neural network model used is improved on the basis of the original CGNet, and the attention mechanism and Gaussian heat map regression concept are introduced. The detailed features of the semantic elements of the instrument image are enhanced by feature fusion, and the classification layer is appropriately deepened when the pixel semantic category prediction is performed at the end. The CG-Net network model is a prior art, so it will not be explained in detail in this embodiment. The first stage sub-network Stage1, the second stage sub-network Stage2, the third stage sub-network Stage3, and the input injection mechanism (input injection) in this neural network model are all from the original CG-Net network model, so they will not be explained in detail. Figure 4 As shown, the model structure of this application specifically includes: the first stage sub-network Stage1, which is used to obtain the original Figure 1 / 2 downsampled feature map; the second stage sub-network Stage2 and the third stage sub-network Stage3 are used to obtain the original Figure 1 / 4, 1 / 8 downsampled feature maps, wherein Stage1, Stage2, and Stage3 are sequentially connected; a Gaussian heat map regression network is used to upsample the feature map output by Stage1, and then match the number of channels with Stage3, so as to map it to the corresponding Gaussian heat map value range [0,1] interval, and finally perform weighted fusion with the output feature map of Stage3 to obtain the feature map to be predicted; a semantic classification network is used to perform semantic prediction and classification on the target feature map to be predicted, and then obtain a semantic prediction map containing semantic information such as scale lines, pointers, and range numbers.
[0050] Among them, the first stage sub-network Stage1 consists of an input layer, three 3×3 convolutional layers and activation layers, and a batch normalization layer connected in sequence; the second stage sub-network Stage2 and the third stage sub-network Stage3 are respectively composed of 3 and 21 stacked guide modules CG Block; the input ends of the first stage sub-network Stage1, the second stage sub-network Stage2, and the third stage sub-network Stage3 are respectively connected to the same input injection mechanism (input injection), and the input injection mechanism (input injection) can connect the input image downsampled by Stage1 and Stage2 with its output feature map, respectively, and then input it into Stage2 and Stage3 to improve the information flow in the network. The guide module CG Block and input injection mechanism (input injection) of the second stage sub-network Stage2 and the third stage sub-network Stage3 in this neural network model are all from the original CG-Net network model, so no further explanation is given.
[0051] In addition, the Gaussian heatmap regression network in this embodiment includes an attention module SENet, two 1×1 convolutional layers, and a Gaussian heatmap connected in sequence. The output end of the first 1×1 convolutional layer is connected to a Sigmoid activation layer, which is then fused with the output end of the third stage subnetwork Stage3 through the Sigmoid activation layer, thereby connecting to the semantic classification network. The semantic classification network includes three 3×3 convolutional layers, an activation layer, a batch normalization layer, and an output layer connected in sequence.
[0052] like Figure 4 As shown in the example, the model retains the three subnetworks and input injection mechanism in CGNet. Stage 1 consists of three 3×3 convolutional layers to obtain the original Figure 1 / 2 downsampled feature map, Stage2 and Stage3 stack 3 and 21 CG Blocks respectively to get the original Figure 1 / 4, 1 / 8 downsampled feature maps. The input injection mechanism additionally concatenates the 1 / 2 and 1 / 4 downsampled input images with the output feature maps of the previous stage and sends them to Stage2 and Stage3 to improve the information flow in the network. In order to share the feature maps output by the Stage1 subnetwork and reduce the number of model parameters, a lightweight attention module SENet is introduced. Here, the feature map output by Stage1 is upsampled by 2 times and sent to SENet to obtain the attention feature map. In order to match the number of channels of the feature map output by Stage3, the number of channels of the attention feature map is transformed by 1×1 convolution, and then mapped to the Gaussian heatmap value range [0,1] interval by the Sigmoid activation function, and then fused with the upsampled 8 times Stage3 output feature map by corresponding element multiplication. In addition, the original CGNet only uses one 1×1 convolutional layer to predict pixel semantic categories. Considering that properly deepening the network can improve performance, the present invention uses three convolutional layers for prediction, including two depthwise separable convolutions (DSConv) and one standard convolution. It should be noted that Figure 4 There are activation layers and batch normalization layers after the convolutional layers. Due to the limited size of the picture, they cannot be fully drawn, so they are supplemented with text here.
[0053] Although the original network stage2 and stage3 both use the attention mechanism idea, it is mainly used to better learn the joint features of local features and global context. The present invention adds a SENet attention module branch after stage1, which is used to regress the Gaussian heatmap. The feature map learned under the supervision of this regression task can be used as a spatial weight to further perform weighted fusion with the output feature map of the original network, thereby further improving the semantic segmentation accuracy. Compared with stage2 and stage3, which are at a higher level, stage1 is at a shallow level of the original network, so the SENet attention module added by the present invention is also an effective supplement to the attention mechanism adopted by the original CGNet.
[0054] Step S2, image correction, fitting an ellipse through the coordinates of the center points of the contours of each scale line in the binary image of the target scale line, and establishing a standard circle corresponding to the ellipse according to a certain method, and then projecting the target instrument image and the binary image of the target scale line, pointer, and range number onto the plane where the target standard circle is located by a certain method, thereby obtaining a corrected image.
[0055] The following will be combined Figure 5 , Figure 6 Step S2 of this embodiment is explained.
[0056] like Figure 5 As shown, step S2 specifically includes:
[0057] S21. Perform morphological corrosion operation on the binary image of the target scale line, and then use the size and position characteristics of the contour to filter out external noise points, so as to obtain an ellipse by fitting the center point coordinates of the remaining contour using the least square method. S22. Obtain the coordinates of the center point of the ellipse, the lengths of the major and minor axes, and the coordinates of the endpoints, and establish a standard circle covering the target ellipse with the major axis of the target ellipse as the diameter; S23. Establish the perspective transformation of the target ellipse and the standard circle, and project the binary image of the target instrument image and the target scale line, pointer, and range number onto the plane where the target standard circle is located to obtain the correction image.
[0058] Among them, Figure 6 As shown, establishing the perspective transformation between the ellipse and the circle requires four sets of corresponding point coordinates. Since scaling and rotation do not affect the reading of the pointer instrument, the present invention adopts an effective method for determining corresponding points. Figure 6 -a shows the ellipse fitting results, Figure 6 -b shows 4 sets of corresponding points between the ellipse and the circle <E i , C i >(i=0,1,2,3) selection method, let the two end points of the major axis of the ellipse be E0 and E1, the two end points of the minor axis be E2 and E3, take the center point O of the ellipse as the center, and the length of the major axis of the ellipse as the diameter to make a standard circle, which intersects with the ellipse at points C0 and C1 (i.e. coincides with E0 and E1 respectively), and the straight line E2E3 where the minor axis of the ellipse is located extends outward to intersect the standard circle at points C2 and C3. The perspective transformation matrix is calculated using these four sets of corresponding point coordinates, and each point on the tilted distorted image can be transformed to the plane where the standard circle is located, thereby achieving image correction.
[0059] Further, combining Figure 7 , Figure 8 and Fig. 9 Step S3 of this embodiment is described in detail.
[0060] Step S3: Reading calculation, identify the calibrated range digital binary image and the original instrument image to obtain the start and end scale readings, obtain the polar coordinate expansion diagram of the calibrated scale line and pointer binary image, locate each scale line and pointer position in the expansion diagram, repair the target scale line, and calculate the relative position of the pointer and the scale line to obtain the final reading of the instrument.
[0061] In this embodiment, in step S3, the corrected target range digital binary image and the original instrument image are specifically identified by an OCR algorithm, thereby obtaining the start and end scale readings of the target image. The OCR algorithm generally determines the position and shape of the characters in the image through image preprocessing such as binarization, grayscale, and filtering, and then converts the characters in the image into text using algorithms such as template matching and feature classification, and finally outputs all the texts to the computer in sequence.
[0062] In addition, in this embodiment, the polar coordinate expansion image of the corrected scale line and pointer binary image is obtained in step S3, which specifically includes: denoising the corrected image of the scale line and the pointer, and merging the denoised scale line and pointer binary images into the same binary image; using polar coordinate transformation to expand the target binary image into a rectangle to obtain the corresponding expansion image.
[0063] like Figure 7 As shown in the figure, on the corrected image, the noise caused by inaccurate semantic segmentation is further removed by using constraints such as the properties of the circle and the contour area. The upper and lower sub-figures are the binary images of the scale line, pointer, and range number after correction and the corresponding denoising results. Among them, the semantic segmentation result of the pointer may be broken. As long as the denoising result retains the section that best reflects its characteristics, in this embodiment, the longest section is selected. As long as it points correctly, it will not affect the calculation of the final reading.
[0064] In addition, in step S3 of this embodiment, the de-noised scale line and the pointer binary image are merged into the same binary image, and the target binary image is expanded into a rectangle using polar coordinate transformation. A Cartesian coordinate system is established with the center of the standard circle as the origin, and the radius of the standard circle is selected as the transformation radius. The transformation is carried out clockwise from the positive half axis of the Y axis so that the starting scale line of the dial is located on the left side of the transformed image. The effect before and after the polar coordinate transformation is shown in the figure. Figure 8 shown.
[0065] like Fig. 9As shown, in step S3, the center line of the pointer and the scale line in the correction image is extracted by Zhang thinning algorithm; the reference value of the distance between the real scale lines is obtained by statistical analysis of the distance set of adjacent scale lines, so as to fill the missing scale lines in the correction image or delete the redundant scale lines according to the target reference value. Compared with the traditional method of locating the pointer by hough transform, the complexity of the operation is reduced while ensuring accuracy, and the repair of the scale lines also makes the subsequent pointer reading calculation more accurate.
[0066] In this embodiment, the reading of the pointer meter can be obtained according to the relative position relationship between the scale line and the pointer, the range size, and the number of scale lines. Assuming that the total number of complete scale lines is N, and the current pointer points between scale line i and scale line i+1 (i=0,1..,N), the normalized meter reading calculation formula is:
[0067]
[0068] Where d is the distance between the pointer and the scale line i, and D is the distance between the scale line i+1 and the scale line i. is the normalized graduation value. Let a and b be the range numbers corresponding to the start and end scales respectively, then the final reading of the pointer meter is: R = a + R norm ×(ba).
[0069] like Fig.10 , Fig.11 As shown, this embodiment also provides the experimental results display and analysis of this method, and the experimental results display and analysis of this embodiment include: (1) image semantic segmentation model ablation comparison experiment; (2) range digital recognition experiment; (3) pointer instrument reading experiment. The experimental results display provided in the embodiment of this application is to better demonstrate the actual effect of this application. The effect display of the method provided in this application is not limited to the above experiments, and can also be demonstrated by other experimental display methods in practice.
[0070] In the embodiments of the present application, mean intersection over union (mIoU) and pixel accuracy (PA) are used as quantitative indicators to measure the performance of semantic segmentation of instrument images.
[0071] mIoU is defined as the ratio of the intersection and union of the two sets of true values and predicted values of the image pixel category:
[0072]
[0073] Among them, K is the number of label categories, TP is the number of positive samples correctly predicted, FP is the number of negative samples predicted as positive samples, and FN is the number of positive samples predicted as negative samples.
[0074] PA is defined as the ratio of correctly classified pixels in the test set to all pixels:
[0075]
[0076] Among them, K represents the number of label categories, p ij It indicates the number of pixels of category i classified as pixels of category j.
[0077] like Fig.10 As shown, the image semantic segmentation model ablation comparison experiment of this application is as follows:
[0078] This embodiment conducts ablation comparison experiments to verify the effectiveness of the CGNet improvement strategy of the present application method, wherein comparison model A is a CGNet improved by using three convolutional layers as classification layers, comparison model B is a CGNet improved by directly inserting the attention mechanism on the basis of comparison model A, and comparison model C is a CGNet improved by introducing the attention mechanism by adding branches on the basis of comparison model A but without using Gaussian heat map regression. Fig.10 From the comparison of parameters, it can be seen that the method of this application introduces the attention mechanism by adding branches and adopts the Gaussian heat map regression idea to effectively improve the segmentation accuracy and achieve the highest mIoU and PA values.
[0079] The range digital recognition experiment of this embodiment is as follows:
[0080] According to the semantic segmentation prediction results of the instrument image, the sub-image corresponding to the range digital area is intercepted from the corrected instrument original image, and then grayed and binarized. The digital is further segmented into single digit characters and decimal points by vertical projection segmentation. The single digit character recognition uses a trained convolutional neural network model.
[0081] In this experimental test, there are 572 groups of range numbers, including 934 single-digit characters, and the recognition accuracy of single-digit characters (0-9) is as high as 99.79%. However, due to the poor binarization effect of a few images due to blur or noise interference, a small number of digits and decimal points are recognized incorrectly, making the overall recognition accuracy of range numbers by group 97.88%. However, since most of the range numbers on the same dial are recognized correctly, their numerical values are sorted and further verified using their arithmetic arrangement relationship, which can ensure the accuracy of all range numbers and use them for final reading calculation.
[0082] The pointer meter reading experiment of this embodiment is as follows:
[0083] This experiment verifies the applicability and stability of the proposed method in the automatic reading of pointer meters. The proposed method is used to calculate the readings of the instrument images in the test set, and the manual reading results are used as the true value to calculate the error between the program reading and the manual reading. The evaluation index is expressed as relative error:
[0084]
[0085] Wherein, m is the instrument reading value of the method of this embodiment, v is the manual reading value, and r is the instrument range.
[0086] The experimental results are as follows Fig.11 As shown in FIG. 1 , the experimental results of some typical test images are listed, wherein the reading values are all retained to two decimal places. It can be seen from the experimental results that the method used in this embodiment has a small relative error in practical application and can meet the accuracy requirements of pointer instrument reading measurement.
[0087] In addition, the model used in this application can achieve an inference speed of up to 27FPS, which can meet the needs of most real-time applications. The model size of only 2.7M is also very suitable for deployment on embedded terminal devices.
[0088] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of the implementation scheme of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not recorded in the present disclosure. The description and examples are regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.
Claims
1. A pointer instrument automatic reading method based on an improved semantic segmentation network, characterized in that: The method comprises: S1: semantic segmentation, obtaining the original image of the target instrument and inputting it into the target neural network model to obtain a semantic prediction image, and respectively obtaining binary images of scale lines, pointers, and range numbers from the target semantic prediction image; S2: image correction, fitting an ellipse through the coordinates of the center points of each scale outline in the binary image of the target scale line, and establishing a standard circle corresponding to the ellipse, and projecting the target instrument image and the binary images of the target scale lines, pointers, and range numbers onto the target standard circle to obtain a corrected image; S3: Reading calculation, identify the corrected range digital binary image and instrument image to obtain the start and end scale readings, obtain the polar coordinate expansion image of the corrected scale line and pointer binary image, locate the scale line and pointer position in the expansion image, repair the target scale line, calculate the relative position of the pointer and the scale line, and obtain the final reading; the neural network model in step S1 includes: the first stage sub-network Stage1, the second stage sub-network Stage2, and the third stage sub-network Stage3 are connected in sequence, respectively used to obtain feature maps with 1 / 2, 1 / 4, and 1 / 8 downsampling sizes of the original image, the Gaussian heat map regression network is used to upsample the feature map output by Stage1, match the number of channels with Stage3, map it to the corresponding Gaussian heat map value range [0,1] interval, and finally perform weighted fusion with the output feature map of Stage3 to obtain the feature map to be predicted, and the semantic classification network is used to perform semantic prediction and classification on the target feature map to be predicted, obtain the scale line, pointer , a semantic prediction map of the range digits; in the step S3, the corrected range digit binary map and the original instrument image are identified, specifically: the corrected target range digit binary map and the original instrument image are identified by the OCR algorithm to obtain the start and end scale readings of the target image; in the step S3, the polar coordinate expansion map of the corrected scale line and pointer binary map is obtained, specifically including: denoising the corrected image of the scale line and the pointer, merging the denoised scale line and pointer binary map onto the same binary map, using polar coordinate transformation to expand the target binary map into a rectangle, and obtaining the corresponding expansion map; in the step S3, the position of each scale line and pointer is located in the expansion map, and the target scale line is repaired, specifically including: extracting the center line of the pointer and the scale line in the expansion map by the image thinning algorithm, obtaining the reference value of the distance between the real scale lines by statistically analyzing the set of distances between adjacent scale lines, and filling the missing scale lines in the expansion map or deleting the redundant scale lines according to the target reference value.
2. The method according to claim 1, characterized in that The step S2 specifically includes: S21, performing a morphological corrosion operation on the target scale line binary image, filtering out external noise points using the size and position characteristics of the contour, and fitting an ellipse using the least squares method using the center point coordinates of the remaining contour; S22, obtaining the center point coordinates of the ellipse and the lengths of the major and minor axes and the endpoint coordinates, and establishing a standard circle covering the target ellipse with the major axis of the target ellipse as the diameter; S23, establishing a perspective transformation to project the target instrument image and the binary image of the target scale line, pointer, and range number onto the target standard circle to obtain a corrected image.
3. The method according to claim 1, characterized in that Before step S1, the method also includes: collecting multiple pictures of instruments, annotating multiple semantic labels in the instrument pictures, obtaining corresponding Gaussian heat map label pictures, and constructing an instrument image data set; using the instrument image data set, using an error back propagation algorithm to train an image semantic segmentation network; wherein the semantic labels in the instrument pictures at least include the 0 scale position point of the instrument, the range digital marking point in the instrument, the pointer fixed point in the instrument, and the pointer top position point in the instrument.
4. The method according to claim 3, characterized in that The labeling of multiple semantic labels in the instrument image specifically includes: extracting the semantic labels of the scale lines, pointers and range numbers in the instrument image, labeling and filling the range number area in a rectangular form; binarizing the semantic labels of the instrument image and detecting the outline of each semantic label, and respectively calculating the corresponding minimum circumscribed rectangle; respectively establishing a perspective transformation of the circumscribed rectangle and the target square, and performing a standard Gaussian heat map projection on the semantic labels of the scale lines, pointers and range numbers through the perspective transformation to obtain a labeled Gaussian heat map label image.
