Pointer type instrument intelligent reading method and system based on image segmentation

Abstract

The invention discloses a pointer type instrument intelligent reading method and system based on image segmentation, and the method comprises five parts: image collection, image preprocessing, dial image segmentation, automatic reading, and optical character recognition, and comprises the steps: firstly segmenting the scale and pointer of a dial based on an improved U-net method; the method comprises the following steps: segmenting a dial plate, inputting a segmented result into a reading module designed in the invention to determine the relative position of a pointer and a scale, detecting key information on the dial plate from an original image through an optical character recognition (OCR) technology, obtaining the maximum range of the dial plate through a set threshold value and a limiting condition, and realizing reading through the matching of the processes. According to the method, the problem of data acquisition in the detection process of pointer type instruments and meters is solved, reading of dial pointers of different specifications is achieved, the stability and robustness of the method are excellent, the recognition accuracy is 99% or above, and the method is of great significance to development of the industries of metering instruments and meters such as aerospace and hydraulic and pneumatic pipelines.

Description

Technical Field

[0001] The invention relates to the technical field of instrument detection, and more specifically to an intelligent reading method of a pointer-type instrument based on image segmentation. Background Art

[0002] With the development of physical manufacturing industry and the continuous improvement of instruments, pointer-type instruments are still used in various fields due to their low manufacturing cost, simple structure, and intuitive response to changing trends. However, a manufacturing workshop may contain several or even dozens of instruments with different scales and produced by different manufacturers. At this time, if manual meter reading is used, not only is the work efficiency low, but it is also easy to cause misreading during the long meter reading process. The existing instrument pointer detection and reading methods have problems such as large detection errors, numerous recognition steps, poor robustness of the identification needle, and poor ability to resist external interference. The recognition of pointer-type instruments through deep learning methods not only ensures accuracy, but also saves manpower consumption. However, most of the commonly used deep learning meter reading methods are based on YOLO. The robustness of these methods is relatively poor, and it is difficult to meet the recognition of different types of instruments in a manufacturing workshop. Therefore, an intelligent reading method for pointer-type instruments based on image segmentation is provided. Summary of the invention

[0003] The present invention solves the problems of large detection error, numerous identification steps, poor robustness of identification of meter pointers, and dark light pointer meter reading in existing meter pointer detection and reading methods;

[0004] In order to achieve the above object, the present invention provides a pointer instrument intelligent reading method based on image segmentation, comprising the following steps:

[0005] Step S1: Capturing the pointer image, the camera transmits the captured image to the local computer in real time via the network;

[0006] Step S2: pre-processing the collected pointer instrument dial image, performing image enhancement, tilt correction, image segmentation, and morphological processing on the image;

[0007] Step S2-1: The preprocessing method includes: compressing the image brightness range and enhancing the image contrast by frequency domain filtering, enhancing the image with uneven illumination, and enhancing the pointer in the image; rotating and scaling the image by affine transformation, correcting the tilt phenomenon in the image, and keeping the instrument panel image horizontal; using edge detection method to detect the edge information of the instrument contour to determine the boundary of the target area, and separating the instrument dial in the image from the background;

[0008] Step S3: semantically segment the pointer and scale of the input image using the improved U-net model semantic segmentation technology. The prediction map obtained by image segmentation contains both position information and feature information.

[0009] Step S3-1: The improved U-net model replaces the U-net maximum pooling layer with a convolutional layer. According to the convolution calculation formula:

[0010]

[0011] In the formula, the input image size is W input ×H input , the output image size is W output ×H output , P is padding, S is step size, and convolution kernel size is W filter ×H filter ;

[0012] Step S3-2: The semantic segmentation technology uses a labeled dataset to divide the dataset into a training set, a test set, and a validation set in proportion. The dataset is expanded using random noise and random flipping in data preprocessing. The cross entropy loss function and Adam optimization method are used to perform multiple rounds of training on a server with a GTX 2070super8G.

[0013] Step S4: Input the segmented result into the reading module of the method to determine the relative position of the pointer and the scale. First, the segmented image is subjected to morphological corrosion by the reading module, and then the pointer is fitted with a straight line to perform scale detection. Secondly, it is determined whether the scale and the pointer coincide with each other, and the number of scales swept by the pointer is determined. Then, the scale percentage and the number of scales are obtained.

[0014] Step S4-1: The process of pointer straight line fitting after image morphological corrosion is as follows:

[0015] First, the segmented image is morphologically eroded, and three eroded convolution kernels are compared. Finally, the elliptical convolution kernel is selected. The eroded image retains more accurate position information based on the original label. After corrosion, the pixel points with a pixel value of 1 on the image are statistically probabilistic Hough line transform for straight line fitting. For Hough transform, the polar coordinate system is used to represent the straight line, and the expression of the straight line can be simplified as:

[0016] r=x cosθ+y sinθ

[0017] In the formula, r is the distance of the polar radius, that is, the vertical distance from the straight line to the origin, θ is the angle between the vertical line and the x-axis, x is the intersection of the vertical line and the x-axis, and y is the intersection of the vertical line and the y-axis. For a point (x0, y0) on the image, a family of straight lines passing through this point can be uniformly defined as:

[0018] r θ =x0 cosθ+y0 sinθ

[0019] At this time, each pair (r θ , θ) represents a straight line passing through the point (x0, y0), and a sine curve will be obtained. This operation is performed on all points in the image. If there are points drawn that intersect in the polar coordinate polar diameter and polar angle plane, it means that they pass through the same straight line;

[0020] Line transformation fits the longest straight line with a pixel value of 1, sets the endpoint of the line segment close to the center point of the figure to A, and the other endpoint to B;

[0021] Step S4-2: Scale detection is to detect the points with a pixel value of 2 through breadth-first search BFS, set the points with a pixel value of 2 adjacent to each scale in each cluster as a class, and finally output all classes, detect the point with the largest distance from point A for each class, and save the position information of all points into a new list L;

[0022] Step S4-3: The process of judging whether the scale and the pointer coincide with each other is as follows;

[0023] First, determine the vector AB, pointing to B, and determine whether the straight-line distance from each point in the list L to 1.5 times the vector AB is greater than the error b=2, that is, two pixel values. If it is greater, it means that the pointer and the scale do not coincide. Then find the two points closest to B in the list L, record them as C and D. At this time, we only know that C and D are on both sides of the straight line AB. Convert the straight line AB into a new function f=ax+by+c, bring the coordinates of C and D into f, set the coordinates greater than 0 to C, and set the coordinates less than 0 to D. At this time, C is on the left side of the straight line AB, and D is on the right side of the straight line AB.

[0024] Generate vector CA and vector AD, calculate the arc cosine by vector dot multiplication to get the size of ∠CAB and ∠CAD, and find the proportion of ∠CAB in ∠CAD, and calculate the proportional size of the pointer in one scale;

[0025] Step S4-4: The process of determining the scale percentage and the scale number is as follows;

[0026] Remove points C and D from list L, re-search the point closest to point C in list L, set the found point as F and remove F from list L, then search for the point closest to F in list L, search sequentially, count once each time F is removed, stop when the distance between F and an element in list L is greater than 2 times the CD distance, and get the number of points removed M;

[0027] The first scale of common pointer pressure gauges is generally small, but in some instruments with smaller ranges, the first scale is the same as other scales. In order to improve the robustness of the entire module, restrictions are imposed when making labels. If the first scale is less than 0.25 times the other scales, the second scale line is omitted from the label. If it is greater, the second scale line is drawn to prevent scale adhesion. At this time, if the second scale line is not drawn, the first scale must be greater than the other scales. If it is drawn, it must be less than or equal to. Therefore, when reading, if the first scale is greater than CD, then m=M+1, and the number of list L is l=L. Otherwise, m=M, and the number of list L is l=L-1. The mathematical expression of the overall reading is:

[0028]

[0029] Where R represents the range;

[0030] If there is a point in list L whose straight line distance to 1.5 times vector AB is less than error b, it means that the pointer and scale are collinear. Set this collinear point to Z, remove Z from list L, and search for the two points closest to B in L. The subsequent method is roughly the same as the above non-collinear method. The only difference is that there is no need to calculate angle B. After determining points C and D, point F can be directly found. The mathematical expression of the reading is:

[0031]

[0032] In the above formula, only the range R is unknown, so the maximum range of the dial is obtained through OCR technology;

[0033] Step S5: Using the optical character recognition (OCR) technology to detect key information on the dial of the original image, and obtaining the maximum range of the dial through the set threshold and restriction conditions;

[0034] The threshold setting should be that the upper limit of the instrument should be greater than the maximum value of the measured value, and the lower limit should be less than the minimum value of the measured value, and an appropriate margin should be left. The amount of margin should be determined according to the variation of the measured parameter; the setting limit condition is usually that 1 / 3 to 2 / 3 of the measuring range of the instrument is the fluctuation range of the measured value;

[0035] Step S5-1: The process of obtaining the maximum range using character recognition OCR technology is as follows:

[0036] The input image is detected through the text area detection in the optical character recognition (OCR) technology to determine the character position on the dial, and then the character information on the dial is obtained through text recognition. The detected character information is filtered to exclude non-pure digital information. Since the digital information of each large scale is a geometric sequence, all the pure numbers obtained are randomly subtracted and the absolute value is taken. The obtained values ​​are randomly ratioed to further eliminate the interference factors of the pure numbers in the dial. Finally, the maximum value of the pure number obtained is taken as the maximum range R.

[0037] In a preferred manner, the image acquisition in step S1 uses an industrial camera to shoot the original picture, and the image supports normal lighting and extremely dark conditions, and is effective for most pointer instruments on the market.

[0038] In a preferred manner, the pixel value of the pointer after segmentation in the prediction image is 1, the pixel value of the scale after segmentation is 2, and the pixel value of the background is 0.

[0039] In the preferred mode, the size of the convolution kernel determines the depth of the network, so when a smaller convolution kernel is selected, the input image size is ensured to be twice the output image.

[0040] In the preferred embodiment, the size of the convolution kernel in this method is 3×3, the step size is 2, and the padding is 1. The above parameters can replace the maximum pooling layer, retaining more image details and original features.

[0041] In the preferred method, the semantic segmentation technology uses a dataset of 748 labeled images, including 200 dial images under extremely dark conditions, and divides the dataset into a training set, a test set, and a validation set in a ratio of 7:2:1.

[0042] An intelligent reading system for pointer-type instruments based on image segmentation includes five parts: an image acquisition module, an image preprocessing module, an automatic reading module, a dial image segmentation module, and an optical character recognition module. The system is specifically divided into the following steps:

[0043] Step M1: The image acquisition module for capturing the pointer image, the camera transmits the captured image to the local computer in real time via the network;

[0044] Step M2: The image preprocessing module is implemented by preprocessing the collected pointer instrument dial image, performing image enhancement, tilt correction, image segmentation, and morphological processing on the image;

[0045] Step M2-1: The preprocessing method includes: first, compressing the image brightness range and enhancing the image contrast simultaneously by frequency domain filtering, enhancing the image with uneven illumination, and enhancing the pointer in the image; second, rotating and scaling the image by affine transformation, correcting the tilt phenomenon in the image, and keeping the instrument panel image horizontal; then, using the edge detection method to detect the edge information of the instrument contour to determine the boundary of the target area, and separating the instrument dial in the image from the background;

[0046] Step M3: The dial image segmentation module is implemented as follows: semantic segmentation is performed on the pointer and scale of the input image using the improved U-net model semantic segmentation technology. The prediction image obtained by image segmentation contains both position information and feature information, where the pixel value of the pointer after segmentation is 1, the pixel value of the scale after segmentation is 2, and the pixel value of the background is 0;

[0047] Step M3-1: The improved U-net model replaces the U-net maximum pooling layer with a convolutional layer. According to the convolution calculation formula:

[0048]

[0049] In the formula, the input image size is W input ×H input , the output image size is W output ×H output , P is padding, S is step size, and convolution kernel size is W filter ×H filter The size of the convolution kernel determines the depth of the network, so when a smaller convolution kernel is selected, the input image size is guaranteed to be twice the output image. According to the formula, the size of the convolution kernel is 3×3, the step size is 2, and the padding is 1. The above parameters can perfectly replace the maximum pooling layer, retaining more image details and original features.

[0050] Step M3-2: The semantic segmentation technology uses a dataset of 748 labeled images, including 200 dial images under extremely dark conditions. The dataset is divided into a training set, a test set, and a validation set in a ratio of 7:2:1. The dataset is expanded using random noise and random flipping methods in data preprocessing. The cross entropy loss function and Adam optimization method are used to train 200 rounds on a server with GTX 2070super 8G.

[0051] Step M4: The automatic reading module is implemented as follows: the segmented result is input into the reading module of the present method to determine the relative position of the pointer and the scale. First, the segmented image is subjected to morphological corrosion by the reading module, and then the pointer is fitted with a straight line to perform scale detection. Secondly, it is determined whether the scale and the pointer coincide with each other, and the number of scales swept by the pointer is determined. Then, the scale percentage and the number of scales are obtained.

[0052] Step M4-1: The process of pointer straight line fitting after image morphological corrosion is as follows:

[0053] First, the segmented image is morphologically corroded, and three types of corroded convolution kernels are compared. Finally, the elliptical convolution kernel is selected. The corroded image retains more accurate position information based on the original label. After corrosion, the pixel points with a pixel value of 1 on the image are statistically probabilistic Hough line transform for straight line fitting. For Hough transform, the polar coordinate system is used to represent the straight line, and the expression of the straight line can be simplified as:

[0054] r=xcosθ+ysinθ

[0055] r is the distance of the polar radius, that is, the vertical distance from the straight line to the origin, θ is the angle between the vertical line and the x-axis, x is the intersection of the vertical line and the x-axis, and y is the intersection of the vertical line and the y-axis. For a point (x0, y0) on the image, a family of straight lines passing through this point can be uniformly defined as:

[0056] r θ =x0cosθ+y0sinθ

[0057] At this time, each pair (r θ ,θ) represents a straight line passing through the point (x0, y0), and a sine curve will be obtained. This operation is performed on all points in the image. If there are points drawn that intersect in the polar coordinate polar diameter and polar angle plane, it means that they pass through the same straight line;

[0058] Line transformation fits the longest straight line with a pixel value of 1, sets the endpoint of the line segment close to the center point of the figure to A, and the other endpoint to B;

[0059] Step M4-2: Scale detection is to detect the points with a pixel value of 2 through breadth-first search BFS, set the points with a pixel value of 2 adjacent to each scale in each cluster as a class, and finally output all classes. For each class, detect the point with the largest distance from point A, and save the position information of all points into a new list L;

[0060] Step M4-3: Determine whether the scale and the pointer coincide with each other.

[0061] First, determine the vector AB, pointing to B, and determine whether the straight-line distance from each point in the list L to 1.5 times the vector AB is greater than the error b=2, that is, two pixel values. If it is greater, it means that the pointer and the scale do not coincide. Then find the two points closest to B in the list L, record them as C and D. At this time, we only know that C and D are on both sides of the straight line AB. Convert the straight line AB into a new function f=ax+by+c, bring the coordinates of C and D into f, set the coordinates greater than 0 to C, and set the coordinates less than 0 to D. At this time, C is on the left side of the straight line AB, and D is on the right side of the straight line AB.

[0062] Generate vector CA and vector AD, calculate the arc cosine by vector dot multiplication to get the size of ∠CAB and ∠CAD, and find the proportion of ∠CAB in ∠CAD, and calculate the proportional size of the pointer in one scale;

[0063] Step M4-4: The process of determining the scale percentage and the scale number is as follows;

[0064] Remove points C and D from list L, re-search the point closest to point C in list L, set the found point as F and remove F from list L, then search for the point closest to F in list L, search sequentially, count once each time F is removed, stop when the distance between F and an element in list L is greater than 2 times the CD distance, and get the number of points removed M;

[0065] At this point, it should be noted that the first scale of common pointer pressure gauges is generally small, but in some instruments with smaller ranges, the first scale is the same as other scales. In order to improve the robustness of the entire module, restrictions are imposed when making labels. If the first scale is less than 0.25 times the other scales, the second scale line is omitted from the label. If it is greater, the second scale line is drawn to prevent scale adhesion. At this time, if the second scale line is not drawn, then the first scale must be greater than the other scales. If it is drawn, it must be less than or equal to; therefore, when reading, if the first scale is greater than CD, then m = M + 1, and the number of list L is l = L. Otherwise, m = M, and the number of list L is l = L-1. The mathematical expression of the overall reading is:

[0066]

[0067] Where R represents the range;

[0068] If there is a point in list L whose straight line distance to 1.5 times vector AB is less than error b, it means that the pointer and scale are collinear. Set this collinear point to Z, remove Z from list L, and search for the two points closest to B in L. The subsequent method is roughly the same as the above non-collinear method. The only difference is that there is no need to calculate angle B. After determining points C and D, point F can be directly found. The mathematical expression of the reading is:

[0069]

[0070] In the above formula, only the range R is unknown, so the maximum range of the dial is obtained through OCR technology;

[0071] Step M5: Use the optical character recognition module OCR technology to detect key information on the dial of the original image, and obtain the maximum range of the dial through the set threshold and restriction conditions;

[0072] The threshold setting should be that the upper limit of the instrument should be greater than the maximum value of the measured value, and the lower limit should be less than the minimum value of the measured value, and an appropriate margin should be left. The amount of margin should be determined according to the variation of the measured parameter; the setting limit condition is usually that 1 / 3 to 2 / 3 of the measuring range of the instrument is the fluctuation range of the measured value;

[0073] Step M5-1: The process of obtaining the maximum range using character recognition OCR technology is as follows:

[0074] The input image is detected through the text area in the optical character recognition module (OCR) technology to determine the position of the characters on the dial, and then the character information on the dial is obtained through text recognition. The detected character information is filtered to exclude non-pure digital information. Since the digital information of each large scale is a geometric sequence, all the pure numbers obtained are randomly subtracted and the absolute value is taken. The obtained values ​​are randomly ratioed to further eliminate the interference factors of the pure numbers in the dial. Finally, the maximum value of the pure numbers obtained is taken as the maximum range R.

[0075] The beneficial effects of the present invention are as follows:

[0076] 1. This method does not rely on a specific form of input image, avoids the complex image preprocessing process in the early stage, can be deployed in different workshops, and meets the needs of practical applications.

[0077] 2. The pointer and scale of the dial can be automatically separated, and the image processing is performed through the reading module to determine whether the scale and the pointer coincide;

[0078] 3. This method has good stability and robustness for dials of different specifications and models;

[0079] 4. This method is applicable to both normal light and extremely dark conditions, and the readings are still accurate under extremely dark conditions;

[0080] 5. The accuracy of the instrument pointer reading test is better than the test results of other deep learning methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 It is an overall working flow chart of an embodiment of a pointer-type instrument intelligent reading method based on image segmentation of the present invention;

[0082] Figure 2 A network structure diagram after improving U-net in an embodiment of a pointer-type instrument intelligent reading method based on image segmentation of the present invention;

[0083] Figure 3 It is a loss value on a training set and a MIoU change graph on a test set in an embodiment of a pointer-type instrument intelligent reading method based on image segmentation of the present invention;

[0084] Figure 4 A flowchart of a reading module design in an embodiment of a pointer-type instrument intelligent reading method based on image segmentation according to the present invention;

[0085] Figure 5 A pointer fitting diagram in an embodiment of a pointer-type instrument intelligent reading method based on image segmentation of the present invention;

[0086] Figure 6 It is a scale position information diagram in an embodiment of a pointer-type instrument intelligent reading method based on image segmentation of the present invention;

[0087] Figure 7 FIG. a is a diagram showing the scale of the pointer and the pointer angle ratio in an embodiment of an intelligent reading method for a pointer-type instrument based on image segmentation of the present invention;

[0088] Figure 8 FIG. b is a diagram showing the scale of the pointer and the proportion of the pointer angle in an embodiment of an intelligent reading method of a pointer-type instrument based on image segmentation of the present invention;

[0089] Fig. 9 The scale through which the pointer passes and the first scale diagram a of the dial in an embodiment of an intelligent reading method of a pointer-type instrument based on image segmentation of the present invention;

[0090] Fig.10 The scale through which the pointer passes and the first scale diagram b of the dial in an embodiment of an intelligent reading method of a pointer-type instrument based on image segmentation of the present invention;

[0091] Fig.11It is an OCR recognition maximum range diagram in an embodiment of a pointer-type instrument intelligent reading method based on image segmentation of the present invention;

[0092] Fig.12 This is a preview image of a dial test in an embodiment of a pointer-type instrument intelligent reading method based on image segmentation of the present invention. DETAILED DESCRIPTION

[0093] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0094] Example

[0095] Step 1: Image acquisition of pointer image. Image acquisition uses industrial cameras to shoot original pictures. Industrial cameras need to have good light adaptability and functions such as automatic exposure and automatic white balance to cope with changes in different lighting environments. They can capture clear images under normal lighting and extremely dark conditions. Ensure that the distance between the camera and the pointer instrument is about 30cm, the angle is maintained at 90°, and the camera is installed facing the instrument dial to capture a complete instrument image and reduce the occurrence of distortion. Pointer instruments are effective for most models on the market. The camera will transmit the captured image to the local computer in real time through the network, and the subsequent processing system will process and identify it.

[0096] Step 2: Preprocess the collected pointer instrument dial image, perform image enhancement, tilt correction, image segmentation, and morphological processing on the image. First, perform logarithmic transformation on the input pointer image, then perform Fourier transformation, convert it from the spatial domain to the frequency domain, design and apply the frequency domain filter, and then perform inverse Fourier transformation on the filtered frequency domain image, convert it from the frequency domain back to the spatial domain, and perform exponential transformation on the inverse transformed logarithmic image to restore the brightness range and contrast of the original image. The above frequency domain filtering method simultaneously compresses the image brightness range and enhances the image contrast, enhances the image with uneven illumination, and enhances the pointer in the image; secondly, the image is rotated and scaled by affine transformation, and the tilt phenomenon in the image is corrected to keep the instrument panel image horizontal; then, the edge detection method is used to detect the edge information of the instrument contour to determine the boundary of the target area and separate the instrument dial in the image from the background.

[0097] Step 3: Use the improved U-net model semantic segmentation technology to semantically segment the pointer and scale of the input image. The prediction map obtained by image segmentation contains both position information and feature information. The pixel value of the pointer after segmentation is 1, the pixel value of the scale after segmentation is 2, and the pixel value of the background is 0. Considering that the task goal is to segment the pointer and scale, the selected semantic segmentation model must first be sensitive enough to small targets;

[0098] The function of the network structure observation is to reduce the size of the feature map, and the improved U-net has the advantage of retaining details. It combines low-level features with high-level semantic information through skip connections, which helps to retain detailed features and boundary information. It is more effective for processing tasks with clear boundaries and small targets. Therefore, the present invention uses a convolution layer to replace the maximum pooling layer. According to the convolution calculation formula:

[0099]

[0100] like Figure 2 , the input image size is W input ×H input , the output image size is W output ×H output , P is padding, S is step size, and convolution kernel size is W filter ×H filter The size of the convolution kernel determines the depth of the network, so when a smaller convolution kernel is selected, the input image size is guaranteed to be twice the output image. According to the formula, the size of the convolution kernel is 3×3, the step size is 2, and the padding is 1. Using the above parameters, the maximum pooling layer can be perfectly replaced, retaining more image details and original features.

[0101] Among them, the scale semantic segmentation technology is: using a data set of 748 labeled images, including 200 dial images under extremely dark conditions, the data set is divided into training set, test set, and validation set in a ratio of 7:2:1, and the data set is expanded using random noise, random flipping and other methods in data preprocessing. The cross entropy loss function and Adam optimization method are used to train 200 rounds on a server with GTX 2070super 8G.

[0102] The accuracy of the model is verified by MIOU (mean intersection over union). The calculation formula of MIOU is:

[0103]

[0104] Where i represents the true value, j represents the predicted value, and k represents the number of categories after removing the background. It is expressed as the training set loss function curve and the test set MIOU curve obtained by measuring i as i after the training is completed. Figure 3 (a) and (b) show that the loss function stabilizes after 100 rounds of training, remaining at around 0.05, and the MIOU stabilizes after 150 rounds of training, remaining at around 87%.

[0105] Step 4: Input the segmented result into the reading module of the present invention to determine the relative position of the pointer and the scale. First, the segmented image is subjected to morphological corrosion by the reading module, and then the pointer is fitted with a straight line to perform scale detection. Then, it is determined whether the scale and the pointer coincide, and the number of scales swept by the pointer is determined, and then the scale percentage and the number of scales are obtained;

[0106] The process of fitting the pointer line after morphological corrosion of the image is as follows: first, the segmented image is morphologically corroded, three types of corrosion convolution kernels are compared, and finally the elliptical convolution kernel is selected. The corroded image retains more accurate position information based on the original label. After corrosion, the pixel points with a pixel value of 1 on the image are subjected to statistical probability Hough line transformation to perform straight line fitting. For Hough transformation, the polar coordinate system is used to represent the straight line, and the expression of the straight line can be simplified to:

[0107] r=xcosθ+ysinθ

[0108] For a point (x0, y0) on an image, a family of straight lines passing through this point can be uniformly defined as:

[0109] r θ =x0 cosθ+y0 sinθ

[0110] At this time, each pair (r θ,θ) represents a straight line passing through the point (x0, y0). For a point (x0, y0), if all the graphs passing through it are drawn in the polar coordinate to polar diameter and angle plane, a sine curve will be obtained. This operation is performed on all points in the image. If there are points drawn that intersect in the polar coordinate to polar diameter and angle plane, it means that they pass through the same straight line;

[0111] like Figure 5 , the line transformation fits the longest straight line with a pixel value of 1, sets the endpoint of the line segment close to the center point of the figure to A, and the other endpoint to B;

[0112] The scale detection is as follows: the points with a pixel value of 2 are detected by breadth-first search (BFS), and the points with a pixel value of 2 in each cluster (each scale) are set as a class, and finally all classes are output. For each class, the point with the largest distance from point A is detected, and the position information of all points is saved in a new list L, such as Figure 6 ;

[0113] The process of judging whether the scale coincides with the pointer is as follows: first determine the vector AB, pointing to B, and judge whether the straight-line distance from each point in the list L to 1.5 times the vector AB is greater than the error b=2 (two pixel values). If it is greater, it means that the pointer and the scale do not coincide. Then find the two points closest to B in the list L, recorded as C and D. At this time, we only know that C and D are on both sides of the straight line AB. Convert the straight line AB into a new function f=ax+by+c, bring the coordinates of C and D into f, set the coordinates greater than 0 to C, and set the coordinates less than 0 to D. At this time, C is on the left side of the straight line AB, and D is on the right side of the straight line AB. Figure 7 ;

[0114] like Figure 8 , generate vector CA and vector AD, calculate the arc cosine by vector dot multiplication to get the size of ∠CAB and ∠CAD, and find the proportion of ∠CAB in ∠CAD, and calculate the proportion of the pointer in one scale;

[0115] The process of determining the scale percentage and the number of scales is as follows: remove points C and D from list L, re-search the point closest to point C in list L, set the found point as F and remove F from list L, then search for the point closest to F in list L, search in sequence, count once each time F is removed, stop when the distance between F and the element in list L is greater than 2 times the CD distance, and get the number of points removed M, as shown in Fig. 9 At this point, it should be noted that the first scale of common pointer pressure gauges is generally small, such as Fig.10However, in some instruments with smaller ranges, the first scale is the same as other scales. In order to improve the robustness of the entire module, a restriction is imposed when making labels. If the first scale is less than 0.25 times the other scales;

[0116] At this time, the second scale line is omitted from the label. If it is greater than CD, the second scale line is drawn to prevent the scales from sticking. At this time, if the second scale line is not drawn, then the first scale must be greater than the other scales. If it is drawn, it must be less than or equal to CD. Therefore, when reading, if the first scale is greater than CD, then m = M + 1, and the number of lists L is l = L. Otherwise, m = M, and the number of lists L is l = L-1. The mathematical expression of the overall reading is:

[0117]

[0118] Where R represents the range.

[0119] If there is a point in list L whose straight line distance to 1.5 times vector AB is less than error b, it means that the pointer and scale are collinear. Set this collinear point to Z, remove Z from list L, and search for the two points closest to B in L. The subsequent method is roughly the same as the above non-collinear method. The only difference is that there is no need to calculate angle B. After determining points C and D, point F can be directly found. The mathematical expression of the reading is:

[0120]

[0121] Observe the above formula, only the range R is an unknown number, so it is necessary to use OCR technology to obtain the maximum range of the dial.

[0122] Step 5: Use optical character recognition (OCR) technology to detect key information on the dial of the original image, and obtain the maximum range of the dial through the set threshold and restriction conditions;

[0123] In actual working conditions, the upper limit of the instrument should be greater than the maximum value of the measured value, and the lower limit should be less than the minimum value of the measured value, and an appropriate margin should be left. The amount of margin should be determined according to the fluctuation of the measured parameter. Usually, 1 / 3 to 2 / 3 of the measuring range of the instrument is more suitable for the fluctuation range of the measured value. Therefore, it is feasible to obtain the maximum range without obstruction from the dial;

[0124] The input image is detected by text area detection in optical character recognition (OCR) technology to determine the position of the characters on the dial, and then the character information on the dial is obtained by text recognition. The detected character information is filtered to exclude non-pure digital information. Since the digital information of each large scale is a geometric progression, all the pure numbers obtained are randomly subtracted and the absolute value is taken, and the obtained values ​​are randomly compared to further eliminate the interference factors of the pure numbers in the dial. Finally, the maximum value of the pure numbers obtained is taken as the maximum range R. The results of text area detection and text recognition are shown in the figure. Fig.11 shown.

[0125] like Fig.12 As shown in the comparison chart after identification, the present invention selects three common instruments in industry and three instruments under extremely dark conditions for experimental testing. The manual readings represent the true values. The data after identification are as follows:

[0126]

[0127] In summary, the recognition accuracy of the present invention is 99%. At the same time, the reading speed of the last three samples is far slower than that of the intelligent reading when reading manually, which further illustrates that the intelligent reading based on deep learning is faster and more accurate than the manual reading, can meet the requirements of most industrial applications, and has high social application value.

[0128] OCR technology can scan ID cards, quickly and accurately identify and extract key information on the ID cards; install a license plate recognition system at the entrance of the parking lot, use OCR technology to automatically identify the license plate numbers of vehicles entering, and compare them with the information in the system database to achieve automatic billing and management; scan the ticket image through OCR software to automatically identify and extract key information on the ticket, such as the amount, date, and payee.

[0129] The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A pointer instrument intelligent reading method based on image segmentation, characterized in that: The following steps are involved: Step S1: Capturing the pointer image, the camera transmits the captured image to the local computer in real time via the network; Step S2: pre-processing the collected pointer instrument dial image, performing image enhancement, tilt correction, image segmentation, and morphological processing on the image; Step S2-1: The preprocessing method includes: compressing the image brightness range and enhancing the image contrast by frequency domain filtering, enhancing the image with uneven illumination, and enhancing the pointer in the image; rotating and scaling the image by affine transformation, correcting the tilt phenomenon in the image, and keeping the instrument panel image horizontal; using edge detection method to detect the edge information of the instrument contour to determine the boundary of the target area, and separating the instrument dial in the image from the background; Step S3: semantically segment the pointer and scale of the input image using the improved U-net model semantic segmentation technology. The prediction map obtained by image segmentation contains both position information and feature information. Step S3-1: The improved U-net model replaces the U-net maximum pooling layer with a convolutional layer. According to the convolution calculation formula: In the formula, the input image size is W input ×H input , the output image size is W output ×H output , P is padding, S is step size, and convolution kernel size is W filter ×H filter ; Step S3-2: The semantic segmentation technology uses a labeled dataset to divide the dataset into a training set, a test set, and a validation set in proportion. The dataset is expanded using random noise and random flipping in data preprocessing. The cross entropy loss function and Adam optimization method are used to perform multiple rounds of training on a server with a GTX 2070super 8G. Step S4: Input the segmented result into the reading module of the method to determine the relative position of the pointer and the scale. First, the segmented image is subjected to morphological corrosion by the reading module, and then the pointer is fitted with a straight line to perform scale detection. Secondly, it is determined whether the scale and the pointer coincide with each other, and the number of scales swept by the pointer is determined. Then, the scale percentage and the number of scales are obtained. Step S4-1: The process of pointer straight line fitting after image morphological corrosion is as follows: First, the segmented image is morphologically eroded, and three eroded convolution kernels are compared. Finally, the elliptical convolution kernel is selected. The eroded image retains more accurate position information based on the original label. After corrosion, the pixel points with a pixel value of 1 on the image are statistically probabilistic Hough line transform for straight line fitting. For Hough transform, the polar coordinate system is used to represent the straight line, and the expression of the straight line can be simplified as: r=xcosθ+ysinθ In the formula, r is the distance of the polar radius, that is, the vertical distance from the straight line to the origin, θ is the angle between the vertical line and the x-axis, x is the intersection of the vertical line and the x-axis, and y is the intersection of the vertical line and the y-axis. For a point (x0, y0) on the image, a family of straight lines passing through this point can be uniformly defined as: r θ =x0cosθ+y0sinθ At this time, each pair (r θ ,θ) represents a straight line passing through the point (x0, y0), and a sine curve will be obtained. This operation is performed on all points in the image. If there are points drawn that intersect in the polar coordinate polar diameter and polar angle plane, it means that they pass through the same straight line; Line transformation fits the longest straight line with a pixel value of 1, sets the endpoint of the line segment close to the center point of the figure to A, and the other endpoint to B; Step S4-2: Scale detection is to detect the points with a pixel value of 2 through breadth-first search BFS, set the points with a pixel value of 2 adjacent to each scale in each cluster as a class, and finally output all classes, detect the point with the largest distance from point A for each class, and save the position information of all points into a new list L; Step S4-3: The process of judging whether the scale and the pointer coincide with each other is as follows; First, determine the vector AB, pointing to B, and determine whether the straight-line distance from each point in the list L to 1.5 times the vector AB is greater than the error b=2, that is, two pixel values. If it is greater, it means that the pointer and the scale do not coincide. Then find the two points closest to B in the list L, record them as C and D. At this time, we only know that C and D are on both sides of the straight line AB. Convert the straight line AB into a new function f=ax+by+c, bring the coordinates of C and D into f, set the coordinates greater than 0 to C, and set the coordinates less than 0 to D. At this time, C is on the left side of the straight line AB, and D is on the right side of the straight line AB. Generate vector CA and vector AD, calculate the arc cosine by vector dot multiplication to get the size of ∠CAB and ∠CAD, and find the proportion of ∠CAB in ∠CAD, and calculate the proportional size of the pointer in one scale; Step S4-4: The process of determining the scale percentage and the scale number is as follows; Remove points C and D from list L, re-search the point closest to point C in list L, set the found point as F and remove F from list L, then search for the point closest to F in list L, search sequentially, count once each time F is removed, stop when the distance between F and an element in list L is greater than 2 times the CD distance, and get the number of points removed M; The first scale of common pointer pressure gauges is generally small, but in some instruments with smaller ranges, the first scale is the same as other scales. In order to improve the robustness of the entire module, restrictions are imposed when making labels. If the first scale is less than 0.25 times the other scales, the second scale line is omitted from the label. If it is greater, the second scale line is drawn to prevent scale adhesion. At this time, if the second scale line is not drawn, the first scale must be greater than the other scales. If it is drawn, it must be less than or equal to. Therefore, when reading, if the first scale is greater than CD, then m=M+1, and the number of list L is l=L. Otherwise, m=M, and the number of list L is l=L-1. The mathematical expression of the overall reading is: Where R represents the range; If there is a point in list L whose straight line distance to 1.5 times vector AB is less than error b, it means that the pointer and scale are collinear. Set this collinear point to Z, remove Z from list L, and search for the two points closest to B in L. The subsequent method is roughly the same as the above non-collinear method. The only difference is that there is no need to calculate angle B. After determining points C and D, point F can be directly found. The mathematical expression of the reading is: In the above formula, only the range R is unknown, so the maximum range of the dial is obtained through OCR technology; Step S5: Using the optical character recognition (OCR) technology to detect key information on the dial of the original image, and obtaining the maximum range of the dial through the set threshold and restriction conditions; The threshold setting should be that the upper limit of the instrument should be greater than the maximum value of the measured value, and the lower limit should be less than the minimum value of the measured value, and an appropriate margin should be left. The amount of margin should be determined according to the variation of the measured parameter; the setting limit condition is usually that 1 / 3 to 2 / 3 of the measuring range of the instrument is the fluctuation range of the measured value; Step S5-1: The process of obtaining the maximum range using character recognition OCR technology is as follows: The input image is detected through the text area detection in the optical character recognition (OCR) technology to determine the character position on the dial, and then the character information on the dial is obtained through text recognition. The detected character information is filtered to exclude non-pure digital information. Since the digital information of each large scale is a geometric sequence, all the pure numbers obtained are randomly subtracted and the absolute value is taken. The obtained values ​​are randomly ratioed to further eliminate the interference factors of the pure numbers in the dial. Finally, the maximum value of the pure number obtained is taken as the maximum range R.

2. According to claim 1, the method for intelligent reading of pointer instruments based on image segmentation is characterized in that: The image acquisition in step S1 uses an industrial camera to shoot the original picture. The image supports normal lighting and extremely dark conditions and is effective for most pointer instruments on the market.

3. The method for intelligent reading of pointer-type instruments based on image segmentation according to claim 1 is characterized in that: In the prediction image, the pixel value of the pointer after segmentation is 1, the pixel value of the scale after segmentation is 2, and the pixel value of the background is 0.

4. According to claim 1, the method for intelligent reading of pointer instruments based on image segmentation is characterized in that: The size of the convolution kernel determines the depth of the network, so when a smaller convolution kernel is selected, the input image size is ensured to be twice the output image.

5. The method for intelligent reading of pointer-type instruments based on image segmentation according to claim 4 is characterized in that: In this method, the size of the convolution kernel is 3×3, the step size is 2, and the padding is 1. Using the above parameters can replace the maximum pooling layer, retaining more image details and original features.

6. The method for intelligent reading of pointer-type instruments based on image segmentation according to claim 1, characterized in that: The semantic segmentation technology uses a dataset of 748 labeled images, including 200 dial images under extremely dark conditions, and divides the dataset into training set, test set, and validation set in a ratio of 7:2:

1.

7. An intelligent reading system for pointer-type instruments based on image segmentation, characterized in that: It includes five parts: image acquisition module, image preprocessing module, automatic reading module, dial image segmentation module, and optical character recognition module. It is specifically divided into the following steps: Step M1: The image acquisition module for capturing the pointer image, the camera transmits the captured image to the local computer in real time via the network; Step M2: The image preprocessing module is implemented by preprocessing the collected pointer instrument dial image, performing image enhancement, tilt correction, image segmentation, and morphological processing on the image; Step M2-1: The preprocessing method includes: first, compressing the image brightness range and enhancing the image contrast simultaneously by frequency domain filtering, enhancing the image with uneven illumination, and enhancing the pointer in the image; second, rotating and scaling the image by affine transformation, correcting the tilt phenomenon in the image, and keeping the instrument panel image horizontal; then, using the edge detection method to detect the edge information of the instrument contour to determine the boundary of the target area, and separating the instrument dial in the image from the background; Step M3: The dial image segmentation module is implemented as follows: semantic segmentation is performed on the pointer and scale of the input image using the improved U-net model semantic segmentation technology. The prediction image obtained by image segmentation contains both position information and feature information, where the pixel value of the pointer after segmentation is 1, the pixel value of the scale after segmentation is 2, and the pixel value of the background is 0; Step M3-1: The improved U-net model replaces the U-net maximum pooling layer with a convolutional layer. According to the convolution calculation formula: In the formula, the input image size is W input ×H input , the output image size is W output ×H output , P is padding, S is step size, and convolution kernel size is W filter ×H filter The size of the convolution kernel determines the depth of the network, so when a smaller convolution kernel is selected, the input image size is guaranteed to be twice the output image. According to the formula, the size of the convolution kernel is 3×3, the step size is 2, and the padding is 1. The above parameters can perfectly replace the maximum pooling layer, retaining more image details and original features. Step M3-2: The semantic segmentation technology uses a dataset of 748 labeled images, including 200 dial images under extremely dark conditions. The dataset is divided into a training set, a test set, and a validation set in a ratio of 7:2:

1. The dataset is expanded using random noise and random flipping methods in data preprocessing. The cross entropy loss function and Adam optimization method are used to train 200 rounds on a server with GTX 2070super 8G. Step M4: The automatic reading module is implemented as follows: the segmented result is input into the reading module of the present method to determine the relative position of the pointer and the scale. First, the segmented image is subjected to morphological corrosion by the reading module, and then the pointer is fitted with a straight line to perform scale detection. Secondly, it is determined whether the scale and the pointer coincide with each other, and the number of scales swept by the pointer is determined. Then, the scale percentage and the number of scales are obtained. Step M4-1: The process of pointer straight line fitting after image morphological corrosion is as follows: First, the segmented image is morphologically eroded, and three eroded convolution kernels are compared. Finally, the elliptical convolution kernel is selected. The eroded image retains more accurate position information based on the original label. After corrosion, the pixel points with a pixel value of 1 on the image are statistically probabilistic Hough line transform for straight line fitting. For Hough transform, the polar coordinate system is used to represent the straight line, and the expression of the straight line can be simplified as: r=xcosθ+ysinθ r is the distance of the polar radius, that is, the vertical distance from the straight line to the origin, θ is the angle between the vertical line and the x-axis, x is the intersection of the vertical line and the x-axis, and y is the intersection of the vertical line and the y-axis. For a point (x0, y0) on the image, a family of straight lines passing through this point can be uniformly defined as: r θ =x0cosθ+y0sinθ At this time, each pair (r θ , θ) represents a straight line passing through the point (x0, y0), and a sine curve will be obtained. This operation is performed on all points in the image. If there are points drawn that intersect in the polar coordinate polar diameter and polar angle plane, it means that they pass through the same straight line; Line transformation fits the longest straight line with a pixel value of 1, sets the endpoint of the line segment close to the center point of the figure to A, and the other endpoint to B; Step M4-2: Scale detection is to detect the points with a pixel value of 2 through breadth-first search BFS, set the points with a pixel value of 2 adjacent to each scale in each cluster as a class, and finally output all classes. For each class, detect the point with the largest distance from point A, and save the position information of all points into a new list L; Step M4-3: Determine whether the scale and the pointer coincide with each other. First, determine the vector AB, pointing to B, and determine whether the straight-line distance from each point in the list L to 1.5 times the vector AB is greater than the error b=2, that is, two pixel values. If it is greater, it means that the pointer and the scale do not coincide. Then find the two points closest to B in the list L, record them as C and D. At this time, we only know that C and D are on both sides of the straight line AB. Convert the straight line AB into a new function f=ax+by+c, bring the coordinates of C and D into f, set the coordinates greater than 0 to C, and set the coordinates less than 0 to D. At this time, C is on the left side of the straight line AB, and D is on the right side of the straight line AB. Generate vector CA and vector AD, calculate the arc cosine by vector dot multiplication to get the size of ∠CAB and ∠CAD, and find the proportion of ∠CAB in ∠CAD, and calculate the proportional size of the pointer in one scale; Step M4-4: The process of determining the scale percentage and the scale number is as follows; Remove points C and D from list L, re-search the point closest to point C in list L, set the found point as F and remove F from list L, then search for the point closest to F in list L, search sequentially, count once each time F is removed, stop when the distance between F and an element in list L is greater than 2 times the CD distance, and get the number of points removed M; At this point, it should be noted that the first scale of common pointer pressure gauges is generally small, but in some instruments with smaller ranges, the first scale is the same as other scales. In order to improve the robustness of the entire module, restrictions are imposed when making labels. If the first scale is less than 0.25 times the other scales, the second scale line is omitted from the label. If it is greater, the second scale line is drawn to prevent scale adhesion. At this time, if the second scale line is not drawn, then the first scale must be greater than the other scales. If it is drawn, it must be less than or equal to; therefore, when reading, if the first scale is greater than CD, then m = M + 1, and the number of list L is l = L. Otherwise, m = M, and the number of list L is l = L-1. The mathematical expression of the overall reading is: Where R represents the range; If there is a point in list L whose straight line distance to 1.5 times vector AB is less than error b, it means that the pointer and scale are collinear. Set this collinear point to Z, remove Z from list L, and search for the two points closest to B in L. The subsequent method is roughly the same as the above non-collinear method. The only difference is that there is no need to calculate angle B. After determining points C and D, point F can be directly found. The mathematical expression of the reading is: In the above formula, only the range R is unknown, so the maximum range of the dial is obtained through OCR technology; Step M5: Use the optical character recognition module OCR technology to detect key information on the dial of the original image, and obtain the maximum range of the dial through the set threshold and restriction conditions; The threshold setting should be that the upper limit of the instrument should be greater than the maximum value of the measured value, and the lower limit should be less than the minimum value of the measured value, and an appropriate margin should be left. The amount of margin should be determined according to the variation of the measured parameter; the setting limit condition is usually that 1 / 3 to 2 / 3 of the measuring range of the instrument is the fluctuation range of the measured value; Step M5-1: The process of obtaining the maximum range using character recognition OCR technology is as follows: The input image is detected through the text area in the optical character recognition module (OCR) technology to determine the position of the characters on the dial, and then the character information on the dial is obtained through text recognition. The detected character information is filtered to exclude non-pure digital information. Since the digital information of each large scale is a geometric sequence, all the pure numbers obtained are randomly subtracted and the absolute value is taken. The obtained values ​​are randomly ratioed to further eliminate the interference factors of the pure numbers in the dial. Finally, the maximum value of the pure numbers obtained is taken as the maximum range R.

Images