A target detection-based full-process automatic reading method for a pointer-type instrument

Abstract

This invention discloses a method for automatic reading of pointer-type instruments throughout the entire process based on target detection, comprising: 1) extracting reading-related information from the instrument dial image using a target detection network; 2) estimating the coordinates of the pointer rotation center point using the scaling digit detection results of the target detection network; 3) fitting the pointer region image obtained from the pointer detection results of the target detection network to the straight line where the pointer is located, and obtaining the original position coordinates of the pointer tip; 4) defining the average distance from all scaling digits to the pointer rotation center point as the radius of the instrument's scaling digit circle, and optimizing the position coordinates of the pointer tip using the instrument's scaling digit circle; 5) matching the pointer tip point and the scaling digits to determine the pointer-related scaling interval; 6) calculating the final instrument reading based on the relevant information of the pointer tip point, the pointer rotation center point, and the pointer-related scaling interval. This invention achieves automatic reading of the instrument throughout the entire process without relying on prior information, while also possessing good detection speed.

Description

Technical Field

[0001] This invention relates to the technical field of intelligent visual inspection, and in particular to a method for automatic reading of pointer-type instruments throughout the entire process based on target detection. Background Technology

[0002] Instruments are a general term for numerical display instruments, and they are important tools used to measure various data in production and daily life, playing a vital role in understanding production and environmental conditions. Although we have entered the digital age and many production scenarios are replacing traditional pointer instruments with digital sensing systems, pointer instruments are still widely used in many traditional industrial scenarios such as power generation and oil and gas production due to their strong anti-interference capabilities, simple structure, and low cost.

[0003] In typical scenarios, manual inspectors conduct on-site inspections and record readings from various instruments. This work is tedious and requires a high degree of concentration, easily leading to fatigue and recording errors. Therefore, using inspection robots to replace manual inspections is of great practical significance, and automatic instrument reading algorithms are the core component and key technology required for such inspection robots.

[0004] Existing methods are mainly divided into traditional image processing methods and deep learning methods. Traditional image processing methods typically involve directly inputting a segmented, high-quality dial image or using Hough circle detection to obtain the dial portion. Then, line detectors such as Hough transform are used to detect the pointer angle. Prior information about the dial is then used, usually through a manually created mapping table between angles and readings. The reading is then obtained based on the pointer angle detection result according to this mapping table. Deep learning methods typically use object detection networks to detect dial regions, and the dial processing is similar to traditional methods. These methods utilize traditional image processing techniques to detect pointer information, resulting in high parameter sensitivity and numerous interferences across the entire dial, which is detrimental to recognition. Furthermore, the dial information relies almost entirely on prior information, lacking general, fully automated reading capabilities. Some deep learning methods employ more complex deep learning models such as OCR models, leading to high computational costs, large memory usage, and long inference times, hindering their deployment in practical applications.

[0005] The shortcomings of the existing methods mentioned above result in problems such as insufficient generalization ability, insufficient robustness, inadequate real-time detection, and insufficient deployment resources to meet the needs of practical engineering applications. Therefore, designing a more universal and comprehensive automatic instrument reading method remains a challenging problem to be solved. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and propose a target detection-based automatic reading method for pointer instruments throughout the entire process. This method can automatically establish dial information based on dial characters and uses an improved pointer extraction and reading calculation method, providing a universal, effective and fast solution for solving the problem of intelligent instrument reading.

[0007] To achieve the above objectives, the technical solution provided by this invention is: a method for automatic reading of pointer-type instruments throughout the entire process based on target detection, comprising the following steps:

[0008] 1) Use a trained object detection network to extract reading-related information from the instrument dial image, the reading-related information including the pointer tip, the pointer rotation center point, and the scale digits;

[0009] 2) Estimate the coordinates of the pointer rotation center point of the instrument using the scaling digital detection results of the target detection network;

[0010] 3) Fit the pointer region image obtained from the pointer detection results of the object detection network to the straight line where the pointer is located, and obtain the original position coordinates of the pointer tip;

[0011] 4) Define the average distance from all scale digits to the pointer rotation center point as the radius of the instrument's scale digit circle, and use this scale digit circle to optimize the pointer tip position coordinates;

[0012] 5) Match the pointer tip with the scale number to determine the pointer-associated scale interval;

[0013] 6) Using the instrument reading calculation formula, the final instrument reading is calculated based on the relevant information of the pointer tip, pointer rotation center point, and pointer associated scale interval.

[0014] Further, in step 1), the target detection network needs to be trained for the instrument automatic reading task. After inputting the instrument dial image, the output results are the bounding box and predicted label of the corresponding dial scale number and pointer area. The bounding box is described by the coordinates of its upper left and lower right corners. The scale number includes two attributes: scale value and position coordinates. The scale value of a scale number is its corresponding predicted label, and the position coordinates are the coordinates of the center point of its corresponding bounding box (x, y, y). s ,y s ), can be determined by the coordinates of the top left corner of the bounding box (x tl ,y tl ) and the coordinates of the lower right corner (x br ,y br The calculation is as follows:

[0015]

[0016]

[0017] Further, in step 2), the position coordinates of the pointer rotation center point of the instrument are the center coordinates of the arc trajectory formed during the rotation of the pointer tip. The scale numbers are also distributed on this circle. The position coordinates of every three scale numbers can determine a unique common circle. The center of this common circle is recorded as a fitting point. The scale numbers are permuted in groups of three, and the fitting point corresponding to each group of scale numbers is calculated. All fitting points constitute a fitting point set. The center coordinates of the fitting point set are used as the estimated result of the pointer rotation center point.

[0018] The specific solution process for the above fitting points is as follows:

[0019] 2.1) The general equation of a circle is defined as:

[0020] (x 2 +y 2 )+Bx+Cy+D=0

[0021] B 2 +C 2 -4*D 2 >0

[0022] In the formula, the coordinates of any point on the circle are (x, y); B, C, and D are the parameter matrices of the equation.

[0023] 2.2) Substituting the known points (x1,y1), (x2,y2), and (x3,y3) into the general equation of a circle, we can obtain a homogeneous system of equations about B, C, and D. If the three points are not collinear, the equations have a unique solution about B, C, and D.

[0024]

[0025] 2.3) The coordinates of the center of the circle (x, y, z) can be derived from the general equation of a circle. c ,y c Substituting the coefficients B, C, and D obtained in step 2.2) into the calculation formula, we can obtain the coordinates of the circle center, which is the coordinates of a fitted point:

[0026]

[0027]

[0028] Furthermore, in step 2), before estimating the pointer rotation center point, the fitted point set needs to be screened and outlier fitted points, i.e., fitted points that are far from other fitted points; a distance index reflecting the dispersion of a fitted point and the fitted point set is defined as the outlier index. If the outlier index of a fitted point is greater than a set threshold, it is determined to be an outlier fitted point and is removed; the calculation process of the outlier index is as follows:

[0029] Calculate the distance from the fitted point to all other fitted points in the set of fitted points;

[0030] The average of the nearest 1 / 4 of all distances is the outlier index of the fitted point.

[0031] Furthermore, in step 3), the original position coordinates of the pointer tip are the coordinates of the pointer tip in the dial image, and the specific process includes:

[0032] 3.1) Extract the pointer mask, use the adaptive threshold segmentation OTU method to binarize the pointer region image obtained by the target detection network and extract all contours in the binary image, and identify and remove contours with small areas as noise;

[0033] 3.2) The PCA principal component analysis method is used to detect the line and obtain the line equation. The line equation is described by the slope k and intercept b of the line. The specific calculation process is as follows:

[0034] 3.2.1) Obtain the position coordinates of all white pixels in the binary image to form a sample set X for line fitting;

[0035] 3.2.2) After centering the sample set X, calculate the covariance matrix XX of the sample set X. T ;

[0036] 3.2.3) Regarding this covariance matrix XX T Perform eigenvalue decomposition and take the eigenvector corresponding to the largest eigenvalue as the projection vector. Select the coordinates of any two points near the sample mean and use the projection vector to find the coordinates of their projection points on the fitted line. Then, the two-point equation of the fitted line can be determined using these two projection points.

[0037] 3.3) Calculate the two intersection points of the line equation and the pointer region boundary box. Select the intersection point that is far from the pointer rotation center point as the original position of the pointer tip, and its coordinates are the original position coordinates of the pointer tip point.

[0038] Further, in step 4), the coordinates of the pointer tip are adjusted to the scale digital circle, the equation of the straight line formed by the original coordinates of the pointer tip and the coordinates of the pointer rotation center is calculated, and the point on the straight line that is a distance from the pointer rotation center to the radius of the instrument scale digital circle is taken as the final pointer tip.

[0039] Furthermore, in step 5), the pointer-associated scale interval refers to the scale interval formed by two scale digits on the dial image, which is the smallest scale interval capable of calculating the pointer reading; the pointer-associated scale interval is calculated by matching the pointer tip and the position of the scale digit, as follows:

[0040] 5.1) Calculate the distance from the pointer tip to each scale number;

[0041] 5.2) The scale interval formed by the two scale numbers closest to the pointer tip is used as the matching result of the pointer-associated scale interval. The scale number with the larger scale is the upper bound of the scale interval, and the scale number with the smaller scale is the lower bound of the scale interval.

[0042] Furthermore, in step 5), since the pointer of the pointer instrument can interfere with the target detection network's prediction of the scale numbers around it, the scale numbers close to the pointer tip are defined as unstable scale numbers; half the length of the diagonal of the bounding box of a scale number is defined as the pointer interference threshold of that scale number. If the distance between the scale number calculated in step 5.1) and the pointer tip is less than the pointer interference threshold, then the scale number is determined to be an unstable scale number, and the unstable scale number does not participate in the pointer association scale interval matching process in step 5.2).

[0043] Furthermore, in step 5), the pointer-associated scaling interval should ensure that the angle formed by the scaling interval is an inferior angle, that is, an angle less than 180 degrees; establish a rectangular coordinate system with the pointer rotation center point as the origin, and verify whether the scaling numbers corresponding to the two pointer-associated scaling intervals obtained by matching are one in the first quadrant and the other in the fourth quadrant. If the above situation exists, retain the scaling number closest to the pointer, remove the second closest scaling number to the pointer and replace it with the scaling number closest to the pointer among the remaining scaling numbers to form a new pointer-associated scaling interval. Then repeat the above verification process until the above situation does not occur, that is, the scaling numbers corresponding to the two pointer-associated scaling intervals are one in the first quadrant and the other in the fourth quadrant.

[0044] Furthermore, in step 6), the angle between any two points in the image is defined with the pointer rotation center point as the vertex, and the line segment connecting these two points and the pointer rotation center point is the angle corresponding to the angle formed by the two sides. The formula for calculating the instrument reading is as follows:

[0045] When the position pointed to by the pointer is greater than the lower bound of the scale interval associated with the pointer, the calculation is performed according to Formula 1:

[0046]

[0047] When the position pointed to by the pointer is less than the lower bound of the scale interval associated with the pointer, the calculation is performed according to Formula 2:

[0048]

[0049] In the formula, v represents the calculated reading value of the instrument. max The scale value, v, represents the lower bound of the scale interval. min The scale value θ represents the upper bound of the scale interval. d θ represents the angle between the pointer tip and the lower bound of the scale interval. u θ represents the angle between the pointer tip and the upper bound of the scale interval. s This represents the angle between the upper and lower bounds of the scale interval;

[0050] The specific calculation process is as follows:

[0051] Calculate the angle θ between the pointer tip and the upper and lower bounds of the scale interval, respectively. u and θ d and the angle θ between the upper and lower bounds of the scale interval s ;

[0052] If the sum of the angles θ between the pointer tip and the upper and lower bounds of the scale interval is... u +θ d Equal to the angle θ between the upper and lower bounds of the scale interval s Or, the sum of the angles θ between the upper and lower bounds of the scale interval and the angle θ between the pointer and the upper bound of the scale interval. s +θ u Equal to the angle θ between the pointer and the lower bound of the scale interval d Then use Formula 1 to calculate;

[0053] If the sum of the angle between the upper and lower bounds of the scale interval and the angle between the pointer and the lower bound of the scale interval is θ s +θ d Equal to the angle θ between the pointer and the upper bound of the scale interval u Then use Formula 2 for calculation.

[0054] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0055] 1. This invention has good generalization ability. The reading method of this invention is based on the basic principle of human eye reading instruments and uses an advanced target detection network, which enables this invention to get rid of the high dependence of similar automatic instrument reading algorithms on prior information of the dial, thereby realizing automatic instrument reading recognition throughout the process, that is, directly obtaining the reading result from any input single pointer instrument dial image.

[0056] 2. The present invention has good detection accuracy. Experimental tests have shown that the recognition accuracy of the method of the present invention can reach 90% for reading test sets that are not limited to instrument types and include complex scene backgrounds. This is a high reading accuracy among similar general instrument reading methods.

[0057] 3. This invention has good real-time performance. The instrument reading stage is based entirely on conventional image processing methods, which greatly reduces the amount of computation compared to methods that use multiple deep learning models. The processing speed is fast. Experimental tests have shown that it can achieve a detection rate of 17 frames per second, which can meet the real-time detection requirements in power monitoring and detection scenarios.

[0058] 4. This invention has good robustness and anti-interference ability. Several improvements have been made to the pointer scale matching and reading calculation formula, which maximizes the robustness and anti-interference ability of the method.

[0059] 5. This invention has high practical value. The object detection network used is a relatively common and mature deep learning model. Many companies have provided embedded devices for object detection networks, which can be quickly deployed on machines based on model deployment tools such as Docker. Compared with some more advanced deep learning model methods, it is easier to deploy. Attached Figure Description

[0060] Figure 1 This is a flowchart of the overall process of the method of the present invention.

[0061] Figure 2 This is a schematic diagram of the bounding box of the target detection network.

[0062] Figure 3 This is a schematic diagram for estimating the fitting points.

[0063] Figure 4 This is a diagram illustrating outlier removal.

[0064] Figure 5 A schematic diagram for obtaining the pointer region.

[0065] Figure 6 This is a schematic diagram of the pointer detection process.

[0066] Figure 7 An optimized diagram of the pointer tip.

[0067] Figure 8 A schematic diagram for removing unstable scaling numbers.

[0068] Figure 9 Diagram to prevent incorrect matching.

[0069] Figure 10 This is a schematic diagram for calculating instrument readings. Detailed Implementation

[0070] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0071] like Figure 1 As shown in the figure, this embodiment discloses a fully automatic reading method for pointer-type instruments based on target detection. First, the method requires a specially trained target detection network, which is a convolutional neural network capable of returning the bounding box of a specific target in the image to be detected. Next, the coordinates of the pointer rotation center point are estimated based on the position coordinates of the scale digits obtained by the target detection network, and the original position coordinates of the pointer tip are estimated based on the pointer region obtained by the target detection network. Then, the position coordinates of the pointer tip are optimized based on the instrument scale digit circle, and the optimized pointer tip and scale digits are matched to determine a pointer-associated scale interval for the final reading calculation. Finally, the final reading result is calculated using an instrument reading calculation formula improved based on the angle reading method.

[0072] In this embodiment, we selected a lightweight FCOS model as the object detection network. This network is obtained by replacing the backbone network of the FCOS model with MobileNetV2, and is a relatively conventional lightweight object detection network. (Theoretically, any mature object detection network can perform the task required by this method after being trained on a dataset labeled with instrument panel objects; the only difference will be in performance.)

[0073] Step 1: Use a trained lightweight FCOS model to extract reading-related information from the instrument panel image. For example... Figure 2 As shown, the left image represents all the scaling digit detection results output by the lightweight FCOS model, and the right image is a schematic diagram of the bounding box of the scaling digit "2". The lightweight FCOS model directly outputs the top left corner P of this bounding box. tl (x tl ,y tl ) and the bottom right corner P br (x br ,y br The coordinates are given, and based on these two coordinates, the coordinates P of the center point S of the bounding box can be calculated. s (x s ,y s The specific calculation formula is as follows:

[0074]

[0075]

[0076] In this method, the position of all other scale numbers is described by the center point of their bounding box (the following description of the scale number position refers to the center point of that scale number).

[0077] Step 2, using the scale digit detection results of the dial image to estimate the coordinates of the pointer rotation center point position of the instrument, can be further subdivided into the following steps:

[0078] 2.1) Calculate a fitting point for every three scale numbers to obtain a set of fitting points;

[0079] The following is combined Figure 3 The diagram illustrating the fitting point estimation provides a detailed description of the steps involved in estimating the pointer rotation center point. For example... Figure 3 As shown in the figure, this illustrates the process of estimating the fitting points using three scale numbers: 6, 7, and 8. These three points are denoted as P. s6 (x6, y6), P s7 (x7,y7), P s8 (x8, y8).

[0080] The equation of the fitted circle is defined as follows:

[0081] (x 2 +y 2 )+Bx+Cy+D=0

[0082] B 2 +C 2 -4*D 2 >0

[0083] The coordinates of any point on the circle are (x, y), and B, C, and D are the parameter matrices of the equation.

[0084] Given point P s6 (x6, y6), P s7 (x7, y7) and P s8 Substituting (x8, y8) into the general equation of a circle yields a system of homogeneous equations about points B, C, and D. This system has a unique solution for points B, C, and D when the three points are not collinear.

[0085]

[0086] The coordinates of the center P can be derived from the general equation of a circle. C (x c ,y cBy substituting the coefficients B, C, and D obtained from the above calculation into the formula, we can obtain the coordinates of the center of the circle, which are also the coordinates of the fitted point.

[0087]

[0088]

[0089] 2.2) Calculate the outlier index for each fitted point and filter out fitted points with outlier indices higher than a threshold. Referring to the k-nearest neighbor method, a distance index reflecting the dispersion of a fitted point and the set of fitted points is defined as the outlier index. If the outlier index of a fitted point is greater than the set threshold, it is determined to be an outlier fitted point and is removed. The specific calculation process of the outlier index is as follows:

[0090] Calculate the distance from the fitted point to all other fitted points in the set of fitted points;

[0091] The average of the nearest 1 / 4 of all distances is the outlier index of the fitted point.

[0092] like Figure 4 As shown, the five white dots on the outer edge of the image represent outliers that have been removed, while the dense black dots at the center of the pointer rotation represent the retained fitted points. In this example, the outlier threshold is set to 80. If this threshold is increased, more points will be identified as retained fitted points; if this threshold is decreased, more fitted points will be identified as outliers.

[0093] 2.3) Calculate the coordinates of the center point of the fitted point set after filtering, which is the final coordinate of the pointer rotation center point. Figure 4 The gray dots at the location of the fitted points represent the final pointer rotation center point, which is obtained by calculating the center position from all the black origin points, i.e., the retained fitted points.

[0094] Step 3: Fit the line containing the pointer using the pointer area image of the dial image to obtain the original position coordinates of the pointer tip. The process of obtaining the pointer area image is as follows: Figure 5 As shown, the bounding box of the pointer region output by the lightweight FCOS model is drawn in the dial image as follows. Figure 5 As shown in the middle left image, the pointer region image is extracted using the coordinates of this bounding box. Figure 5 As shown in the middle right figure.

[0095] Calculating the pointer slope using the pointer region image obtained from the lightweight FCOS model can be divided into the following steps:

[0096] 3.1) Extract the pointer mask, use the adaptive threshold segmentation OTU method to binarize the pointer region image and extract all contours in the binary image, and identify and remove contours with smaller areas as noise;

[0097] The image after binarization is as follows: Figure 6 As shown in (b), the white part in the figure is the calculated pointer mask.

[0098] 3.2) Based on the pointer mask described above, PCA (Principal Component Analysis) is used to detect lines and obtain the line equation. The line equation is described by the slope k and intercept b of the line. The specific calculation process is as follows:

[0099] 3.2.1) Obtain the position coordinates of all white pixels in the binary image to form a sample set X for line fitting;

[0100] 3.2.2) After centering the sample set X, calculate the covariance matrix XX of the sample set. T ;

[0101] 3.2.3) Regarding this covariance matrix XX T Perform eigenvalue decomposition and take the eigenvector corresponding to the largest eigenvalue as the projection vector. You can select any two points near the sample mean and find the coordinates of their projection points on the fitted line. Then you can use these two projection points to determine the two-point equation of the fitted line.

[0102] The projection vector of the pointer mask obtained by the above PCA calculation is [0.982-0.191]. Taking the mean point (390, 116) of the sample set X as the reference point for drawing the fitted line, the fitted line can be drawn as follows. Figure 6 As shown in (c) in the figure.

[0103] 3.3) Calculate the two intersection points of the line equation and the pointer area bounding box, connect the two intersection points and draw the line on the dial image, such as... Figure 6 As shown in (d) in the diagram. Selecting a point that is farthest from the center of the pointer gives us the original position of the pointer tip.

[0104] Step 4: Define the average distance from all scale digits to the pointer rotation center point as the radius of the instrument scale digit circle. Use this radius of the instrument scale digit circle to optimize the pointer tip position coordinates. The relevant optimization formula is as follows:

[0105]

[0106]

[0107]

[0108]

[0109] Where r is the radius of the scale number circle, P c (x c ,y c ) represents the coordinates of the center of the scale number circle (which is also the coordinates of the pointer rotation center point), P e1 Let k be the original position of the pointer tip, and k be the value of P. c and P e1 The slope of the line formed by the two lines is used to calculate the slope of the two points (x, y). e21 ,y e21 ) and (x e22 ,y e22 ) respectively with P e1 The distance between them is used to select the point with the smaller distance as the optimized pointer tip point P. e2 .

[0110] like Figure 7 As shown, the white circle represents the scale numerals of the dial, which are centered on the pointer's rotation point P. C Centered on P, use various scale numbers to represent P. C The average distance is obtained using the radius. P in the diagram... e1 P represents the original position of the pointer tip. e2 This indicates the optimized pointer tip position. As can be seen from the figure, P... e1 and P e2 Collinear, P e2 And it is located on the scale number circle.

[0111] Step 5: Match the pointer tip and the scale number to determine the pointer-associated scale interval; the pointer-associated scale interval refers to the scale interval formed by two scale numbers on the dial image, which is the smallest scale interval that can calculate the pointer reading. This invention does not impose too many restrictions on the positional relationship between the pointer tip and the pointer-associated scale interval. The pointer tip can be located inside or outside the pointer-associated scale interval.

[0112] The pointer-associated scale interval is calculated by matching the pointer tip and the scale number. The specific process is as follows:

[0113] Calculate the distance from the pointer tip to each scale number;

[0114] The scale interval formed by the two scale numbers closest to the pointer tip is used as the matching result of the pointer-associated scale interval. The scale number with the larger scale is the upper bound of the scale interval, and the scale number with the smaller scale is the lower bound of the scale interval.

[0115] Since the pointer of a pointer instrument can interfere with the target detection network's prediction of surrounding scale numbers, scale numbers that are too close to the pointer tip are defined as unstable scale numbers. Half the length of the diagonal of the bounding box of a scale number is defined as the pointer interference threshold for that scale number. If the distance between a scale number and the pointer tip is less than the pointer interference threshold, the scale number is determined to be an unstable scale number and will not participate in the pointer-associated scale interval matching process.

[0116] like Figure 8 As shown, the pointer interference threshold for the scale number "7" is half the diagonal length of the bounding box of the scale number "7". A circle can be drawn with the pointer interference threshold as the radius and the pointer tip as the center. If the position of the scale number "7" is within this circle, then that scale number does not participate in the matching of the pointer-associated scale interval. The final matching result of the pointer-associated scale interval obtained in this figure is the scale numbers "6" and "8", corresponding to the position of point P. u and P d The coordinates of P are (223, 619) and (311, 1362) respectively, where P u P is the upper bound of the scale interval associated with the pointer. d This is the lower bound of the scale interval of the pointer-associated interval.

[0117] Another type of misidentification is as follows: Figure 9 As shown, due to the large distance between the scale numbers, the distance between scale number 8 and scale number 16 is P. e2 The distances are very close, which causes the scale number 4 and the scale number 16 to be matched as the same pointer associated scale interval when calculated from the straight-line distance. If the reading is calculated in this interval, the reading error will occur.

[0118] The above situation typically occurs when the pointer tip is close to the largest or smallest scale number, after establishing a Cartesian coordinate system with the pointer rotation center point as the origin. If this situation occurs, it means that one of the pointer tip and the endpoint of the pointer-related scale interval furthest from the pointer is located in the third quadrant, and the other in the fourth quadrant. This can be determined based on the coordinates of the pointer tip and the upper and lower bounds of the scale interval. If the above situation is confirmed, the scale number farther from the pointer tip is discarded, and the closer scale number is retained. This retained scale number, along with the remaining scale number closest to the pointer tip, forms the pointer-related interval. Figure 9 In the middle, P, which is farther from the pointer tip, was discarded. 16 Then, P8, being the closest remaining scale number to the pointer tip, together with P4, constitutes the correct pointer-associated scale interval.

[0119] Step 6: Using the instrument reading calculation formula, calculate the instrument reading based on the pointer tip, pointer rotation center point, and pointer associated scale interval.

[0120] The angle between any two points in the image is defined as the angle formed by the line segment connecting the two points and the center of rotation of the pointer, with the center of rotation as the vertex. The formula for calculating the instrument reading is as follows:

[0121] When the position pointed to by the pointer is greater than the lower bound of the scale interval associated with the pointer, the calculation is performed according to Formula 1:

[0122]

[0123] When the position pointed to by the pointer is less than the lower bound of the scale interval of the pointer's associated interval, it is calculated according to Formula 2:

[0124]

[0125] In the formula, v represents the calculated reading value of the instrument. max The scale value, v, represents the lower bound of the scale interval. min The scale value θ represents the upper bound of the scale interval. d θ represents the angle between the pointer tip and the lower bound of the scale interval. u θ represents the angle between the pointer tip and the upper bound of the scale interval. s It represents the angle between the upper and lower bounds of the scale interval.

[0126] The following is combined Figure 10 A diagram illustrating pointer reading calculation is provided below, showing the specific calculation process:

[0127] Calculate the angle θ between the pointer tip and the upper and lower bounds of the scale interval, respectively. u =∠P e2 P c P u and θ d =∠P e2 P c P d and the angle θ between the upper and lower bounds of the scale interval. s =∠P u P c P d ;

[0128] If the sum of the angles ∠P from the pointer tip to the upper and lower bounds of the scale interval is... e2 P c P u +∠P e2 P c P d Equal to the angle ∠P between the upper and lower bounds of the scale intervalu P c P d Or, the sum of the angle between the upper and lower bounds of the scale interval and the angle from the pointer to the upper bound of the scale interval, ∠P. u P c P d +∠P e2 P c P u Equal to the angle θ between the pointer and the lower bound of the scale interval d =∠P e2 P c P d Then use Formula 1 to calculate;

[0129] If the sum of the angle between the upper and lower bounds of the scale interval and the angle from the pointer to the lower bound of the scale interval is ∠P u P c P d +∠P e2 P c P d Equal to the angle ∠P between the pointer and the upper bound of the scale interval e2 P c P u Then use Formula 2 for calculation.

[0130] exist Figure 10 In the case shown:

[0131] θ u =∠P e2 P c P u =36.00

[0132] θ d =∠P e2 P c P d =36.15

[0133] θ s =∠P u P c P d =72.16

[0134] Because, ∠P e2 P c P u +∠P e2 P c P d =∠P u P c P d Therefore, Formula 1 is chosen as the calculation formula, and the scale values ​​v of the two scale numbers are substituted in. min =6, v max =8 Entering the formula yields the following calculation result:

[0135]

[0136] v = 7.002

[0137] The obtained reading is approximately equal to 7, which is highly consistent with the reading obtained by the human eye, verifying the effectiveness of the method of the present invention.

[0138] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for full process automatic reading of a pointer type instrument based on target detection, characterized in that, The method comprises the following steps: 1) extracting reading related information of the instrument dial image using a trained target detection network, the reading related information comprising a pointer tip point, a pointer rotation center point and scale numbers; 2) estimating the position coordinates of the pointer rotation center point of the instrument using the scale number detection result of the target detection network; 3) fitting a straight line of the pointer region image obtained using the pointer detection result of the target detection network to obtain original position coordinates of the pointer tip point; 4) defining an average distance of all scale numbers to the pointer rotation center point as a scale number circle radius of the instrument, and optimizing the position coordinates of the pointer tip point using the scale number circle of the instrument, specifically as follows: adjusting the pointer tip point coordinates to the scale number circle, calculating a straight line equation formed by the original coordinates of the pointer tip point and the coordinates of the pointer rotation center point, and taking a point on the straight line with a distance to the pointer rotation center point as a length of the scale number circle radius of the instrument as the final pointer tip point; 5) matching the pointer tip point and the scale numbers to determine a pointer associated scale interval; 6) calculating a final instrument reading based on related information of the pointer tip point, the pointer rotation center point and the pointer associated scale interval using an instrument reading calculation formula.

2. A target detection based automatic reading method for a pointer type instrument throughout the process according to claim 1, characterized in that, In step 1), the target detection network needs to be trained for the instrument automatic reading task, and the output result is the scale number of the corresponding dial and the boundary box and the predicted label of the pointer region after inputting the dial image. The boundary box is described by the coordinates of the two points of the upper left corner and the lower right corner. The scale number includes two attributes: scale value and position coordinates. The scale value of a scale number is its corresponding predicted label, and the position coordinates are the center point coordinates (x s ,y s ) of the corresponding boundary box, which can be calculated from the upper left corner coordinates (x tl ,y tl ) and the lower right corner coordinates (x br ,y br ) of the boundary box. The specific calculation formula is as follows:

3. A target detection based automatic reading method for a pointer type instrument throughout the process according to claim 2, characterized in that, In step 2), the position coordinates of the pointer rotation center point of the instrument are the center coordinates of a circular arc trajectory formed in the rotation process of the pointer tip point, and the scale numbers are also distributed on the circle, and every three scale number position coordinates can determine a unique common circle, and the center of the common circle is recorded as a fitting point; all permutations of three scale numbers are performed, and the fitting points corresponding to each group of scale numbers are calculated, and all the fitting points form a fitting point set, and the center coordinates of the fitting point set are taken as the estimation result of the pointer rotation center point; The specific solving process of the fitting point is as follows: 2.1) defining a general equation of a circle as: (x 2 +y 2 )+Bx+Cy+D=0 B 2 +C 2 -4*D 2 >0 In the equation, the coordinates of any point on the circle are (x, y); B, C and D are parameter matrices of the equation; 2.2) substituting the known points (x1, y1), (x2, y2) and (x3, y3) into the general equation of the circle to obtain a homogeneous equation group about B, C and D, and the equation has a unique solution about B, C and D under the condition that the three points are not collinear; 2.3) According to the general equation of a circle, the formula of the center coordinate (x c ,y c ) can be derived, and the coefficients B, C, D calculated in step 2.2) can be brought in to obtain the center coordinate, that is, a fitting point coordinate:

4. A full process automatic reading method of a pointer type instrument based on target detection according to claim 3, characterized in that, In step 2), the fitting point set needs to be screened to remove outlier fitting points before estimating the pointer rotation center point, that is, the fitting points far from other fitting points; a distance index reflecting the dispersion degree of a fitting point and the fitting point set is defined, which is called an outlier index, if the outlier index of the fitting point is greater than a set threshold, the fitting point is determined as an outlier fitting point, and the point is removed; the calculation process of the outlier index is as follows: calculating the distances from the fitting point to all other fitting points in the fitting point set; selecting the nearest 1 / 4 distances to obtain an average, and the result is the outlier index of the fitting point.

5. A full process automatic reading method of a pointer type instrument based on target detection according to claim 4, characterized in that, In step 3), the original position coordinates of the pointer tip point are the coordinates of the pointer tip point in the dial image, and the specific process comprises: 3.1) Extract the pointer mask, and use the adaptive threshold segmentation OTSU method to binarize the pointer region image obtained by the target detection network and extract all contours in the binary image. The contours with small areas are determined as noise and removed; 3.2) Use the PCA principal component analysis method for straight line detection to obtain a straight line equation, which is described by the slope k and the intercept b of the straight line. The specific calculation process is as follows: 3.2.1) Obtain the position coordinates of all white pixels in the binary image to form a sample set X for straight line fitting; 3.2.2) After sample centering of the sample set X, compute the covariance matrix XX of the sample set X T ; 3.2.3) on the covariance matrix XX T Eigenvalue decomposition is performed on the covariance matrix XX, and the eigenvector corresponding to the largest eigenvalue is taken as the projection vector. The coordinates of any two points near the sample mean are selected, and the projection vector is used to find the projection point coordinates on the fitting straight line. The two-point straight line equation of the fitting straight line can be determined by the two projection points. 3.3) Calculate the two intersection points of the straight line equation and the pointer region bounding box, and select the intersection point far from the pointer rotation center point as the original position of the pointer tip. The coordinates of the pointer tip point are the original position coordinates of the pointer tip point.

6. A target detection based automatic reading method for a pointer type instrument throughout the process according to claim 5, characterized in that, In step 5), the pointer associated scale interval refers to the scale interval formed by two scale numbers on the dial image, which is the smallest scale interval that can calculate the pointer reading; the pointer associated scale interval is calculated by matching the pointer tip point and the scale number position, and the specific process is as follows: 5.1) Calculate the distance from the pointer tip point to each scale number; 5.2) Match the scale interval formed by the two scale numbers closest to the pointer tip point as the matching result of the pointer associated scale interval, wherein the scale number with the larger scale is the upper limit of the scale interval, and the scale number with the smaller scale is the lower limit of the scale interval.

7. A target detection based automatic reading method for a pointer type instrument throughout the process according to claim 6, characterized in that, In step 5), since the pointer of the pointer instrument will interfere with the prediction result of the scale number around it by the target detection network, the scale number close to the pointer tip point is defined as an unstable scale number; define half of the diagonal length of the boundary box of a scale number as the pointer interference threshold of the scale number, if the distance from the scale number to the pointer tip point calculated in step 5.1) is less than the pointer interference threshold, then the scale number is determined as an unstable scale number, which does not participate in the pointer associated scale interval matching process of step 5.2).

8. A target detection based automatic reading method for a pointer type instrument throughout the process according to claim 7, characterized in that, In step 5), the pointer associated scale interval should ensure that the angle formed by the scale interval is the inferior angle, that is, less than 180 degrees; a rectangular coordinate system is established with the pointer rotation center point as the origin, and it is verified whether the scale numbers corresponding to the two pointer associated scale intervals match are one in the first quadrant and the other in the fourth quadrant. If the above situation exists, the scale number closest to the pointer is retained, the second closest scale number to the pointer is removed and replaced with the scale number closest to the pointer among the remaining scale numbers to form a new pointer associated scale interval, and then the above verification process is repeated until the above situation does not occur, that is, the scale numbers corresponding to the two pointer associated scale intervals are one in the first quadrant and the other in the fourth quadrant.

9. A target detection based automatic reading method for a pointer type instrument throughout the process according to claim 8, characterized in that, In step 6), the angle between any two points in the image is defined as the angle corresponding to the angle formed by the two sides of the line segment connected by the two points and the pointer rotation center point as the vertex, and the instrument reading calculation formula is as follows: When the pointer pointed position is greater than the scale interval lower limit of the pointer associated scale interval, formula 1 is used for calculation: When the pointer pointed position is less than the scale interval lower limit of the pointer associated scale interval, formula 2 is used for calculation: where v represents the value of the reading calculated from the instrument reading, v min represents the value of the scale number at the lower bound of the scale interval, v max represents the value of the scale number at the upper bound of the scale interval, θ d represents the angle between the tip of the pointer and the lower bound of the scale interval, θ u represents the angle between the tip of the pointer and the upper bound of the scale interval, θ s represents the angle between the upper and lower bounds of the scale interval; The specific calculation process is as follows: Calculate the angle θ between the upper scale interval boundary and the lower scale interval boundary u and the angle θ between the upper and lower scale interval boundaries d and the angle θ between the upper and lower scale interval boundaries s ; If the sum of the angles θ u +θ d between the upper and lower scale interval limits and the angle of the pointer to the upper scale interval limit is equal to the angle θ s between the upper and lower scale interval limits, or the sum of the angle θ s +θ u between the upper and lower scale interval limits and the angle of the pointer to the upper scale interval limit is equal to the angle θ d of the pointer to the lower scale interval limit, then the calculation is made using Formula 1. If the sum of the angle between the lower and upper bounds of the scale interval and the angle of the pointer to the lower bound of the scale interval θ s + θ d is equal to the angle of the pointer to the upper bound of the scale interval θ u then use Equation 2 for the calculation.

Images