A pointer type instrument dial information recognition method based on target detection

Abstract

This invention discloses a method for recognizing pointer-type instrument dial information based on object detection, comprising: 1) extracting scale digits and pointer information from the instrument dial image using an object detection network; 2) performing non-maximum suppression processing on characters with different labels; 3) dividing adjacent individual characters into character groups to be fused; 4) generating scale digit information based on the individual character information within the character groups to be fused; 5) estimating the pointer rotation center point using the scale digit information; 6) applying an improved adaptive Hough transform line detection method to the pointer region to fit the line where the pointer is located; and 7) calculating the coordinates of the pointer tip based on the pointer rotation center point and the line where the pointer is located. This invention utilizes an object detection network, which can establish complete dial information based on each individual character, and adaptively analyze the pointer region to obtain pointer information. It has good versatility, detection speed, and accuracy, and can effectively improve the performance of instrument reading algorithms.

Description

Technical Field

[0001] This invention relates to the technical field of intelligent visual inspection, and in particular to a method for recognizing information on the dial of a pointer-type instrument based on target detection. Background Technology

[0002] Pointer-type instruments, due to their simple structure, low cost, and strong anti-interference capabilities, are widely used in industrial environments such as petrochemicals and power grid systems, playing a crucial role in the safe and efficient production of these locations. In typical scenarios, companies obtain instrument readings by assigning personnel to conduct on-site inspections. This work is tedious and prone to errors due to human fatigue after prolonged periods. Therefore, with continuous technological advancements, more and more researchers are exploring the use of inspection robots to replace manual inspection workers, achieving full automation of such inspection tasks.

[0003] The automatic instrument reading algorithm is one of the core algorithms of the aforementioned robot. The core problem that this algorithm needs to solve is to obtain the pointer and scale information of the instrument and calculate the instrument reading. Whether reliable instrument-related information can be obtained becomes the key to whether the instrument reading algorithm can complete its reading task.

[0004] Most existing research on acquiring instrument information focuses on obtaining pointer information, and can be mainly divided into two categories: traditional image processing methods and deep learning methods. Traditional image processing methods typically involve preprocessing the image first, then using line fitting methods such as least squares, Hough transform line detectors, or PCA principal component analysis to obtain the pointer line expression from the binarized image. Deep learning methods often use instance segmentation networks to predict the pointer mask, then extract the fitted center line and further calculate the pointer deflection angle. Traditional digital image processing methods are highly sensitive to parameters, often requiring different parameter settings for different scenarios. These parameter setting requirements are often quite strict, resulting in poor robustness. Instance segmentation network-based methods involve large network computations and high data annotation costs, raising the barrier to entry for their application in production.

[0005] Methods for acquiring dial information from instruments are even fewer. Most instrument reading methods treat dial information as prior information, designing reading methods and parameters for specific instruments, or they treat it as quasi-prior information, designing reading methods for a limited number of known instruments, and then determining the instrument type through model label prediction or template matching. A small number of instrument reading methods have explored methods based on OCR models to detect dial information. Methods utilizing prior information cannot be directly extrapolated to new instrument models, resulting in poor universality. For methods using OCR models, excessive use of deep learning models in the entire instrument recognition process leads to slow detection speeds, high resource consumption, and difficult deployment.

[0006] The shortcomings of the existing methods mentioned above lead to problems such as sensitivity to dial information recognition parameters, poor versatility and scalability, high deployment costs, and difficulty in guaranteeing detection accuracy and speed. Therefore, designing a dial information recognition method that is universally applicable, easy to deploy, and fast and reliable remains a challenging problem to be solved. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and propose a target detection-based method for recognizing dial information of pointer instruments. This method can automatically and reliably acquire the dial and pointer information of unknown instruments using a mature target detection network, providing a universal, fast, and robust solution for solving the problem of recognizing instrument dial information through computer vision methods.

[0008] To achieve the above objectives, the technical solution provided by this invention is: a method for recognizing dial information of a pointer-type instrument based on target detection, comprising the following steps:

[0009] 1) Use a trained object detection network to extract character and pointer area information from the instrument panel;

[0010] 2) Perform non-maximum suppression on all obtained characters across different labels, and filter the remaining characters to obtain the effective character detection results;

[0011] 3) Divide adjacent characters into a group of characters to be merged, and arrange the characters in the group of characters to be merged in ascending order of their horizontal coordinates;

[0012] 4) Determine the final digital fusion result of each character group to be fused. The fused number is the scale number of the instrument. All the scale numbers together constitute the scale information of the dial.

[0013] 5) Estimate the coordinates of the pointer rotation center point of the instrument based on the instrument's scale readings;

[0014] 6) An improved adaptive Hough transform line detection method is applied to the pointer region to fit and obtain the pointer centerline;

[0015] 7) Calculate the coordinates of the pointer tip based on the pointer centerline and the pointer rotation center point.

[0016] Furthermore, in step 1), the target detection network is a convolutional neural network. The input is a color image, and the output is the prediction result of a specific target in the image and its bounding box. The bounding box is described by the position coordinates of its upper left and lower right corners. The specific targets that the target detection network trained with the labeled dataset can detect include characters 0 to 9 and pointers, totaling 11 types of targets.

[0017] Furthermore, in step 2), non-maximum suppression needs to be applied to the characters of different detected labels to remove duplicate detections that may be simultaneously identified as two different label characters by the target detection network due to similarity; the specific steps of the non-maximum suppression are as follows:

[0018] 2.1) Divide all characters into three categories: retained characters that have undergone non-maximum suppression, deleted characters that have been removed, and unprocessed characters that have not undergone non-maximum suppression.

[0019] 2.2) Select any character to be processed;

[0020] 2.3) Calculate the Intersection over Union (IoU) between this character and all other characters to be processed, and calculate the Intersection over Union (IoU) of the bounding boxes. The formula for calculating IoU is as follows:

[0021]

[0022] In the formula, S1 and S2 represent the areas of the two bounding boxes in which the IoU is being calculated, respectively;

[0023] 2.4) If the IoU between the character and another character to be processed is greater than a preset threshold, the IoU test determines that there are highly repeated characters. The characters with high confidence are retained and non-maximum suppression is continued. The characters with low confidence are removed as deleted characters. If the IoU between the character and the bounding boxes of all the remaining characters to be processed is less than the threshold, the character is determined to be a retained character.

[0024] 2.5) Repeat steps 2.2) to 2.4) until all characters are reduced to two categories: retained characters and deleted characters. All retained characters are the final valid characters.

[0025] Furthermore, in step 3), the character group to be merged refers to an ordered character group composed of several characters that are identified as collectively forming a scale number; the character position coordinates are defined as the coordinates of the center point of the character bounding box, and the character spacing between two characters is the Euclidean distance between the position coordinates of the two characters. The character group to be merged can be divided by calculating the character spacing. The specific calculation steps are as follows:

[0026] 3.1) Divide all characters into two categories: unmerged and merged;

[0027] 3.2) Randomly select an unmerged character as the initial character of the current character group to be merged. Iterate through all remaining unmerged characters and calculate the character spacing and character bounding box spacing with the initial character respectively. The character bounding box spacing refers to half of the sum of the x-direction distances of the character bounding boxes. If the difference between the character spacing and the character bounding box spacing is greater than the set threshold, it is determined that the two characters belong to the same character group to be merged.

[0028] d p -0.5*(d a +d b )>α

[0029] In the formula, d p Indicates character spacing, d a and d b These represent the horizontal lengths of the two character bounding boxes, and α represents the set threshold, which is 0.25*(d). a +d b 3.3) Elements in the character group to be merged need to be arranged in ascending order according to the horizontal coordinate of the character center point;

[0030] 3.4) If a character is assigned to the current character group to be merged, the loop process in step 3.3) is also required for that character until all characters in the current character group to be merged have been compared and calculated with all remaining unmerged characters, and then the character group to be merged is finally determined.

[0031] 3.5) Repeat steps 3.2) to 3.4) until all characters have been divided into groups of characters to be merged.

[0032] Furthermore, in step 4), the number fusion specifically refers to the process of generating a new scale number using several character information from a character group to be fused, including the following steps:

[0033] 4.1.1) Boundary box merging: Select the coordinates of the outermost edge of each character in all character groups to be merged. The specific calculation formula is as follows:

[0034] x1=min(x 11 ,x 12 ,x13 ,...,x 1i ,...,x 1n )

[0035] y1=min(y 11 ,y 12 ,y 13 ,...,y 1i ,...,y 1n )

[0036] x2=max(x 21 ,x 22 ,x 23 ,...,x 2i ,...,x 2n )

[0037] y2 = max(y 21 ,y 22 ,y 23 ,...,y 2i ,...,y 2n )

[0038] In the formula, (x1,y1) and (x2,y2) represent the coordinates of the top left and bottom right corners of the merged scale number, respectively. 1i ,y 1i ) and (x 2i ,y 2i ) represent the top-left and bottom-right coordinates of the i-th character in a character group of n characters to be merged;

[0039] 4.1.2) Calculate the center point coordinates (x, y) of the scale number based on the new coordinates of the merged bounding box:

[0040]

[0041]

[0042] 4.1.3) Calculate the original scale value of the scale number according to the arrangement order of the character groups to be merged. The calculation formula is as follows:

[0043]

[0044] In the formula, L represents the original fused label of the character group to be merged, which is the original scale value represented by the corresponding scale number. i This represents the label of the i-th character in the character group to be merged.

[0045] Furthermore, in step 4), since the object detection network has difficulty recognizing small targets such as decimal points, it is necessary to predict the number of decimal places of the scale digits based on the arrangement information of the character groups to be fused, and determine the final scale value based on this. The specific calculation process is as follows:

[0046] 4.2.1) Calculate the potential number of decimal places based on the number of characters in the character group to be merged. The potential number of decimal places β is the number of characters in the character group to be merged minus one.

[0047] 4.2.2) If the first element of the character group to be merged is 0 and the number of characters in the character group to be merged is greater than 1, then the character group to be merged represents a decimal scale number. If there is a decimal scale number on the dial, then the instrument is determined to be a decimal scale type instrument.

[0048] 4.2.3) If the instrument is not determined to be a decimal scale type instrument, then the original scale value obtained in step 4.1.3) is the final identified scale value of the scale number;

[0049] 4.2.4) If the instrument is determined to be a decimal scale instrument, the scale of all scale digits on the dial needs to be adjusted. The adjusted scale is the final recognition scale value L. res The specific adjustment formula is as follows:

[0050] L res =L*10 β .

[0051] Furthermore, in step 5), the pointer rotation center point is estimated using the obtained scaling numbers, and the optimal ellipse passing through all scaling numbers is fitted using an ellipse fitting method based on least squares. The center position of this ellipse is the pointer rotation center point position.

[0052] Furthermore, in step 6), an improved adaptive Hough transform line detection method is used to fit the pointer centerline, which includes two parts: adaptive binarization and adaptive Hough transform line detection. The specific process is as follows:

[0053] 6.1) Set different binarization termination thresholds based on the length and width of the pointer region segment, denoted as t. end ;

[0054] 6.2) Initialize a binarization segmentation threshold t for binarization;

[0055] 6.3) Perform binary segmentation on the pointer region image based on a threshold t, and calculate the average gray level g of the pointer region image after binary segmentation;

[0056] 6.4) Compare the average gray level g and the binarization termination threshold tend The size of t, if t end If the value is greater than g, then update the binarization threshold t' = t + Δt using the step size Δt of the binarization segmentation threshold, and repeat steps 6.3) and 6.4) with the new binarization threshold t' until t end The binarization process ends when g < g;

[0057] 6.5) Perform edge detection on the binarized pointer region to obtain a binarized pointer edge image;

[0058] 6.6) The pointer edge image is detected using the Hough transform line detection method. To adapt to input dial images of different sizes, different Hough transform line detection thresholds are set in stages. The detection threshold is set to be larger for larger images and smaller for smaller images.

[0059] 6.7) After the line detection, the results of several line equations are obtained. The center line equations of these lines are then calculated. First, the two endpoints of each line at the same scale are calculated using the line equations obtained by the Hough transform line detection. Then, the coordinates of the center points of the two endpoints are calculated respectively. The line represented by the coordinates of the two center points is the pointer center line.

[0060] Furthermore, in step 7), the two pointer endpoints within the pointer area are obtained by calculating the coordinates of the intersection point between the pointer center line and the pointer area, and the position coordinates of these two pointer endpoints in the dial image are calculated based on the distance in the dial image of the pointer area. The specific calculation process is as follows:

[0061] 7.1) Calculate the coordinates of the intersection points of the pointer center line and the four sides of the pointer area;

[0062] 7.2) Determine whether the coordinates of the intersection point are inside the pointer area, and select the two intersection points of the pointer center line and the edge of the pointer area as pointer endpoints;

[0063] 7.3) Calculate the coordinates of the pointer endpoint in the pointer area coordinate system and the dial area coordinate system based on the x and y coordinates of the upper left corner of the pointer area, where (x... tl ,y tl (x) represents the coordinates of the top-left corner of the pointer area. p ,y p (x) is a pointer endpoint in the pointer region coordinate system. m ,y m () represents the coordinates of the pointer endpoint in the corresponding dial area coordinate system:

[0064]

[0065] Further, in step 7), the distance between the two pointer endpoints under the dial image obtained in step 7.3) and the pointer rotation center point obtained in step 5) is calculated respectively, and the point with the farthest distance is retained as the final pointer tip point coordinates.

[0066] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0067] 1. This invention has good versatility and generalization ability. Compared with instrument reading methods that directly use prior information or quasi-prior information, this invention can significantly reduce the data annotation and parameter adjustment work during scene transformation.

[0068] 2. This invention has good detection accuracy. Compared with similar methods that directly use general OCR models, this invention has specially designed the target in the instrument and replaced the general text detection model in the OCR method with a specially designed digital fusion algorithm, which is more suitable for the needs of instrument reading tasks.

[0069] 3. This invention has good anti-interference capabilities. It has carried out a lot of adaptive processing in the pointer detection and other stages, which can adapt to the dial images captured in different environments, improve the detection quality, and reduce the application condition settings.

[0070] 4. This invention has good real-time performance and uses a relatively mature object detection network for scale recognition. Compared with using OCR or instance segmentation networks, it is easier to deploy on current mainstream deep learning embedded devices and can be more conveniently applied to actual production projects. Attached Figure Description

[0071] Figure 1 This is a flowchart of the overall process of the method of the present invention.

[0072] Figure 2 This is a schematic diagram of the output of the target detection network.

[0073] Figure 3 This is a diagram illustrating character duplication detection.

[0074] Figure 4 This is a flowchart of the NMS processing.

[0075] Figure 5 A diagram illustrating the division of characters to be merged.

[0076] Figure 6 This is a flowchart of the digital integration process.

[0077] Figure 7 This is a schematic diagram of the digital fusion process.

[0078] Figure 8 This is a schematic diagram of digital integration for decimal instruments.

[0079] Figure 9 A schematic diagram of fitting the center point of pointer rotation.

[0080] Figure 10 This is a flowchart for adaptive Hough transform line detection.

[0081] Figure 11 This is a schematic diagram of the adaptive Hough transform line detection process.

[0082] Figure 12 A schematic diagram for calculating the pointer tip. Detailed Implementation

[0083] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0084] For a dial image, all its scale numbers (including the corresponding scale values ​​and position coordinates) together constitute the dial information, and the coordinates of the pointer rotation center point and the pointer tip point together constitute the pointer information. The aforementioned dial information and pointer information together constitute the dial information, which is the final calculation target of this invention.

[0085] like Figure 1 As shown, this embodiment discloses a method for recognizing pointer-type instrument dial information based on target detection. The input instrument dial image is fed into a specially trained target detection network to obtain character and pointer target information from the dial image. For the character targets, a digital fusion process is used to fuse them from several individual characters into scale numbers. First, non-maximum suppression (NMS) is applied between characters with different labels. Then, the NMS-processed characters are divided into groups to be fused. Finally, the proposed fusion strategy is used to perform digital fusion to generate scale numbers. For the pointer targets, an improved adaptive Hough transform line detection method is used to fit the equation of the line where the pointer is located after binarization and Hough transform line detection. Finally, the coordinates of the pointer's rotation center point and the pointer bounding box information obtained from the scale number fitting are used to calculate the coordinates of the pointer's tip.

[0086] Step 1: Use a trained object detection network to extract character and pointer area information from the instrument panel. The direct output of the object detection network is the bounding box of the detected object and its predicted label. The bounding box is described by the coordinates of its upper left and lower right vertices, and the predicted label indicates the meaning of the object. The object detection network required by this method can detect 11 specific object categories, including characters 0-9 and pointers.

[0087] like Figure 2As shown, the left image illustrates the bounding box of each character (each a single character) output by the network, with the predicted character value printed below the bounding box. The right image illustrates the bounding box of the character "0," where the network directly outputs the top-left vertex P of the bounding box. b1 (x b1 ,y b1 ) and the bottom right vertex P b2 (x b2 ,y b2 These two points can be used to calculate the coordinates P of the center point of the bounding box. b (x b ,y b The calculation formula is as follows:

[0088]

[0089]

[0090] Step 2: Perform non-maximum suppression on all obtained characters across different labels, and filter the remaining characters to obtain the effective character detection results.

[0091] Non-maximum suppression (hereinafter referred to as "NMS") is a common post-processing technique for object detection networks. Its main purpose is to remove multiple detection boxes around the same object (these detection boxes have confidence scores exceeding the detection threshold and are therefore detected repeatedly). In general, multi-class object detection uses NMS separately for each class label to remove duplicate detection boxes.

[0092] In this invention, since the object detection network is used to detect individual characters, some characters may be simultaneously identified as multiple different categories of characters due to similarity or interference. For example... Figure 3 As shown, the pointer is so close to the scale number "300" that it obscures the character "3," causing it to be simultaneously identified as "3," "2," and "6" due to interference (the confidence level for all three labels at this position reaches the output threshold of the object detection network). If number fusion is performed at this point, the scale number "32600" will be obtained, which is clearly incorrect. Therefore, all characters output by the object detection network need to undergo another class-insensitive NMS processing to avoid this. Figure 3 Similar situations have occurred in China.

[0093] like Figure 4 As shown, the specific steps of the NMS processing are as follows:

[0094] 2.1) All characters are divided into three categories: retained characters after NMS processing, deleted characters that have been removed, and unprocessed characters to be processed. The IoU test is defined as calculating the intersection-union ratio (IoU) of the bounding boxes of two characters, and then determining whether the two characters are overlapping characters based on whether the IoU exceeds a pre-set threshold. The formula for calculating IoU is as follows:

[0095]

[0096] Where S1 and S2 represent the areas of the two bounding boxes in which IoU is being calculated;

[0097] 2.2) Randomly select one character to be processed as the character to be processed in this loop;

[0098] 2.3) Randomly select one of the remaining characters to be processed that has not yet undergone an IoU test with the current character and perform an IoU test with the current character;

[0099] 2.4) If not identified as a duplicate character, repeat step 2.3) until all remaining characters to be processed have undergone IoU verification with the character, and the character is classified as a retained character. If identified as a duplicate character, compare the confidence scores of the character with those of the duplicate characters, and remove characters with low confidence scores (classified as deleted characters). If the character to be processed in this loop is removed, return to step 2.2); otherwise, repeat step 2.3) until the character completes the NMS processing.

[0100] Finally, the characters processed by NMS are filtered based on the bounding box area to remove characters with too small or too large areas, resulting in a more reliable character output.

[0101] Step 3: Divide several adjacent characters into a group of characters to be merged, and arrange the characters within the group in ascending order of their horizontal coordinates. The group of characters to be merged refers to an ordered group of characters identified as collectively forming a single-value digit. For example... Figure 5 As shown, the characters "6" and "0" can be considered to together form the scale number "60". The purpose of this step is to group all the characters in the dial image that may be combined to form the same scale character together.

[0102] Combination Figure 6 As shown, the specific calculation steps are as follows:

[0103] 3.1) Divide all characters into two categories: unmerged and merged; define the character position coordinates as the coordinates of the center point of the character bounding box, and the character spacing between two characters as the Euclidean distance between the position coordinates of the two characters; define the character bounding box spacing as half of the sum of the x-direction distances of the character bounding boxes;

[0104] 3.2) Randomly select an unmerged character as the initial character of the current character group to be merged (hereinafter referred to as "current character group");

[0105] 3.3) Randomly select one character from the remaining unmerged characters and compare it with the characters in the current character group. The character bounding box spacing and character spacing are then compared. If the difference between the character spacing and the character bounding box spacing is greater than a set threshold, the two characters are determined to belong to the same character group to be merged. The determination formula is as follows:

[0106] d p -0.5*(d a +d b )>α

[0107] In the formula, d p Indicates character spacing, d a and d b These represent the horizontal lengths of the two character bounding boxes, and α represents the set threshold, which is typically 0.25*(d). a +d b This refers to half the spacing between the character bounding boxes;

[0108] like Figure 5 As shown in the example in the middle left figure, the character spacing between the character "6" and the character "0" is d. p =P A P B The horizontal lengths of the two bounding boxes are d respectively. a =P A3 P A4 and d b =P B3 P B4 The character bounding box spacing is (P A3 P A4 +P B3 P B4 ) / 2. According to the above calculation process, "6" and "0" are classified into the same character group. However, after calculation, the character spacing between other characters and "6" or "0" is much larger than the character boundary box spacing between them, so they cannot be classified into the same character group to be merged.

[0109] 3.4) Elements within a merged character group are arranged in ascending order of their horizontal position coordinates, such as... Figure 5 As shown, the character order in the character group to be merged, consisting of the character "6" and the character "0", must be "6, 0", and cannot be "0, 6".

[0110] 3.5) Repeat steps 3.3) to 3.4) until all unmerged characters have been compared with the current character group, and the character group to be merged is finally determined. Note that if a new character is assigned to the current character group, all unmerged characters taken after repeating steps 3.3) to 3.4) must also be compared with the newly assigned character using the above judgment process.

[0111] 3.6) Repeat steps 3.2) to 3.5) until all characters have been divided into groups of characters to be merged.

[0112] Step 4: Determine the final fusion result of each character group to be fused. The fused number is the scale number of the instrument. All the scale numbers together constitute the scale information of the dial.

[0113] The digital fusion process specifically refers to the process of generating a new scale number using several character information from a character group to be fused, combined with... Figure 6 As shown, the process specifically includes the following steps:

[0114] 4.1) Boundary box merging: The coordinates of the outermost edges of each character in all character groups to be merged are selected as the coordinates of the top-left and bottom-right corners of the final bounding box. "Outermost" means that the top-left corner uses the smallest x and y coordinates, and the bottom-right corner uses the largest x and y coordinates. The specific calculation formula is as follows:

[0115] x1=min(x 11 ,x 12 ,x 13 ,...,x 1i ,...,x 1n )

[0116] y1=min(y 11 ,y 12 ,y 13 ,...,y 1i ,...,y 1n )

[0117] x2=max(x 21 ,x 22 ,x 23 ,...,x 2i ,...,x 2n )

[0118] y2 = max(y 21 ,y 22 ,y 23 ,...,y 2i ,...,y 2n )

[0119] Where (x1, y1) and (x2, y2) represent the coordinates of the top-left and bottom-right corners of the merged scale number, respectively. 1i ,y 1i ) and (x 2i ,y 2i ) represent the top-left and bottom-right coordinates of the i-th character in a character group of n characters to be merged;

[0120] like Figure 7 As shown, the three separate characters "3", "0", and "0" are merged into the scale number "300", and the three bounding boxes are integrated into one bounding box;

[0121] 4.2) Calculate the center point coordinates (x, y) of the scale number based on the new coordinates of the merged bounding box:

[0122]

[0123]

[0124] 4.3) Calculate the original scale value of the scale number according to the arrangement order of the character groups to be merged. The calculation formula is as follows:

[0125]

[0126] Where L represents the original fused label of the character group to be merged, which is the original scale value represented by the corresponding scale number. i The label (character value) represents the i-th character in the character group to be merged.

[0127] like Figure 7 As shown, the order of the characters to be merged is "3", "0", "0". Substituting this into the formula in step 4.3), we get:

[0128] L = 3 * 10 3-1 +0*10 3-2 +0*10 3-3 =300

[0129] Therefore, the original scale value of the character group to be merged is calculated to be 300, which is consistent with human visual observation.

[0130] In engineering practice, there is a type of instrument whose scale numbers are decimals. If the original scale value of the character group to be fused is directly used as the recognition result of the scale value, an error will occur. Furthermore, the decimal point, as a small target, is not easily recognized by the target detection network. Therefore, this invention designs the following method for handling decimal problems based on the characteristics of the scale numbers of pointer-type instruments:

[0131] A decimal scale is defined as follows: when the first character in a character arrangement of a character group to be merged is "0" and the number of characters in the character group is greater than 1 (i.e., 0 appears at the maximum number of digits and the original scale value of the character is not 0), the scale number corresponding to the character group to be merged is decimal. If a dial image contains a scale number greater than or equal to a decimal, the instrument is defined as a decimal scale type instrument. Based on the determination result of whether the instrument is a decimal scale type, the final scale value of the scale number is determined. The specific calculation process is as follows:

[0132] 4.4) Calculate the potential number of decimal places based on the number of characters in the character group to be merged. The potential number of decimal places β is the number of characters in the character group to be merged minus one.

[0133] 4.5) If the first element of the character group to be merged is 0 and the number of characters in the character group to be merged is greater than 1, then the character group to be merged represents a decimal scale number. If there is a decimal scale number on the dial, then the instrument is determined to be a decimal scale type instrument.

[0134] 4.6) If the instrument is not identified as a decimal scale instrument, the original scale value obtained in step 4.3) is the final recognized scale value of that scale number; if the instrument is identified as a decimal scale instrument, the scale of all scale numbers on the dial needs to be adjusted, and the adjusted scale is the final recognized scale value L. res The specific adjustment formula is as follows:

[0135] L res =L*10 -β

[0136] like Figure 8 As shown, Figure 8 The character group "0, 4" to be merged shown in (b) satisfies the condition in step 4.5) that "the first element is 0 and the number of characters in the character group to be merged is 2, which is greater than 1". Therefore, this is a decimal scale number. According to step 4.4), the potential number of decimal places β of this character is the number of characters in the character group to be merged minus one, that is, β = 2 - 1 = 1. Therefore, according to the formula in step 4.6), the original scale value L = 4 obtained in step 4.3) is substituted into the scale value for adjustment:

[0137] L = 4 * 10 -1 =0.4

[0138] res

[0139] The final scale value was 0.4, which is consistent with the reading obtained by the human eye.

[0140] Because of the presence of the decimal scale number "0.4", the entire instrument is classified as a decimal scale type instrument. Therefore, all characters must be adjusted in step 4.6). Figure 8 As shown in (c), the character group "1, 6" to be merged did not originally meet the conditions for decimal scale numbers. However, because the instrument was judged to be a decimal scale type instrument, the reading result was also adjusted. The final scale value result is 1.6, which is consistent with the reading of the human eye.

[0141] Step 5: Estimate the pointer rotation center point of the instrument based on the instrument's scale values; fit the optimal ellipse passing through all scale values ​​using the least squares-based ellipse fitting method, and the center of this ellipse is the pointer rotation center point.

[0142] like Figure 9 As shown, the positions of each scale number are marked with a white dot P. xx This indicates that the subscript xx and its corresponding scale value are the same (P). 20 The scale value corresponding to the scale number is 20), and the coordinates of the center position of the fitted ellipse are P. C After making the corresponding white dot, its position is aligned with the height of the pointer's rotation center point as observed by the human eye.

[0143] Step 6 involves performing an improved adaptive Hough transform line detection on the pointer region to fit the pointer centerline. This process mainly includes two parts: adaptive binarization and adaptive Hough transform line detection. Figure 10 and Figure 11 The specific calculation process is as follows:

[0144] 6.1) Set different binarization termination thresholds based on the length and width of the pointer region segment, denoted as t. end .

[0145] In this embodiment, pointer areas with an aspect ratio greater than 4 are set to t. end =30, for pointer areas with an aspect ratio less than 4, set to t end =90, because the pointer occupies a higher proportion in some pointer areas with higher aspect ratios (when the pointer is close to horizontal or vertical). Therefore, setting thresholds in segments can enhance the robustness of the algorithm and achieve a certain degree of adaptive effect.

[0146] 6.2) Initialize a binarization segmentation threshold t for binarization. In this embodiment, t = 10.

[0147] 6.3) Perform binary segmentation on the pointer region image based on a threshold t, and calculate the average gray level g of the pointer region image after binary segmentation;

[0148] 6.4) Compare the average gray level g and the binarization termination threshold t end The size of t, if t end If the value is greater than g, then update the binarization threshold t' = t + Δt using the step size Δt of the binarization segmentation threshold, and repeat steps 6.3) and 6.4) with the new binarization threshold t' until t end The binarization process ends when g < g;

[0149] like Figure 11 As shown in (a), the pointer region image is extracted from the dial image in this embodiment based on the object detection network. After 11 iterations of the method in steps 6.2) to 6.4), the final segmentation threshold t is set to 130, resulting in an average grayscale of approximately 32 for the binary image of the pointer region. The obtained binary image of the pointer region is shown in Figure 1. Figure 11 As shown in (b);

[0150] 6.5) Perform edge detection on the binarized pointer region to obtain a binarized pointer edge image, such as... Figure 11 As shown in (c);

[0151] 6.6) The pointer edge image is detected using the Hough transform line detection method. To adapt to input dial images of different sizes, different Hough transform line detection thresholds are set in stages. For larger images, the detection threshold is set to be larger, and vice versa.

[0152] 6.7) After line detection, several line equations are obtained. The centerline equations of these lines are then calculated. First, the Hough transform is used to calculate the two endpoints of each line at the same scale. Then, the coordinates of the center points of the two endpoints are calculated. The line represented by the coordinates of the two center points is the centerline of the pointer. Figure 11 In the Chinese embodiment, the pointer center line obtained from the above detection is drawn, and the effect is as follows: Figure 11 As shown in (d).

[0153] Step 7: Calculate the coordinates of the pointer tip using the pointer centerline obtained in Step 6) and the pointer rotation center obtained in Step 5).

[0154] First, the two pointer endpoints within the pointer area are obtained by calculating the coordinates of the intersection point between the pointer center line and the pointer area boundary. Then, the position coordinates of these two points in the dial image are calculated based on the distance between them and the pointer area in the dial image. Figure 12 The specific calculation process is as follows:

[0155] 7.1) Calculate the coordinates of the intersection points of the pointer center line and the four edges of the pointer area;

[0156] 7.2) Determine if the intersection point coordinates are inside the pointer area, and select the two intersection points of the pointer center line and the edge of the pointer area as pointer endpoints; for example... Figure 12 The coordinates of the two intersection points obtained from the left subgraph are P. e1 (0,11) and P e2 (270,251);

[0157] 7.3) Calculate the coordinates of the pointer endpoint in the pointer area coordinate system and the dial area coordinate system based on the x and y coordinates of the upper left corner of the pointer area, where (x... tl ,y tl (x) represents the coordinates of the top-left corner of the pointer area. p ,y p (x) represents the coordinates of a pointer endpoint in the pointer region coordinate system. m ,y m () represents the coordinates of the pointer tip in the corresponding dial area coordinate system:

[0158] x m =x p +x tl

[0159] y m =y p +y tl

[0160] Known Figure 12 In the example shown, the coordinates of the pointer bounding box are (179, 295). Therefore, the coordinates of the right subplot can be calculated using the formula above: P e1 '(179,306) and P e2 '(449,546);

[0161] 7.4) Calculate the two pointer endpoints under the dial image obtained in step 7.3) and the pointer rotation center point P calculated in step 5). e1 The distance between (179, 306) is used to select the point with the furthest distance as the final pointer tip coordinates. Figure 12 In the case shown, P e1 'P C The distance is greater than P e2 'P C Distance, therefore P e1 The point is the final pointer tip.

[0162] Based on steps 1 to 7 above, the positions and corresponding scale values ​​of all scale numbers in the instrument dial image have been obtained, as have the coordinates of the pointer tip and the pointer rotation center point. Thus, all dial information of the instrument has been obtained.

[0163] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A target detection based method for pointer instrument dial information recognition, characterized in that, The method comprises the following steps: 1) extracting character and pointer region information of the instrument dial using a trained target detection network; 2) performing non-maximum suppression between different labels on all obtained characters, and screening the remaining characters to obtain effective character detection results; 3) dividing adjacent characters into a to-be-fused character group, and arranging the characters in the to-be-fused character group in ascending order according to the horizontal coordinates; The to-be-fused character group refers to an ordered character group composed of several characters that are identified as jointly constituting a scale number; the character position coordinates are defined as the center point coordinates of the character bounding box, and the character spacing between two characters is the Euclidean distance between the two character position coordinates, and the to-be-fused character group can be divided by calculating the character spacing, and the specific calculation steps are as follows: 3.1) dividing all characters into two categories of unfused and fused; 3.2) taking an unfused character as an initial character of a current to-be-fused character group, calculating the character spacing and the character bounding box spacing between the initial character and all remaining unfused characters respectively, the character bounding box spacing refers to half of the sum of the character bounding box x-direction distance, if the difference between the character spacing and the character bounding box spacing is greater than a set threshold, it is determined that the two characters belong to the same to-be-fused character group; ； In the formula, represents the character spacing, and respectively represent the horizontal length of the two character bounding boxes, represents the set threshold value, which is taken ; 3.3) the elements in the character group to be fused need to be arranged in ascending order of the horizontal coordinates of the character center points; 3.4) if a character is divided into the current to-be-fused character group, the character also needs to undergo the loop process in step 3.3), until all characters in the current to-be-fused character group have completed the comparison calculation with all remaining unfused characters, the to-be-fused character group is finally determined; 3.5) repeating steps 3.2) to 3.4) until all characters have completed the to-be-fused character group division operation; 4) determining the final number fusion result of each to-be-fused character group, and the fused number is the scale number of the instrument, and all scale numbers jointly constitute the scale disc information of the dial; The number fusion specifically refers to a process of generating a new scale number using several character information in a to-be-fused character group, comprising the following steps: 4.1.1) boundary box fusion, selecting the outermost coordinates of each character in all to-be-fused character groups, and the specific calculation formula is as follows: ； wherein, and respectively represent the coordinates of the upper left corner and the lower right corner of the scaled number after fusion, and respectively represent the upper left corner and the lower right corner coordinates of the i-th character in the character group to be fused with n characters. 4.1.2) Calculate the center point coordinates of the scale number according to the new coordinates of the fused bounding box : ; 4.1.3) calculating the original scale value of the scale number according to the arrangement order of the to-be-fused character group, and the calculation formula is as follows: ； wherein, denotes the original fused label of the character group to be fused, i.e. the original scale value corresponding to the scale number representation, denotes the label of the i-th character in the character group to be fused; 5) estimating the pointer rotation center point coordinates of the instrument according to the scale number of the instrument; 6) performing an improved adaptive Hough transform straight line detection on the pointer region to fit the pointer center line; 7) calculating the pointer tip point coordinates according to the pointer center line and the pointer rotation center point.

2. The pointer-type instrument dial information recognition method based on target detection according to claim 1, characterized in that, In step 1), the target detection network is a convolutional neural network, the input is a color image, and the output is the prediction result of a specific target in the image and the bounding box thereof, the bounding box is described by the position coordinates of the upper left corner and the lower right corner, and the specific target that can be detected by the target detection network trained using the labeled data set includes 11 types of targets including characters 0~9 and a pointer.

3. The pointer-type instrument dial information recognition method based on target detection according to claim 2, characterized in that, In step 2), the detected characters of different labels need to be respectively subjected to non-maximum suppression to remove repeated detection caused by similar characters being simultaneously recognized by the target detection network as two different label characters; the non-maximum suppression specifically comprises the following steps: 2.1) dividing all the characters into three categories: retained characters subjected to non-maximum suppression, deleted characters, and to-be-processed characters not subjected to non-maximum suppression; 2.2) selecting a to-be-processed character at random; 2.3) calculating the IoU of the character and all other to-be-processed characters, and calculating the intersection over union IoU of the bounding boxes, the IoU calculation formula being as follows: ； wherein and respectively represent the areas of the two bounding boxes whose IoU is being calculated; 2.4) if the IoU of the character and another to-be-processed character is greater than a preset threshold, it is determined that there is a highly repeated character, and the character with a higher confidence is retained for further non-maximum suppression, and the character with a lower confidence is deleted; if the IoU of the character and all other to-be-processed characters is less than the threshold, the character is determined to be a retained character; 2.5) repeatedly performing steps 2.2) to 2.4) until all the characters are only retained characters and deleted characters, and all the retained characters are the final effective characters.

4. The pointer-type instrument dial information recognition method based on target detection according to claim 3, characterized in that, In step 4), since the target detection network cannot easily recognize small targets such as decimal points, the decimal place of the scale number needs to be predicted based on the arrangement information of the to-be-fused character group, and the final scale value is determined based on the decimal place, the specific calculation process being as follows: 4.2.1) Calculate the potential number of digits after the decimal point according to the number of characters in the group of characters to be fused, the potential number of digits after the decimal point is the number of characters in the group of characters to be fused minus one; 4.2.2) if the first element of the to-be-fused character group is 0 and the number of characters in the to-be-fused character group is greater than 1, it is determined that the to-be-fused character group represents a decimal scale number, and if there is a decimal scale number greater than or equal to one in the dial, it is determined that the instrument is a decimal scale type instrument; 4.2.3) if the instrument is not determined to be a decimal scale type instrument, the original scale value obtained in step 4.1.3) is the final recognized scale value of the scale number; 4.2.4) If the meter is determined to be a decimal scale type meter, then the scale of all the scale numbers on the dial needs to be adjusted to the final recognized scale value. The specific adjustment formula is as follows: , where: 。 5. The pointer-type instrument dial information recognition method based on target detection according to claim 4, characterized in that, In step 5), the scale number is used to estimate the center point of the pointer rotation, and an ellipse fitting method based on the least square method is used to fit the optimal ellipse passing through all the scale numbers, and the center position of the ellipse is the position of the center point of the pointer rotation.

6. The pointer-type instrument dial information recognition method based on target detection according to claim 5, characterized in that, In step 6), an improved adaptive Hough transform line detection method is used to fit the center line of the pointer, including adaptive binarization and adaptive Hough transform line detection, the specific process being as follows: 6.1) Set different binarization termination thresholds according to the aspect ratio of the pointer region, denoted as ; 6.2) initializing a binarization threshold t for binarization; 6.3) performing binarization on the pointer region image based on the threshold t, and calculating the average gray value g of the binarized pointer region image; 6.4) Compare the average gray value g with the binarization termination threshold If then update the binarization threshold by the step size Update the binarization threshold by the step size and perform the calculations of steps 6.3) and 6.4) again with the new binarization threshold until the process of binarization is terminated. 6.5) performing edge detection on the binarized pointer region to obtain a binarized pointer edge image; 6.6) using the Hough transform line detection method to detect the pointer edge image, and setting different Hough transform line detection thresholds in different stages to adapt to different sizes of input dial images, the threshold being set to be large for large-size images, and vice versa. 6.7) After the detection of the straight line equation, the center line equation of the straight line is obtained. First, the two end points of each straight line under the same scale are calculated by using the Hough transform straight line detection equation. Then the coordinates of the center points of the two end points are calculated respectively. The straight line represented by the two center point coordinates is the pointer center line.

7. The pointer-type instrument dial information recognition method based on target detection according to claim 6, characterized in that, In step 7), the intersection coordinates of the pointer center line and the pointer region are calculated to obtain the two pointer end points in the pointer region, and the position coordinates of the two pointer end points in the dial image are calculated according to the distance in the dial image. The specific calculation process is as follows: 7.1) Calculate the intersection coordinates of the pointer center line and the straight line on which the four edges of the pointer region lie respectively; 7.2) Determine whether the intersection coordinates are inside the pointer region, and select the two intersection points of the pointer center line and the pointer region edge as the pointer end points; 7.3) Calculate the coordinates of the pointer end points in the dial area coordinate system from the horizontal and vertical coordinates of the upper left corner of the pointer area, where is the upper left corner coordinate of the pointer area, is one of the pointer end points in the pointer area coordinate system, is the corresponding pointer end point coordinate in the dial area coordinate system. 。 8. The pointer-type instrument dial information recognition method based on target detection according to claim 7, characterized in that, In step 7), the distance between the two pointer end points in the dial image obtained in step 7.3) and the pointer rotation center point calculated in step 5) is calculated respectively, and the point with the farthest distance is kept as the final pointer tip point coordinate.

Images