A method for automatic reading of a pointer instrument for a patrol robot

Abstract

The application discloses a kind of pointer instrument automatic reading method for inspection robot, can replace artificial inspection meter reading, greatly improve efficiency and safety.The specific steps of the present application include: first, combining attention mechanism and adaptive feature fusion module, design instrument detection network, accurately locate pointer table from the perspective of robot, and cut out the pointer table image suitable for automatic reading.Second, a directed pointer detection network is proposed, which can locate the pointer on the dial and accurately fit the tip position of the pointer.Third, a dial text information extraction network based on deep learning is designed, and the scale and unit information of the pointer table are obtained through text detection and filtering algorithm.Finally, a polar coordinate pixel method is proposed to locate the main scale line, and a local angle method is designed to calculate the reading of the pointer instrument.Adequate experiments prove that the present application has high accuracy and robustness in actual factory inspection tasks, and the average global error of reading is only 0.73%.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and is a deep learning-based method for automatically reading pointer meters. It is a complete solution that can be mounted on an inspection robot to replace manual meter reading. This method takes a video stream of an inspection in a real industrial scenario as input and outputs the readings of the pointer meters contained in the video stream. Background Technology

[0002] Pointer-type meters are crucial tools for monitoring manufacturing processes in modern industrial settings. They possess strong resistance to electromagnetic interference, good mechanical stability, and can withstand harsh production environments such as high temperatures and pressures, making them widely used in production sites in industries like petroleum, chemical, and power. However, most pointer-type meters lack data output interfaces, still requiring manual intervention at hazardous engineering sites for data recording. This method is time-consuming, labor-intensive, and inefficient, and can lead to safety accidents if equipment malfunctions are not detected promptly. With advancements in industrial hardware and software in my country, the field of inspection robots has developed rapidly, with robots increasingly replacing manual labor in repetitive and complex tasks. In power and chemical plants, many meters are installed in dangerous locations, compromising the safety and efficiency of meter readers. Inspection robots, with their intelligence and explosion-proof features, automatically identify readings after capturing meter images, making them an effective method for meter reading. Therefore, utilizing inspection robots for automated reading of pointer-type meters is of great significance for improving production efficiency and safety in factories.

[0003] However, instruments in industrial settings often suffer from interference such as blurred dials and stains. Combined with the effects of angle, distance, and lighting during robot image acquisition, this places higher demands on the accuracy and stability of the automatic meter reading algorithms mounted on inspection robots. With the development of computer vision technology, using computer vision for automatic meter reading is gradually becoming a new trend. Currently, many researchers have achieved good results using traditional image processing and machine vision methods for automatic meter reading. However, traditional image processing methods (such as template matching and line detection) are easily affected by other interference factors on the dial, exhibiting instability in harsh industrial environments such as low light, vibration, and reflection. The accuracy of dial recognition and reading is not high enough, lacking universality. Although many researchers have incorporated deep learning methods into automatic meter reading, current methods mostly focus on improving the performance of a specific step in the dial reading process (such as dial positioning and pointer detection). Because different scenarios have different requirements for reading algorithms, a complete automatic reading solution for pointer-type instruments suitable for inspection robots is still lacking.

[0004] Therefore, this invention studies an automatic reading method for pointer-type instruments that is more accurate, more stable, and can be used in inspection robots, which has important practical significance. Summary of the Invention

[0005] This invention proposes a complete solution for automatic pointer-type meter reading, which can be installed on a robot to perform meter reading tasks. Combining the advantages of deep learning and computer vision, the method consists of five stages, including three deep learning-based object detection models and two computer vision-based mathematical models: meter detection and localization, pointer recognition and extraction, meter text information extraction, main scale line localization, and reading calculation. We designed different deep network structures for localization and classification based on the different characteristics of the meter, pointer, and text information. Finally, we established two mathematical models for locating the main scale line and calculating the angle, based on the polar coordinate pixel method and the local angle method, to obtain the final reading. Extensive experiments demonstrate that this method has high accuracy and robustness under robot operating conditions.

[0006] To achieve the above functions, the present invention includes the following steps:

[0007] Step 1: Design and train the YOLO_Meter instrument detection network. Convert robot-captured images into instrument images. During the robot's inspection process, the instrument detection algorithm runs continuously. Every 5 seconds, the robot captures an image from the video and sends it to the instrument detection network. The algorithm detects the presence of instruments in the image, and the camera automatically zooms in, focuses, and crops the image to fit the subsequent readings based on the instrument's location.

[0008] The CSPDarknet network is used as the base network structure, and the SPPF module is used to pool features at different scales and fuse them with the feature pyramid network. The SimAM attention mechanism is added to each C3 module of the backbone network. The ASFF adaptive feature fusion method replaces the Concat and Element-wise operations in the Yolov5 neck layer. Specifically, the operations include the same rescaling and adaptive fusion, and the adaptive fusion weight parameters for the three feature scales are automatically learned.

[0009] Step 2: Based on the cropped instrument image from Step 1, design and train a pointer detection network to obtain the relative position of the pointer on the instrument image. Design a directed pointer detection network (OPDNet) based on the pointer, resulting in bounding boxes that are close to the pointer. Use the cropped instrument image from Step 1 as input, pass it through a ResNet50 residual network consisting of 49 convolutional layers and one fully connected layer, and output the result. Use a Feature Pyramid Network (FPN) to fuse feature maps of different scales from ResNet50. In the Region Proposal Network (RPN), add two parameters, Δα and Δβ, representing the offset of the detection box in the horizontal and vertical directions, to the proposal, resulting in proposal boxes with rotation angles. Use the directed RPN as anchor boxes, and then use Softmax to determine if there is a target in the anchor box, obtaining precise candidate boxes with rotation angles. Subsequently, convert the candidate boxes into feature maps F', and decode the parameters to (x, y, w, h, θ). The feature maps are then trained for classification and regression after passing through two fully connected layers. The trained OPDNet accurately identifies the pointer and the center of rotation of the instrument from the dial image, and obtains the coordinates (x, y) of the pointer recognition box. p ,y p ,w p ,h p ,θ p ) and the coordinates (x) of the instrument center recognition frame c ,y c ,w c ,h c ,θ c Where x and y represent the horizontal and vertical coordinates of the center point of the box, w and h represent the width and height of the box, respectively, and θ represents the rotation angle of the candidate box based on the horizontal direction.

[0010] Step 3: Based on the pointer recognition frame obtained in Step 2, fit the position of the pointer tip in the instrument image. Calculate the quadrant where the pointer is located using the instrument's rotation center and the position coordinates of the pointer detection frame. Then, obtain the coordinates of the pointer tip through mathematical angle transformation. With the rotation center as the origin, the entire image is divided into left and right parts. When the center of the pointer is located on the right side, the coordinates of the pointer tip are calculated as follows:

[0011]

[0012]

[0013] Similarly, when the center of the pointer is to the left of the center of the instrument:

[0014]

[0015]

[0016] Fitting pointer tip position N(x) n ,y n This prepares the groundwork for determining the positional relationship between the pointer tip and the scale line.

[0017] Step 4: Extract instrument text information from the cropped instrument image in Step 1. Propose OCR_Meter, a deep learning-based method for instrument text information recognition and filtering, to obtain the specific values ​​of units, scale markings, and scale line coordinates. The instrument text information extraction task is divided into two parts: text localization and text recognition. A deep neural network, OCR_Meter, is designed to recognize the main scale values ​​and units on the instrument dial. First, image features of the instrument are extracted using MobileNetV3. Then, features at different scales are fused by a feature pyramid network. The activation function f(x) is used to binarize the features, dividing the image into background and text regions, thus completing the text localization. Here, u represents the magnification factor, set to 50.

[0018]

[0019] Next, the small bounding boxes containing the text are fed into ResNet31 to further extract detailed text features. Encoding and decoding are performed by a two-layer LSTM (Long Short-Term Memory) network, and the feature maps are processed through Softmax to obtain the text classification result, completing text recognition. Finally, a filtering algorithm is set based on the character pair recognition result. The filtering algorithm extracts the main tick value and unit, two essential pieces of information, for subsequent calculations.

[0020] Step 5: Continue to determine the positional relationship between the pointer tip and the scale lines to obtain the instrument's specific reading. Calculate the reading using the positional difference between the pointer tip and the main scale line. Define the main scale line as the thickest and longest scale line closest to the main scale value. Use polar coordinate pixel method to locate the main scale line. Since the arrangement of the main scale lines resembles an arc, use the pointer rotation center C(x) identified in Step 2. c ,y c Using (x, y) as the origin, the image is transformed from a Cartesian coordinate system to a polar coordinate system. Then, all the tick marks are arranged almost sequentially on the same straight line. Let the position of each pixel in the original image be (x, y). i ,y j If the polar coordinates are transformed into Cartesian coordinates, then the coordinates of the pixel in the Cartesian coordinate system are (ρ...). (i,j) θ (i,j) ),in:

[0021]

[0022]

[0023] The coordinate system of the original instrument image was transformed into a polar coordinate system.

[0024] Step Six: Using the position of each major tick mark value from Step Five as a reference, smooth the image to the right within a certain area using a Gaussian filter. After processing, determine the region with the densest concentration of black pixels as the major tick mark and record its coordinates. Convert the image back to a Cartesian coordinate system and record the coordinates of the major tick marks as (x...). k ,y k ).

[0025] Step 7: Using the rotation center C(x) obtained in Step 2 c ,y c Step 3 yields the pointer tip N(x) p ,y p Step six yields the positions of each main tick mark (x). k ,y k The final reading of the instrument is calculated. First, the distance between any two major scale marks is calculated using the angle formed between the center point C and the two major scale lines. That is, how much range each degree represents.

[0026] Points A and B are the results of positioning two principal scale lines. The difference between the scale values ​​at points A and B is calculated based on these two points. According to the Law of Cosines:

[0027]

[0028]

[0029] Among them, X k This represents the proportional value corresponding to the positioning result of the k-th main scale line. The positioning results of all main scale lines and their corresponding main scale values ​​are included in the calculation, regardless of whether they are adjacent to each other. The average value is obtained through n calculations to obtain a more accurate range spacing d. Even if the OCR_Meter is interfered with and misses detections occur, adjacent main scale values ​​and main scale lines will still be included in the calculation, ensuring the robustness of this method.

[0030] Step 8: Similarly, according to formula (8), calculate the angle formed between the pointer tip, the adjacent scale line, and the center of the instrument. Determine the positional relationship between the pointer tip and the main scale line. The final reading Y is calculated using the following formula:

[0031] Y=d·∠ACN+X k (10)

[0032] Compared to existing technologies, this invention combines the advantages of deep learning and computer vision, proposing a complete solution for automatic reading of pointer-type instruments for inspection robots, significantly improving the efficiency of meter reading in factories. Specifically, this invention utilizes SimAM and ASFF modules, greatly enhancing its detection rate in harsh environments. It also employs deep learning-based directed target detection to accurately locate the pointer and fit the needle tip position, which is crucial for subsequent accurate readings. Furthermore, this patent utilizes deep learning-based text localization and text recognition to directly extract dial text information, greatly enhancing the versatility and robustness of the method for various dial types. The three deep learning-based models used in this patent, trained on a large amount of data, can essentially achieve full-scene coverage of factory inspections, stably extracting dial, pointer, and dial text information under various interference factors such as dim lighting, reflections, and tilt. The essence of meter reading is determining the positional relationship between the pointer and the scale lines, requiring high accuracy. This patent proposes a polar coordinate pixel method and a local angle method to locate the main scale line at the pixel level, and calculates the instrument reading corresponding to the pointer tip through angle calculation, resulting in a very small reading error. Overall, this invention not only proposes a high-precision automatic reading method for pointer-type instruments, but also optimizes and enhances the method for the characteristics of robot inspection, making industrial meter reading safer and more efficient. Attached Figure Description

[0033] Figure 1 This is an overview diagram of the steps of the present invention.

[0034] Figure 2 This is a flowchart illustrating how the algorithm works in a real-world scenario.

[0035] Figure 3 This is the network structure diagram of the instrument detection algorithm.

[0036] Figure 4 This is a diagram of the pointer detection network structure.

[0037] Figure 5 This is a diagram of the deep network structure for dial text detection.

[0038] Figure 6 This is a schematic diagram of the main scale line positioning method.

[0039] Figure 7 This is a schematic diagram of the indicator calculation model.

[0040] Figure 8-10 This is a schematic diagram of the algorithm results for each step. Detailed Implementation

[0041] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0042] This invention proposes a complete solution for pointer-type automatic meter reading, which can be installed on a robot to perform meter reading tasks. The process is as follows: Figure 1 As shown in the diagram. Specifically, combining the advantages of deep learning and computer vision, this method consists of five stages, including three deep learning-based object detection models and two computer vision-based mathematical models: instrument detection and localization, pointer recognition and extraction, instrument text information extraction, main scale line localization, and reading calculation. We designed different deep network structures for localization and classification based on the different characteristics of the instrument, pointer, and text information. Finally, we established two mathematical models for locating the main scale line and calculating the angle, based on the polar coordinate pixel method and the local angle method, to obtain the final reading. Extensive experiments demonstrate that this method has high accuracy and robustness under robot working conditions. Figure 2 This is a flowchart illustrating how the algorithm of this patent works in a real-world scenario.

[0043] To achieve the above functions, the present invention includes the following steps:

[0044] Step 1: Design and train the YOLO_Meter instrument detection network. This step converts robot-captured images into instrument images. The instrument detection algorithm runs continuously during the robot's inspection process. Every 5 seconds, the robot captures an image from the video and sends it to the instrument detection network. Once the algorithm detects an instrument in the image, the camera automatically zooms in, focuses, and crops the image to fit the subsequent readings based on the instrument's location.

[0045] In real-world industrial scenarios, the YOLOv5 algorithm's ability to identify instruments of varying styles and sizes still needs improvement. This invention improves the YOLOv5 algorithm. The network structure is as follows: Figure 3As shown, we still use CSPDarknet as the basic network structure and use the SPPF module to pool features at different scales, then fuse it with the feature pyramid network. This invention adds the SimAM attention mechanism to each C3 module of the backbone network. As a computational unit, SimAM is a three-dimensional attention mechanism that assigns weights to neurons by calculating the analytical solution of the energy function, making the network pay more attention to important locations. It enhances the expressive power of features in convolutional neural networks, helping the network find instrument features in cluttered backgrounds. After fully extracting features, the network needs to further fuse them. Furthermore, since instruments occupy different proportions in the image, conflicts may arise between features at different scales, posing a challenge to feature fusion. This invention uses the ASFF adaptive feature fusion method to replace the original Yolov5 neck's Concat and Element-wise operations. The specific operations include the same rescaling and adaptive fusion, which can automatically learn the adaptive fusion weight parameters for three feature scales, improving the performance of Yolov_Meter in recognizing small-sized instruments in industrial environments.

[0046] Step Two: Based on the instrument image cropped in Step One, design and train a pointer detection network to obtain the relative position of the pointer on the instrument image. In this patent, we designed a directed pointer detection network (OPDNet) based on the characteristics of the thin and long pointer, and the resulting recognition box can be very close to the pointer. In addition, the rotation center of the pointer instrument is also one of the key information for reading, and we also use this network to identify and locate it. Figure 4 This is a schematic diagram of the OPDNet network structure. We take the cropped instrument image from step one as input, pass it through a ResNet50 residual network consisting of 49 convolutional layers and one fully connected layer, and output the result. This residual network extracts pointer features well with fewer parameters and faster speed. Then, we use a Feature Pyramid Network (FPN) to fuse feature maps of different scales from ResNet50, which greatly improves the saliency of pointer feature representation. Next, we add two parameters Δα and Δβ to the proposal of the Region Proposal Network (RPN) to represent the offset of the detection box. These two parameters represent the offset of the proposal box in the horizontal and vertical directions, and we obtain proposal boxes with rotation angles. Next, we use the oriented RPN as anchor boxes, and then use Softmax to determine whether there is a target in the anchor box to obtain the exact candidate boxes with rotation angles. Subsequently, the candidate boxes are converted into feature maps F', and the parameters are decoded as (x,y,w,h,θ). The feature maps are then trained for classification and regression after passing through two fully connected layers. The trained OPDNet can accurately identify the pointer and the center of rotation of the instrument from the dial image, and obtain the coordinates (x, y) of the pointer recognition box.p ,y p ,w p ,g p ,θ p ) and the coordinates (x) of the instrument center recognition frame c ,y c ,w c ,h c ,θ c Where x and y represent the horizontal and vertical coordinates of the center point of the box, w and h represent the width and height of the box, respectively, and θ represents the rotation angle of the candidate box based on the horizontal direction.

[0047] Step 3: Based on the pointer detection box obtained in Step 2, fit the position of the pointer tip in the instrument image. We use the rotation center of the instrument and the position coordinates of the pointer detection box to calculate the quadrant where the pointer is located. Then, we obtain the coordinates of the pointer tip through mathematical angle transformation. With the rotation center as the origin, the entire image is divided into left and right parts. When the center of the pointer is located on the right side, the coordinates of the pointer tip can be calculated as follows:

[0048]

[0049]

[0050] Similarly, when the center of the pointer is to the left of the center of the instrument:

[0051]

[0052]

[0053] Fitting pointer tip position N(x) n ,y n This laid the necessary groundwork for determining the positional relationship between the pointer tip and the scale line.

[0054] Step Four: Extract instrument text information from the instrument image cropped in Step One. This step is independent of Steps Two and Three and is not sequential. Accurate extraction of instrument text information is crucial for accurate readings. In different industrial scenarios, the types, scales, and units of instruments vary. Many template matching methods are no longer applicable. Therefore, we propose OCR_Meter, a deep learning-based method for instrument text information recognition and filtering, which can obtain specific values ​​for units, scales, and scale line coordinates. The network structure is as follows... Figure 5As shown, this invention divides the task of extracting instrument text information into two parts: text localization and text recognition. We designed a deep neural network, OCR_Meter, which can be used to recognize the main scale values ​​and units on the instrument dial. First, image features of the instrument are extracted using MobileNetV3. Then, features at different scales are fused by a feature pyramid network. Next, we use the activation function f(x) to binarize the features, that is, to divide the image into background and text regions, thereby completing the text localization. Here, u represents the magnification factor, which is set to 50.

[0055]

[0056] Next, the small bounding boxes containing the text are fed into ResNet31 to further extract detailed text features. Encoding and decoding are performed by a two-layer LSTM (Long Short-Term Memory) network, and the feature maps are processed by Softmax to obtain the text classification result, thus completing text recognition. Finally, we set up a filtering algorithm based on the character pair recognition results. The filtering algorithm only extracts the main tick value and unit—two essential pieces of information—for subsequent calculations.

[0057] Step 5: Based on the above steps, we can continue to determine the positional relationship between the pointer tip and the scale lines to obtain the instrument's specific reading. Since accurately locating all scale lines is very difficult, we use the positional difference between the pointer tip and the main scale line to calculate the reading. In this patent, we define the main scale line as the thickest and longest scale line closest to the main scale value. We use the polar coordinate pixel method to locate the main scale line. Since the arrangement of the main scale lines resembles an arc, we use the pointer rotation center C(x) identified in Step 2. c ,y c Using (x, y) as the origin, the image is transformed from a Cartesian coordinate system to a polar coordinate system. Then, all the tick marks are arranged almost sequentially on the same straight line. Let the position of each pixel in the original image be (x, y). i ,y j If the polar coordinates are transformed into Cartesian coordinates, then the coordinates of the pixel in the Cartesian coordinate system are (ρ...). (i,j) θ (i,j) ),in:

[0058]

[0059]

[0060] In this way, the coordinate system of the original instrument image is transformed into a polar coordinate system. For example... Figure 6 As shown.

[0061] Step Six: Using the position of each major tick value from Step Five as a reference (white circle), smooth the image to the right within a certain area using a Gaussian filter. After processing, we determine the region with the densest concentration of black pixels as the major tick line and record its coordinates. We mark them with circles. Subsequently, we transform the image back to a Cartesian coordinate system and record the coordinates of the major tick lines as (x...). k ,y k ).

[0062] Step 7: Using the rotation center C(x) obtained in Step 2 c ,y c Step 3 yields the pointer tip N(x) p ,y p Step six yields the positions of each main tick mark (x). k ,y k The final reading of the instrument is calculated. First, we use the angle formed between the center point C and the two main scale lines to calculate the distance between any two main scales. That is, how much range each degree represents.

[0063] like Figure 7 As shown, points A and B are the results of positioning two main scale lines. We calculate the difference between the scale values ​​at points A and B. According to the Law of Cosines:

[0064]

[0065]

[0066] Among them, X k This represents the proportional value corresponding to the positioning result of the kth main scale line. Similarly, we include the positioning results of all main scale lines and their corresponding main scale values ​​in the calculation, regardless of whether they are adjacent to each other. By averaging the results of n calculations, a more accurate range spacing d can be obtained. Even if OCR_Meter is interfered with and misses detection, adjacent main scale values ​​and main scale lines will still be included in the calculation, ensuring the robustness of this method. Step 8: Similarly, according to formula (8), the angle formed between the pointer tip, adjacent scale lines and the instrument center can be calculated. Based on this, the positional relationship between the pointer tip and the main scale line can be determined. The final reading Y is calculated using the following formula:

[0067] Y=d·∠ACN+X k (10)

[0068] Example

[0069] Data set preparation. In industrial settings such as gas stations and chemical plants, we equipped robots with cameras and storage devices to collect videos during robot inspections.

[0070] The collected videos were processed, and 5000 instrument images with different backgrounds were cropped. Annotations were added to the dials, pointers, and dial text to create an instrument dataset, of which 80% was used for training and 20% for testing. The images in the dataset included various environmental conditions, such as indoor and outdoor, high and low light, blurry, and partially occluded images.

[0071] Configure an instrument recognition training program to train an instrument recognition neural network. This network can accurately acquire the position of instruments in complex industrial backgrounds and guide the robot camera to continuously focus on and magnify the instruments, forming clearer instrument images. The results are as follows: Figure 8 .

[0072] A pointer training and extraction method was configured, and a pointer extraction neural network was trained. After approximately 100 training iterations, a suitable weight file was selected and saved. After training, this method can recognize the pointers on most watch faces and obtain the position of the pointer tips, as shown in the following results. Figure 9 .

[0073] Configure the text recognition training program, perform approximately 100 training iterations, and select and save appropriate weight files. The trained network can then recognize and filter key information on the dial, as shown in the following results. Figure 10 .

[0074] Configure the main scale positioning program and use the various valid information from the aforementioned steps to calculate the readings.

[0075] Input the images from the verification set into the program to complete the reading test.

[0076] The algorithm was integrated into an inspection robot for actual factory acceptance testing, completing various meter reading tasks. Some results are shown in the table below:

[0077]

[0078] Among them, Y m For manual meter readings, Y represents the reading obtained by this patented method, Unit represents the unit of measurement identified by this method, and X represents the reading obtained by manual meter readings. max -X min E represents the maximum range of the instrument. r E represents the relative error of the method in this patent. g This represents the global error of the method in this patent.

[0079]

[0080]

[0081] It can be seen that this method can achieve good results on various dials in the factory, with an average relative error and global error of only 1.86% and 0.73%, respectively, which is sufficient for the inspection and meter reading tasks in the factory.

Claims

1. A method for automatically reading pointer-type instruments used in inspection robots, characterized in that, The method includes the following steps: Step 1: Design and train the instrument detection network Yolo_Meter; convert robot-captured images into instrument images; during robot inspection, the instrument detection algorithm runs continuously; every 5 seconds, the robot captures an image from the video and sends it to the instrument detection network; The instrument detection algorithm detects the presence of an instrument in the image, and the camera automatically zooms in, focuses, and crops the instrument image to be suitable for subsequent readings based on the instrument's position. CSPDarknet is used as the basic network structure, and SPPF module is used to pool features at different scales and fuse them with the feature pyramid network; SimAM attention mechanism is added to each C3 module of the backbone network; ASFF adaptive feature fusion method is adopted to replace the concat and element-wise operation of Yolov5 neck, the specific operation includes the same rescaling and adaptive fusion, and the weight parameters of adaptive fusion at three feature scales are automatically learned. Step 2: Based on the instrument image cropped in Step 1, design and train a pointer detection network to obtain the relative position of the pointer on the instrument image; design a directed pointer detection network OPDNet based on the pointer, and obtain a recognition box close to the pointer; take the instrument image cropped in Step 1 as input, and output it after passing through a ResNet50 residual network consisting of 49 convolutional layers and one fully connected layer. The Feature Pyramid Network (FPN) is used to fuse feature maps of different scales from ResNet50; two parameters representing the detection box offset are added to the proposal of the Region Proposal Network (RPN). This represents the offset of the proposal box in both the horizontal and vertical directions, resulting in a proposal box with a rotation angle. The oriented RPN is used as the anchor box, and then Softmax is used to determine if there is a target within the anchor box, thus obtaining the exact candidate box with the rotation angle. Subsequently, the candidate boxes are converted into feature maps. and decode the parameters as The feature maps are trained for classification and regression after passing through two fully connected layers; The trained OPDNet accurately identifies the pointer and the center of rotation of the instrument from the dial image and obtains the coordinates of the pointer recognition box. Coordinates of the instrument center identification frame ;in, The x and y coordinates of the center point of the box These represent the width and height of the frame, respectively. The candidate bounding boxes are based on the horizontal rotation angle; Step 3: Based on the pointer recognition frame obtained in Step 2, fit the position of the pointer tip in the instrument image; use the rotation center of the instrument and the position coordinates of the pointer detection frame to calculate the quadrant where the pointer is located; then, obtain the coordinates of the pointer tip through mathematical angle transformation; with the rotation center as the origin, the entire image is divided into left and right parts; when the center of the pointer is located on the right side, the coordinates of the pointer tip are calculated as follows: (1)； (2)； Similarly, when the center of the pointer is to the left of the center of the instrument: (3)； (4)； Fitting pointer tip position This prepares the groundwork for determining the positional relationship between the pointer tip and the scale line; Step 4: Extract the instrument text information from the cropped instrument image in Step 1 to obtain the specific values ​​of units, scale, and scale line coordinates. Divide the instrument text extraction task into two parts: text localization and text recognition. Design a deep neural network OCR_Meter to recognize the main scale values ​​and units on the instrument dial. First, extract the image features of the instrument using MobileNetV3. Then, features at different scales are fused by a feature pyramid network. Activation functions are then used... Binarization mapping is performed on the features, that is, the image is divided into background regions and text regions, thereby completing the text localization; whereby... This represents the magnification factor, set to 50; (5)； Next, the small boxes containing the text are input into ResNet31 to further extract the detailed features of the text; encoding and decoding are completed by a two-layer LSTM long short-term memory network, and the feature maps are processed by Softmax to obtain the text classification result, thus completing text recognition; finally, a filtering algorithm is set according to the character pair recognition result; the filtering algorithm extracts the main scale value and unit, two necessary pieces of information, for subsequent calculations; Step 5: Continue to determine the positional relationship between the pointer tip and the scale line to obtain the specific reading of the instrument. Calculate the reading using the positional difference between the pointer tip and the main scale line; define the main scale line as the thickest and longest scale line closest to the main scale value; use the polar coordinate pixel method to locate the main scale line; use the pointer rotation center identified in Step 2. Using the origin of the transformation, the image is converted from a Cartesian coordinate system to a polar coordinate system; then, all tick marks are arranged in order on the same straight line; let the position of each pixel in the original image be... Then, the pixel coordinates in the Cartesian coordinate system after polar coordinate transformation are: ,in: (6)； (7)； The coordinate system of the original instrument image was transformed into a polar coordinate system; Step Six: Using the position of each major tick value from Step Five as a reference, smooth the image to the right within a certain area using a Gaussian filter; after processing, determine the region with the densest concentration of black pixels as the major tick line and record its coordinates; transform the image back to a Cartesian coordinate system and record the coordinates of the major tick lines as follows. ; Step 7: Using the center of rotation obtained in Step 2 The pointer tip obtained in step three The positions of each main scale line obtained in step six Calculate the final reading of the instrument; first, use the center point. The distance between any two major scale points is calculated by the angle formed between the two major scale lines. In other words, each degree represents a certain range; Points A and B are the results of positioning two main scale lines; the difference between the scale values ​​of points A and B is calculated based on these two points; according to the law of cosines: (8)； (9)； in, Representative and the The proportional value corresponding to the positioning result of each main scale line; the positioning results of all main scale lines and their corresponding main scale values ​​are included in the calculation, regardless of whether they are adjacent to each other; the range spacing is obtained by averaging through n calculations. Even if OCR_Meter is interfered with and misses a detection, adjacent major tick values ​​and major tick lines will still participate in the calculation. Step 8: Calculate the angle formed between the pointer tip, adjacent scale lines, and the instrument center according to formula (8); determine the positional relationship between the pointer tip and the main scale line; and finally take the reading. The calculation formula is: (10)。

Images