Mobile terminal circular instrument reading method combining YOLOv8n and perspective correction

Abstract

The invention discloses a mobile terminal circular instrument reading method combining YOLOv8n and perspective correction, and the method comprises the steps: firstly redesigning a lightweight instrument detection model and a key target detection model based on YOLOv8n, reducing a network structure, and generating a lightweight model which is more suitable for a mobile terminal through employing a LiteFFM module integrated with PConv and an LAMP pruning technology; an efficient circular instrument correction method is provided, a correction matrix is constructed through an instrument edge contour point set, distortion caused by shooting inclination is effectively eliminated, and rotation correction is carried out in combination with marks and scale characters. And finally, accurately calculating a reading through an angle enhancement method, and designing an error correction guide mechanism in the APP to optimize user shooting. Experimental verification shows that when the shooting angle is seriously inclined, the correction algorithm can reduce the average relative error by 60.68%. The operation speed of the system on a mobile phone can reach 9 FPS, the average relative error of readings in a complex environment is only 1.76%, the method is superior to an existing advanced method, and high robustness is shown.

Description

Technical Field

[0001] The invention belongs to the technical field of instruments and meters, and specifically discloses a mobile terminal circular instrument reading method combining YOLOv8n and perspective correction. Background Art

[0002] Pointer instruments are widely used due to their simple structure, good stability and low cost. Since automated data collection and transmission are limited by cost and technology, manual inspection is still a common and economical method. In order to solve the problems of large manual reading errors, low efficiency and poor data authenticity, it is of great significance to digitize instruments on mobile phones and use artificial intelligence technology to assist reading to improve efficiency and accuracy.

[0003] In reality, when inspection workers use their mobile phones to read meters, the shooting direction of the mobile phone camera is random and changes all the time. When the shooting direction is not perpendicular to the meter plane, it will cause geometric distortion of the meter image, seriously affecting the subsequent reading processing.

[0004] In order to eliminate the parallax effect of circular instrument images, the "A Method and Device for Recognizing Patrol Pointer-Type Circular Instruments" with publication number CN113378663A uses SFIT feature matching and KNN neural network to perform feature matching between the standard image and the actual target image, thereby correcting the image. However, it requires a known front view template of the instrument to be tested, and has poor versatility.

[0005] The publication number CN115546794A, "A circular instrument dial correction method based on deep learning and OCR technology", uses an instance segmentation deep learning model to obtain the instrument dial ROI image and the corresponding image mask, detects the edge of the instrument image mask and performs ellipse fitting. Then, the rectangular area of ​​the text printed on the dial is used for correction. This method is easily affected by the lighting environment, and the fitting accuracy is limited, which will lead to inaccurate perspective correction.

[0006] The publication number CN118196414A, "A method for intelligent positioning and correction of multiple types of instruments", uses the U-net series algorithm to segment the instrument dial area, fit the four points corresponding to the major and minor axes of the minimum circumscribed ellipse and the minimum circumscribed circle, calculate the homography matrix, and perform perspective transformation correction. However, this method has too few key points, and the fitted ellipse has a large error.

[0007] In the study of instrument reading, traditional visual instrument reading methods usually use morphological processing and threshold segmentation to calculate the reading. The "A Method for Automatic Recognition of Pointer Instrument Reading" with publication number CN118334668A uses histogram equalization, noise reduction, edge detection and Hough line detection to detect the instrument, scale and pointer. However, traditional methods are easily affected by light changes and background interference, and have limited adaptability to different types of instruments.

[0008] In recent years, deep learning methods have been widely used in instrument reading. The publication number CN118172765A, "A method and system for automatic identification of substation instruments based on deep learning", uses YOLOv5 to detect instruments and combines traditional image recognition algorithms to automatically identify readings. However, in the field of target detection and segmentation, it still faces challenges such as complex backgrounds, small-sized targets, and occlusion, which affect the reading accuracy.

[0009] In addition, the existing automatic meter reading system is basically deployed on the PC or embedded system, which has poor flexibility in outdoor environments. In order to overcome the above challenges, the present invention proposes a mobile circular meter reading method that combines lightweight YOLOv8n and perspective correction, and deploys it on the mobile phone. It can perform real-time and accurate readings of circular meters in complex environments. Summary of the invention

[0010] In view of the shortcomings of the above-mentioned existing methods, the present invention proposes a mobile circular instrument reading method that combines lightweight YOLOv8n and perspective correction. The main work is as follows: (1) An improved lightweight YOLOv8n circular instrument and key target detection model is proposed. While maintaining high detection accuracy, the model volume is greatly compressed, making it suitable for deployment on mobile terminals. (2) A tilt correction algorithm and an azimuth correction algorithm based on the instrument contour are proposed, which can restore the distorted instrument image to a standard front view. Combined with the enhanced angle reading method, the reading error is significantly reduced. (3) An automatic instrument reading APP is developed, and an error correction guidance mechanism is designed to help users quickly take pictures of instruments and obtain readings.

[0011] The present invention provides a mobile terminal circular instrument reading method combining YOLOv8n and perspective correction, comprising the following steps:

[0012] (1) Input image:

[0013] Take a photo with your mobile phone or select the original instrument image from the mobile phone album. If it is in real-time reading mode, capture the frame image from the mobile phone camera preview video as the input image;

[0014] (2) Design lightweight instrument detection model and key target detection model:

[0015] Based on YOLOv8n, a lightweight circular instrument detection model and key target detection model were redesigned. First, the YOLOv8n network structure was reduced. Then, the LiteFFM module with integrated Pconv and LAMP pruning technology were used to generate a lightweight model YOLOv8n-CLP that is more suitable for mobile terminals. Finally, a circular instrument detection model was trained to quickly identify the instrument ROI from the original image and crop the instrument ROI. The key target detection model was trained to detect the pointer, scale, mark and center of the circular instrument panel.

[0016] (3) Instrument tilt angle correction:

[0017] The ellipse detection algorithm based on arc segment adjacency matrix is ​​used to identify the ellipse contour of the circular instrument. Then, the correction matrix is ​​calculated from the ellipse coordinate point set using the newly proposed perspective correction algorithm. Finally, the instrument ROI is interpolated using bilinear interpolation to obtain the corrected instrument image to eliminate the distortion caused by the shooting tilt angle.

[0018] (4) Instrument azimuth correction:

[0019] The marking method is combined with the symmetry method. By pasting specific marks and the position characteristics of the original scale characters of the instrument, the instrument is rotated and corrected to eliminate the distortion caused by the azimuth angle.

[0020] (5) Enhanced angle reading:

[0021] The enhanced angle method is used to combine the key target detection model and OCR to obtain the values ​​of the pointer and the adjacent main scales, and the reading value is calculated using the rotation angle between the pointer and the two nearest main scales.

[0022] (6) Mobile system deployment:

[0023] Using the NCNN neural network computing framework, the network model is deployed on the Android side, the instrument correction and reading algorithms are encapsulated into a dynamic link library, the instrument reading system is deployed on the Android phone using JNI, and an error correction guidance mechanism is designed in the APP to optimize the user shooting experience and improve the instrument image quality.

[0024] The present invention designs a mobile circular instrument reading system based on the improved detection model and perspective correction of YOLOv8n, which mainly includes three parts: lightweight detection model, instrument correction and reading, and mobile terminal deployment. First, after streamlining the YOLOv8n model structure, the detection efficiency is improved by the LiteFFM module, and the LAMP pruning operation is performed, which significantly improves the lightweight effect of the model. Secondly, an inclination angle correction algorithm based on the perspective correction of the instrument contour and an azimuth angle correction algorithm based on the symmetry of the marking and scale characters are proposed, which greatly reduces the error caused by the parallax effect, and uses the enhanced angle method to accurately calculate the reading. Finally, the model is deployed on the mobile terminal through the NCNN framework, and an automatic error correction guidance mechanism is integrated to optimize user shooting. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic flow chart of the circular meter reading method of the present invention;

[0026] Figure 2 It is a schematic diagram of the practical application process of the circular instrument reading method of the present invention;

[0027] Figure 3 It is a schematic diagram of the structure of the lightweight model YOLOv8n-CLP of the present invention;

[0028] Figure 4 Schematic diagram of the LiteFFM module structure in the model of the present invention;

[0029] Figure 5 It is a schematic diagram of the instrument, image plane and mobile phone lens of the present invention;

[0030] Figure 6 This is a schematic diagram of the instrument tilt angle correction of the present invention;

[0031] Figure 7 This is a schematic diagram of the instrument azimuth correction of the present invention;

[0032] Figure 8 It is a schematic diagram of the angle enhancement reading method of the present invention;

[0033] Fig. 9 This is a display of the meter reading mobile phone application interface of the present invention. DETAILED DESCRIPTION

[0034] The present invention is further described below with reference to the embodiments and drawings, but is not intended to limit the present invention.

[0035] Example

[0036] Reference Figure 1-2 , a mobile terminal circular instrument reading method combining YOLOv8n[1] improved detection model and perspective correction, comprising the following steps:

[0037] Step (1) Input image: Take a photo with a mobile phone or select the original instrument image from the mobile phone album. If it is in real-time reading mode, capture a frame image from the mobile phone camera preview video as the input image.

[0038] The circular instrument detection model is used to quickly identify the instrument ROI from the original image and crop the instrument ROI. The key target detection model is used to detect the pointer, scale, mark and center of the circular instrument panel.

[0039] Step (2) Design a lightweight instrument detection model and a key target detection model, such as Figure 3 As shown, the specific steps are:

[0040] (2.1) YOLOv8n, the smallest version in the YOLOv8 series, is selected as the basis. In order to meet the deployment efficiency and performance requirements on the mobile terminal, the YOLOv8n model needs to be lightweight and improved. Since the key targets on the instrument and dial have obvious appearance features and the detection task scale is relatively single, the overall network model is first streamlined, and the number of convolution kernels and the number of output feature channels are reduced to 1 / 8 of YOLOv8n. The network after deletion is called a streamlined network.

[0041] (2.2) The LiteFFM module is designed using partial convolution (PConv) [2]. It exploits the redundancy in the identity map and applies regular convolution to only some input channels for spatial feature extraction, while the remaining channels remain unchanged. For continuous or regular memory access, the first or last continuous cp channel is used as the representative of the entire feature map for calculation. Therefore, the FLOPs of PConv is:

[0042] h×w×k 2 ×c p 2 (1)

[0043] The FLOPs of PConv is only 1 / 16 of that of regular convolution. In addition, the memory access amount of PConv is:

[0044] h×w×2c p +k 2 ×c p 2 ≈h×w×2c p (2)

[0045] It is only 1 / 4 of the conventional convolution, so PConv can better utilize the computing power of the device and has excellent spatial feature extraction capabilities;

[0046] The structure of the LiteFFM module designed based on PConv is as follows Figure 4 As shown, while continuing the rich C2f gradient flow, LiteFFM replaces the original network structure with n Lite Bottlenecks. The bottleneck structure consists of two Pconv Model modules, which are responsible for feature compression and high-dimensional feature mapping respectively, and are fused with the input;

[0047] The Pconv Model module contains partial convolutional layers, which only perform convolutional calculations on partial features, splice the uncalculated part with the calculated part, uses batch normalization to accelerate network convergence, and adopts the SiLU activation function to improve the optimization effect of the deep model. Replacing the C2f module with the LiteFFM module can effectively reduce the network complexity and accelerate the inference process;

[0048] (2.3) Use the adaptive pruning method LAMP[3] based on weight magnitude to prune the model. Calculate the LAMP score through the weight magnitude, which can estimate the impact of pruning on the overall model. All connections with LAMP scores less than the target weight will be pruned;

[0049] To uniformly define the LAMP scores for the fully connected layer and the convolutional layer, assume that each weight tensor is converted into a one-dimensional vector, and the weights are sorted in ascending order according to the given index mapping, that is, when u < v, |W[u]| ≤ |W[v]|, where W[u] represents the mapping of index u to W. The LAMP score of the u-th index of the weight tensor W is defined as:

[0050]

[0051] where, W[u] 2 represents the square of the weight magnitude of the target connection, and Σ v≥u (W[u]) 2 represents the sum of the squares of the weight magnitudes of all the remaining connections in this layer.

[0052] LAMP pruning has the advantages of high computational efficiency, no need to adjust hyperparameters, and not relying on specific model knowledge.

[0053] Step (3) Tilt angle correction of the instrument:

[0054] When using a mobile phone to take pictures of the instrument, the degree of freedom of the lens is relatively high, resulting in the captured instrument image being tilted, which will have an adverse impact on subsequent reading recognition. Therefore, it is necessary to correct the tilt angle of the instrument;

[0055] The coordinate systems of the instrument, the virtual imaging plane, and the mobile phone lens are as follows Figure 5 As shown, in the mobile phone coordinate system, the origin O c is located at the center of the lens, and Yc Axis perpendicular to the lens, X c The Z axis is parallel to the lens and the ground, while c The axis is parallel to the lens and perpendicular to the ground;

[0056] The instrument plane and the virtual image plane are parallel to the lens, and the x, y, z axes and X m , Y m , Z m The axis directions are respectively the same as the above X c , Y c and Z c The axis directions coincide;

[0057] Assume θ x ,θ y and θ z Indicates that the instrument is along X m , Y m and Z m The Y axis is counterclockwise m -Z m , X m -Z m and X m -Y m The corresponding angles of rotation on the plane, the corresponding three rotation matrices are R yoz , R xoz , R xoy , the total rotation matrix R:

[0058]

[0059] The correction of the instrument image is divided into the correction of the tilt angle (θ x ,θ z ) and the azimuth angle θ y Correction of

[0060] First, the inclination angle θ x and θ z Correction, the x-axis and z-axis on the virtual image plane are defined as:

[0061]

[0062] where f represents the (uniform) scaling parameter, used to describe the size transformation from the actual instrument to the image plane;

[0063] When R is projected onto the xz virtual imaging plane, the correction matrix H is obtained. x-z It is expressed as:

[0064]

[0065] The circular instrument image projected on the virtual imaging plane has contour pixel intensity only at the coordinates (X, 0, Z), where X and Z represent the two random processes that constitute the coordinates along the x-axis and z-axis, respectively;

[0066] When a meter is placed on X m -Z m When on a plane, all contour edge points collected along the virtual image plane Should meet:

[0067]

[0068] In the formula, Ε[·] represents the statistical expectation, represents the geometric center of the instrument image projected on the xz plane, ρ x =Ε[(XC x ) 2 ],ρ z =Ε[(ZC z ) 2 ], only when the tilt angle θ x and θ z When it is zero, the edge of the contour is a perfect circle, ρ x =ρ z =ρ o =r 2 / 2;

[0069] When the tilt angle is not zero, use Represents the edge points of the ellipse contour of the instrument image projection, set up We can get:

[0070]

[0071] Perform eigenvalue decomposition on the left side:

[0072]

[0073] Where W and Λ represent the corresponding eigenvector and eigenvalue matrix respectively;

[0074] When x =ρ z =ρ o =r 2 / 2, the following formula holds true:

[0075]

[0076] Where V is a unitary matrix;

[0077] For H x-z Singular value decomposition yields:

[0078] Hx-z =USV (10)

[0079] Where U, V and S represent the left singular vector, right singular vector and singular value matrix respectively;

[0080] According to formula 8-11, we get:

[0081] WΛW H =ρ ο USVV H SU H =ρ ο US 2 U H (11)

[0082] According to formula 12, we get:

[0083] W=U, (12)

[0084]

[0085] Finally we get:

[0086] H x-z =SU -H V (14)

[0087] Only the edge points of the instrument contour need to be collected, and the H x-z The distorted instrument image is restored to its original appearance, successfully eliminating the x and θ z Distortion or uneven scaling effects caused by.

[0088] The instrument's tilt angle correction process, such as Figure 6 As shown in the figure, after the instrument is detected, a high-resolution instrument ROI is cropped from the input original image, and the instrument contour ellipse is fitted using the arc segment adjacency matrix based ellipse detection algorithm (AAMED) [4], and the optimal ellipse is found through constraint conditions and verification comprehensive scores;

[0089] Use v to represent the x and z coordinates of the ROI pixel and use the perspective correction matrix H x-z Calculate the corrected coordinates (x rect ,z rect ), from which the corresponding relationship between the tilt instrument ROI and the correction instrument ROI can be established:

[0090]

[0091] in, is the correction matrix H x-z The pseudo matrix of "<>" means that each item of the vector is rounded to the corresponding integer;

[0092] The total number of pixels in the ROI may be smaller than the total number of pixels in the corrected image. Considering the interpolation speed and the clarity of the processed image, the present invention uses bilinear interpolation to interpolate the corrected image, and finally eliminates the pixel value caused by θ x and θ z Corrected image causing distortion.

[0093] Step (4) Instrument azimuth correction:

[0094] The instrument's azimuth angle θ y The correction of the rotation angle is neglected by many studies. Existing methods use the rectangular outline of the text printed on the dial to determine the rotation angle [5], but due to the limitations of specific printing features and text orientation, the correction angle is usually less than 90°.

[0095] The present invention combines the marking method with the symmetry method, and the direction angle correction method is as follows: Figure 7 As shown in the figure, a mark is pasted directly below the instrument surface to add a fixed position information to the instrument. The polar angle between the line connecting the mark and the center of the dial and the vertical downward direction is calculated.

[0096]

[0097] Where (x 1 ,y 1 ) is the coordinate of the center of the dial, (x 2 ,y 2 ) is the coordinate of the marker center, That is, the rotation angle of the correction azimuth;

[0098] For unmarked instruments, the symmetry method is used: by analyzing the characteristics of the symmetrical distribution of all scale characters along the vertical central axis in the upright instrument image, the angle σ between the line connecting two symmetrical key points (such as the starting scale and the maximum range scale) and the horizontal direction is calculated to achieve the correct rotation of the image.

[0099] The marking method has a correction range of 180°, is highly robust, and is suitable for regular inspections.

[0100] The symmetry rule can correct the instrument by its own characteristics without marking, and is suitable for random scenarios.

[0101] The enhanced angle reading method in step (5) uses the rotation angle between the pointer and the two nearest main scales to calculate the reading value R m :

[0102]

[0103] In the formula, R m is the final reading, Y iis the reading value of the ith main scale, Y i+n is the reading value of the i+nth main scale, Y i and Y i+n Obtained by optical character recognition (OCR), α and β are the relative rotation angles between the pointer and the two adjacent main scales, such as Figure 8 shown.

[0104] The coordinates of the dial center, pointer tip, and main scale characters of the corrected instrument image can be obtained through the key target detection network, and the pointer straight line can be determined by connecting the dial center point and the pointer tip. This method avoids the problem of small scales of the instrument being blocked when it is severely tilted, and can cope with the situation where the tilt angle is less than 70°.

[0105] The mobile terminal system deployment in step (6) specifically includes the following steps:

[0106] (6.1) Using NCNN, a high-performance neural network computing framework optimized for mobile phones, the network model is deployed on the Android side. The detection model weight file is converted into ONNX format, and then converted into NCNN param and bin files, and loaded into the Android project. The main program calls the model for detection and recognition. In addition, the instrument calibration and reading algorithm is encapsulated into a dynamic link library, and the circular instrument reading system is deployed on the mobile side using JNI and NDK;

[0107] Application interface such as Fig. 9 As shown in (1), you can use the mobile phone camera to input the meter picture in real time or the photo album to quickly identify the reading. When running on a mobile phone, a single reading takes only 140ms, and the real-time running speed is as high as 7-9FPS (frames per second);

[0108] (6.2) The flexible position and angle of the user when taking photos with a mobile phone will lead to a large randomness of the photos. Although the correction algorithm can effectively restore the instrument image, the present invention integrates an automatic error correction guidance mechanism in the mobile phone application to help users take better instrument images;

[0109] The mechanism includes: when the instrument is tilted too much, the eccentricity of the dial outline ellipse detected is Where a and b are the semi-major axis and semi-minor axis of the ellipse, respectively. If e>0.9 (the tilt angle is greater than 50°), it is considered to be an over-tilted instrument image;

[0110] For excessively tilted instrument images, a lightweight detection model is used to identify the front and back areas of the instrument, and an arrow is displayed on the phone to indicate the direction from the back to the front. The user can move the phone in the direction of the arrow to reduce the tilt parallax. The tilt correction guidance interface is as follows: Fig. 9 (2) as shown.

[0111] Since the mobile phone is not specially designed for meter photography, the meter ROI may occupy a small proportion in the image and the clarity may be insufficient. To ensure the important position of the meter in the image, the application will ensure that the ratio of the detected meter bounding box area to the entire image area is not less than 1 / 64. If the actual detected ratio is lower than the threshold, the application will prompt the user to move closer to the meter or increase the focal length to shoot.

[0112] Experiment and result analysis

[0113] (1) Lightweight detection model

[0114] The circular instrument dataset used in the experiment contains 1,500 images of 12 types of instruments, taken at different angles and backgrounds. The key target dataset consists of 400 calibrated instrument ROIs, and the Albumentation library is used for online data enhancement.

[0115] The experimental environment is uniformly configured as follows: The processor is Core TM i5-12400F, 32GB RAM, GeForce RTX 3060 (12GB) GPU, Windows 11 operating system. PyTorch 1.8 and CUDA 11.7 versions are used for acceleration, and the mobile device is Xiaomi 14 phone.

[0116] The detection accuracy is evaluated using the model average detection accuracy mAP@0.5, the model scale is measured by the number of parameters, the computational complexity is represented by the number of floating-point operations GFLOPs, and the detection speed is evaluated by measuring the average time required for a single image (including pre- and post-processing time and inference time).

[0117] In order to verify the effectiveness of the proposed improvement scheme, ablation experiments were conducted on the circular instrument dataset and the key target dataset to test the lightweight improvement effect. The results are shown in Table 1. The left and right sides of " / " are the results of the instrument detection network and the key target detection network, respectively, and the optimal performance is bold. YOLOv8n-C represents a streamlined network, which achieves a 92% reduction in parameters and an 84% reduction in GFLOPs while maintaining detection accuracy, and increases the detection speed by 24%. YOLOv8n-CL refers to a streamlined network model using the LiteFFM module. After replacing the C2f module of the backbone network, the model's parameter volume and computational complexity are reduced by 10-14%, and the detection speed is also improved. YOLOv8n-CLP is the final model of the present invention, which refers to the model after continuing to use LAMP pruning and compression, with pruning ratios of 1 / 2 and 1 / 3 respectively. After pruning, the model is fine-tuned for 200 cycles to restore the detection accuracy. The results show that the pruned model effectively reduces the scale and complexity and improves the detection speed with almost no loss of accuracy, which is consistent with the pruning ratio.

[0118] In order to fully verify the lightweight advantage of the model of the present invention, the final model is compared with the current mainstream single-stage algorithms SSD, YOLO, etc. The highest accuracy of the instrument detection network is SSD-VGG16, but SSD has weak detection capabilities for small targets and the model size is huge. The highest accuracy of the key target detection network is YOLOv7-tiny, but the model is not lightweight enough. Compared with other models, the final model of the present invention, YOLOv8n-CLP, greatly reduces the model size, complexity and detection time while the detection accuracy drops by less than 0.005. The outstanding lightweight advantage enables the model to be more efficient on the mobile terminal, and can reach a running speed of 40FPS on the mobile phone.

[0119] Table 1 Model comparison experiment

[0120]

[0121] (2) Tilt meter calibration and reading

[0122] In order to verify the effectiveness of the perspective correction algorithm proposed in this paper, this experiment changes the tilt angles (θ x and θ z ), and 10 images were taken every 10° rotation to simulate the situation where the mobile phone lens was not parallel to the instrument plane.

[0123] The common existing correction method is to use characters or scale key points to fit an ellipse and then correct the instrument image through projection transformation. This method is called the key point correction method. The instrument images corrected by the key point correction method and the method in this paper are automatically read on the mobile phone, and the average relative error δ and average reference error γ are introduced to evaluate the reading accuracy:

[0124]

[0125] Where ai is the reading identified by the algorithm, A i is the actual reading, S is the scale range of the pointer instrument, and n is the total number of samples. The reading results of the three instruments are analyzed for errors and the average values ​​are compared, as shown in Table 2.

[0126] The experimental results show that in the slightly tilted area, the correction effect is limited due to the small tilt angle. The average relative error of the readings after the key point correction method and the correction method in this paper is only reduced by 0.1012% and 0.1950% respectively compared with the uncorrected one. When in the severely tilted area, the correction algorithm can significantly improve the reading accuracy of the pointer instrument. The key point correction method reduces the average relative error by 43.08% and the average reference error by 43.77%. The correction algorithm proposed in this paper can reduce the average relative error by 60.68% and the average reference error by 72.23%. The correction method in this paper has a better effect on improving the reading accuracy than the key point correction method. The reason is that the key point correction method relies on the recognition accuracy that is heavily dependent on the key point coordinates. When the instrument is severely tilted, the dial information area is compressed or blocked, which increases the error. The method in this paper uses the elliptical contour features of the instrument to reduce the negative impact of occlusion and incomplete surface information.

[0127] Table 2 Comparison of reading errors before and after correction when tilted

[0128]

[0129] (3) Robustness of the instrument reading algorithm in complex environments

[0130] In order to verify the robustness of the meter reading algorithm proposed in this paper in complex environments, the experiment simulated images of six types of meters in four adverse environments: blur, overexposure, low light, and rainy weather. A comparative experiment was conducted with the semantic segmentation fusion perspective transformation reading method of Hou[6] and the fan-shaped mask area reading method of Yang[7]. The average relative error δ and the average reference error γ were used to evaluate the performance of each algorithm. The results are shown in Table 3.

[0131] The experimental results show that the method proposed in the present invention is only 0.04% higher than Hou's method in a fuzzy environment. In other environments, the average error is much lower than that of Hou and Yang's methods. In four complex environments, the average relative error of the method proposed in the present invention increased by only 0.9387% at most. Thanks to the organic combination of deep learning and traditional image algorithms, the method proposed in the present invention has a high adaptability to the natural environment. Under adverse conditions such as lighting changes, complex backgrounds, instrument tilt, image blur and bad weather, the average relative error of the reading is only 1.76%, showing excellent robustness and stability.

[0132] Table 3 Comparison of reading errors under harsh conditions

[0133]

[0134] References:

[0135] [1] REISD, KUPEC J, HONG J, and A. Daoudi, "Real-time flying object detection with yolov8," arXiv preprint arXiv:2305.09972, 2023;

[0136] [2]CHEN J,KAO S,HE H,et al.Run,Don't walk:Chasing higher FLOPS forfaster neural networks[C] / / Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition.2023:12021-12031;

[0137] [3]LEE J, PARK S, MO S, et al.Layer-adaptive sparsity for the magnitude-based pruning[J].arXiv preprint arXiv:2010.07611,2020;

[0138] [4]MENG C, LI Z, BAI

[0139] [5] Hou Zhuocheng, Ouyang Hua, Hu Xin, et al., Fuzzy pointer instrument correction reading method based on deep learning[J]. Electronic Measurement Technology, 2023, 46(09): 158-165.

[0140] [6]HOU L,WANG S,SUN X,et al.Apointer meter reading recognition methodbased on YOLOX and semantic segmentation technology[J].Measurement,2023,218:113241;

[0141] [7]YANG C,ZHU R,YU X,et al.Real-time reading system for pointer meterbased on YolactEdge[J].Connection Science,2023,35(1):2241669;

[0142] In view of the problem that the existing circular meter reading algorithm is easily affected by adverse factors such as shooting angle and complex environment, the method of the present invention proposes a mobile circular meter reading method combining YOLOv8n and perspective correction, and successfully deploys it on the mobile phone. The experimental results show that the final lightweight detection model has a weight file size of only 375KB, GFLOPs of only 0.4, and the detection speed is increased by 47.83% while maintaining high accuracy. x / θ z ∈[-60°,-20°)∪(20°,60°]), the correction method of the present invention can reduce the average relative error by 60.68% and the average reference error by 72.23%, thus widening the shooting angle of effective reading. In complex environments, the average relative error of the system reading is only 1.76%, which is significantly better than the recently published methods of Hou[6] and Yang[7] and has high robustness. The present invention is lightweight and fast. When running on a mobile phone, a single reading takes only 140ms, and the real-time running speed is as high as 7 to 9 FPS (frames per second).

Claims

1. A mobile circular instrument reading method combining YOLOv8n and perspective correction, characterized in that: The method comprises the following steps: (1) Input image: Take a photo with your mobile phone or select the original instrument image from the mobile phone album. If it is in real-time reading mode, capture the frame image from the mobile phone camera preview video as the input image; (2) Design lightweight instrument detection model and key target detection model: Based on YOLOv8n, a lightweight circular instrument detection model and key target detection model were redesigned. First, the YOLOv8n network structure was reduced. Then, the LiteFFM module with integrated Pconv and LAMP pruning technology were used to generate a lightweight model YOLOv8n-CLP that is more suitable for mobile terminals. Finally, a circular instrument detection model was trained to quickly identify the instrument ROI from the original image and crop the instrument ROI. The key target detection model was trained to detect the pointer, scale, mark and center of the circular instrument panel. (3) Instrument tilt angle correction: The ellipse detection algorithm based on arc segment adjacency matrix is ​​used to identify the ellipse contour of the circular instrument. Then, the correction matrix is ​​calculated from the ellipse coordinate point set using the newly proposed perspective correction algorithm. Finally, the instrument ROI is interpolated using bilinear interpolation to obtain the corrected instrument image to eliminate the distortion caused by the shooting tilt angle. (4) Instrument azimuth correction: The marking method is combined with the symmetry method. By pasting specific marks and the position characteristics of the original scale characters of the instrument, the instrument is rotated and corrected to eliminate the distortion caused by the azimuth angle. (5) Enhanced angle reading: The enhanced angle method is used to combine the key target detection model and OCR to obtain the values ​​of the pointer and the adjacent main scales, and the reading value is calculated using the rotation angle between the pointer and the two nearest main scales. (6) Mobile system deployment: Using the NCNN neural network computing framework, the network model is deployed on the Android side, the instrument correction and reading algorithms are encapsulated into a dynamic link library, the instrument reading system is deployed on the Android phone using JNI, and an error correction guidance mechanism is designed in the APP to optimize the user shooting experience and improve the instrument image quality.

2. The mobile terminal circular meter reading method according to claim 1, characterized in that: The step (2) of designing a lightweight instrument detection model and a key target detection model comprises the following specific steps: (2.1) Reduce the YOLOv8n network structure and reduce the number of convolution kernels and output feature channels to 1 / 8 of YOLOv8n; (2.2) Design the LiteFFM module, which uses partial convolution PConv. Design the LiteFFM module structure based on PConv, and use the LiteFFM module to replace the C2f module. LiteFFM replaces the original network structure with n LiteBottlenecks while continuing the rich gradient flow of C2f. The bottleneck structure consists of two Pconv Model modules, which are responsible for feature compression and feature high-dimensional mapping, and are fused with the input. The Pconv Model module contains some convolutional layers, performs convolution calculations on only some features, concatenates the uncalculated parts with the calculated parts, uses batch normalization to speed up network convergence, and uses the SiLU activation function to improve the optimization effect of the deep model; (2.3) The model is pruned using the adaptive pruning method LAMP based on weight amplitude. The LAMP score is calculated by the weight amplitude, and the impact of pruning on the overall model can be estimated. All connections with LAMP scores less than the target weight will be pruned.

3. The mobile terminal circular meter reading method according to claim 1, characterized in that: The specific method of the inclination angle correction of the instrument in step (3) is: Determine the coordinate system of the instrument, virtual imaging plane and mobile phone lens. In the mobile phone coordinate system, the origin O c Located in the center of the lens, Y c Axis perpendicular to the lens, X c The Z axis is parallel to the lens and the ground, while c The axis is parallel to the lens and perpendicular to the ground; The instrument plane and the virtual image plane are parallel to the lens, and the x, y, z axes and X m , Y m , Z m The axis directions are respectively the same as the above X c , Y c and Z c The axis directions coincide; Correction matrix H x-z The specific calculation method is: When the phone is tilted to shoot the instrument, the coordinate system X is set at the center of the instrument. m , Y m , Z m ; Assume θ x ,θ y and θ z Indicates that the instrument is along X m , Y m and Z m The Y axis is counterclockwise m -Z m , X m -Z m and X m -Y m The corresponding angles of rotation on the plane, the corresponding three rotation matrices are R yoz , R xoz , R xoy , the total rotation matrix R: The tilt angle θ x and θ z During correction, the x-axis and z-axis on the virtual image plane are defined as: where f represents the (uniform) scaling parameter, used to describe the size transformation from the actual instrument to the image plane; When R is projected onto the xz virtual imaging plane, the correction matrix H is obtained. x-z It is expressed as: The circular instrument image projected on the virtual imaging plane has contour pixel intensity only at the coordinates (X, 0, Z), where X and Z represent the two random processes that constitute the coordinates along the x-axis and z-axis, respectively; When a meter is placed on X m -Z m When on a plane, all contour edge points collected along the virtual image plane Should meet: In the formula, Ε[·] represents the statistical expectation, represents the geometric center of the instrument image projected on the xz plane, ρ x =Ε[(XC x ) 2 ],ρ z =Ε[(ZC z ) 2 ], only when the tilt angle θ x and θ z When it is zero, the contour edge is a perfect circle, ρ x =ρ z =ρ o =r 2 / 2; When the tilt angle is not zero, use Represents the edge points of the ellipse contour of the instrument image projection, set up We can get: Perform eigenvalue decomposition on the left side: Where W and Λ represent the corresponding eigenvector and eigenvalue matrix respectively; When x =ρ z =ρ o =r 2 / 2, the following formula holds true: Where V is a unitary matrix; For H x-z Singular value decomposition yields: H x-z =USV (7) Where U, V and S represent the left singular vector, right singular vector and singular value matrix respectively; According to formula 8-11, we get: WΛW H =ρ ο USVV H SU H =ρ ο US 2 U H (8) According to formula 12, we get: W=U,(9) Finally we get: H x-z =SU -H V (11) Only the edge points of the instrument contour need to be collected, and the H x-z The distorted instrument image is restored to its original appearance, successfully eliminating the x and θ z Distortion or uneven scaling effects caused by After the instrument detection model in step (2) detects the instrument, a high-resolution instrument ROI is cropped from the input original image, and the instrument contour ellipse is fitted using an ellipse detection algorithm based on an arc segment adjacency matrix, and the best ellipse is found through constraint conditions and verification comprehensive scores; Use v to represent the x and z coordinates of the ROI pixel and use the perspective correction matrix H x-z Calculate the corrected coordinates (x rect ,z rect ), thereby establishing the corresponding relationship between the tilt instrument ROI and the correction instrument ROI: in, is the correction matrix H x-z The pseudo matrix "<>" means that each item of the vector is rounded to the corresponding integer; The total number of pixels in the ROI may be smaller than the total number of pixels in the corrected image. Considering the interpolation speed and the clarity of the processed image, the present invention uses bilinear interpolation to interpolate the corrected image, and finally eliminates the pixel value caused by θ x and θ z Corrected image causing distortion.

4. The mobile terminal circular meter reading method according to claim 1, characterized in that: Step (4) of the instrument azimuth correction combines the marking method with the symmetry method. A mark is pasted directly below the instrument surface to add a fixed position information to the instrument. The polar angle between the mark and the center of the dial and the vertical downward direction is calculated. Where (x1, y1) is the coordinate of the dial center, (x2, y2) is the coordinate of the mark center, That is, correct the rotation angle of the azimuth; for unmarked instruments, the symmetry method is used: by analyzing the characteristics of the symmetrical distribution of all scale characters along the vertical central axis in the upright instrument image, the angle σ between the connecting line of a set of symmetrical key points and the horizontal direction is calculated, such as the starting scale and the maximum range scale are a set of symmetrical key points, so as to achieve the correct rotation of the image.

5. The mobile terminal circular meter reading method according to claim 1, characterized in that: The enhanced angle reading method in step (5) uses the rotation angle between the pointer and the two nearest main scales to calculate the reading value R m : In the formula, R m is the final reading, Y i is the reading value of the ith main scale, Y i+n is the reading value of the i+nth main scale, Y i and Y i+n Obtained by optical character recognition (OCR), α and β are the relative rotation angles between the pointer and two adjacent main scales.

6. The mobile terminal circular meter reading method according to claim 1, characterized in that: The mobile terminal system deployment in step (6) specifically includes the following steps: (6.1) Using NCNN, a high-performance neural network computing framework optimized for mobile phones, the network model is deployed on the Android side. The detection model weight file is converted into ONNX format, and then converted into NCNN param and bin files, and loaded into the Android project. The main program calls the model for detection and recognition. In addition, the instrument calibration and reading algorithms are encapsulated into dynamic link libraries, and the circular instrument reading system is deployed on the mobile side using JNI and NDK; the instrument pictures are input in real time through the mobile phone camera or the album to quickly identify the readings; (6.2) Integrating an automatic error correction guidance mechanism into mobile phone applications can help users take better instrument images; The mechanism includes: when the instrument is tilted too much, the eccentricity of the dial outline ellipse detected is Where a and b are the semi-major axis and semi-minor axis of the ellipse respectively; if e>0.9, that is, when the tilt angle is greater than 50°, it is considered to be an over-tilted instrument image; For excessively tilted instrument images, a lightweight detection model is used to identify the front and back areas of the instrument, and an arrow is displayed on the phone to indicate the direction from the back to the front. Users can reduce the tilt parallax by moving the phone in the direction of the arrow. To ensure the important position of the instrument in the image, the application will ensure that the ratio of the detected instrument bounding box area to the entire image area is not less than 1 / 64. If the actual detected ratio is lower than the threshold, the application will prompt the user to move closer to the instrument or increase the focal length to shoot.

Images