A pointer instrument recognition and reading method based on deep learning

Abstract

The present invention discloses a pointer-type instrument recognition and reading method based on deep learning. First, the instrument image is obtained by an inspection robot, a data set A is constructed, an instrument target detection model is built and trained, the instrument target detection model is used to identify the data set A, and the image is cropped according to the instrument position information; the cropped image is used to construct a pointer and range segmentation data set B, a pointer target detection model and a range semantic segmentation model are constructed, the pointer and range of the instrument are identified and segmented, and a pointer image and a range image are obtained; the pointer and range images are processed, and numerical calculations are performed according to the angle method to obtain the pointer reading. The present invention uses deep learning to replace the manual reading of the traditional method, solves the problem that the instrument detection is difficult, the extraction speed is slow, and the reading accuracy is low due to factors such as the complex substation environment and the unclear instrument characteristics in the actual environment, and greatly improves the instrument detection speed and the reading accuracy.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and computer vision, and more specifically to a method for identifying and reading a pointer-type instrument based on deep learning. Background Art

[0002] With the rapid development of science and technology, electricity has been integrated into all aspects of our lives. Substations are places in the power system that transform voltage and current, receive and distribute electric energy. It is particularly important to maintain the smooth operation of substations. Pointer instruments have the advantages of strong anti-interference ability, simple structure and low cost. They are widely used in substations and require regular inspection of instruments. However, the current domestic inspection method for instruments in substations is mainly manual inspection, which often causes misreading or missing reading due to factors such as lack of concentration and visual fatigue. The large workload leads to low work efficiency, and some places with dangerous working conditions and high safety risks (strong radiation, high temperature) are not suitable for manual work.

[0003] Substation instruments are of different types and the background is complex. Traditional instrument detection methods have many shortcomings, such as Hough circle detection, speeded up robust features (SURF) method and template image matching. Hough circle detection is easily interfered by objects with similar appearance to instruments. Although SURF instrument detection can detect and extract instruments from the background environment, it mostly relies on high-quality instrument templates and has poor robustness to environmental changes. In view of the shortcomings of manual inspection and traditional algorithms, the use of deep learning-based target detection algorithms can successfully solve the above problems. Summary of the invention

[0004] In order to solve the deficiencies in the prior art, the purpose of the present invention is to provide a pointer instrument recognition and reading method based on deep learning, so as to solve the problems in the background technology and improve the accuracy of the reading.

[0005] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0006] A method for identifying and reading a pointer instrument based on deep learning comprises the following steps:

[0007] Step 1: Obtain instrument images through the inspection robot, perform preprocessing, and use annotation software to manually annotate the dial. The annotated images are divided into a training set and a validation set to construct a data set A.

[0008] Step 2: Build and train the instrument target detection model based on the constructed instrument dataset A;

[0009] Step 3: Use the instrument recognition model to identify the data set A and crop the original image according to the instrument location information;

[0010] Step 4: Use the cropped images obtained in step 3 to construct a pointer and range segmentation dataset B, annotate the pointer and range respectively, and divide the annotated images into a training set and a validation set;

[0011] Step 5: Based on the constructed data set B, build and train the pointer target detection model and the range semantic segmentation model to obtain the pointer recognition model and the range recognition model;

[0012] Step 6: According to the pointer target detection model and the range semantic segmentation model trained in step 5, the pointer and the range of the instrument are identified and segmented to obtain a pointer image and a range image;

[0013] Step 7: Process the range and pointer image identified in step 6, perform numerical calculations according to the angle method, and obtain the pointer reading.

[0014] To further optimize the technical solution, in step 1, the collection of instrument images is required to be carried out under conditions of different backgrounds, lighting, shooting angles, and sizes.

[0015] To further optimize the technical solution, in step 1, the data set is expanded using image enhancement technology.

[0016] To further optimize the technical solution, in step 1, the instrument dial is labeled using the labelImg tool.

[0017] To further optimize the technical solution, in step 1, the collected images are divided so that the training set images and the verification set images account for 90% and 10% of the total number of images, respectively.

[0018] To further optimize the technical solution, in step 2, the instrument recognition model is built based on YOLOv5.

[0019] To further optimize the technical solution, in step 4, the instrument pointer is labeled using the labelImg tool, and the instrument scale lines are labeled using the labelme tool.

[0020] To further optimize the technical solution, in step 5, the pointer recognition model is built based on YOLOv5, and the range recognition model is built based on DEEPLABv3+.

[0021] To further optimize the technical solution, the step 7 further includes the following steps:

[0022] Step 7-1, pre-process the identified range image, including grayscale processing, dilation processing, and binarization processing, and use the OpenCV computer vision open source library to perform ellipse fitting on the obtained range image, and the ellipse center O is the pointer rotation center;

[0023] Step 7-2: Based on step 7-1, make a mask image of the ellipse edge, intersect the mask image with the expansion image of the range, and obtain the fitting curve of the range. According to the convolution principle, define a 3*3 convolution kernel to traverse the curve, find the two endpoints of the range, which are the minimum range point P1 and the maximum range point P2, and use the OpenCV library to calculate the angle from OP1 to OP2 clockwise, and record the angle as θ;

[0024] Step 7-3: mask the identified pointer image to keep only the pointer part;

[0025] Step 7-4: Based on step 7-3, perform OTSU binary processing on the pointer image and use the Canny operator to detect edges, such as Figure 5 As shown, perform HoughLine detection on the edge map to obtain the two endpoints of the pointer. The one closer to point O is endpoint C, and the one farther from point O is endpoint D. Use the OpenCV library to calculate the angle from OP1 clockwise to OD, and record the angle as r;

[0026] Step 7-5: Read the pointer according to its position on the fitting curve and use the angle method to read the value. Where V min Indicates the minimum range value, V max Indicates the maximum range value.

[0027] The beneficial effect of the present invention is that, compared with the prior art, the recognition and reading method of the pointer instrument of the substation proposed by the present invention based on deep learning first uses the YOLOv5 target detection algorithm to locate the instrument, crops the recognized dial, uses the YOLOv5 target detection algorithm to locate the instrument pointer on the cropped dial image, and uses DEEPLABv3+ to segment the instrument scale line, and finally processes the detected pointer and scale line first and uses the angle method to read the reading. In the actual environment, the complex substation environment and the unclear instrument features make the instrument detection difficult, the extraction speed is slow, and the reading accuracy is low. The instrument recognition and reading method proposed by the present invention can greatly improve the instrument detection speed and the reading accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of the method of the present invention;

[0029] Figure 2 It is a schematic diagram of the range segmentation result in the present invention;

[0030] Figure 3 This is a schematic diagram of the results of ellipse fitting of the range according to the present invention;

[0031] Figure 4 It is a schematic diagram of the range curve fitting result of the present invention;

[0032] Figure 5 It is a schematic diagram of the detection result of the HoughLine straight line of the pointer edge graph according to the present invention;

[0033] Figure 6 It is a schematic diagram of the recognition result of the present invention;

[0034] Among them, (a) is a schematic diagram of the pointer at the range position; (b) is a schematic diagram of the pointer and the range result; DETAILED DESCRIPTION

[0035] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] A pointer instrument recognition and reading method based on deep learning, combined with Figure 1 The following steps are shown:

[0037] Step 1: Obtain instrument images through the inspection robot, perform preprocessing, and use annotation software to manually annotate the dial. The annotated images are divided into a training set and a validation set to construct a data set A.

[0038] For the collection of instrument images, it is required to collect under different backgrounds, lighting, shooting angles and sizes.

[0039] The dataset is expanded using image enhancement techniques.

[0040] Use the labelImg tool to label the instrument dial.

[0041] The collected images are divided so that the training set images and the validation set images account for 90% and 10% of the total number of images respectively.

[0042] Step 2: Build and train the instrument target detection model based on the training set in dataset A;

[0043] The instrument target detection model is built based on YOLOv5.

[0044] Step 3: Use the instrument target detection model to identify the data set A and crop the original image according to the instrument location information;

[0045] Step 4: Use the cropped images from step 3 to construct a pointer and range segmentation dataset B, annotate the pointer and range respectively, and divide the annotated images into a training set and a validation set;

[0046] Use the labelImg tool to label the instrument pointer, and use the labelme tool to label the instrument range;

[0047] Step 5: Based on data set B, build and train a pointer target detection model and a range semantic segmentation model to obtain a pointer target detection model and a range semantic segmentation model;

[0048] The pointer recognition model is built based on YOLOv5, and the range recognition model is built based on DEEPLABv3+.

[0049] Step 6: According to the pointer target detection model and range semantic segmentation model trained in step 5, the pointer and range of the instrument are identified and segmented to obtain the pointer image and range image. The range image is shown in Figure 2 As shown;

[0050] Step 7: Process the range and pointer image identified in step 6, perform numerical calculations according to the angle method, and obtain the pointer reading. Specifically, step 7 should also include the following steps:

[0051] Step 7.1: Preprocess the identified range image, including grayscale processing, expansion processing, and binarization processing, such as Figure 3 As shown, the OpenCV computer vision open source library is used to fit the range image to an ellipse, and the ellipse center O is the pointer rotation center;

[0052] Step 7-2: Based on step 7-1, make a mask image of the ellipse edge, intersect the mask image with the expansion image of the range, and get the fitting curve of the range. According to the convolution principle, define a 3*3 convolution kernel to traverse the curve, find the two endpoints of the range, which are the minimum range point P1 and the maximum range point P2, and use the OpenCV library to calculate the angle from OP1 to OP2 clockwise, and record the angle as θ, as follows: Figure 4 As shown;

[0053] Step 7-3: mask the identified pointer image to keep only the pointer part;

[0054] Step 7-4: Based on step 7-3, perform OTSU binary processing on the pointer image and use the Canny operator to detect edges, such as Figure 5 As shown, HoughLine detection is performed on the edge map to obtain the two endpoints C and D of the pointer. The one closer to point O is endpoint C, and the one farther from point O is endpoint D. Since the pointer has a certain width, endpoint C includes C1 and C2, and endpoint D includes D1 and D2, as shown in Figure 6 As shown, use the OpenCV library to calculate the angle from OP1 clockwise to OD1 or from OP1 clockwise to OD2, and record the angle as r;

[0055] Step 7-5: Read the pointer according to its position on the fitting curve and use the angle method to read the value. Where V min Indicates the minimum range value, V max Indicates the maximum range value.

[0056] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A pointer instrument recognition and reading method based on deep learning, characterized in that: The following steps are involved: Step 1: Obtain instrument images through the inspection robot, perform preprocessing, and use annotation software to manually annotate the dial. The annotated images are divided into a training set and a validation set to construct a data set A. Step 2: Build and train the instrument target detection model based on the constructed instrument dataset A; Step 3: Use the instrument target detection model to identify the data set A and crop the original image according to the instrument location information; Step 4: Use the cropped images obtained in step 3 to construct a pointer and range segmentation dataset B, annotate the pointer and range respectively, and divide the annotated images into a training set and a validation set; Step 5: Based on the constructed data set B, build and train the pointer target detection model and the range semantic segmentation model to obtain the pointer target detection model and the range semantic segmentation model; Step 6: According to the pointer target detection model and the range semantic segmentation model trained in step 5, the pointer and the range of the instrument are identified and segmented to obtain a pointer image and a range image; Step 7, processing the range and pointer image identified in step 6, and finally performing numerical calculation according to the angle method to obtain the pointer reading; The step 7 also includes the following steps: Step 7-1, pre-process the identified range image, including grayscale processing, dilation processing, and binarization processing, and use the OpenCV computer vision open source library to perform ellipse fitting on the obtained range image, and the ellipse center O is the pointer rotation center; Step 7-2: Based on step 7-1, make a mask image of the ellipse edge, intersect the mask image with the expansion image of the range, and obtain the fitting curve of the range. According to the convolution principle, define a 3*3 convolution kernel to traverse the curve, find the two endpoints of the range, which are the minimum range point P1 and the maximum range point P2, and use the OpenCV library to calculate the angle from OP1 to OP2 clockwise, and record the angle as θ; Step 7-3, masking the identified pointer image to retain only the pointer part; Step 7-4: Based on step 7-3, perform OTSU binary processing on the pointer image and use the Canny operator to detect edges. Perform HoughLine detection on the edge map to obtain the two endpoints of the pointer. The one closer to point O is endpoint C, and the one farther from point O is endpoint D. Use the OpenCV library to calculate the angle from OP1 to OD in a clockwise direction, and record the angle as r. Step 7-5: Read the pointer according to its position on the fitting curve and use the angle method to read the value. Where V min Indicates the minimum range value, V max Indicates the maximum range value.

2. The pointer instrument recognition and reading method based on deep learning according to claim 1 is characterized in that: In step 1, the collection of instrument images is required to be performed under conditions of different backgrounds, lighting, shooting angles, and sizes.

3. The pointer instrument recognition and reading method based on deep learning according to claim 1 is characterized in that: In step 1, the data set is expanded using image enhancement technology.

4. The method for identifying and reading pointer-type instruments based on deep learning according to claim 1, characterized in that: In step 1, the instrument dial is labeled using the labelImg tool.

5. The pointer instrument recognition and reading method based on deep learning according to claim 1 is characterized in that: In step 1, the collected images are divided so that the training set images and the verification set images account for 90% and 10% of the total number of images respectively.

6. The method for identifying and reading pointer instruments based on deep learning according to claim 1, characterized in that: In step 2, the instrument target detection model is built based on YOLOv5.

7. The method for identifying and reading pointer instruments based on deep learning according to claim 1, characterized in that: In step 4, the instrument pointer is labeled using the labelImg tool, and the instrument range is labeled using the labelme tool.

8. The method for identifying and reading pointer instruments based on deep learning according to claim 1, characterized in that: In step 5, the pointer target detection model is built based on YOLOv5, and the range semantic segmentation model is built based on DEEPLABv3+.

Images