A method for assisting in reading industrial tables based on YOLOv5 and U2-Net frameworks

Abstract

A method for reading industrial watches based on the YOLOv5 and U2-Net framework includes the following steps: S1: Locate the industrial watch in an image using YOLOv5 to obtain an industrial watch image; S2: Segment the pointer and scale lines in the industrial watch image using U2-Net to obtain a pointer and scale line image; S3: Perform preprocessing such as erosion and sharpening on the image to facilitate subsequent processing of the pointer and scale lines; S4: Unfold the dial in the preprocessed image into a rectangular image; S5: Convert the rectangular image obtained in S4 into a one-dimensional array based on the pixel values ​​of the pointer and scale lines, and binarize it using the mean; S6: Locate the position of the pointer relative to the scale lines based on the values ​​of the pointer and scale lines, and determine the dial type and obtain the dial range based on the number of scale lines; S7: Multiply the relative position of the pointer by the range to obtain the target reading. This invention is applicable to any tilted dial and can accurately obtain the dial reading.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, deep learning, and other technologies, and specifically to a method for assisting in reading industrial tables based on the YOLOv5 and U2-Net framework. Background Technology

[0002] Nowadays, almost all power plants need to regularly monitor industrial meter readings to ensure the normal operation of the project and the safety of the surrounding area. However, industrial meters are scattered, manual meter reading is inefficient, and some working environments are not suitable for human operation. To address these issues, robots can be used to patrol the factory and obtain meter data through cameras to automatically read the data and achieve the goal.

[0003] With continuous technological advancements, the use of robots is becoming increasingly widespread and cost-effective, making them particularly suitable for replacing manual identification. In the industrial production sector, intelligent identification of industrial instruments has significant application value. Currently, intelligent identification solutions for industrial instruments primarily involve using industrial cameras to collect image data, followed by image processing algorithms to complete the reading recognition and recording of the instruments. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a method for assisting in reading industrial meters based on the YOLOv5 and U2-Net frameworks. This method enables robots to read industrial meters during navigation, thereby freeing workers from dangerous environments and significantly reducing labor costs. At the same time, this method can reduce the subjective factors brought about by human meter reading, thereby improving the efficiency and accuracy of reading and has good practicality and robustness.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] A method for assisting in reading industrial tables based on YOLOv5 and U2-Net framework includes the following steps:

[0007] Step S1: Input the image captured by the camera into the trained YOLOV5 model, then detect the position of the dial and obtain the target image of the industrial dial;

[0008] Step S2: Input the target image of the dial into the trained U2-Net model, and then segment the pointer and tick marks in the target image. Use green and red to segment the tick marks and pointer respectively. This is to facilitate the subsequent differentiation of the tick marks and pointer based on the pixel values. Finally, output the image segmented with tick marks and pointer.

[0009] Step S3: Perform image preprocessing on the obtained scale line and pointer image, mainly including image scaling to a fixed size, image erosion, and image sharpening, to make the scale lines and pointer information easier to distinguish, and output the result.

[0010] Step S4: Transform the preprocessed image to Cartesian coordinates using polar coordinates, and convert the ring shape into a rectangular image starting from the bottom of the dial ring.

[0011] Step S5: Convert the rectangular image from a two-dimensional array to a one-dimensional array to obtain a one-dimensional array of pointers and tick marks, and perform mean binarization on the one-dimensional array of pointers and tick marks.

[0012] Step S6: Calculate the position of the pointer corresponding to the scale line based on the binarized one-dimensional array obtained in S5, and deduce the dial range based on the value of the scale line array;

[0013] Step S7: Calculate the industrial meter value based on the relative position of the pointer and the range.

[0014] Furthermore, step S1 includes the following steps:

[0015] Step S11: Construct training and test sets: First, select an appropriate number of datasets, requiring different dials and different ranges to be included in the dataset, selecting 80% of the data as the dataset;

[0016] Step S12: Training set image data annotation: Use LabelImg to annotate the images in the training set and select the location of the industrial dial in the image;

[0017] Step S13: Multi-round training of the model: Set the number of training rounds to 10 and the number of iterations to 4;

[0018] Step S14: Verify the training effect using the test set: Verify the test set data.

[0019] Furthermore, in step S2, U2-Net is used to segment the pointer and tick marks. Similarly, LabelImg is used to annotate the images in the training set to mark the positions of the tick marks and pointer. Finally, the images of the segmented tick marks and pointer are displayed with a black background. The tick marks are segmented in green, while the pointer is segmented in red, so that the pointer and tick marks can be distinguished based on pixel values ​​later.

[0020] Furthermore, step S3 involves setting the segmented scale line and pointer images to a fixed size [512, 512], and then performing image erosion. The convolution kernel size of the image erosion is set to 1, which facilitates subsequent readings after preprocessing.

[0021] In step S4, the preprocessed image is transformed from polar coordinates to the corresponding Cartesian coordinate system using formulas (1) and (2), with the dial ring as the reference point.

[0022] x=r+r×cos(θ) (1)

[0023] y=rr×sin(θ) (2)

[0024] Where x is the transformed Cartesian coordinate system, r is the polar radius in polar coordinates, θ is the angle value in polar coordinates, and y is the transformed Cartesian coordinate system.

[0025] Step S5 includes the following steps:

[0026] Step S51: Determine the position of the pointer and tick mark based on the difference in pixel values. The pixel value range of the pointer is set to [70, 80], while the pixel range of the tick mark is set to [100, 200]. The background pixel value is set to 0. Because the pixel value difference is large, it is easy to determine the pointer and tick mark. This reduces the dimension of the two-dimensional array to a one-dimensional array. The length of the one-dimensional array is 512. The one-dimensional array is used to represent the pointer and tick mark respectively.

[0027] Step S52: After identifying the pointer and tick marks, obtain a one-dimensional array of pointers and tick marks. First, calculate the average value of the values ​​in the array, and then iterate through the array to compare whether the values ​​in the array are greater than the average value. If they are greater than the average value, set it to 1; otherwise, set it to 0. Similarly, the length of the binarized pointer array and tick mark array is 512.

[0028] In step S6, the relative position of the pointer is located based on the binarized pointer array and scale line array. First, the starting position of the scale line is found, which is the first two consecutive non-zero values ​​in the scale line. Note that it must be the first two consecutive non-zero values ​​because there may be noise, and the pixel position of the line in the image is also continuously represented. Then, the position of the non-zero value in the pointer array is calculated to obtain the position of the relative scale line. At the same time, the range of the dial is calculated based on the number of scale lines. The range needs to be set in advance.

[0029] In step S7, the dial value is calculated using formula (3) based on the range information and the relative position of the pointer.

[0030] M=t×(l÷s) (3)

[0031] Where M is the dial value, t is the relative position of the pointer in the image, l is the range of the standard dial, and s is the number of scale lines on the standard dial. The range divided by the number of scale lines represents the unit value of one scale division.

[0032] The main advantages of this invention are as follows: This invention can effectively identify the dial in an image, segmenting the pointer and scale lines. Based on the obtained pointer and scale lines, the image segmentation allows for accurate identification of the pointer's position relative to the scale lines, thereby reading the dial's value from the image. It can be applied to reading any tilted industrial dial, offering greater robustness and practicality. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating a method for reading industrial tables using the YOLOv5 and U2-Net framework.

[0034] Figure 2 This is a diagram showing the recognition results of the industrial table.

[0035] Figure 3 This is a diagram showing the segmentation results of the industrial table. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0037] Reference Figures 1-3 A method for assisting in reading industrial tables based on YOLOv5 and U2-Net framework. Figure 1 As shown in the flowchart, the method comprises three modules: using YOLOv5 to identify industrial watches, using U2-Net to segment the pointers and scale lines on the dial, and reading the industrial dial. The method includes the following steps.

[0038] Step S1: The YOLOv5 algorithm takes the entire image as input and directly regresses the position and category of the candidate bounding boxes in the output layer. An image is divided into S×S grid cells, and the detection of the target object is performed using these grid cells. The image acquired by the robot is input into YOLOv5 for dial recognition to obtain the dial from the original image. Figure 2 The recognition results are displayed.

[0039] Step S2: First, use the LabelMe annotation tool to annotate the scale and pointer of the pointer gauge. Taking a pressure gauge as an example, after annotation, a semantic segmentation dataset of 500 pointer gauge images is obtained. The pointer gauge images are then trained using U2-Net. During training, the image size is adjusted to 512×512, the hyperparameter learning rate is set to 0.0001, the batch size for each iteration is 4, and training is completed after 200 epochs. Then, the recognized dial images are input into U2-Net for semantic segmentation, resulting in images of the pointer and scale of the industrial pointer gauge. These images are set to a black background, with red lines representing pointer information and green lines representing scale information. Figure 3 The segmentation results are displayed.

[0040] Step S3: First, the U2-Net model is segmented to obtain tick marks and pointer images, which are set to a fixed size [512, 512]. Then, image erosion is performed, with the convolution kernel size of the image erosion set to 1. This preprocessing facilitates subsequent readings.

[0041] Step S4: First, we need to determine the center of the circle in the image. It should be the image obtained through the dial recognition. So we set the center of the image to the center of the dial [256,256], and define the radius of the circle as 250 and the circumference d of the circle as 1570. Then we set the angle θ and the image formula (1). The preprocessed image is transformed from polar coordinates to the corresponding Cartesian coordinates by using the dial ring as the circle (formula (2) and formula (3)). The transformation effect is shown.

[0042] θ=2×π×t÷d (1)

[0043] x=r+r×cos(θ) (2)

[0044] y=rr×sin(θ) (3)

[0045] Where t is the traversal value of the circle's circumference in the image [1, 1570]; d is the circumference of the ideal circle, set to 1570; x is the converted Cartesian coordinates; r is the polar radius in polar coordinates; θ is the angle value in polar coordinates; and y is the converted Cartesian coordinates.

[0046] Step S5: Convert the rectangular image from a two-dimensional array to a one-dimensional array, obtaining a one-dimensional array of pointers and tick marks, and then perform mean binarization on the one-dimensional array of pointers and tick marks; the process is as follows:

[0047] Step S51: Determine the position of the pointer and tick marks based on their pixel values. The pointer's pixel value range is set to [70, 80], while the tick marks' pixel range is set to [100, 200], and the background pixel value is set to 0. First, create pointer and tick mark arrays with the same length as the image width. Then, traverse the image and record the column values ​​of pixels that match the pixel value range. Accumulate all pixels in that column that match the range into the corresponding pointer and tick mark arrays. Because the pixel value difference is large, it is easy to determine the pointer and tick marks, thus reducing the dimensionality from a two-dimensional array to a one-dimensional array. The length of the one-dimensional array is 512, and the one-dimensional array is used to represent the pointer and tick marks respectively.

[0048] Step S52: After identifying the pointer and tick marks, obtain a one-dimensional array of the pointer and tick marks. First, calculate the average value of the values ​​in the array. Then, traverse the array and compare whether the values ​​in the array are greater than the average value. If they are greater than the average value, set it to 1; otherwise, set it to 0. Similarly, the length of the binarized pointer array and tick mark array is 512. The binarized array will have all columns of the tick marks and pointer in the image as 1, which is convenient for calculating the position of the pointer relative to the tick marks later.

[0049] Step S6: Calculate the position of the pointer corresponding to the scale line based on the binarized one-dimensional array obtained in S5, and deduce the dial range based on the values ​​of the scale line array; the process is as follows:

[0050] Step S61: Based on the binarized pointer array and tick line array, locate the relative position of the pointer. First, when traversing the binarized array, find the starting position of the tick line and record the index of the starting position, which is the first two consecutive non-zero values ​​in the tick line. Note that it must be the first two consecutive non-zero values ​​because there may be noise, and the pixel position of the line in the image is also represented continuously. Then, traverse the binarized array in turn to obtain the center position of each tick line represented by the array.

[0051] At the same time, the center position of the pointer in the image is calculated using the same method. Unlike the scale lines, which are relatively thin, two or more consecutive pixels can be considered as a scale line. The pointer, on the other hand, is relatively thick, so it can be set to consider four or more consecutive pixels as a pointer, and then the center position of the consecutive pixels is calculated.

[0052] This gives us the positions of the scale lines and the pointer in the image.

[0053] Step S62: First, by traversing the scale array, determine the scales on both sides of the pointer's index. Remember the scale that is smaller than and closest to the pointer's index. Calculate the position of the non-zero value in the pointer array using formula (4) to obtain the relative scale position.

[0054] t=i+(ps)÷(Ss) (4)

[0055] Where i is the i-th tick mark before the pointer, p is the index of the pointer, s is the index of the i-th tick mark, and S is the index of the (i+1)-th tick mark.

[0056] Step S7: First, determine the number of scale lines to select the corresponding range and one scale unit value. Then, combine the given relative position of the pointer to calculate the dial value using formula (5).

[0057] M=t×(l÷s) (5)

[0058] Where M is the dial value, t is the relative position of the pointer in the image, l is the range of the standard dial, and s is the number of scale lines on the standard dial. The range divided by the number of scale lines represents the unit value of one scale division.

[0059] The embodiments described in this specification are merely examples of implementations of the inventive concept and are for illustrative purposes only. The scope of protection of this invention should not be considered limited to the specific forms described in these embodiments; rather, it extends to equivalent technical means conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for assisting in reading industrial tables based on YOLOv5 and U2-Net frameworks, characterized in that, The method includes the following steps: Step S1: Input the image captured by the camera into the trained YOLOV5 model, then detect the position of the dial and obtain the target image of the industrial dial; Step S2: Input the target image of the dial into the trained U2-Net model, then segment the pointer and tick marks in the target image, segment the tick marks and pointer with green and red respectively, and finally output the segmented image with tick marks and pointer; Step S3: Perform image preprocessing on the obtained tick mark and pointer image, including image scaling to a fixed size, image erosion, and image sharpening, to make the tick mark and pointer information easier to distinguish, and output the result. Step S4: Transform the preprocessed image to Cartesian coordinates using polar coordinates, and convert the ring shape into a rectangular image starting from the bottom of the dial ring. Step S5: Convert the rectangular image from a two-dimensional array to a one-dimensional array to obtain a one-dimensional array of pointers and tick marks, and perform mean binarization on the one-dimensional array of pointers and tick marks. Step S6: Calculate the position of the pointer corresponding to the scale line based on the binarized one-dimensional array obtained in S5, and deduce the dial range based on the value of the scale line array; Step S7: Calculate the industrial meter value based on the relative position of the pointer and the measuring range; In step S4, the preprocessed image is transformed from polar coordinates to the corresponding Cartesian coordinate system using formulas (1) and (2), with the dial ring as the center. （1）； （2）； Where x is the converted Cartesian coordinate, r is the polar radius in polar coordinates, θ is the angle value in polar coordinates, and y is the converted Cartesian coordinate; Step S5 includes the following steps: Step S51: Determine the position of the pointer and tick mark based on the difference in pixel values. The pixel value range of the pointer is set to [70, 80], while the pixel range of the tick mark is set to [100, 200]. The background pixel value is set to 0. Because the pixel value difference is large, it is easy to determine the pointer and tick mark. This reduces the dimension of the two-dimensional array to a one-dimensional array. The length of the one-dimensional array is 512. The one-dimensional array is used to represent the pointer and tick mark respectively. Step S52: After identifying the pointer and tick marks, obtain a one-dimensional array of pointers and tick marks. First, calculate the average value of the values ​​in the array, and then iterate through the array to compare whether the values ​​in the array are greater than the average value. If they are greater than the average value, set it to 1; otherwise, set it to 0. Similarly, the length of the binarized pointer array and tick mark array is 512.

2. The method for assisting in reading industrial tables based on YOLOv5 and U2-Net framework according to claim 1, characterized in that, Step S1 includes the following steps: Step S11: Construct training and test sets: First, select an appropriate number of datasets, requiring different dials and different ranges to be included in the dataset, and select 80% of the data as the dataset; Step S12: Training set image data annotation: Use LabelImg to annotate the images in the training set and select the positions of the industrial dials in the images; Step S13: Multi-round training of the model: Set the number of training rounds to 10 and the number of iterations to 4; Step S14: Verify the training effect using the test set: Verify the test set data.

3. The method for assisting in reading industrial tables based on YOLOv5 and U2-Net framework according to claim 1 or 2, characterized in that, In step S2, U2-Net is used to segment the pointer and tick marks. Similarly, LabelImg is used to label the images in the training set, marking the positions of the tick marks and pointer. Finally, the images of the segmented tick marks and pointer are used, with a black background. The tick marks are segmented in green, and the pointer is segmented in red, so that the pointer and tick marks can be distinguished based on pixel values ​​later.

4. The method for assisting in reading industrial tables based on YOLOv5 and U2-Net framework according to claim 1 or 2, characterized in that, The process of step S3 is as follows: the segmented scale line and pointer image are set to a fixed size [512,512], and then image erosion is performed. The convolution kernel size of the image erosion is set to 1. This preprocessing facilitates subsequent reading operations.

5. The method for assisting in reading industrial tables based on YOLOv5 and U2-Net framework according to claim 1 or 2, characterized in that, In step S6, the relative position of the pointer is located based on the binarized pointer array and scale line array. First, the starting position of the scale line is found, which is the first two consecutive non-zero values ​​in the scale line. The pixel positions of the line in the image are also continuously represented. Then, the position of the non-zero value in the pointer array is calculated to obtain the position of the relative scale line. At the same time, the range of the dial is determined based on the number of scale lines. The range needs to be set in advance.

6. The method for assisting in reading industrial tables based on YOLOv5 and U2-Net framework according to claim 1 or 2, characterized in that, In step S7, the dial value is calculated using formula (3) based on the range information and the relative position of the pointer. （3）； Where M is the dial value. t It is the relative position of the pointer in the image. l This is the standard dial range. s It is the number of scale lines on a standard dial. The range divided by the number of scale lines represents the unit value of one scale.

Images