Automatic recognition, positioning and reading method of digital instruments based on deep learning

Abstract

The method based on deep learning disclosed in the present invention realizes the recognition, positioning and reading method of digital instruments. First, images with digital instruments are taken, and a total of 1,000 images are collected to form a data set; then the data set is annotated using Labelimg, and then divided into a training set, a test set, and a verification set in a ratio of 6:2:2; the annotated image is sent to the YOLOv3 network model after training and tuning, and then the digital instrument image is intercepted and input into the improved CenterNet network to obtain the digital display area coordinates and tilt angle, and then the image is tilt-corrected according to the tilt angle and the digital display area image is intercepted, and finally the digital display area image is directly input into the CRNN network to obtain the final reading recognition result. Compared with the traditional algorithm, the present invention reduces the influence of instrument tilt, environmental changes, etc. on reading recognition, and has a wider range of application and better robustness.

Description

Technical Field

[0001] The present invention belongs to the technical field of automatic reading of digital instruments, and specifically relates to a method for automatic identification, positioning and reading of digital instruments based on deep learning. Background Art

[0002] Industrial instruments can be divided into digital instruments and pointer instruments according to the display mode. Digital instruments mainly display the readings through various carriers such as liquid crystals and digital tubes. They have the advantages of simple observation and convenient maintenance, and have been increasingly used in industrial production. However, most of the digital instruments for industrial production do not have direct reading interfaces due to information security and cost issues, and cannot realize the automatic collection and reading of the instrument readings. Therefore, the readings of digital instruments in early industrial sites mainly rely on operators to read and record. However, this manual interpretation method often has problems such as low efficiency and susceptibility to subjective factors. At present, with the development of computer technology, the automatic reading recognition algorithm of instruments based on image recognition algorithms has gradually developed. For example, in the Chinese patent CN109255336A "Lightning arrester recognition method based on patrol robot", a two-way projection method combined with an artificial neural network is used for digital segmentation and recognition. However, these traditional image processing methods have high requirements for the captured images. If the image has problems such as large illumination changes, complex background, a wide variety of instruments, and instrument tilt, it is easy to have low recognition accuracy in the later processing. Summary of the invention

[0003] The purpose of the present invention is to provide a method for automatic identification, positioning and reading of digital instruments based on deep learning, which solves the impact of problems such as instrument tilt, environmental changes and a wide variety of instruments on the reading and recognition of digital instruments.

[0004] The technical solution adopted by the present invention is a method for automatic identification, positioning and reading of digital instruments based on deep learning, which is specifically implemented in the following steps:

[0005] Step 1: Take an image with a digital instrument;

[0006] Step 2: Use two types of digital temperature meters and data set expansion methods to collect 1000 digital instrument images and create an instrument data set;

[0007] Step 3: Label the dataset in step 2 using Labelimg labeling software, and then divide the dataset into training set, test set, and validation set in a ratio of 6:2:2;

[0008] Step 4: Train and tune the YOLOv3 network model, and input the image annotated in step 3 into the YOLOv3 network model. The network output is the location information and classification information of the digital instrument in the image.

[0009] Step 5: Train and tune the CenterNet network model, input the location information of the digital instrument output in step 4 into the improved CenterNet network model, and the network output is the location information and tilt angle of the digital instrument display area;

[0010] Step 6, performing a tilt transformation on the display area according to the tilt angle of the digital instrument output in step 5 to obtain a corrected display area image;

[0011] Step 7: Train and tune the CRNN network model, input the display area image corrected in step 6 into the CRNN network model to obtain the final digital meter reading.

[0012] The present invention is also characterized in that

[0013] The two types of digital temperature meters in step 2 are a 0-200°C digital temperature meter and a 0-500°C digital temperature meter, and the data set expansion means are random adjustment of contrast and brightness and random exchange of channels.

[0014] In step 4, the training and tuning is to first use the K-means algorithm to perform cluster analysis on the instrument data set to obtain the anchor size that best matches the instrument data set;

[0015] During training, the network input size is set to 618×618×3, the number of iterations is 50200, the initial learning rate is 0.005, the learning rate decays by 10 times every 10000 times, the optimization algorithm is set to Mini-BGD algorithm, the momentum value is set to 0.9, the decay coefficient is 0.0005, the batch size is set to 64, and the forward propagation is completed in 8 times;

[0016] By observing the loss value change curve of the validation set, when it shows a downward trend and tends to be stable after a certain number of iterations, the model is considered to have converged. If it does not converge, increase or decrease the learning rate by a multiple of 10 and retrain.

[0017] In step 5, the training and tuning is to divide the data set in step 3, set the number of iterations to 150, the initial learning rate to 0.000125, reduce the learning rate by 10 times in the 90th and 120th rounds respectively, the batch size is 8, and the optimization algorithm is set to the Adam algorithm. By observing the loss value change curve of the validation set, when it shows a downward trend and tends to be stable after a certain number of iterations, the model is considered to have converged. If it does not converge, increase or decrease the learning rate by multiples of 10 and retrain.

[0018] In step 5, the improved CenterNet network model is based on the original CenterNet network structure, with the network input size modified to 256×256×3, the backbone extraction network selected to be ResNet-50, and the last layer of downsampling operation in ResNet-50 replaced with a dilated convolution, and an angle regression branch added to the head part;

[0019] Specifically, it consists of two convolutional layers and one ReLU activation layer. The composition of each layer is as follows: the convolution kernel size of the first convolutional layer is 64×3×3, and the feature map output is 64×64×64. The second layer is the ReLU activation layer. The third layer is the convolution layer, i.e., the output layer. The convolution kernel size is 1×1×1, and the feature map output is 1×64×64. Since an angle regression branch is added, the corresponding loss function of the rotation angle regression is added. The minimum absolute value is used as the loss function, i.e., the L1 loss function. The specific formula of the improved CenterNet loss function is as follows:

[0020]

[0021] Among them, L CenterNet is the loss function of CenterNet network, λ Angle is the weight of the angle regression branch loss, which is 0.5 in the network. is the angle prediction value, y n The actual rotation angle value.

[0022] In step 6, the tilt transformation is based on the center point (x0, y0) as the origin, and the rotation matrix is ​​expressed as

[0023]

[0024] Where θ is the tilt angle obtained from the output of step 5.

[0025] The training and optimization in step 7 is to first crawl the images with digital display instruments and digital display temperature meters as keywords from the network pictures through the web crawler, filter and crop the images to obtain images containing only the digital parts, and then crop the images collected in this experiment to obtain the data set;

[0026] The training was divided into two stages. In the first stage, only the processed network images were used for training. The training parameters were set as follows: the image input size was 32×100×3, the batch size was 128, the number of iterations was set to 72, the initial learning rate was 0.005, and the optimization algorithm was the Adam algorithm.

[0027] In the second stage, the data set is divided into training set, validation set and test set in a ratio of 8:1:1. The best model obtained in the first stage is used as the pre-training model, and the training set is used to train the model. The training parameters of the second stage are set as follows: the image input size is 32×100×3, the batch size is 128, the number of iterations is set to 144, the initial learning rate is 0.0005, the learning rate decay coefficient is 0.00004, the optimization algorithm is the Adam algorithm, the first-order moment estimation and second-order moment estimation decay rates are 0.9 and 0.999, and L2 regularization is set;

[0028] By observing the loss value change curve of the validation set, when it shows a downward trend and tends to be stable after a certain number of iterations, the model is considered to have converged. If it does not converge, increase or decrease the learning rate by a multiple of 10 and retrain.

[0029] The beneficial effects of the present invention are:

[0030] (1) The present invention uses a deep learning-based method to realize the recognition, positioning and reading of digital instruments, and introduces an angle prediction branch based on the CenterNet network so that the model can directly regress the tilt angle without using Hough line detection and other methods to calculate the tilt angle separately.

[0031] (2) Compared with traditional algorithms, the present invention reduces the impact of instrument tilt, environmental changes, etc. on readings, and has a wider range of applications and better robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of an experimental scenario of a method for automatic identification, positioning and reading of digital instruments based on deep learning of the present invention;

[0033] Figure 2 It is a flow chart of a method for automatic identification, positioning and reading of digital instruments based on deep learning of the present invention;

[0034] Figure 3 It is a flow chart of digital instrument rough positioning and classification of the digital instrument automatic identification, positioning and reading method based on deep learning of the present invention;

[0035] Figure 4 This is a diagram of the rough positioning and classification results of digital instruments according to the method for automatic identification, positioning and reading of digital instruments based on deep learning of the present invention;

[0036] Figure 5 It is a flow chart of digital display area positioning and tilt angle detection of the method for automatic identification, positioning and reading of digital instruments based on deep learning of the present invention;

[0037] Figure 6 It is a digital display area positioning result diagram of the digital instrument automatic recognition, positioning and reading method based on deep learning of the present invention;

[0038] Figure 7 It is a digital display area recognition flow chart of the method for automatic recognition, positioning and reading of digital instruments based on deep learning of the present invention;

[0039] Figure 8 This is a diagram of digital instrument recognition results based on the deep learning-based digital instrument automatic recognition, positioning and reading method of the present invention.

[0040] In the picture, 1. Camera, 2. Inspection robot, 3. Digital instrument, 4. Wall, 5. Server. DETAILED DESCRIPTION

[0041] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] Schematic diagram of the experimental scenario of the method for automatic identification, positioning and reading of digital instruments based on deep learning of the present invention, as shown in Figure 1 As shown, the camera 1 is connected to the inspection robot 2 and transmits the captured image stream in real time. The inspection robot 2 will travel according to the pre-planned path. After arriving at the specified position, the camera 1 will be adjusted to capture the original image of the wall 4 containing the instrument 3, and then the image will be transmitted to the server 5, which will automatically identify the readings of the LCD digital instrument.

[0043] The flowchart of the method for automatic identification, positioning and reading of digital instruments based on deep learning of the present invention is as follows: Figure 2 As shown, the specific implementation steps are as follows:

[0044] Step 1: The inspection robot 2 moves to a designated position, adjusts the camera 1, and captures an image with a digital instrument 3.

[0045] Step 2: Process Figure 3 As shown in the figure, a 0-200℃ digital temperature meter and a 0-500℃ digital temperature meter were used to collect 1000 digital instrument images and produce the instrument dataset by using the dataset expansion method of random adjustment of contrast and brightness and random exchange of channels.

[0046] Step 3: Use Labelimg annotation software to annotate the dataset in step 2, and then divide the dataset into training set, test set, and validation set in a ratio of 6:2:2.

[0047] Step 4: Train and tune the YOLOv3 network model. First, use the K-means algorithm to perform cluster analysis on the instrument data set to obtain the anchor size that best matches the instrument data set.

[0048] During training, the network input size is set to 618×618×3, the number of iterations is 50200, the initial learning rate is 0.005, the learning rate decays by 10 times every 10000 times, the optimization algorithm is set to Mini-BGD algorithm, the momentum value is set to 0.9, the decay coefficient is 0.0005, the batch size is set to 64, and the forward propagation is completed in 8 times;

[0049] By observing the loss value change curve of the validation set, when it shows a downward trend and stabilizes after a certain number of iterations, the model is considered to have converged. If it does not converge, increase or decrease the learning rate by a multiple of 10 and retrain. After the training is completed, calculate the mAP value and average forward reasoning time of the test set. The higher the mAP value and the shorter the average forward reasoning time, the better the model performance.

[0050] Then the image annotated in step 3 is fed into the YOLOv3 network model, and the network output is the location information and classification information of the digital instrument in the image, such as Figure 4 As shown, finally the instrument image is captured according to the position information.

[0051] Step 5: Process Figure 5 As shown in the figure, the CenterNet network model is trained and tuned. The data set divided in step 3 is set to 150 iterations, the initial learning rate is 0.000125, the learning rate is reduced by 10 times in the 90th and 120th rounds respectively, the batch size is 8, and the optimization algorithm is set to the Adam algorithm. By observing the loss value change curve of the validation set, when it shows a downward trend and tends to be stable after a certain number of iterations, the model is considered to have converged. If it does not converge, the learning rate is increased or decreased by multiples of 10 and retrained;

[0052] Then, the position information of the digital instrument output in step 4 is input into the improved CenterNet network model. That is, based on the original CenterNet network structure, the network input size is modified to 256×256×3, the backbone extraction network selects ResNet-50, and the last layer of downsampling operation in ResNet-50 is replaced by dilated convolution, and an angle regression branch is added to the head part;

[0053] Specifically, it consists of two convolutional layers and one ReLU activation layer. The composition of each layer is as follows: the convolution kernel size of the first convolutional layer is 64×3×3, and the feature map output is 64×64×64. The second layer is the ReLU layer. The third layer is the convolutional layer, i.e., the output layer. The convolution kernel size is 1×1×1, and the feature map output is 1×64×64. Due to the addition of an angle regression branch, the corresponding loss function of the rotation angle regression is added. The minimum absolute value is used as the loss function, i.e., the L1 loss function. The specific formula of the improved CenterNet loss function is as follows:

[0054]

[0055] Among them, L CenterNet is the loss function of CenterNet network, λ Angle is the weight of the angle regression branch loss, which is 0.5 in the network. is the angle prediction value, y n is the actual rotation angle value;

[0056] The network output is a digital instrument displaying the area location information and tilt angle, such as Figure 6 shown.

[0057] Step 6: Perform a tilt transformation on the display area according to the tilt angle predicted in step 5. The center point (x0, y0) is used as the origin during the tilt transformation. The rotation matrix is ​​expressed as

[0058]

[0059] Where θ is the tilt angle predicted in step 5;

[0060] Thus, a corrected display area image is obtained.

[0061] Step 7: Process Figure 7 As shown, the CRNN network model is trained and tuned, and images with digital display instruments and digital display temperature meters as keywords are crawled from the network through a web crawler. The images are screened and cropped to obtain images containing only the digital part. The images collected in this experiment are cropped to obtain a data set.

[0062] The training was divided into two stages. In the first stage, only the processed network images were used for training. The training parameters were set as follows: the image input size was 32×100×3, the batch size was 128, the number of iterations was set to 72, the initial learning rate was 0.005, and the optimization algorithm was the Adam algorithm.

[0063] In the second stage, the experimental data set was divided into training set, validation set, and test set in a ratio of 8:1:1. The best model obtained after the first stage of training was used as the pre-training model, and the training set was used to fine-tune the model. The training parameters of the second stage were set as follows: the image input size was 32×100×3, the batch size was 128, the number of iterations was set to 144, the initial learning rate was 0.0005, the learning rate decay coefficient was 0.00004, the optimization algorithm was the Adam algorithm, the first-order moment estimation and second-order moment estimation decay rates were 0.9 and 0.999, and L2 regularization was set;

[0064] By observing the loss value change curve of the validation set, when it shows a downward trend and tends to be stable after a certain number of iterations, the model is considered to have converged. If it does not converge, increase or decrease the learning rate by a multiple of 10 and retrain. After training, calculate the accuracy of the test set and the average forward reasoning time. The higher the accuracy and the shorter the average forward reasoning time, the better the model performance.

[0065] Then the corrected display area image in step 6 is input into the CRNN network model to obtain the final digital meter reading. The final result is as follows: Figure 8 shown.

[0066] The present invention introduces an angle prediction branch based on the CenterNet network, so that the model can directly regress the tilt angle without using Hough line detection and other methods to calculate the tilt angle. Compared with traditional algorithms, the present invention reduces the impact of instrument tilt, environmental changes and other conditions on reading recognition, and has a wider range of applications and better robustness.

Claims

1. A method for automatic identification, positioning and reading of digital instruments based on deep learning, characterized in that: Please follow the steps below to implement: Step 1, capture an image with a digital instrument (3); Step 2: Use two types of digital temperature meters and data set expansion methods to collect 1000 digital instrument images and create an instrument data set; Step 3: Label the dataset in step 2 using Labelimg labeling software, and then divide the dataset into training set, test set, and validation set in a ratio of 6:2:2; Step 4: Train and tune the YOLOv3 network model, and input the image annotated in step 3 into the YOLOv3 network model. The network output is the location information and classification information of the digital instrument in the image. Step 5: Train and tune the CenterNet network model, input the location information of the digital instrument output in step 4 into the improved CenterNet network model, and the network output is the location information and tilt angle of the digital instrument display area; Step 6, performing a tilt transformation on the display area according to the tilt angle of the digital instrument output in step 5 to obtain a corrected display area image; Step 7: Train and tune the CRNN network model, input the corrected display area image in step 6 into the CRNN network model, and obtain the final digital meter reading; The improved CenterNet network model in step 5 is based on the original CenterNet network structure, with the network input size modified to 256×256×3, the backbone extraction network selected to be ResNet-50, and the last layer of downsampling operation in ResNet-50 replaced by a dilated convolution, and an angle regression branch added to the head part; Specifically, it consists of two convolutional layers and one ReLU activation layer. The composition of each layer is as follows: the convolution kernel size of the first convolutional layer is 64×3×3, and the feature map output is 64×64×64. The second layer is the ReLU activation layer. The third layer is the convolution layer, i.e., the output layer. The convolution kernel size is 1×1×1, and the feature map output is 1×64×64. Since an angle regression branch is added, the corresponding loss function of the rotation angle regression is added. The minimum absolute value is used as the loss function, i.e., the L1 loss function. The specific formula of the improved CenterNet loss function is as follows: in, is the loss function of the CenterNet network, is the weight of the angle regression branch loss, which is 0.5 in the network. is the predicted angle value, is the actual rotation angle value; The tilt transformation described in step 6 is based on the center point is the origin, the rotation matrix is ​​expressed as in The tilt angle obtained by step 5 output; The training and optimization in step 7 is to first crawl the images with digital display instruments and digital display temperature meters as keywords from the network pictures through a web crawler, screen and crop the images to obtain images containing only the digital parts, and then crop the images collected in this experiment to obtain a data set; The training was divided into two stages. In the first stage, only the processed network images were used for training. The training parameters were set as follows: the image input size was 32×100×3, the batch size was 128, the number of iterations was set to 72, the initial learning rate was 0.005, and the optimization algorithm was the Adam algorithm. In the second stage, the data set is divided into training set, validation set and test set in a ratio of 8:1:

1. The best model obtained in the first stage is used as the pre-training model, and the training set is used to train the model. The training parameters of the second stage are set as follows: the image input size is 32×100×3, the batch size is 128, the number of iterations is set to 144, the initial learning rate is 0.0005, the learning rate decay coefficient is 0.00004, the optimization algorithm is the Adam algorithm, the first-order moment estimation and second-order moment estimation decay rates are 0.9 and 0.999, and L2 regularization is set; By observing the loss value change curve of the validation set, when it shows a downward trend and tends to be stable after a certain number of iterations, the model is considered to have converged. If it does not converge, increase or decrease the learning rate by a multiple of 10 and retrain.

2. The method for automatic identification, positioning and reading of digital instruments based on deep learning according to claim 1 is characterized in that: The two types of digital display temperature meters in step 2 are a 0-200° C. digital display temperature meter and a 0-500° C. digital display temperature meter, and the data set expansion means are random adjustment of contrast and brightness and random exchange of channels.

3. The method for automatic identification, positioning and reading of digital instruments based on deep learning according to claim 1 is characterized in that: The training and tuning in step 4 is to first use the K-means algorithm to perform cluster analysis on the instrument data set to obtain the anchor size that best matches the instrument data set; During training, the network input size is set to 618×618×3, the number of iterations is 50200, the initial learning rate is 0.005, the learning rate decays by 10 times every 10000 times, the optimization algorithm is set to Mini-BGD algorithm, the momentum value is set to 0.9, the decay coefficient is 0.0005, the batch size is set to 64, and the forward propagation is completed in 8 times; By observing the loss value change curve of the validation set, when it shows a downward trend and tends to be stable after a certain number of iterations, the model is considered to have converged. If it does not converge, increase or decrease the learning rate by a multiple of 10 and retrain.

4. The method for automatic identification, positioning and reading of digital instruments based on deep learning according to claim 1 is characterized in that: The training and tuning in step 5 is to divide the data set in step 3, set the number of iterations to 150, the initial learning rate to 0.000125, reduce the learning rate by 10 times in the 90th and 120th rounds respectively, the batch size is 8, and the optimization algorithm is set to the Adam algorithm. By observing the loss value change curve of the validation set, when it shows a downward trend and tends to be stable after a certain number of iterations, the model is considered to have converged. If it does not converge, increase or decrease the learning rate by a multiple of 10 and retrain.

Images