Electricity meter image target recognition method based on deep learning

Abstract

The present invention discloses a method for target recognition of electric energy meter images based on deep learning, which solves the problems existing in the prior art that the detection accuracy is greatly affected by the image shooting conditions, the deep learning method has a large amount of computation and is difficult to deploy. The solution is as follows: three training sets are generated using electric energy meter images, a yolox‑nano network is selected as the training network, and a region of interest detection network, a dial information recognition network, and a barcode information recognition network are trained respectively; the image to be detected is input into the region of interest detection network to obtain the dial and barcode images, which are respectively input into the corresponding information recognition network; and the recognition results are calculated according to the corresponding formula. The present invention can accurately identify the reading of the electric energy meter and its barcode information, so as to detect anomalies in a timely manner and take corresponding measures. The computational complexity is small, which makes it easy to deploy on a mobile terminal and has stronger practicality.

Description

Technical Field

[0001] This invention belongs to the field of physics technology and further relates to the field of image processing technology, a method for identifying objects in electricity meter images based on deep learning. This invention can be used to accurately identify the meter readings and barcode information in electricity meter images, enabling timely detection of anomalies and the implementation of appropriate countermeasures. Background Art

[0002] Electricity meter data is crucial user information in power systems. However, in reality, power workers typically obtain this data manually, which not only requires significant manpower and resources but is also prone to errors. Automating the recognition of electricity meter information using image processing technology can save manpower and time while reducing the risk of reading errors, significantly impacting the intelligent development of power systems. Commonly used methods for identifying electricity meter information can be categorized into two types: those based on traditional image processing techniques and those based on deep learning.

[0003] The Guangxi Electric Power Research Institute of Power Grid Co., Ltd., in its patent application, "A Method for Intelligently Reading Electricity Meter Data" (Application No. 201910987804.8, Application Date: October 17, 2019, Publication No. CN110781884A), discloses a method for intelligently reading electricity meter data. The method involves the following steps: first, performing HSV spatial transformation and edge detection on the electricity meter image to obtain an edge image; then, performing morphological processing and connected domain extraction on the edge image. False positives are filtered out based on the aspect ratio of the connected domain to obtain the data region in the electricity meter image; then, using Radon transform to tilt-correct the data region, and using horizontal and vertical projection to identify the data, obtaining the recognition result. While this method can automatically recognize electricity meter data, it still has the disadvantage of requiring high image capture conditions. Conditions such as backlighting and uneven illumination may reduce recognition accuracy.

[0004] Jiangsu Electric Power Information Technology Co., Ltd. disclosed a method for intelligent reading of electric energy meter data in its patent application document "A method for identifying electric energy meter numbers based on YOLOV3 network" (application number: 202010417060.9, application date: 2020.05.18, application publication number: CN111461121A). The implementation steps of this method are as follows: first, use the YOLOv3-tiny network to build a number area positioning network, locate the number, and then crop the target area of ​​the number; finally, build a number recognition network based on the YOLOv3 network, perform number recognition, and obtain the recognition result. This method uses target detection technology to recognize the numbers in the electric energy meter image, with high accuracy and strong stability. However, the method still has the disadvantage that the network used is heavy and not convenient to be deployed on the mobile terminal, which limits its practical application. Summary of the Invention

[0005] The purpose of the present invention is to address the shortcomings of the above-mentioned existing technologies and propose a method for identifying electric energy meter information based on deep learning, which is used to solve the problems that the recognition accuracy is greatly affected by the image shooting conditions, and the YOLOv3 network is too large and difficult to deploy.

[0006] The purpose of the present invention is to improve the accuracy and stability of electricity meter information recognition based on the Yolox-nano network, thereby resolving the problem in the prior art of reduced recognition accuracy due to backlighting and uneven illumination. The present invention also reduces the computational complexity by using depthwise separable convolution to construct the feature extraction module. This solves the problem in the prior art of excessive computational complexity, which makes it inconvenient to deploy on mobile devices and thus limits its practical application.

[0007] To achieve the above object, the present invention specifically implements the steps as follows:

[0008] Step 1: Generate three training sets:

[0009] Step 1.1: Use a camera to capture at least 1,000 electricity meter dials to obtain an image of each meter. Use a rectangular frame to mark the dial and barcode area in each image as the region of interest. All regions of interest form a training set 1.

[0010] Step 1.2: intercept the dial area of ​​each image in training set 1 and obtain the corresponding dial image. Use a rectangular frame to mark each number in each image as the marked dial area. All the marked dial areas are used as training set 2.

[0011] Step 1.3: intercept the barcode region of each image in training set 1 to obtain the corresponding barcode image. For each image, mark each number in the image with a rectangular frame as the marked barcode region. All the marked barcode regions are used as training set 3.

[0012] Step 2: Using the target detection network yolox-nano in the prior art as the recognition network;

[0013] Step 3: Train the recognition network:

[0014] Training set 1, training set 2, and training set 3 are respectively input into the recognition network, and the loss function values ​​of the networks are respectively calculated. The networks are iteratively trained using the gradient descent method, and the parameters of the networks are iteratively updated until the loss function converges. The trained region of interest detection network, dial information recognition network, and barcode information recognition network are respectively obtained;

[0015] Step 4: Output images of different regions through the region of interest detection network:

[0016] The energy meter image to be identified is input into the trained region of interest detection network, which outputs the category of the region of interest in the energy meter image and the coordinates of the upper left corner of the rectangular box containing the region; the dial area image and the barcode area image are cropped from the energy meter image based on the output coordinates;

[0017] Step 5: Get the information of each number through the dial information recognition network and the barcode information recognition network:

[0018] Step 5.1: Input the dial area image into the trained dial information recognition network, and output the category of each number in the dial area and the coordinates of the upper left corner of the rectangular box containing the number;

[0019] Step 5.2: Input the barcode area image into the trained barcode information recognition network, and output the category of each number in the barcode area and the coordinates of the upper left corner of the rectangular box where the number is located;

[0020] Step 6: Output the information recognition results of the dial and barcode:

[0021] Step 6.1: Arrange each rectangular box in the dial area from small to large according to the size of the horizontal coordinate of the upper left corner point, read the category of the number contained in each rectangular box in turn, calculate the dial reading using the dial reading calculation formula, and output the recognition result;

[0022] Step 6.2: Arrange each rectangular box in the barcode area from small to large according to the size of the horizontal coordinate of the upper left corner point, read the category of the number contained in each rectangular box in turn, calculate the barcode reading according to the barcode reading calculation formula, and output the recognition result.

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] First, since the present invention performs electricity meter information recognition based on the yolox-nano network, it overcomes the problem in the prior art that the detection accuracy is greatly affected by the image shooting conditions, making the target recognition accuracy of the present invention higher and more stable.

[0025] Second, since the convolution of the feature extraction module in the recognition network of the present invention is a depth-separable convolution, it overcomes the problems of large number of parameters and computational complexity and difficult deployment in the existing technology, making the present invention use fewer parameters and computational complexity and more convenient to deploy on the mobile terminal. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is an implementation flow chart of the present invention;

[0027] Figure 2 It is a simulation experiment diagram of the present invention. DETAILED DESCRIPTION

[0028] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] Reference Figure 1 , the implementation steps of the embodiment of the present invention are further described in detail.

[0030] Step 1: Generate three training sets.

[0031] An embodiment of the present invention uses a camera to shoot 1,338 electricity meters to obtain 1,338 different electricity meter images corresponding to each electricity meter. The dial and barcode area in each image are marked with a rectangular frame as the region of interest, and all 1,338 regions of interest form a training set 1.

[0032] An embodiment of the present invention is to intercept the dial area of ​​each image in training set 1 to obtain 1338 corresponding different dial images, mark each number in each image with a rectangular frame as the marked dial area, and use all 1338 marked dial areas as training set 2.

[0033] In an embodiment of the present invention, the barcode region of each image in the training set 1 is intercepted to obtain 1338 corresponding different barcode images, each number in each image is marked with a rectangular frame as the marked barcode region, and all 1338 marked barcode regions are used as the training set 3;

[0034] Step 2: Select Identify Network.

[0035] An embodiment of the present invention uses the target detection network yolox-nano in the prior art as a recognition network. The yolox-nano network is composed of a feature extraction module Backbone, a feature fusion module Neck, and a detection module Head connected in series.

[0036] The feature extraction module Backbone includes a Focus module, a depthwise separable convolution module, a residual convolution module and a spatial pyramid pooling module. The Focus module consists of a sampling layer and a Concat layer. Among them, the depthwise separable convolution module includes a group convolution layer GConv with a convolution kernel size of 3*3, a batch normalization layer BN, a nonlinear activation function SiLu, and a point convolution layer with a convolution kernel size of 1*1 in series; the residual convolution block includes a convolution layer Conv with a convolution kernel size of 3*3, a batch normalization layer BN, a nonlinear activation function SiLu and a jump connection layer in series; the spatial pyramid pooling module includes a convolution layer Conv with a convolution kernel size of 3*3, a batch normalization layer BN, a nonlinear activation function SiLu, three parallel pooling layers with pooling kernels of 5*5, 9*9 and 13*13 respectively, and a jump connection layer in series;

[0037] The feature fusion module Neck is composed of a top-down feature pyramid FPN and a bottom-up feature pyramid PAN. FPN is composed of a convolution layer Conv with a convolution kernel size of 3*3, a batch normalization layer BN, a nonlinear activation function SiLu, and an upsampling layer connected in series. PAN is composed of a convolution layer Conv with a convolution kernel size of 3*3, a batch normalization layer BN, a nonlinear activation function SiLu, and a downsampling layer connected in series.

[0038] The detection module Head consists of three parallel detection heads, each of which is composed of a convolution layer with a convolution kernel size of 3*3, a Sigmoid layer, a Concat layer and a reshape layer connected in series.

[0039] Step 3: Train the recognition network to obtain the trained region of interest detection network, dial information recognition network, and barcode information recognition network.

[0040] Training set 1, training set 2 and training set 3 are input into the training network respectively, and the training network is trained using the same training method to obtain the trained region of interest detection network, dial information recognition network and barcode information recognition network respectively.

[0041] The steps of the training method are as follows:

[0042] In the first step, all parameters of the training network are set to random numbers that conform to the normal distribution to obtain an initialized network. The mean of the normal distribution is 0, the variance is 0.02, and the learning rate of the network is set to 0.005;

[0043] In the second step, 8 images are randomly selected from the training set and input into the initialized network. The network parameters are iteratively updated using the gradient descent method until the loss function converges to obtain a trained network.

[0044] The loss function is as follows:

[0045]

[0046] Where L represents the average loss value of the selected samples, Σ represents the summation operation, k represents the index of the selected sample image, k = 1, 2, 3, 4, 5, 6, 7, 8, U represents the expected output of each sample image, U' represents the actual output of each sample image, ∩ represents the intersection operation, and ∪ represents the union operation.

[0047] Step 4: Output images of different regions through the region of interest detection network.

[0048] The electric energy meter image to be identified is input into the trained region of interest detection network to obtain the category of the region of interest in the electric energy meter image and the coordinates of the upper left corner of the rectangular box where the region is located. The dial area image and the barcode area image are cropped from the electric energy meter image according to the output coordinates.

[0049] Step 5: Obtain the information of each number through the dial information recognition network and the barcode information recognition network.

[0050] Step 5.1: Input the dial area image into the trained dial information recognition network to obtain the category of each number in the dial area and the coordinates of the upper left corner of the rectangular box where the number is located.

[0051] In step 5.2, the barcode area image is input into the trained barcode information recognition network to obtain the category of each number in the barcode area and the coordinates of the upper left corner of the rectangular box where the number is located.

[0052] Step 6: Output the information recognition results of the dial and barcode.

[0053] Step 6.1: Calculate the dial information recognition result. Arrange each rectangular box in the dial area from small to large according to the horizontal coordinate of the upper left corner point. Read the category of the number contained in each rectangular box in turn. Calculate the dial reading according to the dial reading calculation formula and output the recognition result.

[0054] The calculation formula of the dial indication is as follows:

[0055]

[0056] Among them, r represents the dial reading, Σ represents the summation operation, C 2i It represents the category of the number contained in the i-th rectangular box read from large to small in the dial image, and m represents the number of rectangular boxes in the dial image.

[0057] Step 6.2: Calculate the barcode information recognition result. Arrange each rectangular box in the barcode area from small to large according to the horizontal coordinate of the upper left corner point. Read the category of the number contained in each rectangular box in turn. Calculate the barcode reading according to the barcode reading calculation formula and output the recognition result.

[0058] The barcode reading calculation formula is as follows:

[0059]

[0060] Among them, r2 represents the barcode number, Σ represents the sum operation, C 3j It represents the category of the number contained in the j-th rectangular box read from the largest to the smallest in the barcode image, and n represents the number of rectangular boxes in the barcode image.

[0061] The effects of the present invention are further described below in conjunction with simulation experiments:

[0062] 1. Simulation experiment conditions:

[0063] The hardware platform of the simulation experiment of the present invention is: the processor is Intel i7-10870H CPU, the main frequency is 2.21 GHz, and the memory is 16 GB.

[0064] The software platform for the simulation experiment of the present invention is: Windows 10 operating system and Python 3.7.

[0065] The input images used in the simulation experiment of the present invention are four images of different electric energy meters, all of which are actually taken, and the resolution of each image is 864*1920. Figure 2 (a) Figure 2 (b) Figure 2 (c) Figure 2 (d) shown.

[0066] 2. Simulation content and result analysis:

[0067] The simulation experiment of the present invention is to use the method of the present invention to Figure 2 Four different electricity meter images are simulated respectively to obtain the dial information recognition results and barcode information recognition results. The dial information recognition results and barcode information recognition results are compared with the corresponding actual electricity meter information to see whether they are consistent. The results are shown in Table 1:

[0068] Table 1 Comparison between the electric energy meter information recognition results and the actual electric energy meter information

[0069]

[0070]

[0071] In Table 1, "T" in "Whether the dial simulation results are consistent with the actual values" and "Whether the barcode simulation results are consistent with the actual values" represents that the measurement results are consistent with the actual values, and "F" represents that the measurement results are inconsistent with the actual values.

[0072] Combining the two rows of “Whether the dial simulation result is consistent with the actual value” and “Whether the barcode simulation result is consistent with the actual value” given in Table 1, it can be seen that the present invention can accurately identify the information of the dial and barcode area of ​​the electricity meter.

Claims

1. A method for electric energy meter image target recognition based on deep learning, characterized in that: The Yolox-nano network was trained using three training sets containing electricity meter information to obtain the trained region of interest detection network, dial information recognition network, and barcode information recognition network. The specific steps of the recognition method include the following: Step 1: Generate three training sets: Step 1.1: Use a camera to capture at least 1,000 electricity meter dials to obtain an image of each meter. Use a rectangular frame to mark the dial and barcode area in each image as the region of interest. All regions of interest form a training set 1. Step 1.2: intercept the dial area of ​​each image in training set 1 and obtain the corresponding dial image. Use a rectangular frame to mark each number in each image as the marked dial area. All the marked dial areas are used as training set 2. Step 1.3: intercept the barcode region of each image in training set 1 to obtain the corresponding barcode image. For each image, mark each number in the image with a rectangular frame as the marked barcode region. All the marked barcode regions are used as training set 3. Step 2: Using the target detection network yolox-nano in the prior art as the recognition network; Step 3: Train the recognition network: Training set 1, training set 2, and training set 3 are respectively input into the recognition network, and the loss function values ​​of the networks are respectively calculated. The networks are iteratively trained using the gradient descent method, and the parameters of the networks are iteratively updated until the loss function converges. The trained region of interest detection network, dial information recognition network, and barcode information recognition network are respectively obtained; Step 4: Output images of different regions through the region of interest detection network: The energy meter image to be identified is input into the trained region of interest detection network, which outputs the category of the region of interest in the energy meter image and the coordinates of the upper left corner of the rectangular box containing the region; the dial area image and the barcode area image are cropped from the energy meter image based on the output coordinates; Step 5: Get the information of each number through the dial information recognition network and the barcode information recognition network: Step 5.1: Input the dial area image into the trained dial information recognition network, and output the category of each number in the dial area and the coordinates of the upper left corner of the rectangular box containing the number; Step 5.2: Input the barcode area image into the trained barcode information recognition network, and output the category of each number in the barcode area and the coordinates of the upper left corner of the rectangular box where the number is located; Step 6: Output the information recognition results of the dial and barcode: Step 6.1: Arrange each rectangular box in the dial area from small to large according to the size of the horizontal coordinate of the upper left corner point, read the category of the number contained in each rectangular box in turn, calculate the dial reading using the dial reading calculation formula, and output the recognition result; Step 6.2: Arrange each rectangular box in the barcode area from small to large according to the size of the horizontal coordinate of the upper left corner point, read the category of the number contained in each rectangular box in turn, calculate the barcode reading according to the barcode reading calculation formula, and output the recognition result.

2. The method for electric energy meter image target recognition based on deep learning according to claim 1, characterized in that: The steps for iteratively training the network using the gradient descent method described in step 3 are as follows: In the first step, all parameters of the training network are set to random numbers that conform to the normal distribution to obtain an initialized network. The mean of the normal distribution is 0, the variance is 0.02, and the learning rate of the network is set to 0.005; In the second step, 8 images are randomly selected from the training set and input into the initialized network. The network parameters are iteratively updated using the gradient descent method until the loss function converges to obtain a trained network. The loss function is as follows: Where L represents the average loss value of the selected samples, Σ represents the summation operation, k represents the index of the selected sample image, k = 1, 2, 3, 4, 5, 6, 7, 8, U represents the expected output of each sample image, U' represents the actual output of each sample image, ∩ represents the intersection operation, and ∪ represents the union operation.

3. The method for electric energy meter image target recognition based on deep learning according to claim 1, characterized in that: The calculation formula for the dial reading in step 6.1 is as follows: Among them, r1 represents the dial reading, Σ represents the summation operation, C 1i It represents the category of the number contained in the i-th rectangular box read from large to small in the dial image, and m represents the number of rectangular boxes in the dial image.

4. The method for electric energy meter image target recognition based on deep learning according to claim 1, characterized in that: The barcode reading calculation formula described in step 6.2 is as follows: Among them, r2 represents the barcode number, Σ represents the sum operation, C 2j It represents the category of the number contained in the j-th rectangular box read from the largest to the smallest in the barcode image, and n represents the number of rectangular boxes in the barcode image.

Images