A power meter screen positioning method and system based on deep learning and data enhancement

Abstract

The application provides a kind of based on deep learning and data enhancement electric energy meter screen positioning method and system, comprising: obtaining different model, state and perspective electric energy meter image dataset, data enhancement is carried out;Depth convolutional neural network is constructed, and total loss function is set;Enhanced dataset is used to train depth convolutional neural network, and total loss function is used to optimize network, and mean square error is used to evaluate model performance;The measured image of electric energy meter is input into the trained depth convolutional neural network, and the coordinates of its screen image and four corner points are extracted and output, and the screen positioning is completed.The application can realize high-precision and high-robustness positioning of electric energy meter liquid crystal screen, is suitable for complex and changeable environment, has good generalization ability and practical value, and provides a basis for subsequent electric energy meter anomaly detection.

Description

Technical Field

[0001] This invention relates to the application of computer vision technology in the field of power equipment inspection, and in particular to a method for locating the screen of an electricity meter based on deep learning and data augmentation. This method is used to accurately detect the corner coordinates of the LCD screen of an electricity meter, providing a foundation for subsequent detection of electricity meter anomalies. Background Technology

[0002] With the continuous advancement of automation in modern manufacturing and the construction of smart grids, automated testing platforms are increasingly widely used in industrial quality control. The automatic testing of LCD screens in electricity meters, as a key link in improving the production and calibration efficiency of electricity metering equipment, is in growing demand. In this type of testing task, accurate and stable positioning of the four corner points of the LCD screen is the foundation and prerequisite for subsequent image analysis operations such as character recognition and defect determination.

[0003] Traditional corner detection methods rely heavily on manual feature extraction and fixed threshold settings, which have the following limitations in practical industrial applications: First, manual inspection is inefficient, taking 3-5 minutes per meter, making it difficult to meet the cycle time requirements of large-scale mass production; second, template-based methods have poor adaptability, requiring the creation of new templates for different models, sizes, or screen layouts of electricity meters, lacking flexibility; third, traditional image processing methods are not robust enough, easily affected by changes in ambient lighting and screen surface reflections, leading to positioning errors or even failures; finally, the average positioning error of conventional corner detection algorithms in this specific scenario often exceeds 10 pixels, making it difficult to meet the positioning requirements of high-precision quality inspection.

[0004] In recent years, deep learning-based corner detection methods have demonstrated the potential for high-precision positioning in complex environments by autonomously learning features from large-scale data. However, existing deep learning solutions for corner detection on electricity meter LCD screens still face several key challenges, limiting their deployment effectiveness in practical automation platforms:

[0005] First, the feature extraction network lacks specificity. The general backbone network does not fully consider the structural characteristics of the electricity meter's LCD screen (such as the shape of the border and the distribution of the character area), making it difficult to effectively extract corner features with strong discriminative power.

[0006] Secondly, the loss function design did not incorporate spatial structural constraints, ignoring the inherent rectangular geometric relationships between the four corners of the screen (such as perpendicularity, parallelism, and diagonal constraints), which affected the geometric accuracy of the positioning results.

[0007] Third, the data augmentation strategies used during training are not compatible with real-world application scenarios and fail to effectively cover typical challenges in electricity meter detection (such as diverse reflections, stains, and posture changes), resulting in insufficient model generalization ability.

[0008] Fourth, some schemes have relatively simple coordinate regression head structure designs, which have poor anti-interference ability against complex backgrounds and are prone to model overfitting or output coordinate fluctuations.

[0009] Therefore, there is an urgent need for a detection solution specifically for corner location of the LCD screen of electricity meters, which can address the shortcomings of existing methods in the above aspects, and achieve high-precision, high-robustness and high-efficiency corner detection, thereby meeting the stringent requirements for positioning performance in automated quality control platforms. Summary of the Invention

[0010] The purpose of this invention is to address the shortcomings of existing methods in the aforementioned aspects by proposing a method for locating the LCD screen of an electricity meter based on deep learning and data augmentation. This method solves the problem of locating the LCD screen of an electricity meter and features high accuracy, high speed, and strong robustness to different experimental environments.

[0011] The technical solution of this invention is:

[0012] This invention provides a method for locating an electricity meter screen based on deep learning and data augmentation, comprising the following steps:

[0013] S1. Obtain image datasets of electricity meters of different models, states, and perspectives, and perform data augmentation;

[0014] S2. Construct a deep convolutional neural network, which includes a feature extraction layer and a regression head, used to extract image features and regress the coordinates of the four corner points of the screen;

[0015] S3. Set the total loss function, including mean square error constraint and direction constraint. The direction constraint is used to ensure that the quadrilateral formed by the predicted corner points surrounds the screen area.

[0016] S4. Train the deep convolutional neural network using an augmented dataset, optimize the network using the total loss function, and evaluate the model performance using mean squared error.

[0017] S5. Input the measured image of the electricity meter into the trained deep convolutional neural network, extract and output its screen image and the coordinates of the four corner points to complete the screen positioning.

[0018] Further, S1 includes: S11, acquiring images of different models of electricity meters in different states and from multiple perspectives, including front, top, bottom, left, and right views, and states including screen lit up when powered on and screen off when powered off; S12, using a labeling tool to label the four corner points of the screen in the acquired images, with the corner points labeled in a clockwise order starting from the top left corner; S13, performing data augmentation on the labeled dataset, including translation, rotation, perspective transformation, and shearing transformation; S14, storing the augmented dataset in a standard format, which includes corner point coordinate information.

[0019] Furthermore, in S13, data enhancements include:

[0020] Translation: Randomly move the image within the image plane to maintain the integrity of the electricity meter screen;

[0021] Rotation: Rotate the image at a random angle, where the angle is within a preset range;

[0022] Perspective transformation simulates the shooting effect from different angles, generating images with tilted perspectives;

[0023] Shearing transformation performs local transformations on the image, enhancing the model's robustness to occlusion.

[0024] Furthermore, in S2, the feature extraction layer employs a 7×7 convolutional layer, a batch normalization layer, multiple residual modules, and a global average pooling layer to output a feature vector of fixed dimensions; the regression head comprises a first linear layer, a first normalization layer, a first activation function layer, a Dropout layer, a second linear layer, a second normalization layer, a second activation function layer, and a third linear layer connected in sequence, mapping the output of the feature extraction layer to an output representing the coordinates of the four corner points.

[0025] Furthermore, the regression head performs the following steps:

[0026] The first linear layer maps the output of the feature extraction layer to the first dimension feature, followed by the first normalization layer and the first activation function layer, where the first activation function layer uses the ReLU activation function;

[0027] The Dropout layer uses a preset dropout rate to prevent overfitting;

[0028] The second linear layer maps the first-dimensional features to the second-dimensional features, followed by the second normalization layer and the second activation function layer, where the second activation function layer uses the GELU activation function.

[0029] The third linear layer maps the second-dimensional features into an output vector representing the coordinates of the four corner points.

[0030] Furthermore, S3 includes:

[0031] S31. Calculate the mean square error between the true corner coordinates and the predicted corner coordinates as the first loss term;

[0032]

[0033] in, i =1 corresponds to the top left corner. i =2 corresponds to the top right corner. i =3 corresponds to the bottom right corner. i =4 corresponds to the bottom left corner;

[0034] S32. The spatial position relationship of the predicted corner points is constrained by the direction constraint term. The constraint term calculates the difference between the horizontal and vertical coordinates of each corner point based on preset conditions. The rectangle enclosed by the corner points can completely contain the screen of the actual electricity meter, thereby ensuring that no screen information is lost.

[0035]

[0036] in, This represents the difference between the output result and the actual result. j =1, 2, 3, 4, k =1, 2, These are the label values ​​for the four corner points. These are the predicted values ​​for the four corner points;

[0037] S33. The ReLU function is used to quantify the error degree of the coordinate difference, and the direction constraint loss is used as the second loss term.

[0038]

[0039] S34. Add the mean square error to the weighted direction constraint loss to obtain the total loss;

[0040] .

[0041] Furthermore, the training of the deep convolutional neural network using an augmented dataset as described in S4 includes:

[0042] S41. Optimize the deep convolutional neural network using the AdamW optimizer and update the parameters based on the total loss function;

[0043] S42. Employ a dynamic learning rate adjustment strategy. When the validation loss no longer decreases over multiple consecutive epochs, reduce the learning rate to a preset ratio.

[0044] S43. During the training process, prediction results are generated on the validation set and a loss curve is plotted every fixed epoch. The loss curve includes total loss, mean squared error and orientation constraint loss.

[0045] S44. Evaluate the performance of the trained model using mean squared error.

[0046] A system employing a method for locating electricity meter screens based on deep learning and data augmentation includes:

[0047] The data acquisition and enhancement module is used to acquire image datasets of electricity meters of different models, states and perspectives, and to perform data enhancement processing on the datasets.

[0048] A network construction module is used to construct a deep convolutional neural network, the network including a feature extraction layer for extracting image features and a regression head for regressing the coordinates of the four corner points of the screen;

[0049] The loss function setting module is used to set the total loss function for the deep convolutional neural network. The total loss function includes mean square error loss for constraining coordinate accuracy and orientation constraint loss for ensuring that the quadrilateral formed by the predicted corner points can surround the screen area.

[0050] The network training and evaluation module is used to train the deep convolutional neural network using an augmented dataset, optimize the network parameters using the total loss function, and evaluate the model performance using mean squared error as an indicator.

[0051] The screen corner localization module is used to input the measured image of the electricity meter into the trained deep convolutional neural network, extract and output its screen image and the coordinates of the four corner points, and complete the screen localization.

[0052] The beneficial effects of this invention are:

[0053] This invention discloses an intelligent solution for accurate positioning of electricity meter screens, which solves the problem of corner detection of electricity meter screens under different models, states and viewing angles. In particular, it addresses the difficulty of accurately identifying screen areas using traditional methods under conditions such as power on or off, multiple viewing angles (such as front, top, and bottom views) and occlusion.

[0054] This invention acquires diverse images of electricity meters and performs data enhancement through translation, rotation, and perspective transformation. It constructs a deep convolutional neural network containing a 7×7 convolutional layer, a residual module, and a regression head. It extracts robust features and regresses the coordinates of the four corner points of the screen, achieving high-precision screen corner point positioning. It is applicable to various electricity meter models and shooting conditions, providing efficient and reliable technical support for automated electricity meter reading and intelligent management.

[0055] In this invention, the total loss function integrates mean square error and orientation constraint loss to ensure that the predicted corner points form quadrilaterals that completely surround the screen area, thus avoiding information loss. The AdamW optimizer and dynamic learning rate strategy are used to optimize the network, significantly improving the model's generalization ability in complex scenarios.

[0056] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0057] The above and other objects, features and advantages of the present invention will become more apparent from the more detailed description of exemplary embodiments of the invention in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the invention.

[0058] Figure 1 This is a flowchart of the energy meter screen positioning method based on deep learning and data augmentation according to the present invention.

[0059] Figure 2 This is a schematic diagram illustrating the data augmentation of the labeled dataset according to the present invention.

[0060] Figure 3 This is a structural diagram of the deep convolutional neural network model of the present invention.

[0061] Figure 4 This is a schematic diagram illustrating the spatial positional relationship of corner points predicted by the direction constraint of the present invention.

[0062] Figure 5 This is a training loss curve for the present invention.

[0063] Figure 6 This is a schematic diagram of the screen localization process of the deep convolutional neural network of the present invention.

[0064] Figure 7 This is a diagram illustrating the positioning effect of a partial energy meter screen according to the present invention. Detailed Implementation

[0065] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0066] Example 1

[0067] Figure 1 This is a flowchart of the energy meter screen positioning method based on deep learning and data augmentation according to the present invention.

[0068] like Figure 1 As shown, the specific process of the electricity meter LCD screen positioning method is as follows:

[0069] Step 1: Image Acquisition, Labeling, and Dataset Augmentation of Electricity Meter Screens. Acquire images of electricity meters in different states and types, including the screen lit up when powered on and the screen off when powered off. Filter and clean the acquired images. Label the pre-filtered electricity meters using LabelMe and process them in VOC dataset format, with the labels being the four corner points of the electricity meter screen. The annotation order starts from the top left corner and proceeds clockwise, and is stored in JSON file format.

[0070] like Figure 2 As shown, in order to simulate the tilt and tilt that may occur in the image acquisition device under real experimental scenarios, data augmentation is performed on the labeled dataset. The augmentation methods include: translation, randomly translating the image in the image plane; rotation, randomly rotating the image with a rotation angle range of -30 degrees to 30 degrees; perspective transformation, simulating the shooting effect from different perspectives; and cropping, cropping the image.

[0071] Step Two, as follows Figure 3 As shown, this is a deep convolutional neural network with enhanced residual structure: the network includes a feature extraction layer (conv1, bn1, relu, maxpool, layer1, layer2, layer3, layer4) and a regression head. The feature extraction layer uses an initial convolutional layer with a stride of 2 and a size of 7×7, a batch normalization layer, and four residual modules, extracting 512-dimensional features through a global average pooling layer. The regression head consists of three linear layers (fully connected layers), a normalization layer, and an activation function layer. Specifically, the structure is as follows: First linear layer: maps the 512-dimensional features to 256 dimensions, followed by the first normalization layer and the ReLU activation function. Dropout layer: with a dropout rate of 0.5 to prevent overfitting. Second linear layer: maps the 256-dimensional features to 128 dimensions, followed by the second normalization layer and the GELU activation function. Third linear layer: maps the 128-dimensional features to 8 dimensions, i.e., the normalized coordinates of the four corner points (each point contains x and y coordinates).

[0072] Step 3, as follows Figure 4 The diagram shows a directional constraint; the directional constraint loss term is used to constrain the spatial relationship of the predicted corner points, ensuring that the quadrilateral formed by the four predicted corner points can enclose the entire screen. The specific constraints are as follows:

[0073]

[0074] in, This represents the difference between the output result and the actual result. j =1, 2, 3, 4, k =1, 2, These are the label values ​​for the four corner points. Predicted values ​​for the four corner points;

[0075] The ReLU function is used to quantify the error level of the coordinate difference, and the orientation constraint loss is obtained as the second loss term.

[0076]

[0077] The total loss is obtained by adding the mean square error to the weighted direction constraint loss.

[0078] .

[0079] Step 4, Model Training and Evaluation: Train the network constructed in Step 2 using the dataset obtained in Step 1, and optimize it using the loss function designed in Step 3. The AdamW optimizer is used during training, and a dynamic learning rate adjustment strategy is set. The loss function curve during the training process is shown in the figure below. Figure 5 As shown, after training, mean squared error is used as the evaluation metric to assess the model.

[0080] like Figure 6 The diagram illustrates the screen localization process of the deep convolutional neural network of this invention. The measured image of the electricity meter is input into the trained deep convolutional neural network, which extracts and outputs the screen image and the coordinates of the four corner points to complete screen localization. The execution process includes the following key steps:

[0081] (1) Image preprocessing: Perform size normalization and tensor transformation operations on the input image.

[0082] (2) Model inference: Load the parameters of the pre-trained network model and perform forward computation of the neural network.

[0083] (3) Coordinate processing: Perform inverse normalization transformation on the output coordinates.

[0084] (4) Result output: Generate standardized JSON format annotation files through the result output module and realize visualization display.

[0085] Table 1 shows the MSE and orientation constraint loss values ​​under different hyperparameters. When the hyperparameter lambda = 0.02, the final test loss is 0.0007, the MSE is 0.0007, and the orientation constraint is 0.0018. The orientation constraint loss decreases compared to when it is 0.01, resulting in the best prediction performance, the smallest MSE, and the orientation constraint loss is also relatively small. Inference speed: The inference speed without preheating is 79ms. After model preheating, the average inference speed for 50 rounds is 20ms / frame.

[0086] Table 1

[0087]

[0088] Example 2

[0089] This invention provides a system for a method of locating electricity meter screens based on deep learning and data augmentation, comprising:

[0090] The data acquisition and enhancement module is used to acquire image datasets of electricity meters of different models, states and perspectives, and to perform data enhancement processing on the datasets.

[0091] A network construction module is used to construct a deep convolutional neural network, the network including a feature extraction layer for extracting image features and a regression head for regressing the coordinates of the four corner points of the screen;

[0092] The loss function setting module is used to set the total loss function for the deep convolutional neural network. The total loss function includes mean square error loss for constraining coordinate accuracy and orientation constraint loss for ensuring that the quadrilateral formed by the predicted corner points can surround the screen area.

[0093] The network training and evaluation module is used to train the deep convolutional neural network using an augmented dataset, optimize the network parameters using the total loss function, and evaluate the model performance using mean squared error as an indicator.

[0094] The screen corner localization module is used to input the measured image of the electricity meter into the trained deep convolutional neural network, extract and output its screen image and the coordinates of the four corner points, and complete the screen localization.

[0095] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A method for locating an electricity meter screen based on deep learning and data augmentation, characterized in that, Includes the following steps: S1. Obtain image datasets of electricity meters of different models, states, and perspectives, and perform data augmentation; S2. Construct a deep convolutional neural network, which includes a feature extraction layer and a regression head, used to extract image features and regress the coordinates of the four corner points of the screen; S3. Set the total loss function, including mean square error constraint and direction constraint. The direction constraint is used to ensure that the quadrilateral formed by the predicted corner points surrounds the screen area. S4. Train the deep convolutional neural network using an augmented dataset, optimize the network using the total loss function, and evaluate the model performance using mean squared error. S5. Input the measured image of the electricity meter into the trained deep convolutional neural network, extract and output its screen image and the coordinates of the four corner points to complete the screen positioning. In S2, the feature extraction layer uses a 7×7 convolutional layer, a batch normalization layer, multiple residual modules and a global average pooling layer to output a feature vector of fixed dimensions. The regression head comprises a first linear layer, a first normalization layer, a first activation function layer, a Dropout layer, a second linear layer, a second normalization layer, a second activation function layer, and a third linear layer connected in sequence, mapping the output of the feature extraction layer to an output representing the coordinates of the four corner points; The regression head performs the following steps: The first linear layer maps the output of the feature extraction layer to the first dimension feature, followed by the first normalization layer and the first activation function layer, where the first activation function layer uses the ReLU activation function; The Dropout layer uses a preset dropout rate to prevent overfitting; The second linear layer maps the first-dimensional features to the second-dimensional features, followed by the second normalization layer and the second activation function layer, where the second activation function layer uses the GELU activation function. The third linear layer maps the second-dimensional features into an output vector representing the coordinates of the four corner points; S3 includes: S31. Calculate the coordinates of the true corner points. With predicted corner coordinates The mean square error is used as the first loss term; ； in, Corresponding to the top left corner, Corresponding to the top right corner, Corresponding to the bottom right corner, Corresponding to the bottom left corner; S32. The spatial position relationship of the predicted corner points is constrained by the direction constraint term. The constraint term calculates the difference between the horizontal and vertical coordinates of each corner point based on preset conditions. The rectangle enclosed by the corner points can completely contain the screen of the actual electricity meter, thereby ensuring that no screen information is lost. ； in, This represents the difference between the output result and the actual result. j =1, 2, 3, 4, k =1, 2, These are the label values ​​for the four corner points. These are the predicted values ​​for the four corner points; S33. The ReLU function is used to quantify the error degree of the coordinate difference, and the direction constraint loss is used as the second loss term. ； S34. Add the mean square error to the weighted direction constraint loss to obtain the total loss; 。 2. The method for locating an electricity meter screen based on deep learning and data augmentation as described in claim 1, characterized in that... S1 includes: S11. Collect images of different models of electricity meters under different states and multiple perspectives. The perspectives include front, top, bottom, left side and right side. The states include the screen lighting up when powered on and the screen turning off when powered off. S12. Use a labeling tool to label the four corner points of the acquired image on the screen. The corner points are labeled in a clockwise order, starting from the top left corner. S13. Perform data augmentation on the labeled dataset, including translation, rotation, perspective transformation and shearing transformation; S14. Store the enhanced dataset in a standard format, which includes corner coordinate information.

3. The method for locating an electricity meter screen based on deep learning and data augmentation as described in claim 2, characterized in that... In S13, data augmentation includes: Translation: Randomly move the image within the image plane to maintain the integrity of the electricity meter screen; Rotation: Rotate the image at a random angle, where the angle is within a preset range; Perspective transformation simulates the shooting effect from different angles, generating images with tilted perspectives; Shearing transformation performs local transformations on the image, enhancing the model's robustness to occlusion.

4. The method for locating an electricity meter screen based on deep learning and data augmentation as described in claim 1, characterized in that... In S4, training the deep convolutional neural network using an augmented dataset includes: S41. Optimize the deep convolutional neural network using the AdamW optimizer and update the parameters based on the total loss function; S42. Employ a dynamic learning rate adjustment strategy. When the validation loss no longer decreases over multiple consecutive epochs, reduce the learning rate to a preset ratio. S43. During the training process, prediction results are generated on the validation set and a loss curve is plotted every fixed epoch. The loss curve includes total loss, mean squared error and orientation constraint loss. S44. Evaluate the performance of the trained model using mean squared error.

5. A system employing the method according to any one of claims 1-4, characterized in that, include: The data acquisition and enhancement module is used to acquire image datasets of electricity meters of different models, states and perspectives, and to perform data enhancement processing on the datasets. A network construction module is used to construct a deep convolutional neural network, the network including a feature extraction layer for extracting image features and a regression head for regressing the coordinates of the four corner points of the screen; The loss function setting module is used to set the total loss function for the deep convolutional neural network. The total loss function includes mean square error loss for constraining coordinate accuracy and orientation constraint loss for ensuring that the quadrilateral formed by the predicted corner points can surround the screen area. The network training and evaluation module is used to train the deep convolutional neural network using an augmented dataset, optimize the network parameters using the total loss function, and evaluate the model performance using mean squared error as an indicator. The screen corner localization module is used to input the measured image of the electricity meter into the trained deep convolutional neural network, extract and output its screen image and the coordinates of the four corner points, and complete the screen localization.

Images