A lightweight pointer meter reading method based on multi-task learning

Abstract

This invention discloses a lightweight pointer meter reading method based on multi-task learning. The method uses GhostNet and MobileNetV2 as lightweight backbone feature extraction networks; the feature fusion layer combines SE attention mechanism, FPN, and transposed convolutional layers to enhance the model's feature representation capability; the output layer employs different network structures for different functional branches. This invention integrates multiple tasks into a single model through multi-task learning, simultaneously optimizing multiple objective functions such as dial positioning, pointer and scale detection, and reading regression. It achieves end-to-end pointer meter positioning, pointer and scale detection, and reading recognition using only a lightweight neural network model, featuring low resource consumption, fast inference speed, and lightweight model, making it particularly suitable for applications in embedded systems.

Description

Technical Field

[0001] This invention belongs to the field of machine vision and deep learning, and in particular, it is a lightweight pointer meter reading method based on multi-task learning. Background Technology

[0002] Pointer-type meters are mostly mechanical meters, widely used in industrial production fields such as power, chemical, and metallurgy due to their reliability and strong electromagnetic interference resistance. However, in current industrial production, readings are generally done manually, which is slow, labor-intensive, and no longer suitable for automation and intelligent development.

[0003] To address the aforementioned issues, several automatic reading methods for pointer meters have been proposed. Currently, most pointer meter reading methods divide the entire reading process into three interconnected stages: dial detection, pointer and scale detection, and reading calculation. For example, Reference 1 (Wan Jilin, Wang Huifang, Guan Minyuan, et al. Automatic Recognition Method for Substation Pointer Meter Reading Based on Faster R-CNN and U-Net [J]. Power System Technology, 2020, 44(8):3097-3105) first uses Faster R-CNN to detect the instrument panel, then uses the semantic segmentation network U-Net to locate the position of the pointer and scale, and finally uses the angle method to realize the final reading calculation; Reference 2 (Zhang X, Dang X, Lv Q, et al. A pointer meter recognition algorithm based on deep learning [C]. 2020 3rd International Conference on Advanced Electronic Materials, Computers and Software Engineering (AEMCSE). IEEE, 2020:283-287) uses traditional image processing methods, such as line fitting and Hough transform, to detect the scale based on the instrument panel detected by Faster R-CNN, and then performs the reading. In this mode, the pointer meter reading task is divided into several independent small tasks, and the errors of each small task are superimposed, which not only consumes a lot of time but also reduces the recognition accuracy. Summary of the Invention

[0004] The purpose of this invention is to address the problems existing in existing pointer meter reading methods by proposing a lightweight pointer meter reading method based on multi-task learning to achieve end-to-end pointer meter reading recognition.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a lightweight pointer meter reading method based on multi-task learning, comprising the following steps:

[0006] Step 1: Construct a lightweight network model based on multi-task learning and train it.

[0007] Step 2: Input the meter image and use the network model obtained in Step 1 to perform inference and obtain the reading result.

[0008] Compared with the prior art, the significant advantages of this invention are:

[0009] (1) Utilizing the idea of ​​multi-task learning, an end-to-end pointer meter reading recognition method is proposed. This method designs a shared structure, enabling a single model to simultaneously perform three different tasks: pointer meter panel detection, pointer and scale detection, and final reading regression. Because the scale digits are detected, this method can directly regress the reading result to the true value rather than a relative value within the [0, 1] interval. Furthermore, due to the mutual constraints and complementarity of multi-task learning, the overall reading performance is superior to traditional methods.

[0010] (2) The end-to-end network is easy to use, and since there is only one model, the overall inference speed is much faster than traditional methods. In addition, the model proposed in this invention is designed entirely for embedded platforms. Each part of the model is designed with a lightweight network, the model size is small, the inference time is short, and it can be directly applied to embedded platforms, making it very suitable for industrial deployment.

[0011] (3) The model trained by the multi-task learning network in this paper is more conducive to model transfer. The shared structure designed in this paper can be transferred to other deep learning tasks after training. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0013] Figure 1 This is a network structure diagram of a pointer meter reading model in one embodiment.

[0014] Figure 2 The figure shows the structure diagram of the multi-scale feature fusion layer in one embodiment. In the figure, (a) is the structure diagram of SCN and (b) is the structure diagram of the overall multi-scale feature fusion layer SFN.

[0015] Figure 3The figure shows the output head structure diagram in one embodiment. (a) is the structure diagram of the instrument panel detection output head, (b) is the structure diagram of the pointer and scale detection output head, and (c) is the structure diagram of the regression reading output head.

[0016] Figure 4 This is a graph showing the result of a single prediction by the model in one embodiment.

[0017] Figure 5 This is a structural diagram of a lightweight pointer meter reading system based on multi-task learning in one embodiment. Detailed Implementation

[0018] This invention considers multi-task learning and designs a novel end-to-end lightweight pointer meter reading method. Multi-task learning can adaptively find relationships between multiple tasks, allowing different tasks to provide each other with additional useful information. Using this additional information, a better-performing and more robust model can be trained.

[0019] This invention addresses multiple tasks related to analog meter readings. Combining multi-task learning methods, it proposes a method for simultaneously performing training and inference for dashboard detection, pointer and scale detection, and reading regression, all using a single lightweight neural network. The method employs GhostNet and MobileNetV2 as lightweight backbone feature extraction networks. The feature fusion layer combines SE attention, FPN, and transposed convolutional layers to enhance the model's feature representation capabilities. The output layer uses different network structures for different functional branches; for example, a heatmap-based anchorless detection method is used for dashboard positioning and pointer and scale detection, while a fully convolutional network is used for reading regression. Unlike existing methods that use multiple models to perform dial positioning, pointer extraction, and reading recognition separately, this invention integrates multiple tasks into a single model. Through multi-task learning, it simultaneously optimizes multiple objective functions such as dial positioning, pointer and scale detection, and reading regression. It can achieve end-to-end pointer meter positioning, pointer and key point detection, and reading recognition using only a lightweight neural network model. It features low resource consumption, fast inference speed, and lightweight model, making it particularly suitable for applications in embedded systems.

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description, in conjunction with the accompanying drawings and embodiments, further illustrates this application. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0021] The present invention proposes a pointer meter reading method based on multi-task learning, the steps of which are as follows:

[0022] Step 1: Construct a lightweight network model based on multi-task learning and train it.

[0023] Step 2: Input the meter image and use the network model obtained in Step 1 to perform inference and obtain the reading result.

[0024] Furthermore, in one embodiment, the specific steps of constructing and training the lightweight network model based on multi-task learning in step 1 are as follows:

[0025] Step 1.1, Network structure design: A lightweight network structure is adopted to ensure that the network structure is simple, has few parameters, and is highly accurate.

[0026] Step 1.2, Loss Function Design: Define appropriate loss functions for the three different output headers.

[0027] Step 1.3: Collect data and train the designed network model.

[0028] In one embodiment, the lightweight network model based on multi-task learning constructed in step 1.1 is as follows: Figure 1 As shown, the overall network structure consists of four parts: a downsampling network (backbone feature extraction network), a multi-scale feature fusion layer, an upsampling network, and an output head. Furthermore, according to... Figure 1 It is known that multi-scale feature fusion layers and upsampling networks can be considered together. Therefore, in one embodiment, the network structure design in step 1.1 includes the following steps:

[0029] Step 1.1.1: A lightweight backbone feature extraction network suitable for pointer meter reading tasks is constructed using lightweight networks such as GhostNet and MobileNetV2. The input image is first processed by the backbone feature extraction network to extract features, and a feature map with a dimension of 320×24×24 is output for subsequent tasks;

[0030] Step 1.1.2: Combining the SE attention mechanism, FPN, and transposed convolution, an SFN layer was designed as a feature fusion and upsampling layer to restore the feature map resolution while enhancing the model's feature representation capability;

[0031] Step 1.1.3: The output head is lightweighted using the idea of ​​depthwise separable convolution. At the same time, different output layer structures are designed for three different tasks: instrument panel detection, pointer and scale detection, and reading regression.

[0032] Furthermore, in one embodiment, the backbone network in step 1.1.1 uses the G-bneck from GhostNet as its smallest module. The Ghost Module is the main bottleneck structure of the lightweight GhostNet network. Based on the output requirements, the bottleneck structure G-neck is mainly divided into two types: 1) G-bneck1 with a stride of 1, maintaining the same output feature map size; 2) G-bneck2 with a stride of 2, using depthwise separable convolutions for width and height compression, where the feature map size is halved. Additionally, an SE attention mechanism can be added to the G-neck to improve the model's feature representation ability. Specifically, its backbone network is set as follows:

[0033] The following modules are listed: 3×3 convolution with stride 2, G-bneck1 module with stride 1 and no SE structure, G-bneck2 module with stride 2 and no SE structure, G-bneck2 module with stride 1 and no SE structure, G-bneck2 module with stride 2 and added SE structure, G-bneck2 module with stride 1 and added SE structure, G-bneck2 module with stride 2 and no SE structure, G-bneck1 module with stride 1 and no SE structure, G-bneck2 module with stride 1 and no SE structure, G-bneck1 module with stride 1 and added SE structure, and G-bneck2 module with stride 2 and added SE structure.

[0034] Furthermore, in one embodiment, the multi-scale feature fusion layer in step 1.1.2 is as follows: Figure 2 As shown in the diagram. This invention combines depthwise separable convolution, multi-scale fusion layer FPN, and SE attention mechanism to design an SFN feature fusion layer, which can be divided into two parts: SCN and transposed convolution. SCN processes the input features to facilitate subsequent fusion, while transposed convolution performs upsampling to increase the resolution of the feature map. The design concept of SCN is derived from the SE module application in MobileNetV3, and it mainly includes two 1×1 convolutions, one depthwise separable convolution, and one SE attention module, as shown in the diagram. Figure 2 As shown in (a) above. Transposed convolution is mainly used to upsample the feature map and restore its resolution. Figure 2 (b) in the diagram is the overall structure of the feature fusion layer SFN, with layers 6, 9, and 13 of the backbone feature extraction network serving as the inputs to SFN.

[0035] Furthermore, in one embodiment, the output header structure of step 1.1.3 is as follows: Figure 3As shown, there are three output heads: a dashboard detection output head, a pointer and scale detection output head, and a reading regression output head. The dashboard detection output head and the pointer and scale detection output head each consist of one depthwise separable convolution layer, one 1x1 convolution layer, and a ReLU activation function; the reading regression output head consists of two depthwise separable convolution layers and one global average pooling layer.

[0036] Each branch of the dashboard detection and pointer and scale detection tasks first undergoes a depthwise separable convolution, then a 1×1 convolution to adjust the number of channels, and finally outputs the corresponding branch result. The output head of the reading regression task first downsamples the feature map using two stacked depthwise separable convolutions, and then uses a global average pooling operation to obtain the final reading result.

[0037] Furthermore, in one embodiment, the specific steps for designing the loss function in step 1.2 are as follows:

[0038] Step 1.2.1: Design the loss function corresponding to the instrument panel detection output;

[0039] Step 1.2.2: Design the loss function corresponding to the pointer and scale detection output;

[0040] Step 1.2.3: Design the loss function corresponding to the reading regression output;

[0041] Step 1.2.4, Design of the overall loss function.

[0042] Furthermore, in one embodiment, the loss function for the dashboard detection output in step 1.2.1 is designed as follows:

[0043] The dashboard detection output head has three branches: a heatmap output branch, a key point offset output branch, and a target length and width output branch, each with a specific dimension. The heatmap information (where n is the total number of instrument categories) is used to locate the target position, and the output dimension is... Keypoint bias information is used to compensate for potential errors in the heatmap; the output dimension is... The target's length and width information are used to obtain the final target bounding box size; Focal Loss is used as the keypoint loss function (i.e., heatmap loss). k The keypoint offset loss function L is constructed using L1 Loss. off Finally, L1 Loss is used to construct the target aspect ratio loss L. size The overall loss detected by the dashboard is the weighted sum of these three factors: L object =λ k L k +λ off L off +λsize L size , where λ k =1,λ off =1,λ size =0.1.

[0044] Furthermore, in one embodiment, the loss function for the pointer and scale detection output in step 1.2.2 is designed as follows:

[0045] This invention locates the pointer and scale positions by using key point information corresponding to the pointer and scale, transforming the overall pointer and scale detection task into a key point detection task. The pointer and scale detection output head has two branches, one with dimension... Heatmap information (a total of 3 types of key points: pointer endpoints, scale points, and dashboard center point) is used to detect key points, and the dimension is... Keypoint bias information is used to compensate for potential errors in keypoint detection. Correspondingly, Focal Loss is also used as the keypoint loss function L. k And using L1 Loss to construct the keypoint offset loss function L off The overall loss function is as follows: L point =λ k L k +λ off L off , where λ k =1,λ off =1.

[0046] Furthermore, in one embodiment, the pointer and regression reading output head and the corresponding loss function in step 1.2.3 are designed as follows:

[0047] The regression reading module has only one branch, which consists of two downsampling networks that downsample the input feature map and output the reading result. Therefore, L2 Loss can be directly applied to this single branch; here, L2 Loss is used. reg replace.

[0048] Furthermore, the overall model loss function in step 1.2.4 is designed as follows:

[0049] L total =λ1L point +λ2L object +λ3L reg λ1, λ2, and λ3 can all be dynamically adjusted.

[0050] Furthermore, in one embodiment, the specific steps of model training described in step 1.3 are as follows:

[0051] Step 1.3.1: Use methods such as flipping, rotating, and cutout to enhance the original data and expand the training dataset.

[0052] Step 1.3.2: Define the relevant parameters for model training. The parameter design is as follows: the optimization method is Adam, the batch size of input images is set to 8, and the initial learning rate is set to 5×10. -4 The learning rate is dynamically adjusted using a sampling cosine decay strategy.

[0053] Step 1.3.3: The focus of model training is dynamically adjusted by freezing parts of the structure or adjusting the corresponding values ​​of λ1, λ2, and λ3.

[0054] Furthermore, in one embodiment, the model inference steps of step 2 are as follows:

[0055] Step 2.1, data preprocessing: The input image is first scaled to 768×768, and then normalized with a mean of [0.49381427, 0.48300525, 0.47918715] and a variance of [0.19970816, 0.20239334, 0.20760854].

[0056] Step 2.2, Model Conversion and Inference: Convert the trained model into a suitable format, import the model and input the preprocessed data to obtain the model's output.

[0057] Step 2.3 involves post-processing the model output. First, the peak values ​​of the heatmap are extracted as center points, and center points with a confidence level less than 0.3 are removed. The final keypoint coordinates are then obtained by combining this with the output bias information. For pointer and scale detection, this keypoint is the final result. For dashboard detection, the length and width information from the dashboard detection output needs to be added to obtain the final bounding box. The final result from the regression reading output is the final reading result.

[0058] Figure 4 This is a result image of a single prediction by the model in one embodiment. After an overall model inference, the dashboard can be detected. Figure 4 The boxed area represents the detected dashboard; three different types of key points are obtained simultaneously, with red representing the center point, green representing the scale endpoints, and yellow representing the pointer points; finally, the model outputs regression readings, which correspond to the red text in the figure.

[0059] Figure 5 This is a structural diagram of a lightweight pointer meter reading system based on multi-task learning in one embodiment. As shown in the diagram, the lightweight pointer meter reading system based on multi-task learning proposed in this invention mainly includes:

[0060] The data acquisition module is used to acquire images of pointer-type meters taken during industrial production.

[0061] The model loading module loads pre-trained models.

[0062] The main module of the software implements functions including image preprocessing, input image model inference to obtain results, output result post-processing, and return result values.

[0063] It should be noted that the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A lightweight pointer type meter reading method based on multi-task learning, characterized in that, The method comprises the following steps: Step 1, constructing a lightweight network model based on multi-task learning and training the same; the specific method for network structure design is as follows: Step 1.1.1, adopting a GhostNet and a MobileNetV2 lightweight network to construct a lightweight backbone feature extraction network; the backbone network takes a G-bneck in the GhostNet as a minimum module, and specifically, the backbone network is sequentially set as follows: a 3x3 convolution with a step of 2, a G-bneck module without an SE structure with a step of 1, a G-bneck module without an SE structure with a step of 2, a G-bneck module without an SE structure with a step of 1, a G-bneck module with an SE structure with a step of 2, a G-bneck module with an SE structure with a step of 1, a G-bneck module without an SE structure with a step of 2, a G-bneck module without an SE structure with a step of 1, a G-bneck module without an SE structure with a step of 1, a G-bneck module with an SE structure with a step of 1, a G-bneck module with an SE structure with a step of 2; Step 1.1.2, combining an SE attention mechanism, an FPN and a transposed convolution to design an SFN layer for serving as a feature fusion and an up-sampling layer, wherein the feature fusion part mainly comprises a 1x1 convolution, a depth separable convolution and an SE attention mechanism module, and the up-sampling layer adopts a transposed convolution to be implemented; Step 1.1.3, adopting the idea of a depth separable convolution to lightweight the output head, and simultaneously designing different output layer structures for three different tasks of instrument panel detection, pointer and scale detection and reading regression; the output head has three: an instrument panel detection output head, a pointer and scale detection output head and a reading regression output head, and the structure design of the three different output heads is as follows: each branch of the instrument panel detection output head and the pointer and scale detection output head is composed of a layer of depth separable convolution, a layer of 1x1 convolution and a Relu activation function; the reading regression output head is composed of two layers of depth separable convolution and a layer of global average pooling; Step 2, inputting a meter image, and performing inference to obtain a reading result by using the network model obtained in step 1.

2. The multi-task learning based light-weight pointer-style meter reading method of claim 1, wherein, The constructing of the lightweight network model based on multi-task learning in step 1 and the training of the same further comprise the following steps: Step 1.2, loss function design, defining appropriate loss functions for the three different output heads, and specifically as follows: Step 1.2.1, designing a loss function corresponding to the instrument panel detection output; Step 1.2.2, designing a loss function corresponding to the pointer and scale detection output; Step 1.2.3, designing a loss function corresponding to the reading regression output; Step 1.2.4, overall loss function design; Step 1.3, collecting data and training the designed network model; the steps for model training are as follows: Step 1.3.1, adopting the methods of up-down flipping, rotation and cutout to perform enhancement on the original data, and expand the training data set; Step 1.3.2, define the learning rate and input training batch in model training, and use appropriate optimization method to optimize model training; Step 1.3.3, dynamically adjust the focus of model training by freezing part of the structure or adjusting the values of λ1, λ2 and λ3.

3. The multi-task learning based light-weight pointer-style meter reading method of claim 2, wherein, The loss function corresponding to the dial detection output in step 1.2.1 is designed as follows: The instrument panel detection output head has three branches, respectively outputting heat map information with a dimension of for positioning a target position, key point bias information with a dimension of for compensating for possible errors in the heat map, and target length-width information with a dimension of for obtaining a final target frame size, wherein n is a total number of instrument categories; a suitable loss function is designed for each branch, first, Focal Loss is used as a key point loss function L k , second, L1Loss is used to construct a key point offset loss function L off , and finally, L1 Loss is used to construct a target length-width loss L size , and the overall loss of the instrument panel detection is the weighted sum of the three: L object = λ k L k + λ off L off + λ size L size , wherein λ k = 1, λ off = 1, and λ size = 0.

1.

4. The method of claim 3, wherein, The loss function corresponding to the pointer and scale detection output in step 1.2.2 is designed as follows: The pointer and scale detection task is converted into a key point detection task, and positions of the pointer and the scale are located by detecting key point information corresponding to the pointer and the scale; the pointer and scale detection output head has two branches, which are heat map information with a dimension of used for detecting key points, and key point offset information with a dimension of used for making up possible errors in key point detection; Focal Loss is used as a key point loss function L k , and L1 Loss is used to construct a key point offset loss function L off ; a loss function of the overall pointer and scale detection is as follows: L point =λ k L k +λ off L off , wherein λ k =1, λ off =1.

5. The multi-task learning based light-weight pointer-style meter reading method of claim 4, wherein, The loss function corresponding to the regression readout output head in step 1.2.3 is designed as follows: The regression reading module has only one branch, which is composed of two down-sampling networks to down-sample the input feature map and finally output the reading result; for this one branch, L2 Loss is directly used here instead of L reg oss.

6. The multi-task learning based light-weight pointer-style meter reading method of claim 5, wherein, The overall loss function of the model in step 1.2.4 is designed as follows: L total = λ1L point + λ2L object + λ3L reg Where λ1, λ2 and λ3 can be dynamically adjusted.

7. The multi-task learning based light-weight pointer-style meter reading method of claim 1, wherein, The input meter image of step 2 is used to infer the readout result by using the network model obtained in step 1, and the specific steps are as follows: Step 2.1, data preprocessing, first scale the input picture to appropriate size, then use the normalization method based on mean and variance to normalize the picture data; Step 2.2, model conversion and inference, convert the trained model to appropriate format, import the model and input the preprocessed data, obtain the output result of the model; Step 2.3, post-processing of the output result of the model, first extract the peak value of the heat map as the center point, and eliminate the center points with confidence less than the threshold, combine the bias information to get the final key point coordinates; If it is pointer and scale detection, this key point is the final result, if it is dial detection, the length and width information in the dial detection output head needs to be added to get the final target box; The final result of the regression readout output head is the final readout result.

Images