Multi-type instrument image classification and reading identification method based on deep learning

Abstract

The invention discloses a multi-type instrument image classification and reading recognition method based on deep learning. According to the method, a deep learning method is used for training an instrument image classification and reading recognition model to classify and recognize multi-type instrument images; according to the model design, a multi-scale instrument detection network based on YOLOv11 in combination with Swin Transformer, a gas meter recognition network based on YOLOv11 small target recognition, a pressure meter pointer scale recognition network based on DeepLabv3 + segmentation, a water meter key point recognition network based on YOLOv11 key point detection and an electric meter digital display character recognition network based on CRNN in combination with Mobile NetV4 and CBAM are designed. A unified instrument image classification and reading recognition model is constructed based on the network, and after annotation data training is used, a to-be-recognized image is input, and then corresponding reading can be output. Through a unified framework compatible with multiple types of instruments, the problems that multiple types of instruments are difficult to process at the same time and efficient operation on embedded equipment is difficult to realize are solved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, specifically to a method for classifying and recognizing multi-type instrument images based on deep learning. Background Technology

[0002] Currently, energy consumption is mainly divided into three parts: buildings, industry, and transportation.

[0003] In the industrial sector, instruments are crucial devices for monitoring and controlling key parameters, essential for ensuring the safety, efficiency, and accuracy of production processes. With the rapid development of industrial automation and intelligent manufacturing, traditional manual reading methods are no longer sufficient to meet the demands of modern industry for efficient, accurate, and real-time energy consumption monitoring. This manual approach is not only inefficient and prone to errors, but also poses significant safety hazards in harsh environments such as high temperatures and pressures. Especially against the backdrop of the current goals of "carbon peaking and carbon neutrality," achieving real-time monitoring and management of energy consumption data such as water, electricity, gas, and pressure has become a core requirement for industrial enterprises to optimize their energy structure and reduce carbon emissions.

[0004] Taking electricity meters as an example, traditional energy consumption monitoring typically uses invasive devices (such as adding sensors to the front end of the meter) for data collection. Although this method can provide real-time data, its complex deployment and high cost make it difficult to scale up on existing equipment. In contrast, deep learning-based automatic meter reading technology uses a non-invasive method, requiring only the acquisition and recognition of meter images without altering the structure of existing equipment, to achieve efficient data reading. This approach not only reduces deployment and maintenance costs but also allows for rapid adaptation to various types of meters (such as water meters, electricity meters, gas meters, and pressure gauges), providing enterprises with flexible energy management solutions.

[0005] Automated instrument reading systems enable businesses to collect energy consumption data in real time, providing data support for energy conservation and emission reduction. By connecting the data to industrial IoT and big data platforms, businesses can analyze energy consumption in real time, identify wasteful processes, optimize energy use, and reduce carbon emissions. Furthermore, this technology can help businesses address the energy efficiency optimization needs of energy-intensive production processes, facilitating the digital and intelligent transformation of energy management.

[0006] In recent years, with the rapid development of deep learning and computer vision technologies, automated instrument reading methods based on image recognition have gradually become one of the research hotspots. However, existing methods still have the following problems: diverse instrument types, poor adaptability to complex scenarios, and difficulty in deploying resource-constrained equipment. Summary of the Invention

[0007] To address the shortcomings of the existing technologies, this invention provides a deep learning-based method for multi-type instrument image classification and reading recognition. By combining an improved YOLOv11 model, a Deeplabv3+ segmentation model, and a CRNN character recognition network, a universal framework for unified detection, category recognition, and automatic reading of various instruments is designed. This solves the problems of existing technologies being unable to handle multiple instrument types simultaneously and being unable to run efficiently on resource-limited embedded devices.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0009] A deep learning-based method for classifying and recognizing multi-type instrument images includes the following steps:

[0010] S1. Acquire images of multiple types of meters, including images of gas meters, pressure gauges, water meters, and electricity meters;

[0011] S2. Design a multi-scale instrument detection network based on the YOLOv11 model architecture and combined with the Swin Transformer module; a gas meter recognition network based on the YOLOv11 model architecture for small target recognition; a pressure gauge pointer scale segmentation and recognition network based on the Deeplabv3+ segmentation model; a water meter key point recognition network based on the YOLOv11 model architecture and using key point recognition; and an electricity meter digital display character recognition network based on the CRNN network and combined with the MobileNetV4 lightweight network and CBAM attention mechanism.

[0012] S3. Based on the multi-scale instrument detection network, gas meter recognition network, pressure gauge pointer scale segmentation recognition network, water meter key point recognition network, and electricity meter digital display character recognition network, construct an instrument image classification and reading recognition model;

[0013] S4. The collected multi-type instrument images are labeled and preprocessed as the training set of the instrument image classification and reading recognition model, and then input into the instrument image classification and reading recognition model for training to obtain the trained instrument image classification and reading recognition model.

[0014] S5. Input the multi-type instrument images to be identified into the trained instrument image classification and reading recognition model, and output the corresponding instrument reading recognition results of the multi-type instrument images to be identified.

[0015] As a preferred embodiment, the instrument image classification and reading recognition model uses the input multi-type instrument images as input to a multi-scale instrument detection network, and combines the YOLOv11 model architecture with Swin The Transformer module extracts multi-scale global features from various types of instrument images, classifying the input images into gas meter images, pressure gauge images, water meter images, and electricity meter images. For gas meter images, a gas meter recognition network is used to identify scale lines and small numerical targets. A reading correction algorithm is then used to fine-tune the readings, resulting in the gas meter image reading recognition result. For pressure gauge images, the Deeplabv3+ segmentation model is used to perform pixel-level semantic segmentation of the pointer and scale regions, calculating the angular relationship between them to obtain the pressure gauge image reading recognition result. For water meter images, key point reading recognition is used, and geometric relationship analysis is employed to obtain the water meter image reading recognition result. For electricity meter images, an improved CRNN network is used for character recognition to obtain the electricity meter image reading recognition result. Thus, the corresponding instrument reading recognition results for each type of instrument image are obtained.

[0016] As a preferred embodiment, the multi-scale instrument detection network based on the YOLOv11 model architecture and combined with the Swing Transformer module includes: designing the backbone network of the Swing Transformer module, designing the CBAM attention mechanism and the neck network of the weighted bidirectional feature pyramid network BiFPN, and designing the detection head with training loss.

[0017] The backbone network includes a Win Transformer module, an SPPF module, a C2PSA module, and three cascaded convolutional layers connected in sequence; wherein the three cascaded convolutional layers include a first convolutional layer, a second convolutional layer, and a third convolutional layer.

[0018] The neck network includes two upsampled feature fusion pyramid modules, three convolutional feature fusion modules, and a CBAM module connected in sequence; wherein, the upsampled feature fusion pyramid module includes cascaded upsampled units, a BiFPN module, and a C2f module, and the convolutional feature fusion module includes cascaded convolutional layers, a BiFPN module, and a C2f module;

[0019] The detection head includes a parallel cross-union loss module and a classification loss module; wherein, the cross-union loss module includes two cascaded convolutional layers, a two-dimensional convolutional layer and a CIoU loss function, and the classification loss module includes two cascaded depthwise separable convolutional units, a two-dimensional convolutional layer and a CLS loss function.

[0020] In the multi-scale instrument detection network, the input of the instrument image classification and reading recognition model serves as the input of the backbone network. The output of the first convolutional layer in the backbone network also serves as the input of the BiFPN module in the second upsampled feature fusion pyramid module of the neck network. The output of the second convolutional layer also serves as the input of the BiFPN module in the first upsampled feature fusion pyramid module of the neck network. The output of the third convolutional layer also serves as the input of the upsampling unit in the first upsampled feature fusion pyramid module of the neck network and the BiFPN module in the third convolutional feature fusion module, respectively. The outputs of the first convolutional feature fusion module, the second convolutional feature fusion module, and the CBAM module in the neck network also serve as the inputs of the detection head, respectively.

[0021] As a preferred embodiment, the gas meter identification network based on the YOLOv11 model architecture for small target recognition includes: designing a backbone network that integrates a parallel patch perception attention mechanism in the C3k2 module, and designing a detection head with adaptive threshold focus loss and a small target detection layer;

[0022] The backbone network includes a convolutional layer, four C3k2_PPA units, an SPPF module, and a C2PSA module connected in sequence; wherein, the C3k2_PPA unit includes a cascaded C3k2_PPA module and a convolutional layer;

[0023] The neck network includes three upsampling C3k2 modules and three convolutional C3k2 modules connected in sequence; wherein, the upsampling C3k2 module includes a cascaded upsampling layer, a feature fusion layer and a C3k2 module, and the convolutional C3k2 module includes a cascaded convolutional layer, a feature fusion layer and a C3k2 module;

[0024] The detection head is used to obtain the reading recognition result of the gas meter image based on the output of the neck network and the small target detection layer, which is then used as the output of the gas meter recognition network.

[0025] In the gas meter recognition network, the input gas meter image is used as the input to the backbone network. The outputs of the C3k2_PPA modules in the first and second C3k2_PPA units of the backbone network are also used as the inputs to the first and second upsampling C3k2 modules in the neck network, respectively. The output of the C2PSA module is also used as the input to the feature fusion layer in the third convolutional C3k2 module of the neck network. The outputs of the first and second upsampling C3k2 modules in the neck network are also used as the inputs to the first and second convolutional C3k2 modules, respectively. The outputs of the third upsampling C3k2 module and the first and second convolutional C3k2 modules are also used as the inputs to the detection head.

[0026] As a preferred embodiment, the pressure gauge pointer scale segmentation and recognition network based on the Deeplabv3+ segmentation model includes: an encoder designed with an ASPP module, and a decoder designed with an Edge Head module and a CBAM module;

[0027] The processing steps of the pressure gauge pointer scale segmentation and recognition network include:

[0028] The input feature map is fed into the encoder of the Deeplabv3+ segmentation model to extract multi-level feature maps. Then, the multi-level feature maps are input into the ASPP module for multi-scale feature extraction. The ASPP module includes a 1×1 convolution branch, a 3×3 dilated convolution branch with a dilation rate of 6, a 3×3 dilated convolution branch with a dilation rate of 12, a 3×3 dilated convolution branch with a dilation rate of 18, and an image pooling branch. The output features of each branch are concatenated to obtain the ASPP fusion feature map, and then channel integration is performed through a convolutional layer. Finally, the integrated ASPP fusion feature map is... The input is fed into the first CBAM module for attention enhancement, adaptively enhancing the feature responses of the slender pointer structure and scale area. The attention-enhanced feature map is then input to the decoder, where upsampling is used to progressively restore the spatial resolution of the feature map. The upsampled feature map is then concatenated with the corresponding low-level feature map from the encoder to achieve multi-scale feature fusion. The fused feature map is then input to the second CBAM module, which outputs the segmentation feature map of the main segmentation branch. Simultaneously, the decoder is connected in parallel with an EdgeHead branch, which includes a third CBAM module. This EdgeHead branch receives the feature map input shared with the main segmentation branch, extracts edge features through convolution stacking, and outputs an edge prediction map. Finally, the segmentation feature map output from the main segmentation branch is fused with the edge prediction map output from the EdgeHead branch to obtain the final segmentation prediction map of the pressure gauge pointer area and scale area, which serves as the reading recognition result of the pressure gauge image.

[0029] As a preferred embodiment, the processing procedure of the water meter key point identification network includes:

[0030] The water meter image is input into a small target detection network based on an improved YOLOv11 for inference, resulting in bounding boxes containing digits, small dials, pointer endpoints, and a zero reference point. These bounding boxes are then sorted by their horizontal coordinates to form an ordered sequence. Based on this sequence, the digits are extracted and concatenated to form the integer part of the reading. Simultaneously, the dial, pointer endpoint, and zero reference point are associated with multiple keypoint groups based on their spatial locations. For each keypoint group, two vectors are constructed with the center of the dial as the center, pointing to the pointer endpoint and the zero reference point. The angle between these vectors is calculated and converted into a circumferential ratio. According to the range weights corresponding to each small dial, the ratio is converted into the decimal part of the reading. Finally, the integer and decimal parts are added together to output the complete water meter reading, which serves as the reading recognition result of the water meter image.

[0031] As a preferred embodiment, the meter digital display character recognition network based on CRNN network and combined with MobileNetV4 lightweight network and CBAM attention mechanism includes: designing convolutional layers of MobileNetV4 lightweight network, recurrent layers of CBAM attention mechanism and transcription layer.

[0032] As a preferred embodiment, the processing procedure of the meter digital display character recognition network includes:

[0033] The input feature map is processed by a lightweight MobileNetV4 network to extract convolutional features, resulting in a convolutional feature map. Then, the convolutional feature map is serialized, unfolding the two-dimensional feature map along its width into a time series S. A CBAM attention mechanism is added to the time series S to generate an enhanced context vector. Next, the enhanced context vector is fed into an LSTM bidirectional long short-term memory network for encoding, further extracting temporal information. Subsequently, the feature sequence output from the recurrent layer is decoded by a transcription layer. Finally, a connection-time classification loss function is used to predict the numerical sequence, generating a numerical string representing the instrument reading result.

[0034] As a preferred embodiment, the instrument image classification and reading recognition model is trained in the following manner:

[0035] Multi-type instrument sample images with pre-annotated instrument types, pointer and scale areas, and digital areas are used as training samples to form a training sample set. These samples are then input into the multi-scale instrument detection network, gas meter recognition network, pressure gauge pointer and scale segmentation recognition network, electricity meter digital display character recognition network, and water meter key point recognition network of the instrument image classification and reading recognition model. Each network is optimized and updated with the objective of minimizing its respective loss function, thereby training the instrument image classification and reading recognition model.

[0036] As a preferred embodiment, the multi-scale instrument detection network adopts the CIOU loss function composed of the CIoU loss function and the CLS loss function; the gas meter identification network and the water meter key point identification network both adopt the adaptive threshold focus loss function; the pressure gauge pointer scale segmentation identification network adopts the Dice loss function; and the electricity meter digital display character identification network adopts the CTC loss function.

[0037] Compared with the prior art, the present invention has the following technical effects:

[0038] (1) The multi-scale instrument detection network of the present invention is based on the YOLOv11 model architecture. Through the multi-task learning framework, it can simultaneously perform instrument type classification and region detection (such as digital region, pointer region and scale region). By introducing the SwinTransformer module, the network's ability to model the global context information of instrument images is enhanced. The CBAM attention mechanism and BiFPN module are combined to achieve multi-scale feature fusion. The CIOU loss function is used to optimize the regression accuracy. This design achieves high-precision classification of multi-type instrument images and can adapt to complex scenes to realize real-time inference.

[0039] (2) The present invention designs a gas meter recognition network based on an improved YOLOv11, introduces a parallel patch perception attention mechanism (C3k2_PPA module) and a small target detection layer, and combines it with an adaptive threshold focus loss function. This design significantly enhances the feature extraction and localization capabilities of high-detail targets such as fine scale lines, numerical characters and triangular indicator marks on the gas meter. The adaptive threshold focus loss function dynamically adjusts the weights of easy and difficult samples during training, effectively alleviating the gradient imbalance problem in small target detection. At the same time, combined with the reading correction algorithm, the gas meter reading accuracy is finally improved to four decimal places, meeting the application requirements of high-precision metering scenarios.

[0040] (3) The pressure gauge pointer scale segmentation and recognition network of the present invention adopts an improved DeepLabv3+ network, embedding CBAM attention mechanism in multiple places after the ASPP module and at the decoding end, and adding an independent Edge Head branch. This multi-stage CBAM module enables the network to adaptively focus on the slender structure of the pointer and the scale area, enhancing the perception of key details and segmentation accuracy. The Edge Head branch specifically extracts and strengthens edge features, effectively improving the segmentation clarity of the pointer and scale boundary, and can maintain stable segmentation performance, especially in low contrast or complex backgrounds. Through segmentation mask post-processing and geometric analysis, the accurate calculation of the pointer angle is realized, thereby completing the highly reliable pressure gauge reading recognition.

[0041] (4) This invention constructs a character recognition network for electricity meter displays based on an improved CRNN architecture and combined with a lightweight MobileNetV4 network and a CBAM attention mechanism. MobileNetV4 serves as a lightweight convolutional feature extractor, which significantly reduces computational complexity and memory usage, making it suitable for resource-constrained embedded deployments. The CBAM attention mechanism is embedded in the recurrent layer to enhance the model's ability to focus on key features in character sequences and improve the robustness of recognition of complex characters or ambiguous regions. While maintaining the advantages of CRNN in processing sequence data, this network further optimizes recognition accuracy and inference efficiency, making it suitable for real-time character recognition of digital display instruments such as electricity meters.

[0042] (5) This invention reuses the improved YOLOv11 small target detection framework, combines key point detection and geometric relationship analysis to realize water meter reading. By using the small target detection capability in the gas meter identification network, it ensures stable detection of multiple discs, pointer endpoints and zero reference points on the water meter face. Through key point spatial association and vector angle calculation, it realizes geometric reading analysis of multi-pointer small dials, avoiding reliance on complex segmentation or template matching. This method separates and processes the integer part (digital characters) and the fractional part (pointer angle) and then merges them to adapt to the multi-range and multi-pointer reading structure of water meters, taking into account both recognition accuracy and system real-time performance. Attached Figure Description

[0043] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0044] Figure 1 This invention discloses a flowchart of a method for classifying and recognizing images of multiple types of instruments based on deep learning;

[0045] Figure 2 This is a schematic diagram illustrating the improved multi-scale instrument detection network architecture in an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of a multi-scale instrument detection network architecture based on YOLOv11 in an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram illustrating the improvement of the gas meter identification network in an embodiment of the present invention;

[0048] Figure 5 This is a schematic diagram of the gas meter identification network architecture based on YOLOv11 in an embodiment of the present invention;

[0049] Figure 6 This is a schematic diagram of the improved Deeplabv3+ network structure in an embodiment of the present invention;

[0050] Figure 7 This is a schematic diagram illustrating the improved CRNN-based model architecture in an embodiment of the present invention;

[0051] Figure 8 This is a schematic diagram of the improved CRNN network structure in an embodiment of the present invention;

[0052] Figure 9 This is a schematic diagram of the multi-type instrument classification and reading recognition algorithm in an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but only to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0054] The present invention will now be described in further detail with reference to the accompanying drawings.

[0055] Figure 1 This invention discloses a deep learning-based method for classifying and recognizing images of various types of instruments. Addressing the problems of diverse instrument types, poor adaptability to complex scenarios, and difficulty in deploying resource-constrained equipment in existing automated instrument reading methods, this method includes the following steps:

[0056] S1. Acquire images of multiple types of meters, including images of gas meters, pressure gauges, water meters, and electricity meters;

[0057] S2. Design a multi-scale instrument detection network based on the YOLOv11 model architecture and combined with the Swin Transformer module, a gas meter recognition network based on the YOLOv11 model architecture for small target recognition, a pressure gauge pointer scale segmentation and recognition network based on the Deeplabv3+ segmentation model architecture, a water meter recognition network based on the YOLOv11 model architecture and using key point recognition, and an electricity meter digital display character recognition network based on the CRNN network and combined with the MobileNetV4 lightweight network and CBAM attention mechanism.

[0058] S3. Based on the multi-scale instrument detection network, gas meter recognition network, pressure gauge pointer scale segmentation recognition network, water meter recognition network, and electricity meter digital display character recognition network, construct an instrument image classification and reading recognition model;

[0059] S4. The collected multi-type instrument images are labeled and preprocessed as the training set of the instrument image classification and reading recognition model, and then input into the instrument image classification and reading recognition model for training to obtain the trained instrument image classification and reading recognition model.

[0060] S5. Input the multi-type instrument images to be identified into the trained instrument image classification and reading recognition model, and output the corresponding instrument reading recognition results of the multi-type instrument images to be identified.

[0061] This invention addresses the problem of automatic identification of multiple types of instruments in industrial environments by proposing a deep learning-based solution. This method improves the YOLOv11 model, the Deeplabv3+ segmentation model, and the CRNN model to construct multi-scale instrument detection networks, gas meter recognition networks, pressure gauge pointer scale segmentation and recognition networks, water meter key point recognition networks, and electricity meter digital display character recognition networks. This results in an instrument image classification and reading recognition model for classifying and recognizing multiple types of instruments, achieving unified classification and reading recognition for both pointer and digital display instruments. This design is particularly suitable for handling images in complex scenarios, such as varying lighting, background interference, and image tilt, while also meeting the real-time inference requirements of resource-constrained devices. Through model quantization and optimization, this invention can be efficiently deployed on embedded devices, significantly improving the automation level and accuracy of instrument readings.

[0062] Furthermore, this invention overcomes the limitations of traditional manual reading methods, such as low efficiency, susceptibility to errors, and safety hazards in harsh environments. Through automated instrument reading technology, it helps enterprises achieve real-time monitoring and management of energy consumption data, optimize energy use, and reduce carbon emissions, thereby supporting the goal of "carbon peaking and carbon neutrality." This method not only improves work efficiency and reduces labor costs but also enhances system security and reliability, making it suitable for various application scenarios such as industrial automation, smart energy management, and smart city monitoring.

[0063] The following is a more detailed description of the deep learning-based method for classifying and recognizing multi-type instrument images.

[0064] 1. Image acquisition and annotation

[0065] In this embodiment, a Raspberry Pi 4B 2GB is used as the main control device. A high-definition camera module is connected via the Raspberry Pi's CSI interface. The camera is mounted in front of the instrument using an auxiliary bracket. The camera position is adjusted by controlling the image size. An SD card is prepared for storing the collected image data. By setting commands on the terminal, the Raspberry Pi camera can be used to take pictures, acquiring images of pointer and digital instruments (such as water meters, electricity meters, gas meters, and pressure gauges). The acquired images are stored. The LabelImg annotation tool is used to annotate the acquired images, including the instrument type, pointer area, and digital area. The Labelme annotation tool is used to annotate the pointer and scale areas of the acquired images.

[0066] 2. Image preprocessing

[0067] First, the resolution of the acquired multi-type instrument images is adjusted by scaling them to the fixed resolution required for the YOLOv11 model input, which is 640×640 pixels. Then, the pixel values ​​of the images are normalized to [0,1]. Gaussian filtering is applied to the acquired multi-type instrument images to remove high-frequency noise caused by camera noise or environmental interference, so as to retain important edge and detail features related to the instruments. To address the problem of low image contrast caused by insufficient lighting or environmental occlusion, histogram equalization or contrast-limited adaptive histogram equalization technology is used to enhance the contrast of pointer, scale, and digital areas in the instrument images, highlighting key information areas. To address potential tilting in the acquired images, edge detection and geometric analysis techniques are used to detect the instrument borders and perform rotation correction. Data augmentation is performed on the images, including random rotation, translation, scaling, flipping, and noise addition.

[0068] 3. Instrument Image Classification and Reading Recognition Model

[0069] The instrument image classification and reading recognition model includes a multi-scale instrument detection network based on the YOLOv11 model architecture and combined with the SwinTransformer module, a gas meter recognition network based on the YOLOv11 model architecture for small target recognition, a pressure gauge pointer scale segmentation and recognition network based on the Deeplabv3+ segmentation model, a water meter key point recognition network based on the YOLOv11 model architecture and using key point recognition, and an electricity meter digital display character recognition network based on the CRNN network and combined with the MobileNetV4 lightweight network and CBAM attention mechanism.

[0070] The instrument image classification and reading recognition model takes the input of multiple types of instrument images as input to a multi-scale instrument detection network, and uses the YOLOv11 model architecture combined with Swin The Transformer module extracts multi-scale global features from various types of instrument images, classifying the input images into gas meter images, pressure gauge images, water meter images, and electricity meter images. For gas meter images, a gas meter recognition network is used to identify scale lines and small numerical targets. A reading correction algorithm is then used to fine-tune the readings, resulting in the gas meter image reading recognition result. For pressure gauge images, the Deeplabv3+ segmentation model is used to perform pixel-level semantic segmentation of the pointer and scale regions, calculating the angular relationship between them to obtain the pressure gauge image reading recognition result. For water meter images, key point reading recognition is used, and geometric relationship analysis is employed to obtain the water meter image reading recognition result. For electricity meter images, an improved CRNN network is used for character recognition to obtain the electricity meter image reading recognition result. Thus, the corresponding instrument reading recognition results for each type of instrument image are obtained.

[0071] The following sections will introduce the multi-scale instrument detection network, gas meter identification network, pressure gauge pointer scale segmentation identification network, water meter key point identification network, and electricity meter digital display character identification network.

[0072] 3.1 Multi-scale instrument detection network

[0073] like Figure 2 As shown, the multi-scale instrument detection network includes a YOLOv11 model architecture combined with the SwinTransformer module. The multi-scale instrument detection network includes: designing the backbone network of the SwinTransformer module, designing the CBAM attention mechanism and the neck network of the weighted bidirectional feature pyramid network BiFPN, and designing the detection head with training loss.

[0074] Specifically, such as Figure 3 As shown, the backbone network includes a Win Transformer module, an SPPF module, a C2PSA module, and three cascaded convolutional layers connected in sequence; wherein the three cascaded convolutional layers include a first convolutional layer, a second convolutional layer, and a third convolutional layer.

[0075] The Swin Transformer unit includes cascaded window layer normalization subunits and perceptual layer normalization subunits. The window layer normalization subunits include cascaded layer normalization layers and a window-based multi-head self-attention mechanism. The perceptual layer normalization subunits include cascaded layer normalization layers and a multilayer perceptron. Each sublayer in the Transformer unit uses residual connections to maintain information flow and stabilize training. The SPPF module includes a cascaded first convolutional layer, three max-pooling layers, a feature concatenation layer, and a second convolutional layer. In the SPPF module, the input feature map is processed by the first convolutional layer for feature extraction. Then, the convolutional feature map is passed through three max-pooling layers with different parameters to capture features at different scales. Next, the output feature maps of the three max-pooling layers are concatenated to fuse multi-scale features. Finally, the concatenated feature map is passed through the second convolutional layer to obtain the output feature map. The C2PSA module includes a cascaded first convolutional layer, N pyramid spatial attention blocks, a feature concatenation layer, and a second convolutional layer. The pyramid spatial attention block includes a cascaded attention layer, a first convolutional layer, and a second convolutional layer. In the pyramid spatial attention block, the input feature map is added to the output of the attention layer through residual connections, and the input of the first convolutional layer is added to the output of the second convolutional layer, thereby preserving the original information and enhancing feature representation.

[0076] The neck network includes two upsampled feature fusion pyramid modules, three convolutional feature fusion modules, and a CBAM module connected in sequence; wherein, the upsampled feature fusion pyramid module includes cascaded upsampled units, a BiFPN module, and a C2f module, and the convolutional feature fusion module includes cascaded convolutional layers, a BiFPN module, and a C2f module;

[0077] The detection head is used to optimize the corresponding instrument reading recognition results of multiple types of instrument images output by the neck network using the loss function CIoU, and uses the optimized corresponding instrument reading recognition results of multiple types of instrument images as the output of the instrument image classification and reading recognition model; the detection head includes a parallel cross-union loss module and a classification loss module; wherein, the cross-union loss module includes two cascaded convolutional layers, a two-dimensional convolutional layer and a CIoU loss function, and the classification loss module includes two cascaded depthwise separable convolutional units, a two-dimensional convolutional layer and a CLS loss function;

[0078] In the multi-scale instrument detection network, the input of the instrument image classification and reading recognition model serves as the input of the backbone network. The output of the first convolutional layer in the backbone network also serves as the input of the BiFPN module in the second upsampled feature fusion pyramid module of the neck network. The output of the second convolutional layer also serves as the input of the BiFPN module in the first upsampled feature fusion pyramid module of the neck network. The output of the third convolutional layer also serves as the input of the upsampling unit in the first upsampled feature fusion pyramid module of the neck network and the BiFPN module in the third convolutional feature fusion module, respectively. The outputs of the first convolutional feature fusion module, the second convolutional feature fusion module, and the CBAM module in the neck network also serve as the inputs of the detection head, respectively.

[0079] This embodiment's network is based on the YOLOv11 model architecture. Through a multi-task learning framework, it can simultaneously perform instrument type classification and region detection (such as digital regions, pointer regions, and scale regions). By introducing the Swing Transformer module, it enhances the network's ability to model global contextual information of instrument images. It combines the CBAM attention mechanism and the BiFPN module to achieve multi-scale feature fusion, and uses the CIOU loss function to optimize regression accuracy. This design achieves high-precision classification of multiple types of instrument images, while also adapting to complex scenarios and enabling real-time inference.

[0080] 3.2 Gas Meter Identification Network

[0081] like Figure 4 As shown, the gas meter identification network based on the YOLOv11 model architecture for small target recognition includes: a backbone network integrating a parallel patch perception attention mechanism in the C3k2 module, a neck network of the C3k2 module, and a detection head with adaptive threshold focus loss and a small target detection layer.

[0082] Specifically, such as Figure 5 As shown, the backbone network includes a convolutional layer, four C3k2_PPA units, an SPPF module, and a C2PSA module connected in sequence; wherein, the C3k2_PPA unit includes cascaded C3k2_PPA modules and convolutional layers;

[0083] The neck network includes three upsampling C3k2 modules and three convolutional C3k2 modules connected in sequence; wherein, the upsampling C3k2 module includes a cascaded upsampling layer, a feature fusion layer and a C3k2 module, and the convolutional C3k2 module includes a cascaded convolutional layer, a feature fusion layer and a C3k2 module;

[0084] The detection head is used to obtain the reading recognition result of the gas meter image based on the output of the neck network and the small target detection layer, which is then used as the output of the gas meter recognition network.

[0085] In the gas meter recognition network, the input gas meter image is used as the input to the backbone network. The outputs of the C3k2_PPA modules in the first and second C3k2_PPA units of the backbone network are also used as the inputs to the first and second upsampling C3k2 modules in the neck network, respectively. The output of the C2PSA module is also used as the input to the feature fusion layer in the third convolutional C3k2 module of the neck network. The outputs of the first and second upsampling C3k2 modules in the neck network are also used as the inputs to the first and second convolutional C3k2 modules, respectively. The outputs of the third upsampling C3k2 module and the first and second convolutional C3k2 modules are also used as the inputs to the detection head.

[0086] In specific implementation, the processing procedure of the gas meter recognition network is as follows: The gas meter image is input into multiple convolutional layers of the backbone network for initial feature extraction; the initial features are sequentially input into multiple C3k2_PPA modules for local detail enhancement, wherein each C3k2_PPA module uses a parallel patch-aware attention mechanism to weight features of different local regions to enhance the response to high-frequency details in small target areas of the gas meter; the features output by the C3k2_PPA modules are input into the SPPF module for multi-scale context aggregation, and then input into the C2PSA module to further enhance the channel and spatial table recognition of key areas. The enhanced multi-scale features are input into multiple C3k2 modules to form feature maps of different levels, and each level of feature map is input into the detection head. The small target detection layer set in the detection head predicts the feature maps of different levels and outputs candidate detection boxes for small targets, including at least the gas meter reading area, the last digit, and the scale, as well as their category confidence. During the training phase, the adaptive threshold focus loss function is used to perform weighted optimization on the classification branch, and together with the localization loss, the network parameters are updated in reverse so that the network assigns higher weights to small target samples that are difficult to detect during the training process, thereby improving the detection accuracy of small targets on the gas meter.

[0087] In the reading inference stage, the gas meter image is input into the trained YOLOv11 small target detection model for forward inference to obtain the bounding box parameters and category information of each detected target. The bounding box parameters and category information are then combined and sorted from left to right according to the horizontal coordinate of the bounding box center, resulting in an ordered target sequence and its corresponding category sequence. Based on the category sequence and the ordered target sequence, the reading structure is analyzed: when a triangular indicator is detected, its position is recorded as an indicator reference; when a scale line target is detected for the first time, it is used as a reading separator. The numeric characters and decimal point characters appearing on the left side of the separator are concatenated sequentially to form a high-order reading string, and the first numeric character appearing on the right side of the separator is determined as the last digit. Simultaneously, the scale line edges on the left and right sides of the separator are collected. The boundary frame forms a left scale set and a right scale set; the decimal point mark in the high-digit reading string is converted into a standard decimal point and then into a numerical value; the reference scale line and the triangular indicator scale line are determined in the right scale set: the right scale lines are sorted by width, and the widest scale line is taken as the reference scale line; the right scale lines are sorted by vertical coordinate, and the distance difference between each right scale line and the triangular indicator mark in the vertical direction is calculated. The smallest distance is taken as the triangular indicator scale line, and the serial numbers of the reference scale line and the triangular indicator scale line in the vertical coordinate sorting sequence are determined respectively; the high-digit reading, the thousandths conversion value of the last digit, and the offset of the reference scale line serial number and the triangular indicator scale line serial number are finely adjusted and compensated by a fixed step size and summed to output the final reading result of the gas meter.

[0088] This embodiment's network design is based on an improved YOLOv11 gas meter recognition network. It introduces a parallel patch-based perceptual attention mechanism (C3k2_PPA module) and a small target detection layer, along with an adaptive threshold focus loss function. This design significantly enhances the feature extraction and localization capabilities for high-detail targets such as fine scale lines, numerical characters, and triangular indicator marks on the gas meter. The adaptive threshold focus loss function dynamically adjusts the weights of easy and difficult samples during training, effectively alleviating the gradient imbalance problem in small target detection. Combined with a reading correction algorithm, the gas meter reading accuracy is ultimately improved to four decimal places, meeting the application requirements of high-precision metering scenarios.

[0089] 3.3 Pressure gauge pointer scale segmentation and recognition network

[0090] like Figure 6As shown, the pressure gauge pointer and scale segmentation and recognition network based on the Deeplabv3+ segmentation model includes: an encoder with an ASPP module, and a decoder with an Edge Head module and a CBAM module. The CBAM module is repeatedly introduced in multiple stages to enhance the network's attention learning of spatial and channel features. In each CBAM module, the spatial attention mechanism focuses on regions in the image, while the channel attention mechanism helps the network concentrate on more important feature channels, thereby improving the perception ability of the pressure gauge scale and pointer. The Edge Head module is used specifically to process edge information in the image. It extracts edge features in the image through convolution operations, further improving the segmentation accuracy of the pressure gauge pointer and scale edges. Through this module, the model can accurately identify the details of the pressure gauge in complex backgrounds, especially when the pressure gauge pointer is small and has low contrast with the background. In this network architecture, the combination of the CBAM module and the Edge Head module effectively improves the segmentation and recognition accuracy of the pressure gauge pointer and scale, ensuring that even in complex backgrounds, the model can accurately extract the key features of the pressure gauge from the image.

[0091] In specific implementation, the pressure gauge pointer scale segmentation and recognition network processes the following: The input feature map is fed into the encoding end of the Deeplabv3+ segmentation model to extract multi-level feature maps; then, the multi-level feature maps are input into the ASPP module for multi-scale feature extraction. The ASPP module includes a 1×1 convolution branch, a 3×3 dilated convolution branch with a dilation rate of 6, a 3×3 dilated convolution branch with a dilation rate of 12, a 3×3 dilated convolution branch with a dilation rate of 18, and an image pooling branch. The output features of each branch are concatenated to obtain the ASPP fusion feature map, and then channelized through convolutional layers. The process involves several steps: First, the integrated ASPP fused feature map is input to the first CBAM module for attention enhancement, adaptively enhancing the feature responses of the slender pointer structure and scale region. The attention-enhanced feature map is then input to the decoder, where upsampling is used to progressively restore the spatial resolution of the feature map. The upsampled feature map is then concatenated with the corresponding low-level feature map from the encoder to achieve multi-scale feature fusion. Next, the fused feature map is input to the second CBAM module, which outputs the segmentation feature map of the main segmentation branch. Simultaneously, the decoder is connected in parallel to an Edge Head branch, including a third CBAM module. This Edge Head branch receives the feature map input shared with the main segmentation branch, extracts edge features through convolutional stacking, and outputs an edge prediction map. Finally, the segmentation feature map output from the main segmentation branch is fused with the edge prediction map output from the Edge Head branch to obtain the final segmentation prediction map of the pressure gauge pointer and scale regions, serving as the reading recognition result for the pressure gauge image.

[0092] Based on the segmented output, the process proceeds to the feature extraction and reading inference stage. First, connected component analysis is performed using the scale area mask to automatically extract the scale ROI. Second, OTSU thresholding and morphological processing are performed on the scale ROI to obtain clear scale digit regions, which are then input into the Swin-Tiny classification network to complete scale value recognition. Next, based on the centroid set of the scale region, the RANSAC elliptic fitting method is used to estimate the dial center position, major and minor axis parameters, and rotation angle to construct the dial coordinate system. Then, combined with the pointer area mask, the skeleton thinning method is used to extract the pointer centerline, and the pointer direction is fitted using the LSC voting method to automatically determine the pointer tip position and calculate the pointer angle parameters. Finally, the pressure gauge reading is calculated based on the pointer angle parameters and the scale mapping relationship, and the pressure value result is output.

[0093] This embodiment employs an improved DeepLabv3+ network, embedding CBAM attention mechanisms at multiple points after the ASPP module and at the decoding end, and adding an independent Edge Head branch. This multi-stage CBAM module enables the network to adaptively focus on the slender structure of the pointer and the scale area, enhancing the perception of key details and segmentation accuracy. The Edge Head branch specifically extracts and strengthens edge features, effectively improving the segmentation clarity of the pointer and scale boundaries, maintaining stable segmentation performance even in low-contrast or complex backgrounds. Through post-processing of the segmentation mask and geometric analysis, accurate calculation of the pointer angle is achieved, thereby completing highly reliable pressure gauge reading recognition.

[0094] 3.4 Water Meter Key Point Identification Network

[0095] The processing steps of the water meter key point identification network include:

[0096] The water meter image is input into the small target detection network based on the improved YOLOv11 for inference, and the detection boxes of the water meter integer digit characters and key structures of the small dial are obtained. All detection boxes are sorted from left to right according to the center horizontal coordinate to form an ordered detection sequence and a corresponding category sequence.

[0097] Structured parsing is performed based on the category sequence: Detection results that can be parsed into numbers are concatenated sequentially to form an integer part string; simultaneously, detection box sets for key points of the small dial disc, zero-position reference point, and pointer endpoint are extracted and summarized; the key points of the small dial disc are sorted from left to right according to their horizontal coordinates, and using the bounding region corresponding to each small dial disc detection box as a constraint, target points falling within the bounding region are retrieved from the set of pointer endpoint key points and the set of zero-position reference point key points, forming a one-to-one correspondence between the disc, pointer endpoint, zero-position reference point, and key point. Key point group; For each key point group, perform small dial reading calculation: Construct a vector from the center of the disc to the pointer endpoint and a vector from the center to the zero reference point, respectively. Calculate the angle between the two vectors, and based on the relative positional relationship between the pointer endpoint and the zero reference point on both sides of the center, convert the angle into a circumferential ratio value from 0 to 1; Convert the circumferential ratio value according to a preset range weight and accumulate it to form a decimal reading, wherein the conversion result of each small dial is rounded to a preset decimal place; Add the integer part reading to the decimal part reading to obtain the final water meter reading and output it.

[0098] This embodiment utilizes an improved YOLOv11 small target detection framework for network reuse, combining key point detection and geometric relationship analysis to achieve water meter readings. By adopting the small target detection capability from the gas meter recognition network, it ensures stable detection of multiple discs, pointer endpoints, and zero reference points on the water meter dial. Through key point spatial association and vector angle calculation, it achieves geometric reading analysis of multi-pointer small dials, avoiding reliance on complex segmentation or template matching. This method separates and then merges the integer part (digital characters) and the fractional part (pointer angle), adapting to the multi-range, multi-pointer reading structure of water meters, balancing recognition accuracy and system real-time performance.

[0099] 3.5. Electricity Meter Digital Display Character Recognition Network

[0100] like Figure 7 As shown, the meter digital display character recognition network based on CRNN network and combined with MobileNetV4 lightweight network and CBAM attention mechanism includes: designing convolutional layers of MobileNetV4 lightweight network, designing recurrent layers of CBAM attention mechanism and transcription layer.

[0101] Specifically, such as Figure 8As shown, the processing procedure of the meter digital display character recognition network is as follows: The input feature map is subjected to convolutional feature extraction through a lightweight MobileNetV4 network to obtain a convolutional feature map; then, the convolutional feature map is serialized, and the two-dimensional feature map is unfolded along the width direction into a time series S; a CBAM attention mechanism is added to the time series S to generate an enhanced context vector; next, the enhanced context vector is fed as input into an LSTM bidirectional long short-term memory network for encoding to further extract temporal information; subsequently, the feature sequence output by the recurrent layer is decoded through a transcription layer; finally, the connection-time classification loss function is used to predict the digital sequence, generating a digital string as the meter reading result.

[0102] This embodiment constructs a character recognition network for electricity meters based on an improved CRNN architecture combined with a lightweight MobileNetV4 network and a CBAM attention mechanism. MobileNetV4 serves as a lightweight convolutional feature extractor, significantly reducing computational complexity and memory usage, making it suitable for resource-constrained embedded deployments. The CBAM attention mechanism is embedded in the recurrent layer, enhancing the model's ability to focus on key features in character sequences and improving the robustness of recognizing complex characters or ambiguous regions. While maintaining the advantages of CRNN in processing sequence data, this network further optimizes recognition accuracy and inference efficiency, making it suitable for real-time character recognition of digital display instruments such as electricity meters.

[0103] 4. Training of Instrument Image Classification and Reading Recognition Model

[0104] In training the instrument image classification and reading recognition model, this embodiment uses the following method:

[0105] Multi-type instrument sample images with pre-annotated instrument types, pointer and scale areas, and digital areas are used as training samples to form a training sample set. These samples are then input into the multi-scale instrument detection network, gas meter recognition network, pressure gauge pointer and scale segmentation recognition network, electricity meter digital display character recognition network, and water meter key point recognition network of the instrument image classification and reading recognition model. Each network is optimized and updated with the objective of minimizing its respective loss function, thereby training the instrument image classification and reading recognition model.

[0106] In specific implementation, the multi-scale instrument detection network adopts a CIOU loss function composed of a CIoU loss function and a CLS loss function, wherein:

[0107] The CIoU loss function is expressed as follows:

[0108] ;

[0109] In the formula, The CIoU loss function; For intersection, union, and comparison; The distance between the center points of the predicted bounding box and the ground truth bounding box is Euclidean distance. The minimum bounding box diagonal length; Aspect ratio consistency metric: These are the weighting coefficients; To predict the bounding box; The true bounding box;

[0110] The CLS loss function is expressed as follows:

[0111] ;

[0112] In the formula, The CLS loss function; Number of categories; True label (one-hot); To predict probabilities;

[0113] The CIOU loss function is then expressed as:

[0114] ;

[0115] In the formula, CIOU loss function; and These are the weighting coefficients used to balance the two types of losses.

[0116] Both the gas meter identification network and the water meter key point identification network adopt an adaptive threshold focus loss function, which is expressed as follows:

[0117] ;

[0118] In the formula, It is an adaptive threshold focus loss function; The weighting factor is adaptive based on the difficulty of the sample. This represents the model's predicted probability for the positive class. The focusing parameter is used to adjust the weights of easy and difficult samples.

[0119] The pressure gauge pointer scale segmentation and recognition network uses the Dice loss function, which is expressed as follows:

[0120] ;

[0121] In the formula, The Dice loss function; For true segmentation labels; To predict probabilities; This is a smoothing term.

[0122] The digital character recognition network for the electricity meter uses the CTC loss function, which is expressed as follows:

[0123] ;

[0124] In the formula, The CTC loss function; For all possible alignment paths; To map to real labels The set of paths; The sequence length; For at any time Output symbols The probability of.

[0125] like Figure 9 As shown, the overall process of the instrument image classification and reading recognition method in this embodiment includes a data acquisition and model training stage, an instrument type discrimination and branch reading stage, and a reading result output stage. Specifically, firstly, instrument images are acquired using a Raspberry Pi camera, and the acquired multi-type instrument images are manually labeled to form a labeled dataset containing instrument category information, gas meter scale lines and small digital targets, pressure gauge pointer and scale area, water meter key points, and electricity meter character area. Then, each sub-network is trained based on the labeled dataset to obtain a multi-scale instrument detection network, a gas meter recognition network, a pressure gauge pointer and scale segmentation recognition network, a water meter key point recognition network, and an electricity meter character recognition network for inference.

[0126] During the inference phase, the input image of the instrument to be identified is first used to detect, locate, and classify the instrument target using an improved YOLOv11 network. Then, based on the classification result, the corresponding reading branch is entered: when identified as a gas meter, the improved YOLOv11 gas meter small target recognition network is used to detect the scale lines and small digital targets, and the reading is corrected based on the detection results to output the gas meter reading; when identified as a pressure gauge, the improved DeepLabv3+ segmentation network is used to perform pixel-level segmentation of the pointer and scale area, and the reading is calculated based on scale ellipse fitting and pointer angle calculation; when identified as a water meter, the YOLOv11 key point detection network is used to detect key points on the small dial, and the reading is calculated using the geometric relationship of the key points and range weights; when identified as an electricity meter, the improved CRNN character recognition network is used to recognize the meter's digital characters and output the meter reading; finally, the reading results from each branch are summarized and output, completing the automatic classification and reading recognition of multiple types of instruments.

[0127] 5. Model Deployment

[0128] In this embodiment, the model structure is optimized by model quantization to reduce the computational complexity of the model, and the model is deployed on a Raspberry Pi to achieve real-time inference.

[0129] The model was deployed on a Raspberry Pi 4B 2GB hardware device, specifically including the following steps: Post-trained quantization (PTQ) was applied in PyTorch, and a trained multi-scale instrument detection network was loaded; fbgemm was used as the quantization configuration; the model was quantized, and a calibration dataset was used to quantize the model; after calibration, the model was converted to a quantized model, and finally the quantized model was saved, thereby reducing the computational complexity of the model to adapt to the limited computing resources of the Raspberry Pi; ONNX was selected as the inference framework, and the quantized model was converted to ONNX inference format; OpenCV library was installed on the Raspberry Pi for image acquisition and processing, and ONNX was installed for model inference; a camera was connected through the Raspberry Pi's CSI interface to acquire real-time instrument images; the acquired instrument images were preprocessed, including cropping, scaling, and normalization; the model deployed on the Raspberry Pi was used for inference to output the instrument's position information, including instrument type, pointer area, or digital area; the recognition results were output to the display screen through the Raspberry Pi's HDMI interface.

[0130] 6. Example

[0131] To further verify the effectiveness of the instrument image classification and reading recognition model proposed in this embodiment, a comparison was made with the model before the improvement. Table 1 shows the comparison of instrument type classification using the multi-scale instrument detection network of this embodiment and the unimproved YOLOv11 model. It can be seen that when using the multi-scale instrument detection network, i.e., the improved YOLOv11 model, to classify instrument types, the classification accuracy reached 99.79%, and the mAP index reached 99.5%, which is higher than the unimproved YOLOv11 model in both accuracy and mAP index. Table 2 shows the comparison of the joint training index of the gas meter recognition network of this embodiment and the gas meter scale and numerical parts of the YOLO series model. The results show that the gas meter recognition network has significantly higher recognition accuracy, recall, and mAP compared with the original model. The accuracy of gas meter recognition is improved to 95.2%, and the mAP of joint training is improved to 99.1%. Table 3 shows the accuracy of gas meter recognition using this embodiment. By comparing the recognition accuracy of different models within the reading error range, it can be clearly seen that the proposed method is superior to YOLOv8, YOLOv11, and YOLOv12 models at all error thresholds. When the error threshold is 0.0002, the accuracy of the proposed method reaches 91.09%; when the error thresholds are 0.0001 and 0, the accuracy reaches 85.97% and 83.01% respectively, both of which are superior to the comparison models. Table 4 shows the comparison effect of gas meter recognition using this embodiment with other models. The results show that the gas meter recognition network outperforms the RT-DETR model and Fast model in terms of recognition accuracy, recall, and mAP. Both the R-CNN and RetinaNet models show significant improvements. Table 5 compares the evaluation metrics of different semantic segmentation datasets used in this embodiment. It can be seen that the improved Deeplabv3+ segmentation model, which uses the pressure gauge pointer and scale segmentation recognition network, achieves a segmentation accuracy of 98.59% on the task of segmenting the pointer and scale regions of the pressure gauge, which is an increase in accuracy compared to the unimproved Deeplabv3+ segmentation model. Table 6 compares the reading recognition of the meter character recognition network used in this embodiment with the unimproved CRNN network model. It can be seen that the improved CRNN network model performs well on the meter character recognition task, achieving an accuracy of 97.5%, which is an increase in accuracy compared to the unimproved CRNN network model.

[0132] Table 1. Comparison of instrument type classification between multi-scale instrument detection networks and the unmodified YOLOv11 model.

[0133] Swin Transformer BiFPN CBAM P / % mAP / % × × × 99.64 99.3 √ × × 99.75 99.4 √ √ × 99.68 99.4 √ √ √ 99.79 99.5

[0134] Table 2 Comparison of Joint Training Indicators for the Scale and Numerical Sections of the Gas Gauge

[0135] Model P / % R / % F1 / % mAP / % YOLOv8 96.6 96.9 96.75 98.4 YOLOv11 95.3 97.6 96.44 98.5 YOLOv12 97.3 97.2 97.25 98.8 This article's method 97.6 99 98.3 99.1

[0136] Table 3 Gas meter recognition accuracy

[0137] Model P / % error (0.0002) P / % error (0.0001) P / % error (0) YOLOv8 88.51 83.54 76.08 YOLOv11 89.3 84.33 78.27 YOLOv12 89.67 85.62 80.13 This article's method 91.09 85.97 83.01

[0138] Table 4 Gas Meter Identification Comparison

[0139] Model P / % R / % F1 / % mAP / % RT-DETR 76.4 64.7 69.97 64.7 Fast R-CNN 93.52 67.78 78.59 63.44 RetinaNet 71.62 66.2 68.8 63.4 This article's method 97.6 99 98.3 99.1

[0140] Table 5. Comparison of Evaluation Metrics for Different Semantic Segmentation Datasets

[0141] Net Name mIou mPA mAcc Unet 88.67 95.87 99.04 Pspnet 84.9 99.35 98.67 Swin-Unet 89.08 96.92 99.12 Deeplabv3+ 92.94 97.43 99.46 Improved Deeplabv3+ 92.62 98.59 99.46

[0142] Table 6 CRNN Ablation Experiment

[0143] MobileNet v4 CBAM ACC / % CER / % × × 94.5 5.3 √ × 96.2 4.0 √ √ 97.5 3.1

[0144] 7. Overview

[0145] This embodiment can cover a variety of common meter types such as water meters, electricity meters, gas meters, and pressure gauges, and has strong versatility, eliminating the need to develop separate solutions for each type of meter. The automated meter reading technology based on image recognition does not require modification of existing meter equipment or the addition of sensors; it only needs to collect images through a camera to achieve readings, reducing deployment costs and complexity. Through automated meter reading technology, enterprises can monitor and manage energy consumption data in real time, optimize energy use, reduce carbon emissions, and help achieve the goal of "carbon peaking and carbon neutrality." It replaces the traditional manual meter reading method, reducing labor costs and errors caused by manual reading, improving work efficiency, and avoiding the safety risks of manual operation in harsh environments.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described with reference to preferred embodiments, those skilled in the art should understand that various changes in form and detail can be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A method for classifying and recognizing multi-type instrument images based on deep learning, characterized in that, Includes the following steps: S1. Acquire images of multiple types of meters, including images of gas meters, pressure gauges, water meters, and electricity meters; S2. Design a multi-scale instrument detection network based on the YOLOv11 model architecture and combined with the Swin Transformer module; a gas meter recognition network based on the YOLOv11 model architecture for small target recognition; a pressure gauge pointer scale segmentation and recognition network based on the Deeplabv3+ segmentation model; a water meter key point recognition network based on the YOLOv11 model architecture and using key point recognition; and an electricity meter digital display character recognition network based on the CRNN network and combined with the MobileNetV4 lightweight network and CBAM attention mechanism. S3. Based on the multi-scale instrument detection network, gas meter recognition network, pressure gauge pointer scale segmentation recognition network, water meter key point recognition network, and electricity meter digital display character recognition network, construct an instrument image classification and reading recognition model; S4. The collected multi-type instrument images are labeled and preprocessed as the training set of the instrument image classification and reading recognition model, and then input into the instrument image classification and reading recognition model for training to obtain the trained instrument image classification and reading recognition model. S5. Input the multi-type instrument images to be identified into the trained instrument image classification and reading recognition model, and output the corresponding instrument reading recognition results of the multi-type instrument images to be identified.

2. The method for classifying and recognizing multi-type instrument images based on deep learning according to claim 1, characterized in that, The instrument image classification and reading recognition model takes the input multi-type instrument images as input to a multi-scale instrument detection network. Using the YOLOv11 model architecture and the Swing Transformer module, it extracts multi-scale global features from the multi-type instrument images, classifying them into gas meter images, pressure gauge images, water meter images, and electricity meter images. For gas meter images, a gas meter recognition network is used to identify scale lines and small numerical targets. A reading correction algorithm is then used to fine-tune the readings, resulting in the gas meter image reading recognition result. For pressure gauge images, a Deeplabv3+ segmentation model is used to perform pixel-level semantic segmentation of the pointer and scale regions, calculating the angular relationship between them to obtain the pressure gauge image reading recognition result. For images classified as water meters, key point reading recognition is used and geometric relationship analysis is performed to obtain the reading recognition results of the water meter images; For images classified as electricity meters, an improved CRNN network is used for character recognition to obtain the reading recognition results of the electricity meter images, thereby obtaining the corresponding meter reading recognition results for multiple types of meter images.

3. The method for classifying and recognizing multi-type instrument images based on deep learning according to claim 1, characterized in that, The multi-scale instrument detection network based on the YOLOv11 model architecture and combined with the Swing Transformer module includes: designing the backbone network of the Swing Transformer module, designing the CBAM attention mechanism and the neck network of the weighted bidirectional feature pyramid network BiFPN, and designing the detection head with training loss. The backbone network includes a Win Transformer module, an SPPF module, a C2PSA module, and three cascaded convolutional layers connected in sequence; wherein the three cascaded convolutional layers include a first convolutional layer, a second convolutional layer, and a third convolutional layer. The neck network includes two upsampled feature fusion pyramid modules, three convolutional feature fusion modules, and a CBAM module connected in sequence; wherein, the upsampled feature fusion pyramid module includes cascaded upsampled units, a BiFPN module, and a C2f module, and the convolutional feature fusion module includes cascaded convolutional layers, a BiFPN module, and a C2f module; The detection head includes a parallel cross-union loss module and a classification loss module; wherein, the cross-union loss module includes two cascaded convolutional layers, a two-dimensional convolutional layer and a CIoU loss function, and the classification loss module includes two cascaded depthwise separable convolutional units, a two-dimensional convolutional layer and a CLS loss function. In the multi-scale instrument detection network, the input of the instrument image classification and reading recognition model serves as the input of the backbone network. The output of the first convolutional layer in the backbone network also serves as the input of the BiFPN module in the second upsampled feature fusion pyramid module of the neck network. The output of the second convolutional layer also serves as the input of the BiFPN module in the first upsampled feature fusion pyramid module of the neck network. The output of the third convolutional layer also serves as the input of the upsampling unit in the first upsampled feature fusion pyramid module of the neck network and the BiFPN module in the third convolutional feature fusion module, respectively. The outputs of the first convolutional feature fusion module, the second convolutional feature fusion module, and the CBAM module in the neck network also serve as the inputs of the detection head, respectively.

4. The method for classifying and recognizing multi-type instrument images based on deep learning according to claim 1, characterized in that, The gas meter identification network based on the YOLOv11 model architecture for small target recognition includes: a backbone network integrating a parallel patch perception attention mechanism in the C3k2 module, and a detection head with adaptive threshold focus loss and a small target detection layer. The backbone network includes a convolutional layer, four C3k2_PPA units, an SPPF module, and a C2PSA module connected in sequence; wherein, the C3k2_PPA unit includes a cascaded C3k2_PPA module and a convolutional layer; The neck network includes three upsampling C3k2 modules and three convolutional C3k2 modules connected in sequence; wherein, the upsampling C3k2 module includes a cascaded upsampling layer, a feature fusion layer and a C3k2 module, and the convolutional C3k2 module includes a cascaded convolutional layer, a feature fusion layer and a C3k2 module; The detection head is used to obtain the reading recognition result of the gas meter image based on the output of the neck network and the small target detection layer, which is then used as the output of the gas meter recognition network. In the gas meter recognition network, the input gas meter image is used as the input to the backbone network. The outputs of the C3k2_PPA modules in the first and second C3k2_PPA units of the backbone network are also used as the inputs to the first and second upsampling C3k2 modules in the neck network, respectively. The output of the C2PSA module is also used as the input to the feature fusion layer in the third convolutional C3k2 module of the neck network. The outputs of the first and second upsampling C3k2 modules in the neck network are also used as the inputs to the first and second convolutional C3k2 modules, respectively. The outputs of the third upsampling C3k2 module and the first and second convolutional C3k2 modules are also used as the inputs to the detection head.

5. The method for classifying and recognizing multi-type instrument images based on deep learning according to claim 1, characterized in that, The pressure gauge pointer scale segmentation and recognition network based on the Deeplabv3+ segmentation model includes: designing an encoder for the ASPP module, and designing decoders for the Edge Head module and the CBAM module; The processing steps of the pressure gauge pointer scale segmentation and recognition network include: The input feature map is fed into the encoder of the Deeplabv3+ segmentation model to extract multi-level feature maps. Then, the multi-level feature maps are fed into the ASPP module for multi-scale feature extraction. The ASPP module includes a 1×1 convolution branch, a 3×3 dilated convolution branch with a dilation rate of 6, a 3×3 dilated convolution branch with a dilation rate of 12, a 3×3 dilated convolution branch with a dilation rate of 18, and an image pooling branch. The output features of each branch are concatenated to obtain an ASPP fusion feature map, which is then channel-integrated through a convolutional layer. Next, the integrated ASPP fusion feature map is fed into the first CBAM module for attention enhancement, adaptively enhancing the feature responses of the thin pointer structure and the scale region. The attention-enhanced feature map is then fed into the decoder, where upsampling is used to progressively restore the spatial resolution of the feature map. The upsampled feature map is then concatenated with the corresponding low-level feature map from the encoder to achieve multi-scale feature fusion. The fused feature map is then fed into the second CBAM module, which outputs the segmentation feature map of the main segmentation branch. Simultaneously, the decoder is connected in parallel with an Edge module including a third CBAM module. The Edge Head branch receives the feature map input shared with the main segmentation branch, extracts edge features through convolution stacking, and outputs an edge prediction map. Finally, the segmentation feature map output by the main segmentation branch is fused with the edge prediction map output by the Edge Head to obtain the final segmentation prediction map of the pressure gauge pointer area and scale area, which serves as the reading recognition result of the pressure gauge image.

6. The method for classifying and recognizing multi-type instrument images based on deep learning according to claim 1, characterized in that, The processing steps of the water meter key point identification network include: The water meter image is input into a small target detection network based on an improved YOLOv11 for inference, resulting in bounding boxes containing digits, small dials, pointer endpoints, and a zero reference point. These bounding boxes are then sorted by their horizontal coordinates to form an ordered sequence. Based on this sequence, the digits are extracted and concatenated to form the integer part of the reading. Simultaneously, the dial, pointer endpoint, and zero reference point are associated with multiple keypoint groups based on their spatial locations. For each keypoint group, two vectors are constructed with the center of the dial as the center, pointing to the pointer endpoint and the zero reference point. The angle between these vectors is calculated and converted into a circumferential ratio. According to the range weights corresponding to each small dial, the ratio is converted into the decimal part of the reading. Finally, the integer and decimal parts are added together to output the complete water meter reading, which serves as the reading recognition result of the water meter image.

7. The method for classifying and recognizing multi-type instrument images based on deep learning according to claim 1, characterized in that, The meter digital display character recognition network based on CRNN network combined with MobileNetV4 lightweight network and CBAM attention mechanism includes: designing convolutional layers of MobileNetV4 lightweight network, recurrent layers of CBAM attention mechanism and transcription layer.

8. The method for classifying and recognizing multi-type instrument images based on deep learning according to claim 7, characterized in that, The processing steps of the digital character recognition network for electricity meters include: The input feature map is processed by a lightweight MobileNetV4 network to extract convolutional features, resulting in a convolutional feature map. Then, the convolutional feature map is serialized, unfolding the two-dimensional feature map along its width into a time series S. A CBAM attention mechanism is added to the time series S to generate an enhanced context vector. Next, the enhanced context vector is fed into an LSTM bidirectional long short-term memory network for encoding, further extracting temporal information. Subsequently, the feature sequence output from the recurrent layer is decoded by a transcription layer. Finally, a connection-time classification loss function is used to predict the numerical sequence, generating a numerical string representing the instrument reading result.

9. The method for classifying and recognizing multi-type instrument images based on deep learning according to claim 1, characterized in that, The instrument image classification and reading recognition model is trained in the following way: Multi-type instrument sample images with pre-annotated instrument types, pointer and scale areas, and digital areas are used as training samples to form a training sample set. These samples are then input into the multi-scale instrument detection network, gas meter recognition network, pressure gauge pointer and scale segmentation recognition network, electricity meter digital display character recognition network, and water meter key point recognition network of the instrument image classification and reading recognition model. Each network is optimized and updated with the objective of minimizing its respective loss function, thereby training the instrument image classification and reading recognition model.

10. The method for classifying and recognizing multi-type instrument images based on deep learning according to claim 9, characterized in that, The multi-scale instrument detection network adopts the CIOU loss function, which is composed of the CIoU loss function and the CLS loss function. The gas meter identification network and the water meter key point identification network both adopt the adaptive threshold focus loss function. The pressure gauge pointer scale segmentation identification network adopts the Dice loss function. The electricity meter digital display character identification network adopts the CTC loss function.

Images