Deep learning-based pointer type instrument reading identification method
By improving the YOLOv8 and CRNN models and combining the U2net model, the problem of insufficient detection accuracy under the complex background of automatic reading of pointer instruments is solved, and efficient and accurate instrument reading is achieved to meet the needs of complex industrial environments.
Patent Information
- Application Number
- CN202411172825.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-29
AI Technical Summary
The prior art has problems such as time-consuming and labor-consuming, effective, and poor real-time performance in the automatic reading of pointer instruments, especially in complex contexts.
Using a deep learning-based method, the improved YOLOv8 model and CRNN model were constructed. By replacing the C2f module of the Backbone part as the FasterC2f module, the EMA attention mechanism was introduced, the Slim-Neck structure and Dyhead detection head were used, and CBAM was added to the CRNN model. Attention mechanism, combined with the U2net model for pointer and scale segmentation to achieve final reading of the instrument.
It significantly improves the detection accuracy and robustness of pointer instruments, and can achieve efficient and accurate instrument reading in complex industrial environments, meeting the real-time and accuracy requirements of industrial applications.
Smart Images

Figure CN119049026A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method for identifying numbers on a pointer-type instrument, and in particular to a method for identifying numbers on a pointer-type instrument based on deep learning. Background Art
[0002] In the fields of industrial production, energy management, etc., accurate and timely reading of instrument data is crucial to ensure production safety, improve production efficiency, and optimize resource utilization. Pointer instruments are one of the most widely used instrument types. The reading of their indications is traditionally done manually, which is time-consuming and labor-intensive, the accuracy is affected by human subjective factors, and the real-time performance is poor. How to realize the automatic reading of the indications of pointer instruments is one of the important research directions in the current field of industrial automation.
[0003] Pointer instrument reading recognition mainly involves three aspects, including instrument positioning (recognition), range recognition, and instrument segmentation reading. In terms of instrument positioning, the YOLO model is traditionally used, which can simultaneously predict multiple bounding boxes and category probabilities in a single forward propagation, suitable for real-time detection tasks. However, the standard YOLO model cannot provide sufficient detection accuracy when facing complex backgrounds and small targets.
[0004] In terms of range recognition, traditional optical character recognition (OCR) technology has limitations when dealing with complex backgrounds and deformed text. In recent years, text recognition methods based on deep learning have made significant progress. Among them, the CRNN model can achieve end-to-end text recognition. It combines the feature extraction ability of convolutional neural networks and the sequence modeling ability of recurrent neural networks, and can effectively process text sequences of indefinite length. However, the traditional CRNN model has the problem of insufficient recognition accuracy when dealing with complex backgrounds. Summary of the invention
[0005] Purpose of the invention: The purpose of the present invention is to provide a pointer instrument indication number recognition method based on deep learning that has high accuracy and robustness and can adapt to complex industrial environments.
[0006] Technical solution: The method for identifying the number indicated by a pointer-type instrument based on deep learning of the present invention comprises:
[0007] (1) Construct instrument detection data set and range identification data set;
[0008] (2) Build an improved YOLOv8 model, including:
[0009] Replace the C2f module in the Backbone part with the FasterC2f module. The FasterC2f module is formed by replacing the Bottleneck module in the C2f module with the FasterBlock module. Add the EMA attention mechanism module after the SPPF layer in the Backbone part.
[0010] Replace the first three concat modules in the Neck part with TGC modules, introduce the Slim-Neck structure in the Neck part, and the Slim-Neck structure includes GSConv and VoV-GSCSP. Replace the Conv in the Neck part with GSConv, replace the first three C2f modules in the Neck part with VoV-GSCSP, and replace the last three C2f modules in the Neck part with FasterC2f modules; among the three replaced FasterC2f modules, add the EMA attention mechanism module after the first two FasterC2f modules;
[0011] In the Head part, use the Dyhead detection head and add a 160×160 small target detection layer;
[0012] (3) Building an improved CRNN model, including: using ResNet18 as the backbone network and adding the CBAM attention mechanism;
[0013] (4) The improved YOLOv8 model is trained using the instrument detection dataset, and the improved CRNN model is trained using the range recognition dataset;
[0014] (5) The trained improved YOLOv8 model is used to detect the input image, and the detection results are respectively input into the trained improved CRNN model and the pre-trained U2net model. The trained improved CRNN model identifies the instrument range, and the pre-trained U2net model performs pointer and scale segmentation. The final reading of the instrument is completed by combining the identified instrument start range scale value and end range scale value.
[0015] Furthermore, step (1) includes: collecting pictures containing pointer instruments, annotating the pictures, wherein the annotation contents include meter, max, and min, which respectively represent the instrument dial area, the instrument range maximum value area, and the instrument range minimum value area, and after the annotation is completed, an instrument detection data set is formed; and a range recognition data set is formed by selecting the max and min areas of the pictures in the instrument detection data set.
[0016] Furthermore, data enhancement is performed when constructing the instrument detection dataset, including random scaling, cropping, arrangement, color jittering, and flipping operations.
[0017] Furthermore, GSConv includes standard convolution, depthwise separable convolution DWConv and shuffle operation. The features of different levels obtained by standard convolution and depthwise separable convolution are concat-joined, and then the information from standard convolution is completely mixed into the output of depthwise separable convolution through shuffle operation.
[0018] VoV-GSCSP includes cross-stage operations like residuals, uses two branches to extract features, and aggregates the feature maps of the branches combined with cross-stage operations like residuals.
[0019] Furthermore, the TGC module has three feature maps of different sizes as input, namely C / 2×C / 2, C×C and 2C×2C; the feature map of C / 2×C / 2 is upsampled by nearest neighbor interpolation to obtain a feature map of size C×C; the feature map of C×C remains unchanged; the feature map of 2C×2C is subjected to adaptive maximum pooling and adaptive average pooling operations to obtain a feature map of size C×C; finally, the three C×C feature maps are connected as the output of the TGC module.
[0020] Furthermore, the Backbone part of the YOLOv8 model includes Conv-1, Conv-2, Conv-3, Conv-4, Conv-5, FasterC2f-1, FasterC2f-2, FasterC2f-3, FasterC2f-4, SPPF and EMA-1; the Neck part includes TGC-1, TGC-2, TGC-3, GSConv-1, GSConv-2, GSConv-3, GSConv-4, VoVGSCSP-1, VoVGSCSP-2, VoVGSCSP-3, Concat-1, Concat-2, Concat-3, FasterC2f-5, FasterC2f-6, FasterC2f-7, EMA-2, EMA-3; the Head part includes Dyhead160×160, Dyhead80×80, Dyhead40×40, Dyhead20×20;
[0021] The output of Conv-1 is passed to Conv-2, and the output of Conv-2 is divided into two paths, which are passed to FasterC2f-1 and TGC-3 respectively; the output of FasterC2f-1 is divided into three paths, which are passed to Conv-3, TGC-3 and TGC-2 respectively; the output of Conv-3 is passed to FasterC2f-2, and the output of FasterC2f-2 is divided into three paths, which are passed to Conv-4, TGC-2 and TGC-1 respectively; the output of Conv-4 is passed to FasterC2f-3, and the output of FasterC2f-3 is divided into two paths, which are passed to Conv-5 and TGC-1 respectively; the output of Conv-5 is passed to FasterC2f-4, and the output of FasterC2f-4 is passed to SPPF, and the output of SPPF is passed to EMA-1; the output of EMA-1 is divided into two paths, which are passed to GSConv-1 and Concat-3 respectively;
[0022] The output of GSConv-1 is passed to TGC-1, the output of TGC-1 is passed to VoVGSCSP-1, the output of VoVGSCSP-1 is divided into two paths, passed to GSConv-2 and Concat-2 respectively; the output of GSConv-2 is passed to TGC-2, the output of TGC-2 is passed to VoVGSCSP-2, the output of VoVGSCSP-2 is divided into two paths, passed to GSConv-3 and Concat-1 respectively; the output of GSConv-3 is passed to TGC-3, the output of TGC-3 is passed to VoVGSCSP-3, the output of VoVGSCSP-3 is divided into two paths, passed to Dyhead160×160 and GSConv-4 respectively; the output of GSConv-4 Passed to Concat-1, the output of Concat-1 is passed to FasterC2f-5, the output of FasterC2f-5 is passed to EMA-2, the output of EMA-2 is divided into two paths, passed to Dyhead80×80 and GSConv-5 respectively; the output of GSConv-5 is passed to Concat-2, the output of Concat-2 is passed to FasterC2f-6, the output of FasterC2f-6 is passed to EMA-3, the output of EMA-3 is divided into two paths, passed to Dyhead40×40 and Concat-3 respectively; the output of Concat-3 is passed to FasterC2f-7, and the output of FasterC2f-7 is passed to Dyhead20×20.
[0023] Furthermore, in step (4), the improved YOLOv8 model uses the SGD optimizer to iteratively update the parameters; the improved CRNN model uses the Adam optimizer to update the parameters and adopts CTCloss as the loss function.
[0024] Furthermore, in step (5), the detection results of the improved YOLOv8 model include the bounding box coordinates and confidence scores of the instrument area, the maximum range value area max, and the minimum range value area min.
[0025] Furthermore, the non-maximum suppression (NMS) algorithm is used to filter overlapping detection frames and retain the detection frame with the highest confidence.
[0026] Furthermore, in step (5), the distance method is used to calculate the final instrument reading, including: calculating the Euclidean distance between the pointer endpoint and each scale point, selecting the scale with the smallest distance as the reading indicated by the pointer; mapping the reading to the actual value according to the range information to obtain the final instrument reading result.
[0027] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:
[0028] (1) In the improved YOLOv8 model, the original C2f module is replaced by the FasterC2f module, which can significantly improve the detection performance; the EMA attention mechanism module is added to the backbone network, the Dyhead detection head is used in the detection head part, and the small target detection layer is added, which can enhance the performance of complex background and small target detection; in the neck structure, the efficient fusion of the three input channels is achieved through the Slim-Neck structure and the TGC module, which further improves the detection accuracy and robustness of the model.
[0029] (2) In the improved CRNN model, ResNet18 is used as the backbone network combined with the CBAM attention mechanism to improve the accuracy and robustness of range recognition.
[0030] In summary, the present invention can realize efficient and accurate reading of the numbers indicated by pointer-type instruments, thereby meeting the high requirements for real-time performance and accuracy in industrial applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 It is a flowchart of a method for identifying the number indicated on a pointer-type instrument based on deep learning provided by an embodiment of the present invention;
[0032] Figure 2 Schematic diagram of the improved YOLOv8 model structure in an embodiment of the present invention;
[0033] Figure 3 is a schematic diagram of the FasterC2f module structure in an embodiment of the present invention;
[0034] Figure 4 Schematic diagram of the structure of the EMA attention mechanism module in an embodiment of the present invention;
[0035] Figure 5Schematic diagram of the Slim-Neck structure in an embodiment of the present invention;
[0036] Figure 6 is a schematic diagram of the structure of a TGC module in an embodiment of the present invention;
[0037] Figure 7 Schematic diagram of the structure of four detection heads in an embodiment of the present invention;
[0038] Figure 8 Schematic diagram of the structure of the Dyhead detection head in an embodiment of the present invention;
[0039] Fig. 9 1 is a comparison chart of the detection results of the improved YOLOv8 model and the original model in an embodiment of the present invention;
[0040] Fig.10 is a comparison chart of the recognition accuracy of the improved CRNN model and the original model in the embodiment of the present invention;
[0041] Fig.11 is a target detection graph in an embodiment of the present invention;
[0042] Fig.12 is a diagram of instrument segmentation in an embodiment of the present invention;
[0043] Fig.13 It is an instrument segmentation conversion diagram in an embodiment of the present invention. DETAILED DESCRIPTION
[0044] The present invention will be further described below in conjunction with the accompanying drawings.
[0045] like Figure 1 As shown, an embodiment of the present invention provides a method for identifying the number indicated on a pointer-type instrument based on deep learning, which specifically includes the following steps:
[0046] (1) Construct instrument detection data set and range identification data set;
[0047] Collect pictures containing pointer instruments. The pictures can be collected on-site in the factory or crawled from the Internet. After the pictures are collected, they need to be screened, including removing duplicate pictures and pictures with problems such as instrument occlusion and difficulty in distinguishing. Then, perform data enhancement on the pictures, including random scaling, cropping, arrangement, color jitter, flipping and other operations. Next, annotate the pictures, and the annotation content includes meter, max, and min, which represent the instrument dial area, the maximum value area of the instrument range, and the minimum value area of the instrument range, respectively. After the annotation is completed, the instrument detection data set is formed, and the data set is divided into a training set, a validation set, and a test set.
[0048] The range recognition dataset is formed by selecting the max and min areas of the images in the instrument detection dataset.
[0049] (2) Build an improved YOLOv8 model, including:
[0050] Replace the C2f module in the Backbone part with the FasterC2f module. The FasterC2f module is formed by replacing the Bottleneck module in the C2f module with the FasterBlock module. Add the EMA attention mechanism module after the SPPF layer in the Backbone part.
[0051] Replace the first three concat modules of the Neck part with TGC modules, introduce the Slim-Neck structure in the Neck part, the Slim-Neck structure includes GSConv and VoV-GSCSP, replace the Conv of the Neck part with GSConv, replace the first three C2f modules of the Neck part with VoV-GSCSP, and replace the last three C2f modules of the Neck part with FasterC2f modules; among the three replaced FasterC2f modules, add the EMA attention mechanism module after the first two FasterC2f modules;
[0052] In the Head part, use the Dyhead detection head and add a 160×160 small target detection layer.
[0053] like Figure 3 As shown in Figure 1, in the FasterC2f module, the replaced FasterBlock module adds two PWConv layers after PConv to form an inverted residual block, expands the number of channels in the middle layer, and includes a shortcut to reuse input features. The PConv layer in the FasterBlock module performs traditional convolution on only some of the input channels to extract spatial features, while other channels remain unchanged.
[0054] like Figure 4 As shown in the figure, the EMA attention mechanism module includes multi-scale parallel sub-networks for establishing short-term and long-term dependencies. Along the channel dimension, two one-dimensional feature codes are concatenated and then connected to a 1×1 convolution with shared weights. After the output is decomposed into two vectors, two nonlinear Sigmoid functions are used to fit the two-dimensional binary distribution of the linear convolution, and C / / G×H×W is used as the input tensor, where C represents the number of channels, H represents the height, W represents the width, and G represents the number of blocks.
[0055] The EMA attention mechanism module also uses two-dimensional global average pooling to encode global spatial information in the output of the 1×1 branch, and converts the 1×1 branch into a feature map before the joint activation mechanism of the channel features.
[0056] The two-dimensional global pooling operation is as follows:
[0057]
[0058] Among them, Z c represents the input feature at the cth channel, x c (i,j) represents the value of the feature map at position (i,j).
[0059] Finally, each set of output feature maps is computed as the aggregation of the two generated spatial attention weight values, and the aggregation result is applied to the input feature maps by multiplication. Figure X The output Y is obtained.
[0060] like Figure 5 As shown in the figure, GSConv includes standard convolution Conv, depthwise separable convolution DWConv and shuffle operation. The features of different levels obtained by standard convolution and depthwise separable convolution are concat-joined, and then the information from standard convolution is completely mixed into the output of depthwise separable convolution through shuffle operation.
[0061] VoV-GSCSP includes cross-stage operations like residuals, uses two branches to extract features, and aggregates the feature maps of the branches combined with cross-stage operations like residuals.
[0062] like Figure 6 As shown in the figure, the TGC module has three feature maps of different sizes as input, namely C / 2×C / 2, C×C and 2C×2C. The TGC module first performs neighbor interpolation upsampling on the smaller C / 2×C / 2 feature map to obtain a feature map of size C×C; the medium-sized C×C feature map remains unchanged; the larger 2C×2C feature map is subjected to adaptive maximum pooling (Maxpolling) and adaptive average pooling (Avgpolling) operations to obtain a feature map of size C×C; finally, the three C×C feature maps are concat spliced together as the output of the TGC module to achieve three-input channel fusion.
[0063] Combination Figure 2, the Small feature map of the TGC-1 module comes from FasterC2f-2, whose channel is 64, the Medium feature map comes from FasterC2f-3, whose channel is 128, and the input of the Large feature map comes from GSConv-1, whose channel is 256. The Small feature map of the TGC-2 module comes from FasterC2f-1, whose channel is 32, the Medium feature map comes from FasterC2f-2, whose channel is 64, and the input of the Large feature map comes from GSConv-2, whose channel is 128. The Small feature map of the TGC-3 module comes from Conv-2, whose channel is 16, the Medium feature map comes from FasterC2f-1, whose channel is 32, and the input of the Large feature map comes from GSConv-3, whose channel is 64.
[0064] like Figure 7 As shown in the figure, the present invention introduces a small target detection layer with a size of 160×160 on the basis of the original three detection layers of the YOLOv8 model. The improved YOLOv8 model includes four detection heads of 160×160, 80×80, 40×40, and 20×20. The four detection heads use Dyhead to fuse richer target features, enhance the perception ability of various targets, and further enhance the model's detection ability of the instrument range area.
[0065] like Figure 8 As shown in Figure 1, the Dyhead detection head used can achieve scale awareness, spatial awareness, and task awareness at the same time. DyHead decomposes the attention function into three sequences, each focusing on one dimension. For a specific three-dimensional feature tensor F∈R L×S×C , F represents the three-dimensional feature tensor, R represents the real number tensor, L represents the spatial perception attention module, S represents the multi-scale perception attention module, and C represents the task perception attention module.
[0066] The description of the attention mechanism is as follows:
[0067] W(F)=π C (π S (π L (F)·F)·F)·F
[0068] Among them, W(F) represents the extraction process of the attention mechanism, π L (.) is the processing of the feature tensor by the scale-aware attention module, π s (.) is the processing of the feature tensor by the spatial perception attention module, π c (.) is the processing of the feature tensor by the task-aware attention module. The three attention modules only act on the L, S and C dimensions of the three-dimensional tensor F respectively.
[0069] Combination Figure 2 The Backbone part of the YOLOv8 model includes Conv-1, Conv-2, Conv-3, Conv-4, Conv-5, FasterC2f-1, FasterC2f-2, FasterC2f-3, FasterC2f-4, SPPF and EMA-1; the Neck part includes TGC-1, TGC-2, TGC-3, GSConv-1, GSConv-2, GSConv-3, GSConv-4, VoVGSCSP-1, VoVGSCSP-2, VoVGSCSP-3, Concat-1, Concat-2, Concat-3, FasterC2f-5, FasterC2f-6, FasterC2f-7, EMA-2, EMA-3; the Head part includes Dyhead160×160, Dyhead80×80, Dyhead40×40, Dyhead20×20.
[0070] The output of Conv-1 is passed to Conv-2, and the output of Conv-2 is divided into two paths, which are passed to FasterC2f-1 and TGC-3 respectively. The output of FasterC2f-1 is divided into three paths, which are passed to Conv-3, TGC-3 and TGC-2 respectively. The output of Conv-3 is passed to FasterC2f-2, and the output of FasterC2f-2 is divided into three paths, which are passed to Conv-4, TGC-2 and TGC-1 respectively. The output of Conv-4 is passed to FasterC2f-3, and the output of FasterC2f-3 is divided into two paths, which are passed to Conv-5 and TGC-1 respectively. The output of Conv-5 is passed to FasterC2f-4, and the output of FasterC2f-4 is passed to SPPF, and the output of SPPF is passed to EMA-1. The output of EMA-1 is divided into two paths, which are passed to GSConv-1 and Concat-3 respectively.
[0071] The output of GSConv-1 is passed to TGC-1, and the output of TGC-1 is passed to VoVGSCSP-1. The output of VoVGSCSP-1 is divided into two paths and passed to GSConv-2 and Concat-2 respectively. The output of GSConv-2 is passed to TGC-2, and the output of TGC-2 is passed to VoVGSCSP-2. The output of VoVGSCSP-2 is divided into two paths and passed to GSConv-3 and Concat-1 respectively. The output of GSConv-3 is passed to TGC-3, and the output of TGC-3 is passed to VoVGSCSP-3. The output of VoVGSCSP-3 is divided into two paths and passed to Dyhead160×160 and GSConv-4 respectively. The output of GSConv-4 is passed to Concat-1, and the output of Concat-1 is passed to FasterC2f-5, and the output of FasterC2f-5 is passed to EMA-2. The output of EMA-2 is divided into two paths and passed to Dyhead80×80 and GSConv-5 respectively. The output of GSConv-5 is passed to Concat-2, the output of Concat-2 is passed to FasterC2f-6, the output of FasterC2f-6 is passed to EMA-3, the output of EMA-3 is divided into two paths, passed to Dyhead40×40 and Concat-3 respectively. The output of Concat-3 is passed to FasterC2f-7, and the output of FasterC2f-7 is passed to Dyhead20×20.
[0072] The experimental results of the improved YOLOv8 model and the original model are shown in Table 1 and Fig. 9 shown.
[0073] Table 1
[0074]
[0075] (3) Construct an improved CRNN model, including: using ResNet18 as the backbone network and adding the CBAM attention mechanism. The specific network is shown in Table 2;
[0076] Table 2
[0077]
[0078] The improved CRNN model crops and preprocesses the detected instrument range area, including image denoising, contrast enhancement, and size adjustment.
[0079] The improved CRNN model is used to identify the range of the detected instrument. The sequence features are extracted through the bidirectional LSTM layer, and the final range text is decoded using the CTC decoder. The loss function is constructed through the calculation of the forward and backward algorithms, and finally trained through the Back Propagation algorithm.
[0080] When the input sequence is S, the multiplication of the probability of the label sequence at each time t is the output probability result of CTC, that is, the probability when the output sequence is S can generate the probability of a label sequence π of the sequence S.
[0081] Resnet18 is used as the backbone to reduce overfitting. The CBAM attention mechanism is used to improve detection accuracy. The channel attention module performs global average pooling and global maximum pooling on the input feature map. The channel attention weight is multiplied channel by channel with the original feature map to obtain the enhanced spatial attention module. The two feature maps are spliced in the channel dimension and input into the convolution layer to generate the spatial attention weight; the spatial attention weight is multiplied element by element with the channel enhanced feature map to obtain the final enhanced feature map.
[0082] The improved CRNN model can more effectively extract and utilize image features, and improve the accuracy and robustness of range recognition. The comparison results of the recognition accuracy between the improved CRNN model and the original model are shown in Table 3 and Fig.10 shown.
[0083] Table 3
[0084]
[0085] (4) The improved YOLOv8 model is trained using the instrument detection dataset, and the improved CRNN model is trained using the range recognition dataset;
[0086] The improved YOLOv8 model uses SGD optimizer parameter iterative update, 200 training rounds, momentum parameter 0.937, and Batchsize set to 32 to ensure improved model training performance.
[0087] The improved CRNN model uses the Adam optimizer to update parameters and adopts CTCloss as the loss function. The model parameters are continuously optimized through 100 rounds of iterative training.
[0088] (5) The trained improved YOLOv8 model is used to perform instrument detection on the input image. The detection results are respectively input into the trained improved CRNN model and the pre-trained U2net model. The trained improved CRNN model identifies the instrument range, and the pre-trained U2net model performs pointer and scale segmentation. The final reading of the instrument is completed by combining the identified instrument start range scale value and end range scale value.
[0089] Object Detection:
[0090] The detection results of the improved YOLOv8 model include the bounding box coordinates and confidence scores of the meter area meter, the maximum range value area max, and the minimum range value area min. The detection results are used for subsequent range identification and meter readings. The target detection process accurately identifies and locates the meter range area in the image, providing the original image for the subsequent max and min area image capture of range identification. The target detection diagram is shown in the figure below. Fig.11 shown.
[0091] There may be overlapping detection results in multiple detection frames output by the model. Using the non-maximum suppression (NMS) algorithm to filter overlapping detection frames and retain the detection frame with the highest confidence can ensure that the final output detection result is optimal.
[0092] Range identification:
[0093] Range recognition aims to accurately identify the values of the max and min areas marked on the instrument. The bounding box recorded by the target detection defines the specific position coordinates of the maximum range value area max and the minimum range value area min on the instrument. The max and min areas are extracted from the original input image using the bounding box coordinates obtained by target detection, and then the cropping operation is performed to remove irrelevant background information in the image, retaining only the areas related to max and min, reducing background interference and improving recognition accuracy. The image areas of the cropped max and min areas are adjusted to a uniform size to adapt to the input requirements of the subsequent model. Ensure that all input images have the same size and proportion to facilitate batch processing and model training.
[0094] The improved CRNN model is used to perform range recognition on the cropped and resized max and min areas, and the CTC decoder is used to decode the final range text, and the recognized range information is output, including the range value and decimal point.
[0095] Meter reading:
[0096] The instrument image is segmented using the U2Net model (the U2Net model is an existing technology). The segmented instrument image is post-processed to extract the pointer position and scale information. The U2Net model is used to perform semantic segmentation on the input instrument image to obtain the accurate segmentation results of the pointer and scale. The instrument segmentation map is shown in Fig.12 As shown, the instrument segmentation diagram further obtains Fig.13 The instrument segmentation conversion diagram is shown. In the instrument segmentation conversion diagram, each peak point represents a scale line, the blue line represents the position of the pointer, and the orange line represents the distribution of each scale line. By accumulating the number of these peak points, the total number of scales of the instrument can be calculated. According to the proportional relationship of the scales involved in the blue line, by calculating the proportion of the pointer reading to the range, the number of scale lines on the left side of the pointer and the total number of scale lines are calculated using the distance method, and the result is the scale ratio of the instrument.
[0097] Specifically, the number of scale lines on the left side of the pointer and the total number of scale lines are combined with the previously identified starting scale and ending scale information, and are applied to the distance method for calculation. The result is the reading value of the instrument. Among them, the core idea of the distance method is to calculate the distance between the pointer and each scale, and determine the reading indicated by the pointer by finding the scale with the smallest distance. The Euclidean distance between the pointer endpoint and each scale point is calculated, and the scale with the smallest distance is selected as the reading indicated by the pointer. Finally, the reading is mapped to the actual value according to the range information to obtain the final instrument reading result. The present invention can effectively handle different types of pointer instruments and improve the accuracy and reliability of the reading.
[0098] The distance method formula is as follows:
[0099]
[0100] Among them, value is the numerical value of the instrument's indication, L is the starting scale of the pointer instrument, R represents the ending scale, d1 is the number of scales from the leftmost starting scale to the pointer, and d2 is the total number of scales from the leftmost starting scale to the rightmost ending scale.
[0101] The present invention mainly relates to the improvement of the instrument recognition model and the range recognition model. The improved YOLOv8 model is used to remove complex background information and only retain the instrument graphics. The improved CRNN model is used to realize high-precision recognition of the range scale of the pointer-type instrument. The pointer and scale are segmented by the U2net model, and the instrument's starting range scale value and ending range scale value are combined to complete the final reading of the instrument. The present invention can adapt to complex industrial environments, has good generalization ability and robustness, and can be widely used in industrial automation, energy management, equipment monitoring and other fields.
Claims
1. A method for identifying the number displayed on a pointer instrument based on deep learning, characterized in that: include: (1) Construct instrument detection data set and range identification data set; (2) Build an improved YOLOv8 model, including: Replace the C2f module in the Backbone part with the FasterC2f module. The FasterC2f module is formed by replacing the Bottleneck module in the C2f module with the FasterBlock module. Add the EMA attention mechanism module after the SPPF layer in the Backbone part. Replace the first three concat modules in the Neck part with TGC modules, introduce the Slim-Neck structure in the Neck part, and the Slim-Neck structure includes GSConv and VoV-GSCSP. Replace the Conv in the Neck part with GSConv, replace the first three C2f modules in the Neck part with VoV-GSCSP, and replace the last three C2f modules in the Neck part with FasterC2f modules; among the three replaced FasterC2f modules, add the EMA attention mechanism module after the first two FasterC2f modules; In the Head part, use the Dyhead detection head and add a 160×160 small target detection layer; (3) Building an improved CRNN model, including: using ResNet18 as the backbone network and adding the CBAM attention mechanism; (4) The improved YOLOv8 model is trained using the instrument detection dataset, and the improved CRNN model is trained using the range recognition dataset; (5) The trained improved YOLOv8 model is used to detect the input image, and the detection results are respectively input into the trained improved CRNN model and the pre-trained U2net model. The trained improved CRNN model identifies the instrument range, and the pre-trained U2net model performs pointer and scale segmentation. The final reading of the instrument is completed by combining the identified instrument start range scale value and end range scale value.
2. The method for identifying the number displayed on a pointer-type instrument based on deep learning according to claim 1, characterized in that: Step (1) comprises: collecting pictures containing pointer instruments, annotating the pictures, wherein the annotation contents include meter, max, and min, which respectively represent the instrument dial area, the instrument range maximum value area, and the instrument range minimum value area, and after the annotation is completed, an instrument detection data set is formed; and a range recognition data set is formed by selecting the max and min areas of the pictures in the instrument detection data set.
3. The method for identifying the number displayed on a pointer-type instrument based on deep learning according to claim 2 is characterized in that: Data enhancement is performed when constructing the instrument detection dataset, including random scaling, cropping, arrangement, color jittering, and flipping operations.
4. The method for identifying the number displayed on a pointer-type instrument based on deep learning according to claim 1, characterized in that: GSConv includes standard convolution, depthwise separable convolution DWConv and shuffle operation. The features of different levels obtained by standard convolution and depthwise separable convolution are concat-joined, and then the information from standard convolution is completely mixed into the output of depthwise separable convolution through shuffle operation. VoV-GSCSP includes cross-stage operations like residuals, uses two branches to extract features, and aggregates the feature maps of the branches combined with cross-stage operations like residuals.
5. The method for identifying the number displayed on a pointer-type instrument based on deep learning according to claim 4 is characterized in that: The TGC module has three feature maps of different sizes as input, namely C / 2×C / 2, C×C and 2C×2C; the feature map of C / 2×C / 2 is upsampled by nearest neighbor interpolation to obtain a feature map of size C×C; the feature map of C×C remains unchanged; the feature map of 2C×2C is subjected to adaptive maximum pooling and adaptive average pooling operations to obtain a feature map of size C×C; finally, the three C×C feature maps are connected as the output of the TGC module.
6. The method for identifying the number displayed on a pointer-type instrument based on deep learning according to any one of claims 1 to 5, characterized in that: The Backbone part of the YOLOv8 model includes Conv-1, Conv-2, Conv-3, Conv-4, Conv-5, FasterC2f-1, FasterC2f-2, FasterC2f-3, FasterC2f-4, SPPF and EMA-1; the Neck part includes TGC-1, TGC-2, TGC-3, GSConv-1, GSConv-2, GSConv-3, GSConv-4, VoVGSCSP-1, VoVGSCSP-2, VoVGSCSP-3, Concat-1, Concat-2, Concat-3, FasterC2f-5, FasterC2f-6, FasterC2f-7, EMA-2, EMA-3; the Head part includes Dyhead160×160, Dyhead80×80, Dyhead40×40, Dyhead20×20; The output of Conv-1 is passed to Conv-2, and the output of Conv-2 is divided into two paths, which are passed to FasterC2f-1 and TGC-3 respectively; the output of FasterC2f-1 is divided into three paths, which are passed to Conv-3, TGC-3 and TGC-2 respectively; the output of Conv-3 is passed to FasterC2f-2, and the output of FasterC2f-2 is divided into three paths, which are passed to Conv-4, TGC-2 and TGC-1 respectively; the output of Conv-4 is passed to FasterC2f-3, and the output of FasterC2f-3 is divided into two paths, which are passed to Conv-5 and TGC-1 respectively; the output of Conv-5 is passed to FasterC2f-4, and the output of FasterC2f-4 is passed to SPPF, and the output of SPPF is passed to EMA-1; the output of EMA-1 is divided into two paths, which are passed to GSConv-1 and Concat-3 respectively; The output of GSConv-1 is passed to TGC-1, the output of TGC-1 is passed to VoVGSCSP-1, the output of VoVGSCSP-1 is divided into two paths, passed to GSConv-2 and Concat-2 respectively; the output of GSConv-2 is passed to TGC-2, the output of TGC-2 is passed to VoVGSCSP-2, the output of VoVGSCSP-2 is divided into two paths, passed to GSConv-3 and Concat-1 respectively; the output of GSConv-3 is passed to TGC-3, the output of TGC-3 is passed to VoVGSCSP-3, the output of VoVGSCSP-3 is divided into two paths, passed to Dyhead160×160 and GSConv-4 respectively; the output of GSConv-4 Passed to Concat-1, the output of Concat-1 is passed to FasterC2f-5, the output of FasterC2f-5 is passed to EMA-2, the output of EMA-2 is divided into two paths, passed to Dyhead80×80 and GSConv-5 respectively; the output of GSConv-5 is passed to Concat-2, the output of Concat-2 is passed to FasterC2f-6, the output of FasterC2f-6 is passed to EMA-3, the output of EMA-3 is divided into two paths, passed to Dyhead40×40 and Concat-3 respectively; the output of Concat-3 is passed to FasterC2f-7, and the output of FasterC2f-7 is passed to Dyhead20×20.
7. The method for identifying the number displayed on a pointer-type instrument based on deep learning according to claim 1, characterized in that: In step (4), the improved YOLOv8 model uses the SGD optimizer to iteratively update the parameters; the improved CRNN model uses the Adam optimizer to update the parameters and adopts CTCloss as the loss function.
8. The method for identifying the number displayed on a pointer-type instrument based on deep learning according to claim 2, characterized in that: In step (5), the detection results of the improved YOLOv8 model include the bounding box coordinates and confidence scores of the instrument area, the maximum range value area max, and the minimum range value area min.
9. The method for identifying the number displayed on a pointer-type instrument based on deep learning according to claim 8, characterized in that: Use the non-maximum suppression (NMS) algorithm to filter overlapping detection frames and retain the detection frame with the highest confidence.
10. The method for identifying the number displayed on a pointer-type instrument based on deep learning according to claim 1, characterized in that: In step (5), the distance method is used to calculate the final instrument reading, including: calculating the Euclidean distance between the pointer endpoint and each scale point, selecting the scale with the smallest distance as the reading indicated by the pointer; mapping the reading to the actual value according to the range information to obtain the final instrument reading result.
Citation Information
Patent Citations
Container weak and small serial number target detection and identification method based on deep learning
CN117253154A
Pointer type instrument detection method and system based on deep learning
CN118379741A
Pointer type instrument reading method based on adaptive pooling and attention fusion
CN118397612A
Cited By
Image recognition and positioning method of component embedded part based on target detection
CN119672717A
An image recognition and positioning method for component embedded parts based on target detection
CN119672717B
A digital display meter reading recognition method and device based on a target recognition model
CN122676517A
