Liquid level instrument reading method and system, electronic device, and medium

Abstract

The application discloses a liquid level type instrument reading method and system, electronic equipment and a medium. The reading method comprises the following steps: performing image segmentation processing on a liquid level type instrument image according to vertical texture information of the instrument image, so as to determine a scale panel region and a liquid level column region in the instrument image; performing text recognition on the scale panel region, so as to obtain a scale number recognition result; performing image processing on the liquid level column region, so as to obtain a liquid level height; and determining a reading result of the liquid level type instrument according to the liquid level height and the scale number recognition result. There is strong vertical texture between the scale panel region and the liquid level column region. The image segmentation processing is performed on the instrument image according to the vertical texture information of the liquid level type instrument image. The segmentation line is determined in combination with the vertical texture information and the text detection result, the accuracy of segmentation is improved, and the accuracy of liquid level type instrument reading is improved.

Description

Technical Field

[0001] This invention relates to the field of instrument reading, specifically to a method and system for reading liquid level instruments, electronic equipment, and media. Background Technology

[0002] With the rapid development of industry, level gauges are used in industrial production to measure temperature or water level. However, the workload of reading these gauges is enormous. Currently, most level gauge readings are done manually. However, in the process of reading gauges by eye, workers need to combine the gauge scale lines with the liquid level to make a comprehensive judgment, resulting in a significant time consumption. Furthermore, the environment in which the gauges are located may not be comfortable for humans, thus posing safety hazards. Therefore, the automatic identification of level gauges has become a topic of common concern in industry and academia, and many researchers have proposed efficient automatic gauge reading methods. With the popularization and high accuracy of deep learning, deep learning-based gauge reading algorithms have become the mainstream solution to this type of problem.

[0003] Current technologies rely solely on traditional image processing methods to identify liquid level gauges, which is not intelligent enough. They acquire gauges using pre-defined templates, resulting in specificity but limited generalization. For liquid level gauges with insufficiently distinctive features, the detection performance is limited.

[0004] Existing technologies also target the measuring cylinder area for identification, requiring manual cutting out of the measuring cylinder area and identification of the liquid level height based on the heat map output by the model. This requires a lot of manual intervention, which is inconvenient. The recognition accuracy is highly correlated with the position of the liquid level in the measuring cylinder, and it is only applicable to specific scenarios of measuring cylinder liquid level identification, and is not compatible with liquid level gauge identification. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the defect of the prior art in that it is impossible to accurately identify the scale value of liquid level instruments, and to provide a liquid level instrument reading method and system, electronic equipment and medium.

[0006] The present invention solves the above-mentioned technical problems through the following technical solution:

[0007] Firstly, a method for reading a liquid level gauge is provided, the method including:

[0008] The instrument image is segmented based on the vertical texture information of the liquid level gauge image to determine the scale panel area and the liquid level column area in the instrument image;

[0009] Text recognition is performed on the scale panel area to obtain the scale number recognition result;

[0010] Image processing is performed on the liquid level column area to obtain the liquid level height;

[0011] The reading result of the liquid level instrument is determined based on the liquid level height and the scale number recognition result.

[0012] Optionally, the step of performing image segmentation processing on the instrument image based on the vertical texture information of the level gauge image to determine the scale panel area and the level column area in the instrument image includes:

[0013] The Sobel algorithm is used to perform a convolution operation on the liquid level instrument image to obtain a texture activation map; wherein, the vertical texture information includes the texture activation map;

[0014] The Otsu thresholding method was used to perform image segmentation on the texture activation map to obtain the image segmentation result;

[0015] Morphological processing is performed on the image segmentation results to obtain a morphological image;

[0016] The scale panel area and liquid level column area in the instrument image are determined based on the texture intensity of the morphological image.

[0017] Optionally, before the step of performing image segmentation processing on the instrument image based on the vertical texture information of the level gauge image, the method further includes:

[0018] Acquire the original image of the liquid level instrument;

[0019] The original image is input into the instrument detection model to extract the liquid level instrument image from the original image based on the first output result of the instrument detection model; wherein, the instrument detection model includes a feature extraction sub-model, a first region generation network, and a second region generation network; the feature extraction sub-model is used to perform general feature extraction on the original image, the first region generation network is used to determine candidate regions of the liquid level instrument in the original image based on the general features, and the second region generation network is used to perform feature extraction, category classification, and bounding box regression on the candidate regions, and output the first output result.

[0020] Optionally, the feature extraction sub-model includes a first convolutional neural network and a pyramid network; the first convolutional neural network is used to extract features from the original image to obtain feature maps of different sizes; the pyramid network is used to fuse the feature maps of different sizes to obtain the general features of the original image.

[0021] Optionally, the text recognition of the scale panel area includes:

[0022] The scale panel area is input into a text detection model to determine the text probability map and adaptive threshold map of the scale panel area based on the output of the text detection model; wherein, the text detection model includes a text feature extraction network and a differentiable binarization network; the text feature extraction network is used to perform differentiable binarization processing on the text feature map to obtain the text probability map and the adaptive threshold map;

[0023] The text probability map and the adaptive threshold map are binarized to obtain an approximate binary map corresponding to the original image;

[0024] The binary image is processed to obtain the text region in the scale panel area; wherein the text region is a bounded polygon image with numbers;

[0025] Text recognition is performed on the text region.

[0026] Optionally, the text recognition of the text region includes:

[0027] The text region is input into a text recognition model to determine the scale number recognition result based on the second output result of the text recognition model; wherein, the text recognition model includes a second convolutional neural network and a recurrent neural network;

[0028] The second convolutional neural network is used to extract image features of the text region;

[0029] The recurrent neural network is used to obtain the sequence information of the text region based on the image features, and to perform CTC encoding based on the sequence information to obtain the scale number recognition result.

[0030] Optionally, determining the reading result of the level gauge based on the liquid level height and the scale digit recognition result includes:

[0031] The scale line recognition result in the scale panel area is determined based on the text area;

[0032] Based on the text area and the scale line recognition results, the range between the scales in the scale panel area is determined;

[0033] The reading result of the instrument is determined based on the liquid level height, the scale line recognition result, and the range.

[0034] Optionally, determining the scale line recognition result in the scale panel area based on the text area includes:

[0035] The text regions are sorted according to their coordinates, and then traversed from top to bottom.

[0036] The recognition result of the scale lines in the scale panel area is determined based on the vertical coordinate of the center of the text area.

[0037] Secondly, a reading system for a liquid level gauge is provided, the reading system comprising:

[0038] The segmentation module is used to perform image segmentation processing on the instrument image based on the vertical texture information of the liquid level instrument image, so as to determine the scale panel area and the liquid level column area in the instrument image;

[0039] The recognition module is used to perform text recognition on the scale panel area to obtain the scale number recognition result;

[0040] The processing module is used to perform image processing on the liquid level column area to obtain the liquid level height;

[0041] The determination module is used to determine the reading result of the liquid level instrument based on the liquid level height and the scale number recognition result.

[0042] Thirdly, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and for running on the processor, wherein the processor executes the computer program to implement the method for reading a level gauge as described in the first aspect.

[0043] Based on common knowledge in the field, the above-mentioned preferred conditions can be combined arbitrarily to obtain various preferred embodiments of the present invention.

[0044] The positive and progressive effects of this invention are as follows: it detects and recognizes text on the instrument, extracts the liquid level height, outputs the reading results, and shows strong vertical texture between the scale panel area and the liquid level column area. Based on the vertical texture information of the liquid level instrument image, it performs image segmentation processing on the instrument image, and combines the vertical texture information and the text detection results to determine the segmentation line, thereby improving the accuracy of segmentation and thus improving the accuracy of liquid level instrument reading. Attached Figure Description

[0045] Figure 1 A first flowchart of a method for reading a liquid level instrument provided in an embodiment of the present invention;

[0046] Figure 2 A second flowchart of a method for reading a liquid level instrument provided in an embodiment of the present invention;

[0047] Figure 3 An application scenario effect diagram of the Faster R-CNN network structure for a liquid level gauge reading method provided in an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of the FPN network structure for a liquid level gauge reading method provided in an embodiment of the present invention;

[0049] Figure 5 The image shows an application scenario of the DBNet text detection network for a liquid level gauge reading method provided in an embodiment of the present invention.

[0050] Figure 6 A schematic diagram illustrating the difference between differentiable binarization (DB) and standard binarization (SB) in a liquid level gauge reading method provided in an embodiment of the present invention;

[0051] Figure 7 The image shows an application scenario effect of the CRNN text recognition network for a liquid level instrument reading method provided in an embodiment of the present invention.

[0052] Figure 8 A schematic diagram of CRNN text recognition preprocessing for a liquid level gauge reading method provided in an embodiment of the present invention;

[0053] Figure 9 A flowchart of a conventional algorithm for reading a liquid level instrument, provided in an embodiment of the present invention;

[0054] Figure 10 A schematic diagram of a liquid level gauge, including the liquid level column, scale panel, liquid level line, range, and key scale line, provided for an embodiment of the present invention for reading a liquid level gauge.

[0055] Figure 11 A schematic diagram illustrating the calculation of the vertical edge of a liquid level gauge reading method provided in an embodiment of the present invention;

[0056] Figure 12 A morphological processing schematic diagram of a liquid level gauge reading method provided in an embodiment of the present invention;

[0057] Figure 13 A binary histogram schematic diagram of a liquid level gauge reading method provided in an embodiment of the present invention;

[0058] Figure 14 A statistical and segmented scale panel and liquid level column schematic diagram of a liquid level gauge reading method provided in an embodiment of the present invention;

[0059] Figure 15 A flowchart illustrating the extraction of the red line segment from the left scale panel of a liquid level gauge reading method provided in an embodiment of the present invention;

[0060] Figure 16 A numerical calculation effect diagram of a liquid level gauge reading method provided in an embodiment of the present invention;

[0061] Figure 17 This is a schematic diagram of the structure of a liquid level gauge reading system provided in an embodiment of the present invention.

[0062] Figure 18 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0063] The present invention will be further illustrated by way of embodiments below, but the present invention is not limited to the scope of the embodiments described herein.

[0064] like Figure 1 As shown, the method for reading a liquid level instrument provided in this embodiment of the invention includes the following steps:

[0065] S11. Perform image segmentation processing on the instrument image based on the vertical texture information of the liquid level instrument image to determine the scale panel area and liquid level column area in the instrument image.

[0066] The scale panel area includes the portions containing scale readings on both sides of the level column, the level column area includes the area in the middle of the level gauge, and the scale panel area includes the left scale panel and the right scale panel.

[0067] In one embodiment, image segmentation processing is performed on the instrument image based on the vertical texture information of the level gauge image to determine the scale panel area and the level column area in the instrument image, including:

[0068] The Sobel algorithm is used to perform a convolution operation on the liquid level gauge to obtain a texture activation map, in which the vertical texture information includes the texture activation map;

[0069] The Otsu thresholding method was used to segment the texture activation map, and the image segmentation results were obtained.

[0070] Morphological processing is performed on the image segmentation results to obtain a morphological image;

[0071] The scale panel area and level column area of ​​the instrument image summary are determined based on the texture intensity of the morphological image.

[0072] Image processing methods are used to divide the level gauge into three parts: the level column, the left scale panel, and the right scale panel. Strong vertical textures exist between the scale panel area and the level column area. Vertical texture information is extracted for segmentation of the level column and scale panel. Fragmented vertical textures on the level gauge dial can affect the segmentation results; therefore, image processing methods are used to remove texture noise. An adaptive threshold segmentation algorithm is used to distinguish between strong and weak vertical textures. Then, morphological processing methods are used to extract vertical textures exceeding a fixed length. The fixed length is set according to actual conditions; in this embodiment, the fixed length is 50, resulting in a strong vertical texture activation map. In the horizontal direction, the coordinates of the numbers near the center of the scale panel almost coincide with the segmentation line between the level column and the scale panel. Combining the strong vertical texture activation map with the segmentation line improves the accuracy of segmentation.

[0073] The Sobel algorithm is used to perform convolution operations on the enhanced image to obtain a texture activation map. The texture activation map satisfies the following characteristics: in the vertical direction of the original image, if there is strong texture, the corresponding activation map has a high gray value; if the texture is weak, the corresponding activation map has a low gray value. After obtaining the activation map, in order to eliminate the spots in the image, the Otsu thresholding method is used to segment the image to obtain a binary image. After obtaining the binary image, the activation values ​​at the scale numbers of the liquid level gauge image are relatively high, which affects the segmentation effect. Morphological processing is performed on the binary image to improve the robustness of the algorithm. After obtaining the morphological image, the texture intensity of each column of the image is counted, and the scale panel area and liquid level column area in the gauge image are determined based on the texture intensity of the morphological image.

[0074] There are strong vertical textures between the scale panel area and the liquid level column area. Based on the vertical texture information of the liquid level instrument image, the instrument image is segmented. The segmentation line is determined by combining the vertical texture information and the text detection results, thereby improving the accuracy of segmentation and thus improving the accuracy of liquid level instrument reading.

[0075] In one embodiment, before the step of performing image segmentation processing on the instrument image based on the vertical texture information of the level gauge image, the method further includes:

[0076] Acquire the original image of the liquid level gauge; input the original image into the gauge detection model to extract the liquid level gauge image from the original image based on the first output result of the gauge detection model; wherein, the gauge detection model includes a feature extraction sub-model, a first region generation network and a second region generation network; the feature extraction sub-model is used to perform general feature extraction on the original image, the first region generation network is used to determine the candidate region of the liquid level gauge in the original image based on the general features, and the second region generation network is used to perform feature extraction, category classification and bounding box regression on the candidate region, and output the first output result.

[0077] In one embodiment, the feature extraction sub-model includes a convolutional neural network (CNN).

[0078] The instrument detection model adopts the two-stage object detection algorithm Faster R-CNN, which includes the following two stages: The first stage consists of two networks: a convolutional neural network (CNN) and a region generation network (RPN). The CNN is used to extract general features. In this embodiment of the invention, the general feature extraction network of the instrument detection model is ResNet-50. The RPN network consists of a convolutional network and classification and regression networks. The first region generation network includes a convolutional network and a regression network. The second stage uses the second region generation network to extract features from the candidate regions from the RPN and perform category classification, bounding box regression, and output the image of the dial part of the liquid level instrument.

[0079] In one embodiment, the feature extraction sub-model includes a first convolutional neural network and a pyramid network. The first convolutional neural network is used to extract features from the original image to obtain feature maps of different sizes. The pyramid network is used to fuse the feature maps of different sizes to obtain general features of the original image.

[0080] The camera is used to acquire images of the liquid level gauge. As the distance between the camera and the liquid level gauge changes, the size of the gauge in the image changes significantly. To improve the robustness of the model to targets of different sizes, a feature pyramid network is added to the general feature extraction stage of Faster R-CNN. The first convolutional neural network extracts features from the original image to obtain feature maps of different sizes. The pyramid feature network fuses the feature maps of different sizes to obtain the general features of the original image.

[0081] S12. Perform text recognition on the scale panel area to obtain the scale number recognition results.

[0082] The scale digit recognition results include the scale values ​​on the scale panel.

[0083] In one embodiment, text recognition of the scale panel region includes: inputting the scale panel region into a text detection model to determine a text probability map and an adaptive threshold map of the scale panel region based on the output of the text detection model; wherein the text detection model includes a text feature extraction network and a differentiable binarization network; the text feature extraction network is used to perform differentiable binarization processing on the text feature map to obtain a text probability map and an adaptive threshold map; the text probability map and the adaptive threshold map are binarized to obtain an approximate binary map corresponding to the original image; image processing is performed on the binary map to obtain a text region in the scale panel region; wherein the text region is a bounded polygon image with numbers; and text recognition is performed on the text region.

[0084] This invention utilizes algorithm components from the SenseTime AI computing platform, employing a segmentation-based text detection model, specifically the DBNet (Differentiable Binarization network). DBNet is a single-stage text detection network composed of a text feature extraction network (CNN) and a differentiable binarization network (DBNet). The CNN network extracts text feature maps, which are then used as input to DBNet, outputting a text probability map and an adaptive threshold map. Next, the text probability map and the adaptive threshold map undergo numerical transformation (Differentiable Binarization, DB) to obtain an approximate binary map. Finally, image processing methods are used to post-process the approximate binary map to obtain the text region in the scale panel area. The text region is a bounding polygon image containing numbers, and text recognition is performed on the text region.

[0085] In one embodiment, text recognition of a text region includes: inputting the text region into a text recognition model to determine a scale digit recognition result based on a second output result of the text recognition model; wherein the text recognition model includes a second convolutional neural network and a recurrent neural network; the second convolutional neural network is used to extract image features of the text region; the recurrent neural network is used to obtain sequence information of the text region based on the image features, and to perform CTC encoding (Connectionist temporal classification) based on the sequence information to obtain the scale digit recognition result.

[0086] This invention employs a text recognition model, specifically a text recognition network (CRNN). The CRNN consists of three parts: first, a second convolutional neural network (CNN) is used to extract image features; then, these features are input into a recurrent neural network (RNN) to process the sequence information in the image. The RNN used in this invention is a Long Short-Term Memory (LSTM) network; finally, the output of the RNN is CTC encoded to enhance the model's robustness to texts of varying lengths. Since the scale digit recognition results output by DBNet are rotated rectangles, preprocessing is required before text recognition. For each recognition result, its maximum bounding rectangle without angles is first calculated, and then the image inside the rectangle is extracted and fed into the CRNN for recognition.

[0087] S13. Perform image processing on the liquid level column area to obtain the liquid level height.

[0088] Liquid level column images are typically RGB images, where R represents red, G represents green, and B represents blue. Liquid level column images are mainly composed of red and white. In this embodiment of the invention, the liquid level height, i.e., the vertical coordinate of the liquid level line in the image, is determined by comparing the relative relationships of the color channels of each row of the image.

[0089] S14. Determine the reading result of the liquid level instrument based on the liquid level height and scale number recognition results.

[0090] In one embodiment, determining the reading result of a level gauge based on the liquid level height and the scale digit recognition result includes: determining the scale line recognition result in the scale panel area based on the text area; determining the range between scales in the scale panel area based on the text area and the scale line recognition result; and determining the reading result of the gauge based on the liquid level height, the scale line recognition result, and the range.

[0091] The text area is used to determine the scale line recognition result in the scale panel area, extract the reading scale position, calculate the adjacent reading scale positions, convert them into numbers for absolute difference calculation, select the mode of the absolute difference as the actual range, and determine the instrument reading result based on the liquid level height, scale line recognition result and range.

[0092] In one embodiment, determining the tick mark recognition result in the tick mark panel area based on the text area includes: sorting the text area according to the coordinates of the text area and traversing the text area in a top-to-bottom order; and determining the tick mark recognition result in the tick mark panel area based on the ordinate of the center of the text area.

[0093] The text detection boxes are sorted according to their coordinates. The text detection boxes are traversed from top to bottom. For each text detection box, the position with the closest vertical coordinate to the center of the text detection box is selected as the scale line position. This operation is repeated until the number of reading scale line positions is the same as the number of text detection boxes. The average value of the distance difference between adjacent reading scale lines is calculated. The liquid level height position and the reading scale line position are compared to obtain the scale line recognition result in the scale panel area.

[0094] The traditional algorithm based on image algorithms and rules includes image enhancement, calculation of vertical edges, adaptive threshold segmentation, morphological processing, statistical segmentation of liquid level columns and scale panels, extraction of liquid level height, extraction of line segments from the scale panel, statistical and numerical calculations. It has good compatibility for liquid level instrument recognition in similar scenarios. For an input image containing a liquid level instrument, the system can automatically locate the liquid level instrument dial and output the corresponding reading, and the system output reading accuracy is greater than or equal to 93%.

[0095] The following is combined Figure 2 The following further explains the method for reading level gauges:

[0096] The reading of liquid level gauges mainly consists of four steps: 1. Detecting liquid level gauges in an image using a target detection network and cropping the liquid level gauge image based on the detection results; 2. Performing text detection on the cropped liquid level gauge and extracting the digital scale on the liquid level gauge; 3. Performing text recognition on the digital scale on the liquid level gauge; 4. Using the liquid level gauge image, text detection, and text recognition results as input to a traditional algorithm and outputting the reading result.

[0097] In the instrument detection stage, the detection of the dial of the level gauge is considered as a target detection problem. Regarding the target detection algorithm, this embodiment of the invention uses algorithm components from the SenseTime AI computing platform. For instrument dial detection, a two-stage target detection algorithm, Faster R-CNN, is employed. The Faster R-CNN network structure is as follows: Figure 3 As shown, the target detection algorithm comprises two stages: the first stage consists of a Convolutional Neural Network (CNN) and a Region Proposal Network (RPN). The CNN extracts general features and generates feature maps. In this embodiment, the instrument detection model uses ResNet-50 as its general feature extraction network. The RPN network typically consists of a simple convolutional network and classification and regression networks. The second stage extracts features from the candidate regions from the RPN and uses a classifier for category classification and bounding box regression. RoI pooling refers to Region of Interest pooling. As the distance between the camera and the liquid level gauge changes, the size of the gauge in the image changes significantly. To improve the model's robustness to targets of different sizes, such as... Figure 4 As shown, this scheme adds a Feature Pyramid Network (FPN) to the general feature extraction stage (CNN) of Faster R-CNN.

[0098] In the scale digit detection stage, scale digit detection is treated as a text detection problem. Regarding text detection, this embodiment of the invention utilizes algorithm components from the SenseTime AI computing platform. Based on text detection, it employs the segmentation-based text detection network DBNet. A schematic diagram of the DBNet network is shown below. Figure 5As shown, DBNet is a single-stage text detection network consisting of a text feature extraction network (CNN) and a differentiable binarization network (DBNet). Element-wise summation is used for element-wise summation, up-sampling with ratio N is used for upsampling by a ratio of N, and convolution is used for convolution. For the input image, the CNN network is first used to extract text feature maps; then, the text feature maps are used as input to DBNet, outputting a text probability map and an adaptive threshold map; next, the text probability map and the adaptive threshold map are subjected to Differentiable Binarization (DB) to obtain an approximate binary map; finally, image processing methods are used to post-process the approximate binary map to obtain the text region in the scale panel area, where the text region is an circumscribed polygon image with numbers.

[0099] Classic text detection networks typically use a fixed threshold to distinguish whether text exists at the output location. However, this threshold should change according to the information at the current location. This invention transforms the "binarization of the image using a fixed threshold" operation into a learnable process, thereby enabling the model to have stronger discriminative power against positive and negative samples. The difference between differentiable binarization (DB) and standard binarization (SB) is as follows: Figure 6 As shown, differentiable binarization provides stronger discrimination between positive and negative samples. The formula for calculating differentiable binarization is:

[0100]

[0101] Where r(i,j) is Figure 5 The approximate binary map is the result at coordinate (i,j), p(i,j) is the result of the text probability map at coordinate (i,j), t(i,j) is the result of the adaptive threshold map at coordinate (i,j), and k is a hyperparameter, which defaults to 50.

[0102] In the scale digit recognition stage, this embodiment of the invention uses algorithm components from the SenseTime AI computing platform. For the text recognition part, it employs the classic text recognition network CRNN, the structure of which is as follows: Figure 7As shown, the input image and convolutional feature maps are processed in the convolutional layers; deep bidirectional long short-term memory (LSTM) is performed in the recurrent layers; and per-frame predictions and predicted sequences are performed in the transcription layer. The CRNN network consists of three parts: first, image features are extracted using a convolutional neural network (CNN); then, the image features are input into a recurrent neural network (RNN) to process the sequence information in the image; the RNN network used in this embodiment is a long short-term memory (LSTM) network; finally, the output of the RNN network is CTC encoded to make the model more robust to text of different lengths. Figure 8 As shown, since the detection results output by DBNet are rotated rectangles, preprocessing is required before text recognition. For each detection result, its maximum bounding rectangle without angle is first calculated, and then the image inside the rectangle without angle is cropped and sent to the CRNN network for recognition.

[0103] In the traditional algorithm stage, the first three stages yield the location of the level gauge in the image, the level gauge scale readings, and the result, such as... Figure 9 and Figure 10 As shown, the traditional algorithm stage consists of two parts: image processing and rule-based strategies. It includes image enhancement, calculation of vertical edges, adaptive threshold segmentation, morphological processing, statistical segmentation of the liquid level column and scale panel, extraction of liquid level height, extraction of the red line segment of the left scale panel, statistics and numerical calculation. The algorithm details of each module of the traditional algorithm are described in detail below.

[0104] Image enhancement: Image enhancement is performed to improve the robustness of the algorithm. Different liquid level gauges have inconsistent brightness due to different lighting environments, making the pixel value distribution of their images more uniform, thereby improving the robustness of the algorithm. In this embodiment of the invention, histogram equalization is used to enhance the input liquid level gauge image to obtain an enhanced image.

[0105] Calculating the vertical edge: Using image processing methods, the level gauge is divided into three parts: the level column, the left scale panel, and the right scale panel (the level column refers to the area of ​​liquid level change in the middle of the level gauge, and the scale panel refers to the parts containing the scale readings on both sides of the level column). A schematic diagram of the level column and scale panel is shown below. Figure 10As shown, there is a strong vertical texture between the scale panel and the liquid level column. This embodiment of the invention extracts the vertical texture information for segmentation of the liquid level column and the scale panel. This embodiment of the invention uses the Sobel algorithm to perform a convolution operation on the enhanced image to obtain a texture activation map. The texture activation map satisfies the following characteristics: in the vertical direction of the original image, if there is a strong texture, the corresponding activation map has a high gray value; if the texture is weak, the corresponding activation map has a low gray value. The Sobel operator and convolution operation are as follows: Figure 11 As shown.

[0106] Adaptive thresholding segmentation: The fragmented vertical textures on the level gauge dial can affect the segmentation results. After obtaining the activation image, in order to eliminate noise in the image, image processing methods are used to remove texture noise. The Otsu thresholding method is used to segment the image to obtain a binary image, as follows: First, the adaptive thresholding segmentation algorithm is used to distinguish between strong and weak vertical textures. Then, morphological processing methods are used to extract vertical textures exceeding a fixed length (this value is 50) to obtain a strong vertical texture activation image.

[0107] Morphological processing: After obtaining the binary image, the activation values ​​at the scale digits of the level gauge image may be high, thus affecting the segmentation effect. To improve the robustness of the algorithm, this project performs morphological processing on the binary image. The steps of morphological processing are as follows: Figure 12 As shown: Set the template size to 1*50 (width*height), and set the internal values ​​of the template to 1; sequentially traverse the adaptive segmentation image, and use the template to perform a logical AND operation on the adaptive segmentation image, where && represents the logical AND, and the small square in front of the logical AND is the image, for example, a 1*50 pixel image, to obtain the processed image.

[0108] The liquid level column and scale panel are statistically segmented: In the horizontal direction, the coordinates of the scale numbers near the center of the scale panel almost coincide with the dividing line between the liquid level column and the scale panel. Therefore, the dividing line is determined by comprehensively considering the strong vertical texture activation map and the text detection results, thus better segmenting the liquid level column and the scale panel. After obtaining the morphological image, the texture intensity of each column of the image needs to be statistically analyzed. The detailed steps are as follows: Statistical histograms are obtained by summing the pixels in each column of the morphological image; then, the mean and variance of the histogram are calculated, and the threshold is obtained by subtracting the variance from the mean; for each column value in the histogram, if its value is less than the threshold, it is set to 0, otherwise it is set to 1, resulting in a binary histogram. Ideally, the binary histogram should look like this: Figure 13 As shown. To eliminate the uncertainty of statistical values, the dividing lines between the liquid level column and the scale panel are extracted by combining the results of binary histogram and text detection of scale numbers, as shown. Figure 14As shown, ideally, the text detection results are located on both sides of the liquid level column. First, select the detection box located on the left side of the image center, and calculate the mean horizontal coordinates of the left and right edges of each detection box. Then, select the detection box located on the right side of the image center, and calculate the mean horizontal coordinates of the left and right edges of each detection box. Finally, sort the four means in ascending order to obtain four base values, i.e. Figure 14 [A,B,C,D]; For the coordinates of the second cardinal (B), which is usually the right coordinate of the detection box to the left of the image center, calculate the x-coordinate of the nearest binary histogram with a value of 1, and use it as the left dividing line; For the coordinates of the third cardinal (C), which is usually the left coordinate of the detection box to the left of the image center, calculate the x-coordinate of the nearest binary histogram with a value of 1, and use it as the right dividing line; Based on the coordinates of the left and right dividing lines, the liquid level instrument image is divided into three parts: the left scale panel, the liquid level column, and the right scale panel.

[0109] Liquid level height extraction: After obtaining the liquid level column, the liquid level height is extracted using image processing methods. The liquid level height is as follows: Figure 10 As shown, liquid level column images are typically RGB images, where R represents red, G represents green, and B represents blue. The liquid level column image is primarily composed of red and white. Therefore, this method determines the liquid level height (the vertical coordinate of the liquid level line in the image) by comparing the relative relationships of the color channels in each row of the image, as detailed below: 1. For each row of the image, calculate the sum of the R components, G components, and B components from top to bottom; 2. For the current row, multiply the smaller value between the sum of the G components and the sum of the B components by 1.5 to obtain the baseline value. If the sum of the R components is greater than the baseline value, and the sum of the G components and the sum of the B components are both smaller than the sum of the R components, then the horizontal coordinate of the current row is the liquid level height, and the calculation ends. 3. If condition 2 is not met, select the next row and repeat steps 1 and 2 until the bottom of the image.

[0110] Extracting key tick marks from the left tick panel: After obtaining the left tick panel, extract its key tick marks. Since the left tick panel contains both red tick marks and black tick marks, this embodiment of the invention uses a method combining color and texture information to extract the red tick marks. Figure 15As shown. The detailed steps of the key scale line extraction algorithm are as follows: Extracting the red component: Scanning from left to right and from top to bottom, extracting the color information of each pixel. If the R value is greater than both the G and B values, then the value of that position in the color activation map is 255; otherwise, it is 0. Extracting texture information: Converting the image to grayscale, firstly, using the horizontal Sobel operator to extract horizontal edge information, then performing adaptive thresholding, and finally performing morphological processing (at this time, the morphological processing direction is horizontal, opposite to the direction in the morphological processing steps) to obtain a texture binary map. Finally, performing a bitwise AND operation between the color activation map and the texture binary map yields the final horizontal red component activation map. At this point, the highlighted areas of the red component activation map are those that simultaneously satisfy the condition of having the largest R value and strong horizontal texture edges.

[0111] Statistics: After obtaining the horizontal red component activation map, a histogram can be calculated based on the grayscale values ​​of the activation map, using image rows as units. The method for calculating the histogram is the same as the statistical method used in the steps of statistically analyzing and segmenting the level column and scale panel. For level gauges, under ideal conditions, a statistical histogram with 11 peaks can be obtained.

[0112] Numerical calculation includes three steps: First, the reading scale position is extracted using the text detection results and the statistical histogram obtained in the statistical step; then, the actual range between the reading scales (representing the true meaning of the distance between two scales) is obtained based on the text detection and text recognition results; finally, the reading of the level gauge is obtained using the liquid level height position, key scale line position, and actual range obtained in the liquid level extraction step. The detailed process is as follows... Figure 16As shown. 1. In this embodiment of the invention, the text detection results of the scale numbers are used to extract the reading scale position, as detailed below: First, the text detection boxes are sorted according to coordinates, and then the text detection boxes are traversed in a top-to-bottom order. For each text detection box, the position with a value of 1 in the statistical histogram and the closest vertical coordinate to the center of the text detection box is taken as the scale line position. The above steps are repeated until the number of reading scale line positions is consistent with the number of text detection boxes. Finally, the mean value of the distance difference between adjacent reading scale lines is calculated, and the standard interval length is set to I. 2. After sorting the text detection boxes, the range difference between adjacent text detection boxes is first calculated. Specifically, the text recognition results of adjacent text detection boxes are extracted, converted into numbers, and the absolute difference is calculated (in this embodiment of the invention, the text recognition result at the top of the level gauge is "1m", and in this step, the text recognition result is converted to 100). Finally, the mode of the absolute difference is selected as the actual range, set as M. 3. Compare the liquid level height obtained in the liquid level height extraction step with the reading scale position. Initialize the counter to 0. Compare the liquid level height position with the scale position in descending order. If the scale position is greater than the liquid level height position, increment the counter by 1 and continue comparing subsequent scale positions. If the scale position is less than the liquid level height position, calculate the vertical coordinate difference between the liquid level height and the previous scale position, and set it as x. If the counter value is n at this time, the final reading result is M*(n+x / I).

[0113] This invention also provides a liquid level gauge reading system, such as... Figure 17 As shown, the reading system includes:

[0114] The segmentation module 171 is used to perform image segmentation processing on the instrument image based on the vertical texture information of the liquid level instrument image, so as to determine the scale panel area and the liquid level column area in the instrument image;

[0115] The recognition module 172 is used to perform text recognition on the scale panel area to obtain the scale number recognition result;

[0116] The processing module 173 is used to perform image processing on the liquid level column area to obtain the liquid level height;

[0117] The determination module 174 is used to determine the reading result of the liquid level instrument based on the liquid level height and the scale number recognition result.

[0118] In one embodiment, the segmentation module is further configured to:

[0119] The Sobel algorithm is used to perform a convolution operation on the liquid level instrument image to obtain a texture activation map; wherein, the vertical texture information includes the texture activation map;

[0120] The Otsu thresholding method was used to perform image segmentation on the texture activation map to obtain the image segmentation result;

[0121] Morphological processing is performed on the image segmentation results to obtain a morphological image;

[0122] The scale panel area and liquid level column area in the instrument image are determined based on the texture intensity of the morphological image.

[0123] In one embodiment, the reading system further includes a detection module for: acquiring the original image of the level gauge;

[0124] The original image is input into the instrument detection model to extract the liquid level instrument image from the original image based on the first output result of the instrument detection model; wherein, the instrument detection model includes a feature extraction sub-model, a first region generation network, and a second region generation network; the feature extraction sub-model is used to perform general feature extraction on the original image, the first region generation network is used to determine candidate regions of the liquid level instrument in the original image based on the general features, and the second region generation network is used to perform feature extraction, category classification, and bounding box regression on the candidate regions, and output the first output result.

[0125] Optionally, the feature extraction sub-model includes a first convolutional neural network and a pyramid network; the first convolutional neural network is used to extract features from the original image to obtain feature maps of different sizes; the pyramid network is used to fuse the feature maps of different sizes to obtain the general features of the original image.

[0126] In one embodiment, the recognition module is further configured to: input the scale panel region into a text detection model to determine a text probability map and an adaptive threshold map of the scale panel region based on the output of the text detection model; wherein, the text detection model includes a text feature extraction network and a differentiable binarization network; the text feature extraction network is configured to perform differentiable binarization processing on the text feature map to obtain a text probability map and an adaptive threshold map;

[0127] The text probability map and the adaptive threshold map are binarized to obtain an approximate binary map corresponding to the original image;

[0128] The binary image is processed to obtain the text region in the scale panel area; wherein the text region is a bounded polygon image with numbers;

[0129] Text recognition is performed on the text region.

[0130] In one embodiment, the identification module is further used for:

[0131] The text region is input into a text recognition model to determine the scale number recognition result based on the second output result of the text recognition model; wherein, the text recognition model includes a second convolutional neural network and a recurrent neural network;

[0132] The second convolutional neural network is used to extract image features of the text region;

[0133] The recurrent neural network is used to obtain the sequence information of the text region based on the image features, and to perform CTC encoding based on the sequence information to obtain the scale number recognition result.

[0134] In one embodiment, the determining module is further configured to:

[0135] The scale line recognition result in the scale panel area is determined based on the text area;

[0136] Based on the text area and the scale line recognition results, the range between the scales in the scale panel area is determined;

[0137] The reading result of the instrument is determined based on the liquid level height, the scale line recognition result, and the range.

[0138] In one embodiment, the determining module is further configured to:

[0139] The text regions are sorted according to their coordinates, and then traversed from top to bottom.

[0140] The recognition result of the scale lines in the scale panel area is determined based on the vertical coordinate of the center of the text area.

[0141] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0142] This invention also provides an electronic device such as Figure 18 As shown, it includes a memory, a processor, and a computer program stored in the memory and used to run on the processor. When the processor executes the computer program, it implements the liquid level gauge reading method described in any of the above embodiments. Figure 18The electronic device 180 shown is merely an example and should not be construed as limiting the functionality or scope of the embodiments of the present invention. Figure 18 As shown, the electronic device 180 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 180 may include, but are not limited to: at least one processor 181, at least one memory 182, and a bus 183 connecting different system components (including memory 182 and processor 181).

[0143] Bus 183 includes a data bus, an address bus, and a control bus.

[0144] The memory 182 may include volatile memory, such as random access memory (RAM) 1821 and / or cache memory 1822, and may further include read-only memory (ROM) 1823.

[0145] The memory 182 may also include a program tool 1825 (or utility) having a set (at least one) program module 1824, such program module 1824 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0146] The processor 181 executes various functional applications and data processing by running computer programs stored in the memory 182, such as the liquid level instrument reading method described in any of the above embodiments.

[0147] Electronic device 180 can also communicate with one or more external devices 184. This communication can be performed via input / output (I / O) interface 185. Furthermore, the model-generated electronic device 180 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 186. Figure 18 As shown, network adapter 186 communicates with other modules of electronic device 180 via bus 183. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with electronic device 180, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0148] This invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements the liquid level gauge reading method provided in any of the above embodiments.

[0149] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.

Claims

1. A method for reading a liquid level instrument, characterized in that, The recognition method includes: The instrument image is segmented based on the vertical texture information of the liquid level gauge image to determine the scale panel area and the liquid level column area in the instrument image; Text recognition is performed on the scale panel area to obtain the scale number recognition result; Image processing is performed on the liquid level column area to obtain the liquid level height; Based on the liquid level height and the scale digit recognition result, the reading result of the liquid level instrument is determined; The step of performing image segmentation processing on the instrument image based on the vertical texture information of the liquid level gauge image to determine the scale panel area and the liquid level column area in the instrument image includes: The Sobel algorithm is used to perform a convolution operation on the liquid level instrument image to obtain a texture activation map; wherein, the vertical texture information includes the texture activation map; The Otsu thresholding method was used to perform image segmentation on the texture activation map to obtain the image segmentation result; Morphological processing is performed on the image segmentation results to obtain a morphological image; The scale panel area and liquid level column area in the instrument image are determined based on the texture intensity of the morphological image; The text recognition process for the scale panel area includes: The scale panel area is input into a text detection model to determine the text probability map and adaptive threshold map of the scale panel area based on the output of the text detection model; wherein, the text detection model includes a text feature extraction network and a differentiable binarization network; the text feature extraction network is used to perform differentiable binarization processing on the text features to obtain the text probability map and adaptive threshold map; The text probability map and the adaptive threshold map are binarized to obtain an approximate binary map corresponding to the original image. The binary image is processed to obtain the text region in the scale panel area; wherein the text region is a bounded polygon image with numbers; Perform text recognition on the text region; The text recognition of the text region includes: The text region is input into a text recognition model to determine the scale number recognition result based on the second output result of the text recognition model; wherein, the text recognition model includes a second convolutional neural network and a recurrent neural network; The second convolutional neural network is used to extract image features of the text region; The recurrent neural network is used to obtain the sequence information of the text region based on the image features, and to perform CTC encoding based on the sequence information to obtain the scale number recognition result.

2. The method for reading a liquid level instrument as described in claim 1, characterized in that, Before the step of performing image segmentation processing on the instrument image based on the vertical texture information of the liquid level instrument image, the method further includes: Acquire the original image of the liquid level gauge; The original image is input into the instrument detection model to extract the liquid level instrument image from the original image based on the first output result of the instrument detection model; wherein, the instrument detection model includes a feature extraction sub-model, a first region generation network, and a second region generation network; the feature extraction sub-model is used to perform general feature extraction on the original image, the first region generation network is used to determine candidate regions of the liquid level instrument in the original image based on the general features, and the second region generation network is used to perform feature extraction, category classification, and bounding box regression on the candidate regions, and output the first output result.

3. The method for reading a liquid level instrument as described in claim 2, characterized in that, The feature extraction sub-model includes a first convolutional neural network and a pyramid network; the first convolutional neural network is used to extract features from the original image to obtain feature maps of different sizes; the pyramid network is used to fuse the feature maps of different sizes to obtain the general features of the original image.

4. The method for reading a level gauge as described in claim 1, characterized in that, The step of determining the reading result of the level gauge based on the liquid level height and the scale digit recognition result includes: The scale line recognition result in the scale panel area is determined based on the text area; Based on the text area and the scale line recognition results, the range between the scales in the scale panel area is determined; The reading result of the instrument is determined based on the liquid level height, the scale line recognition result, and the range.

5. The method for reading a liquid level instrument as described in claim 4, characterized in that, The step of determining the scale line recognition result in the scale panel area based on the text area includes: The text regions are sorted according to their coordinates, and then traversed from top to bottom. The recognition result of the scale lines in the scale panel area is determined based on the vertical coordinate of the center of the text area.

6. A reading system for a liquid level instrument, characterized in that, The recognition system includes: The segmentation module is used to perform image segmentation processing on the instrument image based on the vertical texture information of the liquid level instrument image, so as to determine the scale panel area and the liquid level column area in the instrument image; The recognition module is used to perform text recognition on the scale panel area to obtain the scale number recognition result; The processing module is used to perform image processing on the liquid level column area to obtain the liquid level height; The determination module is used to determine the reading result of the liquid level instrument based on the liquid level height and the scale number recognition result; The segmentation module is further configured to: perform a convolution operation on the liquid level instrument image using the Sobel algorithm to obtain a texture activation map; wherein, the vertical texture information includes the texture activation map; perform image segmentation processing on the texture activation map using the Otsu thresholding method to obtain an image segmentation result; perform morphological processing on the image segmentation result to obtain a morphological image; and determine the scale panel region and liquid level column region in the instrument image based on the texture intensity of the morphological image. The recognition module is further configured to: input the scale panel region into a text detection model to determine a text probability map and an adaptive threshold map of the scale panel region based on the output of the text detection model; wherein, the text detection model includes a text feature extraction network and a differentiable binarization network; the text feature extraction network is used to perform differentiable binarization processing on the text features to obtain a text probability map and an adaptive threshold map; perform binarization processing on the text probability map and the adaptive threshold map to obtain an approximate binary map corresponding to the original image; perform image processing on the binary map to obtain a text region in the scale panel region; wherein, the text region is a bounded polygon image with numbers; and perform text recognition on the text region. The recognition module is further configured to: input the text region into a text recognition model to determine the scale number recognition result based on the second output result of the text recognition model; wherein, the text recognition model includes a second convolutional neural network and a recurrent neural network; the second convolutional neural network is used to extract image features of the text region; the recurrent neural network is used to obtain sequence information of the text region based on the image features, and to perform CTC encoding based on the sequence information to obtain the scale number recognition result.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and for running on the processor, characterized in that, When the processor executes the computer program, it implements the method for reading the level gauge as described in any one of claims 1-5.

Images