Digital display meter reading processing method and system, terminal and medium

Abstract

The invention discloses a digital display meter reading processing method and system, a terminal and a medium, and relates to the field of image processing, and the main points of the technical scheme are that all meter area images including a digital display area in a target image are extracted; extracting first mask information of a dial plate area in the meter area image by adopting a deep learning target detection algorithm; extracting second mask information of a reading area in the meter area image by adopting a text detection algorithm; analyzing a mask coverage degree between a mask area in the first mask information and a mask area in the second mask information, and screening out a common mask area exceeding a coverage degree threshold to form a mask reading area; and identifying a character sequence of each mask area in the mask reading area, and analyzing the character sequences to obtain a reading identification result. According to the method, part of interference outside the non-dial area can be filtered, especially false detection of non-texts outside the non-dial area is filtered out, and the accuracy and reliability of reading processing of the digital display meter are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and more specifically, to a method, system, terminal and medium for processing readings of digital display type meters. Background Art

[0002] There are many existing methods for reading digital meters, such as: matching characters in an input image through predefined character templates or samples; reading preconfigured character target boxes, classifying them one by one, and generating meter readings; detecting all individual characters in an image, and combining the individual characters to generate meter readings.

[0003] However, the template matching-based methods in the prior art can only process clear and regular fonts, have high requirements for character shape and layout, and have poor adaptability to characters of different fonts, sizes, and tilt angles. A large number of templates are required to cover all possible character changes, which increases the cost of calculation and storage. The single character classification method requires pre-labeling of all digital areas. The accuracy of the pre-labeling affects the recognition accuracy of the model; when the camera position changes, all point data needs to be reconfigured. When the number of meters is large, it often requires huge manpower and is difficult to maintain. In the method of detecting a single character, the detection of a single character will lose contextual information, such as the order and continuity of the numbers, which may lead to final recognition errors, especially when there is overlap or small spacing between numbers. The training data cannot cover all character types, and missing characters will also affect the detection accuracy of the target.

[0004] Therefore, how to research and design a digital meter reading processing method, system, terminal and medium that can overcome the above-mentioned defects is a problem that we urgently need to solve. Summary of the invention

[0005] In order to solve the deficiencies in the prior art, the purpose of the present invention is to provide a method, system, terminal and medium for processing digital meter readings, which adopts a text detection algorithm that does not require pre-training and has higher text recognition accuracy to identify and extract the second mask information of all text information in the meter area, and then adopts a pre-trained deep learning target detection algorithm to extract the first mask information of the dial area in the meter area image, and then filters out some interference outside the non-dial area according to the mask coverage between the mask area in the first mask information and the mask area in the second mask information, especially to complete the filtering out of non-text false detection outside the non-dial area, thereby improving the accuracy and reliability of digital meter reading processing.

[0006] The above technical objectives of the present invention are achieved through the following technical solutions:

[0007] In a first aspect, a method for processing digital meter readings is provided, comprising the following steps:

[0008] Extract all meter area images containing digital display areas in the target image;

[0009] A deep learning target detection algorithm is used to extract the first mask information of the dial area in the meter area image;

[0010] A text detection algorithm is used to extract the second mask information of the reading area in the meter area image;

[0011] Analyzing the mask coverage between the mask area in the first mask information and the mask area in the second mask information, and screening out all common mask areas whose mask coverage exceeds the coverage threshold to form a mask reading area;

[0012] The character sequences of each mask area in the mask reading area are identified, and the reading recognition result is obtained after parsing the character sequence.

[0013] Furthermore, the process of analyzing the mask coverage between the mask area in the first mask information and the mask area in the second mask information is specifically as follows:

[0014] Calculate the intersection area between the mask area A in the first mask information and the mask area B in the second mask information;

[0015] The mask coverage between the mask area A in the first mask information and the mask area B in the second mask information is calculated as the ratio of the intersection area to the area of ​​the mask area B.

[0016] Furthermore, the common mask area is specifically:

[0017] If the mask coverage between the mask area A in the first mask information and the mask area B in the second mask information is greater than the coverage threshold, then:

[0018] The intersection area between mask area A and mask area B is taken as the common mask area.

[0019] Furthermore, the first mask information is extracted using a meter detection model constructed based on deep learning target detection algorithm training;

[0020] Among them, deep learning target detection algorithms include YOLO and Faster R-CNN algorithms.

[0021] Furthermore, the second mask information is extracted using a text detection model constructed based on a text detection algorithm;

[0022] Among them, text detection algorithms include PSENet, EASTNET and DBNet algorithms.

[0023] Furthermore, the process of identifying the character sequence of each mask area in the mask reading area is specifically as follows:

[0024] Calculate the minimum bounding rectangle of each mask area in the mask reading area, and start from the rectangular center of the minimum bounding rectangle, and expand 25% of the width and height of the minimum bounding rectangle in four directions at the same time to obtain an enlarged bounding rectangle;

[0025] An affine transformation matrix is ​​calculated based on the enlarged bounding rectangle, and the mask area is rotated according to the affine transformation matrix to obtain a rotated mask area;

[0026] The rotated mask area is scaled proportionally to the set pixel size to obtain the final mask area;

[0027] The final masked area is input as an image into the pre-built reading recognition model for recognition analysis to obtain the character sequence of each masked area.

[0028] Furthermore, the method also uses a cyclic generative adversarial network to perform data enhancement on the data set for training the reading recognition model. The specific process is as follows:

[0029] Collect pictures of real meter readings, annotate the readings, and build a real data set;

[0030] Generate digital images corresponding to standard fonts based on the readings marked in the real data set to construct a standard data set;

[0031] The training cycle generates an adversarial model to transfer the style of the standard dataset to the real dataset;

[0032] Randomly generate several standard font digital images, and use the cyclic generative adversarial model to generate the final real dataset.

[0033] In a second aspect, a digital meter reading processing system is provided, the system is used to implement a digital meter reading processing method as described in any one of the first aspects, comprising:

[0034] Meter detection module, used to extract all meter area images containing digital display areas in the target image;

[0035] A dial detection module, used to extract first mask information of a dial area in a meter area image by using a deep learning target detection algorithm;

[0036] A reading detection module, used for extracting second mask information of a reading area in the meter area image by using a text detection algorithm;

[0037] A mask screening module, used for analyzing the mask coverage between the mask area in the first mask information and the mask area in the second mask information, and screening out all common mask areas whose mask coverage exceeds the coverage threshold to form a mask reading area;

[0038] The reading recognition module is used to recognize the character sequence of each mask area in the mask reading area, and obtain the reading recognition result after parsing the character sequence.

[0039] In a third aspect, a computer terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for processing digital meter readings as described in any one of the first aspects is implemented.

[0040] In a fourth aspect, a computer-readable medium is provided, on which a computer program is stored, and the computer program is executed by a processor to implement a digital meter reading processing method as described in any one of the first aspects.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. A method for processing readings of a digital meter provided by the present invention adopts a text detection algorithm that does not require pre-training and has a higher text recognition accuracy to identify and extract the second mask information of all text information in the meter area, and then adopts a pre-trained deep learning target detection algorithm to extract the first mask information of the dial area in the meter area image, and then according to the mask coverage between the mask area in the first mask information and the mask area in the second mask information, part of the interference outside the non-dial area is filtered out, especially completing the filtering of non-text false detection outside the non-dial area, thereby improving the accuracy and reliability of the processing of readings of digital meters;

[0043] 2. In order to solve the problem of difficulty in acquiring vertical data, the present invention uses a cyclic generative adversarial model to expand and enhance the training data of the reading recognition model. Compared with traditional sample data amplification processing, the present invention not only amplifies the background of the sample data, but also amplifies the content in the sample data, thereby improving the accuracy and generalization ability of the meter readings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

[0045] Figure 1 is a flow chart of Embodiment 1 of the present invention;

[0046] Figure 2 is a schematic diagram of meter area detection in Example 1 of the present invention;

[0047] Figure 3 is a schematic diagram of dial area detection in Example 1 of the present invention;

[0048] Figure 4 is a schematic diagram of the reading area detection in Example 1 of the present invention;

[0049] Figure 5 is a schematic diagram of the mask reading area in Example 1 of the present invention;

[0050] Figure 6 It is a system block diagram in Example 2 of the present invention. DETAILED DESCRIPTION

[0051] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0052] Embodiment 1: A method for processing digital meter readings, such as Figure 1 As shown, the following steps are included:

[0053] S1: Extract all meter area images containing digital display areas in the target image;

[0054] S2: Use deep learning target detection algorithm to extract the first mask information of the dial area in the meter area image;

[0055] S3: extracting the second mask information of the reading area in the meter area image using a text detection algorithm;

[0056] S4: analyzing the mask coverage between the mask area in the first mask information and the mask area in the second mask information, and screening out all common mask areas whose mask coverage exceeds the coverage threshold to form a mask reading area;

[0057] S5: Identify the character sequence of each mask area in the mask reading area, and obtain the reading recognition result after parsing the character sequence.

[0058] In step S1, an advanced deep learning target detection algorithm is used to train and build a meter detection model, such as YOLO, Faster R-CNN, etc., and then the target image is input into the meter detection model to obtain the location and category information of all meters under a complex background. Since the present invention is for digital display meters, it is necessary to filter out the meter area containing the digital display area based on the category information, and then extract the meter area image from the target image based on the location, such as Figure 2 As shown, the area marked by the target box in the figure is the meter area image.

[0059] In step S2, a deep learning target detection algorithm, such as YOLO, Faster R-CNN, etc., can be used to train and build a dial area detection model. Then, the meter area image in step S1 is input into the dial area detection model to obtain the first mask information of the dial area in the meter area image, such as Figure 3 As shown, the meter area is the panel area with digital recording and display function.

[0060] In step S3, the meter area image is input into a text detection model constructed based on a text detection algorithm, and the second mask information of the reading area in the meter area image can be extracted, such as Figure 4 As shown in Figure 2. Text detection algorithms include but are not limited to PSENet, EASTNET and DBNet algorithms.

[0061] For example, the constructed text detection model specifically adopts the text detection model based on the ResNet series network architecture under the PaddleOCR framework. The model introduces a differentiable binarization module to ensure high accuracy and stability when identifying digital meter readings;

[0062] DBNet (Differentiable Binarization Network) is a segmentation-based text detection algorithm. The algorithm realizes the function of automatic network training threshold by training threshold map and using differentiable binarization function. The output of DBNet includes probability map and threshold map, through which approximate binary map can be generated to obtain the location of text area.

[0063] PSENet (Progressive Scale Expansion Network) is a text detection algorithm based on pixel embedding and shape regression. The algorithm generates bounding boxes of text regions by gradually expanding them, effectively solving the problem of detecting text of arbitrary shapes. PSENet mainly consists of two key modules: Proposal Localization Network (PLN) and Proposal Refinement Network (PRN). PLN is responsible for generating preliminary text region candidate boxes, while PRN is responsible for refining and correcting the candidate boxes.

[0064] In addition, the present invention calculates the average pixel confidence of the mask area and filters the candidate masks with low confidence.

[0065] In step S4, the overlapping area ratios of all reading areas and the dial area may be calculated, and character objects outside the dial area may be filtered to obtain a final reading area.

[0066] The process of analyzing the mask coverage between the mask area in the first mask information and the mask area in the second mask information is specifically as follows: calculating the intersection area between the mask area A in the first mask information and the mask area B in the second mask information; and calculating the mask coverage between the mask area A in the first mask information and the mask area B in the second mask information by the ratio of the intersection area to the area of ​​the mask area B.

[0067] For example, Figure 3 The mask area in the first mask information has 3, and Figure 4 There are 7 mask areas in the second mask information. If the mask coverage of the mask area "6.509" in the first mask information and all the mask areas in the second mask information is less than 0.8, it means that the mask area "6.509" is an incorrect recognition and needs to be deleted from the first mask information. Otherwise, it means that it is correctly recognized and the mask area "6.509" can be used as part of the mask reading area.

[0068] The common mask area is specifically: if the mask coverage between the mask area A in the first mask information and the mask area B in the second mask information is greater than the coverage threshold, then the intersection area between the mask area A and the mask area B is used as the common mask area.

[0069] through Figure 3 The first mask information in Figure 4 The mask reading area formed by the second mask information is as follows Figure 5 shown.

[0070] In step S5, the process of identifying the character sequence of each mask area in the mask reading area is specifically as follows:

[0071] S51: Calculate the minimum bounding rectangle of each mask area in the mask reading area, and start from the rectangle center of the minimum bounding rectangle, expand 25% of the width and height of the minimum bounding rectangle in four directions at the same time, to obtain an expanded bounding rectangle;

[0072] S52: calculating an affine transformation matrix based on the enlarged circumscribed rectangle, and rotating the mask region according to the affine transformation matrix to obtain a rotated mask region;

[0073] S53: scaling the rotated mask area to a set pixel size in equal proportion to obtain a final mask area;

[0074] S54: The final mask area is input as an image into a pre-built reading recognition model for recognition analysis to obtain a character sequence of each mask area.

[0075] For example, according to the height of the reading rectangle, the rotated reading area is scaled proportionally so that the image height is 48 pixels and the width does not exceed 320 pixels. The area with a width less than 320 pixels is filled and sent to the reading recognition model for analysis; according to the reading category sequence of length 20 output by the reading recognition model, adjacent numbers are clustered to obtain the final recognition result. The output category index is 0 to 12, 0 represents the start symbol, 1 to 10 represents 10 numbers from 0 to 9, 11 represents the decimal point ".", and 12 represents the end symbol.

[0076] For example, the reading model outputs 0, 1, 1, 1, 2, 2, 1, 1, 1, 1, 12, 12... Among them, 0 is the start character and can be ignored during parsing, and 12 is the end character. The first 0 does not need to be parsed.

[0077] Several identical and consecutive characters are merged. Three consecutive 1s are parsed as a single "0", two consecutive 2s are parsed as a "1", and four consecutive 1s are parsed as a single "0". 12 is the terminator, and subsequent characters stop parsing and are finally merged into "010".

[0078] Considering that the content of digital meter readings generally changes only when a fault or abnormality occurs, it is difficult to obtain sample data. To this end, the present invention also uses a cyclic generative adversarial network to perform data enhancement on the data set of the training reading recognition model. The specific process is: collect real meter reading pictures, annotate the readings, and construct a real data set; generate digital pictures corresponding to standard fonts based on the readings annotated in the real data set, and construct a standard data set; train a cyclic generative adversarial model to perform style migration from the standard data set to the real data set; randomly generate several standard font digital pictures, and use the cyclic generative adversarial model to generate the final real data set.

[0079] It should be noted that standard fonts include but are not limited to Microsoft YaHei, Songti, etc.

[0080] Compared with traditional sample data augmentation processing, the present invention not only amplifies the background of the sample data, but also amplifies the content in the sample data, solving the problem of small data volume and single sequence in real data sets, and generating images similar to the real data but with different styles, thereby expanding the diversity and quantity of training data and improving the generalization ability of the model.

[0081] Embodiment 2: A digital meter reading processing system, the system is used to implement a digital meter reading processing method recorded in embodiment 1, such as Figure 6 As shown, it includes a meter detection module, a dial detection module, a reading detection module, a mask screening module and a reading recognition module.

[0082] Among them, the meter detection module is used to extract all meter area images containing digital display areas in the target image; the dial detection module is used to extract the first mask information of the dial area in the meter area image by using a deep learning target detection algorithm; the reading detection module is used to extract the second mask information of the reading area in the meter area image by using a text detection algorithm; the mask screening module is used to analyze the mask coverage between the mask area in the first mask information and the mask area in the second mask information, and screen out all common mask areas whose mask coverage exceeds the coverage threshold to form a mask reading area; the reading recognition module is used to identify the character sequence of each mask area in the mask reading area, and obtain the reading recognition result after parsing the character sequence.

[0083] The present invention also describes a computer terminal, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, a digital meter reading processing method as described in Example 1 is implemented.

[0084] The present invention also describes a computer-readable medium on which a computer program is stored. The computer program is executed by a processor to implement a digital meter reading processing method as described in Example 1.

[0085] Working principle: The present invention adopts a text detection algorithm that does not require pre-training and has higher text recognition accuracy to identify and extract the second mask information of all text information in the meter area, and then adopts a pre-trained deep learning target detection algorithm to extract the first mask information of the dial area in the meter area image, and then filters out some interference outside the non-dial area according to the mask coverage between the mask area in the first mask information and the mask area in the second mask information, especially to complete the filtering out of non-text false detections outside the non-dial area, thereby improving the accuracy and reliability of digital meter reading processing.

[0086] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0087] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0088] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0089] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0090] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for processing readings of a digital meter, characterized in that: The following steps are involved: Extract all meter area images containing digital display areas in the target image; A deep learning target detection algorithm is used to extract the first mask information of the dial area in the meter area image; A text detection algorithm is used to extract the second mask information of the reading area in the meter area image; Analyzing the mask coverage between the mask area in the first mask information and the mask area in the second mask information, and screening out all common mask areas whose mask coverage exceeds the coverage threshold to form a mask reading area; The character sequences of each mask area in the mask reading area are identified, and the reading recognition result is obtained after parsing the character sequence.

2. A method for processing digital meter readings according to claim 1, characterized in that: The process of analyzing the mask coverage between the mask area in the first mask information and the mask area in the second mask information is specifically as follows: Calculate the intersection area between the mask area A in the first mask information and the mask area B in the second mask information; The mask coverage between the mask area A in the first mask information and the mask area B in the second mask information is calculated as the ratio of the intersection area to the area of ​​the mask area B.

3. A method for processing digital meter readings according to claim 1, characterized in that: The common mask area is specifically: If the mask coverage between the mask area A in the first mask information and the mask area B in the second mask information is greater than the coverage threshold, the intersection area between the mask area A and the mask area B is used as the common mask area.

4. A method for processing digital meter readings according to claim 1, characterized in that: The first mask information is extracted using a meter detection model built based on deep learning target detection algorithm training; Among them, deep learning target detection algorithms include YOLO and Faster R-CNN algorithms.

5. A method for processing digital meter readings according to claim 1, characterized in that: The second mask information is extracted using a text detection model constructed based on a text detection algorithm; Among them, text detection algorithms include PSENet, EASTNET and DBNet algorithms.

6. A method for processing digital meter readings according to claim 1, characterized in that: The process of identifying the character sequence of each mask area in the mask reading area is specifically as follows: Calculate the minimum bounding rectangle of each mask area in the mask reading area, and start from the rectangular center of the minimum bounding rectangle, and expand 25% of the width and height of the minimum bounding rectangle in four directions at the same time to obtain an enlarged bounding rectangle; An affine transformation matrix is ​​calculated based on the enlarged bounding rectangle, and the mask area is rotated according to the affine transformation matrix to obtain a rotated mask area; The rotated mask area is scaled proportionally to the set pixel size to obtain the final mask area; The final masked area is input as an image into the pre-built reading recognition model for recognition analysis to obtain the character sequence of each masked area.

7. A method for processing digital meter readings according to claim 6, characterized in that: This method also uses a cyclic generative adversarial network to perform data augmentation on the dataset for training the reading recognition model. The specific process is as follows: Collect pictures of real meter readings, annotate the readings, and build a real data set; Generate digital images corresponding to standard fonts based on the readings marked in the real data set to construct a standard data set; The training cycle generates an adversarial model to transfer the style from the standard dataset to the real dataset; Randomly generate several standard font digital images, and use the cyclic generative adversarial model to generate the final real dataset.

8. A digital meter reading processing system, characterized in that: The system is used to implement a digital meter reading processing method as described in any one of claims 1 to 7, comprising: Meter detection module, used to extract all meter area images containing digital display areas in the target image; A dial detection module, used to extract first mask information of a dial area in a meter area image by using a deep learning target detection algorithm; A reading detection module, used for extracting second mask information of a reading area in the meter area image by using a text detection algorithm; A mask screening module, used for analyzing the mask coverage between the mask area in the first mask information and the mask area in the second mask information, and screening out all common mask areas whose mask coverage exceeds the coverage threshold to form a mask reading area; The reading recognition module is used to recognize the character sequence of each mask area in the mask reading area, and obtain the reading recognition result after parsing the character sequence.

9. A computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, a digital meter reading processing method as described in any one of claims 1 to 7 is implemented.

10. A computer readable medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement a digital meter reading processing method as described in any one of claims 1 to 7.

Images