Full-automatic pointer type meter counting value identification method based on OCR and key point detection

Abstract

The invention provides a full-automatic pointer type meter counting value recognition method based on OCR and key point detection. A system of the method comprises a meter detection module, a key point detection module, a digital recognition module and a reading analysis module. The method comprises the following steps: step 1, training neural network models in a meter detection module, a key point detection module and a digital recognition module in advance; step 2, acquiring and inputting an image containing a to-be-identified pointer meter, and detecting a dial plate area of the to-be-identified meter by using a meter detection module; 3, using a key point detection module to detect a pointer rotation axis point, a pointer tail end point, a dial scale starting point and a dial scale ending point for the identified dial region sub-image; step 4, detecting coordinates and numerical values of all scale numbers of the recognized dial region sub-image by using an OCR (Optical Character Recognition) algorithm; and step 5, according to the obtained forehead key point information and the scale reading information, using a reading analysis module to obtain a final meter reading.

Description

Technical Field

[0001] The invention relates to the technical field of pointer meter count value image recognition, and in particular to a fully automatic pointer meter count value recognition method based on OCR and key point detection. Background Art

[0002] There are a large number of meters in industrial production environments, and being able to monitor the status of these meters in a timely manner is very important for safe production. Although networked electronic meter technology has matured, in many scenarios, due to environmental factors such as electromagnetic interference, traditional pointer-type meters are still widely used. The monitoring of these meters still relies on manual inspections and readings, which increases the workload of personnel on the one hand, and makes it difficult to detect abnormalities in the first place on the other hand, increasing the risk of system failure.

[0003] At present, image recognition technology is developing rapidly, and many methods for identifying pointer meter readings based on camera videos have been proposed. However, these methods still rely partially on human participation when used, and there is no fully automatic method yet. For example, the "Dial Positioning and Automatic Reading Pointer Meter Count Recognition Method and System" (Patent Publication No. CN112508098B) uses a key point detection network to obtain the key point position in the entire image, and uses the key point position to infer the meter position; the key point information is input into the indication reading network to obtain the pointer deflection angle information; the reading is calculated based on the pointer deflection angle information and the pre-configured meter range; but it performs key point detection on the entire image, and then groups the key points belonging to the same meter according to the positional relationship; if the meter occupies too small an area of ​​the entire image, the key point detection will be greatly reduced. The accuracy of point detection is reduced, thereby increasing the numerical error of the final recognition; the algorithm obtains the rotation angle of the pointer from the image analysis, and it also needs to cooperate with the pre-configured meter range to calculate the actual reading of the meter. Each meter needs to be manually marked once, and the flexibility is poor; such as: "A pointer meter recognition method and device based on image" (patent number and publication number CN116071604A); the dial position of the pointer meter is obtained by using the target detection algorithm; the dial is segmented into a scale image, a pointer image, and a text image using a segmentation algorithm; the center and radius of the meter are fitted according to the scale image, and the The algorithm calculates the angle between the pointer and the nearest scale area; however, it uses a segmentation algorithm to obtain scales, pointers, numbers and other areas from the detected dial; the difficulty and manpower consumption of annotating segmented data are much greater than annotating key points and detection frames, which means that the cost of training a good segmentation model is very high, and if the types of meters in the application scenario increase, the cost will be further magnified; the scales segmented by the algorithm must be scales that match the numbers, but the number of scales on the dial is usually greater than the number of numbers, and the scales are all short segments. The difference between the scales corresponding to the numbers and other scales is also very small, and it is difficult to ensure that the trained model can output The algorithm does not provide a corresponding solution for the scale that completely matches the numbers. In fact, this problem is also difficult to handle. For this reason, the applicant proposes a fully automatic pointer meter count value recognition method based on OCR and key point detection according to the needs of pointer meter count value recognition. The target detection algorithm and the OCR algorithm are used to automatically identify the scale value of the pointer meter, and the center point of the scale digital area is used as the scale point of the dial. The linear constraints inherent in the scale value and the scale angle are used to calculate the fitting function of the scale value and the scale angle. There is no need to mark the scale range for each meter to be identified, which can reduce the workload of personnel and improve practicality. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a fully automatic pointer meter count recognition method based on OCR and key point detection. The system of the method includes a meter detection module, a key point detection module, a digital recognition module and a reading analysis module; a text detection model and an OCR algorithm are used to automatically identify the scale readings of the pointer meter to be identified, and there is no need to mark the scale range for each meter to be identified, which can reduce the workload of personnel and improve practicality; during identification, only the scale numbers need to be detected, avoiding the cost of pre-training the scale point detection model, and using the inherent constraints of the scale points and scale values ​​to estimate the function of the angle and the scale, avoiding the disadvantage of the difficulty of scale point detection, and the recognition is fast and accurate.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A fully automatic pointer meter count value recognition method based on OCR and key point detection is characterized in that: the system of the fully automatic pointer meter count value recognition method based on OCR and key point detection includes a meter detection module, a key point detection module, a digital recognition module and a reading analysis module; the recognition method steps are:

[0007] Step 1: Pre-train the neural network models in the meter detection module, key point detection module, and digital recognition module;

[0008] Step 2: collect input images containing pointer meters to be identified, and use a meter detection module to detect the dial area of ​​the meter to be identified;

[0009] Step 3: Use the key point detection module to detect the pointer rotation axis point, pointer end point, dial scale start point and dial scale end point of the identified dial area sub-image;

[0010] Step 4: Use the OCR algorithm to detect the coordinates and values ​​of all scale numbers in the identified dial area sub-image;

[0011] Step 5: Obtain the key point information and scale reading information according to the steps, and use the reading analysis module to obtain the final meter reading.

[0012] Furthermore, the specific process of step 1 of the fully automatic pointer meter count value recognition method based on OCR and key point detection is:

[0013] 1) Using a fixed camera and a handheld device, collect a set of images I containing various pointer meters;

[0014] 2) For each image I i Manually mark the rectangular boxes of the dial areas of all pointer meters, and the marked set is The coordinates of the upper left corner and the lower right corner of the j-th pointer of the i-th picture;

[0015] 3) Use the yolov5 algorithm to train the pointer meter detection model M1 using the image dataset I and the label dataset B;

[0016] 4) Using the image dataset I and the label dataset B, for each B i ∈B, from image I i Take the sub-image from And mark the pointer rotation axis point, pointer end point, dial scale starting point, and dial scale end point in each picture;

[0017] 5) Use a collection of images and the key point label set in 4), using the HRNet algorithm to train the key point detection model M2;

[0018] 6) Image Mark each picture with a rectangular box containing all the scale numbers, and each rectangular box completely contains all the numbers on the scale reading, including the positive and negative signs and the decimal point;

[0019] 7) Using image datasets and the label set in 6), and use the yolov5 algorithm to train the scale digit detection model M3.

[0020] Furthermore, the specific process of step 2 in the fully automatic pointer meter count value recognition method based on OCR and key point detection is:

[0021] 1) Input the pointer meter image to be identified, scale the image to 640*640 pixels, and multiply the value of each pixel by 1 / 255 to normalize the input image;

[0022] 2) Take the normalized input image and input it into the pointer meter detection model M1 to obtain the output of the network; the output format is a list containing multiple detection results, each of which contains a rectangular box of a pointer meter and a confidence level;

[0023] 3) Perform non-maximum suppression on the detection results, remove duplicate detection results, and remove results with confidence levels less than a certain threshold to obtain a rectangular frame of the pointer meter to be identified.

[0024] Furthermore, the specific process of step 3 in the fully automatic pointer meter count value recognition method based on OCR and key point detection is:

[0025] 1) Take the rectangular frame of the pointer meter to be identified obtained in step 2 and cut out the corresponding meter sub-image;

[0026] 2) Take the cropped meter sub-image, scale the image to 192*192 pixels, and multiply the value of each pixel by 1 / 255 to obtain the normalized input image;

[0027] 3) Input the normalized input image into the key point detection model M2 to obtain the pointer rotation axis point, pointer end point, dial scale starting point, and dial scale end point, which are recorded as P and c ,P m ,P s ,P e .

[0028] Furthermore, the specific process of step 4 in the fully automatic pointer meter count value recognition method based on OCR and key point detection is:

[0029] 1) Take the meter sub-image cropped in step 3, scale the image to 640*640 pixels, and multiply the value of each pixel by 1 / 255 to obtain the normalized input image;

[0030] 2) Input the normalized input image into the scale digit detection model M3 to obtain the network output; the output format is a list containing multiple detection results, each of which contains a rectangular box of a scale digit and a confidence level;

[0031] 3) Take the test result Bs in 2), and for each Bs=[x min ,y min ,x max ,y max ], calculate the coordinates of its center point as:

[0032]

[0033] Calculate the Euclidean distance from the center point of all digital rectangular frames to the rotation axis of the pointer, remove the digital rectangular frames with abnormal distances, and obtain the set of real scale digital frames;

[0034] 4) Take the real scale digital outer frame in 3), cut out the scale digital sub-image from the meter sub-image in step 3 for each outer frame, apply the OCR algorithm to recognize its text content, and convert it into a numerical value.

[0035] Furthermore, the specific process of step 5 in the fully automatic pointer meter count value recognition method based on OCR and key point detection is:

[0036] 1) Take the four key points P in step 3 c ,P m ,P s ,P e , vector v0 = P s -P c , vector v1 = Pm -P c , then the rotation angle of the pointer is the angle between vectors v1 and v0;

[0037] 2) Take the center point and scale value of the scale number rectangle in step 4, and use the least squares method to calculate the fitting straight line between the center point angle and the scale value, with the horizontal axis being the scale value and the vertical axis being the angle of the scale number;

[0038] 3) Take the linear function of the angle and scale value in 2), calculate its inverse function, that is, the function of the value with respect to the angle, substitute the pointer rotation angle in 1), and the final reading of the pointer meter is obtained.

[0039] The benefits of this application are:

[0040] 1. The fully automatic pointer meter reading recognition method based on OCR and key point detection uses the text detection model and OCR algorithm to automatically identify the scale reading of the pointer meter to be identified. There is no need to mark the scale range for each meter to be identified, which can reduce the workload of personnel and improve practicality;

[0041] 2. The fully automatic pointer meter counter value recognition method based on OCR and key point detection only needs to detect the scale numbers, avoiding the cost of training the scale point detection model in the early stage;

[0042] 3. The fully automatic pointer meter counter value recognition method based on OCR and key point detection uses the inherent constraints of scale points and scale values ​​to estimate the function of angle and scale, avoiding the disadvantage of difficult scale point detection;

[0043] 4. The fully automatic pointer meter count value recognition method based on OCR and key point detection uses the center point of the scale digital frame to calculate the angle of the scale, reducing the parameters of the algorithm model and the difficulty of preliminary training. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the algorithm flow of the present invention;

[0045] Figure 2 This is an example diagram of the algorithm of the present invention. DETAILED DESCRIPTION

[0046] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0047] like Figure 1-2As shown, a fully automatic pointer meter count value recognition method based on OCR and key point detection is shown, which is characterized in that: the system of the fully automatic pointer meter count value recognition method based on OCR and key point detection includes a meter detection module, a key point detection module, a digital recognition module and a reading analysis module; the steps of the recognition method are:

[0048] Step 1: Train the neural network models in the meter detection module, key point detection module, and digital recognition module in advance; the specific process is as follows:

[0049] 1) Using a fixed camera and a handheld device, collect a set of images I containing various pointer meters;

[0050] 2) For each image I i Manually mark the rectangular boxes of the dial areas of all pointer meters, and the marked set is The coordinates of the upper left corner and the lower right corner of the j-th pointer of the i-th picture;

[0051] 3) Use the yolov5 algorithm to train the pointer meter detection model M1 using the image dataset I and the label dataset B;

[0052] 4) Using the image dataset I and the label dataset B, for each B i ∈B, from image I i Take the sub-image from And mark the pointer rotation axis point, pointer end point, dial scale starting point, and dial scale end point in each picture;

[0053] 5) Use a collection of images and the key point label set in 4), using the HRNet algorithm to train the key point detection model M2;

[0054] 6) Image Mark each picture with a rectangular box containing all the scale numbers, and each rectangular box completely contains all the numbers on the scale reading, including the positive and negative signs and the decimal point;

[0055] 7) Using image datasets and the label set in 6), use the yolov5 algorithm to train the scale digit detection model M3;

[0056] Step 2: collect input images containing pointer meters to be identified, and use meter detection modules to detect the dial area of ​​the meter to be identified; the specific process is as follows:

[0057] 1) Input the pointer meter image to be identified, scale the image to 640*640 pixels, and multiply the value of each pixel by 1 / 255 to normalize the input image;

[0058] 2) Take the normalized input image and input it into the pointer meter detection model M1 to obtain the output of the network; the output format is a list containing multiple detection results, each of which contains a rectangular box of a pointer meter and a confidence level;

[0059] 3) Perform non-maximum suppression on the detection results, remove duplicate detection results, and remove results with confidence levels less than a certain threshold to obtain a rectangular frame of the pointer meter to be identified

[0060] Step 3: Use the key point detection module to detect the pointer rotation axis point, pointer end point, dial scale start point and dial scale end point of the identified dial area sub-image; the specific process is as follows:

[0061] 1) Take the rectangular frame of the pointer meter to be identified obtained in step 2 and cut out the corresponding meter sub-image;

[0062] 2) Take the cropped meter sub-image, scale the image to 192*192 pixels, and multiply the value of each pixel by 1 / 255 to obtain the normalized input image;

[0063] 3) Input the normalized input image into the key point detection model M2 to obtain the pointer rotation axis point, pointer end point, dial scale starting point, and dial scale end point, which are recorded as P and c ,P m ,P s ,P e ;

[0064] Step 4: Use the OCR algorithm to detect the coordinates and values ​​of all scale numbers in the identified dial area sub-image;

[0065] The specific process is:

[0066] 1) Take the meter sub-image cropped in step 3, scale the image to 640*640 pixels, and multiply the value of each pixel by 1 / 255 to obtain the normalized input image;

[0067] 2) Input the normalized input image into the scale digit detection model M3 to obtain the network output; the output format is a list containing multiple detection results, each of which contains a rectangular box of a scale digit and a confidence level;

[0068] 3) Take the test result Bs in 2), and for each Bs=[x min ,y min ,x max ,y max ], calculate the coordinates of its center point as:

[0069]

[0070] Calculate the Euclidean distance from the center point of all digital rectangular frames to the rotation axis of the pointer, remove the digital rectangular frames with abnormal distances, and obtain the set of real scale digital frames;

[0071] 4) Take the real scale digital frame in 3), cut out the scale digital sub-image from the meter sub-image in step 3 for each frame, apply the OCR algorithm to recognize its text content, and convert it into a numerical value

[0072] Step 5: According to the key point information and scale reading information obtained in step 1, the final meter reading is obtained using the reading analysis module; the specific process is as follows:

[0073] 1) Take the four key points P in step 3 c ,P m ,P s ,P e , vector v0 = P s -P c , vector v1 = P m -P c , then the rotation angle of the pointer is the angle between vectors v1 and v0;

[0074] 2) Take the center point and scale value of the scale number rectangle in step 4, and use the least squares method to calculate the fitting straight line between the center point angle and the scale value, with the horizontal axis being the scale value and the vertical axis being the angle of the scale number;

[0075] 3) Take the linear function of the angle and scale value in 2), calculate its inverse function, that is, the function of the value with respect to the angle, substitute the pointer rotation angle in 1), and the final reading of the pointer meter is obtained.

[0076] The fully automatic pointer meter value recognition method based on OCR and key point detection shown in the figure uses the target detection algorithm and the OCR algorithm to automatically identify the scale value of the pointer meter, and takes the center point of the scale number area as the scale point of the dial, and uses the inherent linear constraints of the scale value and the scale angle to calculate the fitting function of the scale value and the scale angle; uses the OCR algorithm to automatically identify the scale value of the meter without the need to pre-configure the scale value; uses the center point of the scale number frame to calculate the angle of the scale, reducing the parameters of the algorithm model and the difficulty of preliminary training; when using it, there is no need to mark the scale range for each meter to be identified, which can reduce the workload of personnel and improve practicality; only the scale numbers need to be detected, avoiding the cost of preliminary training of the scale point detection model; uses the inherent constraints of the scale points and scale values ​​to estimate the function of the angle and the scale, avoiding the disadvantage of the difficulty of scale point detection, and has wide applicability and accurate and fast recognition.

[0077] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. A fully automatic pointer meter count value recognition method based on OCR and key point detection, characterized by: The system of the fully automatic pointer meter count value recognition method based on OCR and key point detection includes a meter detection module, a key point detection module, a digital recognition module and a reading analysis module; the recognition method steps are: Step 1: Train the neural network models in the meter detection module, key point detection module, and digital recognition module in advance; Step 2: collect input images containing pointer meters to be identified, and use a meter detection module to detect the dial area of ​​the meter to be identified; Step 3: Use the key point detection module to detect the pointer rotation axis point, pointer end point, dial scale start point and dial scale end point of the identified dial area sub-image; Step 4: Use the OCR algorithm to detect the coordinates and values ​​of all scale numbers in the identified dial area sub-image; Step 5: Obtain the key point information and scale reading information according to the steps, and use the reading analysis module to obtain the final meter reading.

2. The fully automatic pointer meter count value recognition method based on OCR and key point detection according to claim 1 is characterized in that: The specific process of step one is: 1) Using a fixed camera and a handheld device, collect a set of images I containing various pointer meters; 2) For each image I i Manually mark the rectangular boxes of the dial areas of all pointer meters, and the marked set is The coordinates of the upper left corner and the lower right corner of the j-th pointer of the i-th picture; 3) Use the yolov5 algorithm to train the pointer meter detection model M1 using the image dataset I and the label dataset B; 4) Using the image dataset I and the label dataset B, for each B i ∈B, from image I i Take the sub-image from And mark the pointer rotation axis point, pointer end point, dial scale starting point, and dial scale end point in each picture; 5) Use a collection of images and the key point label set in 4), using the HRNet algorithm to train the key point detection model M2; 6) Image Mark each picture with a rectangular box containing all the scale numbers, and each rectangular box completely contains all the numbers on the scale reading, including the positive and negative signs and the decimal point; 7) Using image datasets and the label set in 6), and use the yolov5 algorithm to train the scale digit detection model M3.

3. The fully automatic pointer meter count value recognition method based on OCR and key point detection according to claim 1 is characterized in that: The specific process of step 2 is as follows: 1) Input the pointer meter image to be identified, scale the image to 640*640 pixels, and multiply the value of each pixel by 1 / 255 to normalize the input image; 2) Take the normalized input image and input it into the pointer meter detection model M1 to obtain the output of the network; The output format is a list containing multiple test results, each of which contains a pointer meter rectangle and confidence level; 3) Perform non-maximum suppression on the detection results, remove duplicate detection results, and remove results with confidence levels less than a certain threshold to obtain a rectangular frame of the pointer meter to be identified.

4. The fully automatic pointer meter count value recognition method based on OCR and key point detection according to claim 1 is characterized in that: The specific process of step three is: 1) Take the rectangular frame of the pointer meter to be identified obtained in step 2 and cut out the corresponding meter sub-image; 2) Take the cropped meter sub-image, scale the image to 192*192 pixels, and multiply the value of each pixel by 1 / 255 to obtain the normalized input image; 3) Input the normalized input image into the key point detection model M2 to obtain the pointer rotation axis point, pointer end point, dial scale starting point, and dial scale end point, which are recorded as P and c ,P m ,P s ,P e .

5. The fully automatic pointer meter count value recognition method based on OCR and key point detection according to claim 1 is characterized in that: The specific process of step 4 is as follows: 1) Take the meter sub-image cropped in step 3, scale the image to 640*640 pixels, and multiply the value of each pixel by 1 / 255 to obtain the normalized input image; 2) Input the normalized input image into the scaled digit detection model M3 to obtain the output of the network; The output format is a list containing multiple test results, each of which contains a rectangular box with scale numbers and confidence level; 3) Take the test result Bs in 2), and for each Bs=[x min ,y min ,x max ,y max ], calculate the coordinates of its center point as: Calculate the Euclidean distance from the center point of all digital rectangular frames to the rotation axis of the pointer, remove the digital rectangular frames with abnormal distances, and obtain the set of real scale digital frames; 4) Take the real scale digital outer frame in 3), cut out the scale digital sub-image from the meter sub-image in step 3 for each outer frame, apply the OCR algorithm to recognize its text content, and convert it into a numerical value.

6. The fully automatic pointer meter count value recognition method based on OCR and key point detection according to claim 1 is characterized in that: The specific process of step five is as follows: 1) Take the four key points P in step 3 c ,P m ,P s ,P e , vector v0 = P s -P c , vector v1 = P m -P c , then the rotation angle of the pointer is the angle between vectors v1 and v0; 2) Take the center point and scale value of the scale number rectangle in step 4, and use the least squares method to calculate the fitting straight line between the center point angle and the scale value, with the horizontal axis being the scale value and the vertical axis being the angle of the scale number; 3) Take the linear function of the angle and scale value in 2), calculate its inverse function, that is, the function of the value with respect to the angle, substitute the pointer rotation angle in 1), and the final reading of the pointer meter is obtained.

Images