A Method and System for Recognizing Pointer Instrument Readings in an Inspection Scenario

Abstract

The present invention discloses a method and system for recognizing the readings of pointer instruments in an inspection scenario. The method includes: acquiring an image of a pointer instrument; locating the pointer and the instrument in the pointer instrument image based on a positioning detection model, and locating the scale readings of the instrument in the image; fitting a dial image in the pointer instrument image by using an OCR direction detection model based on the above-positioned data; performing correction operations on the dial image based on plane rotation and spatial flipping to obtain a corrected dial image; performing pointer straight line fitting and digital recognition of the scale readings on the corrected dial image, and the digital recognition uses an OCR recognition model; calibrating the digital recognition result based on the prior information of the instrument; and obtaining the instrument reading based on the calibration result and the pointer straight line fitting result. The present invention can accurately perform instrument positioning and correction, ensuring the success rate and accuracy of the final reading of the pointer instrument.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent robots, and in particular relates to a method and system for identifying pointer instrument readings in an inspection scenario. Background Art

[0002] Pointer instruments are important indicating instruments and meters, and are widely used in industrial sites, such as pressure gauges, voltmeters, ammeters, thermometers, etc. These instruments are used in industrial sites to measure and indicate environmental parameters, production index parameters, equipment operation parameters, etc. Reading and recording the values ​​of these instruments, i.e., meter reading, is an essential part of the production process. In traditional industrial production sites, the reading of instrument values ​​is done by workers going to the site to copy, which requires a lot of manpower to complete this task. In addition, in some complex and dangerous production environments, manual interpretation and data recording are more difficult.

[0003] With the rapid development of image processing technology and the widespread application of highly integrated mobile inspection robots in industrial production sites, meter reading methods and approaches have undergone a major change. It uses the visual camera carried by the inspection robot to collect instrument pictures or videos at the production site, and then uses the relevant algorithms in the image processing field to analyze the instrument pictures, interpret and store them in the warehouse to complete the record. This method can save manpower to a great extent. The patent text (CN111598109A) provides a substation pointer meter reading intelligent recognition method, which obtains the instrument reading through traditional image processing technology (including image correction, grayscale, edge detection, circular transformation detection dial, Hough line detection detection pointer); the patent text (CN111931776 A) provides a dual-pointer instrument reading method based on deep learning, which uses deep learning methods to locate the instrument, and then performs Hough circle detection on the instrument image, and at the same time performs elliptical connected domain screening, and combines the two methods to obtain the circular dial of the instrument. The method is limited by the Hough circle detection method and is not suitable for target imaging rotation and flipping scenes, and is not well applied in practice.

[0004] In summary, the existing pointer meter automatic recognition methods all have certain defects. In actual machine room inspection applications, it is easy to be difficult to recognize the target image in different lighting, angles, and postures. It is almost difficult to perform effective recognition under conditions such as partial occlusion, blurred imaging, and image tilt, so it is difficult to meet the requirements of actual applications. Summary of the invention

[0005] In view of the defects existing in the above-mentioned prior art, the present invention provides a method for identifying pointer instrument readings in an inspection scenario, comprising the following steps:

[0006] Step S101, obtaining a pointer instrument image;

[0007] Step S103: Locate the pointer and the instrument in the pointer instrument image based on the positioning detection model, and locate the scale reading of the instrument in the image;

[0008] Step S105: Based on the above-positioned data, use the OCR direction detection model to fit the dial image in the pointer instrument image;

[0009] Step S107: Perform correction operations on the dial image based on plane rotation and space flipping to obtain the corrected dial image;

[0010] Step S109: Based on the corrected dial image, perform pointer straight line fitting and digital recognition of the scale reading. The digital recognition uses the OCR recognition model;

[0011] Step S111: Calibrate the digital recognition result based on the prior information of the instrument;

[0012] Step S113: Obtain the instrument reading based on the calibration result and the pointer straight line fitting result.

[0013] Among them, the positioning detection model and the OCR direction detection model are trained using N sample images with target instruments collected historically.

[0014] Among them, the step S105 includes:

[0015] According to the inclusion relationship between the positions of the detected instrument, pointer, and scale reading in the image, screen and associate the detected scale reading and pointer with the instrument, and eliminate the detection results outside the target.

[0016] Among them, the step S107 includes:

[0017] Based on the OCR direction detection model, obtain the horizontal rotation direction of the scale reading;

[0018] Combined with the prior information of the instrument, obtain the imaging angle of the instrument;

[0019] Then, according to the position information of the located scale reading, fit the equation of the circle or ellipse where the scale is located on the instrument, and obtain the spatial rotation angle;

[0020] Finally, correct the instrument image according to the obtained imaging angle and rotation angle.

[0021] Among them, the pointer fitting in the step S109 includes:

[0022] Perform image space transformation, image morphology processing, and data fitting on the pointers in the dial image respectively to obtain a straight line and / or straight line segment equation containing position information that is parallel to the pointer direction and has the best coincidence degree;

[0023] And determine the direction of obtaining the pointer based on the position information.

[0024] Wherein, the prior information of the instrument includes at least one of the type, measurement range, and scale distribution of the instrument.

[0025] Wherein, the image morphological processing includes:

[0026] Based on the boundary extraction method, segment the pointer area from the image transformed in the image space, and perform an opening operation to obtain a connected area including the pointer area;

[0027] Based on the skeleton extraction operation, obtain the skeleton of the pointer according to the connected area,

[0028] Obtain a set of points near the skeleton in the two-dimensional plane according to the skeleton of the pointer.

[0029] Wherein, the digital recognition of the scale reading is performed using paddleOCR.

[0030] Wherein, the step S111 includes:

[0031] Determine that the scale reading of the instrument is an arithmetic sequence according to the prior information of the instrument;

[0032] Sort the results of digital recognition along the fitting circle in the clockwise direction with the starting point at the lower left corner of the dial image according to the scale reading on the fitting circle;

[0033] Fit the digital recognition results in order or at an interval of one scale value into the corresponding arithmetic sequence, and calculate the common difference of the arithmetic sequence;

[0034] Compare the common difference and the adjacent difference of the digital recognition results in turn to determine whether there is misrecognition in the digital recognition result of the current scale reading, and average the digital recognition results of the scale readings with misrecognition according to the digital recognition results of the adjacent scale readings;

[0035] Obtain a set of calibrated scale reading values.

[0036] Wherein, the step S111 includes:

[0037] Determine that the scale reading of the instrument is a geometric sequence according to the prior information of the instrument;

[0038] Sort the results of digital recognition along the fitting circle in the clockwise direction with the starting point at the lower left corner of the dial image according to the scale reading on the fitting circle;

[0039] According to the digital recognition results, fit them into corresponding geometric sequences in order or at an interval of one scale value, and calculate the common ratio of the geometric sequences;

[0040] Compare the common ratio and the adjacent quotient of the digital recognition results in turn to determine whether there is misrecognition in the digital recognition result of the current scale reading. For the scale reading with misrecognition, multiply and take the square root according to the digital recognition results of adjacent scale readings;

[0041] Obtain a set of calibrated scale reading values.

[0042] The present invention also provides a recognition system for pointer instrument readings in an inspection scenario, which includes:

[0043] An image acquisition module, which is used to acquire pointer instrument images;

[0044] An image detection module, which is used to locate the pointer and the instrument in the pointer instrument image based on a positioning detection model, and locate the scale reading of the instrument in the image;

[0045] An image fitting module, which is used to fit the dial image in the pointer instrument image based on the above-positioned data by using an OCR direction detection model;

[0046] An image processing module, which is used to perform correction operations on the dial image based on plane rotation and space flipping to obtain a corrected dial image;

[0047] A dial recognition module, which is used to perform pointer straight line fitting and digital recognition of scale readings based on the corrected dial image. Digital recognition uses an OCR recognition model;

[0048] A scale calibration module, which is used to calibrate the digital recognition results based on the prior information of the instrument;

[0049] A result acquisition module, which is used to obtain the instrument reading based on the calibration result and the pointer straight line fitting result.

[0050] Compared with the prior art, the method for reading pointer instrument readings based on object detection and OCR of the present invention can adapt to scenarios where the target (pointer) imaging rotates and flips, etc., and can accurately perform instrument positioning and correction, ensuring the success rate and accuracy of the final reading of the pointer instrument. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features, and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the drawings, several embodiments of the present disclosure are shown in an exemplary rather than restrictive manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0052] Figure 1It is a flowchart showing a method for identifying the reading of a pointer instrument in an inspection scenario according to an embodiment of the present invention;

[0053] Figure 2 It is a flowchart showing the identification method according to an embodiment of the present invention;

[0054] Figure 3 It is a flowchart showing the process of obtaining a straight-line equation according to an embodiment of the present invention;

[0055] Figure 4a It is a schematic diagram showing the scale calculation by the angle method according to an embodiment of the present invention Figure 1 ;

[0056] Figure 4b It is a schematic diagram showing the scale calculation by the angle method according to an embodiment of the present invention Figure 2 ;

[0057] Figure 5 It is a schematic diagram of an identification system for the reading of a pointer instrument in an inspection scenario according to an embodiment of the present invention. Detailed implementation manners

[0058] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0059] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "the" and "said" used in the embodiments of the present invention and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. "Multiple" generally includes at least two.

[0060] It should also be noted that the term "comprises", "comprising" or any other variation thereof is intended to cover a non-exclusive inclusion, such that a commodity or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such commodity or device. Without further limitation, an element defined by the phrase "comprising an..." does not exclude the presence of additional identical elements in the commodity or device comprising said element.

[0061] The optional embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0062] Embodiment 1

[0063] As Figure 1 shown, an embodiment of the present invention provides a method for recognizing the reading of a pointer instrument in an inspection scenario, including the following steps:

[0064] Step S101, obtain an image of the pointer instrument;

[0065] Step S103, based on the positioning detection model, locate the pointer and the instrument in the pointer instrument image, and locate the scale indication of the instrument in the image;

[0066] Step S105, based on the above-positioned data, use the OCR direction detection model to fit the dial image in the pointer instrument image; wherein, the positioning detection model and the OCR direction detection model are trained using N sample images with target instruments collected historically;

[0067] Step S107, perform correction operations on the dial image based on plane rotation and space flipping to obtain the corrected dial image; specifically, it may include:

[0068] Obtain the horizontal rotation direction of the scale indication based on the OCR direction detection model;

[0069] Combine the prior information of the instrument to obtain the imaging angle of the instrument;

[0070] Then, according to the position information of the located scale indication, fit the equation of the circle or ellipse where the scale is located on the instrument, and obtain the space rotation angle;

[0071] Finally, correct the instrument image according to the obtained imaging angle and space rotation angle;

[0072] Generally, the imaging angle can be understood to include two directions: horizontal and vertical. The imaging angle in this embodiment is the rotation angle in the horizontal direction of the image;

[0073] Step S109, perform pointer straight line fitting and digital recognition of the scale indication on the corrected dial image, and the digital recognition uses the OCR recognition model; the pointer fitting may include:

[0074] Perform image space transformation, image morphology processing and data fitting on the pointers in the dial image respectively to obtain a straight line and / or straight line segment equation containing position information that is parallel to the pointer direction and has the best coincidence degree;

[0075] And judge and obtain the direction of the pointer based on the position information;

[0076] In the actual application scenario, this embodiment segments the pointer area from the dial image, and performs an open-then-close process to obtain a connected area including the pointer area, and then uses the skeleton extraction method in the morphological processing to obtain the skeleton of the pointer (which can be understood as the central axis of the image, the skeleton of a rectangle is the central axis of its rectangle, the skeleton of a circle is its center, and the skeleton of a straight line is itself); so far, a group of points located near the central axis of the pointer (image area) on the two-dimensional plane can be obtained, and a straight line and / or straight line segment equation containing position information that is parallel to the pointer direction and has the best overlap is obtained based on the least squares method straight line fitting;

[0077] Among them, digital recognition is performed using paddleOCR;

[0078] Step S111, calibrating the digital recognition result based on the instrument prior information;

[0079] Step S113: Obtaining the meter reading based on the calibration result and the pointer straight line fitting result.

[0080] The prior information of the embodiment of the present invention refers to some information about the pointer instrument, including at least one of the type, measurement range and scale distribution of the instrument, etc. The prior information is the instrument's own attributes, and is a parameter predefined and stored individually or by category.

[0081] For example:

[0082] ① Instrument type: (such as voltage / current / pressure...)

[0083] ②Instrument measurement range: (0-100)

[0084] ③ Scale distribution: (such as clockwise / counterclockwise, arithmetic / exponential, increasing / decreasing. For arithmetic type pointer instruments, the scale value distribution can be expressed as: a n =a 1 +(n-1)*d, where a 1 Indicates the scale value of the starting position / zero point scale value, a n It represents the value of the nth scale, d represents the difference between every two adjacent scales, also called tolerance. For clockwise instruments, d is a positive number; for counterclockwise instruments, d is a negative number).

[0085] Embodiment 2

[0086] Based on the above embodiment, this embodiment may further include the following contents:

[0087] The OCR direction detection model in the embodiment of the present invention is a classification model composed of a Mobilnet network. There are two types of methods in OCR recognition: one is the end-to-end learning method, and the other is non-end-to-end. End-to-end means using a network (model) to input an image and output the text recognition result. A non-end-to-end network is composed of multiple networks (models) or parts, specifically including: a text detection model (i.e., an OCR detection model, which outputs the text position result), a text direction detection model (i.e., an OCR direction detection model, which outputs the text angle result), and a text recognition model (i.e., an OCR recognition model, which outputs the final text result).

[0088] The OCR direction detection model in the embodiment of the present invention refers to the text direction detection model in the above non-end-to-end method. Specifically, the MobileNet network (model) is adopted.

[0089] When actually applying the OCR direction detection model in this embodiment, step S105 may specifically include:

[0090] According to the inclusion relationship between the positions of the detected instrument, pointer, and scale reading in the image, the detected scale reading and pointer are screened and associated with the instrument, and the detection results outside the target are eliminated.

[0091] Among them, the screening and association are to judge the attribution of the pointer, scale, and instrument according to the inclusion relationship between the positions of the detected instrument, pointer, and scale reading in the image. For the convenience of understanding, it should be noted in this embodiment that the image to be detected (image) will contain multiple instruments, and the above detection model will detect multiple instruments, multiple pointers, and multiple scale readings. Therefore, the purpose of this screening and association is to determine the attribution of the instrument, pointer, and number.

[0092] Embodiment III

[0093] On the basis of the above embodiment, this embodiment may further include the following content:

[0094] For the convenience of understanding the technical solution of this embodiment, this embodiment provides a method for recognizing the reading of a pointer instrument in an inspection scenario in combination with an actual application scenario, and will be described in detail as Figure 2 shown, this method may include the following steps:

[0095] Step (1) Image acquisition. In this step, N sample images with target instruments are collected at the inspection site, and after annotation, an image data set is obtained, and the training set and test set are divided for the subsequent training of the positioning detection model and the OCR model (including the OCR direction detection model and the OCR recognition model), as well as algorithm testing.

[0096] Step (2) Instrument detection. This step includes the detection of the instrument panel, the detection of the instrument scale readings, and the detection of the instrument pointer. It refers to obtaining the instrument components of interest (targets) from the image to be detected, obtaining their position information on the image. Generally, a rectangular box is used to frame them. Specifically, the yolov5 model is used to complete the instrument detection task.

[0097] Yolov5 is a mainstream object detection method in the current field of deep learning. After a series of developments or revisions such as yoloV1-V5, excellent results have been achieved in the detection accuracy and speed on various public datasets for object detection, and it has received much attention in the scientific research field and industrial production sites.

[0098] Yolov5 is an end-to-end object detection network. Compared with the traditional multi-stage processing process, it only uses a deep network to complete the inference and achieve the object detection function. For a determined network framework model, its inference or detection output result depends on the design of its input (samples) and output (labels) during the training process. The samples and labels used to train the model are collectively referred to as training data or training samples. Here, the training samples are obtained from pictures.

[0099] During model training, a batch of pictures including pointer instruments and labels marked with position information (specifically, the dial code, scale code, pointer code, and position representing the detection target on the dial) are input.

[0100] During model inference (i.e., detection), if the input is an RGB three-channel picture, its output is the position and code of the target of interest on the picture.

[0101] Step (3) Association of scale, pointer and instrument. This step is to judge the ownership of these candidate boxes according to the inclusion relationship between the positions of the detected instrument, pointer, and scale readings in the picture, and then screen and associate the detected scale readings and pointers with the instrument, and eliminate the detection results (candidate boxes) outside the target of interest.

[0102] Algorithmically, this inclusion relationship can be measured by IOU (Intersection over Union). Among the multiple detected pointers (candidate boxes), when the IOU value of a certain pointer (candidate box) and the instrument (box) is non-zero and the largest, it is determined that the pointer belongs to the instrument, so as to achieve the association; for the scale (candidate box), the candidate box with non-zero IOU belongs to the instrument. Among them, IOU includes GIOU, DIOU, and CIOU.

[0103] Step (4) Instrument Direction Judgment and Correction. This step refers to judging the rotation and flipping of the plane and spatial directions of the detected instrument area on the image, and then correcting / righting it to obtain a more regular instrument image on the imaging for subsequent processing.

[0104] In this step, the horizontal rotation direction of the scale reading is obtained by using the text direction detection model in the OCR (Optical Character Recognition) algorithm, and the imaging angle of the instrument is obtained by combining the prior information of the instrument. Then, according to the position information of the located scale reading, the equation of the circle or ellipse where the scale on the instrument is located is fitted, and the spatial rotation angle is obtained. Finally, the instrument image is corrected according to the obtained flipping and rotation angles. Specifically,

[0105] First, according to the associated information in step (3), extract a certain instrument and its corresponding scale (frame) and the picture area within the frame.

[0106] Second, use the OCR direction detection model (the model can be viewed in the scale OCR recognition part) to process the scale image of the instrument in turn to obtain the direction of the numbers in the picture, or the angle relative to the x-axis direction of the image. Thus, judge the rotation angle of the instrument imaging relative to the horizontal direction.

[0107] Then, calculate the center point coordinates of the scale frame one by one to obtain a two-dimensional array [(x1, y1), (x2, y2),..., (xN, yN)] of the image plane. Then, use the Least Square Method (LSM) to fit the equation of the circle or ellipse where the scale is located, x^2 / a^2 + y^2 / b^2 = 1, (a > b > 0), and obtain the equation of the largest inscribed circle of the ellipse. Use perspective transformation to solve the inclination angle from the ellipse equation to the largest inscribed circle equation as the flipping angle of the instrument in the spatial imaging.

[0108] Finally, according to the obtained spatial flipping angle and plane rotation angle, use affine transformation (also known as affine mapping) to correct the instrument image.

[0109] Step (5) Pointer Fitting. This step refers to expressing the pointer with a directed line segment on the image to obtain the direction information and position information of the pointer. By extracting the pointer main body within the detection frame, through image space transformation, image morphological processing and data fitting, a straight line / straight line segment equation (position information) parallel to the pointer direction and with the best coincidence degree is obtained, and the direction information of the pointer is obtained by combining the digital position judgment. Specifically,

[0110] The position information of the pointer can be represented by a straight-line equation in the two-dimensional image space: Ax + By + C = 0, where x ∈ [x1, x2], and x1 and x2 are the minimum and maximum values of the straight-line segment equation in the horizontal direction (x direction). As Figure 3 shown, the process of obtaining this straight-line equation is as follows:

[0111] First, extract the pointer area image by extracting the detected pointer box.

[0112] Second, convert the entire image of the pointer area into a grayscale space. And use the adaptive binarization method to differentiate the pointer body and the background at the pixel level.

[0113] Then, use the boundary extraction method of image morphological processing to segment the pointer area from the background and perform an opening operation to obtain a connected area including the pointer area. Then, use the skeleton extraction method in morphological processing to obtain the skeleton of the pointer (which can be understood as the central axis of the image. The skeleton of a rectangle is the central axis on its rectangle, the skeleton of a circle is its center, and the skeleton of a straight line is itself). So far, a set of points near the central axis of the pointer (image area) in the two-dimensional plane can be obtained.

[0114] Finally, perform a straight-line equation fitting on this set of skeleton points to calculate the equation of a straight line representing the pointer. The specific method used can be hough line fitting, least squares method LSTM, gradient descent method, etc.

[0115] Among them, the boundary extraction in this embodiment is performed on the binarized image. Specifically, if the 8-neighborhood of a point is all 1, then this point can be considered an internal point rather than a boundary point, and such points can be deleted. The remaining points are the boundary points. The specific steps are as follows: Erode the image with a 3*3 structuring element to obtain the internal points, and then subtract the eroded image from the original image to obtain the boundary of the image.

[0116] The opening operation is as follows: Use the structuring element B (a two-dimensional structuring element can be understood as a two-dimensional matrix, and the values of the matrix elements are 0 or 1) to perform an opening operation on the set A (the image pixel block, which is essentially also a two-dimensional matrix), defined as:

[0117]

[0118] The skeleton extraction operation is as follows: Adopt the method based on the maximum disk. Specifically, the skeleton of the target is composed of the centers of all inscribed disks within the target. That is, fit a set of inscribed circles to the obtained connected area, and then extract the center positions, which can be used as the skeleton of the pointer.

[0119] Among them, the direction information of the pointer in this embodiment is to determine the starting point (the end of the instrument pointer) and the ending point (the needle tip segment of the instrument pointer) of the straight line segment. Specifically, on the image, these two points refer to the two endpoints (x1, y1) and (x2, y2) of the straight line equation obtained by the above fitting. For whether these two points belong to the end or the needle tip, it is necessary to judge by means of the center position of the scale calculated in the above direction calibration: select the endpoint with the closest Euclidean distance to the center position on the image as the pointer end, and the farthest one as the needle tip.

[0120] Step (6) Scale OCR recognition, that is, use the text recognition model in OCR to process the image within the scale (frame) to obtain the numbers or characters of the instrument scale.

[0121] The scale recognition link is implemented using the paddleOCR framework. It is a set of practical OCR algorithms / tools libraries. The framework provides models including text detection, text direction detection, and character recognition. The models have the characteristics of being ultra-lightweight (not exceeding 10M), supporting user-defined training, being able to run on multiple systems such as Linux, Windows, MacOS, and supporting PIP for quick installation and use.

[0122] For the instrument direction judgment and correction and the scale OCR recognition link, the text direction detection model and the text recognition model of it are respectively adopted.

[0123] The text direction detection model and the text recognition model are respectively designed as two end-to-end networks. The models need to be trained with the labeled scale reading pictures. The training data of the former is a batch of pictures containing only a set of numbers / characters and combinations, and the label is the angle (0-360) of the characters (or character combinations) in the picture relative to the horizontal direction. The training data of the latter is a batch of pictures including one or more sets of numbers / characters and combinations (constituted by pictures of simulated Arabic numeral data and instrument scale pictures labeled in the actual scene, etc.), and the label is the position of the characters and the numbers and characters (referring to [01234656789.]).

[0124] Step (7) Scale reading calibration. This step is to calibrate the scale readings obtained by the OCR recognition model to avoid misrecognition by the OCR recognition model. This kind of misrecognition is caused by factors such as noise, artifacts, and occlusion in the image. The calibration process combines the prior information of the scale instrument to calibrate or screen the recognition results of the instrument recognized by the OCR recognition model.

[0125] Based on the instrument being directionally calibrated, this step sorts the detected scales in the clockwise direction; taking a single-pointer instrument as an example, it often follows the "clockwise direction" rule, that is, the readings of the instrument increase / decrease in the "clockwise direction". Moreover, this kind of increasing and decreasing rule belongs to an arithmetic sequence or a geometric sequence.

[0126] This step mainly includes:

[0127] Determine that the scale readings of the instrument are an arithmetic sequence or a geometric sequence according to the prior information of the instrument;

[0128] When it is determined according to the prior information of the instrument that the scale readings of the instrument are an arithmetic sequence;

[0129] First, sort the results after scale reading recognition according to the scale frame, starting from the lower left corner of the picture on the fitted circle, and sort the recognition results of the scales in a clockwise direction along the fitted circle. Secondly, fit arithmetic sequences in order or at an interval of one scale value respectively, and calculate the common difference of the arithmetic sequences. Finally, compare the common difference with the adjacent difference of the scales (recognized numbers) in turn to determine whether there is misrecognition in the current recognized scale recognition result, and average the misrecognized scales according to the adjacent scale recognition results. Finally, a set of ordered (clockwise or counterclockwise) and calibrated scale values are obtained.

[0130] For this calibration process, in this embodiment, when the characters ".10", "20", "30", "40", "50" are recognized by paddleOCR and ".10" is calibrated to the number 10 as an example. First, in this embodiment, it is known from the prior information (scale distribution) of the instrument that the scale of this instrument satisfies the arithmetic sequence type. Therefore, it is first necessary to locate the misrecognized characters. Here, in this embodiment, the recognized characters are sorted clockwise according to their positions in the image, and the characters '.10', '20', '30', '40', '50' are obtained, and the results of converting the characters to numbers are 0.1, 20, 30, 40, 50. Then, according to the general term formula of the arithmetic sequence, substitute it to calculate the common difference, and the common difference can be obtained as 10. According to the difference between adjacent two items or two items separated by one item, it is judged that the number 0.1 corresponding to ".10" is incorrect and other items are correct. Therefore, it is calibrated to 10 again.

[0131] The instrument of the geometric sequence type is similar, and the difference is that the difference calculation process is replaced by a quotient calculation.

[0132] That is, when it is determined according to the prior information of the instrument that the scale readings of the instrument are a geometric sequence;

[0133] First, sort the results after scale reading identification according to the scale frame. Starting from the bottom left corner of the picture on the fitted circle, sort the scale recognition results in a clockwise direction along the fitted circle. Secondly, fit a geometric sequence in order or with an interval of one scale value, and calculate the common ratio of the geometric sequence. Finally, compare the common ratio with the adjacent quotient of the scale (recognized number) in turn to determine whether there is misrecognition in the current recognized scale result. For the scale with misrecognition, multiply and take the square root according to the adjacent scale recognition results. Finally, obtain a set of ordered (clockwise or counterclockwise) scale values that have passed the reading verification.

[0134] Step (8) Reading calculation can be carried out by using the angle method. The principle is as follows:

[0135] Assume that the scale reading is on a plane circle (same scale line), and any two scale points on the instrument dial are denoted as a and b, and a and b are not duplicate scales, and the corresponding scale values are Va and Vb respectively. At the same time, the linear equation obtained by fitting the pointer is Ax + By + C = 0, then the line intersects with these two scale lines at point c in the image plane (if the line is parallel to the scale connection ab, reselect one of the points to ensure there is an intersection with the pointer line). In the scenario, the position distributions of the scale and the pointer are roughly divided into Figure 4a and Figure 4b two types;

[0136] Then the current reading of the instrument is calculated as:

[0137] V = Va + ∠aoc / ∠aob * (Vb - Va) (as Figure 4a shown)

[0138] Or

[0139] V = Va - ∠aoc / ∠aob * (Vb - Va) (as Figure 4b shown)

[0140] Based on the above formula, the current reading of the instrument can be obtained;

[0141] For the values of the two scale points required in this embodiment, they can be selected from the calibrated scales;

[0142] Considering the constraints or influences of image resolution, imaging interference, etc., for any two arbitrarily selected scale points, the calculated results have a large deviation. Therefore, use the two scale points closest to the pointer to calculate the reading by the angle method. The position information of the calculated scale position, the pointer equation, and the pointer tip endpoint are required. Specifically, measure the distance by calculating the shortest distance from the center position of each scale frame to the pointer line (line end) / or the Euclidean distance from the pointer endpoint, and select the two scale points with the smallest Euclidean distance.

[0143] Example 4

[0144] As Figure 5 shown, the present invention also provides a recognition system for the reading of pointer instruments in an inspection scenario, which includes:

[0145] An image acquisition module for acquiring pointer instrument images;

[0146] An image detection module for positioning the pointer and instrument in the pointer instrument image based on a positioning detection model, and positioning the scale indication of the instrument in the image;

[0147] An image fitting module for fitting the dial image in the pointer instrument image by using an OCR direction detection model based on the above-positioned data;

[0148] An image processing module for performing correction operations on the dial image based on plane rotation and spatial flipping to obtain a corrected dial image;

[0149] A dial recognition module for performing pointer straight line fitting and digital recognition of the scale indication on the corrected dial image, and the digital recognition uses an OCR recognition model;

[0150] A scale calibration module for calibrating the digital recognition result based on the prior information of the instrument;

[0151] A result acquisition module for obtaining the instrument reading based on the calibration result and the pointer straight line fitting result.

[0152] Example 5

[0153] The present disclosure provides a non-volatile computer storage medium storing computer-executable instructions that can execute the method steps described in the above embodiments.

[0154] It should be noted that the above-mentioned computer-readable medium in the present disclosure can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and this computer-readable signal medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0155] The above-mentioned computer-readable medium can be included in the above-mentioned electronic device; or it can exist independently and not be assembled into the electronic device.

[0156] The computer program code for performing the operations of the present disclosure can be written in one or more programming languages or combinations thereof. The above-mentioned programming languages include object-oriented programming languages - such as Java, Smalltalk, C++, and also include conventional procedural programming languages - such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN) - or, it can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0157] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur in a different order than that noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0158] The units involved in the embodiments described in the present disclosure can be implemented in software or in hardware. Among them, the name of the unit does not, in some cases, constitute a limitation on the unit itself.

[0159] The preferred embodiments of the present invention have been described above to make the spirit of the present invention clearer and easier to understand, and are not intended to limit the present invention. Any modifications, substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection defined by the appended claims of the present invention.

Claims

1. A method for recognizing the readings of pointer instruments in an inspection scenario, characterized in that, it includes the following steps: Step S101, obtain an image of the pointer instrument; Step S103, based on a positioning detection model, locate the pointer and the instrument in the pointer instrument image, and locate the scale readings of the instrument in the image; Step S105, based on the above-positioned data, use an OCR direction detection model to fit the dial image in the pointer instrument image, including: screening and associating the detected scale readings and the pointer with the instrument according to the inclusion relationship between the positions of the detected instrument, pointer, and scale readings in the image, and eliminating the detection results outside the target; Step S107, perform correction operations on the dial image based on plane rotation and space flipping to obtain a corrected dial image, including: obtaining the horizontal rotation direction of the scale readings based on the OCR direction detection model; combining the prior information of the instrument to obtain the imaging angle of the instrument; then, according to the position information of the located scale readings, fitting the equation of the circle or ellipse where the scales are located on the instrument to obtain the space rotation angle; finally, correct the instrument image according to the obtained imaging angle and space rotation angle; Step S109, perform pointer straight line fitting and digital recognition of the scale readings based on the corrected dial image, and digital recognition uses an OCR recognition model; Step S111, based on the prior information of the instrument, calibrate the digital recognition result, including: determining that the scale readings of the instrument are an arithmetic sequence or a geometric sequence according to the prior information of the instrument; Step S113, obtain the instrument reading based on the calibration result and the pointer straight line fitting result.

2. The method according to claim 1, characterized in that, the positioning detection model and the OCR direction detection model are trained using N sample images with target instruments collected historically.

3. The method according to claim 1, characterized in that, the prior information of the instrument includes at least one of the type, measurement range, and scale distribution of the instrument.

4. The method according to claim 1, characterized in that, the pointer fitting in step S109 includes: respectively performing image space transformation, image morphology processing, and data fitting on the pointer in the dial image to obtain a straight line and / or straight line segment equation containing position information that is parallel to the pointer direction and has the best coincidence degree; and judging and obtaining the direction of the pointer based on the position information.

5. The method according to claim 4, characterized in that, the image morphology processing includes: based on a boundary extraction method, segment the pointer area from the image after image space transformation, and perform an opening operation to obtain a connected area including the pointer area; based on a skeleton extraction operation, obtain the skeleton of the pointer according to the connected area, and obtain a set of points near the skeleton in the two-dimensional plane according to the skeleton of the pointer.

6. The method according to claim 1, characterized in that, step S111 includes: determining that the scale readings of the instrument are an arithmetic sequence according to the prior information of the instrument; Sort the results of digital recognition according to the scale readings. Starting from the lower left corner of the dial image on the fitted circle, sort the digital recognition results of the scales in a clockwise direction along the fitted circle; Fit the digital recognition results into corresponding arithmetic progressions in order or with an interval of one scale value, and calculate the common difference of the arithmetic progressions; Compare the common difference with the adjacent differences of the digital recognition results in turn to determine whether there is misrecognition in the digital recognition results of the current scale reading. For the scale readings with misrecognition, perform average processing according to the digital recognition results of adjacent scale readings; Obtain a set of calibrated scale readings.

7. The method according to claim 1, characterized in that the step S111 includes: Determine that the scale readings of the instrument are a geometric progression according to the prior information of the instrument; Sort the results of digital recognition according to the scale readings. Starting from the lower left corner of the dial image on the fitted circle, sort the digital recognition results of the scales in a clockwise direction along the fitted circle; Fit the digital recognition results into corresponding geometric progressions in order or with an interval of one scale value, and calculate the common ratio of the geometric progressions; Compare the common ratio with the adjacent quotients of the digital recognition results in turn to determine whether there is misrecognition in the digital recognition results of the current scale reading. For the scale readings with misrecognition, perform multiplication and square root processing according to the digital recognition results of adjacent scale readings; Obtain a set of calibrated scale readings.

8. An identification system for reading pointer instruments in an inspection scenario for implementing the method according to any one of claims 1-7, characterized in that it includes: An image acquisition module for acquiring pointer instrument images; An image detection module for positioning the pointer and the instrument in the pointer instrument image based on a positioning detection model, and positioning the scale readings of the instrument in the image; An image fitting module for fitting the dial image in the pointer instrument image using an OCR direction detection model based on the above-positioned data; An image processing module for performing correction operations on the dial image based on plane rotation and space flipping to obtain a corrected dial image; A dial identification module for performing pointer straight line fitting and digital recognition of scale readings on the corrected dial image, and the digital recognition uses an OCR recognition model; A scale calibration module for calibrating the digital recognition results based on the prior information of the instrument; A result acquisition module for obtaining the instrument reading based on the calibration result and the pointer straight line fitting result.

Images