A pointer instrument reading method and inspection equipment based on image recognition

Abstract

The present invention discloses a pointer instrument reading method based on image recognition, including obtaining instrument information to obtain a detection model and then obtain an identification dial image and a dial detection center; using Hough transform to obtain a detection straight line; if the number of detection straight lines is at least two, respectively compare the first distance and the preset spacing to obtain a class-excellent straight line; if the number of class-excellent straight lines is at least two, project a light beam toward the pointer instrument to obtain a verification dial image and use Hough transform to obtain a verification straight line, and then obtain a straight line where the class-excellent straight line and the verification straight line coincide as the pointer straight line; obtain a needle axis boundary box and obtain a pointer boundary box based on the pointer straight line; based on the instrument information, obtain the pointer reading according to the pointer boundary box and the needle axis boundary box. The pointer instrument reading method based on image recognition provided by the present invention has low calculation difficulty, simple algorithm, accurate reading, and strong applicability.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a pointer-type instrument reading method and inspection equipment based on image recognition. Background Art

[0002] Pointer instruments are common instruments used to measure and display physical quantities such as pressure, temperature, and liquid level. They are widely used in industries such as petrochemicals, electricity, pharmaceuticals, metallurgy, and aerospace to ensure the normal operation and safety of the system. Accurate reading of pointer instruments is crucial to ensuring the normal operation of production equipment, stable product quality, and the safety of the process. Therefore, inspection of pointer instruments is a necessary part of industrial production management.

[0003] With the development of machine vision technology, the inspection work of pointer instruments has gradually changed from manual inspection to inspection equipment with machine vision technology. Therefore, automatic reading of precision pointer instruments based on machine vision has become an important key common technology, which is of great significance to improving safety management level, improving production efficiency and reducing labor costs.

[0004] In production applications, the needle axis of some pointer instruments is long, and the pointer installation position is far away from the scale plane. Figure 4 Due to the influence of the environment or lighting in the workshop, the projection of the pointer often appears on the scale plane. When reading with machine vision technology, the pointer shadow reading or error is often output, which affects the safety management level and production efficiency, and may even cause safety failures in serious cases.

[0005] Early researchers eliminated shadows by establishing physical models, in which the physical model was established based on the assumption that illumination is constant. However, this method is only applicable to eliminating shadows in high-quality images that meet the assumptions. However, the operating environment of pointer instruments in production is changeable, and this method cannot be widely used. Another method is based on the characteristics of shadow images and manually designs features to describe shadows. However, the ability to describe shadows based on manually designed features is limited and usually fails when applied.

[0006] In recent years, a shadow detection method based on convolutional neural networks has been proposed. Although the accuracy of shadow detection has been continuously improved, it requires the support of a large number of training samples or training data. Despite this, the existing methods may still mistakenly identify black objects as shadows. In addition, the pointers of pointer-type instruments in production applications are mostly dark or black, and shadows and pointers are even more difficult to distinguish in the use environment.

[0007] Hough transform is a feature extraction and detection method widely used in image analysis, computer vision, etc. It can be used to extract straight lines, circles and other shapes in images. In pointer instruments, the pointer is linear and the needle shaft is circular. If edge accuracy detection is not performed, Hough transform straight line detection and Hough transform circle detection can be quickly used for extraction. However, the pointer shape usually becomes gradually thinner or pointed from the needle shaft to the pointer vertex, and high-precision detection or complex algorithms are required to extract the precise pointer pointing. Therefore, it is limited in production applications that require reading accuracy. Summary of the invention

[0008] In order to solve the above problems in the prior art, the present invention provides a pointer instrument reading method based on image recognition, which has low computational difficulty, simple algorithm, can accurately read pointer instruments in a projection environment, and has strong applicability.

[0009] In order to achieve the above-mentioned object of the invention, the present invention provides a pointer instrument reading method based on image recognition, comprising the following steps:

[0010] Obtain recognition model and detection model library according to training data set;

[0011] Acquire an identification picture according to the pointer instrument, wherein the image content of the identification picture includes the pointer instrument;

[0012] Recognize the recognition picture according to the recognition model to obtain instrument information;

[0013] Obtaining a detection model from the detection model library according to the instrument information;

[0014] Detect the recognition picture according to the detection model to obtain a recognition dial boundary box;

[0015] Obtaining an identified dial image and a dial detection center according to the identified dial boundary frame;

[0016] Based on the identified dial image, a detection straight line is obtained by using Hough transform;

[0017] If the number of the detection straight lines is at least two, obtaining a preset spacing;

[0018] respectively acquiring a first distance, where the first distance is the distance from the detection center of the dial to the detection straight line;

[0019] The first distance and the preset distance are compared respectively, and a detection straight line whose first distance is smaller than the preset distance is a quasi-optimal straight line;

[0020] If the number of the class-optimal straight lines is at least two, then the class-optimal straight lines are saved;

[0021] Projecting a light beam toward the pointer instrument, and acquiring a verification image based on the pointer instrument after the light beam is projected;

[0022] Detect the verification image according to the detection model to obtain a verification dial boundary box;

[0023] Obtaining a verification dial image according to the verification dial boundary box;

[0024] Based on the verification dial image, a verification straight line is obtained by using Hough transform;

[0025] The class-optimal straight line and the verification straight line are compared respectively, and the straight line where the class-optimal straight line and the verification straight line coincide with each other is the pointer straight line;

[0026] Based on the pointer straight line, detecting the identified dial image according to the detection model to obtain a pointer bounding box;

[0027] Detect the identified dial image according to the detection model to obtain a needle axis boundary box;

[0028] Based on the instrument information, a pointer reading is obtained according to the pointer bounding box and the needle axis bounding box.

[0029] Furthermore, the steps of obtaining the pointer straight line are:

[0030] When the number of the detection straight lines is unique, the detection straight line is the pointer straight line;

[0031] When the number of the detected straight lines is at least two, the optimal straight line is obtained:

[0032] If the number of the class-optimal straight lines is unique, the class-optimal straight line is the pointer straight line;

[0033] If the number of the class-optimal straight lines is at least two, the verification straight line is obtained: a straight line where the class-optimal straight line and the verification straight line overlap is a pointer straight line.

[0034] Furthermore, the step of obtaining a straight line where the class-optimal straight line coincides with the verification straight line is:

[0035] Projecting a light beam toward the pointer instrument at a first position, the first position being located on the same side of the at least two quasi-optimal straight lines;

[0036] Obtaining a first verification image based on the pointer instrument after the light beam is projected at the first position;

[0037] Detecting the first verification image according to the detection model to obtain a first verification dial boundary box;

[0038] Crop the first verification dial boundary frame to obtain an initial first verification dial image;

[0039] Correcting and processing the initial first verification dial image to obtain a first verification dial image;

[0040] Obtain a first verification straight line by using Hough transform based on the first verification dial image;

[0041] Compare the first verification straight line and the optimal straight line respectively, retain the optimal straight line that coincides with the first verification straight line, and obtain an updated straight line;

[0042] When the number of the update straight lines is unique, the update straight line is the pointer straight line;

[0043] When the number of the update straight lines is at least two:

[0044] Projecting a light beam toward the pointer instrument at a second position, the second position being located on the other side of the at least two quasi-optimal straight lines;

[0045] Obtaining a second verification image based on the pointer instrument behind the light beam projected at the second position;

[0046] Detecting the second verification image according to the detection model to obtain a second verification dial boundary box;

[0047] Crop the second verification dial boundary frame to obtain an initial second verification dial image;

[0048] Correcting and processing the initial second verification dial image to obtain a second verification dial image;

[0049] Obtain a second verification straight line by using Hough transform based on the second verification dial image;

[0050] The second verification straight line and the update straight line are compared respectively, and the update straight line that coincides with the second verification straight line is the pointer straight line.

[0051] Furthermore, the step of obtaining the needle axis bounding box is:

[0052] Based on the identified dial image, a detection circle is obtained by using Hough transform;

[0053] When the number of the detection circles is unique, the detection circle is the needle axis circle;

[0054] When the number of the detection circles is at least two, acquiring a second distance, where the second distance is a distance from a center of the detection circle to a detection center of the dial;

[0055] The detection circle with the smallest second distance is the needle axis circle.

[0056] Further, the detection model includes an instrument dial detection model and a dial content detection model;

[0057] The instrument dial detection model is used to obtain a dial bounding box according to a picture, wherein the picture includes an identification picture and a verification picture, and the dial bounding box includes an identification dial bounding box and a verification dial bounding box:

[0058] Detecting the identification picture according to the instrument dial detection model to obtain the identification dial boundary box;

[0059] Detecting the verification image according to the instrument dial detection model to obtain the verification dial boundary box;

[0060] The dial content detection model is used to obtain the pointer bounding box and the needle axis bounding box according to the identified dial image.

[0061] Further, the step of obtaining the pointer reading according to the needle axis bounding box and the pointer bounding box is:

[0062] Based on the needle axis bounding box, obtaining the needle axis center according to a center point calculation formula;

[0063] Based on the pointer bounding box, obtaining the pointer center according to a center point calculation formula;

[0064] Obtaining a pointer vertex according to the pointer bounding box, the needle axis center and the pointer center;

[0065] The pointer angle is obtained according to the pointer vertex and the needle axis center; the pointer angle is:

[0066] ;

[0067] In the formula, is the pointer angle, is the horizontal coordinate of the pointer vertex, is the vertical coordinate of the pointer vertex, is the horizontal coordinate of the needle axis center, The ordinate of the needle axis center, is the ratio of pi;

[0068] Acquire a minimum scale reading, a maximum scale reading, a minimum scale angle, and a maximum scale angle based on the instrument information;

[0069] The pointer reading is obtained according to the pointer angle, the minimum scale reading, the maximum scale reading, the minimum scale angle and the maximum scale angle; the pointer reading is:

[0070] ;

[0071] In the formula, r For pointer reading, is the minimum scale reading, is the maximum scale reading, is the minimum scale angle, is the maximum scale angle, is the pointer angle.

[0072] Furthermore, the center point calculation formula is:

[0073] ;

[0074] In the formula, is the horizontal coordinate of the center point of the bounding box, is the ordinate of the center point of the bounding box, is the bounding box, is the horizontal coordinate of the upper left corner of the bounding box, is the ordinate of the upper left corner of the bounding box, is the horizontal coordinate of the lower right corner of the bounding box, The ordinate of the lower right corner of the bounding box.

[0075] Furthermore, the step of obtaining the pointer vertex according to the pointer bounding box, the needle axis center and the pointer center is:

[0076] when , hour:

[0077] ;

[0078] when , hour:

[0079] ;

[0080] when , hour:

[0081] ;

[0082] when , hour:

[0083] ;

[0084] when , hour:

[0085] ;

[0086] when , hour:

[0087] ;

[0088] when , hour:

[0089] ;

[0090] when , hour:

[0091] ;

[0092] In the formula, is the horizontal coordinate of the pointer vertex, is the vertical coordinate of the pointer vertex, is the horizontal coordinate of the center of the dial, is the ordinate of the dial center, is the horizontal coordinate of the pointer center, is the ordinate of the pointer center, is the horizontal coordinate of the upper left corner of the pointer bounding box, is the ordinate of the upper left corner of the pointer bounding box, is the horizontal coordinate of the lower right corner of the pointer bounding box, The ordinate of the lower right corner of the pointer's bounding box.

[0093] An inspection device, used to implement the pointer-type instrument reading method based on image recognition, comprising a camera, a light shield, a floodlight, a control module and a calculation module;

[0094] The camera is used to take a picture of the pointer instrument and transmit it to the calculation module;

[0095] The floodlight is used to project a light beam toward the pointer instrument;

[0096] The light shield is used to prevent the projected light beam from directly hitting the photographing part of the camera when the projector projects a light beam;

[0097] The control module is used to control the camera to take pictures of the pointer instrument and transmit the pictures to the calculation module according to the instructions of the calculation module, and to control the floodlight to project a light beam toward the pointer instrument according to the instructions of the calculation module;

[0098] The computing module is used to store and run computer-readable instructions, and the computer-readable instructions execute the pointer instrument reading method based on image recognition when running.

[0099] Furthermore, the inspection equipment also includes a support, a support rod and a swing arm;

[0100] The camera is installed on an axial end surface of the support, and the light shield is arranged around the outside of the camera;

[0101] The support rod is installed at the other axial end of the support to support the support;

[0102] One end of the swing arm is rotatably matched with the support, and the other end of the swing arm is provided with the floodlight. When one end of the swing arm rotates along the support, the floodlight rotates around the camera outside the light shield.

[0103] The beneficial effects of the present invention are:

[0104] 1. The present invention only requires a moderate amount of training data for identifying the instrument dial, pointer and needle shaft, and only requires basic Hough transform straight line detection and Hough transform circle detection to achieve accurate reading of pointer instruments in a projection environment. It does not require a large amount of training data support or a high-precision and complex Hough transform algorithm, and has low computational difficulty and a simple algorithm.

[0105] 2. The present invention can realize the reading of pointer instruments with projection in the use environment through deep learning and Hough transform, and combined with projection light beam, with high reading accuracy and easy promotion and application;

[0106] 3. The present invention does not require a large amount of training data support and has a simple algorithm, which is convenient for integration in the same device or storage medium for reading various types of pointer instruments and has strong applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] Figure 1 It is a flow chart of the pointer instrument reading method based on image recognition of the present invention;

[0108] Figure 2 A flow chart of obtaining a straight line that coincides with a verification straight line in the present invention;

[0109] Figure 3 It is a front view of the pointer instrument in the present invention;

[0110] Figure 4 It is a schematic diagram of the longitudinal section structure of the pointer type instrument in the present invention;

[0111] Figure 5 A schematic diagram of obtaining an identified dial image according to an identified picture in the present invention;

[0112] Figure 6 It is a schematic diagram of a dial image recognition when the number of detection straight lines is two and the number of class-excellent straight lines is unique in the present invention;

[0113] Figure 7 for Figure 6 A schematic diagram of the partially enlarged structure at center A;

[0114] Figure 8 It is a schematic diagram of a dial image recognition when the number of detection lines is two and the number of quasi-excellent lines is two in the present invention;

[0115] Fig. 9 for Figure 8 A schematic diagram of the partially enlarged structure at B in the middle;

[0116] Fig.10 This is a schematic diagram of a dial image recognition when the number of detection circles is two in the present invention;

[0117] Fig.11 for Fig.10 A schematic diagram of the partially enlarged structure at C in the middle;

[0118] Fig.12 A schematic diagram of an image of a dial for identification when the number of detected circles is unique in the present invention;

[0119] Fig.13 It is a schematic diagram of the minimum scale angle, the maximum scale angle and the pointer angle in the present invention;

[0120] Fig.14 is a schematic diagram of a pointer boundary box and a needle axis boundary box in the present invention;

[0121] Fig.15 It is a schematic diagram of the partial structure of the inspection equipment in the present invention;

[0122] Fig.16 The present invention is a schematic diagram of the positional relationship between the inspection equipment and the pointer-type instrument during the implementation of the pointer-type instrument reading method based on image recognition of the present invention.

[0123] Among them, the figure markings are: 1. pointer instrument; 11. scale plane; 12. pointer; 13. needle axis; 140. dial detection center; 15. scale; 151. minimum scale; 152. maximum scale; 2. inspection equipment; 21. support; 22. camera; 23. light shield; 24. floodlight; 25. swing arm; 26. support rod; 3. identification picture; 41. identification dial boundary box; 42. pointer boundary box; 43. needle axis boundary box; 5. identification dial image. DETAILED DESCRIPTION

[0124] In order to clearly illustrate the technical features of this solution, this solution is described below through a specific implementation method.

[0125] See also Figure 1 The embodiment of the present invention provides a pointer instrument reading method based on image recognition, comprising the following steps:

[0126] A recognition model and a detection model library are obtained according to the training data set; the training data set includes a recognition data set and a detection data set, a recognition model is obtained according to the instrument recognition data set, and a detection model is obtained according to the detection data set; the detection model library includes at least one detection model; the detection model includes an instrument dial detection model and a dial content detection model; the instrument dial detection model is used to infer a dial bounding box according to a picture, the picture includes a recognition picture and a verification picture, and the dial bounding box includes a recognition dial bounding box and a verification dial bounding box: the recognition picture is detected according to the instrument dial detection model to obtain a recognition dial bounding box; the verification picture is detected according to the instrument dial detection model to obtain a verification dial bounding box; the dial content detection model is used to infer the needle shaft (actually the exposed part of the needle shaft) and the pointer according to the recognition dial image (or verification dial image) and obtain the pointer bounding box and the needle shaft bounding box; preferably, the recognition model and the detection model are obtained by the YOLOv8 algorithm;

[0127] The identification picture 3 is obtained according to the pointer instrument, and the image content of the identification picture includes the pointer instrument 1; a digital camera or a video camera can be used to take a picture of the pointer instrument 1 to obtain the identification picture 3. When taking the picture, the photographing part of the digital camera or the video camera should face the dial of the pointer instrument 1 as much as possible. Fig.16 , in order to reduce the difficulty of subsequent processing and improve the reading accuracy;

[0128] Still see Figure 1 , according to the recognition model, the recognition picture 3 is identified to obtain the instrument information;

[0129] Obtaining a detection model from a detection model library according to instrument information;

[0130] The recognition image is detected according to the detection model to obtain the recognition dial boundary box 41; the detection model used in this process is the instrument dial detection model;

[0131] The identified dial image 5 and the dial detection center 140 are obtained according to the identified dial boundary frame 41; specifically, see Figure 5 , according to the detection model, the recognition image 3 is detected to obtain the recognition dial boundary frame 41, and the recognition dial boundary frame 41 is cropped to obtain the initial recognition dial image; the initial recognition dial image is corrected to obtain the recognition dial image 5; preferably, the initial recognition dial image is corrected by affine transformation to obtain the dial detection center 140, and the dial detection circle can also be obtained by performing Hough transformation circle detection on the recognition dial image 5 to obtain the dial detection center 140, which is a prior art and will not be repeated here;

[0132] Based on the identification of the dial image 5, the Hough transform is used to obtain the detection line L; the threshold of the Hough transform line detection is set, and the straight line equivalent to the pointer length is detected by the preset Hough transform line detection threshold to obtain the detection straight line and count it; in actual application scenarios, different pointer instruments 1 or the same pointer instrument 1 will receive different illumination angles and illumination intensities at different times, and one or more pointer shadows may appear on the scale plane 11: see Figure 4 , the plane where the pointer 12 is located is parallel to the scale plane 11. Under ambient lighting, the projection of the pointer 12 on the scale plane 11 is parallel to the pointer. Under different lighting angles, the projection of the pointer may appear enlarged or elongated or projected out of the scale plane 11. When it is higher than the threshold of Hough transform straight line detection, it is not recorded as a detection straight line. At the same time, the scale line is lower than the threshold of Hough transform straight line detection and is not recorded as a detection straight line. Therefore, mutually parallel detection straight lines can be obtained by presetting the threshold of Hough transform straight line detection. Therefore, the detection straight line includes the straight line where the pointer is located, or the straight line where the pointer is located and the straight line where the pointer shadow is located (the pointer shadow within the threshold range of Hough transform straight line detection);

[0133] If the number of detected straight lines L is at least two, the preset spacing is obtained ;

[0134] Get the first distance respectively , the first distance is the distance from the dial detection center 140 to the detection line L; the first distance is calculated between each detection line and the dial detection center , respectively ~ (No. 1~ The distance between the detection line and the detection center of the dial);

[0135] Compare the first distance respectively and preset spacing , the first distance Smaller than preset spacing Right now The detection line is the quasi-optimal straight line L', For the The first distance between the detection line and the detection center of the dial, For the detection lines; The detection straight line is the line where the pointer shadow is located, so it can be directly excluded;

[0136] If the number of class-optimal lines L' is at least two, see Figure 8 , 9 , detect straight line With detection straight line The corresponding first distance All smaller than the preset spacing , then save the class-optimal straight line L';

[0137] Projecting a light beam toward the pointer instrument, and acquiring a verification image based on the pointer instrument after the light beam is projected;

[0138] The verification image is detected according to the detection model to obtain the verification dial boundary box; the detection model used in this process is the instrument dial detection model;

[0139] The verification dial image is obtained according to the verification dial bounding box; the step of obtaining the verification dial image is similar to the step of obtaining the identification dial image, the verification dial bounding box is obtained by detecting the verification picture according to the detection model, and the verification dial bounding box is cropped to obtain the initial verification dial image; the initial verification dial image is corrected to obtain the verification dial image;

[0140] Based on the verification dial image, Hough transform is used to obtain the verification straight line;

[0141] The class-optimal straight line L' and the verification straight line are compared respectively, and the line where the class-optimal straight line L' and the verification straight line coincide is the pointer straight line; when a light beam is projected toward the pointer-type instrument 1, the illumination angle is changed to change the position of the pointer shadow, that is, the position of the straight line where the pointer shadow is located is changed, and after the illumination angle is changed, the position of the pointer remains unchanged, that is, the position of the straight line where the pointer is located remains unchanged, and the pointer straight line can be obtained according to whether the class-optimal straight line L' coincides with the verification straight line;

[0142] Based on the pointer straight line, the dial image 5 is detected and identified according to the detection model to obtain the pointer boundary box 42; the dial image 5 is detected and identified according to the detection model to obtain the needle axis boundary box 43, see Fig.14 ; The detection model used in this process is the dial content detection model;

[0143] Based on the meter information, a pointer reading is obtained from a pointer bounding box 42 and a needle axis bounding box 43 .

[0144] The bounding box is a rectangular box. The output information of the bounding box includes the horizontal coordinate of the upper left corner point of the bounding box, the vertical coordinate of the upper left corner point of the bounding box, the horizontal coordinate of the lower right corner point of the bounding bar, the vertical coordinate of the lower right corner point of the bounding box, and the bounding box information (for example, the bounding box is a pointer or a needle axis). When the upper left corner point and the lower right corner point of the bounding box are both known, the center of the line between the upper left corner point and the lower right corner point is the center point of the bounding box: that is, the center point of the needle axis bounding box is the needle axis center, and the center point of the pointer bounding box is the pointer center;

[0145] The steps to get the pointer reading based on the needle axis bounding box and the pointer bounding box are:

[0146] Based on the needle axis bounding box, the needle axis center is obtained according to the center point calculation formula; compared with the dial detection center obtained by non-precise methods such as calibration and Hough transform circle detection, the needle axis center is used to calculate the pointer angle to ensure accurate reading and improve the confidence of the reading result;

[0147] Based on the pointer bounding box, the pointer center is obtained according to the center point calculation formula;

[0148] Get the pointer vertex according to the pointer bounding box, the needle axis center and the pointer center;

[0149] Get the pointer angle based on the pointer vertex and the needle axis center, see Fig.13 ; The pointer angle is:

[0150] ;

[0151] In the formula, is the pointer angle, is the horizontal coordinate of the pointer vertex, is the vertical coordinate of the pointer vertex, is the horizontal coordinate of the needle axis center, The ordinate of the needle axis center, is pi;

[0152] Get the minimum scale reading, maximum scale reading, minimum scale angle and maximum scale angle based on the instrument information. Fig.13 ; After the instrument information is determined, the scale 15 and the corresponding angle on the pointer instrument are determined, that is, the reading and angle corresponding to the minimum scale 151 and the reading and angle corresponding to the maximum scale 152 are determined;

[0153] The pointer reading is obtained based on the pointer angle, the minimum scale reading, the maximum scale reading, the minimum scale angle and the maximum scale angle; the pointer reading is:

[0154] ;

[0155] In the formula, r For pointer reading, is the minimum scale reading, is the maximum scale reading, is the minimum scale angle, is the maximum scale angle, is the pointer angle.

[0156] The center point calculation formula is:

[0157] ;

[0158] In the formula, is the horizontal coordinate of the center point of the bounding box, is the ordinate of the center point of the bounding box, is the bounding box, is the horizontal coordinate of the upper left corner of the bounding box, is the ordinate of the upper left corner of the bounding box, is the horizontal coordinate of the lower right corner of the bounding box, The ordinate of the lower right corner of the bounding box.

[0159] When the pin axis bounding box When the acquisition is successful, according to the center point calculation formula, the needle axis center point is:

[0160] ;

[0161] In the formula, is the horizontal coordinate of the center point of the needle axis, is the ordinate of the needle axis center point, is the needle axis bounding box, is the horizontal coordinate of the upper left corner of the needle axis bounding box, is the ordinate of the upper left corner of the needle axis bounding box, is the horizontal coordinate of the lower right corner of the needle axis bounding box, The ordinate of the lower right corner of the needle axis bounding box.

[0162] Likewise, when the pointer bounding box When the acquisition is successful, according to the center point calculation formula, the center point of the pointer is:

[0163] ;

[0164] In the formula, is the horizontal coordinate of the center point of the pointer, is the ordinate of the center point of the pointer, is the pointer bounding box, is the horizontal coordinate of the upper left corner of the pointer bounding box, is the ordinate of the upper left corner of the pointer bounding box, is the horizontal coordinate of the lower right corner of the pointer bounding box, The ordinate of the lower right corner of the pointer's bounding box.

[0165] For the convenience of explanation, the above pointer center point calculation is described in the form of a pointer bounding box, and the horizontal coordinate of the upper left corner of the pointer bounding box is expressed as , the vertical coordinate of the upper left corner of the pointer bounding box is expressed as , the horizontal coordinate of the lower right corner of the pointer bounding box is expressed as , the vertical coordinate of the lower right corner of the pointer bounding box is expressed as , and the following pointer vertex calculation is described in the form of conventional coordinate points, and the horizontal coordinate of the upper left corner of the pointer bounding box is expressed as , the vertical coordinate of the upper left corner of the pointer bounding box is expressed as , the horizontal coordinate of the lower right corner of the pointer bounding box is expressed as , the vertical coordinate of the lower right corner of the pointer bounding box is expressed as :

[0166] When both the pointer center and the needle axis center are known, the pointer vertex can be obtained according to the upper left corner point, lower right corner point of the pointer bounding box and the positional relationship between the pointer center and the needle axis center: that is, the steps for obtaining the pointer vertex according to the pointer bounding box, the needle axis center and the pointer center are:

[0167] when , hour:

[0168] ;

[0169] when , hour:

[0170] ;

[0171] when , hour:

[0172] ;

[0173] when , hour:

[0174] ;

[0175] when , hour:

[0176] ;

[0177] when , hour:

[0178] ;

[0179] when , hour:

[0180] ;

[0181] when , hour:

[0182] ;

[0183] In the formula, is the horizontal coordinate of the pointer vertex, is the vertical coordinate of the pointer vertex, is the horizontal coordinate of the center of the dial, is the ordinate of the dial center, is the horizontal coordinate of the pointer center, is the ordinate of the pointer center, is the horizontal coordinate of the upper left corner of the pointer bounding box, is the ordinate of the upper left corner of the pointer bounding box, is the horizontal coordinate of the lower right corner of the pointer bounding box, The ordinate of the lower right corner of the pointer's bounding box.

[0184] In some preferred embodiments, the steps of obtaining the pointer straight line are:

[0185] When the number of detection lines is unique, the detection lines are pointer lines. Figure 3 , 13 ;

[0186] When the number of detected lines is at least two, obtain the optimal line:

[0187] If the number of class-optimal lines is unique, the class-optimal line is the pointer line, see Figure 6 , 7 , detect straight line The corresponding first distance Smaller than preset spacing For the class of optimal straight lines, the detection straight line The corresponding first distance Greater than the preset spacing Then detect the straight line It is not a class-optimal straight line, that is, the number of class-optimal straight lines is unique, and the detection line For the pointer straight line;

[0188] If the number of class-optimal lines is at least two, see Figure 8 , 9 , obtain the verification straight line: the straight line where the superior straight line coincides with the verification straight line is the pointer straight line.

[0189] When the illumination angle changes, the detection straight line corresponding to the pointer does not change, while the detection straight line corresponding to the pointer shadow changes; the pointer shadow is finally eliminated by changing the angle of the projected light beam to confirm the detection straight line where the pointer is located; the confirmation of the pointer straight line can be achieved by projecting the light beam once or multiple times; if only one light beam is projected, the verification dial image includes a first verification dial image, the verification dial boundary box includes a first verification dial boundary box, the initial verification dial image includes an initial first verification dial image, the verification dial image includes the first verification dial image, and the verification straight line includes the first verification straight line; if two light beams are projected, the verification dial image includes a first verification dial image and a second verification dial image, the verification dial boundary box includes a first verification dial boundary box and a second verification dial boundary box, the initial verification dial image includes an initial first verification dial image and an initial second verification dial image, the verification dial image includes a first verification dial image and a second verification dial image, and the verification straight line includes a first verification straight line and a second verification straight line; and so on.

[0190] See also Figure 2 , the steps to obtain a straight line that coincides with the validation straight line are:

[0191] Projecting a light beam toward the pointer instrument at a first position, the first position being located on the same side of at least two quasi-optimal straight lines, for excluding a pointer shadow located on a first side of the pointer, the first side of the pointer being opposite to the first position;

[0192] Obtaining a first verification image based on the pointer instrument behind the projected light beam at the first position;

[0193] Detect the first verification image according to the detection model to obtain a first verification dial boundary box;

[0194] Crop the first verification dial boundary frame to obtain an initial first verification dial image;

[0195] Correcting and processing the initial first verification dial image to obtain a first verification dial image;

[0196] Based on the first verification dial image, a first verification straight line is obtained by using Hough transform;

[0197] Compare the first verification line and the optimal line respectively, retain the optimal line that coincides with the first verification line, and obtain the updated line ;

[0198] When updating the line When the number of is unique, update the straight line It is a straight line for the pointer, i.e. when the light beam is projected towards the pointer instrument at the first position, the position of the pointer shadow just changes;

[0199] When the number of update lines is at least two, that is, when a light beam is projected toward the pointer instrument at the first position, the pointer shadow position does not change:

[0200] Projecting a light beam toward the pointer instrument at a second position, the second position being located on the other side of at least two quasi-optimal straight lines, for excluding a pointer shadow located on a second side of the pointer, the second side of the pointer being opposite to the second position;

[0201] Obtaining a second verification image based on the pointer instrument behind the projected light beam at the second position;

[0202] Detect the second verification image according to the detection model to obtain a second verification dial bounding box;

[0203] Crop the second verification dial boundary box to obtain an initial second verification dial image;

[0204] Correcting and processing the initial second verification dial image to obtain a second verification dial image;

[0205] Based on the second verification dial image, a second verification straight line is obtained by using Hough transform;

[0206] Compare the second verification line and the update line respectively , an update line that coincides with the second verification line The pointer is a straight line.

[0207] By sequentially projecting light beams on both sides of at least two optimal straight lines, all pointer shadows can be eliminated, which is convenient to operate, reduces operating procedures and facilitates control.

[0208] In practical application:

[0209] When the features of the needle shaft exposed part are obvious and can be directly distinguished from the needle shaft shadow, the needle shaft bounding box can be directly inferred by the dial content detection model;

[0210] When the features of the exposed part of the needle shaft are not obvious, it is necessary to exclude the circle where the needle shaft shadow is located before obtaining the needle shaft circle; at this time, the steps to obtain the needle shaft boundary box are:

[0211] Based on the recognition of the dial image, a detection circle is obtained by using Hough transform; a threshold of Hough transform circle detection is set, and a circle having a diameter equivalent to the exposed part of the needle shaft is detected by using the preset threshold of Hough transform circle detection to obtain the detection circle and count the number;

[0212] When the number of detected circles is unique, see Fig.12 At this time, from the main viewing angle, the shadow part of the needle shaft is blocked by the exposed part of the needle shaft. Through the Hough transform circle detection, only the circle where the exposed part of the needle shaft is located can be detected. The detection circle (detection circle ) is unique, that is, the detection circle is the needle axis circle;

[0213] When the number of detected circles is at least two, see Fig.10 , 11 At this time, the circle where the exposed part of the needle shaft and the circle where the shadow of the needle shaft are located can be obtained through Hough transform circle detection, that is, the number of detection circles is at least two, and the second distance is obtained. The second distance is the distance from the center of the detection circle to the detection center of the dial: the second distance is obtained according to the detection circle and the detection center of the dial , respectively ~ , are the distances between the centers of the 1st to rth needle axis detection circles and the center coordinates of the dial, Fig.10 , 11 There are two detection circles, namely detection circle and detection circle , detection circle The corresponding second distance Smaller than detection circle The corresponding second distance , that is, the detection circle is the needle axis circle;

[0214] The detection circle with the second smallest distance is the needle axis circle.

[0215] See also Fig.15 , a patrol device 2, used to implement a pointer-type instrument reading method based on image recognition, including a camera 22, a light shield 23, a floodlight 24, a control module and a calculation module;

[0216] The camera 22 is used to take a picture of the pointer instrument 1 to obtain an identification picture 3 or a verification picture and transmit it to the computing module; preferably, when taking pictures, the camera 22 is facing the instrument dial of the pointer instrument 1, see Fig.16 , in order to reduce the difficulty of subsequent processing and improve the reading accuracy;

[0217] The floodlight 24 is used to project a light beam toward the pointer instrument 1;

[0218] The light shield 23 is used to prevent the projected light beam from directly hitting the photographing part of the camera 22 when the projector 24 projects the light beam;

[0219] The control module is used to control the movement of the inspection device 2, to control the camera to take pictures of the pointer instrument and transmit them to the calculation module according to the instructions of the calculation module, and to control the floodlight to project a light beam toward the pointer instrument according to the instructions of the calculation module;

[0220] The computing module is used to store and run computer-readable instructions. When the computer-readable instructions are run, a pointer instrument reading method based on image recognition is executed, which is used to control the camera 22 to take pictures of the pointer instrument 1 and to control the floodlight to project a light beam toward the pointer instrument.

[0221] Furthermore, the inspection device 2 also includes a support 21, a support rod 26 and a swing arm 25;

[0222] A camera 22 is installed on an axial end surface of the support 21, and a light shield 23 is arranged around the outer side of the camera 22;

[0223] The support rod 26 is installed at the other axial end of the support 21 to support the support 21;

[0224] One end of the swing arm 25 is rotatably matched with the support 21, and a floodlight 24 is arranged at the other end of the swing arm 25. When one end of the swing arm 25 rotates along the support 21, the floodlight 24 rotates around the camera 22 outside the light shield 23. Preferably, the camera 22 is located at the center of the end surface of one axial end, and the light shield 23 is arranged circumferentially along the support 21; the structure is simple, easy to control, and can ensure that the light shield 23 has a light shielding effect on the camera 22 when the floodlight 24 rotates;

[0225] When in use, under the control module, the inspection device 2 inspects the pointer instrument 1 at the site at a predetermined position according to the preset route. The control module receives the instruction of the calculation module to control the camera 22 to take a picture of the needle instrument 1 and transmits it to the calculation module to obtain the identification picture 3. Preferably, the support rod 26 is arranged at the lifting end of the lifting mechanism, and the height of the camera 22 is achieved by controlling the lifting mechanism to improve the applicability of the inspection device 2. The lifting mechanism can be a hydraulic rod or other mechanism that can realize the lifting function. This is a prior art and will not be repeated here.

[0226] The computing module is used to:

[0227] Obtain recognition model and detection model library according to training data set;

[0228] Obtain identification image 3 according to the pointer instrument;

[0229] Identify the image 3 according to the recognition model to obtain the instrument information;

[0230] Select the detection model library to obtain the detection model according to the instrument information;

[0231] Detect and identify the image 3 according to the detection model to obtain an identification dial boundary box 41;

[0232] Obtaining the identified dial image 5 and the dial detection center according to the identified dial boundary box;

[0233] Obtain detection lines based on the identification of the dial image using Hough transform, and determine the number of detection lines; if the number of detection lines is at least two, obtain a preset spacing;

[0234] respectively obtain the first distance;

[0235] The first distance and the preset spacing are compared respectively, and the detection straight line whose first distance is smaller than the preset spacing is a class-excellent straight line, and the number of class-excellent straight lines is determined; if the number of class-excellent straight lines is at least two, the class-excellent straight lines are saved;

[0236] Projecting a light beam toward the pointer instrument, and acquiring a verification image based on the pointer instrument after the light beam is projected;

[0237] Detect the verification image according to the detection model to obtain the verification dial bounding box;

[0238] Obtain a verification dial image according to the verification dial bounding box;

[0239] Based on the verification dial image, Hough transform is used to obtain the verification straight line;

[0240] Compare the superior straight line and the verification straight line respectively, and the line where the superior straight line and the verification straight line coincide is the pointer straight line;

[0241] Based on the pointer straight line, the dial image 3 is detected and identified according to the detection model to obtain the pointer boundary box 42;

[0242] Detect and identify the dial image 3 according to the detection model to obtain the needle axis boundary box 43;

[0243] Based on the meter information, a pointer reading is obtained from a pointer bounding box 42 and a needle axis bounding box 43 .

[0244] Specifically, the calculation module is used to:

[0245] When the number of detected straight lines is unique, the detected straight line is determined to be a pointer straight line;

[0246] When the number of detected lines is at least two, obtain the optimal line:

[0247] If the number of class-optimal straight lines is unique, the class-optimal straight line is determined to be the pointer straight line;

[0248] If the number of the class-optimal straight lines is at least two, obtain the verification straight line: determine that the straight line where the class-optimal straight line coincides with the verification straight line is the pointer straight line.

[0249] The calculation module is also used to:

[0250] Send an instruction to the control module, so that the control module controls the floodlight 24 to project a light beam at the first position. Specifically, the control module controls the swing arm 25 to rotate along the support 21, so that the floodlight 24 rotates to the first position and projects a light beam (the control module controls the swing arm 25 to rotate along the support 21, which is a prior art and will not be repeated here). Preferably, when the floodlight 24 is at the first position, the line projection (the first line projection is the projection of the line connecting the center point of the floodlight 24 and the center point of the camera 22 photographing part on the scale plane 11) is perpendicular to the optimal straight line, that is, the line projection is perpendicular to the center line of the pointer, so that the control module controls the camera 22 to obtain the first verification picture at the first position;

[0251] Detect the first verification image according to the detection model to obtain a first verification dial boundary box;

[0252] Crop the first verification dial boundary frame to obtain an initial first verification dial image;

[0253] Correcting and processing the initial first verification dial image to obtain a first verification dial image;

[0254] Based on the first verification dial image, a first verification straight line is obtained by using Hough transform;

[0255] The first verification straight line and the optimal straight line are compared respectively, and the optimal straight line that coincides with the first verification straight line is retained to obtain an updated straight line;

[0256] When the number of update lines is unique, the update line is a pointer line;

[0257] When the number of update lines is at least two:

[0258] Sending instructions to the control module, so that the control module controls the floodlight 24 to project a light beam at the second position. Specifically, the control module controls the swing arm 25 to rotate along the support 21, so that the floodlight 24 rotates to the second position and projects a light beam. Similarly, preferably, when the floodlight 24 is at the second position, the projection of the connecting line is perpendicular to the quasi-optimal straight line, so that the control module controls the camera 22 at the second position to obtain a second verification picture;

[0259] Detect the second verification image according to the detection model to obtain a second verification dial bounding box;

[0260] Crop the second verification dial boundary box to obtain an initial second verification dial image;

[0261] Correcting and processing the initial second verification dial image to obtain a second verification dial image;

[0262] Based on the second verification dial image, a second verification straight line is obtained by using Hough transform;

[0263] The second verification straight line and the update straight line are compared respectively, and the update straight line that coincides with the second verification straight line is the pointer straight line.

[0264] The calculation module is also used to:

[0265] Based on the recognition of the dial image, the detection circle is obtained by using Hough transform;

[0266] When the number of the detection circle is unique, it is confirmed that the detection circle is the needle axis circle;

[0267] When the number of the detection circles is at least two, a second distance is acquired; and it is confirmed that the detection circle with the smallest second distance is the needle axis circle.

[0268] The system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0269] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0270] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A pointer instrument reading method based on image recognition, characterized in that: The following steps are involved: Obtain recognition model and detection model library according to training data set; Acquire an identification picture according to the pointer instrument, wherein the image content of the identification picture includes the pointer instrument; Recognize the recognition picture according to the recognition model to obtain instrument information; Obtaining a detection model from the detection model library according to the instrument information; Detect the recognition picture according to the detection model to obtain a recognition dial boundary box; Obtaining an identified dial image and a dial detection center according to the identified dial boundary frame; Based on the identified dial image, a detection straight line is obtained by using Hough transform; If the number of the detection straight lines is at least two, obtaining a preset spacing; respectively acquiring a first distance, where the first distance is the distance from the detection center of the dial to the detection straight line; The first distance and the preset distance are compared respectively, and a detection straight line whose first distance is smaller than the preset distance is a quasi-optimal straight line; If the number of the class-optimal straight lines is at least two, then the class-optimal straight lines are saved; Projecting a light beam toward the pointer instrument, and acquiring a verification image based on the pointer instrument after the light beam is projected; Detect the verification image according to the detection model to obtain a verification dial boundary box; Obtaining a verification dial image according to the verification dial boundary box; Based on the verification dial image, a verification straight line is obtained by using Hough transform; The class-optimal straight line and the verification straight line are compared respectively, and the straight line where the class-optimal straight line and the verification straight line coincide with each other is the pointer straight line; Based on the pointer straight line, detecting the identified dial image according to the detection model to obtain a pointer bounding box; Detect the identified dial image according to the detection model to obtain a needle axis boundary box; Based on the instrument information, a pointer reading is obtained according to the pointer bounding box and the needle axis bounding box.

2. The method for reading a pointer-type instrument based on image recognition according to claim 1, characterized in that: When the number of the detection straight lines is unique, the detection straight line is the pointer straight line; If the number of the class-optimal straight lines is unique, the class-optimal straight line is the pointer straight line.

3. The method for reading pointer-type instruments based on image recognition according to claim 1, characterized in that: Projecting a light beam toward the pointer instrument includes projecting a light beam toward the pointer instrument at a first position, or projecting a light beam toward the pointer instrument includes projecting a light beam toward the pointer instrument at a first position and projecting a light beam toward the pointer instrument at a second position; Projecting a light beam toward the pointer instrument at a first position, the first position being located on the same side of the at least two quasi-optimal straight lines; Obtaining a first verification image based on the pointer instrument after the light beam is projected at the first position; Detecting the first verification image according to the detection model to obtain a first verification dial boundary box; Crop the first verification dial boundary frame to obtain an initial first verification dial image; Correcting and processing the initial first verification dial image to obtain a first verification dial image; Obtain a first verification straight line by using Hough transform based on the first verification dial image; Compare the first verification straight line and the optimal straight line respectively, retain the optimal straight line that coincides with the first verification straight line, and obtain an updated straight line; When the number of the update straight lines is unique, the update straight line is the pointer straight line; When the number of the update straight lines is at least two: Projecting a light beam toward the pointer instrument at a second position, the second position being located on the other side of the at least two quasi-optimal straight lines; Obtaining a second verification image based on the pointer instrument behind the light beam projected at the second position; Detecting the second verification image according to the detection model to obtain a second verification dial boundary box; Crop the second verification dial boundary frame to obtain an initial second verification dial image; Correcting and processing the initial second verification dial image to obtain a second verification dial image; Obtain a second verification straight line by using Hough transform based on the second verification dial image; The second verification straight line and the update straight line are compared respectively, and the update straight line that coincides with the second verification straight line is the pointer straight line.

4. The method for reading pointer-type instruments based on image recognition according to claim 1, characterized in that: The steps for obtaining the needle axis bounding box are: Based on the identified dial image, a detection circle is obtained by using Hough transform; When the number of the detection circles is unique, the detection circle is the needle axis circle; When the number of the detection circles is at least two, acquiring a second distance, where the second distance is a distance from a center of the detection circle to a detection center of the dial; The detection circle with the smallest second distance is the needle axis circle.

5. The method for reading pointer-type instruments based on image recognition according to claim 1, characterized in that: The detection model includes an instrument dial detection model and a dial content detection model; The instrument dial detection model is used to obtain a dial bounding box according to a picture, wherein the picture includes an identification picture and a verification picture, and the dial bounding box includes an identification dial bounding box and a verification dial bounding box: Detecting the identification picture according to the instrument dial detection model to obtain the identification dial boundary box; Detecting the verification image according to the instrument dial detection model to obtain the verification dial boundary box; The dial content detection model is used to obtain the pointer bounding box and the needle axis bounding box according to the identified dial image.

6. The method for reading pointer-type instruments based on image recognition according to claim 1, characterized in that: The steps of obtaining the pointer reading according to the needle axis bounding box and the pointer bounding box are: Based on the needle axis bounding box, obtaining the needle axis center according to a center point calculation formula; Based on the pointer bounding box, obtaining the pointer center according to a center point calculation formula; Obtaining a pointer vertex according to the pointer bounding box, the needle axis center and the pointer center; The pointer angle is obtained according to the pointer vertex and the needle axis center; the pointer angle is: ； In the formula, is the pointer angle, is the horizontal coordinate of the pointer vertex, is the vertical coordinate of the pointer vertex, is the horizontal coordinate of the needle axis center, The ordinate of the needle axis center, is the ratio of pi; Acquire a minimum scale reading, a maximum scale reading, a minimum scale angle, and a maximum scale angle based on the instrument information; The pointer reading is obtained according to the pointer angle, the minimum scale reading, the maximum scale reading, the minimum scale angle and the maximum scale angle; the pointer reading is: ； In the formula, r For pointer reading, is the minimum scale reading, is the maximum scale reading, is the minimum scale angle, is the maximum scale angle, is the pointer angle.

7. The method for reading pointer-type instruments based on image recognition according to claim 6, characterized in that: The center point calculation formula is: ； In the formula, is the horizontal coordinate of the center point of the bounding box, is the ordinate of the center point of the bounding box, is the bounding box, is the horizontal coordinate of the upper left corner of the bounding box, is the ordinate of the upper left corner of the bounding box, is the horizontal coordinate of the lower right corner of the bounding box, The ordinate of the lower right corner of the bounding box.

8. The method for reading pointer-type instruments based on image recognition according to claim 6, characterized in that: The steps of obtaining the pointer vertex according to the pointer bounding box, the needle axis center and the pointer center are: when , hour: ； when , hour: ； when , hour: ； when , hour: ； when , hour: ； when , hour: ； when , hour: ； when , hour: ； In the formula, is the horizontal coordinate of the pointer vertex, is the vertical coordinate of the pointer vertex, is the horizontal coordinate of the center of the dial, is the ordinate of the dial center, is the horizontal coordinate of the pointer center, is the ordinate of the pointer center, is the horizontal coordinate of the upper left corner of the pointer bounding box, is the ordinate of the upper left corner of the pointer bounding box, is the horizontal coordinate of the lower right corner of the pointer bounding box, The ordinate of the lower right corner of the pointer's bounding box.

9. A patrol inspection device, used to implement the pointer instrument reading method based on image recognition according to any one of claims 1 to 8, characterized in that: It includes a camera, a light shield, a floodlight, a control module and a computing module; The camera is used to take a picture of the pointer instrument and transmit it to the calculation module; The floodlight is used to project a light beam toward the pointer instrument; The light shield is used to prevent the projected light beam from directly hitting the photographing part of the camera when the projector projects a light beam; The control module is used to control the camera to take pictures of the pointer instrument and transmit the pictures to the calculation module according to the instructions of the calculation module, and to control the floodlight to project a light beam toward the pointer instrument according to the instructions of the calculation module; The computing module is used to store and run computer-readable instructions, and the computer-readable instructions execute the pointer instrument reading method based on image recognition when running.

10. The inspection device according to claim 9, characterized in that: It also includes a support, a support rod and a swing arm; The camera is installed on an axial end surface of the support, and the light shield is arranged around the outside of the camera; The support rod is installed at the other axial end of the support to support the support; One end of the swing arm is rotatably matched with the support, and the other end of the swing arm is provided with the floodlight. When one end of the swing arm rotates along the support, the floodlight rotates around the camera outside the light shield.

Images