Pointer-type instrument reading recognition method, device and terminal device

Abstract

This application is applicable to the field of instrument recognition technology, and provides a method, device and terminal device for recognizing the readings of pointer-type instruments. The method includes: segmenting the scales and pointers of a pointer-type instrument according to a pointer-type instrument image to obtain a plurality of scale lines and a pointer; performing text recognition on the pointer-type instrument image to obtain a plurality of scale values; matching the scale values with the scale lines according to the characteristic information of the scale values and the characteristic information of the scale lines to determine the scale values corresponding to each scale line; respectively fitting the scale lines and the pointer to obtain a scale fitting circle and a pointer fitting straight line; determining the intersection point of the pointer fitting straight line and the scale fitting circle and the scale interval where the pointer is located according to the scale fitting circle, the pointer fitting straight line and the scale values corresponding to each scale; determining the reading of the pointer-type instrument according to the scale interval where the pointer is located and the intersection point. This application can achieve accurate segmentation of the pointer and the scale, and can accurately calculate the reading of the instrument.

Description

Technical Field

[0001] The present application relates to the technical field of instrument recognition, and particularly to a method, device and terminal device for recognizing readings of pointer-type instruments. Background Art

[0002] As an important tool in the field of measurement and control, instruments are widely used in different fields. Common instruments in the power system include pressure gauges, oil temperature gauges, ammeters, etc. The traditional method is to rely on humans to read the values, which is time-consuming and laborious. Therefore, it is necessary to design a method that can automatically read the values. Scholars at home and abroad have conducted a large number of studies on automatically reading the scales of instruments, and most of the studies are based on traditional image processing methods, such as using Canny to detect the contour information of the instrument, Hough Transform to detect circular instruments, and the least squares method to fit the pointer and the scale. The traditional algorithm detects the pointer and the scale through thresholds, and then reads the reading of the pointer through the radian. However, this algorithm has poor generalization ability and is easily affected by angles, lighting, etc., resulting in unreliable recognition results. Summary of the Invention

[0003] In view of this, the embodiments of the present application provide a method, device and terminal device for recognizing readings of pointer-type instruments, which can achieve accurate segmentation of the pointer and the scale, and can accurately calculate the reading of the instrument, and show good robustness and generalization ability in the recognition of readings of different types of instruments.

[0004] To achieve the above object, the present application adopts the following technical solutions:

[0005] In a first aspect, the embodiments of the present application provide a method for recognizing readings of pointer-type instruments, including:

[0006] Segmenting the scale and the pointer of the pointer-type instrument according to the pointer-type instrument image to obtain a plurality of scale lines and a pointer;

[0007] Performing text recognition on the pointer-type instrument image to obtain a plurality of scale values;

[0008] Matching the plurality of scale values with the plurality of scale lines according to the characteristic information of the scale values and the characteristic information of the scale lines to determine the scale value corresponding to each scale line; wherein, the characteristic information of the scale values includes the position, angle and direction of the scale values, and the characteristic information of the scale lines includes the position, angle and direction of the scale lines;

[0009] Fitting the plurality of scale lines and the one pointer respectively to obtain a scale fitting circle and a pointer fitting straight line;

[0010] Determine the intersection point of the pointer fitting line and the scale fitting circle and the scale interval where the pointer is located according to the scale fitting circle, the pointer fitting line, and the scale values corresponding to each scale;

[0011] Determine the reading of the pointer-type instrument according to the scale interval where the pointer is located and the intersection point.

[0012] Based on the first aspect, in some embodiments, the matching of the multiple scale values and the multiple scale lines according to the characteristic information of the scale values and the characteristic information of the scale lines to determine the scale values corresponding to each scale line includes:

[0013] Determine the first position coordinates x i and y i of the i-th scale value, as well as the first angle θ i and the first direction v i ;

[0014] Determine the second position coordinates x j and y j of the j-th scale line, as well as the second angle θ j and the first direction v j ;

[0015] According to the formula calculate the Euclidean distance between the characteristic information c i of the i-th scale value and the characteristic information c j of the j-th scale line;

[0016] According to the formula calculate the similarity s ij between the i-th scale value and the j-th scale line, where σ is the standard deviation of c i and c j ;

[0017] According to calculate the weight between the i-th scale value and the j-th scale line to obtain the weights between each scale value and all scale lines;

[0018] Match the scale line with the largest weight with the i-th scale value, thereby obtaining the scale line matched with each scale value.

[0019] Based on the first aspect, in some embodiments, the fitting of the multiple scale lines and the one pointer respectively to obtain a scale fitting circle and a pointer fitting line includes:

[0020] Determine the center point of each scale line;

[0021] Use the least squares method to fit the center points of each scale line to obtain the scale fitting circle;

[0022] Perform a Hough transform on the pointer to generate the pointer fitting line, and the pointer fitting line intersects the scale fitting circle at two points.

[0023] Based on the first aspect, in some embodiments, the determining the intersection points of the pointer fitting line and the scale fitting circle and the scale interval where the pointer is located according to the scale fitting circle, the pointer fitting line, and the scale values corresponding to each scale includes:

[0024] Determine the target point on the pointer that is farthest from the center of the scale fitting circle;

[0025] Determine the two intersection points of the scale fitting circle and the pointer fitting line, and determine the target intersection point closest to the target point among the two intersection points;

[0026] Calculate the distance between the center point of each scale line and the target intersection point, and use the scale interval formed by the two scale lines corresponding to the two smallest distances as the scale interval where the pointer is located.

[0027] Based on the first aspect, in some embodiments, the determining the reading of the pointer-type instrument according to the scale interval where the pointer is located and the intersection points includes:

[0028] Determine the starting position and the ending position of the scale interval;

[0029] Calculate the first radian α between the starting position and the ending position, and the second radian β between the starting position and the target intersection point;

[0030] According to Determine the reading of the pointer-type instrument;

[0031] where value is the reading of the pointer-type instrument, maxvalue is the scale value corresponding to the ending position of the scale interval, minvalue is the scale value corresponding to the starting position of the scale interval, and x 1 and y 1 are the coordinates of the starting position of the scale interval, x 2 and y 2 are the coordinates of the ending position of the scale interval, x 3 and y 3 are the coordinates of the target intersection point, x c and y c are the coordinates of the center of the scale fitting circle.

[0032] Based on the first aspect, in some embodiments, before the step of segmenting the scale and the pointer of the pointer-type instrument according to the pointer-type instrument image, the pointer-type instrument reading recognition method further includes:

[0033] Obtain an original image, where the original image contains a pointer-type instrument;

[0034] Based on an improved YOLOv9 model, determine the position of the pointer-type instrument from the original image, and extract the pointer-type instrument image;

[0035] Among them, in the improved YOLOv9 model, the P3 feature map and the conv-reg module, conv-cls module, and concat module connected to the P3 feature map are deleted;

[0036] The determination process of the ahchor parameter in the conv-reg module of the P4 feature map and P5 feature map of the improved YOLOv9 model is as follows: Obtain the sizes of the instruments after annotation in multiple pointer-type instrument images, and calculate the multiple instrument sizes through the mean algorithm to obtain an ahchor box that can represent the pointer-type instrument.

[0037] Based on the first aspect, in some embodiments, the step of segmenting the scale and pointer of the pointer-type instrument according to the pointer-type instrument image to obtain multiple scale lines and a pointer includes:

[0038] Segment the pointer-type instrument image through the U2Net network algorithm to obtain multiple scale lines and a pointer;

[0039] Among them, only the En_1 layer, En_3 layer, En_5 layer, and En_6 layer of the U2Net network algorithm are used to extract the scale line features and pointer features in the pointer-type instrument image.

[0040] Based on the first aspect, in some embodiments, the step of performing text recognition on the pointer-type instrument image to obtain multiple scale values includes:

[0041] Extract features from the pointer-type instrument image through convolution and sampling operations to obtain a feature map;

[0042] Perform a deconvolution operation on the feature map to obtain a probability map;

[0043] Process the feature map to determine the pixel threshold corresponding to each pixel point in the feature map;

[0044] According to the pixel threshold corresponding to each pixel point in the feature map, perform a differentiable binarization operation on the probability map to obtain a binarized map;

[0045] Based on the binarized map, perform text region segmentation on the pointer-type instrument image to obtain a text image region containing text content;

[0046] Perform text recognition on each text image region to obtain multiple scale values.

[0047] In a second aspect, an embodiment of the present application provides a pointer-type instrument reading recognition device, including:

[0048] A segmentation module, configured to segment the scale and the pointer of the pointer-type instrument according to the pointer-type instrument image to obtain multiple scale lines and a pointer;

[0049] A text recognition module, configured to perform text recognition on the pointer-type instrument image to obtain multiple scale values;

[0050] A matching module, configured to match the multiple scale values with the multiple scale lines according to the feature information of the scale values and the feature information of the scale lines, and determine the scale value corresponding to each scale line; wherein, the feature information of the scale value includes the position, angle, and direction of the scale value, and the feature information of the scale line includes the position, angle, and direction of the scale line;

[0051] A fitting module, configured to respectively fit the multiple scale lines and the one pointer to obtain a scale fitting circle and a pointer fitting straight line;

[0052] A scale interval determination module, configured to determine the intersection point of the pointer fitting straight line and the scale fitting circle and the scale interval where the pointer is located according to the scale fitting circle, the pointer fitting straight line, and the scale value corresponding to each scale;

[0053] A reading determination module, configured to determine the reading of the pointer-type instrument according to the scale interval where the pointer is located and the intersection point.

[0054] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the pointer-type instrument reading recognition method described in any one of the first aspects above are implemented.

[0055] The beneficial effects of the embodiments of the present application compared with the prior art are:

[0056] The above-mentioned method for identifying the reading of a pointer-type instrument first segments the scale and the pointer of the pointer-type instrument according to the pointer-type instrument image to obtain multiple scale lines and a pointer, and performs text recognition on the pointer-type instrument image to obtain multiple scale values. Then, the scale values and the scale lines are matched according to the characteristic information of the scale values and the characteristic information of the scale lines. Next, the scale lines and the pointer are respectively fitted to obtain a scale fitting circle and a pointer fitting straight line. According to the scale fitting circle, the pointer fitting straight line, and the scale values corresponding to each scale, the intersection point of the pointer fitting straight line and the scale fitting circle and the scale interval where the pointer is located are determined. Finally, according to the scale interval where the pointer is located and the intersection point, the reading of the pointer-type instrument is determined. The embodiments of the present application can achieve accurate segmentation of the pointer and the scale lines, and can accurately calculate the reading of the instrument, and have good robustness and generalization ability. Description of the Drawings

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to these drawings.

[0058] Figure 1 It is a schematic flowchart of the method for identifying the reading of a pointer-type instrument provided by the embodiments of the present application;

[0059] Figure 2 It is a schematic diagram of the segmented scale lines and the pointer provided by the embodiments of the present application;

[0060] Figure 3 It is a schematic diagram of the matching effect between the scale lines and the scale values provided by the embodiments of the present application;

[0061] Figure 4 It is a schematic diagram of the scale fitting circle and the pointer fitting straight line provided by the embodiments of the present application;

[0062] Figure 5 It is a schematic diagram of the scale fitting circle and the pointer fitting straight line presented on the pointer-type instrument provided by the embodiments of the present application;

[0063] Figure 6 It is a schematic diagram of the process of calculating the reading of a pointer-type instrument provided by the embodiments of the present application;

[0064] Figure 7 It is a schematic diagram of the recognition effect of the pressure gauge reading provided by the embodiments of the present application;

[0065] Figure 8 It is a schematic diagram of the recognition effect of the oil level gauge reading provided by the embodiments of the present application;

[0066] Figure 9 It is a schematic diagram showing the recognition effect of the voltmeter reading provided by an embodiment of the present application;

[0067] Figure 10 It is a schematic structural diagram of a pointer-type instrument reading recognition device provided by an embodiment of the present application;

[0068] Figure 11 It is a schematic diagram of a terminal device provided by an embodiment of the present application. Specific embodiments

[0069] The present application will be described more clearly below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the role of the present application, but do not limit the present application in any form. It should be noted that those of ordinary skill in the art can make several deformations and improvements without departing from the concept of the present application. These all belong to the protection scope of the present application.

[0070] To make the purpose, technical solution and advantages of the present application clearer, the following will be described through specific embodiments in conjunction with the accompanying drawings.

[0071] See Figure 1 , the pointer-type instrument reading recognition method provided by the embodiment of the present application may include the following steps:

[0072] Step 101, segment the scale and pointer of the pointer-type instrument according to the pointer-type instrument image to obtain a plurality of scale lines and a pointer.

[0073] In some embodiments, step 101 may include: segmenting the pointer-type instrument image through the U2Net network algorithm to obtain a plurality of scale lines and a pointer; wherein, only the En_1 layer, En_3 layer, En_5 layer and En_6 layer of the U2Net network algorithm are used to extract the scale line features and pointer features in the pointer-type instrument image.

[0074] Specifically, the U2Net network algorithm uses the RSU module to extract image features. This RSU module can capture more context information, fuse features of different scales, and combine residual connections and the U-Net structure to effectively improve the feature extraction ability. Figure 2 They are a plurality of scale lines and a pointer obtained by segmentation. However, the original U2Net network structure has 6 layers, which is relatively redundant and prone to overfitting. In the present application, only the En_1, En_3, En_5, and En_6 layers of the U2Net network are used to extract instrument features. On the basis of maintaining the recognition accuracy, the inference speed is accelerated, making it more efficient in extracting the significant regions of the image. By segmenting the pointer-type instrument image, the precise positions of each scale line and the pointer on the pointer-type instrument can be obtained, laying a foundation for subsequent reading recognition.

[0075] Step 102: Perform text recognition on the pointer-type instrument image to obtain multiple scale values.

[0076] In some embodiments, Step 102 may include: extracting features from the pointer-type instrument image through convolution and sampling operations to obtain a feature map; performing a deconvolution operation on the feature map to obtain a probability map, where the probability map represents the probability that each pixel point in the feature map is text; processing the feature map to determine the pixel threshold corresponding to each pixel point in the feature map; performing a differentiable binarization operation on the probability map according to the pixel threshold corresponding to each pixel point in the feature map to obtain a binarized map; based on the binarized map, performing text region segmentation on the pointer-type instrument image to obtain a text image region containing text content; and performing text recognition on each text image region to obtain multiple scale values.

[0077] Specifically, the process of extracting features from the pointer-type instrument image through convolution and sampling operations to obtain a feature map may include:

[0078] Step A1: Perform convolution downsampling on the pointer-type instrument image to obtain a first feature layer, a second feature layer, a third feature layer, and a fourth feature layer. The size of the fourth feature layer is one-fourth of the pointer-type instrument image, the size of the third feature layer is one-eighth of the pointer-type instrument image, the size of the second feature layer is one-sixteenth of the pointer-type instrument image, and the size of the first feature layer is one-thirty-second of the pointer-type instrument image.

[0079] Step A2: Upsample the first feature layer by a factor of two and then add the corresponding pixel values to the second feature layer to obtain a sixth feature layer. Upsample the sixth feature layer by a factor of two and then add the corresponding pixel values to the third feature layer to obtain a seventh feature layer. Upsample the seventh feature layer by a factor of two and then add the corresponding pixel values to the fourth feature layer to obtain an eighth feature layer.

[0080] Step A3: Perform the following processing on the fifth feature layer to the eighth feature layer: perform 3×3 convolution on the eighth feature layer, perform 3×3 convolution on the seventh feature layer and then upsample it by a factor of two, perform 3×3 convolution on the sixth feature layer and then upsample it by a factor of four, and perform 3×3 convolution on the fifth feature layer and then upsample it by a factor of eight. Connect the processed fifth feature layer, sixth feature layer, seventh feature layer, and eighth feature layer together through the concat function to generate the above-mentioned feature map.

[0081] In some embodiments, the process of performing deconvolution on the feature map to obtain the probability map is as follows: Step B1, the feature map first passes through a convolutional layer to compress the number of channels to 1 / 4 of the input, and then passes through BN and relu, and the size of the obtained feature map is one-fourth of the pointer-type instrument image; Step B2, perform a 2×2 deconvolution operation on the obtained feature map, and the size of the obtained feature map is one-half of the pointer-type instrument image; Step B3, perform a 2×2 deconvolution operation on the feature map, the number of channels of the output feature map is 1, and the size of the obtained feature map is the same as that of the pointer-type instrument image; Step B4, process the feature map obtained in Step B3 through the sigmoid function to obtain the probability map.

[0082] In some embodiments, the above-mentioned differentiable binarization operation is performed on the probability map according to the pixel threshold corresponding to each pixel point in the feature map to obtain the binarized map, including: By Obtain the binarized map, where B i,j is the pixel value of the pixel point at the i-th row and j-th column in the binarized map, P i,j is the pixel value of the pixel point at the i-th row and j-th column in the probability map, T i,j is the pixel threshold corresponding to the pixel point at the i-th row and j-th column in the feature map, and k is the magnification factor, for example, k = 50.

[0083] For text recognition of each text image region, the CRNN+CTC text recognition algorithm can be used. This algorithm can perform end-to-end training, that is, directly predict the final text sequence from the original image without preprocessing steps such as artificial feature extraction or character segmentation, simplifies the model training process, improves the training efficiency, can handle text sequences of any length, is not limited by the number of characters in the input image, and is suitable for recognizing text in various scenarios, such as text in natural scenes, table text, long text, etc.

[0084] Step 103, according to the feature information of the scale values and the feature information of the scale lines, match the multiple scale values with the multiple scale lines to determine the scale value corresponding to each scale line.

[0085] Among them, the characteristic information of the scale value includes the position, angle, and direction of the scale value, and the characteristic information of the scale line includes the position, angle, and direction of the scale line. The direction of the scale value and the scale line is from the inside of the instrument to the outside. The angle of the scale value can be the angle corresponding to the connection line between the center of the scale value and the center of the circle in a coordinate system with the center of the circle fitted by the scale value as the origin. For example, the angle between the connection line between the center of the scale value and the center of the circle and the coordinate axis of the coordinate system. The angle of the scale line can be the angle corresponding to the connection line between the center of the scale line and the center of the circle in a coordinate system with the center of the circle fitted by the scale line as the origin. For example, the angle between the connection line between the center of the scale line and the center of the circle and the coordinate axis of the coordinate system. Among them, the circle fitted by the scale value and the circle fitted by the scale line have the same center.

[0086] In some embodiments, step 103 may specifically include:

[0087] Determine the first position coordinates x i and y i of the i-th scale value, as well as the first angle θ i and the first direction v i ;

[0088] Determine the second position coordinates x j and y j of the j-th scale line, as well as the second angle θ j and the first direction v j ;

[0089] According to the formula calculate the Euclidean distance between the characteristic information c i of the i-th scale value and the characteristic information c j of the j-th scale line;

[0090] According to the formula calculate the similarity s ij between the i-th scale value and the j-th scale line, where σ is the standard deviation of c i and c j used to measure the degree of dispersion of c i and c j ;

[0091] According to calculate the weight between the i-th scale value and the j-th scale line, and obtain the weights between each scale value and all scale lines;

[0092] Match the scale line with the largest weight with the i-th scale value, and thus obtain the scale line matched with each scale value.

[0093] Among them, the position of the scale value can be the position of the center point of the scale value, and the position of the scale line can be the position of the center point of the scale line. In this step, by matching the position, angle, and direction of the scale value and the position, angle, and direction of the scale line, the matching accuracy between the scale and the pointer can be improved, and the matching result is as Figure 3 shown.

[0094] Step 104: Fit each of the multiple scale lines and the one pointer respectively to obtain a scale fitting circle and a pointer fitting straight line.

[0095] In some embodiments, step 104 may include: determining the center point of each scale line; using the least squares method to fit the center points of each scale line to obtain the scale fitting circle; performing a Hough transform on the pointer to generate the pointer fitting straight line, and the pointer fitting straight line intersects the scale fitting circle at two points.

[0096] In this step, for each scale line obtained by segmenting the instrument, the center point of the scale line mask is taken, and the least squares method is used to determine an optimal fitting circle, that is, the scale fitting circle as Figure 4 shown, which can ensure that the sum of the squares of the distances from the scale fitting circle to all center points reaches the minimum, so as to obtain an accurate circle to represent the scale dial of the instrument. For the pointer part obtained by segmentation, the Hough transform is used for straight line fitting, that is, the pointer fitting straight line as Figure 4 shown. The Hough transform can effectively detect straight lines from the image and maintain high accuracy even in the presence of noise. By this method, a straight line representing the pointer of the instrument can be determined. The scenario of presenting the fitted scale fitting circle and pointer fitting straight line on the pointer-type instrument is as Figure 5 shown.

[0097] Step 105: Determine the intersection points of the pointer fitting straight line and the scale fitting circle and the scale interval where the pointer is located according to the scale fitting circle, the pointer fitting straight line, and the scale values corresponding to each scale.

[0098] In some embodiments, step 105 may include: determining a target point on the pointer that is farthest from the center of the scale fitting circle; determining the two intersection points of the scale fitting circle and the pointer fitting straight line, and determining the target intersection point closest to the target point among the two intersection points; calculating the distances between the center points of each scale line and the target intersection point, and taking the scale interval formed by the two scale lines corresponding to the two smallest distances as the scale interval where the pointer is located.

[0099] Step 106: Determine the reading of the pointer-type instrument according to the scale interval where the pointer is located and the intersection points.

[0100] In some embodiments, step 106 may include: determining the starting position and the ending position of the scale interval; calculating a first radian α between the starting position and the ending position, and a second radian β between the starting position and the target intersection point; according to determine the reading of the pointer-type instrument.

[0101] where value is the reading of the pointer-type instrument, maxvalue is the scale value corresponding to the ending position of the scale interval, minvalue is the scale value corresponding to the starting position of the scale interval, and x 1 and y 1 are the coordinates of the starting position of the scale interval, x 2 and y 2 are the coordinates of the ending position of the scale interval, x 3 and y 3 are the coordinates of the target intersection point, x c and y c are the coordinates of the center of the scale fitting circle.

[0102] As Figure 6 shown, the pointer fitting line and the scale fitting circle have two intersection points, so it is first necessary to confirm the quadrant (also known as the scale interval) where the pointer is located. The point on the pointer mask that is farthest from the center c(x c ,y c ) of the scale fitting circle is the quadrant where the pointer is located. The intersection points of the pointer fitting line and the scale fitting circle are obtained by substituting the pointer fitting line equation into the equation of the scale fitting circle to form a quadratic equation, and solving this equation can obtain the coordinates p 3 (x 3 ,y 3 ); calculate the angular values of the center points of each scale line, so as to determine the two scales adjacent to p 3 , that is, the starting position p 1 (x 1 ,y 1 ) and the ending position p 2 (x 2 ,y 2 ) of the scale area where the pointer is located. Calculate the radian α from p 1 to p 2 , and the radian β from p 1 to p 3 , and finally calculate the pointer reading according to the ratio of α to β and the starting value and ending value of the scale interval where the pointer is located.

[0103] Specifically, the reading of the pointer-type instrument can be determined according to . Where value is the reading of the pointer-type instrument, maxvalue is the ending position p of the scale interval2 The corresponding scale value, where minvalue is the starting position p of the scale range 1 The corresponding scale value, and x 1 and y 1 are the starting position p of the scale range 1 coordinates, x 2 and y 2 are the ending position p of the scale range 2 coordinates, x 3 and y 3 are the coordinates of the target intersection point p 3 coordinates, x c and y c are the coordinates of the center c of the scale fitting circle.

[0104] In some embodiments, before step 101, the above method for identifying pointer instrument readings may further include: obtaining an original image that contains a pointer instrument; determining the position of the pointer instrument from the original image based on an improved YOLOv9 model, and extracting the pointer instrument image. Among them, in the improved YOLOv9 model, the P3 feature map and the conv-reg module, conv-cls module, and concat module connected to the P3 feature map are deleted; the determination process of the anchor parameter in the conv-reg module of the P4 feature map and P5 feature map of the improved YOLOv9 model is: obtaining the sizes of the instrument after annotation in multiple pointer instrument images, and calculating the multiple instrument sizes through the mean algorithm to obtain an anchor box that can represent the pointer instrument.

[0105] The YOLOv9 model introduces pioneering technologies such as programmable gradient information (PGI) and generalized efficient layer aggregation network (GELAN), marking a major advancement in the field of real-time object detection. Since the object to be recognized is a meter rather than a small object, the anchor parameter in the border regression module conv-reg in the original YOLOv9 model, replacing the small object anchor with the size of the instrument after analyzing the dataset (including multiple pointer instrument images), can calculate an anchor box that best represents the pointer instrument through the K-mean algorithm, thereby improving the meter detection accuracy. Since the size of the meter is relatively fixed, the P3 feature map in the original YOLOv9 model is used to detect smaller objects, so the P3 feature map in the original YOLOv9 model and the conv-reg module, conv-cls module, and concat module connected to the P3 feature map are deleted, thereby improving its detection speed faster. Applying the improved YOLOV9 model to meter detection ensures both detection speed and detection quality.

[0106] See Figures 7 to 9 , when this application is applied to different types of instrument images such as pressure gauges, oil level gauges, and voltmeters, it can demonstrate good robustness and generalization ability.

[0107] For the above-mentioned method for identifying the reading of a pointer-type instrument, first, the scale and pointer of the pointer-type instrument are segmented based on the pointer-type instrument image to obtain multiple scale lines and a pointer, and text recognition is performed on the pointer-type instrument image to obtain multiple scale values. Then, the scale values and scale lines are matched according to the characteristic information of the scale values and the characteristic information of the scale lines. Next, the scale lines and the pointer are respectively fitted to obtain a scale fitting circle and a pointer fitting straight line. According to the scale fitting circle, the pointer fitting straight line, and the scale values corresponding to each scale, the intersection point of the pointer fitting straight line and the scale fitting circle and the scale interval where the pointer is located are determined. Finally, according to the scale interval where the pointer is located and the intersection point, the reading of the pointer-type instrument is determined. The embodiments of this application can achieve accurate segmentation of the pointer and scale lines, and can accurately calculate the reading of the instrument, and have good robustness and generalization ability.

[0108] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The order of execution of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of this application.

[0109] Corresponding to the method for identifying the reading of a pointer-type instrument described in the above embodiments, Figure 10 The structural block diagram of the device for identifying the reading of a pointer-type instrument provided by the embodiments of this application is shown. For the sake of convenience of description, only the parts related to the embodiments of this application are shown.

[0110] See Figure 10 , the embodiments of this application provide a device for identifying the reading of a pointer-type instrument, including a segmentation module 201, a text recognition module 202, a matching module 203, a fitting module 204, a scale interval determination module 205, and a reading determination module 206.

[0111] The segmentation module 201 is used to segment the scale and pointer of the pointer-type instrument according to the pointer-type instrument image to obtain multiple scale lines and a pointer.

[0112] The text recognition module 202 is used to perform text recognition on the pointer-type instrument image to obtain multiple scale values.

[0113] The matching module 203 is used to match the multiple scale values and the multiple scale lines according to the characteristic information of the scale values and the characteristic information of the scale lines, and determine the scale values corresponding to each scale line. Among them, the characteristic information of the scale values includes the position, angle, and direction of the scale values, and the characteristic information of the scale lines includes the position, angle, and direction of the scale lines.

[0114] The fitting module 204 is configured to respectively fit the plurality of scale lines and the one pointer to obtain a scale fitting circle and a pointer fitting straight line.

[0115] The scale interval determination module 205 is configured to determine the intersection points of the pointer fitting straight line and the scale fitting circle and the scale interval where the pointer is located according to the scale fitting circle, the pointer fitting straight line, and the scale values corresponding to the respective scales.

[0116] The reading determination module 206 is configured to determine the reading of the pointer-type instrument according to the scale interval where the pointer is located and the intersection points.

[0117] Optionally, the matching module 203 is specifically configured to:

[0118] Determine the first position coordinates x i and y i of the i-th scale value, as well as the first angle θ i and the first direction v i ;

[0119] Determine the second position coordinates x j and y j of the j-th scale line, as well as the second angle θ j and the first direction v j ;

[0120] Calculate the Euclidean distance between the feature information c of the i-th scale value and the feature information c i of the j-th scale line according to the formula j ;

[0121] Calculate the similarity s between the i-th scale value and the j-th scale line according to the formula ij , where σ is the standard deviation of c i and c j ;

[0122] Calculate the weight between the i-th scale value and the j-th scale line according to to obtain the weights between each scale value and all scale lines;

[0123] Match the scale line with the largest weight between the i-th scale value, thereby obtaining the scale line matched with each scale value.

[0124] Optionally, the fitting module 204 is specifically configured to:

[0125] Determine the center point of each scale line;

[0126] Use the least squares method to fit the center points of each scale line to obtain the scale fitting circle;

[0127] Perform the Hough transform on the pointer to generate the pointer fitting line, and the pointer fitting line intersects the scale fitting circle at two points.

[0128] Optionally, the scale interval determination module 205 is specifically configured to:

[0129] Determine the target point on the pointer that is farthest from the center of the scale fitting circle;

[0130] Determine the two intersection points of the scale fitting circle and the pointer fitting line, and determine the target intersection point closest to the target point among the two intersection points;

[0131] Calculate the distance between the center point of each scale line and the target intersection point, and use the scale interval formed by the two scale lines corresponding to the two smallest distances as the scale interval where the pointer is located.

[0132] Optionally, the reading determination module 206 is specifically configured to:

[0133] Determine the starting position and the ending position of the scale interval;

[0134] Calculate the first radian α between the starting position and the ending position, and the second radian β between the starting position and the target intersection point;

[0135] According to Determine the reading of the pointer-type instrument;

[0136] where value is the reading of the pointer-type instrument, maxvalue is the scale value corresponding to the ending position of the scale interval, minvalue is the scale value corresponding to the starting position of the scale interval, and x 1 and y 1 are the coordinates of the starting position of the scale interval, x 2 and y 2 are the coordinates of the ending position of the scale interval, x 3 and y 3 are the coordinates of the target intersection point, x c and y c are the coordinates of the center of the scale fitting circle.

[0137] Optionally, the above pointer-type instrument reading recognition device further includes an image extraction module, and the image extraction module is used to:

[0138] Obtain the original image, where the original image contains a pointer-type instrument;

[0139] Determine the position of the pointer-type instrument from the original image based on the improved YOLOv9 model, and extract the pointer-type instrument image;

[0140] Among them, in the improved YOLOv9 model, the P3 feature map and the conv-reg module, conv-cls module, and concat module connected to the P3 feature map are deleted;

[0141] The determination process of the ahchor parameter in the conv-reg module of the P4 feature map and P5 feature map of the improved YOLOv9 model is as follows: Obtain the sizes of the instruments after annotation in multiple pointer-type instrument images, and calculate the multiple instrument sizes through the mean algorithm to obtain an ahchor box that can represent the pointer-type instrument.

[0142] Among them, after the image extraction module extracts the pointer-type instrument image, the segmentation module 201 further segments the scale and pointer of the pointer-type instrument according to the pointer-type instrument image to obtain multiple scale lines and a pointer.

[0143] Optionally, the segmentation module 201 is specifically used for:

[0144] Segment the pointer-type instrument image through the U2Net network algorithm to obtain multiple scale lines and a pointer;

[0145] Among them, only the En_1 layer, En_3 layer, En_5 layer, and En_6 layer of the U2Net network algorithm are used to extract the scale line features and pointer features in the pointer-type instrument image.

[0146] Optionally, the text recognition module 202 is specifically used for:

[0147] Extract features from the pointer-type instrument image through convolution and sampling operations to obtain a feature map;

[0148] Perform a deconvolution operation on the feature map to obtain a probability map;

[0149] Process the feature map to obtain an adaptive threshold map;

[0150] Perform a differentiable binarization operation on the probability map and the adaptive threshold map to obtain a binarized map;

[0151] Based on the binarized map, perform text region segmentation on the pointer-type instrument image to obtain a text image region containing text content;

[0152] Perform text recognition on each text image region to obtain multiple scale values.

[0153] Figure 11 It is a schematic diagram of a terminal device provided by an embodiment of the present invention. AsFigure 11 As shown, the terminal device 300 of this embodiment includes a processor 310 and a memory 320. A computer program that can run on the processor 310 is stored in the memory 320, such as a pointer-type instrument reading recognition program. When the processor 310 executes the computer program, it implements the steps in the above-described embodiment of the pointer-type instrument reading recognition method, such as Figure 1 101 to 106 shown. Alternatively, when the processor 310 executes the computer program, it implements the functions of each module / unit in the above-described device embodiments, such as Figure 10 the functions of the splitting module 201 to the reading determination module 206 shown.

[0154] Exemplarily, the computer program can be split into one or more modules / units. The one or more modules / units are stored in the memory 320 and executed by the processor 310 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program in the terminal device 300. For example, the computer program can be split into a splitting module, a text recognition module, a matching module, a fitting module, a scale interval determination module, and a reading determination module.

[0155] The terminal device 300 can be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The terminal device may include, but is not limited to, a processor 310 and a memory 320. Those skilled in the art can understand that Figure 11 these are merely examples of the terminal device 300 and do not constitute a limitation on the terminal device 300. It may include more or fewer components than shown in the figure, or combine certain components, or have different components. For example, the terminal device may further include input / output devices, network access devices, a bus, etc.

[0156] The so-called processor 320 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0157] The memory 320 may be an internal storage unit of the terminal device 300, such as a hard disk or memory of the terminal device 300. The memory 320 may also be an external storage device of the terminal device 300, such as a plug-in hard disk equipped on the terminal device 300, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 320 may also include both the internal storage unit of the terminal device 300 and an external storage device. The memory 320 is used to store the computer program and other programs and data required by the terminal device. The memory 320 may also be used to temporarily store data that has been output or is to be output.

[0158] In the above embodiments, the descriptions of the various embodiments have their own emphases. For parts not detailed or recorded in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0159] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for identifying readings of a pointer instrument, characterized in that: include: Segmenting the scale and the pointer of the pointer instrument according to the pointer instrument image to obtain a plurality of scale lines and a pointer; Performing text recognition on the pointer instrument image to obtain multiple scale values; According to the characteristic information of the scale values ​​and the characteristic information of the scale lines, the multiple scale values ​​are matched with the multiple scale lines to determine the scale values ​​corresponding to the respective scale lines; wherein the characteristic information of the scale values ​​includes the position, angle and direction of the scale values, and the characteristic information of the scale lines includes the position, angle and direction of the scale lines; Fitting the multiple scale lines and the one pointer respectively to obtain a scale fitting circle and a pointer fitting straight line; According to the scale fitting circle, the pointer fitting straight line and the scale values ​​corresponding to each scale, determining the intersection point of the pointer fitting straight line and the scale fitting circle and the scale interval where the pointer is located; Determine the reading of the pointer instrument according to the scale interval where the pointer is located and the intersection point; The matching of the multiple scale values ​​with the multiple scale lines according to the characteristic information of the scale values ​​and the characteristic information of the scale lines to determine the scale values ​​corresponding to the respective scale lines includes: Determine the first position coordinate x of the i-th scale value i and i , and the first angle θ i and the first direction v i ; Determine the second position coordinate x of the jth tick mark j and j , and the second angle θ j and the first direction v j ; According to the formula Calculate the characteristic information c of the i-th scale value i and the characteristic information c of the jth scale line j The Euclidean distance between According to the formula Calculate the similarity s between the i-th scale value and the j-th scale line ij , σ is c i With c j The standard deviation of according to Calculate the weight between the i-th scale value and the j-th scale line, and obtain the weight between each scale value and all scale lines; The scale line with the largest weight between the i-th scale value is matched, thereby obtaining a scale line matching each scale value.

2. The method for identifying pointer meter readings according to claim 1, characterized in that: The step of fitting the multiple scale lines and the one pointer respectively to obtain a scale fitting circle and a pointer fitting straight line comprises: Determine the center point of each tick mark; The center point of each scale line is fitted using the least square method to obtain the scale fitting circle; Performing Hough transformation on the pointer to generate the pointer fitting straight line, wherein the pointer fitting straight line intersects with the scale fitting circle at two points.

3. The method for identifying pointer meter readings according to claim 1, characterized in that: The step of determining the intersection of the pointer fitting line and the scale fitting circle and the scale interval where the pointer is located according to the scale fitting circle, the pointer fitting line and the scale values ​​corresponding to each scale, comprises: Determine the target point on the pointer that is farthest from the center of the scale fitting circle; Determine two intersection points of the scale fitting circle and the pointer fitting straight line, and determine a target intersection point closest to the target point among the two intersection points; The distance between the center point of each scale line and the target intersection is calculated, and the scale interval formed by the scale lines corresponding to the two smallest distances is used as the scale interval where the pointer is located.

4. The method for identifying pointer meter readings according to claim 3, characterized in that: Determining the reading of the pointer instrument according to the scale interval where the pointer is located and the intersection point includes: Determine the starting position and the ending position of the scale interval; Calculating a first arc α between the starting position and the ending position, and a second arc β between the starting position and the target intersection point; according to Determine the reading of pointer instruments; Where value is the reading of the pointer instrument, maxvalue is the scale value corresponding to the end position of the scale interval, minvalue is the scale value corresponding to the start position of the scale interval, and x1 and y1 are the coordinates of the starting position of the scale interval, x2 and y2 are the coordinates of the ending position of the scale interval, x3 and y3 are the coordinates of the target intersection, x c and c are the coordinates of the center of the scale fitted circle.

5. The method for identifying pointer meter readings according to claim 1, characterized in that: Before the step of segmenting the scale and the pointer of the pointer instrument according to the pointer instrument image, the pointer instrument reading recognition method further includes: Acquire an original image, wherein the original image includes a pointer instrument; Determine the position of the pointer instrument from the original image based on the improved YOLOv9 model, and extract the pointer instrument image; Among them, the P3 feature map and the conv-reg module, conv-cls module and concat module connected to the P3 feature map are deleted in the improved YOLOv9 model; The process of determining the ahchor parameters in the conv-reg module in the P4 feature map and the P5 feature map of the improved YOLOv9 model is as follows: obtaining the annotated meter sizes in multiple pointer meter images, calculating the multiple meter sizes by the mean algorithm, and obtaining the ahchor box that can represent the pointer meter.

6. The method for identifying pointer meter readings according to claim 1, characterized in that: The step of segmenting the scale and the pointer of the pointer instrument according to the pointer instrument image to obtain a plurality of scale lines and a pointer includes: The pointer instrument image is segmented by the U2Net network algorithm to obtain multiple scale lines and a pointer; Among them, only the En_1 layer, En_3 layer, En_5 layer and En_6 layer of the U2Net network algorithm are used to extract the scale line features and pointer features in the pointer instrument image.

7. The method for identifying pointer meter readings according to claim 1, characterized in that: The text recognition is performed on the pointer instrument image to obtain multiple scale values, including: Extracting features of the pointer meter image through convolution and sampling operations to obtain a feature map; Performing a deconvolution operation on the feature map to obtain a probability map; Processing the feature map to determine a pixel threshold corresponding to each pixel in the feature map; According to the pixel threshold corresponding to each pixel point in the feature map, a differentiable binarization operation is performed on the probability map to obtain a binarized map; Based on the binarized image, the pointer instrument image is segmented into text regions to obtain a text image region containing text content; Text recognition is performed on each text image area to obtain multiple scale values.

8. A pointer-type instrument reading recognition device, characterized in that: include: A segmentation module is used to segment the scale and pointer of the pointer instrument according to the pointer instrument image to obtain multiple scale lines and a pointer; A text recognition module, used for performing text recognition on the pointer instrument image to obtain multiple scale values; A matching module, used for matching the multiple scale values ​​with the multiple scale lines according to the characteristic information of the scale values ​​and the characteristic information of the scale lines, and determining the scale values ​​corresponding to the respective scale lines; wherein the characteristic information of the scale values ​​includes the position, angle and direction of the scale values, and the characteristic information of the scale lines includes the position, angle and direction of the scale lines; A fitting module, used for fitting the multiple scale lines and the one pointer respectively to obtain a scale fitting circle and a pointer fitting straight line; A scale interval determination module, used to determine the intersection of the pointer fitting line and the scale fitting circle and the scale interval where the pointer is located according to the scale fitting circle, the pointer fitting line and the scale values ​​corresponding to each scale; A reading determination module, used to determine the reading of the pointer instrument according to the scale interval where the pointer is located and the intersection point; The matching module is specifically used for: Determine the first position coordinate x of the i-th scale value i and i , and the first angle θ i and the first direction v i ; Determine the second position coordinate x of the jth tick mark j and j , and the second angle θ j and the first direction v j ; According to the formula Calculate the characteristic information c of the i-th scale value i and the characteristic information c of the jth scale line j The Euclidean distance between According to the formula Calculate the similarity s between the i-th scale value and the j-th scale line ij , σ is c i With c j The standard deviation of according to Calculate the weight between the i-th scale value and the j-th scale line, and obtain the weight between each scale value and all scale lines; The scale line with the largest weight between the i-th scale value is matched, thereby obtaining a scale line matching each scale value.

9. A terminal device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the steps of the pointer meter reading recognition method as described in any one of claims 1 to 7 are implemented.

Images