Multi-feature fusion pointer type meter automatic identification method and system

Abstract

The invention discloses a multi-feature fusion pointer type meter automatic identification method and system. The method comprises the steps of obtaining a to-be-identified instrument image; constructing a dial region recognition model and a feature point extraction model; using the dial region recognition model to extract a dial region in a to-be-recognized instrument image, and generating a dial image and an instrument type; a feature point extraction model detects scale line feature points and pointer feature points in the dial image; fitting a dial plate circular area through the scale line feature points to obtain dial plate circle center coordinates; obtaining two scale line feature points nearest to the pointer feature point through the spatial position relation; establishing a mapping relation between scale line feature points and dial template feature points when a shielding inclination condition occurs, and mapping pointer features to a dial circular area; and the instrument reading is determined through the coordinate information of the pointer characteristics on the circle, the characteristics of the two nearest scale marks of the pointer and the instrument type. According to the invention, the reading of the pointer meter can be accurately and rapidly identified.

Description

Technical Field

[0001] The present invention belongs to the technical field of meter reading, and in particular relates to a multi-feature fused pointer type meter automatic identification method and system. Background Art

[0002] Industrial measuring instruments are now mainly divided into digital instruments and pointer instruments. Nowadays, with the advancement of industrial digitization and intelligence, digital instruments are becoming more and more common in industry. However, relying on strong anti-electromagnetic interference ability and mechanical stability, pointer instruments are still the main status monitoring equipment in the power industry. There are many types of pointer instruments in substations, including pressure gauges, ammeters, oil temperature gauges, etc. Substations need to monitor and collect instrument parameters in real time to prevent accidents caused by exceeding the parameters to ensure the stability and safety of the production process. For example, record the oil temperature meter range data to ensure that the oil temperature is within the normal range, and record the leakage current meter data. If it is in the abnormal range, the system will prompt a warning.

[0003] The existing pointer instrument reading recognition detection methods mainly include traditional methods and deep learning methods. Among the traditional methods, the Chinese invention patent application with publication number CN111507186A discloses a substation pointer instrument reading recognition method, which includes the following steps: dial scale area detection, digital position detection, digital recognition, pointer detection and reading recognition, and the instrument reading recognition is performed based on the digital recognition and pointer detection results, wherein the tilt and return recognition process of the recognized numbers can verify the correctness of the scale area detection, and the distance calculation method based on the scale numbers is used in the reading recognition process; the dial scale area detection adopts a circle detection method based on Hough transform, and the digital position at the scale line is detected in a scanning manner, and the rotated numbers are recognized using a template matching method.

[0004] The main problem of the traditional pointer meter reading recognition method of instrument image preprocessing, scale line extraction, and pointer outline extraction is that it is easy to cause errors in scale line and pointer outline extraction. Based on this, some methods replace the method of dial scale line and pointer outline extraction with a method based on deep learning. The Chinese invention patent application with publication number CN115661446A discloses a system and method for automatically reading pointer-type instrument readings based on deep learning, and the method includes the following steps: training a YOLOv5s deep learning model through historical substation scene pictures, and the trained YOLOv5s deep learning model is an instrument picture target detection network; inputting the substation scene picture into the instrument picture target detection network to obtain a target instrument picture; inputting the historical substation scene picture into the instrument picture target detection network to obtain a historical instrument picture, and inputting the historical instrument picture into an improved PSPNet semantic segmentation model for model training to obtain a trained PSPNet semantic segmentation model for autonomous detection of a target area; inputting the target instrument picture into the trained PSPNet semantic segmentation model to obtain an area of ​​interest, and the area of ​​interest includes a scale area and a pointer area; and obtaining the instrument readings according to the relative position relationship between the scale area and the pointer area.

[0005] The most important thing about this method is to obtain the coordinate information of the scale line, pointer, and end scale line. When the instrument is blocked and the start scale or end scale is not detected, the reading recognition accuracy will be greatly reduced, and there will be a large reading error. The use of YOLOv5s and the improved PSPNet model to process images will increase the computational complexity of the system, affect the recognition efficiency, and affect the real-time performance and response speed of the system in resource-constrained environments (such as embedded devices), especially in scenarios that require fast and continuous reading. Summary of the invention

[0006] The present invention provides a multi-feature fusion pointer type meter automatic identification method and system, aiming to solve the problems of insufficient identification efficiency and low accuracy in complex environments.

[0007] In order to solve the above technical problems, the identification method proposed in the present invention comprises the following steps:

[0008] Obtaining the image of the instrument to be identified;

[0009] Construct a dial area recognition model and a feature point extraction model. The dial area recognition model is a YOLOX model. The instrument image to be recognized is input into the YOLOX model, and the YOLOX model outputs the bounding box of the instrument dial area and the corresponding instrument type.

[0010] Use the dial area recognition model to extract the dial area in the instrument image to be recognized, and generate the dial image and instrument type;

[0011] The feature point extraction model detects the scale line feature points and the pointer feature points in the dial image;

[0012] Fit the circular area of ​​the dial through the scale line feature points to obtain the coordinates of the center of the dial; obtain the two scale line feature points closest to the pointer feature point through the spatial position relationship;

[0013] When tilt or occlusion occurs, a mapping relationship between the feature points of the scale lines and the feature points of the dial template is established, and the pointer features are mapped to the dial template;

[0014] The pointer feature points are mapped to the fitted circular area of ​​the dial, and the meter reading is determined by the coordinate information of the pointer feature points on the circle, the feature points of the two nearest scale lines of the pointer, and the instrument type.

[0015] Preferably, after obtaining the coordinates of the center of the dial and the two nearest scale line feature points, the method classifies the dial image quality according to the obtained results into two categories: the dial area has defects and the dial area is normal;

[0016] If there are defects in the dial area, a mapping relationship between the scale feature points and the dial template feature points is established, and the pointer features are mapped to the dial template;

[0017] If the dial area is normal, proceed to the next step without any processing.

[0018] Preferably, the method for the feature point extraction model to detect the scale line feature points and the pointer feature points is:

[0019] Use a convolutional network to extract features from the input image and generate a set of basic feature maps;

[0020] Generate a set of universal basis masks in the input image, which are not tied to any specific object and represent all possible object shapes and sizes;

[0021] Perform object detection for each tick mark area and pointer area of ​​interest, and predict and generate the corresponding bounding box, category label, and mask coefficient;

[0022] Generate instance masks of tick marks and pointers by linearly combining the base mask and the predicted mask coefficients;

[0023] A contour detection algorithm or an edge detection algorithm is used to extract scale line feature points and pointer feature points from the instance mask; the scale line feature points are extracted by extracting the two endpoints of the scale line according to the shape of the scale line to indicate the position and direction of the scale, and the pointer feature points are extracted by extracting the tip position of the pointer and the relative position of the pointer in the image.

[0024] Preferably, the calculation method of the coordinates of the center of the dial is:

[0025] The equation of a circle is expressed as:

[0026] (xx c ) 2 +(yy c ) 2 =r 2

[0027] In the formula, (x c ,y c ) is the coordinate of the center of the circle, r is the radius of the circle, and (x, y) is the known coordinate of the characteristic point of the scale line;

[0028] By using the least squares method, find the optimal (x c ,y c ,r):

[0029]

[0030] Where n is the number of known scale feature points;

[0031] Through algebraic transformation, the equation of the circle is linearized, and the circle equation is expanded first:

[0032]

[0033] Let A = -2x c , B=-2y c , The equation becomes:

[0034] Ax+By+C=-(x 2 +y 2 )

[0035] Solve A, B, and C using the least squares method;

[0036] Calculate the center coordinates and radius of the circle based on A, B, and C, where the center coordinates are expressed as The radius of a circle is expressed as

[0037] Preferably, the method of obtaining the two nearest neighboring scale line feature points of the pointer feature point through the spatial position relationship is as follows:

[0038] Get the coordinates of the pointer feature points and the coordinate set of the scale line feature points;

[0039] Traverse the set of scale line feature points and calculate the distance between each scale line feature point and the pointer feature point one by one;

[0040] The two scale line feature points with the smallest distance are extracted as the nearest neighbor scale line feature points.

[0041] Preferably, the distance between the scale feature point and the pointer feature point is calculated as follows:

[0042]

[0043] Where d(p,s) is the distance between the pointer feature point p and the scale feature point s, and x p ,y p is the coordinate of the pointer feature point p, x s ,y s is the coordinate of the scale feature point s.

[0044] Preferably, the method of mapping the pointer feature to the dial template is:

[0045] Construct a template dial and extract the template scale feature points and template pointer feature points;

[0046] The coordinates of the scale feature points and pointer feature points in the dial image are standardized to the same scale as the template dial;

[0047] Perform feature point matching to match the scale line feature points and pointer feature points in the dial image with the template scale line feature points and template pointer feature points;

[0048] If there are defects in the dial image, the template dial is used to complete the dial image, and the completed feature points are combined with the pointer feature points.

[0049] Preferably, the feature point matching method is:

[0050] The initial correspondence between the feature points in the actual dial image and the feature points in the template dial is obtained by the nearest neighbor search algorithm, and the displacement vector of each pair of matching points is calculated;

[0051] Use RANSAC or the least squares method based on affine transformation to estimate the transformation parameters from the actual dial to the template dial. The transformation parameters are expressed as follows:

[0052] Where a represents the cosine component of the scaling and rotation angle in the x direction, b represents the sine component of the scaling and rotation angle in the x direction, tx, ty represent the translation of the feature point in the x and y directions after the transformation, and (x, y) represents the coordinates of the feature point in the dial image;

[0053] The feature points of the dial image are mapped to the template dial coordinate system through transformation parameters. The mapping is expressed as follows:

[0054] (x′,y′)=T(x,y)

[0055] Where (x′, y′) represents the coordinates of the feature points of the dial image after transformation in the template dial coordinate system.

[0056] Preferably, the method for determining the meter reading is:

[0057] According to the geometric relationship between the point on the circle and the center of the circle, the pointer feature point is stretched toward the center of the circle to the circle to obtain the coordinates of the pointer feature point on the circle;

[0058] Calculate the pointer angle based on the center of the circle and the coordinates on the circle; map the pointer angle to the scale value based on the scale range of the instrument dial. The mapping method is:

[0059]

[0060] In the formula, S p is the scale value after mapping, θ p is the pointer angle, θ is the scale range of the instrument dial, scale max is the maximum scale value of the instrument;

[0061] Use the coordinates of the two nearest scale line feature points of the pointer to calibrate the scale value obtained by mapping;

[0062] The actual scale value of the nearest scale point after calibration is calculated by linear interpolation based on the ratio of the pointer between the two scale points. The calculation method is:

[0063]

[0064] In the formula, S final is the final reading, S k1 is the scale value of the first nearest neighbor scale line, S k2 The second most

[0065] The scale value of the neighboring scale line, θ k1 is the angle of the first nearest neighbor tick mark, θ k2 is the angle of the second nearest tick mark.

[0066] In another aspect of the present invention, a multi-feature fusion pointer type meter automatic identification system is further provided. The system is used to implement the multi-feature fusion pointer type meter automatic identification method as described in the first aspect of the present invention, comprising:

[0067] Image acquisition module, used to obtain image data of the instrument to be identified and pre-process the image data, including grayscale conversion, noise removal and size adjustment;

[0068] The dial area recognition module determines the instrument type information based on the YOLOX model, identifies the dial area in the instrument image, and extracts the dial image;

[0069] The feature point extraction module is used to detect the scale line feature points and pointer feature points in the dial image and output the pixel coordinate positions of the feature points; the scale line feature points are used for dial scale distribution analysis, and the pointer feature points are used to locate the current reading direction;

[0070] The circular fitting module fits the circular area of ​​the dial based on the coordinates of the feature points of the scale lines through the least squares method, calculates the coordinates of the center of the dial and the radius, and provides basic geometric data for subsequent feature mapping;

[0071] The nearest neighbor feature point calculation module is used to calculate the Euclidean distance between the pointer feature point and all the scale line feature points, determine the two scale line feature points closest to the pointer feature point, and output the coordinates of the two scale line feature points;

[0072] A feature mapping module is used to map the coordinates of the pointer feature points to the fitted circular area of ​​the dial, and to establish a mapping relationship between the scale feature points and the dial template feature points;

[0073] The data processing and reading calculation module is used to calculate the reading corresponding to the pointer based on the mapped coordinates of the pointer feature point on the circle, the coordinates of the two nearest scale line feature points of the pointer, the coordinates of the center of the dial and the instrument type information, and complete the reading analysis through geometric angle relationship and linear interpolation method.

[0074] Compared with the prior art, the present invention has the following technical effects:

[0075] 1. The automatic recognition method proposed in the present invention is fully automated from image acquisition to final reading generation, without manual intervention, which greatly improves efficiency. Through the classification and recognition of instrument types and template support, it can be easily extended to various instrument types in industry, laboratories or other scenarios, and has a wider application prospect.

[0076] 2. The automatic recognition method proposed in the present invention fits the circular area of ​​the dial through the least squares method, and maps the pointer feature points to the fitting circle, which effectively eliminates the problem of inaccurate readings caused by image distortion, tilt or extraction errors. Through the relationship between the pointer feature points and the nearest neighbor scale line feature points, combined with geometric angle calculations, the scale corresponding to the pointer is further accurately located, avoiding errors that may be caused by rough readings.

[0077] 3. The automatic recognition method proposed in the present invention establishes a mapping relationship between the scale line feature points and the template feature points. Even if the instrument dial is tilted or partially blocked, the system can still complete the scale calibration and pointer position recovery based on known features; the system combines the instrument type information and can adapt to the reading requirements of different types of instruments such as uniformly distributed scales and non-uniformly distributed scales, thereby expanding the scope of application scenarios.

[0078] 4. The automatic recognition method proposed in the present invention improves the accuracy and robustness of instrument recognition by combining deep learning and geometric fitting. It identifies the dial area and determines the instrument type based on a deep learning model (such as YOLOX). It can adapt to a variety of dial styles more robustly and reduce dependence on specific instrument types. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 It is a schematic diagram of the flow chart of the identification method of the present invention. DETAILED DESCRIPTION

[0080] In order to make the objectives, technical solutions and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in combination with specific embodiments of the present application and with reference to the accompanying drawings.

[0081] Embodiment 1

[0082] This embodiment is a multi-feature fusion pointer type meter automatic identification method, such as Figure 1 As shown, the steps include steps one to seven:

[0083] Step 1: Obtain the image of the instrument to be identified. The image of the instrument to be identified can be obtained by a camera set above or in front of the instrument. The camera can be an ordinary industrial camera, a network camera, or a dedicated high-resolution camera.

[0084] Step 2: Build a dial area recognition model and feature point extraction model.

[0085] It will be understood by those skilled in the art that building a dial area recognition model and a feature point extraction model includes building a model framework and training and tuning the model. During the training process, a large number of image data sets containing different types of instruments are used to train the YOLOX model. The data set needs to include labeled instrument dial areas (borders) and instrument types (such as voltmeters, ammeters, oil level gauges, etc.) labels. When in use, the substation scene image is input into the trained YOLOX model, and the YOLOX model outputs the bounding box of the instrument dial area and the corresponding instrument type. The detection results can be further removed by non-maximum suppression (NMS) to remove redundant frames.

[0086] For the feature point extraction model, it also includes the process of obtaining data sets, labeling, training, and tuning. The instance segmentation algorithm is used to train the instrument dial area. The goal is to label the feature areas of the scale lines and pointers. For the scale line feature points, find the boundary points of all the scale lines in the mask and calculate the midpoints of these points as the scale line feature points. For the pointer feature points, determine the pointer feature points by detecting the thinnest part of the pointer mask (usually at the tip of the pointer).

[0087] In this embodiment, the dial area recognition model constructed is a YOLOX model. When used, the instrument image to be recognized is input into the YOLOX model, and the YOLOX model outputs the boundary box of the instrument dial area and the corresponding instrument type.

[0088] Step three: Use the dial area recognition model to extract the dial area in the instrument image to be recognized, and generate the dial image and instrument type.

[0089] Step 4: The feature point extraction model detects the scale line feature points and pointer feature points in the dial image.

[0090] The method for the feature point extraction model to detect the scale feature points and the pointer feature points is:

[0091] Use a convolutional network to extract features from the input image and generate a set of basic feature maps;

[0092] Generate a set of universal basis masks in the input image, which are not tied to any specific object and represent all possible object shapes and sizes;

[0093] Perform object detection for each tick mark area and pointer area of ​​interest, and predict and generate the corresponding bounding box, category label, and mask coefficient;

[0094] Generate instance masks of tick marks and pointers by linearly combining the base mask and the predicted mask coefficients;

[0095] A contour detection algorithm or an edge detection algorithm is used to extract scale line feature points and pointer feature points from the instance mask; the scale line feature points are extracted by extracting the two endpoints of the scale line according to the shape of the scale line to indicate the position and direction of the scale, and the pointer feature points are extracted by extracting the tip position of the pointer and the relative position of the pointer in the image.

[0096] Those skilled in the art can understand that, for the pointer feature points and scale line feature points initially obtained, the accuracy of the feature points can be further optimized through the center points of the circumscribed rectangular frame.

[0097] Step 5: Fit the circular area of ​​the dial through the scale line feature points to obtain the coordinates of the center of the dial; obtain the two nearest scale line feature points of the pointer feature point through the spatial position relationship.

[0098] In this step, firstly, a circle is fitted using the least square method based on the coordinates of the feature points of the scale lines, and the parameters of the circle output include the coordinates of the center of the circle and the radius.

[0099] The calculation method of the coordinates of the center of the dial is:

[0100] The equation of a circle is expressed as:

[0101] (xxc ) 2 +(yy c ) 2 =r 2

[0102] In the formula, (x c ,y c ) is the coordinate of the center of the circle, r is the radius of the circle, and (x, y) is the known coordinate of the characteristic point of the scale line;

[0103] By using the least squares method, find the optimal (x c ,y c ,r):

[0104]

[0105] Where n is the number of known scale feature points;

[0106] Through algebraic transformation, the equation of the circle is linearized and becomes:

[0107] Ax+By+C=-(x 2 +y 2 )

[0108] Solve A, B, and C using the least squares method;

[0109] Calculate the center coordinates and radius of the circle based on A, B, and C, where the center coordinates are expressed as The radius of a circle is expressed as

[0110] The process of solving A, B, and C using the least squares method is as follows:

[0111] Ax+By+C=-(x 2 +y 2 ) is expressed in matrix form:

[0112] M·θ=b

[0113] Where M is the matrix of the coordinates of the scale feature points, and n is the number of scale feature points:

[0114]

[0115] θ is the parameter vector to be solved:

[0116]

[0117] b is the known right-hand side vector:

[0118] The least squares method is solved by the following formula:

[0119] θ=(M T M) -1 M T b

[0120] Substitute the solved θ vector into the transformed circle equation as the fitted circular area of ​​the dial. In this solution process, you can use the numpy library in Python or other dedicated fitting libraries, such as scipy.optimize.least_squares.

[0121] The specific method of obtaining the two nearest scale line feature points of the pointer feature point through the spatial position relationship is as follows:

[0122] Get the coordinates of the pointer feature points and the coordinate set of the scale line feature points;

[0123] Traverse the scale line feature point set, calculate the distance between each scale line feature point and the pointer feature point one by one, store the results as a list, and record the corresponding scale line feature point index;

[0124] Sort the distance list from small to large, retain the index of the original scale line feature point, and after sorting, the indexes corresponding to the first two items in the list are the two nearest neighbor scale line feature points. Extract the two scale line feature points with the smallest distance as the nearest neighbor scale line feature points.

[0125] When extracting the two nearest neighboring scale feature points, if there are multiple scale feature points with the same distance from the pointer feature point (such as symmetrical points on a circle), they can be selected according to the angle or sequence priority rule. If the distance between a scale feature point and the pointer feature point is zero, then the point is directly used as one of the nearest neighbor points.

[0126] The distance calculation method between the scale feature point and the pointer feature point is:

[0127]

[0128] Where d(p,s) is the distance between the pointer feature point p and the scale feature point s, and x p ,y p For pointer

[0129] The coordinates of the point p, x s ,y s is the coordinate of the scale feature point s.

[0130] Step six, establish a mapping relationship between the scale feature points and the dial template feature points, and map the pointer features to the dial template.

[0131] The method for mapping the pointer features to the dial template is:

[0132] Construct a template dial and extract the template scale feature points and template pointer feature points; the template dial contains complete scale line positions and pointer pointing rules, and the feature points of the template dial contain the coordinates and annotation values ​​of all scale lines, as well as the default position of the pointer (such as the starting point); extract feature points from the template (such as the midpoint of the scale line, the center of the dial) and save them as a template feature point set. The template feature points in the template feature point set are represented by a unified coordinate system to facilitate comparison with the actual dial.

[0133] The coordinates of the scale feature points and pointer feature points in the dial image are standardized to the same scale as the template dial; in the dial image (actual image), the scale feature points and pointer feature points are extracted by instance segmentation, and the coordinates of the feature points of the actual dial are normalized to the same scale as the template dial for matching. The normalization method is based on the center and radius of the dial.

[0134] If there is tilt or occlusion, feature point matching is performed to match the scale feature points and pointer feature points in the dial image with the template scale feature points and template pointer feature points;

[0135] If there are defects in the dial image, use the template dial to complete the dial image and combine the completed feature points with the pointer feature points. Defects include the dial being partially covered, highlights or shadows in some areas of the dial, resulting in the dial image being partially unrecognized.

[0136] Furthermore, the feature point matching method in the above mapping method is:

[0137] The initial correspondence between the feature points in the actual dial image and the feature points in the template dial is obtained through the nearest neighbor search algorithm, and the displacement vector of each pair of matching points is calculated to estimate the transformation parameters of the entire dial; the nearest neighbor search algorithm used in this embodiment includes the KNN (K-Nearest Neighbor) algorithm or the Euclidean distance algorithm.

[0138] Use RANSAC or the least squares method based on affine transformation to estimate the transformation parameters from the actual dial to the template dial. The transformation parameters include rotation, scaling, and translation, which are expressed as follows:

[0139]

[0140] Where a represents the cosine component of the scaling and rotation angle in the x direction, b represents the sine component of the scaling and rotation angle in the x direction, tx, ty represent the translation of the feature point in the x and y directions after the transformation, and (x, y) represents the coordinates of the feature point in the dial image;

[0141] The feature points of the dial image are mapped to the template dial coordinate system through transformation parameters. The mapping is expressed as follows:

[0142] (x′,y′)=T(x,y)

[0143] Where (x′, y′) represents the coordinates of the feature points of the dial image after transformation in the template dial coordinate system.

[0144] Specifically, assuming a feature point (x, y) = (10, 20), the template center needs to be translated (tx, ty) = (5, -3), rotated 30°, that is, θ = 30°, corresponding to cos30° = 0.866, sin30° = 0.5, and no scaling, that is, sx = sy = 1, the construction transformation calculation is as follows:

[0145] a=0.866

[0146] b=0.5

[0147] Rotate and scale:

[0148] x rot =0.866·10-0.5·20=-2.34

[0149] y rot =0.5·10+0.866·20=22.32

[0150] Pan:

[0151] x′=-2.34+5=2.66

[0152] y′=22.32-3=19.32

[0153] Finally, the transformed coordinates of the feature point are (2.66, 19.32).

[0154] This step includes the restoration of the occluded area. First, the completion is performed. The completion methods include template-assisted completion and interpolation completion. In template-assisted completion, if some scale feature points of the actual dial are missing due to occlusion, the corresponding scale feature point positions can be directly obtained from the template dial. In interpolation completion, for the missing scale feature points, the matched adjacent feature points are used to complete the completion through linear interpolation or polynomial fitting. Finally, a comprehensive update is performed to combine the completed feature points with the pointer feature points, and the subsequent circular fitting and reading derivation are continued.

[0155] In some embodiments of the present invention, the matching error after mapping is also evaluated, and the mapping accuracy is verified by calculating the matching error (such as mean square error) between the actual dial feature points after mapping and the template feature points. If the error is large, the initial matching point is readjusted.

[0156] Step 6 is an optional step in this embodiment. The logic of whether to execute step 6 is: after obtaining the coordinates of the center of the dial and the two nearest scale line feature points in step 5, the dial image quality is classified according to the obtained results, and divided into two categories: the dial area has defects and the dial area is normal. Whether to execute step 6 is selected according to the classification results. Defects include occlusion or reflection. Occlusion is usually manifested as the features of certain areas in the image are covered or blurred. It may be caused by foreign objects or equipment problems. It can be judged by edge detection and contour analysis methods, such as using Canny edge detection or Sobel operator to extract the edge of the image. If the edge distribution is sparse or the edge loss in the local area is obvious, it is determined that there is occlusion. Reflection is usually manifested as the brightness of certain areas is significantly higher than the normal range, and is usually unevenly distributed. The image can be converted into HSV or Lab color space first, the brightness channel (V channel or L channel) can be extracted, and the brightness histogram or brightness distribution of the pixels in the image can be calculated. If there are too many highlighted pixels (the proportion of pixels with brightness values ​​close to the maximum value is too high), it may indicate the presence of reflections, that is, if the proportion of highlighted pixels exceeds the judgment threshold preset by the threshold and is concentrated in a certain area, it is determined that reflections exist.

[0157] If there are defects in the dial area, a mapping relationship between the scale feature points and the dial template feature points is established, and the pointer features are mapped to the dial template, that is, in response to the classification result that there are defects in the dial area, step six is ​​executed, and then step seven is executed. In response to the classification result that the dial area is normal, no processing is performed and the next step is continued, that is, step seven is directly executed after step five.

[0158] Step seven, map the pointer feature points to the fitted circular area of ​​the dial, and determine the instrument reading through the coordinate information of the pointer feature points on the circle, the feature points of the two nearest scale lines of the pointer, and the instrument type.

[0159] In this step, the known parameters include the fitted dial circular area parameters (center coordinates, radius), pointer feature point coordinates, pointer nearest two scale line feature point coordinates, instrument type (voltmeter, ammeter, oil level gauge, circular dial scale range, total number of scale lines, etc.).

[0160] The method for determining the meter reading is:

[0161] According to the geometric relationship between the point on the circle and the center of the circle, the pointer feature point is stretched toward the center of the circle to obtain the coordinates of the pointer feature point on the circle (x′ p ,y p ′):

[0162]

[0163] Calculate the pointer angle based on the center of the circle and the coordinates on the circle. Define the center of the circle as the far point, the horizontal positive direction as 0 degrees, and the pointer angle θp It can be calculated by the inverse tangent function:

[0164] θ p = atan2(y′ p -y c ,x′ p -x c )

[0165] If the result of the above formula is negative, add 360 degrees to ensure that the angle is positive;

[0166] According to the scale range of the instrument dial, the pointer angle is mapped to the scale value. The mapping method is:

[0167]

[0168] In the formula, S p is the scale value after mapping, θ p is the pointer angle, θ is the scale range of the instrument dial, scale max is the maximum scale value of the instrument;

[0169] Use the coordinates of the two nearest scale line feature points of the pointer to calibrate the scale value obtained by mapping;

[0170] The actual scale value of the nearest scale point after calibration is calculated by linear interpolation based on the ratio of the pointer between the two scale points. The calculation method is:

[0171]

[0172] θ k1 =atan2(y k1 -y c ,x k1 -x c )

[0173] θ k2 =atan2(y k2 -y x ,x k2 -x c )

[0174] In the formula, S final is the final reading, S k1 is the scale value of the first nearest neighbor scale line, S k2 is the scale value of the second nearest neighbor scale line, θ l1 is the angle of the first nearest neighbor tick mark, θ k2 is the angle of the second nearest tick mark, (x k1 ,y l1 ) is the coordinate of the first nearest neighbor tick mark, (x k2 ,yk2 ) are the coordinates of the second nearest tick mark.

[0175] In the calculation of the above final reading, if the pointer completely coincides with a certain scale line, the value corresponding to the scale line is directly used as the reading; if the distribution between the scale lines is uneven, the interpolation calculation method needs to be adjusted according to the template data provided by the instrument type.

[0176] Final meter reading S final , combined with the meter type information output, for example, meter type: ammeter, unit A, 0-100 scale, reading: 75.3, then the output: current 75.3A.

[0177] Embodiment 2

[0178] This embodiment is a multi-feature fusion pointer type meter automatic identification system, which is used to implement the multi-feature fusion pointer type meter automatic identification method as described in Example 1, including:

[0179] Image acquisition module, used to obtain image data of the instrument to be identified and pre-process the image data, including grayscale conversion, noise removal and size adjustment;

[0180] The dial area recognition module determines the instrument type information based on the YOLOX model, identifies the dial area in the instrument image, and extracts the dial image;

[0181] The feature point extraction module is used to detect the scale line feature points and pointer feature points in the dial image and output the pixel coordinate positions of the feature points; the scale line feature points are used for dial scale distribution analysis, and the pointer feature points are used to locate the current reading direction;

[0182] The circular fitting module fits the circular area of ​​the dial based on the coordinates of the feature points of the scale lines through the least squares method, calculates the coordinates of the center of the dial and the radius, and provides basic geometric data for subsequent feature mapping;

[0183] The nearest neighbor feature point calculation module is used to calculate the Euclidean distance between the pointer feature point and all the scale line feature points, determine the two scale line feature points closest to the pointer feature point, and output the coordinates of the two scale line feature points;

[0184] A feature mapping module is used to map the coordinates of the pointer feature points to the fitted circular area of ​​the dial, and to establish a mapping relationship between the scale feature points and the dial template feature points;

[0185] The data processing and reading calculation module is used to calculate the reading corresponding to the pointer based on the mapped coordinates of the pointer feature point on the circle, the coordinates of the two nearest scale line feature points of the pointer, the coordinates of the center of the dial and the instrument type information, and complete the reading analysis through geometric angle relationship and linear interpolation method.

[0186] The above is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several modifications and improvements without departing from the inventive concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A multi-feature fusion pointer type meter automatic identification method, characterized in that: The following steps are involved: Obtaining the image of the instrument to be identified; Construct a dial area recognition model and a feature point extraction model. The dial area recognition model is a YOLOX model. The instrument image to be recognized is input into the YOLOX model, and the YOLOX model outputs the bounding box of the instrument dial area and the corresponding instrument type. Use the dial area recognition model to extract the dial area in the instrument image to be recognized, and generate the dial image and instrument type; The feature point extraction model detects the scale line feature points and the pointer feature points in the dial image; Fit the circular area of ​​the dial through the scale line feature points to obtain the coordinates of the center of the dial; obtain the two scale line feature points closest to the pointer feature point through the spatial position relationship; The pointer feature points are mapped to the fitted circular area of ​​the dial, and the meter reading is determined by the coordinate information of the pointer feature points on the circle, the feature points of the two nearest scale lines of the pointer, and the instrument type.

2. According to the multi-feature fusion pointer type meter automatic identification method of claim 1, it is characterized in that: After obtaining the coordinates of the center of the dial and the two nearest scale line feature points, the method classifies the dial image quality according to the obtained results into two categories: the dial area has defects and the dial area is normal; If there are defects in the dial area, a mapping relationship between the scale feature points and the dial template feature points is established, and the pointer features are mapped to the dial template; If the dial area is normal, proceed to the next step without any processing.

3. The method for automatic identification of pointer type meters by multi-feature fusion according to claim 1 is characterized in that: The method for the feature point extraction model to detect the scale feature points and the pointer feature points is: Use a convolutional network to extract features from the input image and generate a set of basic feature maps; Generate a set of universal basis masks in the input image, which are not tied to any specific object and represent all possible object shapes and sizes; Perform object detection for each tick mark area and pointer area of ​​interest, and predict and generate the corresponding bounding box, category label, and mask coefficient; Generate instance masks of tick marks and pointers by linearly combining the base mask and the predicted mask coefficients; A contour detection algorithm or an edge detection algorithm is used to extract scale line feature points and pointer feature points from the instance mask; the scale line feature points are extracted by extracting the two endpoints of the scale line according to the shape of the scale line to indicate the position and direction of the scale, and the pointer feature points are extracted by extracting the tip position of the pointer and the relative position of the pointer in the image.

4. The method for automatic identification of pointer type meters by multi-feature fusion according to claim 1 is characterized in that: The calculation method of the coordinates of the center of the dial is: The equation of a circle is expressed as: (x-x c ) 2 +(y-y c ) 2 =r 2 In the formula, (x c ,y c ) is the coordinate of the center of the circle, r is the radius of the circle, and (x, y) is the known coordinate of the characteristic point of the scale line; By using the least squares method, find the optimal (x c ,y c ,r): Where n is the number of known scale feature points; Through algebraic transformation, the equation of the circle is linearized and becomes: Ax+By+C=-(x 2 +y 2 ) Solve A, B, and C using the least squares method; Calculate the center coordinates and radius of the circle based on A, B, and C, where the center coordinates are expressed as The radius of a circle is expressed as 5. The method for automatic identification of pointer type meters by multi-feature fusion according to claim 1 is characterized in that: The specific method of obtaining the two nearest scale line feature points of the pointer feature point through the spatial position relationship is as follows: Get the coordinates of the pointer feature points and the coordinate set of the scale line feature points; Traverse the set of scale line feature points and calculate the distance between each scale line feature point and the pointer feature point one by one; The two scale line feature points with the smallest distance are extracted as the nearest neighbor scale line feature points.

6. The method for automatic identification of pointer type meters by multi-feature fusion according to claim 5 is characterized in that: The distance calculation method between the scale feature point and the pointer feature point is: Where d(p,s) is the distance between the pointer feature point p and the scale feature point s, and x p ,y p is the coordinate of the pointer feature point p, x s ,y s is the coordinate of the scale feature point s.

7. The method for automatic identification of pointer type meters by multi-feature fusion according to claim 1 is characterized in that: The method for mapping the pointer features to the dial template is: Construct a template dial and extract the template scale feature points and template pointer feature points; The coordinates of the scale feature points and pointer feature points in the dial image are standardized to the same scale as the template dial; Perform feature point matching to match the scale line feature points and pointer feature points in the dial image with the template scale line feature points and template pointer feature points; If there are defects in the dial image, the template dial is used to complete the dial image, and the completed feature points are combined with the pointer feature points.

8. The method for automatic identification of pointer type meters by multi-feature fusion according to claim 7 is characterized in that: The feature point matching method is: The initial correspondence between the feature points in the actual dial image and the feature points in the template dial is obtained by the nearest neighbor search algorithm, and the displacement vector of each pair of matching points is calculated; Use RANSAC or the least squares method based on affine transformation to estimate the transformation parameters from the actual dial to the template dial. The transformation parameters are expressed as follows: Where a represents the cosine component of the scaling and rotation angle in the x direction, b represents the sine component of the scaling and rotation angle in the x direction, tx, ty represent the translation of the feature point in the x and y directions after the transformation, and (x, y) represents the coordinates of the feature point in the dial image; The feature points of the dial image are mapped to the template dial coordinate system through transformation parameters. The mapping is expressed as follows: (x′,y′)=T(x,y) Where (x′, y′) represents the coordinates of the feature points of the dial image after transformation in the template dial coordinate system.

9. The method for automatic identification of pointer type meters by multi-feature fusion according to claim 1 is characterized in that: The method for determining the meter reading is: According to the geometric relationship between the point on the circle and the center of the circle, the pointer feature point is stretched toward the center of the circle to the circle to obtain the coordinates of the pointer feature point on the circle; Calculate the pointer angle based on the center of the circle and the coordinates on the circle; map the pointer angle to the scale value based on the scale range of the instrument dial. The mapping method is: In the formula, S p is the scale value after mapping, θ p is the pointer angle, θ is the scale range of the instrument dial, scale max is the maximum scale value of the instrument; Use the coordinates of the two nearest scale line feature points of the pointer to calibrate the scale value obtained by mapping; The actual scale value of the nearest scale point after calibration is calculated by linear interpolation based on the ratio of the pointer between the two scale points. The calculation method is: In the formula, S final is the final reading, S k1 is the scale value of the first nearest neighbor scale line, S k2 is the scale value of the second nearest neighbor scale line, θ k1 is the angle of the first nearest neighbor tick mark, θ k2 is the angle of the second nearest tick mark.

10. A multi-feature fusion pointer type meter automatic identification system, characterized in that: The system is used to implement the multi-feature fusion pointer type meter automatic identification method according to any one of claims 1 to 9, comprising: Image acquisition module, used to obtain image data of the instrument to be identified and pre-process the image data, including grayscale conversion, noise removal and size adjustment; The dial area recognition module determines the instrument type information based on the YOLOX model, identifies the dial area in the instrument image, and extracts the dial image; The feature point extraction module is used to detect the scale line feature points and pointer feature points in the dial image and output the pixel coordinate positions of the feature points; the scale line feature points are used for dial scale distribution analysis, and the pointer feature points are used to locate the current reading direction; The circular fitting module fits the circular area of ​​the dial based on the coordinates of the feature points of the scale lines through the least squares method, calculates the coordinates of the center of the dial and the radius, and provides basic geometric data for subsequent feature mapping; The nearest neighbor feature point calculation module is used to calculate the Euclidean distance between the pointer feature point and all the scale line feature points, determine the two scale line feature points closest to the pointer feature point, and output the coordinates of the two scale line feature points; A feature mapping module is used to map the coordinates of the pointer feature points to the fitted circular area of ​​the dial, and to establish a mapping relationship between the scale feature points and the dial template feature points; The data processing and reading calculation module is used to calculate the reading corresponding to the pointer based on the mapped coordinates of the pointer feature point on the circle, the coordinates of the two nearest scale line feature points of the pointer, the coordinates of the center of the dial and the instrument type information, and complete the reading analysis through geometric angle relationship and linear interpolation method.

Images