Pointer type instrument reading identification method based on template semantic key point matching

Abstract

The invention discloses a pointer type instrument reading recognition method based on template semantic key point matching. The method comprises the steps that a target detection data set, a semantic matching data set and a category template data set are constructed based on collected pointer type instrument images; constructing and training a target detection model and a semantic matching model; inputting a to-be-recognized pointer type instrument image into the trained target detection model, and obtaining an instance position and category and a pointer tail end position; inputting the instance image and the category template image into a trained semantic matching model, and obtaining semantic matching points of the category template key points in the to-be-recognized instance image; homography transformation is carried out on semantic matching points of the category template key points in the to-be-recognized instance image, and instrument reading is calculated according to the category template key points, specific scale value information of the category template key points and the tail end position of the pointer. According to the method, more accurate instrument reading can be obtained in a complex environment, and the robustness and the universality of reading identification can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of pointer-type instrument reading recognition, and in particular to a pointer-type instrument reading recognition method based on template semantic key point matching. Background Art

[0002] Pointer instruments are widely used in many fields such as industry, transportation and medicine, such as speedometers, thermometers, pressure gauges, etc. Especially in some complex industrial environments with radiation, toxicity, high temperature, etc., it is often necessary to check the readings of instruments to confirm the normal and safe production. The readings of these instruments are usually done manually, and these instruments are often numerous and widely distributed. The manual reading method is time-consuming, inefficient, and labor-intensive. Therefore, it is of great practical significance to use computer vision technology to automatically identify the readings of pointer instruments.

[0003] In recent years, with the development of computer vision technology, there have been some attempts to solve this problem. Depending on whether a template is required, these methods can be divided into two categories: template-free methods and template matching-based methods.

[0004] The overall process of the template-free method is as follows: use the target detection method to detect the instrument instance to be identified in the image; use the detection method to obtain the pointer position and scale position information for each instrument instance; obtain the scale reading through OCR text recognition and other technologies, and establish a corresponding relationship with the scale position; and calculate the reading based on the above information.

[0005] However, in the template-free method, one of the most critical steps is to establish the correspondence between the scale reading and the scale position. Scale reading is usually achieved using OCR text recognition technology, which has high requirements on image clarity, occlusion, and light distribution on the image. However, these requirements cannot be met in real production scenarios, which easily leads to reading recognition failure. In addition, since it does not utilize the additional prior knowledge provided by the template information, its scalability and flexibility are poor.

[0006] The overall process of the template matching-based method is as follows: for each type of instrument, select a clear, complete picture with good lighting conditions and shooting angle as a template, and mark the key scale positions and the pointer rotation center; for an image to be identified, first determine each pointer instrument instance to be detected in the image through manual annotation or target detection, and crop the instrument part from the image; use the feature point extraction algorithm to process the cropped image and the template image, and obtain the corresponding feature point pairs between the two images through the matching algorithm; calculate the image transformation matrix according to the feature point pair algorithm, and transform the cropped image to fit the viewing angle characteristics of the template image; finally, extract the pointer, and calculate the reading based on the marked point information of the template image.

[0007] In the template matching-based method, one of the most critical steps is to establish the correspondence between the image to be identified and the template image, which requires extracting the feature points on the two images and matching them to obtain the feature point pairs, and then calculating the transformation matrix based on the feature point pairs. Existing methods have made many explorations and attempts to apply different feature point extraction and matching algorithms, such as using scale invariant feature transform (SIFT), accelerated robust features (SURF), ORB (Oriented FAST and Rotated BRIEF), KAZE, accelerated KAZE (AKAZE) and other feature point extraction algorithms, using similarity measurement methods such as Euclidean distance and Hamming distance, and using brute force matching, K nearest neighbor matching, grid-based motion statistics (GMS) matching and other matching algorithms to improve the accuracy of transformation. These methods can achieve good results when the target is the same and the background environment and lighting conditions are similar. However, when facing actual production scenarios, the ability of feature point extraction algorithms is often insufficient to support the complexity of the real environment, such as shooting angles and distances that are too different from the template image, variable lighting conditions, complex background environments, etc. This leads to frequent incorrect pairing of extracted feature points, resulting in completely wrong transformation results and reading results. Summary of the invention

[0008] The purpose of the present invention is to address the deficiencies of the prior art and to propose a pointer meter reading recognition method based on template semantic key point matching.

[0009] The object of the present invention is achieved through the following technical solution: a pointer meter reading recognition method based on template semantic key point matching, comprising the following steps:

[0010] (1) Collect pointer instrument images and classify them according to their appearance and range information; then build object detection datasets, semantic matching datasets, and category template datasets based on pointer instrument images;

[0011] (2) constructing a target detection model and a semantic matching model, and using the target detection dataset to train the target detection model, and using the semantic matching dataset to train the semantic matching model, to obtain the trained target detection model and semantic matching model;

[0012] (3) Input the image of the pointer instrument to be identified into the trained object detection model to obtain the instrument instance location and category as well as the pointer end position;

[0013] (4) According to the position of the instrument instance, the pointer instrument image to be identified is cropped into the instrument instance image to be identified, and the corresponding category template image and category template key points are selected from the category template data set according to the instrument instance category; the instrument instance image to be identified and the category template image are input into the trained semantic matching model to obtain the semantic matching points of the category template key points in the instrument instance image to be identified;

[0014] (5) Performing homography transformation on the semantic matching points of the category template key points in the instrument instance image to be identified, and calculating the instrument reading based on the category template key points and their specific scale value information and the pointer end position.

[0015] Furthermore, the step (1) includes the following sub-steps:

[0016] (1.1) Collecting pointer instrument images, and classifying the pointer instrument images with the same appearance and range information into the same type;

[0017] (1.2) Constructing an object detection dataset, specifically: for the collected pointer instrument images, annotating each pointer instrument instance in the pointer instrument image in the form of a bounding box, and annotating the pointer end of each pointer instrument instance in the form of a point;

[0018] (1.3) Constructing a semantic matching dataset, specifically: for the annotated pointer instrument images in the target detection dataset, crop each pointer instrument instance into a separate instrument instance image according to the annotated bounding box; for each instrument instance image, annotate at least 4 non-collinear semantic key points with serial numbers, including the key scale position points on the instrument panel, the pointer rotation center point, and the user-defined semantic information point; among them, the same semantic key points of the same type of instrument instance images have the same serial number;

[0019] (1.4) Construct a category template dataset, specifically: select a clear, complete, and well-viewing and well-illuminated instrument instance image from each category of the semantic matching dataset as the template image for this category, use its annotated pointer rotation center point and key scale position points as the key points of the category template, and annotate each key scale position point with specific scale value information.

[0020] Furthermore, in step (2), constructing the target detection model specifically includes:

[0021] YOLOX is selected as the basic structure of the target detection model, and its regression prediction head is improved to construct the target detection model; the target detection model includes a backbone network, a feature pyramid network and a prediction layer, wherein the backbone network is used to obtain a multi-scale feature map of the pointer instrument image; the feature pyramid network is used to perform multi-scale feature fusion on the multi-scale feature map to obtain a fused feature map; the prediction layer includes a classification prediction head and a regression prediction head, the classification prediction head is used to output the probability of each category, and the regression prediction head is used to output the predicted bounding box encoding coordinates, the predicted bounding box foreground probability and the pointer end position encoding coordinates;

[0022] Furthermore, in step (2), constructing a semantic matching model specifically includes:

[0023] CHMNet is selected as the basic structure of the semantic matching model, which includes a high-dimensional correlation calculation module, a convolutional Hough matching module and a flow field and key point conversion module, wherein the high-dimensional correlation calculation module includes a feature extractor, a multi-scale processing module and a size adjustment layer; the convolutional Hough matching module includes a first convolutional Hough matching layer, a maximum pooling layer and a second convolutional Hough matching layer; the flow field and key point conversion module includes a flow field construction module and a key point conversion module, the flow field construction module is implemented by using a Gaussian kernel soft-argmax operation, and the key point conversion module is implemented by a weighted soft sampler;

[0024] The instrument instance image and template image to be identified are input into the semantic matching model, and firstly pass through the feature extractor of the high-dimensional correlation calculation module to obtain the multi-scale feature map corresponding to the two images; the multi-scale feature map corresponding to the two images is subjected to correlation calculation by the multi-scale processing module to obtain the multi-scale feature correlation tensor; the multi-scale feature correlation tensor is interpolated through the size adjustment layer to adjust the feature correlation tensors of different scales to a uniform spatial size; the multi-scale feature correlation tensor with uniform spatial size output by the high-dimensional correlation calculation module is sent to the convolutional Hough matching module, and first passes through the first convolutional Hough matching layer for voting to determine the candidate matches of each feature in the translation space and the scaling space, and obtains the first matching score tensor at each position of each scale feature map; the matching score tensors at each position of all scale feature maps are retained through the maximum pooling layer. The candidate match with the highest score at each position of the feature map is used to obtain the second matching score tensor at each position of each scale feature map; the second matching score tensors at each position of all scale feature maps are aggregated and refined through the second convolutional Hough matching layer to obtain the final matching score tensor corresponding to the two images; the final matching score tensors corresponding to the two images are successively passed through the flow field construction module and the key point conversion module, and the dense flow field is obtained through the Gaussian kernel soft-argmax operation of the flow field construction module, which represents the mapping relationship between each pixel in the template image and the corresponding pixel in the instrument instance image to be identified; and then the final matching result between each template key point in each template image and the queried semantic key point in the instrument instance image to be identified is calculated from the dense flow field through the weighted soft sampler of the key point conversion module, that is, the semantic matching point corresponding to the semantics of the template key point.

[0025] Furthermore, in step (2), the target detection model is trained using the target detection dataset, specifically including:

[0026] After uniform scaling, the pointer instrument images in the target detection dataset are input into the target detection model, and the probability of each category, the predicted bounding box coding coordinates, the predicted bounding box foreground probability and the predicted pointer end position coding coordinates are output; the probability of each category, the predicted bounding box coding coordinates, the predicted bounding box foreground probability and the predicted pointer end position coding coordinates output by the prediction layer are post-processed to obtain the predicted pointer instrument instance category, the pointer instrument instance position expressed in the form of bounding box coordinates and the pointer end position coordinates expressed in the form of point coordinates; the predicted pointer instrument instance category is compared with the corresponding real pointer instrument instance The binary cross entropy loss is calculated as the category discrimination loss function of the target detection model; the intersection-over-union loss is calculated according to the predicted bounding box coordinate form of the pointer instrument instance position and its corresponding real bounding box coordinates; the L2 loss is calculated according to the predicted point coordinate form of the pointer end position coordinates and its corresponding real pointer end position coordinates; the total loss function of the target detection model is calculated according to the category discrimination loss function, the intersection-over-union loss and the L2 loss, and the parameters of the target detection model are adjusted with minimizing the total loss function of the target detection model as the optimization goal until the maximum training round is reached to obtain a trained target detection model.

[0027] Furthermore, the calculation formula of the total loss function of the target detection model is:

[0028] loss1 = -log(IoU(B bg , B pred ))+||P bg -P pred ||2+L(y bg ,y pred )

[0029] Among them, loss1 represents the total loss function of the target detection model; B bg represents the true bounding box, B pred represents the predicted bounding box of the pointer instrument instance, -log(IoU(·)) represents the intersection over union loss, and IoU(·) represents the intersection over union of two bounding boxes; P bg Indicates the actual pointer end position coordinates, P pred represents the predicted pointer end position coordinates, ‖·‖2 represents the L2 loss; y bg represents the true category, y pred represents the predicted instance category, and L(·) represents the category discrimination loss function.

[0030] Furthermore, in step (2), the semantic matching model is trained using the semantic matching dataset, specifically including:

[0031] Any instrument instance image in the semantic matching dataset is used as the instrument instance image to be identified, and another instrument instance image of the same type as the instrument instance image to be identified in the semantic matching dataset is used as the template image; the instrument instance image to be identified and the template image are input into the semantic matching model to obtain predicted semantic key points that semantically match the template key points; the loss function of the semantic matching model is calculated based on the predicted semantic key points and their corresponding true semantic key points, and the parameters of the semantic matching model are adjusted with minimizing the loss function of the semantic matching model as the optimization goal until the maximum number of training rounds is reached to obtain a trained semantic matching model.

[0032] Furthermore, the calculation formula of the loss function of the semantic matching model is:

[0033]

[0034] Among them, loss2 represents the loss function of the semantic matching model, M represents the number of semantic point pairs marked in the semantic matching dataset, and k′ m represents the predicted semantic key point output by the semantic matching model for the mth query key point, k m Represents the true semantic key point corresponding to the m-th query point.

[0035] Furthermore, the step (5) includes the following sub-steps:

[0036] (5.1) Based on the key points of the category template and their semantic matching points in the instrument instance image to be identified, the random sampling consensus method is used to calculate the homography matrix H of the instrument instance image to be identified relative to the category template image 3×3 :

[0037]

[0038] Where (x, y) represents the key point of the category template in the category template image, and (x′, y′) represents the semantic matching point in the instrument instance image to be identified;

[0039] (5.2) Use the homography matrix H obtained in step (5.1) to calculate the coordinates of the pointer end position obtained by the target detection model 3×3 Calculate the pointer end position coordinates after homography transformation;

[0040] (5.3) Based on the category template key points and the pointer end position coordinates after homography transformation, the pointer vector P is obtained with the template pointer rotation center point as the starting point and the pointer end position coordinates after homography transformation as the end point; the scale vector group S = {S1, S2, …, S i ,…,S n};

[0041] (5.4) Calculate the direction angle θ of the pointer vector P relative to each scale vector in the scale vector set S according to the following formula i :

[0042]

[0043] (5.5) Based on the direction angle θ calculated in step (5.4) i , select the direction angle with the smallest absolute value from the negative direction angles and record it as θ s , and record the scale value corresponding to the key point of its scale vector in the category template annotation information s ; Select the direction angle with the smallest absolute value from the non-negative direction angles and record it as θ t , and record the scale value corresponding to the key point of its scale vector in the category template annotation information t ;

[0044] (5.6) Calculate the meter reading according to the following formula:

[0045]

[0046] Among them, result represents the instrument reading.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] (1) The present invention is based on a deep learning semantic matching method, which supports users to customize template key information to a certain extent, and can match corresponding semantic point pairs in the instance image to be identified. It can improve the flexibility of the recognition method of the present invention in processing different types of pointer instruments and the adaptability and accuracy to complex background environments, and can obtain relatively accurate readings of pointer instruments in an environment that is closer to the actual production scene.

[0049] (2) After collecting the images of the pointer instrument, the present invention constructs a target detection data set, a semantic matching data set and template information according to the user's requirements. The template information of the pointer instrument can support user-defined data to a certain extent. The present invention constructs a target detection model and a semantic matching model for pointer instrument reading recognition. Then, the target detection model can be used to identify the instance information and the pointer end position of each pointer instrument. The semantic matching model can be used to identify the semantic corresponding points between each instrument instance and the key points marked in its category template. The semantic matching point pairs are then used to calculate the homography matrix and transform the pointer end coordinates in the instance. By constructing a pointer vector and a scale vector group, the direction angle can be calculated to obtain the final reading of the pointer instrument.

[0050] (3) The present invention can support users to customize the template data of pointer instruments to a certain extent, making up for the shortcomings of existing methods in scalability and flexibility for different instrument types; and can normally identify pointer instrument readings in more complex production environments, improving the robustness and versatility of the identification method described in the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flow chart of the pointer instrument reading recognition method based on template semantic key point matching of the present invention;

[0052] Figure 2 Schematic diagram of the construction of the target detection dataset, semantic matching dataset, and category template dataset in the present invention;

[0053] Figure 3 It is a functional flow chart of the target detection model and the semantic matching model in the present invention;

[0054] Figure 4 The present invention is a flow chart for calculating the final reading of the pointer instrument. DETAILED DESCRIPTION

[0055] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims. It should be understood that the above general description and the detailed description that follows are merely exemplary and explanatory and cannot limit the present application.

[0056] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0057] It should be understood that, although the term first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of..." or "when..." or "in response to determination". Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process and method including a series of elements include not only those elements, but also include other elements not clearly listed, or also include elements inherent to such process and method. In the absence of more restrictions, the elements limited by the statement "including one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.

[0058] The present invention is described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.

[0059] See also Figure 1 The pointer meter reading recognition method based on template semantic key point matching of the present invention specifically comprises the following steps:

[0060] (1) Collect pointer instrument images and classify them according to the appearance and range of the instruments; then build object detection datasets, semantic matching datasets, and category template datasets based on the pointer instrument images, such as Figure 2 shown.

[0061] (1.1) Collect images of pointer instruments and classify the images of pointer instruments with the same appearance, range and other information into the same type.

[0062] It should be understood that after collecting the pointer instrument images, they need to be classified according to the appearance and range of the instruments, and the instrument images with the same appearance, range and other information are classified into the same type.

[0063] (1.2) Constructing a target detection dataset, specifically: for the collected pointer instrument images, annotating each pointer instrument instance in the pointer instrument image in the form of a bounding box, and annotating the pointer end of each pointer instrument instance in the form of a point.

[0064] (1.3) Construct a semantic matching dataset, specifically: for the annotated pointer instrument images in the target detection dataset, crop each pointer instrument instance into a separate instrument instance image according to the annotated bounding box; for each instrument instance image, annotate at least 4 non-collinear semantic key points with serial numbers, including the key scale position points on the instrument panel, the pointer rotation center point, and the user-defined semantic information point. Among them, the same type of instrument instance images have the same serial number for the same semantic key points. For example, in the same type of instrument instance images, point 1 is the pointer rotation center point, point 2 is the zero scale point, and so on.

[0065] (1.4) Construct a category template dataset, specifically: select a clear, complete, and well-viewing and well-illuminated instrument instance image from each category of the semantic matching dataset as the template image for this category, use its annotated pointer rotation center point and key scale position points as the key points of the category template, and annotate each key scale position point with specific scale value information.

[0066] (2) Construct a target detection model and a semantic matching model, and use the target detection dataset to train the target detection model, and use the semantic matching dataset to train the semantic matching model, so as to obtain the trained target detection model and semantic matching model.

[0067] It should be understood that in order to recognize the readings of pointer instruments, it is necessary to build a target detection model and a semantic matching model, whose functions and processes are as follows: Figure 3 As shown in the figure. The target detection model has two tasks: one is to detect all pointer instrument instances in the pointer instrument image, including instance categories and instance positions; the other is to detect the pointer end position of each pointer instrument instance for use in subsequent recognition steps. The task of the semantic matching model is to match the semantic corresponding points on the instrument instance image to be identified according to the annotated semantic key points with serial numbers. The annotated semantic key points with serial numbers generally include the key scale position points on the instrument panel and the pointer rotation center point. Of course, they can also include user-defined semantic information points.

[0068] In this embodiment, constructing a target detection model specifically includes: selecting YOLOX as the basic structure of the target detection model, and improving its regression prediction head to construct the target detection model; the target detection model includes a backbone network, a feature pyramid network (FPN) and a prediction layer, wherein the backbone network is used to obtain a multi-scale feature map of the pointer instrument image; the feature pyramid network is used to perform multi-scale feature fusion on the multi-scale feature map to obtain a fused feature map with richer semantic information; the prediction layer includes a classification prediction head and a regression prediction head, the classification prediction head is used to output the probability of each category, and the regression prediction head is used to output the predicted bounding box encoding coordinates, the predicted bounding box foreground probability and the pointer end position encoding coordinates.

[0069] It should be understood that the basic YOLOX model structure cannot predict the end position of the pointer. Therefore, the regression prediction head of the prediction layer needs to be improved to expand the output dimension of a single pointer instrument image from [h, w, 4] to [h, w, 6], where h and w represent the height and width of the input image (i.e., the pointer instrument image input to the target detection model). The pointer end position decoding method is consistent with the YOLOX target position decoding method, so the regression prediction head can output the predicted bounding box encoding coordinates and the pointer end position encoding coordinates.

[0070] Furthermore, the target detection data set is used to train the target detection model, specifically including: after uniformly scaling the pointer instrument image in the target detection data set, it is input into the target detection model, and firstly passes through the backbone network to obtain a multi-scale feature map. Then the multi-scale feature map is subjected to multi-scale feature fusion through the feature pyramid network to obtain a fused feature map with richer semantic information. Finally, the fused feature map enters the prediction layer, and the prediction result output by the prediction layer is obtained through the decoupled classification prediction head and regression prediction head; wherein, the probability of each category is obtained through the classification prediction head, and the predicted bounding box encoding coordinates, the predicted bounding box foreground probability and the predicted pointer end position encoding coordinates are obtained through the regression prediction head. Afterwards, the probabilities of each category, the predicted bounding box encoding coordinates, the predicted bounding box foreground probability and the predicted pointer end position encoding coordinates output by the prediction layer are post-processed through post-processing operation methods such as non-maximum suppression (NMS) to obtain the predicted pointer instrument instance category, the pointer instrument instance position expressed in the form of bounding box coordinates and the pointer end position coordinates expressed in the form of point coordinates, such as Figure 3As shown. According to the predicted pointer instrument instance category and its corresponding real pointer instrument instance category, the binary cross entropy loss is calculated as the category discrimination loss function of the target detection model. The original bounding box loss function is improved, that is, the L2 loss is added to the original bounding box loss function to replace the original bounding box loss function. Specifically, the intersection-over-union loss is calculated according to the predicted pointer instrument instance position expressed in the form of bounding box coordinates and its corresponding real bounding box coordinates; the L2 loss is calculated according to the predicted pointer end position coordinates expressed in the form of point coordinates and its corresponding real pointer end position coordinates. The total loss function of the target detection model is calculated based on the category discrimination loss function, the intersection-over-union loss and the L2 loss. The total loss function of the target detection model is minimized as the optimization goal, and the parameters of the target detection model are adjusted until the maximum training round is reached to obtain a trained target detection model.

[0071] It should be noted that when post-processing the category probabilities, predicted bounding box encoding coordinates, predicted bounding box foreground probabilities and predicted pointer end position encoding coordinates output by the prediction layer, after multiplying the probability of each category with the predicted bounding box foreground probability, the category corresponding to the maximum value is selected as the final predicted pointer instrument instance category.

[0072] Furthermore, the calculation formula of the total loss function of the target detection model is:

[0073] loss1 = -log(IoU(B bg ,B pred ))+||P bg -P pred ||2+L(y bg ,y pred )

[0074] Among them, loss1 represents the total loss function of the target detection model; B bg represents the true bounding box, B pred represents the predicted bounding box of the pointer instrument instance, -log(IoU(·)) represents the intersection over union loss, and IoU(·) represents the intersection over union of two bounding boxes; P bg Indicates the actual pointer end position coordinates, P pred represents the predicted pointer end position coordinates, ‖·‖2 represents the L2 loss (i.e., L2 paradigm); bg represents the true category, y pred Indicates the predicted instance category, and L(·) represents the category discrimination loss function. The true bounding box coordinates and the true pointer end position coordinates in the above formula are the bounding box form coordinates and the pointer end coordinates of the pointer instrument instance marked when constructing the target detection dataset.

[0075] In this embodiment, constructing a semantic matching model specifically includes: selecting CHMNet (Convolutional Hough Matching Network) as the basic structure of the semantic matching model, the semantic matching model includes a high-dimensional correlation calculation module, a convolutional Hough matching module and a flow field and key point conversion module, wherein the high-dimensional correlation calculation module includes a feature extractor, a multi-scale processing module and a size adjustment layer; the convolutional Hough matching module includes a first convolutional Hough matching layer, a maximum pooling layer and a second convolutional Hough matching layer; the flow field and key point conversion module includes a flow field construction module and a key point conversion module, the flow field construction module is implemented by using a Gaussian kernel soft-argmax operation, and the key point conversion module is implemented by a weighted soft sampler.

[0076] Specifically, the instrument instance image and the template image to be identified are input into the semantic matching model, and firstly pass through the feature extractor of the high-dimensional correlation calculation module to obtain the multi-scale feature map corresponding to the two images; the multi-scale feature map corresponding to the two images is subjected to correlation calculation by the multi-scale processing module to obtain the multi-scale feature correlation tensor; the multi-scale feature correlation tensor is interpolated by the size adjustment layer to adjust the feature correlation tensors of different scales to a unified spatial size. The multi-scale feature correlation tensor with a unified spatial size output by the high-dimensional correlation calculation module is sent to the convolutional Hough matching module, and firstly passes through the first convolutional Hough matching layer for voting to determine the candidate matches of each feature in the translation space and the scaling space, and obtains the first matching score tensor at each position of each scale feature map; the matching score tensors at each position of all scale feature maps are retained by the maximum pooling layer with the highest score candidate match at each position of each scale feature map, and obtain the second matching score tensor at each position of each scale feature map; the second matching score tensors at each position of all scale feature maps are aggregated and refined by the second convolutional Hough matching layer to obtain the final matching score tensor corresponding to the two images. The final matching score tensors corresponding to the two images pass through the flow field construction module and the key point conversion module in turn. The dense flow field is obtained through the Gaussian kernel soft-argmax operation of the flow field construction module, which represents the mapping relationship between each pixel in the template image and the corresponding pixel in the instrument instance image to be identified. The weighted soft sampler of the key point conversion module is then used to calculate the final matching result between each template key point in each template image and the semantic key point queried in the instrument instance image to be identified from the dense flow field, that is, the semantic matching point corresponding to the template key point semantics, and then obtain the corresponding semantic matching point pair.

[0077] Furthermore, the workflow of the high-dimensional correlation calculation module specifically includes: the instrument instance image to be identified and the template image are firstly passed through a feature extractor, such as selecting a pre-trained convolutional network ResNet-101 as the feature extractor of the high-dimensional correlation calculation module, to obtain the original feature map corresponding to the two images, and then the original feature map is reduced three times, each time the length and width are reduced to the original The convolution layers with a convolution kernel size of 3×3 are used to obtain feature maps of three scales, and the multi-scale feature maps corresponding to the two images output by the feature extractor can be obtained. Then the multi-scale feature maps corresponding to the two images are subjected to the following dot product operation by the multi-scale processing module to calculate the correlation coefficient between all scale features corresponding to the two images, so as to obtain the correlation tensor of all scale features:

[0078]

[0079] Among them, m and n represent the spatial scales of the two images (i.e., the instrument instance image to be identified and the template image), x m and x′ n Indicates the position coordinates of the two images at the corresponding spatial scale of the subscript, C mn (x m ,x′ n ) represents the coordinate x m and x′ n The correlation coefficient of the feature represented by m (x m ) and F′ n (x′ n ) represents the features of the position coordinates of the spatial scale feature maps corresponding to the subscripts of the two images, and RELU represents the linear rectification activation function. Finally, all scale feature related tensors are interpolated through the size adjustment layer to adjust the feature related tensors of different scales to a unified spatial size, that is, each group of 4-dimensional feature related tensors C ij Through interpolation operations, they are adjusted to a uniform spatial size and together form a 6-dimensional feature-related tensor.

[0080] Furthermore, the workflow of the convolutional Hough matching module specifically includes: the multi-scale feature correlation tensor with uniform spatial size output by the high-dimensional correlation calculation module, that is, the 6-dimensional feature correlation tensor, first passes through the first convolutional Hough matching layer for voting to determine the candidate matches of each feature in the translation space and the scaling space, and obtains all the 6-dimensional matching score tensors at each position of the feature map. The 6-dimensional matching score tensor retains the candidate matches with the highest scores at different scales at each position through the maximum pooling layer to obtain a 4-dimensional matching score tensor. Finally, the second convolutional Hough matching layer is used to aggregate the 4-dimensional matching score tensors of different scales, and the result is further refined to obtain the final 4-dimensional matching score tensor. Among them, in this embodiment, the first convolutional Hough matching layer is a 6-dimensional convolutional Hough matching layer, and the second convolutional Hough matching layer is a 4-dimensional convolutional Hough matching layer.

[0081] It should be understood that the use of the second convolutional Hough matching layer can process features of different scales and fuse the output results to aggregate visual information at different levels; and the second convolutional Hough matching layer is used to further refine the aggregated results to obtain a more accurate matching result, that is, the final 4-dimensional matching score tensor.

[0082] Furthermore, the workflow of the flow field and key point conversion module specifically includes: the final 4-dimensional matching score tensor output by the convolutional Hough matching module first passes through the flow field construction module, and a dense flow field is obtained by using a Gaussian kernel soft-argmax operation to represent the mapping relationship between each pixel in the template image and the corresponding pixel in the instrument instance image to be identified. The dense flow field output by the flow field construction module then passes through the key point conversion module, and a weighted soft sampler is used to calculate the final matching result between each template key point in each template image and the semantic key point queried in the instrument instance image to be identified from the dense flow field, that is, the semantic matching point corresponding to the template key point semantics, and then obtains the corresponding semantic matching point pair. Among them, the calculation formula of the weighted soft sampler is expressed as:

[0083]

[0084] Among them, k represents the semantic key point queried in the instrument instance image to be identified, represents the weight for the key point k at the image position (i, j), and τ is the distance threshold, which can be adjusted according to actual needs.

[0085] Furthermore, the semantic matching model is trained using the semantic matching dataset, specifically including: taking any instrument instance image in the semantic matching dataset as the instrument instance image to be identified, and taking another instrument instance image of the same type as the instrument instance image to be identified in the semantic matching dataset as the template image; inputting the instrument instance image to be identified and the template image into the semantic matching model to obtain predicted semantic key points that semantically match the template key points; calculating the loss function of the semantic matching model based on the predicted semantic key points and their corresponding real semantic key points, taking minimizing the loss function of the semantic matching model as the optimization goal, adjusting the parameters of the semantic matching model until the maximum number of training rounds is reached, and obtaining a trained semantic matching model.

[0086] It should be noted that, when two instance images are input into the semantic matching model, the semantic matching model can be used to query the semantic corresponding points of the semantic key points queried in one image in the other image.

[0087] Furthermore, the loss function of the semantic matching model is calculated as follows:

[0088]

[0089] Among them, loss2 represents the loss function of the semantic matching model, M represents the number of semantic point pairs marked in the semantic matching dataset, and k′ m represents the predicted semantic key point output by the semantic matching model for the mth query key point, k m It represents the real semantic key point corresponding to the m-th query point, which is determined according to the semantic key points annotated in the semantic matching dataset.

[0090] (3) Input the image of the pointer instrument to be identified into the trained object detection model to obtain the instrument instance location and category as well as the pointer end position.

[0091] (4) According to the position of the instrument instance, the pointer instrument image to be identified is cropped into the instrument instance image to be identified, and the corresponding category template image and category template key points are selected from the category template dataset according to the instrument instance category; the instrument instance image to be identified and the category template image are input into the trained semantic matching model to obtain the semantic matching points of the category template key points in the instrument instance image to be identified.

[0092] (5) Performing homography transformation on the semantic matching points of the category template key points in the instrument instance image to be identified, and calculating the instrument reading based on the category template key points and their specific scale value information and the pointer end position.

[0093] It should be noted that the instance information of each pointer instrument in the pointer instrument image to be identified, that is, the instrument instance position and category and the pointer end position, is obtained through the target detection model, and the semantic matching point pair information of each instance about the template custom key point is obtained through the semantic matching model. Figure 4 As shown, the final reading result is obtained through the following steps.

[0094] (5.1) Based on the semantic matching point pairs, i.e., the key points of the category template and their semantic matching points in the instrument instance image to be identified, the random sampling consensus method (RANSAC) is used to calculate the homography matrix H of the instrument instance image to be identified relative to the category template image. 3×3 :

[0095]

[0096] Among them, (x, y) represents the category template key point in the category template image, and (x′, y′) represents the semantic matching point in the instrument instance image to be identified. Since the number of non-collinear semantic key points annotated in each category template image is required to be no less than 4 when the semantic matching dataset and the category template dataset are constructed, the equation can be solved uniquely.

[0097] (5.2) Use the homography matrix H obtained in step (5.1) to calculate the coordinates of the pointer end position obtained by the target detection model 3×3 Calculate the pointer end position coordinates after homography transformation.

[0098] Specifically, the homography matrix H obtained in step (5.1) 3×3 By multiplying the coordinates of the pointer end position obtained by the target detection model (i.e., the coordinates of the pointer end position obtained in step (3)), the coordinates of the pointer end position after homography transformation can be obtained.

[0099] (5.3) Based on the category template key points and the pointer end position coordinates after homography transformation, the pointer vector P is obtained with the template pointer rotation center point as the starting point and the pointer end position coordinates after homography transformation as the end point; the scale vector group S = {S1, S2, …, S i ,…,S n}.

[0100] (5.4) Calculate the direction angle θ of the pointer vector P relative to each scale vector in the scale vector set S according to the following formula i :

[0101]

[0102] In the above formula, θ i<0 means the pointer vector P is in the i-th scale vector S i In the clockwise direction, θ i >0 means the pointer vector P is in the i-th scale vector S i The counterclockwise direction, θ i =0 indicates that the pointer vector P and the i-th scale vector S i Collinear.

[0103] (5.5) Based on the direction angle θ calculated in step (5.4) i , select the direction angle with the smallest absolute value from the negative direction angles and record it as θ s , and record the scale value corresponding to the key point of its scale vector in the category template annotation information s ; Select the direction angle with the smallest absolute value from the non-negative direction angles and record it as θ t , and record the scale value corresponding to the key point of its scale vector in the category template annotation information t .

[0104] (5.6) Calculate the meter reading according to the following formula:

[0105]

[0106] Among them, result represents the instrument reading.

[0107] In summary, the identification method described in the present invention can support users to customize the template data of pointer instruments to a certain extent, making up for the shortcomings of existing methods in scalability and flexibility for different instrument types; and it can normally identify pointer instrument readings in more complex production environments, thereby improving the robustness and versatility of the identification method described in the present invention.

[0108] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A pointer meter reading recognition method based on template semantic key point matching, characterized in that: The following steps are involved: (1) Collect images of pointer instruments and classify them according to their appearance and range information; Then, we build target detection dataset, semantic matching dataset and category template dataset based on pointer instrument images; (2) constructing a target detection model and a semantic matching model, and using the target detection dataset to train the target detection model, and using the semantic matching dataset to train the semantic matching model, to obtain the trained target detection model and semantic matching model; (3) Input the image of the pointer instrument to be identified into the trained object detection model to obtain the instrument instance location and category as well as the pointer end position; (4) cropping the pointer instrument image to be identified into the instrument instance image to be identified according to the instrument instance position, and selecting the corresponding category template image and category template key points in the category template dataset according to the instrument instance category; Input the instrument instance image to be identified and the category template image into the trained semantic matching model to obtain the semantic matching points of the category template key points in the instrument instance image to be identified; (5) Performing homography transformation on the semantic matching points of the category template key points in the instrument instance image to be identified, and calculating the instrument reading based on the category template key points and their specific scale value information and the pointer end position.

2. The method for identifying pointer meter readings based on template semantic key point matching according to claim 1, characterized in that: The step (1) comprises the following sub-steps: (1.1) Collecting pointer instrument images, and classifying the pointer instrument images with the same appearance and range information into the same type; (1.2) Constructing an object detection dataset, specifically: for the collected pointer instrument images, annotating each pointer instrument instance in the pointer instrument image in the form of a bounding box, and annotating the pointer end of each pointer instrument instance in the form of a point; (1.3) Constructing a semantic matching dataset, specifically: for the annotated pointer instrument images in the target detection dataset, crop each pointer instrument instance into a separate instrument instance image according to the annotated bounding box; for each instrument instance image, annotate at least 4 non-collinear semantic key points with serial numbers, including the key scale position points on the instrument panel, the pointer rotation center point, and the user-defined semantic information point; among them, the same semantic key points of the same type of instrument instance images have the same serial number; (1.4) Construct a category template dataset, specifically: select a clear, complete, and well-viewing and well-illuminated instrument instance image from each category of the semantic matching dataset as the template image for this category, use its annotated pointer rotation center point and key scale position points as the key points of the category template, and annotate each key scale position point with specific scale value information.

3. The method for identifying pointer meter readings based on template semantic key point matching according to claim 1, characterized in that: In the step (2), building the target detection model specifically includes: YOLOX is selected as the basic structure of the target detection model, and its regression prediction head is improved to construct the target detection model; the target detection model includes a backbone network, a feature pyramid network and a prediction layer, wherein the backbone network is used to obtain a multi-scale feature map of the pointer instrument image; the feature pyramid network is used to perform multi-scale feature fusion on the multi-scale feature map to obtain a fused feature map; the prediction layer includes a classification prediction head and a regression prediction head, the classification prediction head is used to output the probability of each category, and the regression prediction head is used to output the predicted bounding box encoding coordinates, the predicted bounding box foreground probability and the pointer end position encoding coordinates.

4. The method for identifying pointer meter readings based on template semantic key point matching according to claim 1, characterized in that: In the step (2), constructing a semantic matching model specifically includes: CHMNet is selected as the basic structure of the semantic matching model, which includes a high-dimensional correlation calculation module, a convolutional Hough matching module and a flow field and key point conversion module, wherein the high-dimensional correlation calculation module includes a feature extractor, a multi-scale processing module and a size adjustment layer; the convolutional Hough matching module includes a first convolutional Hough matching layer, a maximum pooling layer and a second convolutional Hough matching layer; the flow field and key point conversion module includes a flow field construction module and a key point conversion module, the flow field construction module is implemented by using a Gaussian kernel soft-argmax operation, and the key point conversion module is implemented by a weighted soft sampler; The instrument instance image and template image to be identified are input into the semantic matching model, and firstly pass through the feature extractor of the high-dimensional correlation calculation module to obtain the multi-scale feature map corresponding to the two images; the multi-scale feature map corresponding to the two images is subjected to correlation calculation by the multi-scale processing module to obtain the multi-scale feature correlation tensor; the multi-scale feature correlation tensor is interpolated through the size adjustment layer to adjust the feature correlation tensors of different scales to a uniform spatial size; the multi-scale feature correlation tensor with uniform spatial size output by the high-dimensional correlation calculation module is sent to the convolutional Hough matching module, and first passes through the first convolutional Hough matching layer for voting to determine the candidate matches of each feature in the translation space and the scaling space, and obtains the first matching score tensor at each position of each scale feature map; the matching score tensors at each position of all scale feature maps are retained through the maximum pooling layer. The candidate match with the highest score at each position of the feature map is used to obtain the second matching score tensor at each position of each scale feature map; the second matching score tensors at each position of all scale feature maps are aggregated and refined through the second convolutional Hough matching layer to obtain the final matching score tensor corresponding to the two images; the final matching score tensors corresponding to the two images are successively passed through the flow field construction module and the key point conversion module, and the dense flow field is obtained through the Gaussian kernel soft-argmax operation of the flow field construction module, which represents the mapping relationship between each pixel in the template image and the corresponding pixel in the instrument instance image to be identified; and then the final matching result between each template key point in each template image and the queried semantic key point in the instrument instance image to be identified is calculated from the dense flow field through the weighted soft sampler of the key point conversion module, that is, the semantic matching point corresponding to the semantics of the template key point.

5. The method for identifying pointer meter readings based on template semantic key point matching according to claim 1, characterized in that: In the step (2), the target detection model is trained using the target detection dataset, specifically including: After uniform scaling, the pointer instrument images in the target detection dataset are input into the target detection model, and the probability of each category, the predicted bounding box coding coordinates, the predicted bounding box foreground probability and the predicted pointer end position coding coordinates are output; the probability of each category, the predicted bounding box coding coordinates, the predicted bounding box foreground probability and the predicted pointer end position coding coordinates output by the prediction layer are post-processed to obtain the predicted pointer instrument instance category, the pointer instrument instance position expressed in the form of bounding box coordinates and the pointer end position coordinates expressed in the form of point coordinates; the predicted pointer instrument instance category is compared with the corresponding real pointer instrument instance The binary cross entropy loss is calculated as the category discrimination loss function of the target detection model; the intersection-over-union loss is calculated according to the predicted bounding box coordinate form of the pointer instrument instance position and its corresponding real bounding box coordinates; the L2 loss is calculated according to the predicted point coordinate form of the pointer end position coordinates and its corresponding real pointer end position coordinates; the total loss function of the target detection model is calculated according to the category discrimination loss function, the intersection-over-union loss and the L2 loss, and the parameters of the target detection model are adjusted with minimizing the total loss function of the target detection model as the optimization goal until the maximum training round is reached to obtain a trained target detection model.

6. The method for identifying pointer meter readings based on template semantic key point matching according to claim 5, characterized in that: The calculation formula of the total loss function of the target detection model is: loss1 = log(IoU(B) bg ,B pred ))+‖P bg -P pred ‖2+L(and bg ,and pred ) Among them, loss1 represents the total loss function of the target detection model; B bg represents the true bounding box, B pred represents the predicted bounding box of the pointer instrument instance, -log(IoU(·)) represents the intersection over union loss, and IoU(·) represents the intersection over union of two bounding boxes; P bg Indicates the actual pointer end position coordinates, P pred represents the predicted pointer end position coordinates, ‖·‖2 represents the L2 loss; y bg represents the true category, y pred represents the predicted instance category, and L(·) represents the category discrimination loss function.

7. The method for identifying pointer meter readings based on template semantic key point matching according to claim 1, characterized in that: In the step (2), the semantic matching model is trained using the semantic matching dataset, specifically including: Any instrument instance image in the semantic matching dataset is used as the instrument instance image to be identified, and another instrument instance image of the same type as the instrument instance image to be identified in the semantic matching dataset is used as the template image; the instrument instance image to be identified and the template image are input into the semantic matching model to obtain predicted semantic key points that semantically match the template key points; the loss function of the semantic matching model is calculated based on the predicted semantic key points and their corresponding true semantic key points, and the parameters of the semantic matching model are adjusted with minimizing the loss function of the semantic matching model as the optimization goal until the maximum number of training rounds is reached to obtain a trained semantic matching model.

8. The method for identifying pointer meter readings based on template semantic key point matching according to claim 7, characterized in that: The calculation formula of the loss function of the semantic matching model is: Among them, loss2 represents the loss function of the semantic matching model, M represents the number of semantic point pairs marked in the semantic matching dataset, and k ′ m represents the predicted semantic key point output by the semantic matching model for the mth query key point, k m Represents the true semantic key point corresponding to the m-th query point.

9. The method for identifying pointer meter readings based on template semantic key point matching according to claim 1, characterized in that: The step (5) comprises the following sub-steps: (5.1) Based on the key points of the category template and their semantic matching points in the instrument instance image to be identified, the random sampling consensus method is used to calculate the homography matrix H of the instrument instance image to be identified relative to the category template image 3×3 : Among them, (x, y) represents the category template key point in the category template image, (x ′ ,y ′ ) represents the semantic matching point in the instrument instance image to be identified; (5.2) Use the homography matrix H obtained in step (5.1) to calculate the coordinates of the pointer end position obtained by the target detection model 3×3 Calculate the pointer end position coordinates after homography transformation; (5.3) Based on the category template key points and the pointer end position coordinates after homography transformation, the pointer vector P is obtained with the template pointer rotation center point as the starting point and the pointer end position coordinates after homography transformation as the end point; the scale vector group S = {S1, S2, …, S i ,…,S n }; (5.4) Calculate the direction angle θ of the pointer vector P relative to each scale vector in the scale vector set S according to the following formula i : (5.5) Based on the direction angle θ calculated in step (5.4) i , select the direction angle with the smallest absolute value from the negative direction angles and record it as θ s , and record the scale value corresponding to the key point of its scale vector in the category template annotation information s ; Select the direction angle with the smallest absolute value from the non-negative direction angles and record it as θ t , and record the scale value corresponding to the key point of its scale vector in the category template annotation information t ; (5.6) Calculate the meter reading according to the following formula: Among them, result represents the instrument reading.

Images