A pointer instrument reading recognition method based on template semantic key point matching

Abstract

The application discloses a pointer instrument reading recognition method based on template semantic key point matching, and the method comprises the following steps: constructing a target detection dataset, a semantic matching dataset and a category template dataset based on collected pointer instrument images; constructing and training a target detection model and a semantic matching model; inputting a to-be-recognized pointer instrument image into the trained target detection model to obtain an instance position, a category and a pointer end position; inputting an instance image and a category template image into the trained semantic matching model to obtain semantic matching points of category template key points in the to-be-recognized instance image; performing homography transformation on the semantic matching points of the category template key points in the to-be-recognized instance image, and calculating an instrument reading according to the category template key points, specific scale value information of the category template key points and the pointer end position. The application can obtain more accurate instrument readings in a complex environment, and is beneficial to improving the robustness and universality of reading recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pointer instrument reading recognition, and particularly relates to a pointer instrument reading recognition method based on template semantic key point matching. BACKGROUND

[0002] Pointer instruments are widely used in many fields such as industry, transportation and medical treatment, such as speedometer, temperature meter, pressure gauge and the like. In particular, in some complex industrial environments with radiation, toxicity, high temperature and the like, it is often necessary to confirm the normality and safety of production by checking the reading of the instrument. The reading of these instruments is usually manually performed, and the instruments are often numerous and widely distributed, so the manual reading method is time-consuming, low in efficiency and high in labor intensity. Therefore, it is of important practical significance to automatically recognize the reading of the pointer instrument by using computer vision technology.

[0003] In recent years, with the development of computer vision technology, there have been some solutions to try to solve this problem. According to whether a template needs to be set, these methods can be divided into two categories: template-free method and template matching based method.

[0004] Among them, the overall process of the template-free method is as follows: a target detection method is used to detect the instrument instance to be recognized in the image; the pointer position and scale position information of each instrument instance are obtained by using a detection method; the scale reading is obtained by using OCR text recognition technology, and a corresponding relationship with the scale position is established; and the reading is calculated by comprehensively considering the above information.

[0005] However, in the template-free method, one of the most critical steps is to establish the corresponding relationship between the scale reading and the scale position. The scale reading is usually realized by using OCR text recognition technology, which has a high requirement for the clarity, occlusion and light distribution on the image, and the actual production scene cannot meet these requirements many times, so the reading recognition is prone to failure. In addition, since no additional prior knowledge provided by the template information is used, the expansibility and flexibility are poor.

[0006] The overall process of the template matching based method is as follows: a clear, complete, good light condition and shooting angle picture is selected as a template for each type of instrument, and the key scale position and pointer rotation center are labeled; for an image to be recognized, each pointer instrument instance to be detected in the image is determined by manual labeling or target detection, and the instrument part is cropped from the image; the cropped picture and the template picture are processed by using a feature point extraction algorithm, and the corresponding feature point pairs between the two images are obtained by using a matching algorithm; the image transformation matrix is calculated according to the feature point pair algorithm, the cropped image is transformed to fit the perspective feature of the template image; and finally the pointer is extracted, and the reading is calculated according to the labeled point information of the template image.

[0007] In the template matching based method, one of the most critical steps is to establish the correspondence between the image to be identified and the template image, which requires extracting feature points on the two images and matching them to obtain feature point pairs, and then calculating the transformation matrix according to the feature point pairs. The existing methods have made a lot of explorations and attempts for different feature point extraction and matching algorithms, such as using Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), ORB (Oriented FAST and Rotated BRIEF), KAZE, Accelerated-KAZE (AKAZE) and other feature point extraction algorithms, using Euclidean distance, Hamming distance and other similarity measurement methods, and using different combinations of matching algorithms such as brute force matching, K nearest neighbor matching, Grid-based Motion Statistics (GMS) matching, etc. to improve the transformation accuracy. These methods can achieve good results under the condition that the target is the same, the background environment and the light condition are similar, but in the face of actual production scenes, the ability of the feature point extraction algorithm is often insufficient to support the complexity of the real environment, such as too large shooting angle and distance from the template image, variable light conditions, complex background environment, etc. This leads to the frequent pairing errors of the extracted feature points, resulting in completely wrong transformation results and reading results. SUMMARY

[0008] The present application aims at the deficiencies of the prior art and proposes a pointer type instrument reading recognition method based on template semantic key point matching.

[0009] The object of the present application is achieved by the following technical solution: a pointer type instrument reading recognition method based on template semantic key point matching, comprising the following steps:

[0010] (1) Collecting pointer type instrument images and classifying them according to the appearance and range information of the instruments; then constructing a target detection dataset, a semantic matching dataset and a category template dataset based on the pointer type instrument images;

[0011] (2) Constructing a target detection model and a semantic matching model, and training the target detection model using the target detection dataset and the semantic matching model using the semantic matching dataset, to obtain the trained target detection model and semantic matching model;

[0012] (3) input the to-be-identified pointer instrument image into the trained target detection model, to obtain the instrument instance position, the instrument instance category, and the pointer end position;

[0013] (4) according to the instrument instance position, cut the to-be-identified pointer instrument image into a to-be-identified instrument instance image, according to the instrument instance category, select a corresponding category template image and a category template key point in the category template dataset; input the to-be-identified instrument instance image and the category template image into the trained semantic matching model, to obtain a semantic matching point of the category template key point in the to-be-identified instrument instance image;

[0014] (5) perform a homography transformation on the semantic matching point of the category template key point in the to-be-identified instrument instance image, and calculate the instrument reading according to the category template key point, the specific scale value information of each key point, and the pointer end position.

[0015] Further, the step (1) comprises the following sub-steps:

[0016] (1.1) collect pointer instrument images, and divide the pointer instrument images with consistent appearance and range information into the same type;

[0017] (1.2) construct a target detection dataset, specifically: for the collected pointer instrument images, label each pointer instrument instance in the pointer instrument image in the form of a bounding box, and label the pointer end of each pointer instrument instance in the form of a point;

[0018] (1.3) construct a semantic matching dataset, specifically: for the pointer instrument images labeled in the target detection dataset, cut each pointer instrument instance into a separate instrument instance image according to the labeled bounding box; for each instrument instance image, label at least 4 non-collinear semantic key points with serial numbers, including key scale position points on the instrument panel, a pointer rotation center point, and user-defined semantic information points; wherein the serial numbers of the same semantic key points of instrument instance images of the same type are the same;

[0019] (1.4) construct a category template dataset, specifically: select a clear, complete, and good in perspective and lighting condition instrument instance image from the instrument instance images of each category in the semantic matching dataset as a category template image, label the pointer rotation center point and the key scale position points of the category template image as the category template key points, and label the specific scale value information of each key scale position point.

[0020] Further, in the step (2), the construction of the target detection model comprises:

[0021] YOLOX is selected as the basic structure of the target detection model, and the regression prediction head thereof is improved to construct the target detection model; the target detection model comprises a backbone network, a feature pyramid network and a prediction layer, wherein the backbone network is used to obtain multi-scale feature maps of the pointer instrument image; the feature pyramid network is used to perform multi-scale feature fusion on the multi-scale feature maps to obtain a fused feature map; the prediction layer comprises a classification prediction head and a regression prediction head, the classification prediction head is used to output class probabilities, and the regression prediction head is used to output predicted bounding box encoding coordinates, predicted bounding box foreground probabilities and pointer tip position encoding coordinates;

[0022] Further, in the step (2), the constructing the semantic matching model specifically comprises:

[0023] CHMNet is selected as the basic structure of the semantic matching model, and the semantic matching model comprises a high-dimensional correlation calculation module, a convolutional Hough matching module and a flow field and key point conversion module, wherein the high-dimensional correlation calculation module comprises a feature extractor, a multi-scale processing module and a size adjustment layer; the convolutional Hough matching module comprises a first convolutional Hough matching layer, a maximum pooling layer and a second convolutional Hough matching layer; the flow field and key point conversion module comprises a flow field construction module and a key point conversion module, the flow field construction module is realized by using a Gaussian kernel soft-argmax operation, and the key point conversion module is realized by using a weighted soft sampler;

[0024] The instrument instance image to be identified and the template image are input into the semantic matching model, and first pass through the feature extractor of the high-dimensional correlation calculation module to obtain the multi-scale feature maps corresponding to the two images; the multi-scale feature maps corresponding to the two images are subjected to correlation calculation by the multi-scale processing module to obtain a multi-scale feature correlation tensor; the multi-scale feature correlation tensor is subjected to interpolation operation by the size adjustment layer to adjust the feature correlation tensors of different scales to a unified spatial size; the multi-scale feature correlation tensors of the spatial size output by the high-dimensional correlation calculation module are input into the convolution Hough matching module, first pass through the first convolution Hough matching layer for voting to determine the candidate matches of each feature in the translation space and the scaling space, and obtain the first matching score tensor at each position of each scale feature map; the matching score tensors at each position of all scale feature maps are retained by the maximum pooling layer to obtain the highest score of each position of each scale feature map, and obtain the second matching score tensor at each position of each scale feature map; the second matching score tensors at each position of all scale feature maps are aggregated and refined by the second convolution Hough matching layer to obtain the final matching score tensor corresponding to the two images; the final matching score tensor corresponding to the two images is sequentially subjected to the flow field construction module and the key point conversion module, the dense flow field is obtained by the Gaussian kernel soft-argmax operation of the flow field construction module, and the mapping relationship of each pixel in the template image to the corresponding pixel in the instrument instance image to be identified is represented; then the weighted soft sampler of the key point conversion module is used to calculate the final matching result between each template key point in each template image and the semantic key point to be queried in the instrument instance image to be identified, i.e. the semantic matching point corresponding to the semantic of the template key point.

[0025] Further, in the step (2), the target detection model is trained using the target detection dataset, specifically comprising:

[0026] After uniformly scaling the pointer instrument images in the target detection dataset, they are input into the target detection model. The model outputs the probability of each category, the coded coordinates of the predicted bounding box, the probability of the predicted foreground bounding box, and the coded coordinates of the predicted pointer end position. Post-processing is performed on the probability of each category, the coded coordinates of the predicted bounding box, the probability of the predicted foreground bounding box, and the coded coordinates of the predicted pointer end position output from the prediction layer to obtain the predicted pointer instrument instance category, the pointer instrument instance position in bounding box coordinate form, and the pointer end position coordinates in point coordinate form. Based on the predicted pointer instrument instance category and its corresponding real pointer instrument instance... The binary cross-entropy loss is calculated as the class discrimination loss function of the object detection model. The cross-union ratio (CUNR) loss is calculated based on the predicted bounding box coordinates of the pointer instrument instance and its corresponding ground truth bounding box coordinates. The L2 loss is calculated based on the predicted pointer end position coordinates and its corresponding ground truth pointer end position coordinates. The total loss function of the object detection model is calculated based on the class discrimination loss function, the CUNR loss, and the L2 loss. The parameters of the object detection model are adjusted with the goal of minimizing the total loss function of the object detection model until the maximum number of training epochs is reached, and a well-trained object detection model is obtained.

[0027] Furthermore, the formula for calculating the total loss function of the target detection model is as follows:

[0028] loss1 = -log(IoU(B) bg B pred ))+||P bg -P pred ||2+L(y bg y pred )

[0029] Where loss1 represents the total loss function of the object detection model; B bg B represents the true bounding box. pred P represents the bounding box of the predicted pointer instrument instance, -log(IoU(·)) represents the crossover ratio loss, and IoU(·) represents the crossover ratio of the two bounding boxes; bg P represents the actual coordinates of the pointer's end position. pred This represents the coordinates of the predicted pointer's end position; ||·||² represents the L2 loss; y bg Representing the true category, y pred L represents the predicted instance class, and L(·) represents the class discrimination loss function.

[0030] Furthermore, in step (2), the semantic matching model is trained using a semantic matching dataset, specifically including:

[0031] The instrument instance image in any of the semantic matching dataset is taken as a to-be-identified instrument instance image, and another instrument instance image of the same type as the to-be-identified instrument instance image in the semantic matching dataset is taken as a template image; the to-be-identified instrument instance image and the template image are input into the semantic matching model to obtain predicted semantic key points matched with the template key points; a loss function of the semantic matching model is calculated according to the predicted semantic key points and the corresponding real semantic key points, and the loss function of the semantic matching model is minimized as an optimization objective, the parameters of the semantic matching model are adjusted until a maximum training round is reached, and a trained semantic matching model is obtained.

[0032] Further, the loss function of the semantic matching model is calculated according to the following formula:

[0033]

[0034] Wherein, loss2 represents the loss function of the semantic matching model, M represents the number of marked semantic points in the semantic matching dataset, k' m represents the predicted semantic key point output by the semantic matching model for the mth query key point, k m represents the real semantic key point corresponding to the mth query point.

[0035] Further, the step (5) includes the following sub-steps:

[0036] (5.1) According to the category template key points and the semantic matching points in the to-be-identified instrument instance image, the homography matrix H 3×3 of the to-be-identified instrument instance image relative to the category template image is calculated using the random sample consensus method.

[0037]

[0038] Wherein, (x, y) represents the category template key point in the category template image, and (x', y') represents the semantic matching point in the to-be-identified instrument instance image.

[0039] (5.2) The coordinates of the pointer end position obtained by the target detection model are subjected to homographic transformation using the homography matrix H 3×3 obtained in step (5.1) to obtain the homographic transformed pointer end position coordinates.

[0040] (5.3) Based on the category template key points and the homographic transformed pointer end position coordinates, the pointer vector P is obtained with the template pointer rotation center point as the starting point and the homographic transformed pointer end position coordinates as the ending point; and the scale vector group S = {S1, S2, …, S i ,…,S n} is obtained with the template pointer rotation center point as the starting point and the template key scale position point as the ending point.

[0041] (5.4) Calculate the direction angle θ of the pointer vector P relative to each scale vector in the scale vector group S according to the following formula i :

[0042]

[0043] (5.5) Based on the direction angle θ calculated in step (5.4) i , select the direction angle with the smallest absolute value from the negative result direction angle, and record it as θ s , and record the scale value corresponding to the key point of its scale vector in the category template annotation information as scale s ; select the direction angle with the smallest absolute value from the non-negative direction angle, and record it as θ t , and record the scale value corresponding to the key point of its scale vector in the category template annotation information as scale t ;

[0044] (5.6) Calculate the instrument reading according to the following formula:

[0045]

[0046] Where, result represents the instrument reading.

[0047] Compared with the prior art, the beneficial effects of the present application are:

[0048] (1) The present application is based on a deep learning semantic matching method, which supports users to define the template key information to a certain extent, and can match the corresponding semantic point pairs in the to-be-recognized instance image, which can improve the flexibility of the recognition method for different types of pointer instruments, and the adaptability and accuracy for complex background environment, and can get relatively accurate readings of the pointer instrument in a more realistic production scene environment.

[0049] (2) After collecting the pointer instrument image, the present application constructs a target detection data set, a semantic matching data set and a template information according to the user's requirements. The template information of the pointer instrument can support the user to define the data to a certain extent. The present application constructs a target detection model and a semantic matching model for pointer instrument reading recognition. Then, the target detection model can recognize the instance information and the pointer end position of each pointer instrument, and the semantic matching model can recognize the semantic corresponding points of each instrument instance and the labeled key points in the category template. Then, the semantic matching point pair is used to calculate the homography matrix and transform the pointer end coordinates in the instance. By constructing the pointer vector and the scale vector group, the direction angle can be calculated to obtain the final reading of the pointer instrument.

[0050] (3) The application can support users to customize the template data of the pointer instrument to a certain extent, make up for the lack of extensibility and flexibility of the existing method for different instrument types, and can normally identify the reading of the pointer instrument in a more complex production environment, thereby improving the robustness and universality of the identification method. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 The flow chart of the pointer instrument reading identification method based on template semantic key point matching of the application;

[0052] Figure 2 The construction schematic diagram of the target detection data set, the semantic matching data set and the category template data set in the application;

[0053] Figure 3 The function flow chart of the target detection model and the semantic matching model in the application;

[0054] Figure 4 The flow chart of calculating the final reading of the pointer instrument in the application. DETAILED DESCRIPTION

[0055] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers are used to indicate the same elements throughout the several views. The following exemplary embodiments are described with reference to the drawings, wherein like numbers represent the same elements throughout the several views. The embodiments described in the following exemplary embodiments do not represent all the implementations in accordance with this application. Instead, they only illustrate possible implementations in accordance with some aspects of this application, as detailed in the appended claims. It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not restrictive of this application.

[0056] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in this application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0057] It should be understood that, although the terms first, second, third, etc. can be employed in this application to describe various information, these information should not be limited to these terms. These terms are only used to differentiate one piece of information from another. For example, without departing from the scope of the application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if' as used herein can be interpreted to mean "when" or "in response to determining" as in this manner. Moreover, the terms "comprise", "comprises" or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or devices that include a series of elements include not only those elements, but also other elements not explicitly listed, or other elements inherent in such processes, methods, articles or devices. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, article or device that includes the element.

[0058] The application will be described in detail below with reference to the drawings. The features in the following examples and embodiments can be combined with each other without conflict.

[0059] Referring to Figure 1 The pointer type instrument reading recognition method based on template semantic key point matching of the application specifically comprises the following steps:

[0060] (1) Collecting pointer type instrument images, and classifying them according to the appearance and range of the instruments and other information; then constructing a target detection data set, a semantic matching data set and a category template data set based on the pointer type instrument images, as shown in Figure 2 .

[0061] (1.1) Collecting pointer type instrument images, and dividing the pointer type instrument images with consistent appearance and range and other information into the same type.

[0062] It should be understood that after collecting the pointer type instrument images, the pointer type instrument images need to be classified according to the appearance and range of the instruments and other information first, and the instrument images with consistent appearance, range and other information are divided into the same type.

[0063] (1.2) Constructing a target detection data set, specifically: for the collected pointer type instrument images, labeling each pointer type instrument instance in the pointer type instrument images in the form of a bounding box, and labeling the pointer tip of each pointer type instrument instance in the form of a point.

[0064] (1.3) Constructing a semantic matching dataset, specifically: for the pointer instrument image in the target detection dataset that has been labeled, each pointer instrument instance is cropped into a separate instrument instance image according to the labeled bounding box; for each instrument instance image, at least 4 non-collinear semantic key points with serial numbers are labeled, including key scale position points on the instrument panel, pointer rotation center points and user-defined semantic information points. Among them, the same type of instrument instance image has the same serial number of the same semantic key points. For example, in the same type of instrument instance image, No. 1 point is the pointer rotation center point, No. 2 point is the zero scale point, and so on.

[0065] (1.4) Constructing a category template dataset, specifically: selecting a clear, complete, good view angle and light condition instrument instance image from each category of instrument instance image in the semantic matching dataset as the category template image, and labeling the pointer rotation center point and key scale position point of the category template image as the category template key point, and labeling the specific scale value information of each key scale position point.

[0066] (2) Constructing a target detection model and a semantic matching model, and training the target detection model using the target detection dataset and training the semantic matching model using the semantic matching dataset to obtain the trained target detection model and semantic matching model.

[0067] It should be understood that in order to identify the pointer instrument reading, it is necessary to construct a target detection model and a semantic matching model, and their functions and processes are as shown in Figure 3 . Among them, the task of the target detection model has two, one is to detect all pointer instrument instances in the pointer instrument image, including instance category and instance position; the second is to detect the pointer end position of each pointer instrument instance for subsequent identification steps. The task of the semantic matching model is to match the semantic corresponding points on the instrument instance image to be identified according to the labeled semantic key points with serial numbers, which generally include key scale position points on the instrument panel and pointer rotation center points, and of course, can also include user-defined semantic information points.

[0068] In this embodiment, the target detection model is constructed specifically as follows: YOLOX is selected as the basic structure of the target detection model, and the regression prediction head thereof is improved to construct the target detection model; the target detection model comprises a backbone network, a feature pyramid network (FPN), and a prediction layer, wherein the backbone network is used to obtain multi-scale feature maps of the pointer instrument image; the feature pyramid network is used to perform multi-scale feature fusion on the multi-scale feature maps to obtain a fused feature map with richer semantic information; the prediction layer comprises a classification prediction head and a regression prediction head, the classification prediction head is used to output class probabilities, and the regression prediction head is used to output predicted bounding box encoding coordinates, predicted bounding box foreground probabilities, and pointer tip position encoding coordinates.

[0069] It should be understood that the basic YOLOX model structure cannot predict the pointer tip position, and therefore the regression prediction head of the prediction layer needs to be improved to expand the output dimension of a single pointer instrument image from [h, w, 4] to [h, w, 6], wherein h and w represent the height and width of the input image (i.e., the pointer instrument image input into the target detection model). The pointer tip position decoding mode is consistent with the YOLOX target position decoding mode, and therefore the regression prediction head can output predicted bounding box encoding coordinates and pointer tip position encoding coordinates.

[0070] Further, the target detection model is trained using a target detection dataset, specifically as follows: after the pointer instrument images in the target detection dataset are subjected to a uniform scaling operation, they are input into the target detection model, and first obtain multi-scale feature maps through the backbone network. Then, the multi-scale feature maps are subjected to multi-scale feature fusion through the feature pyramid network to obtain a fused feature map with richer semantic information. Finally, the fused feature map enters the prediction layer, and the prediction results output by the prediction layer are obtained through the decoupled classification prediction head and regression prediction head; wherein the class probabilities are obtained through the classification prediction head, and the predicted bounding box encoding coordinates, predicted bounding box foreground probabilities, and predicted pointer tip position encoding coordinates are obtained through the regression prediction head. Then, the class probabilities, predicted bounding box encoding coordinates, predicted bounding box foreground probabilities, and predicted pointer tip position encoding coordinates output by the prediction layer are post-processed through post-processing operation methods such as non-maximum suppression (NMS) to obtain the predicted pointer instrument instance class, pointer instrument instance position in the form of bounding box coordinates, and pointer tip position coordinates in the form of point coordinates, as shown in Figure 3The binary cross-entropy loss is calculated according to the predicted pointer type instrument instance category and the corresponding real pointer type instrument instance category as the category discrimination loss function of the target detection model. The original bounding box loss function is improved, that is, the L2 loss is added to replace the original bounding box loss function. Specifically, the intersection over union loss is calculated according to the predicted pointer type instrument instance position in the form of bounding box coordinates and the corresponding real bounding box coordinates; the L2 loss is calculated according to the predicted pointer end position coordinates in the form of point coordinates and the corresponding real pointer end position coordinates. The total loss function of the target detection model is calculated according to the category discrimination loss function, the intersection over union loss and the L2 loss, and the parameters of the target detection model are adjusted to minimize the total loss function of the target detection model as the optimization goal until the maximum training round is reached, and the trained target detection model is obtained.

[0071] It should be noted that when the various types of probabilities, predicted bounding box encoding coordinates, predicted bounding box foreground probabilities and predicted pointer end position encoding coordinates output by the prediction layer are post-processed, the category corresponding to the maximum value is selected as the final predicted pointer type instrument instance category after the various types of probabilities are multiplied by the predicted bounding box foreground probabilities.

[0072] Further, the calculation formula of the total loss function of the target detection model is:

[0073] loss1 = -log(IoU(B bg ,B pred ))+||P bg -P pred ||2+L(y bg ,y pred )

[0074] Wherein, loss1 represents the total loss function of the target detection model; B bg represents the real bounding box, B pred represents the predicted pointer type instrument instance bounding box, -log(IoU(·)) represents the intersection over union loss, and IoU(·) represents the intersection over union of two bounding boxes; P bg represents the real pointer end position coordinates, P pred represents the predicted pointer end position coordinates, and ‖·‖2 represents the L2 loss (i.e. L2 norm); y bg represents the real category, y pred represents the predicted instance category, and L(·) represents the category discrimination loss function. The real bounding box coordinates and the real pointer end position coordinates in the above formula are the bounding box form coordinates and the pointer type instrument instance pointer end coordinates labeled when the target detection data set is constructed.

[0075] In this embodiment, the constructing the semantic matching model specifically comprises: selecting CHMNet (Convolutional Hough Matching Network) as the basic structure of the semantic matching model, the semantic matching model comprising a high-dimensional correlation calculation module, a convolutional Hough matching module, and a flow field and key point conversion module, wherein the high-dimensional correlation calculation module comprises a feature extractor, a multi-scale processing module, and a size adjustment layer; the convolutional Hough matching module comprises a first convolutional Hough matching layer, a max-pooling layer, and a second convolutional Hough matching layer; the flow field and key point conversion module comprises a flow field construction module and a key point conversion module, the flow field construction module being realized by using a Gaussian kernel soft-argmax operation, and the key point conversion module being realized by a weighted soft sampler.

[0076] Specifically, the to-be-recognized instrument instance image and the template image are input into the semantic matching model, and first pass through the feature extractor of the high-dimensional correlation calculation module to obtain the multi-scale feature maps corresponding to the two images; the multi-scale feature maps corresponding to the two images are subjected to correlation calculation by the multi-scale processing module to obtain a multi-scale feature correlation tensor; the multi-scale feature correlation tensor is subjected to interpolation operation by the size adjustment layer to adjust the feature correlation tensors of different scales to a unified spatial size. The multi-scale feature correlation tensor with a unified spatial size output by the high-dimensional correlation calculation module is sent to the convolutional Hough matching module, first passes through the first convolutional Hough matching layer to determine the candidate matches of each feature in the translation space and the scaling space, and obtains the first matching score tensor at each position of each scale feature map; the matching score tensors at each position of all scale feature maps retain the highest scoring candidate match at each position of each scale feature map by the max-pooling layer, and obtain the second matching score tensor at each position of each scale feature map; the second matching score tensors at each position of all scale feature maps are aggregated and refined by the second convolutional Hough matching layer to obtain the final matching score tensor corresponding to the two images. The final matching score tensor corresponding to the two images passes through the flow field construction module and the key point conversion module in turn, obtains a dense flow field by the Gaussian kernel soft-argmax operation of the flow field construction module, and represents the mapping relationship from each pixel in the template image to the corresponding pixel in the to-be-recognized instrument instance image; and then calculates the final matching result between each template key point in each template image and the semantic key point queried in the to-be-recognized instrument instance image, i.e., the semantic matching point corresponding to the semantic of the template key point, by the weighted soft sampler of the key point conversion module from the dense flow field, and further obtains the corresponding semantic matching point pair.

[0077] Further, the workflow of the high-dimensional correlation calculation module specifically comprises: the instrument instance image to be recognized and the template image first pass through a feature extractor, such as a pre-trained convolutional network ResNet-101 selected as the feature extractor of the high-dimensional correlation calculation module, to obtain the original feature maps corresponding to the two images, and the original feature maps are reduced by 3 times, each time reducing the length and width to 1 / 4 of the original and a 3x3 convolutional layer is used to obtain feature maps of three scales, that is, the multi-scale feature maps corresponding to the two images output by the feature extractor are obtained. Then the multi-scale feature maps corresponding to the two images pass through a multi-scale processing module to calculate the correlation coefficients between the features of all scales corresponding to the two images through the following dot product operation, so as to obtain the feature correlation tensors of all scales:

[0078]

[0079] wherein m and n represent the spatial scales corresponding to the two images (i.e. the instrument instance image to be recognized and the template image), x m and x′ n represent the position coordinates of the two images at the subscript corresponding spatial scales, C mn (x m ,x′ n ) represents the correlation coefficient of the features represented by x m and x′ n , F m (x m ) and F′ n (x′ n ) represent the features of the position coordinates of the feature maps of the two images at the subscript corresponding spatial scales, and RELU represents a linear rectifier activation function. Finally, all the feature correlation tensors of different scales are adjusted to a unified spatial size through interpolation operation by a size adjustment layer, that is, each group of 4-dimensional feature correlation tensors C ij is adjusted to a unified spatial size through interpolation operation, and together forms a 6-dimensional feature correlation tensor.

[0080] Further, the workflow of the convolutional Hough matching module specifically comprises: the multi-scale feature correlation tensor of uniform spatial size output by the high-dimensional correlation calculation module, i.e., the 6-dimensional feature correlation tensor, is first subjected to voting by a first convolutional Hough matching layer to determine the candidate matches of each feature in the translation space and the scaling space, and obtain a 6-dimensional match score tensor of each position on the feature map. The 6-dimensional match score tensor is subjected to a max-pooling layer to retain the highest scoring candidate match at each position and different scales, and obtain a 4-dimensional match score tensor. Finally, a second convolutional Hough matching layer is used to aggregate the 4-dimensional match score tensors of different scales, and further refined operation is performed on the result to obtain the final 4-dimensional match score tensor. In this embodiment, the first convolutional Hough matching layer is a 6-dimensional convolutional Hough matching layer, and the second convolutional Hough matching layer is a 4-dimensional convolutional Hough matching layer.

[0081] It should be understood that the second convolutional Hough matching layer can be used to process features of different scales and fuse the output results to aggregate visual information of different levels, and the second convolutional Hough matching layer is used to further refine the aggregation result to obtain a more accurate match result, i.e., the final 4-dimensional match score tensor.

[0082] Further, the workflow of the flow field and key point conversion module specifically comprises: the final 4-dimensional match score tensor output by the convolutional Hough matching module is first subjected to a flow field construction module to obtain a dense flow field by using a Gaussian kernel soft-argmax operation, which represents the mapping relationship from each pixel in the template image to the corresponding pixel in the to-be-recognized instrument instance image. The dense flow field output by the flow field construction module is then subjected to a key point conversion module to calculate the final match result between each template key point in each template image and the semantic key point to be queried in the to-be-recognized instrument instance image, i.e., the semantic matching point corresponding to the semantic of the template key point, by using a weighted soft sampler, and then obtain the corresponding semantic matching point pair. The calculation formula of the weight of the weighted soft sampler is:

[0083]

[0084] wherein k represents the semantic key point to be queried in the to-be-recognized instrument instance image, represents the weight at the image position (i, j) of the key point k, and τ is a distance threshold value which can be adjusted according to actual needs.

[0085] Further, the semantic matching model is trained using the semantic matching dataset, specifically including: taking any one of the instrument instance images in the semantic matching dataset as a to-be-identified instrument instance image, and taking another instrument instance image of the same type as the to-be-identified instrument instance image in the semantic matching dataset as a template image; inputting the to-be-identified instrument instance image and the template image into the semantic matching model to obtain predicted semantic key points matched with the template key points; calculating a loss function of the semantic matching model according to the predicted semantic key points and the corresponding real semantic key points, taking minimizing the loss function of the semantic matching model as an optimization objective, adjusting parameters of the semantic matching model until a maximum training round is reached, and obtaining the trained semantic matching model.

[0086] It should be noted that the two instance images are input into the semantic matching model, and the semantic matching model can query the semantic corresponding points of the semantic key points in one image in the other image.

[0087] Further, the loss function of the semantic matching model is calculated according to the following formula:

[0088]

[0089] Wherein, loss2 represents the loss function of the semantic matching model, M represents the number of pairs of semantic points marked in the semantic matching dataset, k' m represents the predicted semantic key point output by the semantic matching model for the mth query key point, k m represents the real semantic key point corresponding to the mth query point, which is specifically determined according to the semantic key points marked in the semantic matching dataset.

[0090] (3) inputting the to-be-identified pointer instrument image into the trained target detection model to obtain the instrument instance position, the instrument instance category, and the pointer end position.

[0091] (4) cutting the to-be-identified pointer instrument image into a to-be-identified instrument instance image according to the instrument instance position, selecting a corresponding category template image and category template key points in the category template dataset according to the instrument instance category, and inputting the to-be-identified instrument instance image and the category template image into the trained semantic matching model to obtain the semantic matching points of the category template key points in the to-be-identified instrument instance image.

[0092] (5) performing homography transformation on the semantic matching points of the category template key points in the to-be-identified instrument instance image, and calculating the instrument reading according to the category template key points, the specific scale value information of the category template key points, and the pointer end position.

[0093] It should be noted that the process involves obtaining instance information for each pointer instrument in the image to be identified through an object detection model, including the instrument instance location and category, as well as the pointer end position. Then, a semantic matching model is used to obtain the semantic matching point pairs for each instance regarding the template's custom key points. For example... Figure 4 As shown, the final reading result is obtained through the following steps.

[0094] (5.1) Based on the semantic matching point pairs, i.e., the category template key points and their semantic matching points in the image of the instrument instance to be identified, the homography matrix H of the image of the instrument instance to be identified relative to the category template image is calculated using the Random Sample Consensus (RANSAC) method. 3×3 :

[0095]

[0096] Here, (x,y) represents the category template keypoints in the category template image, and (x′,y′) represents the semantic matching points in the instrument instance image to be identified. Since the semantic matching dataset and the category template dataset are constructed such that each category template image has at least 4 non-collinear semantic keypoints, the equation can be solved uniquely.

[0097] (5.2) The coordinates of the pointer end position obtained by the target detection model are used with the homography matrix H obtained in step (5.1). 3×3 Calculate the coordinates of the pointer's end position after homography transformation.

[0098] Specifically, the homography matrix H obtained in step (5.1) is... 3×3 Multiplying the coordinates of the pointer end position obtained by the target detection model (i.e., the coordinates of the pointer end position obtained in step (3)) by the homography transformation, we can obtain the pointer end position coordinates after homography transformation.

[0099] (5.3) Based on the key points of the category template and the coordinates of the pointer's end position after homography transformation, the pointer vector P is obtained with the center point of template pointer rotation as the starting point and the coordinates of the pointer's end position after homography transformation as the ending point; the scale vector group S = {S1, S2, ..., S...} is obtained with the center point of template pointer rotation as the starting point and the key scale position points of the template as the ending point. i ,…,S n}

[0100] (5.4) Calculate the direction angle θ of the pointer vector P relative to each scale vector in the scale vector group S according to the following formula. i :

[0101]

[0102] In the above formula, θ i<0 indicates that the pointer vector P is at the i-th tick vector S i clockwise direction, θ i >0 indicates that the pointer vector P is at the i-th tick vector S i The counterclockwise direction, θ i =0 indicates that the pointer vector P and the i-th tick vector S are parallel. i Collinear.

[0103] (5.5) Based on the direction angle θ calculated in step (5.4) i The direction angle with the smallest absolute value among the negative direction angles is denoted as θ. s And record the scale value of the key point corresponding to its scale vector in the category template annotation information. s The direction angle with the smallest absolute value among the non-negative direction angles is selected and denoted as θ. t And record the scale value of the key point corresponding to its scale vector in the category template annotation information. t .

[0104] (5.6) Calculate the instrument reading according to the following formula:

[0105]

[0106] Here, result represents the instrument reading.

[0107] In summary, the identification method of this invention allows users to customize the template data of pointer instruments to a certain extent, making up for the shortcomings of existing methods in terms of scalability and flexibility for different instrument types; and it can correctly identify pointer instrument readings in more complex production environments, improving the robustness and versatility of the identification method of this invention.

[0108] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for reading a pointer type instrument based on template semantic key point matching, characterized in that, The method comprises the following steps: (1) collecting a pointer instrument image and classifying the image according to the appearance and range information of the instrument; Then, a target detection dataset, a semantic matching dataset and a category template dataset are constructed based on the pointer instrument image; (2) constructing a target detection model and a semantic matching model, training the target detection model using the target detection dataset, and training the semantic matching model using the semantic matching dataset to obtain the trained target detection model and semantic matching model; in the step (2), the semantic matching model is constructed and comprises: CHMNet is selected as the basic structure of the semantic matching model, the semantic matching model comprises a high-dimensional correlation calculation module, a convolutional Hough matching module and a flow field and key point conversion module, wherein the high-dimensional correlation calculation module comprises a feature extractor, a multi-scale processing module and a size adjustment layer; the convolutional Hough matching module comprises a first convolutional Hough matching layer, a maximum pooling layer and a second convolutional Hough matching layer; the flow field and key point conversion module comprises a flow field construction module and a key point conversion module, the flow field construction module is realized by using a Gaussian kernel soft-argmax operation, and the key point conversion module is realized by using a weighted soft sampler; The to-be-recognized instrument instance image and the template image are input into the semantic matching model, the feature extractor of the high-dimensional correlation calculation module is used to obtain the multi-scale feature maps corresponding to the two images; the multi-scale feature maps corresponding to the two images are subjected to correlation calculation through the multi-scale processing module to obtain a multi-scale feature correlation tensor; the multi-scale feature correlation tensor is subjected to interpolation operation through the size adjustment layer to adjust the feature correlation tensors of different scales to a unified spatial size; the multi-scale feature correlation tensors of the unified spatial size output by the high-dimensional correlation calculation module are input into the convolutional Hough matching module, the first convolutional Hough matching layer is used to perform voting to determine the candidate matches of each feature in the translation space and the scaling space, and a first matching score tensor at each position of each scale feature map is obtained; the matching score tensors at each position of all scale feature maps are subjected to maximum pooling through the maximum pooling layer to retain the candidate match with the highest score at each position of each scale feature map, and a second matching score tensor at each position of each scale feature map is obtained; the second matching score tensors at each position of all scale feature maps are aggregated and refined through the second convolutional Hough matching layer to obtain a final matching score tensor corresponding to the two images; the final matching score tensor corresponding to the two images is sequentially subjected to the flow field construction module and the key point conversion module, the Gaussian kernel soft-argmax operation of the flow field construction module is used to obtain a dense flow field representing the mapping relationship between each pixel in the template image and the corresponding pixel in the to-be-recognized instrument instance image; and the weighted soft sampler of the key point conversion module is used to calculate the final matching result between each template key point in each template image and the semantic key point to be queried in the to-be-recognized instrument instance image, i.e., the semantic matching point corresponding to the semantic of the template key point, from the dense flow field. (3) input the to-be-identified pointer instrument image into the trained target detection model to obtain instrument instance positions, categories, and pointer end positions; (4) cut the to-be-identified pointer instrument image into a to-be-identified instrument instance image according to the instrument instance positions, and select a corresponding category template image and category template key points from the category template dataset according to the instrument instance category; input the to-be-identified instrument instance image and the category template image into the trained semantic matching model to obtain semantic matching points of the category template key points in the to-be-identified instrument instance image; (5) perform homography transformation on the semantic matching points of the category template key points in the to-be-identified instrument instance image, and calculate an instrument reading according to the category template key points, specific scale value information of each key point, and the pointer end position; the step (5) comprises the following sub-steps: (5.1) According to the category template key points and the semantic matching points thereof in the instrument instance image to be recognized, a homography matrix of the instrument instance image to be recognized relative to the category template image is calculated using a random sample consensus method : ； wherein, represents a category template key point in the category template image, represents a semantic matching point in the instrument instance image to be recognized; (5.2) using the homography matrix obtained in step (5.1) on the coordinates of the pointer tip position obtained by the object detection model calculating the homography-transformed pointer tip position coordinates; (5.3) Based on the category template key point and the single transformation of the end of the pointer position coordinates, the end of the pointer position coordinates is obtained as the starting point of the template pointer rotation center point, and the end of the pointer position coordinates is obtained as the end of the template key scale position point ; the template pointer rotation center point is taken as the starting point, and the template key scale position point is taken as the end point to obtain the scale vector group ; (5.4) The pointer vector is calculated according to the following formula with respect to the set of scale vectors the direction angle of each scale vector : ； (5.5) Based on the orientation angle calculated in step (5.4) The direction angle with the smallest absolute value among the negative direction angles is denoted as . And record the corresponding tick values ​​of the key points of its tick vector in the category template annotation information. The direction angle with the smallest absolute value among the non-negative direction angles is denoted as . And record the corresponding tick values ​​of the key points of its tick vector in the category template annotation information. ; (5.6) calculate the instrument reading according to the following formula: ； wherein represents an instrument reading.

2. The method of claim 1, wherein, the step (1) comprises the following sub-steps: (1.1) collect pointer instrument images, and divide pointer instrument images with consistent appearances and ranges into the same type; (1.2) construct a target detection dataset, specifically: for the collected pointer instrument images, label each pointer instrument instance in the pointer instrument image in the form of a bounding box, and label the pointer end of each pointer instrument instance in the form of a point; (1.3) construct a semantic matching dataset, specifically: for the pointer instrument images labeled in the target detection dataset, cut each pointer instrument instance into a separate instrument instance image according to the labeled bounding box; for each instrument instance image, label at least four non-collinear semantic key points with serial numbers, including key scale position points on the instrument panel, pointer rotation center points, and user-defined semantic information points; wherein the serial numbers of the same semantic key points of instrument instance images of the same type are the same; (1.4) construct a category template dataset, specifically: select a clear, complete, and well-angled and well-lit instrument instance image from the instrument instance images of each category in the semantic matching dataset as a category template image, label the pointer rotation center points and key scale position points of the category template image as category template key points, and label specific scale value information for each key scale position point.

3. The method of claim 1, wherein, In the step (2), the target detection model is constructed by selecting YOLOX as the basic structure of the target detection model and improving the regression prediction head, and the target detection model comprises a backbone network, a feature pyramid network, and a prediction layer, wherein the backbone network is used to obtain multi-scale feature maps of the pointer instrument image; the feature pyramid network is used to perform multi-scale feature fusion on the multi-scale feature maps to obtain a fused feature map; and the prediction layer comprises a classification prediction head and a regression prediction head, wherein the classification prediction head is used to output category probabilities, and the regression prediction head is used to output predicted bounding box encoding coordinates, predicted bounding box foreground probabilities, and pointer end position encoding coordinates. In the step (2), the target detection dataset is used to train the target detection model, specifically comprising:

4. The method of claim 1, wherein, ​ The pointer instrument image in the target detection dataset is input into the target detection model after uniform scaling, and the output includes category probability, predicted bounding box encoded coordinates, predicted bounding box foreground probability and predicted pointer end position encoded coordinates; the category probability, predicted bounding box encoded coordinates, predicted bounding box foreground probability and predicted pointer end position encoded coordinates output by the prediction layer are post-processed to obtain the predicted pointer instrument instance category, pointer instrument instance position in the form of bounding box coordinates and pointer end position coordinates in the form of point coordinates; the binary cross-entropy loss is calculated according to the predicted pointer instrument instance category and the corresponding real pointer instrument instance category as the category discrimination loss function of the target detection model; the intersection over union loss is calculated according to the predicted pointer instrument instance position in the form of bounding box coordinates and the corresponding real bounding box coordinates; the L2 loss is calculated according to the predicted pointer end position coordinates in the form of point coordinates and the corresponding real pointer end position coordinates; the total loss function of the target detection model is calculated according to the category discrimination loss function, the intersection over union loss and the L2 loss, and the total loss function of the target detection model is minimized as the optimization goal to adjust the parameters of the target detection model until the maximum training round is reached, and the trained target detection model is obtained.

5. The method of claim 4, wherein, The calculation formula of the total loss function of the target detection model is: ； wherein, represents a total loss function of the target detection model; represents a real bounding box, represents a predicted pointer gauge instance bounding box, represents an intersection over union loss, represents an intersection over union of two bounding boxes; represents a real pointer end position coordinate, represents a predicted pointer end position coordinate, represents an L2 loss; represents a real class, represents a predicted instance class, represents a class discrimination loss function.

6. The method of reading a pointer type instrument based on template semantic key point matching according to claim 1, characterized in that, In the step (2), the semantic matching model is trained using the semantic matching dataset, specifically including: Any one instrument instance image in the semantic matching dataset is taken as a to-be-identified instrument instance image, and another instrument instance image of the same type in the semantic matching dataset is taken as a template image; the to-be-identified instrument instance image and the template image are input into the semantic matching model to obtain predicted semantic key points matched with the template key points; the loss function of the semantic matching model is calculated according to the predicted semantic key points and the corresponding real semantic key points, and the loss function of the semantic matching model is minimized as the optimization goal to adjust the parameters of the semantic matching model until the maximum training round is reached, and the trained semantic matching model is obtained.

7. The method of claim 6, wherein, The calculation formula of the loss function of the semantic matching model is: ； wherein, a loss function representing a semantic matching model, a number of marked semantic point pairs in a semantic matching data set, a predicted semantic key point output by the semantic matching model for the key point of the query, a true semantic key point corresponding to the query point.

Images