Template matching method based on twin network and accurate center point positioning

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a template matching method based on a twin network and accurate center point positioning. Comprising the following steps: S1, acquiring a training data set, namely a template and a search image; s2, a template matching network based on a twin network and accurate center point positioning is constructed, the training data set in the S1 is input into the template matching network, and the template matching network is composed of a twin feature extraction network, a dynamic contraction cross-correlation network and a center point positioning network which are sequentially cascaded; and S3, training the template matching network in the S2 by adopting a loss function. According to the method, the template matching task serves as a center point positioning problem to be processed, the problem that the difference exists between the template and the search image can be better solved, and the positioning precision of the target center point and the robustness of template matching are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a template matching method based on a twin network and precise center point positioning. Background Art

[0002] Template matching is an important technology in computer vision and is widely used in target detection, face recognition, video tracking, similarity matching, and medical imaging. It uses the target template image as prior knowledge, uses the local features of the image to match and locate the position of the target in the search image. Due to differences in shooting time, angle, and imaging equipment, there are often grayscale differences (even from different sources), scale differences, rotation differences, and perspective differences between the template image and the reference image, which bring great challenges to the template matching task.

[0003] Traditional template matching algorithms mainly perform similarity matching based on pixel-level information of corresponding regions. These algorithms perform well under single scene conditions. For example, the grayscale correlation method requires that the images involved in the matching must have similar grayscale characteristics, which is suitable for homologous image matching tasks. The gradient correlation method has better adaptability to grayscale distortion. The mutual information method is suitable for heterogeneous image matching. However, these algorithms are difficult to accurately locate in practical applications because the matching task between the template image and the search image usually involves multiple application scenarios, such as low imaging resolution, large changes in target scale and shape, complex scene clutter, and usually accompanied by more occlusion and interference. In order to solve the difficulty of applying the algorithm in complex scenes, many improved methods based on template matching algorithms have emerged in recent years, such as methods based on local features such as SIFT, SURF, and FAST, methods based on global features such as HOG, LBP, and Haar, and methods based on new similarity metrics. Although these methods have improved the matching accuracy in specific scenes, they still have great limitations. The robustness of template matching is still greatly affected by problems such as heterogeneity and scale differences, and it is easy to fail to match the target.

[0004] In some practical tasks, the prediction accuracy of the target center is more critical. Anchor-based template matching methods locate the target by predicting bounding boxes, and bounding boxes with the same confidence have different prediction accuracy for the target center. These methods indirectly predict the target center point through anchor boxes, with low accuracy. In addition, anchor-based methods have the disadvantages of requiring a large number of anchor boxes, imbalanced positive and negative samples, and manually crafted anchor boxes, which not only increase the complexity of processing, but also reduce the robustness and accuracy of the model. Center-based anchor-free methods directly predict the target center, but these methods cannot effectively utilize the prior information in the anchor box, resulting in inaccurate positioning. Summary of the invention

[0005] In response to the problems existing in existing template matching methods, the present invention provides a template matching method based on a twin network and precise center point positioning. This method treats the template matching task as a center point positioning task, which can better solve the problem of differences between the template and the search image, effectively improve the accuracy of target center point positioning, and enhance the robustness of template matching in complex scenes.

[0006] In order to achieve the above technical effects, the technical solutions adopted by the present invention are as follows:

[0007] A template matching method based on a twin network and precise center point positioning, comprising:

[0008] S1: Get the training dataset, i.e., the template and the search image;

[0009] Construct a template matching network based on the twin network and precise center point positioning, and input the training data set of S1 into the template matching network;

[0010] S3: Use the loss function to train the template matching network of S2.

[0011] Preferably, S2 comprises:

[0012] S2.1: Build a twin feature extraction network, input the template and search image, and output the corresponding feature map;

[0013] S2.2: Build a dynamic shrinkage cross-correlation network to perform cross-correlation fusion on the feature maps output by S2.1;

[0014] S2.3: Construct a center point positioning network to predict the center point information of the template in the search image based on the output of the dynamic shrinkage cross-correlation network in S2.2.

[0015] Preferably, the twin feature extraction network of S2.1 is composed of two convolutional neural networks with shared parameters and the same structure, which take the template T and the search image S as inputs and output the corresponding feature maps. and

[0016] Preferably, the convolutional neural network of the twin feature extraction network is based on the standard ResNet50, and is limited as follows:

[0017] (1) The standard ResNet50 consists of 5 convolutional groups and fully connected layers, with the last fully connected layer removed;

[0018] (2) Add a 3×3 deconvolution layer with a stride of 1 and 128 channels after the fifth convolution group, and two 3×3 deconvolution layers with a stride of 2 and 128 channels;

[0019] (3) A feature map fusion module is added after each deconvolution layer between the feature map and the feature map of the same scale corresponding to the standard ResNet50.

[0020] Preferably, the dynamic shrinkage cross-correlation network of S2.2 is composed of a cascade of a depth-wise convolution module and a dynamic shrinkage attention module;

[0021] The depth-wise convolution module uses the template feature map as the convolution kernel and performs depth-wise convolution with the search image feature map to obtain a preliminary fusion result;

[0022] The dynamic shrinkage attention module is used to improve the attention to high-response channels based on the preliminary fusion results, and remove some feature areas in the feature map that are similar to the target to obtain the final fusion result.

[0023] Preferably, the center point positioning network of S2.3 is composed of a key point positioning module and an offset prediction module, wherein the key point positioning module is composed of a center point positioning branch and a corner point positioning branch, and the offset prediction module is composed of a center point offset prediction branch, and both use the feature map after cross-correlation fusion as input.

[0024] Preferably, the center point positioning branch is composed of three 3×3 deconvolution layers and one 3×3 convolution layer, and the final output channel number is 1 center point heat map Indicates the probability that each position is predicted as the target center point, which is used to roughly locate the target center point;

[0025] The corner point localization branch consists of three 3×3 deconvolution layers and one 3×3 convolution layer, and the final output is a corner point heat map with 4 channels. Indicates the probability that each position is predicted to be the corresponding corner point of the target, which is used to accurately locate the center point of the target based on rough positioning;

[0026] The center point offset prediction branch consists of three 3×3 deconvolution layers and one 3×3 convolution layer, and the final output channel number is 2 center point offset prediction map Represents the offset error of each predicted position, which is used to make the final correction to the target center point based on precise positioning.

[0027] Preferably, the loss function of S3 is:

[0028] L=λ1L cen +λ2L cor +λ3L off

[0029] Among them, L cen is the center point positioning loss, L cor is the corner location loss, L offis the center point offset prediction loss, λ1, λ2 and λ3 are weight hyperparameters, taking λ1=1, λ2=1, λ3=1

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. The template matching solution based on the twin network and precise center point positioning provided by the present invention processes the template matching task as a center point positioning task, which can better solve the problem of differences between the template and the search image, and effectively improve the positioning accuracy of the target center point and the robustness of template matching;

[0032] 2. This paper adds a deconvolution layer and feature map fusion operation after the standard ResNet50 network, and proposes a new feature extraction network that can extract multi-scale features of the input template and the search image, effectively improving the ability to resolve scale differences;

[0033] 3. The present invention combines the depth-wise convolution module with the proposed dynamic shrinkage attention module and proposes a new cross-correlation operation for feature fusion, which can effectively improve the accuracy of center point positioning;

[0034] 4. The present invention represents the target corresponding to the template on the search image with its center point, completely abandons the prediction of the target border, and uses the target corner point information to assist the center point in locating the target, which can effectively improve the positioning accuracy of the target center point. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0036] In the attached picture:

[0037] Figure 1 Schematic diagram of the network structure of the template matching method;

[0038] Figure 2 Schematic diagram of the twin feature extraction network structure used in the present invention;

[0039] Figure 3 Schematic diagram of the structure of the dynamic shrinking attention module used in the present invention;

[0040] Figure 4 This is a schematic diagram of output optimization during the center point positioning network test used in the present invention.

[0041] Figure 5 Some matching results are obtained using the method provided by the present invention. DETAILED DESCRIPTION

[0042] The following is combined with Figure 1-Figure 5 The preferred embodiments of the present invention are described. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0043] Example:

[0044] A template matching method based on a twin network and precise center point positioning, comprising:

[0045] S1: Get the training dataset, namely the template T and the search image S.

[0046] S2: Figure 1 , build a template matching network based on the twin network and precise center point positioning, and input the training data set of S1 into the template matching network. Including:

[0047] S2.1: Construct a twin feature extraction network, input the template and search image, and output the corresponding feature map. The twin feature extraction network is composed of two convolutional neural networks with shared parameters and the same structure, which take the template T and search image S as input and output the corresponding feature maps. and

[0048] The convolutional neural network of the twin feature extraction network is based on the standard ResNet50, with the following restrictions:

[0049] (1) The standard ResNet50 consists of 5 convolutional groups and fully connected layers, with the last fully connected layer removed;

[0050] (2) Add a 3×3 deconvolution layer with a stride of 1 and 128 channels after the fifth convolution group, and two 3×3 deconvolution layers with a stride of 2 and 128 channels;

[0051] (3) A feature map fusion module is added after each deconvolution layer between the feature map and the feature map of the same scale corresponding to the standard ResNet50.

[0052] S2.2: Build a dynamic shrinkage cross-correlation network and perform cross-correlation fusion on the feature maps output by S2.1.

[0053] The dynamic shrinkage cross-correlation network is composed of a cascade of depth-wise convolutional modules and dynamic shrinkage attention modules;

[0054] The depth-wise convolution module uses the template feature map as the convolution kernel and performs depth-wise convolution with the search image feature map to obtain a preliminary fusion result;

[0055] The dynamic shrinkage attention module is used to improve the attention to high-response channels based on the preliminary fusion results, and remove some feature areas in the feature map that are similar to the target to obtain the final fusion result.

[0056] The dynamic shrinking attention module is modified based on the parameter-free attention mechanism. The specific modification is: adding a soft thresholding operation after the parameter-free attention mechanism, and the two are cascaded to form the dynamic shrinking attention module. The detailed structure of the dynamic shrinking attention module used is as follows: Figure 3 shown.

[0057] S2.3: Construct a center point positioning network to predict the center point information of the template in the search image based on the output of the dynamic shrinkage cross-correlation network in S2.2.

[0058] The center point localization network consists of a key point localization module and an offset prediction module. The key point localization module consists of a center point localization branch and a corner point localization branch, and the offset prediction module consists of a center point offset prediction branch. Both use the feature map after cross-correlation fusion as input.

[0059] The center point positioning branch consists of three 3×3 deconvolution layers and one 3×3 convolution layer, and the final output is a center point heat map with 1 channel. Indicates the probability that each position is predicted as the target center point, which is used to roughly locate the target center point;

[0060] The corner point localization branch consists of three 3×3 deconvolution layers and one 3×3 convolution layer, and the final output is a corner point heat map with 4 channels. Indicates the probability that each position is predicted to be the corresponding corner point of the target, which is used to accurately locate the center point of the target based on rough positioning;

[0061] The center point offset prediction branch consists of three 3×3 deconvolution layers and one 3×3 convolution layer, and the final output channel number is 2 center point offset prediction map Represents the offset error of each predicted position, which is used to make the final correction to the target center point based on precise positioning.

[0062] S3: Use the loss function to train the template matching network of S2.

[0063] In this example, template-search image pairs extracted from four datasets, ImageNet DET, ImageNet VID, COCO, and GOT-10K, are used as training data. The sizes of template and search images are 127x127 and 255x255 pixels, respectively. The entire network is trained for 20 rounds on four Titan XP GPUs using stochastic gradient descent (SGD). In the first 5 rounds, the learning rate is increased from 0.001 to 0.005 at equal intervals, and in the following 15 rounds, the learning rate is exponentially decayed from 0.005 to 0.0005. We freeze the parameters of the backbone network in the first 10 rounds, and in the remaining 10 rounds, we train the entire network in an end-to-end manner. Weight decay and momentum are set to 0.0001 and 0.9, respectively.

[0064] The loss function used in training is:

[0065] L=λ1L cen +λ2L cor +λ3L off

[0066] Among them, L cen is the center point positioning loss, L cor is the corner location loss, L off is the center point offset prediction loss, λ1, λ2 and λ3 are weight hyperparameters, taking λ1=1, λ2=1, λ3=1.

[0067]

[0068] Among them, p hm Represents the predicted center point heat map, represents the true label of the center point heat map, p x,y Indicates the probability that the (x, y) position is the target center point, represents the true label, α and β are weight hyperparameters, α=2, β=4. If the point is mapped back to the input search image and is the center point of the target, then is 1, otherwise

[0069]

[0070] where p hm represents the predicted corner point heat map, represents the true label of the corner heat map, p x,y Indicates the probability that the (x, y) position is the corresponding corner point of the target, represents the true label, α and β are weight hyperparameters, α=2, β=4. If the point is mapped back to the input search image and is the corner point corresponding to the target, then is 1, otherwise

[0071]

[0072] Where N represents the number of training samples, SmoothL1Loss represents the L1 loss, p represents the predicted pixel error due to the network step size, represents the true label, is calculated as the difference between the coordinates of the point in the output center point heat map and the coordinates of the point after it is mapped back to the input search image.

[0073] During testing, the template-search image pair is still used as the input of the model, but the image size does not need to be fixed. The output of the model is a set of center point heat maps, corner point heat maps, center point offsets and their corresponding confidence scores. The center point with the highest confidence score is selected as the preliminary positioning result, and then the four corner points with the highest confidence scores are selected to adjust the preliminary positioning result. Finally, the positioning result is corrected by its corresponding center point offset to obtain the precise position of the center point of the template in the search image. The adjustment strategy for the preliminary positioning result of the corner point pair is as follows:

[0074] p opt =(1-ω)p corner +ωp center

[0075] Among them, p opt represents the adjusted center point positioning result, p corner represents the center point position obtained by the four corner points, p center represents the initial positioning result of the center point, ω is the weight hyperparameter, and ω=0.9 is taken.

[0076] Figure 4 What is given is a schematic diagram of the output optimization of the center point positioning network during testing.

[0077] Test results:

[0078] Figure 5 Some matching results obtained by using the method provided by the present invention are given.

[0079] Table 1 shows the comparison results of the method provided by the present invention and some existing template matching methods on the BBS105 dataset and the self-built Hard350 dataset, where Ours represents the method provided by the present invention. Since our method completely abandons the prediction of the target bounding box and only predicts the target center point, three evaluation indicators MCE, SR5 and SR10 are used in the experiment. The smaller the MCE data and the larger the SR5 and SR10 data, the better the performance of the template matching method used.

[0080] Table 1 Test results of different methods on BBS dataset and Hard350 dataset

[0081]

[0082] The center error (CE) is used as the evaluation metric for a single sample. MCE is defined as the average center error of all image pairs. SR5 and SR10 are defined as the ratio of successfully recognized samples to the total number of test samples, where recognition is considered successful when the center error is less than 5 and 10, respectively. CE is given by the following formula:

[0083]

[0084] Among them, (x p ,y p ) and (x g ,y g ) are the predicted center point coordinates and the true center point coordinates respectively.

[0085] It can be seen from the experimental results that the method provided by the present invention has better performance than other template matching methods.

[0086] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A template matching method based on a twin network and precise center point positioning, characterized in that: include: S1: Get the training dataset, i.e., the template and the search image; S2: Build a template matching network based on the twin network and precise center point positioning, and input the training data set of S1 into the template matching network; S3: Use the loss function to train the template matching network of S2.

2. According to claim 1, a template matching method based on a twin network and precise center point positioning is characterized in that: S2 includes: S2.1: Build a twin feature extraction network, input the template and search image, and output the corresponding feature map; S2.2: Build a dynamic shrinkage cross-correlation network to perform cross-correlation fusion on the feature maps output by S2.1; S2.3: Construct a center point positioning network to predict the center point information of the template in the search image based on the output of the dynamic shrinkage cross-correlation network in S2.

2.

3. The template matching method based on twin network and precise center point positioning according to claim 2 is characterized in that: The twin feature extraction network of S2.1 is composed of two convolutional neural networks with the same structure and shared parameters, taking the template T and the search image S as input, and outputting the corresponding feature maps and 4. The template matching method based on twin network and precise center point positioning according to claim 3 is characterized in that: The convolutional neural network of the twin feature extraction network is based on the standard ResNet50, with the following restrictions: (1) The standard ResNet50 consists of 5 convolutional groups and fully connected layers, with the last fully connected layer removed; (2) Add a 3×3 deconvolution layer with a stride of 1 and 128 channels after the fifth convolution group, and two 3×3 deconvolution layers with a stride of 2 and 128 channels; (3) A feature map fusion module is added after each deconvolution layer between the feature map and the feature map of the same scale corresponding to the standard ResNet50.

5. The template matching method based on twin network and precise center point positioning according to claim 4, characterized in that: The dynamic shrinkage cross-correlation network of S2.2 is composed of a cascade of depth-wise convolutional modules and dynamic shrinkage attention modules; The depth-wise convolution module uses the template feature map as the convolution kernel and performs depth-wise convolution with the search image feature map to obtain a preliminary fusion result; The dynamic shrinkage attention module is used to improve the attention to high-response channels based on the preliminary fusion results, and remove some feature areas in the feature map that are similar to the target to obtain the final fusion result.

6. The template matching method based on twin network and precise center point positioning according to claim 5, characterized in that: The center point localization network of S2.3 consists of a key point localization module and an offset prediction module. The key point localization module consists of a center point localization branch and a corner point localization branch, and the offset prediction module consists of a center point offset prediction branch. Both use the feature map after cross-correlation fusion as input.

7. The template matching method based on twin network and precise center point positioning according to claim 6, characterized in that: The center point positioning branch consists of three 3×3 deconvolution layers and one 3×3 convolution layer, and the final output is a center point heat map with 1 channel. Indicates the probability that each position is predicted as the target center point, which is used to roughly locate the target center point; The corner point localization branch consists of three 3×3 deconvolution layers and one 3×3 convolution layer, and the final output is a corner point heat map with 4 channels. Indicates the probability that each position is predicted to be the corresponding corner point of the target, which is used to accurately locate the center point of the target based on rough positioning; The center point offset prediction branch consists of three 3×3 deconvolution layers and one 3×3 convolution layer, and the final output channel number is 2 center point offset prediction map Represents the offset error of each predicted position, which is used to make the final correction to the target center point based on precise positioning.

8. The template matching method based on twin network and precise center point positioning according to claim 7, characterized in that: The loss function of S3 is: L=λ1L cen +λ2L cor +λ3L off Among them, L cen is the center point positioning loss, L cor is the corner location loss, L off is the center point offset prediction loss, λ1, λ2 and λ3 are weight hyperparameters, taking λ1=1, λ2=1, λ3=1.

Images