A template matching method based on a twin network and accurate center point positioning

Abstract

The present application belongs to the technical field of image processing, and particularly relates to a template matching method based on a twin network and accurate center point positioning. The method comprises the following steps: S1, obtaining a training data set, i.e. a template and a search image; S2, constructing a template matching network based on a twin network and accurate center point positioning, and inputting the training data set of S1 into the template matching network, wherein the template matching network is composed of three parts, i.e. a twin feature extraction network, a dynamic shrink cross-correlation network and a center point positioning network, which are sequentially cascaded; and S3, training the template matching network of S2 by using a loss function. The method can better solve the problem of differences between the template and the search image, effectively improve the positioning accuracy of the target center point and the robustness of the template matching.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a template matching method based on Siamese networks and precise center point localization. Background Technology

[0002] Template matching is an important technique in computer vision, widely used in object detection, face recognition, video tracking, similarity matching, and medical imaging. It uses a target template image as prior knowledge, leveraging local features of the image to match and locate the target within the search image. However, due to differences in shooting time, angle, and imaging equipment, template and reference images often exhibit differences in grayscale (even heterogeneous sources), scale, rotation, and viewpoint. These differences pose significant challenges to template matching tasks.

[0003] Traditional template matching algorithms primarily rely on pixel-level information within corresponding regions for similarity matching. These algorithms perform well under single-scene conditions; for example, gray-level correlation methods require images participating in the matching to have similar gray-level characteristics, making them suitable for matching images from the same source. Gradient correlation methods are more adaptable to gray-level distortion. Mutual information methods are suitable for matching images from different sources. However, these algorithms struggle to accurately locate specific targets in practical applications because matching template and search images typically involves multiple scenarios, such as low imaging resolution, large variations in target scale and shape, complex scene clutter, and frequent occlusion and interference. To address the difficulties in applying these algorithms in complex scenes, many improved template matching methods have emerged in recent years, including methods based on local features such as SIFT, SURF, and FAST, methods based on global features such as HOG, LBP, and Haar, and methods based on new similarity metrics. While these methods improve matching accuracy in specific scenarios, they still have significant limitations. The robustness of template matching remains greatly affected by issues such as different sources and scale differences, making target matching prone to failure.

[0004] In some practical tasks, the accuracy of target center prediction is more critical. Anchor-box-based template matching methods locate targets by predicting bounding boxes, but bounding boxes with the same confidence level have varying accuracy in predicting the target center. These methods indirectly predict the target center point through anchor boxes, resulting in lower accuracy. Furthermore, anchor-box-based methods suffer from drawbacks such as requiring a large number of anchor boxes, imbalanced positive and negative samples, and the need for manually crafted anchor boxes. This not only increases processing complexity but also reduces the model's robustness and accuracy. Anchor-box-less methods based on center points directly predict the target center, but these methods cannot effectively utilize prior information within the anchor boxes, leading to inaccurate localization. Summary of the Invention

[0005] To address the problems of existing template matching methods, this invention provides a template matching method based on Siamese networks and precise center point localization. This method treats the template matching task as a center point localization task, which can better solve the problem of differences between the template and the search image, effectively improve the accuracy of target center point localization, and enhance the robustness of template matching in complex scenarios.

[0006] To achieve the above-mentioned technical effects, the technical solution adopted by the present invention is as follows:

[0007] A template matching method based on Siamese networks and precise center point localization includes:

[0008] S1: Obtain the training dataset, i.e., the template and the search image;

[0009] Construct a template matching network based on Siamese network and precise center point localization, and input the training dataset of S1 into the template matching network;

[0010] S3: Train the template matching network of S2 using the loss function.

[0011] Preferably, S2 includes:

[0012] S2.1: Construct a twin feature extraction network, input template and search image, and output corresponding feature maps;

[0013] S2.2: Construct a dynamically shrinking cross-correlation network to perform cross-correlation fusion on the feature maps output from S2.1;

[0014] S2.3: Construct a center point localization network to predict the center point information of the template in the search image based on the output of the dynamically shrinking cross-correlation network in S2.2.

[0015] Preferably, the twin feature extraction network of S2.1 consists of two concatenated convolutional neural networks with shared parameters and identical structures, taking the template T and the search image S as inputs respectively, and outputting the corresponding feature maps respectively. and

[0016] Preferably, the convolutional neural network for the Siamese feature extraction network is based on the standard ResNet50, with the following limitations:

[0017] (1) The standard ResNet50 consists of 5 convolutional groups and fully connected layers. Remove the last fully connected layer.

[0018] (2) After the 5th convolutional group, add a 3×3 deconvolutional layer with a stride of 1 and 128 channels, and two 3×3 deconvolutional layers with a stride of 2 and 128 channels.

[0019] (3) Add a feature map fusion module after each deconvolution layer between the feature map and the feature map of the same scale corresponding to the standard ResNet50.

[0020] Preferably, the dynamically shrinking cross-correlation network of S2.2 is composed of a depthwise convolutional module and a dynamically shrinking attention module cascaded together;

[0021] The depthwise convolution module uses the template feature map as the convolution kernel and performs depthwise convolution with the search image feature map to obtain preliminary fusion results;

[0022] The dynamic shrinking attention module is used to increase attention to high-response channels based on the initial fusion results and remove some feature regions in the feature map that are similar to the target, so as to obtain the final fusion result.

[0023] Preferably, the center point localization network of S2.3 consists of a key point localization module and an offset prediction module. The key point localization module consists of a center point localization branch and a corner point localization branch, and the offset prediction module consists of a center point offset prediction branch. Both use the cross-correlation fused feature map as input.

[0024] Preferably, the center point localization branch consists of three 3×3 deconvolutional layers and one 3×3 convolutional layer, ultimately outputting a center point heatmap with one channel. This represents the probability that each location is predicted to be the target center point, and is used to roughly locate the target center point;

[0025] The corner localization branch consists of three 3×3 deconvolutional layers and one 3×3 convolutional layer, ultimately outputting a corner heatmap with 4 channels. This indicates the probability that each position is predicted to be the corresponding corner point of the target, and is used to accurately locate the center point of the target based on the rough positioning.

[0026] The center point offset prediction branch consists of three 3×3 deconvolutional layers and one 3×3 convolutional layer, ultimately outputting a center point offset prediction map with 2 channels. This represents the offset error of each predicted position, used for final correction of the target center point based on precise positioning.

[0027] Preferably, the loss function for S3 is:

[0028] L=λ1L cen +λ2L cor +λ3L off

[0029] Among them, L cen It is the center point localization loss, L cor It is the corner point localization loss, L offThis is the center point offset prediction loss, where λ1, λ2, and λ3 are weight hyperparameters, taken as λ1 = 1, λ2 = 1, and λ3 = 1.

[0030] Compared with the prior art, the beneficial effects of the present invention are:

[0031] 1. The template matching scheme based on Siamese network and precise center point localization provided by the present invention treats the template matching task as a center point localization task, which can better solve the problem of differences between the template and the search image, and effectively improve the localization accuracy of the target center point and the robustness of template matching;

[0032] 2. This invention adds a deconvolution layer and feature map fusion operation after the standard ResNet50 network, proposing a new feature extraction network that can extract multi-scale features from the input template and search image, effectively improving the ability to solve scale differences;

[0033] 3. This invention combines the depthwise convolution module with the proposed dynamic shrinking attention module, and proposes a new cross-correlation operation for feature fusion, which can effectively improve the accuracy of center point localization;

[0034] 4. This invention represents the target corresponding to the template on the search map using its center point, completely eliminating the prediction of the target's border. It uses the target's corner point information to assist the center point in locating the target, which can effectively improve the positioning accuracy of the target's center point. Attached Figure Description

[0035] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0036] In the attached diagram:

[0037] Figure 1 This is a schematic diagram of the network structure for the template matching method;

[0038] Figure 2 This is a schematic diagram of the twin feature extraction network structure used in this invention;

[0039] Figure 3 This is a schematic diagram of the dynamic contraction attention module structure used in this invention;

[0040] Figure 4 This is a schematic diagram illustrating the output optimization during testing of the center point localization network used in this invention.

[0041] Figure 5 To obtain some matching results using the method provided by this invention. Detailed Implementation

[0042] The following is in conjunction with the appendix Figures 1-5 The preferred embodiments of the present invention will be described herein. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0043] Example:

[0044] A template matching method based on Siamese networks and precise center point localization includes:

[0045] S1: Obtain the training dataset, i.e., template T and search image S.

[0046] S2: As Figure 1 A template matching network based on Siamese networks and precise center point localization is constructed, and the training dataset of S1 is input into the template matching network. This includes:

[0047] S2.1: Construct a Siamese feature extraction network, taking a template and a search image as input, and outputting the corresponding feature maps. The Siamese feature extraction network consists of two cascaded convolutional neural networks with shared parameters and identical structures. The template T and the search image S are taken as inputs, and the outputs are the corresponding feature maps. and

[0048] The convolutional neural network for the Siamese feature extraction network is based on the standard ResNet50, with the following limitations:

[0049] (1) The standard ResNet50 consists of 5 convolutional groups and fully connected layers. Remove the last fully connected layer.

[0050] (2) After the 5th convolutional group, add a 3×3 deconvolutional layer with a stride of 1 and 128 channels, and two 3×3 deconvolutional layers with a stride of 2 and 128 channels.

[0051] (3) Add a feature map fusion module after each deconvolution layer between the feature map and the feature map of the same scale corresponding to the standard ResNet50.

[0052] S2.2: Construct a dynamically shrinking cross-correlation network to perform cross-correlation fusion on the feature maps output from S2.1.

[0053] The dynamically shrinking cross-correlation network is composed of a depthwise convolutional module and a dynamically shrinking attention module cascaded together.

[0054] The depthwise convolution module uses the template feature map as the convolution kernel and performs depthwise convolution with the search image feature map to obtain preliminary fusion results;

[0055] The dynamic shrinking attention module is used to increase attention to high-response channels based on the initial fusion results and remove some feature regions in the feature map that are similar to the target, so as to obtain the final fusion result.

[0056] The dynamic shrinking attention module is derived from the parameterless attention mechanism. Specifically, a soft thresholding operation is added after the parameterless attention mechanism, and the two are cascaded to form the dynamic shrinking attention module. The detailed structure of the dynamic shrinking attention module is as follows: Figure 3 As shown.

[0057] S2.3: Construct a center point localization network to predict the center point information of the template in the search image based on the output of the dynamically shrinking cross-correlation network in S2.2.

[0058] The center point localization network consists of a key point localization module and an offset prediction module. The key point localization module consists of a center point localization branch and a corner point localization branch, and the offset prediction module consists of a center point offset prediction branch. Both use the cross-correlation fused feature map as input.

[0059] The center point localization branch consists of three 3×3 deconvolutional layers and one 3×3 convolutional layer, ultimately outputting a center point heatmap with 1 channel. This represents the probability that each location is predicted to be the target center point, and is used to roughly locate the target center point;

[0060] The corner localization branch consists of three 3×3 deconvolutional layers and one 3×3 convolutional layer, ultimately outputting a corner heatmap with 4 channels. This indicates the probability that each position is predicted to be the corresponding corner point of the target, and is used to accurately locate the center point of the target based on the rough positioning.

[0061] The center point offset prediction branch consists of three 3×3 deconvolutional layers and one 3×3 convolutional layer, ultimately outputting a center point offset prediction map with 2 channels. This represents the offset error of each predicted position, used for final correction of the target center point based on precise positioning.

[0062] S3: Train the template matching network of S2 using the loss function.

[0063] This example uses template-search image pairs extracted from four datasets: ImageNet DET, ImageNet VID, COCO, and GOT-10K, as training data. The template and search image sizes are 127x127 and 255x255 pixels, respectively. The entire network is trained for 20 epochs on four Titan XP GPUs using stochastic gradient descent (SGD). In the first 5 epochs, the learning rate increases at equal intervals from 0.001 to 0.005, while in the subsequent 15 epochs, the learning rate decays exponentially from 0.005 to 0.0005. We freeze the parameters of the backbone network in the first 10 epochs, and train the entire network end-to-end in the remaining 10 epochs. Weight decay and momentum are set to 0.0001 and 0.9, respectively.

[0064] The loss function used for training is:

[0065] L=λ1L cen +λ2L cor +λ3L off

[0066] Among them, L cen It is the center point localization loss, L cor It is the corner point localization loss, L off λ1, λ2, and λ3 are the center point offset prediction loss, and λ1, λ2, and λ3 are the weight hyperparameters, with λ1 = 1, λ2 = 1, and λ3 = 1.

[0067]

[0068] Where, p hm This represents the predicted center point heatmap. p represents the true label of the center point heatmap. x,y This represents the probability that the position (x, y) is the center point of the target. Let α and β represent the true label, and α and β be the weight hyperparameters, respectively, with α = 2 and β = 4. If this point, when mapped back to the input search image, is the center point of the target, then... =1, otherwise

[0069]

[0070] Where p hm This represents the predicted corner heatmap. p represents the true label of the corner heatmap. x,y This represents the probability that the position (x, y) is the corner point corresponding to the target. Let α and β represent the true label, and α and β be the weight hyperparameters, with α = 2 and β = 4. If this point, when mapped back to the input search image, is the corner point corresponding to the target, then... =1, otherwise

[0071]

[0072] Where N represents the number of training samples, SmoothL1Loss represents the L1 loss, and o p This represents the predicted pixel error due to the network step size. Indicates the true label, The calculation method is: the difference between the coordinates of the point in the output center point heatmap and the coordinates of the point after it is mapped back to the input search image.

[0073] During testing, template-search image pairs are still used as input to the model, but the image size does not need to be fixed. The model output is a set of center point heatmaps, corner point heatmaps, center point offsets, and their corresponding confidence scores. The center point with the highest confidence score is selected as the initial localization result. Then, the four corner points with the highest confidence scores are selected to adjust the initial localization result. Finally, the localization result is corrected using their corresponding center point offsets to obtain the precise location of the template's center point in the search image. The adjustment strategy for the initial corner point localization result is as follows:

[0074] p opt =(1-ω)p corner +ωp center

[0075] Where, p opt p represents the adjusted center point location result. corner p represents the position of the center point obtained from the four corner points. center This represents the initial positioning result of the center point, where ω is the weight hyperparameter, and we take ω = 0.9.

[0076] Figure 4 The diagram shown is an output optimization diagram of the center point localization network during testing.

[0077] Test results:

[0078] Figure 5 The following are some matching results obtained using the method provided in this invention.

[0079] Table 1 presents the comparison results between the method provided in this invention and some existing template matching methods on the BBS105 dataset and the self-built Hard350 dataset, where Ours represents the method provided in this invention. Since our method completely abandons the prediction of the target bounding box and only predicts the target center point, three evaluation metrics, MCE, SR5, and SR10, were used in the experiment. The smaller the MCE data and the larger the SR5 and SR10 data, the better the performance of the template matching method used.

[0080] Table 1. Test results of different methods on the BBS dataset and Hard350 dataset.

[0081]

[0082] Central error (CE) is used as the evaluation metric for a single sample. MCE is defined as the average central error of all image pairs. SR5 and SR10 are defined as the ratios of successfully identified samples to the total number of test samples, where a central error less than 5 and 10, respectively, indicates successful identification. CE is given by the following formula:

[0083]

[0084] Among them, (x p ,y p ) and (x g ,y g These are the predicted center point coordinates and the actual center point coordinates, respectively.

[0085] The experimental results show that the method provided by this invention has better performance compared with other template matching methods.

[0086] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A template matching method based on twin network and precise center point positioning, characterized in that: include: S1: Obtain the training dataset, i.e., the template and the search image; S2: Construct a template matching network based on Siamese network and precise center point localization, and input the training dataset from S1 into the template matching network; S2 includes: S2.1: Construct a twin feature extraction network, input template and search image, and output corresponding feature maps; S2.2: Construct a dynamically shrinking cross-correlation network to perform cross-correlation fusion on the feature maps output from S2.1; S2.3: Construct a center point localization network to predict the center point information of the template in the search image based on the output of the dynamically shrinking cross-correlation network in S2.2; The center point localization network in S2.3 consists of a key point localization module and an offset prediction module. The key point localization module consists of a center point localization branch and a corner point localization branch, and the offset prediction module consists of a center point offset prediction branch. Both use the cross-correlation fused feature map as input. The center point positioning branch is composed of three 3x3 deconvolution layers and a 3x3 convolution layer, and finally outputs a center point heat map with a channel number of 1 , indicating the probability of each position being predicted as a target center point, for roughly positioning the target center point; The corner localization branch consists of three 3×3 deconvolutional layers and one 3×3 convolutional layer, ultimately outputting a corner heatmap with 4 channels. , which represents the probability that each position is predicted to be the corresponding corner point of the target, and is used to accurately locate the center point of the target based on the rough positioning; The center point offset prediction branch consists of three 3×3 deconvolutional layers and one 3×3 convolutional layer, ultimately outputting a center point offset prediction map with 2 channels. , representing the offset error of each predicted position, is used to make a final correction to the target center point based on accurate positioning; S3: Train the template matching network of S2 using the loss function; The loss function for S3 is: ； in, It is the center point positioning loss. It is the corner point positioning loss. It is the center point offset prediction loss. , and It is a weight hyperparameter, take , , .

2. The template matching method based on Siamese networks and precise center point localization according to claim 1, characterized in that: The Siamese feature extraction network of S2.1 consists of two cascaded convolutional neural networks with shared parameters and identical structures. The template T and the search image S are used as inputs, and the outputs are the corresponding feature maps. and .

3. The template matching method based on Siamese networks and precise center point localization according to claim 2, characterized in that: The convolutional neural network for the Siamese feature extraction network is based on the standard ResNet50, with the following limitations: (1) The standard ResNet50 consists of 5 convolutional groups and a fully connected layer, with the last fully connected layer removed; (2) After the 5th convolutional group, add a 3×3 deconvolutional layer with a stride of 1 and 128 channels, and two 3×3 deconvolutional layers with a stride of 2 and 128 channels. (3) Add a feature map fusion module after each deconvolution layer between the feature map and the feature map of the same scale corresponding to the standard ResNet50.

4. The template matching method based on Siamese networks and precise center point localization according to claim 3, characterized in that: The S2.2 dynamic shrinking cross-correlation network is composed of a depthwise convolutional module and a dynamic shrinking attention module cascaded together; The depthwise convolution module uses the template feature map as the convolution kernel and performs depthwise convolution with the search image feature map to obtain preliminary fusion results; The dynamic shrinking attention module is used to increase attention to high-response channels based on the initial fusion results and remove some feature regions in the feature map that are similar to the target, so as to obtain the final fusion result.

Images