Template matching system and matching method based on bipartite graph and graph attention mechanism

Abstract

The present invention relates to a template matching method based on a bipartite graph and a graph attention mechanism, which is characterized by: constructing an MLF network module for extracting fusion features of a template image and a search image; constructing a BGAM network module for encoding the relationship between template nodes and search nodes, and aggregating corresponding local information for each search node and template node to generate a response graph for positioning the template; the system includes a network construction module, a learning and pre-training module, and a calculation and positioning module. Beneficial effects: The present invention realizes local-to-local information transmission between the template image and the search image, and the MLF network module improves the representation capability of the template target by fusing features representing the meaning of different levels of the target. The method can accurately and stably realize template matching in many actual complex scenarios, reduces the amount of calculation and video memory usage, and is conducive to deployment in engineering practice.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image processing, and in particular to a template matching system and a matching method based on a bipartite graph and a graph attention mechanism. Background Art

[0002] Template matching is one of the most basic tasks in the field of computer vision and image processing technology. Given a template image, template matching technology can automatically estimate the location and scale information of the target in the search image. This technology is widely used in image matching, landmark recognition, 3D reconstruction and military fields. Although the existing template matching technology has made great progress, it is still a challenging task to achieve accurate and robust template matching in complex scenes due to occlusion, deformation, background changes and many other reasons.

[0003] Traditional template matching algorithms, such as the sum of squared differences, the sum of absolute differences, and normalized cross-correlation, all start from constructing a more accurate similarity metric to improve the accuracy of template matching. These algorithms use pixels as the basic feature unit and calculate similarity through a sliding window. When the background of the template changes, the accuracy of the algorithm will be greatly reduced.

[0004] In recent years, with the development of deep learning methods, especially the outstanding performance of convolutional neural networks (CNN) in computer vision and image processing fields such as target classification and target detection, the deep convolution features of images have also been gradually used in template matching tasks. Some existing algorithms have solved the problems of target occlusion, non-rigid deformation or target background changes to a certain extent, but due to the complexity of the designed similarity measurement, the algorithm has a huge amount of calculation. Moreover, the existing template matching algorithm using CNN matches the search image features as a whole, ignoring the local correspondence between the template image and the search image. Therefore, when the shape, posture and background of the target change, the accuracy of the matching is difficult to guarantee. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the above-mentioned technology and to provide a template matching system and matching method based on bipartite graph and graph attention mechanism, which improves the accuracy of template matching when the shape, posture and background of the target change, reduces the amount of calculation and video memory occupancy, and is conducive to deployment in engineering practice.

[0006] To achieve the above-mentioned purpose, the present invention adopts the following technical scheme: a template matching system based on bipartite graph and graph attention mechanism, including a network construction module, a learning and pre-training module and a calculation and positioning module,

[0007] The network construction module is used to construct a multi-level feature fusion MLF module under the twin network framework, which is used to extract and model the fusion features of the template image and the search image, and construct a BGAM module;

[0008] The learning and pre-training module is used to pre-train the twin network model using a large-scale data image dataset, and learn the multi-level fusion features of the template image and the search image, as well as the graph attention aggregation features in the BGAM module, so as to obtain the local correspondence information between the template image and the search image;

[0009] The calculation and positioning module is used to use the pre-trained twin network model to extract the fusion features of the template image and the search image from the template branch and the search branch respectively and input them into the BGAM network module to generate a response map for positioning the template.

[0010] A template matching method based on bipartite graph and graph attention mechanism is proposed. The MLF network module is constructed to extract the fusion features of the template image and the search image. The BGAM network module is constructed to encode the relationship between the template node and the search node, and the corresponding local information is aggregated for each search node and template node to generate a response graph for positioning the template. The specific steps are as follows:

[0011] 1) Under the twin network framework, a multi-level feature fusion MLF network module is constructed based on the CNN convolutional neural network model;

[0012] 2) Under the twin network framework, a local information matching BGAM network module based on bipartite graph and graph attention mechanism is constructed. The relationship between template nodes and search nodes is encoded through bipartite graph and graph attention mechanism. By learning the attention coefficient, the corresponding local information is aggregated for each search node and template node, and the response graph of the positioning template is generated for subsequent template positioning;

[0013] 3) The MLF network module and the BGAM network module construct a twin network model, use a large-scale data set to pre-train the constructed twin network model, and adjust the parameters in the twin network model;

[0014] 4) the pre-trained MLF network module uses the output results of different convolutional layers of the CNN convolutional neural network as the input of the MLF network module to extract the fusion features of the template image and the search image;

[0015] 5) Inputting the fusion features of the template image and the fusion features of the search image into the BGAM network module, performing complete bipartite graph modeling on the input fusion features and encoding the local feature relationship between the template node and the search node, respectively obtaining the graph attention aggregation features of the template image and the search image, calculating the similarity score between the two sub-graphs by means of convolution cross-correlation, and generating a response graph of the positioning template;

[0016] 6) Locate the position and scale information of the template image according to the response graph, and the area with the highest similarity score is the best matching area.

[0017] Further, in step 4), the output sizes of different convolutional layers of the CNN convolutional neural network are adjusted to the same size by bicubic interpolation, wherein each convolutional feature is multiplied by a weight coefficient, and the formula of the multi-level feature fusion MLF network module is expressed as:

[0018]

[0019] Among them, || means connection in the channel dimension, α i (i=1,2,3,...,n) are the features C of different convolutional layers i The weight coefficient of i (i=1,2,3,...,n)∈[0,1],

[0020] Furthermore, in step 5), the fusion feature F of the template image t and the fusion feature F of the search image s Each 1×1×c grid in is regarded as a node, which represents the local information of the feature map, where c represents the number of feature channels; let V t To include F t The set of all nodes in , let S t To include S t The set of all nodes in , then the complete bipartite graph describing the local feature relationship between the template target and the search area can be defined as:

[0021]

[0022] The two subgraphs of a complete bipartite graph G are defined as: and For each (i,j)∈E, let e ij Represents node i∈V t and node j∈V s The relevant score in and is the feature vector of node i and node j;

[0023] In order to adaptively learn the correlation between nodes, the node features are linearly transformed, and the inner product between the transformed feature vectors is taken to calculate the correlation score, that is, Where W t and W s is the linear transformation matrix;

[0024] In order to balance the amount of information transferred to the search area, a softmax function is used to ij Normalize and get the graph attention coefficient At this time, a ij The meaning of can be described as: how much "attention" should each node i on the template feature give to the search node j;

[0025] Furthermore, in step 5), the complete bipartite graph G is constructed by using the subgraph G t All nodes in G are passed to the subgraph G s The attention of the jth node in the calculation node j is the attention aggregation feature. Where W v is the matrix used for linear transformation; the graph attention mechanism is expanded using multi-head attention. K independent attention mechanisms are used to perform calculations separately, and then the calculated features are connected in series to obtain a size of Graph Attention Aggregation Features

[0026]

[0027] Similarly, the calculation from the subgraph G s All nodes are mapped to the subgraph G t The attention mechanism aggregation feature of the i-th node can be expressed as

[0028]

[0029] Beneficial effects: Compared with the prior art, the present invention realizes local-to-local information transfer between the template image and the search image based on the complete bipartite graph modeled by the BGAM network module. Moreover, in terms of feature representation, the MLF network module improves the representation capability of the template target by fusing the features representing the meanings of different levels of the target. This method can accurately and stably realize template matching in many actual complex scenes. Compared with other methods, it improves the accuracy of template matching, reduces the amount of calculation and video memory usage, and is conducive to deployment in engineering practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is the system structure diagram of the present invention

[0031] Figure 2 It is a schematic diagram of the network framework of the present invention;

[0032] Figure 3 It is a schematic diagram of the multi-level feature fusion module (MLF) of the present invention;

[0033] Figure 4 It is a schematic diagram of the bipartite graph and graph attention mechanism module (BGAM) of the present invention. DETAILED DESCRIPTION

[0034] In order to be able to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other. In the following description, many specific details are set forth in order to fully understand the present invention, and the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the art without making creative work are within the scope of protection of the present invention. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0035] See also Figure 1 The present invention provides a template matching method based on bipartite graph and graph attention mechanism, and the method mainly includes the construction of MLF module and BGAM module. The method includes the following steps:

[0036] S101, see Figure 2 In the twin network framework, a multi-level feature fusion MLF network module is constructed based on a convolutional neural network model. The MLF network module uses the output features of multiple convolutional layers of the convolutional neural network as the input of the MLF network module to construct fusion features for extracting template images and search images.

[0037] In this step, the pre-trained VGG19 network is used as the backbone convolutional neural network. Figure 2 As shown in the figure, the output features of the three convolutional layers Conv1_2, Conv2_2 and Conv3_4 of the VGG19 network are extracted as input to build a multi-level feature fusion module for template matching. At this time, the multi-level feature fusion module can be described as:

[0038]

[0039] Among them, || means connection in the channel dimension, α i (i=1,2,3) are the features C of different convolutional layers i The weight coefficient of i (i=1,2,3)∈[0,1],

[0040] S102, see Figure 3 In the twin network framework, a local information matching network BGAM module based on bipartite graph and graph attention mechanism is constructed. The relationship between template nodes and search nodes is encoded through bipartite graph and graph attention mechanism, and the corresponding local information is aggregated for each search node and template node by learning the attention coefficient.

[0041] In this step, the template image feature F t and search image features F s Each 1×1×c grid in is regarded as a node, which represents the local information of the feature map, where c represents the number of feature channels. Let V t To include F t The set of all nodes in , let S t To include S t The set of all nodes in , then the complete bipartite graph describing the local feature relationship between the template target and the search area can be expressed as:

[0042]

[0043] Then the two subgraphs of the complete bipartite graph G can be represented as: and For each (i,j)∈E, let e ij Represents node i∈V t and node j∈V s The relevant score in and is the feature vector of node i and node j.

[0044] In this step, if a local feature in the search area is more similar to a local feature of the template, it is more likely to be a foreground target, and more target information should be transferred to the corresponding node of the search image during the matching process. ij It should be proportional to the similarity of the two node features. The present invention uses the inner product between features as a measure of the correlation score. In order to adaptively learn the correlation between nodes, the node features are first linearly transformed, and then the inner product between the transformed feature vectors is taken to calculate the correlation score, that is, using the formula Calculate, where Wt and W s is a linear transformation matrix. In order to balance the amount of information transferred to the search area, a softmax function is used to transform e ij Normalize and get the graph attention coefficient

[0045] In this step, we use the subgraph G t All nodes in G are passed to the subgraph G s The attention of the jth node in the calculation node j is the attention aggregation feature. Where W v is the matrix used for linear transformation.

[0046] In order to stabilize the graph attention learning process, the multi-head attention method is used to expand the graph attention mechanism. K independent attention mechanisms are used to perform calculations separately, and then the calculated features are connected in series to obtain a size of Graph Attention Aggregation Features Similarly, the calculation from the subgraph G s All nodes are mapped to the subgraph G t The attention mechanism aggregates features of the i-th node

[0047] S103, pre-training the constructed twin network model using a large-scale data set, and adjusting the parameters in the network model. The twin network model mainly includes an MLF network module and a BGAM network module.

[0048] In this step, offline training uses GOT-10K, COCO, ImageNet DET, and ImageNet VID as training datasets. Randomly select two frames from the same video sequence and combine them into a pair of template image and search image. During training, use template image patches and search image patches of fixed size.

[0049] In this step, for the network parameters of the multi-level feature extraction part, the network parameters of VGG19 trained on the large-scale dataset ImageNet are selected for initialization and frozen, and no longer trained. The parameters in the MLF module and the GATM module are randomly initialized. The proposed network model is trained using an end-to-end method, and the SGD optimization method is used for training. The initial value of the learning rate is 0.01, the decay coefficient is 0.87, and a total of 100 epochs are trained.

[0050] S104, using the pre-trained MLF network module in the twin network model, extracting the fusion features of the template image and the search image from the template branch and the search branch respectively.

[0051] S105, input the template image fusion features and the search image fusion features described in S104 into the BGAM network module to obtain the graph attention aggregation features of the template image and the search image respectively. Finally, the similarity score between the two sub-graphs is calculated by convolution cross-correlation to generate a response graph for locating the template.

[0052] S106, locating the position and scale information of the template image according to the response map generated in S105, and the area with the highest similarity score is the best matching area.

[0053] The advantages of the technical solution provided by the present invention are described below by taking the adopted BBS data set as an example.

[0054] The BBS dataset has been widely used for quantitative evaluation of template matching algorithms. The dataset contains 35 color video sequences selected from the OTB dataset. Three pairs of images are randomly collected from each color video sequence. The frame distance between each pair of images is 20, forming a 105-pair image dataset for template matching algorithm evaluation.

[0055] In order to evaluate the performance of the template matching method, the area under the success rate curve (AUC) is used as a measure of the overall accuracy of template matching. The experiment compares the method proposed by the present invention with the methods that have performed well in recent years. Compared with the QATM method and the Deep-DIM method, the AUC of the template matching method is improved by 6.23% and 3.77%, respectively. In addition to the template matching accuracy, the experiment uses the entire BBS data set to compare and evaluate the computational efficiency of the template matching method with the QATM method and the Deep-DIM method. The GPU video memory occupancy of the template matching method is 1064 Mib, and the average calculation time is 0.11s, which is lower than that of the QATM method and the Deep-DIM method. The reason is that the other two methods need to calculate the complex similarity measurement from the template image to the search image, while the template matching method calculates the feature similarity through convolution cross-correlation, which is simple in principle and easy to implement. In addition, all calculations of the template matching method from feature extraction to the final similarity score map can be performed on the GPU processor, making the algorithm superior to the other two methods in terms of computational efficiency.

[0056] See also Figure 4 The present invention also proposes a twin network target tracking system based on a convolutional self-attention module, wherein the system comprises:

[0057] A network construction module, wherein the construction module is used to first construct a multi-layer feature fusion (MLF) module under the twin network framework to extract the fusion features of the template image and the search image, and then to model the similarity matching problem between features based on a complete bipartite graph. At the same time, a graph attention mechanism is introduced to construct a bipartite graph and graph attention mechanism (BGAM) module for matching between local information;

[0058] A learning and pre-training module, which is used to pre-train the twin network model using a large-scale data image dataset, learn the multi-level fusion features of the template image and the search image, and the graph attention aggregation features in the BGAM module to obtain the local correspondence information between the template image and the search image, and adjust the parameters in the twin network model, mainly including the parameters in MLF and BGAM;

[0059] The calculation and positioning module is used to use the pre-trained twin network model to extract the fusion features of the template image and the search image from the template branch and the search branch respectively. Then the template image fusion features and the search image fusion features are input into the BGAM network module to obtain the graph attention aggregation features of the template image and the search image respectively. Finally, the similarity score between the two sub-images is calculated by convolution cross-correlation to generate a response map for locating the template. Finally, the position and scale information of the template image are located by the response map generated by the twin network model, and the area with the highest similarity score is the best matching area.

[0060] The template matching system based on bipartite graph and graph attention mechanism, wherein, in the learning and pre-training module, two frames are randomly selected from the same video sequence during pre-training, and they are combined into a pair of template image and search image. During the training process, template image blocks and search image blocks of fixed sizes are used. To this end, the original data is processed as follows:

[0061] For the template image, it is cropped in the original image according to the real bounding box, and the size of the cropped template image is recorded as (h t ,w t ), then max(h t ,w t ) as the side length to fill it to make it a square, and finally adjust its size to s according to the scale factor r. t ×s t, where the proportional factor r can be expressed as:

[0062]

[0063] Among them, h t and w t They are the height and width of the template target respectively.

[0064] For the search image, its original image size is equal to that of the template image, so the original image is first resized according to the scale factor r. Assume that the true bounding box of the target contained in the search image is b q =[l q ,t q ,w q ,h q ], then the adjusted true bounding box is b q '=[rl q ,rt q ,rw q ,rh q ]. For the resized search image, crop it according to the bounding box value given by the cropping formula, and the size of the cropped search image block is s q ×s q , where the clipping formula can be expressed as:

[0065]

[0066]

[0067] rw q , rh q ]

[0068] Among them, l q and t q are the distances from the left and top of the target ground truth bounding box respectively; w q and h q are the width and height of the target’s true bounding box, respectively; τ(x,y) represents a randomly generated integer in the range [x,y].

[0069] The above reference embodiments provide a detailed description of a template matching system and matching method based on a bipartite graph and a graph attention mechanism, which are illustrative rather than restrictive. Several embodiments may be listed according to the limited scope. Therefore, changes and modifications without departing from the overall concept of the present invention should fall within the scope of protection of the present invention.

Claims

1. A template matching method based on bipartite graph and graph attention mechanism, characterized by: Based on the template matching system, an MLF network module is constructed to extract the fusion features of the template image and the search image; Construct a BGAM network module to encode the relationship between template nodes and search nodes, aggregate the corresponding local information for each search node and template node, and generate a response graph for positioning templates; the specific steps are as follows: 1) Under the twin network framework, a multi-level feature fusion MLF network module is constructed based on the CNN convolutional neural network model; 2) Under the twin network framework, a local information matching BGAM network module based on bipartite graph and graph attention mechanism is constructed. The relationship between template nodes and search nodes is encoded through bipartite graph and graph attention mechanism. By learning the attention coefficient, the corresponding local information is aggregated for each search node and template node, and the response graph of the positioning template is generated for subsequent template positioning; 3) The MLF network module and the BGAM network module construct a twin network model, use a large-scale data set to pre-train the constructed twin network model, and adjust the parameters in the twin network model; 4) the pre-trained MLF network module uses the output results of different convolutional layers of the CNN convolutional neural network as the input of the MLF network module to extract the fusion features of the template image and the search image; 5) Inputting the fusion features of the template image and the fusion features of the search image into the BGAM network module, performing complete bipartite graph modeling on the input fusion features and encoding the local feature relationship between the template node and the search node, respectively obtaining the graph attention aggregation features of the template image and the search image, calculating the similarity score between the two sub-graphs by means of convolution cross-correlation, and generating a response graph of the positioning template; 6) Locate the position and scale information of the template image according to the response graph, and the area with the highest similarity score is the best matching area; In step 4), the output sizes of different convolutional layers of the CNN convolutional neural network are adjusted to the same size by bicubic interpolation, wherein each convolutional feature is multiplied by a weight coefficient, and the formula of the multi-level feature fusion MLF network module is expressed as: Among them, || means connection in the channel dimension, α i (i=1,2,3,...,n) are the features C of different convolutional layers i The weight coefficient of i (i=1,2,3,...,n)∈[0,1], 2. The template matching method based on bipartite graph and graph attention mechanism according to claim 1, characterized in that: In step 5), the fusion feature F of the template image t and the fusion feature F of the search image s Each 1×1×c grid in is regarded as a node, which represents the local information of the feature map, where c represents the number of feature channels; let V t To include F t The set of all nodes in , let S t To include S t The set of all nodes in , then the complete bipartite graph describing the local feature relationship between the template target and the search area can be defined as: The two subgraphs of a complete bipartite graph G are defined as: and For each (i,j)∈E, let e ij Represents node i∈V t and node j∈V s The relevant score in and is the feature vector of node i and node j; In order to adaptively learn the correlation between nodes, the node features are linearly transformed, and the inner product between the transformed feature vectors is taken to calculate the correlation score, that is, Where W t and W s is the linear transformation matrix; In order to balance the amount of information transferred to the search area, a softmax function is used to ij Normalize and get the graph attention coefficient At this time, a ij The meaning can be described as: how much "attention" each node i on the template feature should give to the search node j.

3. The template matching method based on bipartite graph and graph attention mechanism according to claim 1 or 2, characterized in that: Step 5) The complete bipartite graph G is constructed by using the subgraph G t All nodes in G are passed to the subgraph G s The attention of the jth node in the calculation node j is the attention aggregation feature. Where W v is the matrix used for linear transformation; the graph attention mechanism is expanded using multi-head attention. K independent attention mechanisms are used to perform calculations separately, and then the calculated features are connected in series to obtain a size of Graph Attention Aggregation Features Similarly, the calculation from the subgraph G s All nodes are mapped to the subgraph G t The attention mechanism aggregation feature of the i-th node can be expressed as 4. A template matching system based on bipartite graph and graph attention mechanism, implemented based on the matching method of claim 1, characterized in that: The template matching system includes a network construction module, a learning and pre-training module, and a calculation and positioning module. The network construction module is used to construct a multi-level feature fusion MLF module under the twin network framework, which is used to extract and model the fusion features of the template image and the search image, and construct a BGAM module; The learning and pre-training module is used to pre-train the twin network model using a large-scale data image dataset, and learn the multi-level fusion features of the template image and the search image, as well as the graph attention aggregation features in the BGAM module, so as to obtain the local correspondence information between the template image and the search image; The calculation and positioning module is used to use the pre-trained twin network model to extract the fusion features of the template image and the search image from the template branch and the search branch respectively and input them into the BGAM network module to generate a response map for positioning the template.

Images