A redundancy removal method for target tracking model based on Siamese network

Abstract

This paper discloses a method for removing redundancy from a target tracking model based on a twin network. The method first feeds a template frame and a detection frame into the network for convolutional feature extraction, generating corresponding template and detection feature maps. A cross-correlation operation is then used to perform template matching on the classification and regression branches, respectively, to obtain matching feature maps. The region proposal network is then optimized and de-redundancy designed, ultimately yielding a response feature map. This method simplifies the network structure and reduces network parameters without sacrificing the functionality of this portion of the network, saving computational and memory overhead during both training and inference of the target tracking network model.

Description

Technical Field

[0001] The present invention belongs to the technical field of target tracking, and in particular relates to a de-redundancy method for a target tracking model based on a twin network. Background Art

[0002] In the field of target tracking, speed and accuracy are two core metrics for evaluating algorithm performance. The application of tracking methods requires balanced improvements in both accuracy and speed. However, most existing tracking methods focus on improving tracking accuracy while ignoring or even sacrificing tracking speed.

[0003] The SiamRPN (Siamese region proposal network) algorithm is a real-time target tracking algorithm proposed by Li et al. in 2018. It has excellent balanced performance and is favored by academia and industry. The network structure of SiamRPN can be divided into the following Figure 1 The two parts shown on the left and right are the Siamese network and the Region Proposal Network. The Siamese network extracts image features from the template frame and the detection frame. The Region Proposal Network is divided into two branches: classification and regression. The classification branch distinguishes the object from the background, while the regression branch aims to obtain more accurate object size and location. The Siamese network generally uses a common feature extraction network such as AlexNet and VGGNet.

[0004] Regardless of the classification branch or the regression branch, the key step of template matching is involved. Figure 2 As shown in the figure, during the inference calculation in the target tracking model, for the tracked video frame sequence, the target in the first frame is generally manually labeled as the template frame, and the subsequent frames are used as detection frames. The template frame is sent to the network for convolutional feature extraction and mapping through different convolutional layers to generate weights for the classification branch and regression branch respectively. These two weights serve as matching templates for subsequent frames. When the subsequent frames are sent to the network as detection frames, feature extraction of the backbone network is also performed first to generate the corresponding feature map. The backbone network here shares weights with the backbone network of the template frame. The feature map obtained from the detection frame is then sent to the classification branch and regression branch respectively. After passing through the convolution layer, the matching template generated by the template frame is used as the convolution kernel for convolution operation. Finally, the response map of each branch is generated, and the predicted target bounding box is obtained through post-processing. Summary of the Invention

[0005] Aiming at the imbalance between tracking speed and tracking accuracy of most current twin tracking algorithms, especially the difficulty of speed to meet application requirements, the present invention proposes a target tracking model de-redundancy method based on twin network.

[0006] The technical solution adopted by the present invention to solve the technical problem is: a target tracking model redundancy removal method based on a twin network, comprising the following steps:

[0007] Step S1: The first frame of the video frame sequence of the tracked target that is manually annotated is used as the template frame, and the subsequent frames are used as detection frames. They are sent to the network for convolutional feature extraction to generate weights for the classification branch and regression branch, and the corresponding template feature map z and detection feature map x are generated; the template feature map z is used as the input of the classification branch in the region proposal network, and the template feature map z is mapped through the convolution layer. At the same time, the dimension is increased to increase the number of channels to 2k, and the output template feature map is obtained. Where k is the number of pre-selected anchor boxes with different aspect ratios; at the same time, the detection feature map x is taken as input, and the detection feature map x is mapped through the convolution layer to obtain the output feature map The template feature map z is used as the input of the regression branch in the region proposal network, and the template feature map z is mapped through the convolution layer. At the same time, the dimension is increased to increase the number of channels to 4k, and the output template feature map is obtained. Where k is the number of pre-selected anchor boxes with different aspect ratios; at the same time, the detection feature map x is taken as input, and the detection feature map x is mapped through the convolution layer to obtain the output feature map Use cross-correlation operation to perform template matching on the classification branch and regression branch respectively:

[0008]

[0009]

[0010] Among them, ★ indicates related operations. and Represented by template feature maps and The response feature map obtained by performing group convolution operation as the convolution kernel;

[0011] Step S2, optimize the region proposal network: Based on the SiamRPN network, add a deep cross-correlation module to the cross-correlation convolution operation, first use the two parallel front-end convolution layers and a deep cross-correlation module in the region proposal network as a matching module, and find the two response feature maps. and The similar features in the image are used to facilitate the classification and positioning of the target in the subsequent network, and then the 1×1 convolution dimension feature map is mapped through the dimension mapping module in the region proposal network, thereby achieving foreground and background classification and target coordinate regression respectively;

[0012] Step S3, de-redundancy design: De-redundancy is performed on the parallel branches of the SiamRPN structure. According to the functions implemented by the modules in the framework, the cross-correlation module and its front-end convolutional layer in the region proposal network are recorded as matching modules, and the 1×1 convolutional layer is recorded as a dimension mapping module. The matching module and the dimension mapping module are shared in the network structure. The template features and detection features of the twin network are output as related feature maps to obtain shared matching information. The feature maps are then sent to the classification branch and regression branch in the region proposal network for feature map dimension mapping. Finally, different parameters are obtained to respectively realize foreground and background classification and target coordinate regression;

[0013] In step S4, the shared matching module and dimension mapping module are used to remove half of the redundant network structure of the tiny-SiamRPN network structure obtained after the above de-redundancy, and the matching and dimension mapping of the initial frame features and subsequent frame features are completed to obtain a matching feature map.

[0014] Furthermore, the response feature map output by the classification branch in step S1 has 2k channels, of which the target and background each occupy k channels, and their values ​​represent the positive and negative activation values ​​of the corresponding positions; the feature map output by the regression branch has 4k channels, corresponding to the four parameters of the k candidate box regression adjustment: dx, dy, dw, dh, representing the displacement and scaling parameters of the corresponding anchor box and the real box.

[0015] Furthermore, the redundancy removal in step S3 is performed by f mod The 3×3 convolution represents the replacement of f mod1 and f mod2 , f det The 3×3 convolution represents the replacement of f det1 and f det2 , f Xcorr1 and f Xcorr2 The module is reduced to f Xcorr , using the public mapping module f c Replace f c1 and f c2 ; When performing network structure redundancy removal, the parallel network branch f mod1 and f mod2 , f det1 and f det2 , f Xcorr1 and f Xcorr2 , f c1 and f c2 The two layers are merged and a common module is used to implement the matching and mapping functions, so that the number of convolutional layers in this part is half of that before de-redundancy.

[0016] The beneficial effects of the present invention are: the tracking algorithm model based on the twin network of the present invention simplifies the network structure by removing redundancy, improves the overall tracking performance of the algorithm, and optimizes the network structure of the tracking algorithm by using the network structure designed with redundancy, thereby saving the network's computing cost and inference time, so that the network can obtain a faster inference speed without significantly reducing performance; the method of the present invention has certain adaptability and scalability, and can be applied to tracking algorithms based on twin network structures and other network models with parallel structures to enhance their tracking speed and comprehensive tracking performance; the present invention simplifies the network structure and reduces the network parameters without losing the function of this part of the network, so that the target tracking network model saves computing and memory overhead in both training and inference processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a framework diagram of the SiamRPN network in the prior art;

[0018] Figure 2 This is the network reasoning framework diagram of SiamRPN in the prior art;

[0019] Figure 3 It is a network framework diagram of the baseline algorithm in the existing technology;

[0020] Figure 4 This is a framework diagram of the SiamRPN region proposal network in the prior art;

[0021] Figure 5 This is the framework diagram of the tiny-SiamRPN region proposal network of the present invention;

[0022] Figure 6 This is the overall network framework diagram of the tiny-SiamRPN of the present invention. DETAILED DESCRIPTION

[0023] To further illustrate the purpose and technical solutions of the present invention, the present invention will be further described in detail below with reference to the specific embodiments of the accompanying drawings. The following embodiments are only used to illustrate the present invention and do not constitute a limitation of the present invention.

[0024] The present invention discloses a method for removing redundancy of a target tracking model based on a twin network, which includes the following steps.

[0025] (1) In the region proposal network, the input is set as the template feature map z and the detection feature map x. In the classification branch, z is first mapped through the convolution layer to obtain the output feature map At the same time, the dimension is increased, and the number of channels is increased to 2k (k refers to the number of pre-selected anchor boxes with different aspect ratios). Similarly, the detection image x is also passed through the convolution layer, but the number of channels is not changed, and we get Similarly, convolution mapping is also performed in the regression branch. The difference is that when mapping z, the number of channels is increased to 4k, and the feature map is obtained. And the detection feature map with unchanged number of channels

[0026] Both the classification branch and the regression branch involve the key step of template matching. In the region proposal network, template matching is performed using cross-correlation operations. The “★” in the figure represents the related operations. The specific operations of template matching in the classification branch and the regression branch are as follows:

[0027]

[0028]

[0029] The above two formulas are respectively based on the template feature map and As the convolution kernel, group convolution is performed, and "*" represents the group convolution operation to obtain the corresponding response feature map. and The response feature map output by the classification branch has 2k channels, of which the target and background each occupy k channels, and their values ​​represent the positive and negative activation values ​​​​of the corresponding positions; the feature map output by the regression branch has 4k channels, corresponding to the four parameters of the k candidate box regression adjustment: dx, dy, dw, dh, representing the displacement and scaling parameters of the corresponding anchor box and the real box.

[0030] When the region proposal network is inferring, such as Figure 2 As shown in the figure, for a video frame sequence of a tracking target, the target is generally manually labeled in the first frame and used as a template frame, while the subsequent frames are used as detection frames. When the template frame is sent to the network, convolutional feature extraction is first performed, and then through different convolutional layer mappings, weights for the classification branch and regression branch are generated respectively. These two weights serve as matching templates for subsequent frames. When the subsequent frames are sent to the network as detection frames, feature extraction of the backbone network is also performed first to generate corresponding feature maps. The backbone network used here shares weights with the backbone network of the template frame. The feature maps obtained from the detection frame are then sent to the classification branch and regression branch respectively. After passing through the convolution layer, the matching template generated by the template frame is used as the convolution kernel for convolution operations. Finally, the response maps of each branch are generated, and the predicted target bounding box is obtained through post-processing.

[0031] (2) In SiamRPN++, a depth-wise cross-correlation module (DW-XCorr) is proposed for the cross-correlation convolution operation. The depth-wise cross-correlation layer uses a channel-by-channel convolution method to reduce the number of parameters of the feature map obtained by the cross-correlation operation and enhance the stability of training. Therefore, the redundancy removal method of the present invention continues to use the depth-wise cross-correlation module, and finally obtains the baseline algorithm structure, such as Figure 3 shown.

[0032] In the network architecture based on the baseline algorithm, the region proposal network can be mainly divided into two parts: the first part consists of two parallel front-end convolutional layers and a deep cross-correlation module, and the second part is a 1×1 dimension mapping network. The role of the first part in the region proposal network is to match the template frame features input to the module with the detection frame features, and find similar features in the two feature maps to facilitate the subsequent network classification and target location. This part exists in parallel in the classification branch and the regression branch, and there is a high degree of similarity in terms of network structure, parameters, and functionality. At the same time, after the two branches complete the cross-correlation matching to obtain the feature map, the second part of the region proposal network needs to perform 1×1 convolution dimension mapping in the mapping network. This part of the network also has a partially redundant structure.

[0033] (3) Aiming at the possible redundant structure of the parallel branches of SiamRPN, tiny-SiamRPN has carried out a redundancy removal design on its structure. Figure 4 As shown in the figure, based on the functions implemented by the modules in the framework, the deep cross-correlation layer and its front-end convolutional layer in the region proposal network are denoted as the matching module, and the 1×1 convolutional layer is denoted as the dimension mapping module. Since the matching module exists in the upper and lower branches of the region proposal network, both its classification and regression branches require input from both template features and detection features. Therefore, during actual inference, the matching module calculations must also be performed twice. This may result in redundant network structure and waste inference computation time. This similar network structure is marked with a dashed box in the figure and mainly includes the feature matching module and part of the dimension mapping module.

[0034] The optimization strategy for the region proposal network is to share the network structure of the matching module and the dimension mapping module. Figure 5As shown in the figure, the matching module in the original region proposal network is integrated before the region proposal network. This module can complete its template matching function. At this time, the template frame feature map and detection frame feature map obtained by the twin network do not need to be divided into two parallel branches and fed into the region proposal network. Instead, the template features and detection features of the twin network are output as related feature maps to obtain shared matching information. This feature map is then fed into the classification branch and regression branch of the region proposal network for feature map dimension mapping. Finally, different parameters are obtained to achieve foreground and background classification and target coordinate regression respectively.

[0035] from Figure 5 As can be seen from the figure, the optimized region proposal network completes feature template matching before channel mapping. This simplifies the network structure and reduces network parameters without sacrificing this part of the network functionality. This saves computational and memory overhead during both training and inference of the tracking algorithm network model. In summary, when removing redundancy from the network structure, the main focus is on merging parallel network branches and using common modules to implement their matching and mapping functions.

[0036] from Figure 5 It can also be seen that the redundant region proposal network completes the feature template matching before channel mapping, which simplifies the network structure and reduces the network parameters without losing the function of this part of the network, so that the tracking algorithm network model saves computing and memory overhead in both training and inference. mod The 3×3 convolution represented by f mod1 and f mod2 , f det The 3×3 convolution represented by f det1 and f det2 , f Xcorr1 and f Xcorr2 The module is reduced to f Xcorr , f c1 and f c2 Using the public mapping module f c In short, when performing network structure redundancy removal, the parallel network branch f mod1 and f mod2 , f det1 and f det2 , f Xcorr1 and f Xcorr2 , f c1 and f c2 The two layers are merged and a common module is used to implement the matching and mapping functions, so that the number of convolutional layers in this part is half of that before de-redundancy.

[0037] (4) After the above redundancy removal design, we get tiny-SiamRPN, whose network structure is as follows Figure 6As shown in the figure, the backbone network remains unchanged from the baseline algorithm. The main difference is that the parallel structure of the region proposal network uses a de-redundant design and shares the matching module and dimension mapping module, collectively referred to as the shared module in the figure. In tiny-SiamRPN, the shared module removes half of the network structure, completing the matching of initial frame features with subsequent frame features and performing dimension mapping to obtain the matching feature map. Finally, the network only needs to use the dimension mapping layer with different parameters to achieve classification and regression.

[0038] Table 1 shows the simplified network model of tiny-SiamRPN. The left table shows the network structure of the baseline tracking algorithm, while the right table shows the network structure of the tracking algorithm. The table primarily represents the convolutional and pooling layers in the neural network. The convolutional layer is represented by "conv<convolution kernel size> - <number of convolution kernels>," and the pooling layer is represented by "maxpool." Other special modules include the depthwise cross-correlation module, represented by "DW-Xcorr." The structures that have been adjusted by the left and right algorithms are highlighted in bold.

[0039] The tracking network has two inputs: the template branch receives the target image patch from the initial frame, with an image size of 112×112×3, while the detection branch receives the search region image from the subsequent frame, with an image size of 224×224×3. The following section mainly describes the adjustments made due to the improved network structure. The improved network merges the matching modules, combining the cross-correlation matching structure required by the region proposal network into a single one. A network layer comparison table of the baseline tracking algorithm and tiny-SiamRPN is shown below:

[0040]

[0041] As shown in the table, the four conv3-64 convolutional layers are reduced to two, and the two DW-XCorr layers are reduced to one, corresponding to the redundant design of the region proposal network structure. The two conv1-64 convolutional layers in the baseline network are combined into a single convolutional layer. The network's final output response map contains the number of pre-selected boxes, k, which is set to 3 in this paper.

[0042] The present invention is based on the basic network framework of SiamRPN (Siamese Region Proposal Network), and mainly performs network redundancy design on its region proposal network, and finally obtains a redundant tracking model based on the twin network, which is hereinafter referred to as tiny-SiamRPN. When performing network structure redundancy, the parallel network branches are mainly merged, and the common modules are used to realize their matching and mapping functions.

[0043] A comparative experiment was conducted on the above redundancy removal design to verify the effectiveness of the design method. The experiment was tested on a unified dataset and verified the tracking improvement effect of tiny-SiamRPN in multiple scenarios.

[0044] The tracking performance evaluation metrics used in this experiment are success rate, norm precision, and frame rate (FPS). Among the accuracy evaluation metrics, intersection over union (IoU) and pixel error (Pixel Error) are two basic evaluation metrics. Success rate and norm precision are defined around these two metrics.

[0045] The center pixel error refers to the Euclidean distance between the center of the target's predicted bounding box and the center of the target's actual bounding box. The calculation formula is: Where x and y are the horizontal and vertical coordinates of the center pixel of the prediction box, respectively. G ,y G The center pixel error generally reflects the accuracy of the tracking algorithm's prediction of the target position. The smaller the distance between the predicted position and the target's true position, the more accurate the algorithm's prediction. Conversely, the larger the center pixel error, the lower the algorithm's prediction accuracy.

[0046] The meaning of the intersection-to-union ratio is the ratio of the intersection and union of the area predicted by the algorithm and the actual target area. The calculation formula is: Formula A G Represents the area of ​​the predicted region, A T Represents the area of ​​the true target region. It is also known that the intersection over union (IoU) ∈ [0, 1] is close to 1, indicating that the predicted box and the actual box are almost overlapping, indicating that the algorithm has high prediction accuracy in that frame. Conversely, when the IoU value is close to 0, it means that the predicted box and the actual target box have almost no overlapping area, indicating that the algorithm has low prediction accuracy in that frame.

[0047] Accuracy refers to the proportion of frames in which the algorithm accurately predicts the target position during the entire tracking process. The accuracy of the prediction is determined by the fact that the error of the center pixel calculated by the algorithm in the frame is less than the set error threshold. The specific calculation formula for accuracy is: In the formula, the bool value is 1 if the conditions in the brackets are met, and 0 if they are not met. In the total number of frames n, the center pixel error PixelError calculated for the i-th frame i If it is less than the set error threshold T1 (set to 40 pixels), the prediction of the i-th frame is considered accurate, and the bool function count is 1.

[0048] The ratio of the final predicted accurate frames to the total number of frames is found through this statistic, which is the calculated precision. The average precision refers to the accuracy of the tracking video frame calculated when excluding the completely blocked or out-of-view frames. The calculation formula is Where n1 is the number of valid frames remaining after removing scenes such as complete occlusion from the total number of frames. The remaining statistical calculation method is consistent with the accuracy calculation method. The larger the value, the higher the accuracy of the algorithm.

[0049] The success rate refers to the ratio of the number of frames successfully predicted by the algorithm to the total number of video frames during the tracking process. The basis for judging the success of a frame prediction is whether the intersection-over-union ratio calculated by the tracking algorithm in that frame is greater than the set threshold. The specific calculation formula for the success rate is: In the formula, if the bool value meets the conditions in the brackets, it is 1, and if it does not meet the conditions, it is 0. In the total number of frames n, the intersection over union (IoU) of the i-th frame is i If it is greater than the set threshold T2 (set to 0.4), the frame is counted as the number of frames predicted successfully. The greater the success rate of the calculated number of predicted successful frames compared to the total number of frames, the higher the overall accuracy of the algorithm.

[0050] On the same tracking test set, the improvement effect of tiny-SiamRPN is shown in the following table:

[0051]

[0052] After implementing the network redundancy removal design, tiny-SiamRPN improved its frame rate by 78.7 FPS while maintaining its success rate and average precision. This experiment validates the effectiveness of the redundancy removal design proposed in this paper.

[0053] The dataset and platform environment of this experiment are as follows: the training dataset is ILSVRC, and the test dataset is a self-built dataset, which includes 257 videos of 16 categories of targets such as airplanes, ships, cars, and pedestrians, with a total of 580,453 images. The hardware and software platform include: Intel(R) Core(TM) i7-8700CPU@3.20GHz*12; GPU: Force GTX1080Ti; operating system: Ubuntu14.04LTS; deep learning framework: PyTorch.

[0054] In this experiment, the learning rate of the training weights is set to 0.005, the weight decay rate is 0.0001, each batch contains 256 images, the total number of iterations is 17550, the number of epochs is set to 40, and the number of iterations of a single epoch is 390.

[0055] It is easy for those skilled in the art to understand that the above description is only a preferred use case of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A target tracking model redundancy removal method based on a Siamese network, characterized by: Includes the following steps Step S1: The first frame in the video frame sequence of the tracked target is used as the template frame, and the subsequent frames are used as detection frames. These frames are fed into the backbone network for convolutional feature extraction to generate the corresponding template feature map z and detection feature map x. The template feature map z is used as the input of the classification branch in the region proposal network, and the template feature map z is mapped through the convolution layer. At the same time, the dimension is increased to increase the number of channels to 2k, and the output template feature map is obtained. Where k is the number of pre-selected anchor boxes with different aspect ratios; at the same time, the detection feature map x is taken as input, and the detection feature map x is mapped through the convolution layer to obtain the output feature map The template feature map z is used as the input of the regression branch in the region proposal network, and the template feature map z is mapped through the convolution layer. At the same time, the dimension is increased to increase the number of channels to 4k, and the output template feature map is obtained. Where k is the number of pre-selected anchor boxes with different aspect ratios; at the same time, the detection feature map x is taken as input, and the detection feature map x is mapped through the convolution layer to obtain the output feature map Use cross-correlation operation to perform template matching on the classification branch and regression branch respectively: Among them, ★ indicates related operations. and Respectively represent the template feature map and The response feature map obtained by performing group convolution operation as the convolution kernel; Step S2: Based on the SiamRPN network, a deep cross-correlation module is added to the cross-correlation convolution operation. First, the two parallel front-end convolution layers and a deep cross-correlation module in the region proposal network are used as matching modules to find the two response feature maps. and The similar features in the image are then mapped to feature maps of 1×1 convolutional dimension through the dimension mapping module in the region proposal network to achieve foreground and background classification and target coordinate regression respectively; Step S3: De-redundancy of the parallel branches of the SiamRPN structure. The cross-correlation module and its front-end convolutional layer in the region proposal network are recorded as the matching module, and the 1×1 convolutional layer is recorded as the dimension mapping module. The matching module and the dimension mapping module are shared in the network structure. The template features and detection features of the twin network are output as related feature maps to obtain shared matching information. The feature maps are then sent to the classification branch and regression branch in the region proposal network for feature map dimension mapping. In step S4, the shared matching module and dimension mapping module are simplified by half of the tiny-SiamRPN network structure to complete the matching of the initial frame features with the subsequent frame features and the dimension mapping to obtain a matching feature map.

2. The target tracking model de-redundancy method based on a twin network according to claim 1, characterized in that: The response feature map output by the classification branch in step S1 has 2k channels, of which the target and background each occupy k channels, and their values ​​represent the positive and negative activation values ​​of the corresponding positions; the feature map output by the regression branch has 4k channels, corresponding to the four parameters of the k candidate box regression adjustment: dx, dy, dw, dh, representing the displacement and scaling parameters of the corresponding anchor box and the real box.

3. The target tracking model de-redundancy method based on a twin network according to claim 2, characterized in that: The redundancy removal in step S3 is performed by f mod The 3×3 convolution represents the replacement of f mod1 and f mod2 , f det The 3×3 convolution represents the replacement of f det1 and f det2 , f Xcorr1 and f Xcorr2 The module is reduced to f Xcorr , using the public mapping module f c Replace f c1 and f c2 ; When performing network structure redundancy removal, parallel network branch f mod1 and f mod2 , f det1 and f det2 , f Xcorr1 and f Xcorr2 , f c1 and f c2 The two layers are merged and a common module is used to implement the matching and mapping functions, so that the number of convolutional layers in this part is half of that before de-redundancy.

Images