A network lightweight visual target tracking method

Abstract

The application discloses a network lightweight visual target tracking method, prunes a target tracking network using a Transformer attention architecture, and thus improves the performance of the tracking method. The structured pruning process of the target region is consistent with the reduction of the target search region, coarse positioning is performed using pruning, and then fine positioning is performed using target regression, so that the performance of the visual target tracking method is effectively improved, the efficiency of target tracking is improved, the calculation amount of target tracking is reduced, and the method is easy to implement on a hardware deployment end.

Description

Technical Field

[0001] This invention relates to the field of neural network model compression technology, and in particular to a lightweight visual target tracking method for networks. Background Technology

[0002] Visual object tracking (VOT) aims to determine the location of a target in subsequent video frames given initial information about the target. This task is a classic task in the field of computer vision (CV) and has significant application value in areas such as video surveillance and autonomous driving.

[0003] The Transformer is a model that utilizes attention mechanisms to improve training speed. The Transformer encoder consists of a self-attention layer, a position feedforward network, a layer normalization module, and a remaining connector. The query is the query component, the key is the component being compared, and the dot product of the two reflects their similarity.

[0004] The formula for the self-attention layer is shown below.

[0005]

[0006] Q, K, and V are linear transformations learned from the input, named query, key, and value, respectively, with h used for normalization. The dot product of query and key is transformed using the transfer function softmax to obtain the attention weights. The feedforward layer consists of an MLP applied to each timestamp.

[0007] Transformer pruning is divided into structured pruning and unstructured pruning. Unstructured pruning refers to using pruning as an effective means to remove redundant parameters in the network to improve network performance. However, it is difficult to effectively deploy the sparsity of unstructured pruning in hardware, so neural network accelerators struggle to improve the processing efficiency of algorithms. Structured pruning, on the other hand, refers to the structured removal of network parameters. This type of method has received considerable attention due to its ease of implementation.

[0008] Because visual target tracking tasks face many complex environments (occlusion, out-of-view, deformation, jitter, etc.), existing neural network-based target tracking algorithms often adopt backbone networks with stronger feature extraction performance, which reduces the computational efficiency of the network. Therefore, people have begun to seek some lightweight network methods to accelerate the inference speed of the network. Pruning has attracted attention as an important means, but the current pruning method is to remove Transformer attention heads by a fixed proportion (Ye B, Chang H, Ma B, et al. Joint feature learning and relation modeling for tracking: A one-stream framework[C] / / Computer Vision–ECCV 2022:17th European Conference,Tel Aviv,Israel,October 23–27,2022,Proceedings,PartXXII.Cham:Springer Nature Switzerland,2022:341-357.), which brings two problems:

[0009] First, the hyperparameters cannot be adjusted to a fixed ratio, and the number of parameters removed each time is fixed. It cannot be adjusted according to the target size or other factors, leaving room for performance improvement.

[0010] Secondly, fixed-ratio pruning methods require sorting all objects before pruning. Without a dedicated module, sorting operations cannot be efficiently implemented during hardware deployment.

[0011] The network for object tracking directly adopts the backbone network for object detection, resulting in differences between tasks.

[0012] Current target tracking networks with Transformer architecture as the backbone use a fixed-ratio pruning algorithm to narrow the target search range. However, the fixed ratio requires ranking the importance of all target regions, which requires a lot of computational resources and is difficult to implement on the hardware side. Therefore, it is necessary to develop a more reasonable algorithm to improve the efficiency of target localization and tracking. Summary of the Invention

[0013] The technical problem to be solved by the present invention is to provide a lightweight network-based visual target tracking method that addresses the shortcomings of existing technologies, effectively improves the performance of visual target tracking methods, increases the efficiency of target tracking, and reduces the computational load of target tracking.

[0014] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a lightweight network-based visual target tracking method, comprising the following steps:

[0015] S1. Obtain the block features of the target template and the block features of the search region;

[0016] S2. Concatenate the block features of the target template and the block features of the search region to obtain the concatenated feature X. t ; for the cascaded feature X t Perform multi-head attention feature mapping to obtain the enhanced feature x after weighted mapping by the attention layer. t ; t = 2, 3, ..., T; T is the number of video frames;

[0017] S3. Define several layers of the Transformer-based neural network as pruning layers; for the m-th pruning layer, if feature x t The soft mask value s of the i-th block feature (m) (t i () greater than the threshold θ (m) If so, the feature continues to propagate forward in the neural network;

[0018] S4. Repeat step S3 until the final pruned features are obtained; perform regression localization on the final pruned features to obtain the target location prediction value of the current frame, i.e., the t-th frame of the video.

[0019] S5. Determine whether frame t is the last frame. If not, end the process. If yes, extract the target position template features predicted in the current frame and the search area features of the next frame, and return to step S2 until all video frames have been processed.

[0020] This invention leverages the consistency between the structured pruning process of the target region and the narrowing of the target search region. It uses pruning for coarse localization and then target regression for fine localization, effectively improving the performance of visual target tracking methods, increasing the efficiency of target tracking, reducing the computational load of target tracking, requiring fewer hardware resources, and is easy to implement on the hardware deployment side.

[0021] In this invention, step S1, the process of determining the target template and the search area includes: assuming that at a certain moment, the image I of the t-th frame of the video... t If the size is N×N, then the image I in the (t-1)th frame of the video is centered on the target position determined in the previous moment. t-1 Extract an image block with a length and width of a first set number of pixels as the target template, and then... t Extract an image block with a length and width of the second set number of pixels as the search area.

[0022] In this invention, in step S2, the feature x enhanced by the attention layer weighted mapping... t The expression is:

[0023]

[0024] Where ReLU() is a non-linear activation function, LN() is a linear connection operation, softmax is a function that normalizes the weights to a range between 0 and 1, d represents the block feature dimension, and x n Represents cascaded features X t The features after the nth attention head mapping, W k W q and W v These represent the mapping parameters for the features corresponding to the index, queue, and passed value, respectively, and N is the dimension of the original image corresponding to the target template and the search region.

[0025] In this invention, in step S3, in the m-th pruning layer, feature x t The soft mask value s of the i-th block feature (m) (t i The formula for calculating ) is: Wherein, s(x i ) = s (m) (t i ), where L represents the sum of the number of block features in the target template and the number of block features in the search region, and A(x t ,x t ) is x t The probability of self-attention of the block features contained therein, where A(x) t ,x t ) (i,j) x represents t The i-th block feature t i and x t The j-th block feature t j The probability of the association between them. W k W q These represent the mapping parameters for the corresponding features of the index and queue, respectively, and N is the dimension of the original image corresponding to the target template and the search region.

[0026] In this invention, before step S1, the method further includes: training a Transformer-based neural network. The specific implementation process of training the Transformer-based neural network includes: using the block features of the target template and the block features of the search region as inputs to the neural network, aiming to minimize the loss function, using soft masks in the forward propagation process, and learning the weight parameters and pruning threshold of the neural network.

[0027] In this invention, the loss function of the Transformer-based neural network... Set as: in, Locate the loss to the target. To compare the losses, For regular loss.

[0028] Comparative loss The expression is: CrossEntropy represents the cross-entropy loss function, x p and x f These represent the label values ​​of the positive and negative sample spaces learned, respectively. Positive samples refer to the block features of the target template, while negative samples refer to the features extracted from the non-target template regions of the image.

[0029] Regular loss The expression is: Among them, M (m) This represents the hard mask of the m-th pruned layer, and the loss function is... That is, the sum of the 1-norms of the l-hard mask. M (m) (t i ) represents the hard mask M of the m-th pruning layer. (m) The i-th block feature t i The corresponding hard mask value, s (m) (t i ) represents the feature t of the i-th block in the m-th pruning layer. i The value of the corresponding soft mask.

[0030] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention prunes the target tracking network using the Transformer architecture, thereby improving the performance of the tracking method. By leveraging the consistency between the structured pruning process of the target region and the reduction of the target search area, coarse localization is achieved through pruning, followed by fine localization using target regression. This effectively improves the performance of the visual target tracking method, increases the efficiency of target tracking, reduces the computational load of target tracking, and is easily implemented on the hardware deployment side. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the target tracking method according to an embodiment of the present invention;

[0032] Figure 2 This is a schematic diagram of the pruning mask in an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Figure 1 This is the overall architecture of this embodiment. The target tracking algorithm network structure adopted in this embodiment is based on the Transormer architecture (Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, AN, ... & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.). Figure 1 As shown. The algorithm mainly consists of three parts: block feature extraction, pruned feature forward propagation, and target localization. The target tracking algorithm proposed in this embodiment is an online tracking algorithm, that is, it processes continuously input images (such as video surveillance images, mobile phone images, color three-channel RGB images, etc.) frame by frame. Assume that the t-th frame of the video is a color image of size N×N pixels. The target position (x) needs to be determined based on the (t-1)th frame. t-1 ,y t-1 In this embodiment of the invention, the location is taken as the center at I. t-1 Extract an image patch with dimensions of 128 pixels as the target template. Yu I t Extract an image patch with dimensions of 256 pixels wide and 256 pixels high as the target search region.

[0035] This invention uses a single-layer convolutional neural network to extract block features (tokens), with a kernel width of 16 pixels and a stride of 16 pixels. After convolution, the first and second dimension features are merged to obtain the target template features for frame t. Search area characteristics Subsequently, the 64 block features of the target template and the 256 block features of the search region are concatenated together to form a feature set with a length of 320 and a feature dimension of 768.

[0036] Feature X tAfter feature mapping using Transformer multi-attention heads, weighted attention weights are learned to correlate the search region and the target template region. In this embodiment, the number of attention heads N is set to 4, and the parameters of each attention head are independent and not shared. and These represent the mapping parameters corresponding to the key, query, and value, respectively. This represents the mapping parameters to the target space. The enhanced feature x is obtained after weighted mapping by the Transformer attention layer. t As shown in Formula 1.

[0037]

[0038] ReLU represents a non-linear activation layer, LN represents a linear connection layer, and the softmax function normalizes the weights to between 0 and 1. d equals 768, representing the block feature dimension. x n Representing feature X t Features after the nth attention head mapping.

[0039] The i-th block feature t i and the j-th block feature t j The probability of a correlation between features x can be expressed as a function x. t The magnitude of the self-attention at matrix index (i,j) is shown in the following equation.

[0040]

[0041] This formula shows that the computational complexity of attention is proportional to the square of the number of block features. Therefore, redundant features need to be removed, i.e., pruning. In this embodiment of the invention, the importance of the i-th block feature is defined as the degree of relevance of other block features to this feature, as defined in the following formula.

[0042]

[0043] Here, L represents the number of block features (i.e., the sum of the number of block features in the target template and the number of block features in the search region), and the correlation is the average of the correlation coefficients between all attention heads and block features. In existing literature, pruning is to sort by the magnitude of the correlation coefficient s and select the top k block features (e.g., 30) to reduce the total number of block features. Unlike previous methods, this embodiment of the invention uses a method based on a learnable threshold (Kim S, Shen S, Thorsley D, et al. Learned token pruning for transformers[J]. arXiv preprint arXiv:2107.00910,2021.) to remove redundant block features.

[0044] The structured pruning method employed in this embodiment of the invention involves learning a threshold with a length equal to the number of pruning layers (l) and a channel count of 1 during the forward propagation process. With θ (m) The threshold value for the m-th pruning layer is used to determine whether the feature at the corresponding location should be forward propagated. The mask required for pruning is as follows: Figure 2 As shown.

[0045] The value of the soft mask is the correlation value s, and the value of the hard mask is obtained by thresholding the soft mask, as shown in the following formula.

[0046]

[0047] Here M (m) (t i ) represents the feature t of the i-th block in the m-th pruning layer. i The resulting hard mask value, s (m) (t i ) represents the feature t of the i-th block in the m-th pruning layer. i The resulting soft mask value. The physical meaning of this formula is that if feature x in the m-th pruned layer... t The soft mask value s of the i-th block feature (m) (t i ), s (m) (t i () greater than the threshold θ (m) If the feature continues to propagate forward in the neural network, then the feature block will not propagate forward.

[0048] For example, suppose a neural network has 50 layers, resulting in a large computational load. To improve computational efficiency, layers 4, 12, 24, 36, and 42 are artificially designated as pruning layers. In this case, layer 4 corresponds to the first pruning layer. Thresholding and masks are applied to the pruning layers to reduce computational load.

[0049] The total loss function required for training the Transformer-based neural network used in this embodiment of the invention is divided into three parts, including the target localization loss. Comparative loss Sum of regularization loss As shown in the following formula.

[0050]

[0051] In the embodiment of this invention, when training the neural network to extract target features, the MoCo memory pool structure (He K, Fan H, Wu Y, et al. Momentum contrast for unsupervised visual representation learning[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2020:9729-9738.) is used, such as... Figure 1 As shown. The template features are about to be defined. Features extracted from non-target template regions (of the same size as the template region) in the image are used as positive samples, while features extracted from these regions are used as negative samples. The network aims to learn the ability to distinguish between positive and negative samples, thereby improving the target localization performance of the neural network under unsupervised conditions. During training, a contrastive loss function is used, as shown in the following equation.

[0052]

[0053] Here, CrossEntropy represents the cross-entropy loss function, x p and x f These represent the label values ​​of the positive and negative sample spaces learned, respectively.

[0054] In addition, to prevent disordered pruning of the network, this embodiment of the invention applies a regularization constraint to the pruning threshold, making the threshold required for pruning learnable. The loss function is shown below.

[0055]

[0056] Here M (m) This represents the hard mask of the m-th pruned layer, and the loss function is... This is the sum of the 1-norms of the hard mask, so the network will learn the threshold in the direction of hard mask sparsity.

[0057] Network training and inference:

[0058] During the training phase, when the network forward propagates features (the features here refer to the block features of the target template and the block features of the search region during training), the masks used during forward propagation are all soft masks (to learn the threshold parameters). Therefore, the network can backpropagate the loss values ​​and learn the network's weight parameters and pruning threshold parameters.

[0059] In this embodiment of the invention, no pruning is performed on the neural network during the training phase; only the threshold parameters and the network's mapping parameters are learned.

[0060] In the inference phase, coarse localization is first performed using pruning. The network forward propagates through hard masks, and the search region features become increasingly sparse with each forward propagation. Subsequently, the network performs fine localization. The search region features after hard mask pruning are sparse features, which are then fed into the regression layer for pixel-level precise target localization. The steps of the inference phase are shown below.

[0061] Step 1: Initialize the network (i.e., train the neural network); extract target template features based on the target ground truth in the given initial frame.

[0062] From frame 2 to the last frame T of the video, assuming the current frame is t, the target template features are: The search area features are

[0063] Step 2: Concatenate the target template features with the search region features to obtain feature x. t Feature enhancement is achieved by incorporating a multi-head attention mechanism.

[0064] Step 3: Perform pruning operations at the m-th pruning layer of the network and propagate the features forward.

[0065] Step 4: Perform regression localization on the pruned features in the final output and output the predicted target location value (x) for this frame. t ,y t ).

[0066] Step 5: Determine if t is the last frame. If not, proceed to step 6; otherwise, proceed to step 7.

[0067] Step 6: Extract the template features of the predicted target location in the current frame. and the search region features of the next frame. Then proceed to step 2.

[0068] Step 7: End.

[0069] In this embodiment of the invention, the original images used in the training phase can be images acquired at historical moments. The original images used in the inference phase are images acquired in real time. The processes for acquiring the target template and search region are similar in both the training and inference phases.

[0070] The performance improvements achieved on current publicly available authoritative target tracking benchmark datasets are shown in the table below. OSTrack is the selected benchmark Transformer algorithm, which uses a sorting pruning method. The learnable threshold pruning method proposed in this embodiment is Ostrack+. On the three test datasets, the optimized algorithm has comparable accuracy to the unoptimized algorithm, while achieving a 1.88x speedup.

[0071] Table 1 shows the results of the method of the present invention on three data test sets.

[0072]

[0073] Another embodiment of the present invention provides a terminal device corresponding to the method of the above embodiments. The terminal device may be a processing device for a client, such as a mobile phone, a laptop, a tablet computer, a desktop computer, etc., to execute the method of the above embodiments.

[0074] The terminal device in this embodiment includes a memory, a processor, and a computer program stored in the memory; the processor executes the computer program in the memory to implement the steps of the method in Embodiment 1 described above.

[0075] In some implementations, the memory may be high-speed random access memory (RAM), and may also include non-volatile memory, such as at least one disk storage device.

[0076] In other implementations, the processor can be any type of general-purpose processor, such as a central processing unit (CPU) or a digital signal processor (DSP), and there is no limitation here.

[0077] Another embodiment of the present invention provides a computer-readable storage medium corresponding to the methods of the above embodiments, wherein a computer program / instructions are stored thereon. When the computer program / instructions are executed by a processor, they implement the steps of the methods of the above embodiments.

[0078] A computer-readable storage medium can be a tangible device that holds and stores instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof.

[0079] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0080] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0081] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0082] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0083] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A lightweight network-based visual target tracking method, characterized in that, Includes the following steps: S1. Obtain the block features of the target template and the block features of the search region; S2. Concatenate the block features of the target template and the block features of the search region to obtain concatenated features. ; for the cascaded features Perform multi-head attention feature mapping to obtain features enhanced by weighted mapping through the attention layer. ; t = 2, 3, ..., T; T represents the number of video frames; S3. Define several layers of the Transformer-based neural network as pruning layers; for the m-th pruning layer, if the features The Soft mask values ​​of individual block features Greater than the threshold If the feature is true, it continues to propagate forward in the Transformer-based neural network; otherwise, the first feature is false. Individual block features do not propagate forward; S4. Repeat step S3 until the final pruned features are obtained; Regression localization is performed on the final pruned features to obtain the target location prediction value of the current frame, i.e., the t-th frame of the video. S5. Determine whether frame t is the last frame. If not, end the process. If yes, extract the target position template features predicted in the current frame and the search area features of the next frame, and return to step S2 until all frames of video have been processed. In step S3, in the m-th pruning layer, the features The Soft mask values ​​of individual block features The calculation formula is: ;in, = L represents the sum of the number of block features in the target template and the number of block features in the search region. yes The probability of self-attention of the block features included, where express The Middle Block features and The Middle Block features The probability of the association between them. , , These represent the mapping parameters for the corresponding features of the index and queue, respectively, and N is the dimension of the original image corresponding to the target template and the search region.

2. The lightweight visual target tracking method according to claim 1, characterized in that, In step S1, the process of determining the target template and the search area includes: assuming the image of the t-th frame of a video at a certain moment... Size is Then, taking the target position determined at the previous moment as the center, the image at the (t-1)th frame of the video... Extract an image block with a length and width of a first set number of pixels as the target template. Extract an image block with a length and width of the second set number of pixels as the search area.

3. The lightweight visual target tracking method according to claim 1, characterized in that, In step S2, the features are enhanced by weighted mapping after attention layer. The expression is: ； in, It is a non-linear activation function. For linear connection operations, To normalize the weights to a function between 0 and 1, Represents the block feature dimension. Representing cascaded features After the first Features after attention head mapping , and These represent the mapping parameters for the features corresponding to the index, queue, and passed value, respectively, and N is the dimension of the original image corresponding to the target template and the search region.

4. The lightweight visual target tracking method according to claim 1, characterized in that, Before step S1, the process also includes: training a Transformer-based neural network; the specific implementation process of training the Transformer-based neural network includes: using the block features of the target template and the block features of the search region as inputs to the Transformer-based neural network, aiming to minimize the loss function, using soft masks in the forward propagation process, and learning the weight parameters and pruning threshold of the Transformer-based neural network.

5. The lightweight visual target tracking method according to claim 4, characterized in that, The loss function of the Transformer-based neural network Set as: ;in, Locate the loss to the target. To compare the losses, For regular loss.

6. The lightweight visual target tracking method according to claim 5, characterized in that, Comparative loss The expression is: ; Represents the cross-entropy loss function. and These represent the label values ​​of the positive and negative sample spaces learned, respectively. Positive samples refer to the block features of the target template, while negative samples refer to the features extracted from the non-target template regions of the image.

7. The lightweight visual target tracking method according to claim 5, characterized in that, Regular loss The expression is: ;in, Indicates the first Hard mask for each pruned layer, loss function That is The sum of the norm of the hard mask, , Indicates the first Hard mask of a pruning layer The Middle Block features The corresponding hard mask value, Indicates the first The first pruning layer Block features The value of the corresponding soft mask.

Images