A method for target tracking in sequential images based on scale-equivariant convolutional Siamese networks

Abstract

The present invention discloses a method for tracking a target in a sequence image based on a scale-equivariant convolutional twin network, which belongs to the technical field of target tracking in a sequence image. The method comprises step 1: training the network learnable parameters of LSE‑SiamRPN to determine the optimal values ​​of the network learnable parameters; step 2: determining the bounding box of the tracked target in the first frame image, inputting the template image into the template branch of the twin network, and extracting the original scale features of the target; step 3: inputting the next frame image, inputting the search area into the search branch of the twin network, and extracting the multi-scale features of the search area; step 4: performing deep cross-correlation between the original scale features of the target and the multi-scale features of the search area, and then using the multi-channel response map as the input of the bounding box regression subnetwork to obtain the bounding box of the target; and then going to step 3. The present invention improves the accuracy of learning the similarity metric between the target and the background, and enhances the adaptability to changes in target size.

Description

Technical Field

[0001] The present invention relates to the technical field of sequential image target tracking in computer vision, and in particular to a sequential image target tracking method based on a scale-equivariant convolutional twin network. Background Art

[0002] Object tracking in sequence images is one of the most challenging basic tasks in the field of computer vision. Its purpose is to accurately, continuously and stably locate the target of interest in the image frames of the image sequence. In the past five years, the mainstream object tracking algorithms can be roughly divided into two categories: algorithms based on correlation filters and algorithms based on twin networks. Compared with the former, the latter has higher accuracy. The earliest proposed tracking algorithm based on correlation filters is the minimum mean square error output and filter (MOSSE) algorithm. Subsequently, algorithms based on Circulant Structure with Kernels (CSK), algorithms based on Kernelized Correlation Filter (KCF), algorithms based on Discriminative Scale Space (DSST) and Discriminative Scale Space Tracker (DSST), and spatially regularized discriminative correlation filters (SRDCF) have appeared one after another.

[0003] The tracking algorithm based on the twin network uses deep learning technology to learn the similarity measurement between the template of the tracked target and the background area in the frame, and calculates the similarity value based on this measurement, and takes the position of the background area with the largest similarity value with the target template as the position of the target. The earliest algorithm of this type is the target tracking algorithm based on the fully convolutional twin network (SiamFC, Fully-Convolutional Siamese Networks for Object Tracking), followed by the target tracking algorithm based on the region proposal network (Region Proposal Network, RPN) (HighPerformance Visual Tracking with Siamese Region Proposal Network, SiamRPN), which transforms the similarity calculation problem of SiamFC into a regression and classification problem by introducing the region proposal network (RPN) in the field of target detection, thereby improving the speed and accuracy. Next is the Siamese network tracking algorithm (Deeper and Wider Siamese Networks for Real-time Visual Tracking, SiamDW) based on a deeper and wider network. It replaces the backbone network used to extract features in SiamFC and SiamRPN with Resnet after center cropping, further improving the accuracy and robustness of tracking at the cost of a certain tracking speed. With the improvement of RPN technology, trackers such as SiamBAN have proposed an anchor-free strategy and no longer use anchors. Compared with anchor-based trackers (such as SiamRPN, SiamDW, etc.), it avoids the hyperparameter settings related to anchor boxes and is more flexible and universal. Therefore, the tracking algorithm based on the twin network framework and using the region proposal network (RPN) is the most accurate and applicable algorithm in the current field of target tracking in sequential images.

[0004] However, in real tracking scenarios, the change of target scale (size) is a common problem. If the algorithm assumes that the target scale is constant, then when the target scale actually becomes smaller, the background information will be mistakenly introduced into the target template, causing the offset or error of target positioning. When the target scale becomes larger, only part of the target area can be located, affecting the tracking accuracy. Based on the twin network framework, this type of tracking algorithm using the region proposal network (RPN) assumes that the similarity measure between the target template and the background area does not change with the change of the target scale, or uses a single scale similarity measure to select the size of the background area with the maximum similarity as the scale of the target during the tracking process; or pre-generates a number of target templates of a limited number of scales, determines the corresponding background areas respectively, and selects the size of the background area with the maximum similarity as the scale of the target. The latter pre-defines the limited number of possible values ​​of the target scale and cannot adapt to the continuous change of the target scale. The assumption that the similarity metric of the former does not change with the change of the target scale has been refuted by the experiment of Ivan Sosnovik et al. in Sosnovik I, Moskalev A, Smeulders A. Scale Equivariance Improves Siamese Tracking [C] / / . 2021IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2021-2021: 2764–2773., that is, the features extracted from the traditional convolutional network will change nonlinearly with the change of the target scale, resulting in the change of the similarity metric. Ivan Sosnovik et al. introduced the scale equivariant convolution operation in the framework of SiamFC to realize the scale equivariance of similarity metric learning, and proposed a scale equivariant full convolution twin network target tracking method (SE-SiamFC, Scale Equivariance SiamFC), which achieved higher tracking accuracy than the original SiamFC. At present, there is no method that uses scale equivariant convolution to solve the problem of target scale change in the framework of the twin network tracking algorithm (SiamRPN) using RPN.

[0005] Aiming at the problem that the existing SiamRPN tracking algorithm based on the twin network framework and using the region proposal network (RPN) cannot adapt to the change of the target scale, the present invention proposes a twin network target tracking method (LSE-SiamRPN, Local-Scale-EquivariantSiamRPN) based on the local scale equivariant structure and using the region proposal network (RPN). The scale convolution layer composed of scale convolution kernels is used to construct a local scale equivariant convolution structure to extract multi-scale features from the video sequence frames; the bounding box regression subnetwork composed of the classification branch, the center metric branch and the regression branch is used to convert the multi-scale features after deep cross-correlation into a single-channel response map, thereby determining the bounding box where the target is located. Summary of the invention

[0006] The purpose of the present invention is to propose a sequence image target tracking method based on a scale-equivariant convolutional twin network, which is characterized by comprising the following steps:

[0007] Step 1: Use a public data set to train the network learnable parameters of LSE-SiamRPN to determine the optimal values ​​of the network learnable parameters; the LSE-SiamRPN includes a twin sub-network, a deep cross-correlation sub-network, and a bounding box regression sub-network;

[0008] Step 2: Determine the bounding box of the tracked target in the first frame image, obtain the template image, input the template image into the template branch of the twin network, and extract the original scale features of the target;

[0009] Step 3: Input the next frame image, obtain the search area, input the search area into the search branch of the twin network, and extract the multi-scale features of the search area; then go to step 4;

[0010] Step 4: Perform deep cross-correlation on the original scale features of the target in step 2 and the multi-scale features of the search area in step 3 to obtain a multi-channel response map, and then use the multi-channel response map as the input of the bounding box regression subnetwork. The classification branch, center metric branch and regression branch of the bounding box regression subnetwork jointly obtain the bounding box of the target; then go to step 3.

[0011] The twin network adopts a local scale equivariant convolution structure; the local scale equivariant convolution structure includes several scale convolution layers, each scale convolution layer is composed of several scale convolution kernels, each scale convolution kernel is obtained by linear weighted summation of a set of basis functions, and each basis function is defined as follows:

[0012]

[0013] Among them, H n and H mare the nth and mth order Hermite polynomials respectively; u and v are the horizontal and vertical coordinates of each element with the center of the kernel as the origin; A is the normalization constant; σ is the scale parameter.

[0014] The step 2 is specifically as follows: taking the center position of the bounding box of the target as the center, cutting the image area larger than the bounding box as the template image; calculating the pixel average value of each channel of the first frame image, and if part of the cut image area is outside the boundary of the first frame image, filling it with the pixel average value; scaling the template image to the size set at the input end of the template branch of the twin network, and inputting it into the template branch, and then passing through the network feedforward, obtaining the original scale feature of the target at the output end of the template branch

[0015] The step 3 is specifically as follows: taking the center position of the bounding box of the target in the previous frame image as the center, cutting the image area larger than the bounding box as the search area, if part of the cut image area is outside the boundary of the current frame image, it is filled with the pixel average value of each channel of the first frame image; scaling the search area to the size set at the input end of the search branch of the twin network, and inputting it into the search branch, and then through the network feedforward, obtaining the multi-scale features of the search area at the output end of the search branch

[0016] The classification branch is composed of a plurality of two-dimensional convolutional layers stacked together, and the classification branch output is dual-channel; the center metric branch is composed of a plurality of two-dimensional convolutional layers stacked together, and the center metric branch output is single-channel; the regression branch is composed of a plurality of two-dimensional convolutional layers stacked together, and the regression branch output is four-channel.

[0017] The step 4 is specifically as follows:

[0018] Multi-scale features of the search area and the original scale characteristics of the target Perform channel-by-channel cross-correlation to obtain a multi-channel response map R;

[0019] The bounding box regression subnetwork inputs the multi-channel response map R into the classification branch, the center metric branch, and the regression branch in parallel;

[0020] Summarize the probability that the position in the search area corresponding to each element of the classification branch of the bounding box regression subnetwork belongs to the target, the proximity of the position in the search area corresponding to each element of the center measurement branch to the target center, and the weight coefficient of the bounding box predicted by each element of the regression branch to obtain a single channel response map q;

[0021] According to the location of the maximum value of the single-channel response map q, the bounding box of the target in the current frame image is calculated.

[0022] The value q of the single channel response graph q at position (i, j) ij for:

[0023] q ij =cls ij ×cen ij ×p ij

[0024] Among them, cls ij is the probability that the position in the search area corresponding to each element of the classification branch belongs to the target; cen ij The center metric branch is the proximity between the position of each element in the search area and the target center; ij The weight coefficient of the bounding box predicted for each element of the regression branch;

[0025] The maximum value q of the single channel response graph q max for:

[0026]

[0027] The beneficial effects of the present invention are:

[0028] The present invention improves the accuracy of the twin network's similarity metric learning between the target and the background when the target size changes, and enhances the ability to adapt to changes in target size. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flowchart of the sequential image target tracking method based on the scale-equivariant convolutional twin network.

[0030] Figure 2 Schematic diagram of LSE-SiamRPN network data flow.

[0031] Figure 3 It is the stage1 module structure and parameters of each layer.

[0032] Figure 4 It is the stage2 module structure and parameters of each layer.

[0033] Figure 5 It is the stage3 module structure and parameters of each layer.

[0034] Figure 6 This is the structural diagram of the NeckDown module.

[0035] Figure 7 The structure and layer parameters of the classification branch and center metric branch of the bounding box regression subnetwork.

[0036] Figure 8 is the structure of the regression branch and the parameters of each layer.

[0037] Fig. 9 (a) and (b) are the OTB100 OPE success rate and OTB100 OPE accuracy curves, respectively.

[0038] Fig.10 (a) and (b) are the “scale change” OPE success rate and “scale change” OPE accuracy curves, respectively.

[0039] Fig.11 (a) and (b) are the “fast moving” OPE success rate and “fast moving” OPE accuracy curves, respectively.

[0040] Fig.12 (a) and (b) are the “low resolution” OPE success rate and “low resolution” OPE accuracy curves, respectively.

[0041] Fig.13 (a) and (b) are the OPE success rate and accuracy curves of “in-plane rotation” OPE, respectively.

[0042] Fig.14 (a) and (b) are the “motion blur” OPE success rate and “motion blur” OPE accuracy curves, respectively.

[0043] Fig.15 (a) and (b) are the “occlusion” OPE success rate and “occlusion” OPE accuracy curves, respectively.

[0044] Fig.16 (a) and (b) are the curves of UAV123 OPE success rate and UAV123 OPE accuracy, respectively.

[0045] Fig.17 (a) and (b) are the OPE success rate and accuracy curves of “scale change” on UAV123 and UAV123, respectively. DETAILED DESCRIPTION

[0046] The present invention proposes a sequence image target tracking method based on a scale-equivariant convolutional twin network. The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0047] Figure 1 Flowchart of the method for tracking target in sequential images based on scale-equivariant convolutional twin network. The data flow diagram of the LSE-SiamRPN network constructed in this embodiment is as follows: Figure 2 As shown in the figure, the template branch and search branch of the twin network share the same network structure and weight coefficients, and are composed of three modules: stage1, stage2, and stage3 respectively.

[0048] The stage1 module consists of four layers: SESConv_Z2_H, BatchNorm3d, ReLU and MaxPool3d. Its structure and parameters are as follows: Figure 3 As shown in the figure, the description of each layer is as follows:

[0049] The SESConv_Z2_H layer is used to extract multiple scale features from the input channels. If the number of input and output channels of this layer is C in , C out , the number of scale parameters is K s , then the working process of this layer is as follows: construct K s The basis function set Ψ corresponding to the scale parameter σ σ , generating C out ×K s A scale convolution kernel κ with learnable parameters σ , the size of each core is C in ×V×V, convolve these kernels with the input channels to obtain (C out ×K s )×U×U, and compress it into C out The size is K s ×U×U output channels. Where V represents κ σ The scale of the last two dimensions is pre-set, and U represents the scale of the last two dimensions obtained by the actual convolution of the result channel. In this embodiment, K is set s =3, and the corresponding three scale parameters are 0.9, 1.26, and 1.764 respectively.

[0050] The BatchNorm3d layer is a three-dimensional batch normalization layer that adjusts the value of each input channel to an output with a mean of 0 and a variance of 1. ReLU (Rectified Linear Unit) is an activation function layer that performs a rectified linear transformation on the input. The MaxPool3d layer performs a three-dimensional maximum pooling operation on each input channel.

[0051] The stage2 module consists of SESConv_H_H, SESConv_H_H_1x1, BatchNorm3d, ReLU and MaxPool3d. Its structure and parameters of each layer are as follows: Figure 4 As shown, the descriptions of SESConv_H_H and SESConv_H_H_1x1 are as follows:

[0052] The SESConv_H_H layer is used to extract cross-scale features from the input channels. If the number of input and output channels of this layer is C in , C out , the number of scale parameters is K s, the cross-scale filter size is ss, then the working process of this layer is as follows: construct K s The basis function set Ψ corresponding to the scale parameter σ σ , generate ss scale convolution kernels κ with learnable parameters σ , the size of each core is C out ×K s ×C in ×V×V, respectively convolve with the input channel and accumulate, and get (C out ×K s )×U×U, and compress it into C out The size is K s ×U×U output channels. Where V represents κ σ The pre-set scale of the last two dimensions, U represents the scale of the last two dimensions obtained by the actual convolution of the result channel.

[0053] The SESConv_H_H_1x1 layer is used to perform 1x1 convolution on the input containing the scale channel and 1x1 convolution on the features of each scale respectively.

[0054] The stage3 module consists of SESConv_Z2_H, SESConv_H_H, SESConv_H_H_1x1, SESMaxPool, Conv2d, BatchNorm3d, BatchNorm2d, ReLU and MaxPool3d. Its structure and parameters of each layer are as follows: Figure 5 As shown in the figure, the SESMaxPool layer performs the maximum pooling operation on the scale dimension, Conv2d is a two-dimensional convolutional layer, and BatchNorm2d is a two-dimensional batch normalization layer.

[0055] Step 1: Use public data sets to train the network learnable parameters of LSE-SiamRPN. The LSE-SiamRPN network proposed in this invention consists of three parts: a twin network, a deep cross-correlation, and a bounding box regression sub-network. Different from the existing SiamRPN-type tracking algorithms, the twin network of LSE-SiamRPN adopts a local scale equivariant convolution structure instead of the common non-scale equivariant convolution network, such as Alexnet, Resnet, etc.;

[0056] The so-called scale equivariant property means that when the target scale in the input image changes, the scale in the feature space changes proportionally. The local scale equivariant convolutional network used in the present invention includes several scale convolutional layers, each of which is composed of several scale convolution kernels. Each scale convolution kernel is obtained by linear weighted summation of a set of basis functions. Each basis function is defined as follows:

[0057]

[0058] Among them, H n is an n-order Hermite polynomial, u, v are the coordinates of each element with the core center as the origin, and A is a normalization constant. For a specific scale parameter σ, by incrementally iterating n, m, a set of basis functions can be obtained:

[0059] Ψ σ ={ψ σ00 ,ψ σ01 ,ψ σ10 ,ψ σ11 ,} (2)

[0060] The scale convolution kernel κ corresponding to the scale parameter σ σ is a linear weighted combination of basis functions in this set:

[0061]

[0062] where ω i is the weighting coefficient, which is a learnable parameter.

[0063] In a scale convolution layer, multiple scale convolution kernels κ corresponding to different scale parameters σ are used. σ By performing convolution, features of different scales can be extracted from the input. By stacking scale convolution layers, a local scale equivariant convolutional network can be constructed to form the twin network of LSE-SiamRPN.

[0064] This embodiment uses the COCO (Common Objects in Context) dataset for training. The COCO dataset is a public dataset provided by Microsoft for image object recognition. It has three types of annotations: object instances, object keypoints, and image captions. This embodiment only uses object instances. The object instance annotations used contain two files: instances_train2017.json and instances_val2017.json, which are used for training and verification respectively. Each file consists of 5 sections:

[0065]

[0066] The "info" section is a summary of the data set and has the following structure:

[0067]

[0068] The "licenses" section is an array containing multiple license instances, which describes the copyright information related to the dataset.

[0069] The "images" section is an array containing multiple image instances, each of which describes an image file in the dataset, for example:

[0070]

[0071]

[0072] The "categories" segment is an array containing multiple category instances, each of which describes an object category in the COCO dataset (the COCO dataset contains a total of 80 object categories):

[0073]

[0074] The "annotations" section is an array containing multiple annotation instances, each of which describes the details of an object that appears in the image file:

[0075]

[0076] If iscrowd = 0, then this annotation instance contains only one object, and segmentation is a polygon array consisting of the coordinates of the object's location in adjacent order; if iscrowd = 1, then this annotation instance contains a group of objects, and segmentation is an RLE (Run Length Encoding) encoded area, indicating the precise location of these objects. area is the area of ​​the object. bbox stores the bounding box of the object in the form of an array, which are the x and y coordinates of the upper left corner of the bounding box and the width and height of the bounding box.

[0077] In this embodiment, various objects annotated by object instances in the COCO dataset are used as image targets for training the network. The annotation instance in the "annotations" section of the json file is used to determine the center position (tcx, tcy) of each target bounding box, as well as the width tw and height th from the "bbox" field of the instance. The size s_z of the target image area and the size s_x of the background image area cut out from the corresponding image (determined by "image_id") are calculated according to the following process:

[0078] First, calculate the size of the enlarged target box according to the following formula:

[0079]

[0080] context_amount is an expansion coefficient used to ensure that the target area is completely cropped. In this embodiment, context_amount=0.5.

[0081] Then calculate s_z:

[0082]

[0083] In this embodiment, the cropped target image area is scaled to a square target template image with a width of exemplar_size=127 (exemplar_size×exemplar_size is the length and width of the input image of the twin network template branch in this embodiment), and the background image area is scaled to a square background image with a width of instanc_size=255 (instanc_size×instanc_size is the length and width of the input image of the twin network search branch in this embodiment), and the ratio scale_z of the target template image and the target image area is calculated according to the following formula:

[0084]

[0085] Finally, calculate the size s_x of the background image area:

[0086]

[0087] From the corresponding image, the area centered at (tcx, tcy) with a size of s_z is affine transformed according to the following transformation matrix to obtain the target template image:

[0088]

[0089] From the corresponding image, the area centered at (tcx, tcy) and with a size of s_x is affine transformed according to the following transformation matrix to obtain the background area image:

[0090]

[0091] Thus, 859,999 pairs of target template and background area images were obtained from a total of 118,287 images. For the background area images in each pair of images, random sample enhancement operations such as translation, scaling, blurring, and color shift were performed; for the target template images, random sample enhancement operations such as translation, scaling, and color shift were performed to enhance the robustness of the algorithm during training.

[0092] Among them, the target template image is randomly scaled in width and height with a probability uniformly distributed within [95%, 105%]; it is randomly translated in the height and width directions of the image with a probability uniformly distributed within [-4, 4] pixels; the RGB three-channel value of each pixel of the target template image is offset according to [4.64467907, 4.73710203, 4.88324118]·υ, [1.72091413, 0.19879334, -1.82968581]·υ, [-0.55919361, 0.98062831, -0.41940627]·υ, where υ is the value of a three-dimensional random variable that conforms to the three-dimensional standard normal distribution, and · represents the vector inner product.

[0093] The background area image is randomly scaled in width and height with a uniform probability within [82%, 118%], and the size of the bounding box bbox of the target corresponding to the background area image in the original image is scaled accordingly; the background area image is randomly translated in the height and width directions of the image with a uniform probability within [-64, 64] pixels, and the position of the bounding box bbox of the target corresponding to the background area image in the original image is translated accordingly; the background area image is randomly translated in the height and width directions of the image with a uniform probability within [-64, 64] pixels, and the position of the bounding box bbox of the target corresponding to the background area image in the original image is translated accordingly; The RGB three-channel values ​​of each pixel are offset according to [4.64467907, 4.73710203, 4.88324118]·υ, [1.72091413, 0.19879334, -1.82968581]·υ, [-0.55919361, 0.98062831, -0.41940627]·υ, where υ is a three-dimensional random vector of a three-dimensional standard normal distribution and · represents the vector inner product; with a probability of 20%, the background area image is blurred by two-dimensional convolution. The size of the convolution kernel ks is a number randomly selected with equal probability from [5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45], and the value of each element in the middle column of the convolution kernel is set to The values ​​of the elements in the middle row are set to wx is a random floating point number uniformly distributed in the range [0.0, 1.0).

[0094] The target template and background area image pair after sample enhancement constitute the network input part of a training sample, and the corresponding target bounding box bbox in the original image is the network output part of the training sample.

[0095] Output from the regression branch of the network The position of the predicted target bounding box is calculated, and then the IOU loss L between it and the target real bounding box bbox is calculated. IOU :

[0096]

[0097] The Intersection in the above formula calculates the area of ​​the intersection of two bounding boxes, and Union calculates the area of ​​the union of two bounding boxes.

[0098] The loss function L of the regression branch reg is defined as

[0099]

[0100] Where I(T(i,j)) is defined as:

[0101]

[0102] In the above formula, T(i,j) and The explanation is in formula (19).

[0103] Based on the position of the predicted bounding box, its center metric C(i,j) can be calculated:

[0104]

[0105] And define the loss function L of the center metric branch cen :

[0106]

[0107] The loss function L of the classification branch cls Defined as the classification branch output cls after Softmax transformation ij , and the cross entropy loss function between the true target background label.

[0108] Based on the loss function L of the regression branch, central metric branch and classification branch defined above reg , L cen and L cls , we can define the total target loss function Loss during training:

[0109] Loss = L cls +λ1×L cen +λ2×L reg (15)

[0110] The weights λ1 and λ2 are respectively set to λ1=1 and λ2=3 in this embodiment.

[0111] This embodiment implements the network in Ubuntu 18.04, NVIDIA GeForce TITAN X GPU, Python 3.6.2, Pytorch 1.2.1, CUDA 9.0 environment, and takes the loss function Loss in formula (15) as the minimization objective function, and uses the stochastic gradient descent method to train and adjust the learnable parameters of the network. Among them, the initial values ​​of the learnable parameters of the template branch and the search branch of the twin network are set to the parameters of the corresponding part of the pre-trained network provided by Sosnovik I, Moskalev A, Smeulders A. Scale Equivariance Improves Siamese Tracking [C] / / . 2021IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2021-2021: 2764–2773. During the training process, only the learnable parameters of the module stage3 are adjusted, while the parameters of stage1 and stage2 remain unchanged.

[0112] The total number of training iterations is set to 20, the batch size is set to 16 samples, and the learning rate is set to start from the initial learning rate of 0.0005 in the first epoch and gradually increase to reach the base learning rate of 0.005 in the sixth epoch. After that, the learning rate gradually decays and returns to 0.0005 in the 20th epoch. The specific learning rates for 20 epochs are as follows:

[0113] 0.0005,0.00079245,0.00125594,0.00199054,0.00315479,0.005,0.00424171,0.00359843,0.0030527,0.00258974,0.00219699,0.0018638,0.00158114,0.00134135,0.00113792,0.00096535,0.00081895,0.00069475,0.00058938,0.0005.

[0114] Step 2: Determine the bounding box of the tracked target in the first frame, obtain the target template image, input the template image into the template branch of the twin network, and extract the original scale features of the target;

[0115] Centered on the target bounding box, crop an image area slightly larger than the bounding box as the target template image, calculate the average value of the pixels in each channel of the first frame image, and fill it with the pixel average value if part of the cropped area is outside the boundary of the first frame image. Scale the template image to the size set at the input of the twin network template branch, input it to the twin network template branch, and after the network feedforward, the original scale feature of the target can be obtained at its output

[0116] According to formulas (4) to (8), the template image of the target is obtained from the first frame image according to the target's bounding box, and input into the template branch, as follows: Figure 2 As shown, in the twin network template branch of this embodiment, stage1( Figure 3 ), stage2( Figure 4 ), stage3( Figure 5 ), is set to extract features at three scales at the same time, and the corresponding three scale parameters are 0.9, 1.26, and 1.764 respectively. After being flattened along the scale dimension by the NeckDown module and reduced in dimension, the size of each output channel is adjusted to 7×7 by bicubic interpolation through the template interpolation module to obtain the original scale features. The structure of the NeckDown module is as follows Figure 6 As shown in the figure, the SESFlatten layer tiles the features along the scale dimension, and the ConvTranspose2d layer is a two-dimensional deconvolution layer.

[0117] Step 3: Get the next frame image, take the center position of the bounding box of the target in the previous frame image as the center, get the search area image, input the search area image into the search branch of the twin network, and extract the multi-scale features of the search area;

[0118] Centered on the center of the bounding box of the target in the previous frame, crop an image area larger than the bounding box as the search area image. If part of the cropped area is outside the boundary of the current frame image, fill it with the average value of the pixels in each channel of the first frame image. Scale the search area to the size set by the input of the twin network search branch and input it to the twin network search branch. After the network feedforward, multiple scale features of the search area can be obtained at its output.

[0119] If the number of selected parameters of different scales is K, then K features of different scales F1(X), F2(X), ..., F of the search area image X can be extracted from the twin network. K (X), concatenate the features of these K scales along the channel dimension to obtain the multi-scale features of the search area Right now:

[0120]

[0121] Among them, Cat is the splicing function;

[0122] According to formulas (4) to (9), the search area image is obtained from the current frame image based on the bounding box of the target in the previous frame image and input into the search branch. Figure 2 As shown, in the twin network search branch of this embodiment, stage1( Figure 3 ), stage2( Figure 4 ), stage3( Figure 5 ), is set to extract features at three scales at the same time (K=3), and the corresponding three scale parameters are 0.9, 1.26, and 1.764 respectively. In order to adapt to the possible size changes of the target in this frame image, after tiling along the scale dimension and dimensionality reduction by the NeckDown module, the size of each output channel of the feature is adjusted to 0.9, 1.26, and 1.764 times of 31×31 respectively through the search area interpolation module in a bicubic interpolation manner, and the three scale features F1(X), F2(X) and F3(X) of the search area are obtained. These three resized features are spliced ​​along the channel dimension to obtain the multi-scale features of the search area. Right now:

[0123]

[0124] Step 4: Perform deep cross-correlation on the original scale features of the target and the multi-scale features of the search area, and use the obtained multi-channel response map as the input of the bounding box regression subnetwork. The classification branch, center metric branch and regression branch of the subnetwork jointly obtain the bounding box of the target.

[0125] The depth cross-correlation is the channel-by-channel cross-correlation of the input features, that is,

[0126]

[0127] Here * represents the deep cross-correlation operation, and R is the multi-channel response map obtained by deep cross-correlation.

[0128] The bounding box regression subnetwork inputs R into the classification branch, centrality metric branch, and regression branch in parallel.

[0129] The regression branch consists of several two-dimensional convolutional layers stacked together to output is a four-channel, w and h are the number of columns and rows of each channel, respectively, where each element T(i,j) = (l,t,r,b), l,t,r,b respectively represent the distance from the corresponding position (x,y) in the search area to the left, top, right, and bottom sides of its target prediction bounding box. Assuming that the coordinates of the upper left corner and the lower right corner of the target prediction bounding box are (x0,y0) and (x1,y1) respectively, (x,y) is the position coordinate in the search area corresponding to the element (i,j), then T(i,j) meets the following formula:

[0130]

[0131] The upper left corner position x0, y0, width bw, and height bh of the predicted bounding box can be calculated through T(i, j):

[0132]

[0133] And then calculate the coefficient of change of the area s of the bounding box predicted by element (i, j) and the area s′ of the bounding box of the target in the previous frame and the aspect ratio of the bounding box predicted by element (i, j) The coefficient of change of the aspect ratio a′ of the target’s bounding box in the previous frame According to s_c and r_c, the weight coefficient p of the bounding box predicted by element (i, j) can be calculated ij :

[0134]

[0135] Where P reg is a preset constant.

[0136] The classification branch consists of several two-dimensional convolutional layers, and the output of the last layer is is a dual channel, which respectively represents the score of the corresponding position in the search area belonging to the target and the background, where w and h represent the width and height of each channel respectively. By performing Softmax transformation, we can obtain the probability cls that the position in the search area corresponding to each element (i, j) in R belongs to the target. ij .

[0137] The center metric branch consists of several two-dimensional convolutional layers stacked together, and the output It is a single channel, reflecting the proximity between the position of each element (i, j) in the search area and the target center. ij .

[0138] Summarize the cls of the classification branch, center metric branch, and regression branch of the bounding box regression subnetwork ij ,cen ij With pij , we can get the single channel response graph q, the value q at the (i,j) position ij It can be calculated according to formula (22).

[0139] q ij =cls ij ×cen ij ×p ij (twenty two)

[0140] Finally, according to the maximum value of the single channel response graph q The location of the target is calculated to obtain the bounding box of the target in this frame image.

[0141] In this embodiment, the K=3 scale features F1(X), F2(X) and F3(X) of the search area are respectively compared with the original scale features of the target. Perform deep cross-correlation, and then adjust the size of each output channel to 25×25 through bicubic interpolation; finally, concatenate the interpolation results along the channel dimension to obtain the multi-channel response map R.

[0142] The structure and parameters of each layer of the classification branch and the center measurement branch of the bounding box regression subnetwork of this embodiment are as follows: Figure 7 As shown in the figure, the structure of the regression branch and the parameters of each layer are as follows Figure 8 As shown in Figure 2, the GroupNorm layer is the group normalization layer proposed in Wu Y, He K. GroupNormalization, 2018. It divides 256 channels into 32 groups for normalization. The classification branch and the center metric branch share Figure 7 The output of the regression branch can be obtained by feeding the bounding box regression subnetwork to the multi-channel response map R. Output of the classification branch Output of the centrality metric branch Then we get T(i,j) of each element (i,j) in the output channel, cls ij and cen ij According to formulas (20) and (21), p ij In this embodiment, the constant P in formula (21) is set to reg = 0.04. According to formula (22), the single channel response graph q can be calculated.

[0143] In this embodiment, the maximum value of the single channel response graph q is calculated. Before finding the location, we first enlarge the 25×25 q to 196×196 q_up by bicubic interpolation, obtain the position coordinates max_r_up and max_c_up of the maximum value in q_up, and calculate its displacement (disp_r, disp_c) relative to the center of the search area according to formula (4).

[0144]

[0145] Since there is a proportional relationship between the size of the search area and the size of the current frame image, the displacement calculated by formula (4) is converted into a displacement in pixels of the current frame image: Since the search area is obtained with the center of the bounding box of the target in the previous frame as the center, this displacement is also the maximum value of the single-channel response map q. The displacement relative to the center of the target position in the previous frame is added to the center position coordinates of the target in the previous frame to obtain the center position coordinates of the target in this frame image.

[0146] When calculating the bounding box size of the target in the current frame, the 25×25 T(i, j) is enlarged to 196×196 T_up(i, j) by bicubic interpolation. Then the height th and width tw of the target bounding box in pixels in the current frame image can be calculated according to formula (5):

[0147]

[0148] The following experiments on public data sets illustrate the performance advantages of this embodiment compared with existing tracking algorithms. On the OTB100 and UAV123 data sets, the OPE Benchmark series of indicators are used as the benchmark, and the well-known kernel correlation filter (KCF), DSST, STRUCK, CSK, OAB, Staple, SAMF, MEEM, SRDCF, and ECO tracking algorithms are compared, as shown below:

[0149] On the OTB100 dataset, the OPE (One Pass Evaluation) success rate plot compared with KCF_Gauss, KCF_LinearHog, STRUCK, CSK, and OAB algorithms is shown in the figure below: Fig. 9 As shown in (a), it can be seen that under most overlap threshold values, the success rate of this embodiment is higher than that of the comparison algorithm; the OPE precision (Precision plot) compared with KCF_Gauss, KCF_LinearHog, DSST, STRUCK, and CSK algorithms is shown in Fig. 9As shown in (b), it can be seen that under most center point location error thresholds, the accuracy of this embodiment is higher than that of the comparison algorithm. The OPE performance curves of the comparison algorithm on the six subdivision challenge data sets of scale variation, fast motion, low resolution, in-plane rotation, motion blur, and occlusion are compared as shown in the figure below. Figures 10 to 15 As shown, it can be seen that in all the comparison figures, the success rate and accuracy of this embodiment are higher than the comparison algorithms in most cases. This embodiment achieved an average success rate of 51.0% and an accuracy of 69.7% on the OTB100 dataset, exceeding all the comparison algorithms.

[0150] On the UAV123 dataset, the OPE success rate of this embodiment and the SRDCF, MEEM, SAMF, DSST, STRUCK, KCF, and CSK tracking algorithms is shown in the figure below: Fig.16 (a) shows the OPE accuracy curve. Fig.16 (b) is shown. The OPE performance curve comparison of the comparison algorithm on the scale variation sub-classification dataset of the UAV123 dataset is shown in the figure below. Fig.17 As shown. It can be seen in all the comparison figures that the success rate and accuracy of this embodiment are higher than the comparison algorithms in most cases. This embodiment achieved an average success rate of 49.7% and an accuracy of 70.0% on the UAV123 dataset, which also exceeded all the comparison algorithms.

[0151] Test data on a public data set used to compare algorithm performance show that, compared with existing tracking algorithms, the reliability and accuracy of tracking of the algorithm proposed in the present invention are greatly improved when the target size changes.

[0152] This embodiment is only a preferred specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for tracking target in sequential images based on a scale-equivariant convolutional twin network, characterized in that: The following steps are involved: Step 1: Use a public data set to train the network learnable parameters of LSE-SiamRPN to determine the optimal values ​​of the network learnable parameters; the LSE-SiamRPN includes a twin sub-network, a deep cross-correlation sub-network, and a bounding box regression sub-network; Step 2: Determine the bounding box of the tracked target in the first frame image, obtain the template image, input the template image into the template branch of the twin network, and extract the original scale features of the target; The step 2 specifically includes: taking the center position of the bounding box of the target as the center, cutting out an image area larger than the bounding box as a template image; Calculate the pixel average of each channel of the first frame image. If the partially cropped image area is outside the boundary of the first frame image, fill it with the pixel average. Scale the template image to the size set by the input end of the template branch of the twin network and input it to the template branch. After the network is fed forward, the original scale feature of the target is obtained at the output end of the template branch. Step 3: Input the next frame image, obtain the search area, input the search area into the search branch of the twin network, and extract the multi-scale features of the search area; Then go to step 4; The step 3 is specifically as follows: taking the center position of the bounding box of the target in the previous frame image as the center, cutting the image area larger than the bounding box as the search area, if part of the cut image area is outside the boundary of the current frame image, it is filled with the pixel average value of each channel of the first frame image; scaling the search area to the size set at the input end of the search branch of the twin network, and inputting it into the search branch, and then through the network feedforward, obtaining the multi-scale features of the search area at the output end of the search branch Step 4: Perform deep cross-correlation on the original scale features of the target in step 2 and the multi-scale features of the search area in step 3 to obtain a multi-channel response map, and then use the multi-channel response map as the input of the bounding box regression subnetwork. The classification branch, center metric branch and regression branch of the bounding box regression subnetwork jointly obtain the bounding box of the target; Then go to step 3.

2. According to claim 1, the method for tracking target in sequential images based on scale-equivariant convolutional twin networks is characterized in that: The twin network adopts a local scale equivariant convolution structure; the local scale equivariant convolution structure includes several scale convolution layers, each scale convolution layer is composed of several scale convolution kernels, each scale convolution kernel is obtained by linear weighted summation of a set of basis functions, and each basis function is defined as follows: Among them, H n and H m are the nth and mth order Hermite polynomials respectively; u and v are the horizontal and vertical coordinates of each element with the center of the kernel as the origin; A is the normalization constant; σ is the scale parameter.

3. The method for tracking target in sequential images based on scale-equivariant convolutional twin networks according to claim 1, characterized in that: The classification branch is composed of a plurality of two-dimensional convolutional layers stacked together, and the classification branch output is dual-channel; the center metric branch is composed of a plurality of two-dimensional convolutional layers stacked together, and the center metric branch output is single-channel; the regression branch is composed of a plurality of two-dimensional convolutional layers stacked together, and the regression branch output is four-channel.

4. The method for tracking target in sequential images based on a scale-equivariant convolutional twin network according to claim 1, characterized in that: The step 4 is specifically as follows: Multi-scale features of the search area and the original scale characteristics of the target Perform channel-by-channel cross-correlation to obtain a multi-channel response map R; The bounding box regression subnetwork inputs the multi-channel response map R into the classification branch, the center metric branch, and the regression branch in parallel; Summarize the probability that the position in the search area corresponding to each element of the classification branch of the bounding box regression subnetwork belongs to the target, the proximity between the position in the search area corresponding to each element of the center measurement branch and the target center, and the weight coefficient of the bounding box predicted by each element of the regression branch, and obtain the single-channel response map q; According to the location of the maximum value of the single-channel response map q, the bounding box of the target in the current frame image is calculated.

5. The method for tracking target in sequential images based on a scale-equivariant convolutional twin network according to claim 4, characterized in that: The value q of the single channel response graph q at position (i, j) ij for: q ij =cls ij ×cen ij ×p ij Among them, cls ij is the probability that the position in the search area corresponding to each element of the classification branch belongs to the target; cen ij The center metric branch is the proximity between the position of each element in the search area and the target center; ij The weight coefficient of the bounding box predicted for each element of the regression branch; The maximum value q of the single channel response graph q max for:

Images