A lightweight target tracking method

Abstract

The application discloses a kind of lightweight target tracking methods, the method first obtains the video of target tracking, defines initial template and dynamic template, and constructs template set.Secondly, read search image, template set and search image are input into pre-processing module, obtain output result a, and input output result a into target prediction module, the center point and boundary box of target are predicted.Then search feature is input into online target predictor to obtain prediction result b, and b is linearly fused with the center point to obtain the final center point prediction result, complete target tracking.Finally, search image and prediction result b are input into template updater, and the template updater determines whether search image is used as new dynamic template according to update strategy.The application reduces the calculation amount and parameter amount of the model, enhances the ability to deal with complex environment and large target change scene, and improves the success rate and accuracy of tracking.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of target tracking in computer image, and particularly relates to a lightweight target tracking method. BACKGROUND

[0002] Target tracking is an important research direction in the field of computer vision, and is widely used in video surveillance, automatic driving, human-computer interaction and other fields. The purpose of target tracking algorithm is to automatically track a specific target in a video sequence, which is usually an object or a pedestrian. In recent years, the development of deep learning technology has significantly improved the accuracy of target tracking algorithms based on neural networks, which has become the mainstream method of target tracking algorithms.

[0003] The fully convolutional classification regression neural network SiamCAR is a target tracking algorithm based on deep learning. It first adopts the structure of a twin network, processes the template image and the search image through a weight-shared feature extraction network, and fuses the intermediate features of multiple layers to obtain two feature maps, and calculates the similarity feature map between the two feature maps. Secondly, a feature enhancement network stacked by multiple ordinary convolutions is used to enhance the similarity feature map. Finally, the center point of the target and the size of the target are predicted from the enhanced similarity feature map to realize target tracking. However, due to the complex structure of the SiamCAR model, the large amount of calculation and the large number of parameters, its application in some scenes with limited running resources is limited, and the SiamCAR model does not update the template image during tracking, which makes its accuracy low in complex environments and large target changes. SUMMARY

[0004] In view of the above problems, the present application optimizes the SiamCAR algorithm based on a feature enhancement network with depth separable convolution and spatio-temporal attention fusion, and a runtime template updating method, and proposes a lightweight target tracking method, which reduces the calculation amount and the number of parameters of the model.

[0005] A lightweight target tracking method, comprising the following steps:

[0006] Step 1: Obtain a video to be tracked, define the first frame of the video as an initial template, copy the initial template to define the first dynamic template, and place the initial template and the dynamic template in a set to construct a template set. Each frame image except the first frame is sequentially used as a search image for target tracking.

[0007] Step 2: Read the search image, input the template set and the search image into a preprocessing module to obtain an output result a. The preprocessing module includes a feature extractor and a feature similarity calculator.

[0008] Step 2.1: In the preprocessing module, the template set extracts features using the feature extractor t1 to obtain initial features and dynamic features.

[0009] The search image extracts features through the same feature extractor t2 as t1 to obtain search features.

[0010] The structures of the two feature extractors t1 and t2 are based on ResNet50, and the last layer of the network is removed.

[0011] Step 2.2: The three features extracted in step 2.1 are input into the feature similarity calculator for similarity calculation. The feature similarity calculator calculates the similarity of the initial features, dynamic features, and search features respectively, and selects the feature with the highest similarity as the output result a.

[0012] Step 3: The output result a is input into the target prediction module to predict the center point and bounding box of the target.

[0013] Step 3.1: In the target prediction module, a is input into the feature enhancer b1, and then the output features of b1 are input into the target box predictor to obtain the bounding box of the target.

[0014] Step 3.2: a is input into the same feature enhancer b2 as b1, and then the output features of b2 are input into the target center predictor to obtain the center point of the target.

[0015] Step 4: The search features are input into the online target predictor to obtain the prediction result b.

[0016] Then b is linearly fused with the center point prediction result in step 3.2 to obtain the final center point prediction result, and the target tracking is completed according to the bounding box obtained in step 3.1 and the final center point prediction result.

[0017] Step 5: The search image and the prediction result b in step 4 are input into the template updater, and the template updater decides whether to update the dynamic template according to the update strategy.

[0018] The present application makes the following contributions:

[0019] 1. The present application reduces the computational complexity and parameter quantity of the model through the feature enhancement network in the feature enhancer, which combines depth separable convolution and spatio-temporal attention.

[0020] 2. The present application adds a template updating operation in the tracking stage, which enhances the ability of the algorithm to deal with complex environments and large target changes, and improves the success rate and accuracy of the tracking algorithm.

[0021] 3. The application reduces the calculation amount of the model by 68.3% and the parameter amount by 71%, and improves the success rate and precision rate of tracking in the public data set OTB100. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more intuitively illustrate the details of the application, the following will briefly introduce the drawings.

[0023] Figure 1 is the overall network structure diagram of the application;

[0024] Figure 2 is the flow chart of the specific operation process of the application;

[0025] Figure 3 is the intersection-over-union curve diagram of the application and SiamCAR on the OTB100 data set;

[0026] Figure 4 is the accuracy curve diagram of the application and SiamCAR on the OTB100 data set. DETAILED DESCRIPTION

[0027] The application will be further described below in combination with the drawings. The overall network structure of the application is shown in Figure 1 , the running flow chart is shown in Figure 2 , and the specific steps are as follows:

[0028] Step 1: Obtain the video to be tracked, define the first frame image of the whole video as the initial template, copy a copy as the initial dynamic template, and place the initial template and the dynamic template in a set and define it as the template set. Except for the first frame, each frame image is sequentially used as a search image for target tracking.

[0029] Step 2: As shown in Figure 1 , read the search image, and input the template set and the search image into the preprocessing module. The preprocessing module includes a feature extractor and a feature similarity calculator, which are respectively used to extract image features and calculate the similarity of input features for subsequent prediction.

[0030] Firstly, the template set and the search image are respectively extracted by the same feature extractor, and the initial feature and the dynamic feature are extracted from the template set; the search feature is extracted from the search image. The structure of the feature extractor is based on ResNet50. Since the tracking task does not need to output classification when extracting features and the relatively coarse-grained features can better meet the needs of the tracking task, the last layer network of ResNet50 is removed. Moreover, unlike SiamCAR using a multi-feature fusion feature extractor, the present application only uses the final output result for subsequent operations on the basis of removing the last layer network of ResNet50. The above method reduces the parameter quantity and the calculation quantity of the feature extractor.

[0031] Then, the feature similarity calculator calculates the similarity of the initial template and the dynamic template in the template set after extracting the features and the search image respectively, and selects the highest similarity result as the output result a. The feature calculator is composed of a single layer convolution, and the initial template and the dynamic template are respectively used as the convolution kernel to input the search image for convolution operation to achieve the purpose of calculating the similarity, generating two similarity results, and taking the feature with the larger similarity peak value as the final output result a.

[0032] Step 3: input a into the target prediction module to predict the center point and the bounding box of the target. The target prediction module is composed of a feature enhancer, a target box predictor and a target center predictor. The feature enhancer is used to enhance the output result a in step 2 to improve the accuracy and precision of subsequent prediction. The network structure of the feature enhancer is based on depth separable convolution and spatial-temporal attention CBAM, and the complete structure is sequentially composed of depth separable convolution, group normalization, ReLU activation function, spatial-temporal attention CBAM, group normalization and ReLU activation function, wherein the size of the convolution kernel used in the depth convolution process of the depth separable convolution is 3*3. Compared with the ordinary convolution used in SiamCAR, the calculation amount of the depth separable convolution is reduced to 1 / 9. The use of group normalization reduces the influence of the batchsize parameter on the accuracy of the present application. The present application uses spatial-temporal attention CBAM to improve the accuracy at the cost of a small amount of calculation. The size of the convolution kernel used by the spatial-temporal attention CBAM is 7*7.

[0033] Firstly, a is input into the feature enhancer b1, and then the output feature of b1 is input into the target center predictor to obtain the center point of the target. The target center predictor is composed of a single layer convolution, and the size of the convolution kernel is 3*3. The target center predictor takes the enhanced feature as the input and outputs the predicted target center point, which is used to determine the center position of the target.

[0034] In parallel, input a into the same feature enhancer b2 as b1, and then input the output feature of b2 into the target box predictor to obtain the bounding box of the target. The bounding box is used to represent the size of the target. The target box predictor is composed of a single-layer convolution, and the size of the convolution kernel is 3*3. The target box predictor takes the enhanced feature as input and outputs the predicted target bounding box, which is used to frame the size of the target.

[0035] Step 4: input the search image into the template update module, output the prediction result b, and linearly fuse the prediction result in step 3.2 to obtain the final center point prediction result. As shown in Figure 1 The template update module is composed of a feature extractor, an online target predictor, and a template updater. The feature extractor is the same as the feature extractor used in step 2. The online target predictor is the Classifier used in the DROL tracking algorithm. The Classifier is composed of a spatial-temporal attention and a normal convolution, and the specific order is normal convolution (convolution kernel is 1*1)-spatial-temporal attention CBAM module-normal convolution (convolution kernel is 4*4). The Classifier can provide the prediction result as an auxiliary in step 3 of the target box predictor and the target center predictor to further improve the accuracy. The template updater combines the confidence score given by the online target predictor and the update strategy set in the template updater to determine whether to use the search image as a new dynamic template.

[0036] Further, the template update module takes the search image as input. First, the search feature is input into the online target predictor to obtain the confidence score of the target center prediction. The confidence score has the same size as the output of the center predictor in step 3. Then, the two are linearly fused to obtain the final center prediction result. The linear fusion formula is as follows:

[0037] o = (1-λ)O cen + λO C

[0038] Where o represents the final center prediction result, O cen represents the result of the target center predictor, and O C represents the result of the online target predictor, and λ is a floating point number in [0, 1], representing the fusion weight.

[0039] Step 5: the template update module determines whether to use the search image as a new dynamic template according to the update strategy, and returns to step 2. The update strategy can be described as follows: first, count from the start of the target tracking method, and select the search image with the highest online target predictor confidence score in T u frames every T u frames, and record the prediction result d of the corresponding search image c in step 3. Tu is a positive integer, representing the interval of dynamic template updating, the present application sets T u = 5. Then, the intersection over union (IOU) score of the rectangular prediction frame composed of the prediction result and the target actual bounding box is calculated. If the intersection over union is greater than the set threshold TAU_R, the search image c is taken as the new dynamic template, and the present application sets TAU_R = 0.6. If the intersection over union is not greater than the threshold TAU_R, the dynamic template is not updated. The above updating strategy makes the present application different from SiamCAR in that the present application uses not only the initial template but also the dynamic template. Since the dynamic template is updated in real time during the running process and the IOU intersection over union result is taken as the reference to avoid the pollution of the subsequent tracking process by the samples that are not conducive to tracking, the dynamic template provides the latest state (center position and size) of the tracked target for the algorithm, so that the dynamic template can output a calculation result with a higher peak when the similarity calculation is performed with the search image in step 2, thereby enhancing the ability of the algorithm to deal with the scene with complex environment and large target change and improving the success rate and accuracy of the tracking algorithm.

[0040] Figure 3 The tracking success rate of the present application and the SiamCAR method is shown. The tracking success rate index refers to the intersection over union of the predicted bounding box and the real bounding box. The higher the intersection over union, the more accurate the predicted bounding box. The horizontal axis represents a specific intersection over union threshold, and greater than the threshold is considered to be successful tracking, and the corresponding vertical axis represents the tracking success rate of the whole video under the threshold. The value in the lower left corner is the average tracking success rate of the present application and the SiamCAR method. The present application improves the average tracking success rate by 0.7% compared with SiamCAR.

[0041] Figure 4 The tracking accuracy of the present application and the SiamCAR method is shown. The tracking accuracy index is the error (pixels) of the predicted center point and the real center point. The smaller the error, the more accurate the predicted center point. The horizontal axis represents a specific error threshold, and less than the threshold is considered to be accurate tracking, and the corresponding vertical axis represents the tracking accuracy of the whole video under the threshold. The value in the lower left corner is the tracking accuracy of the present application and the SiamCAR method when the error threshold is 20. The present application improves the average tracking accuracy by 3.1% compared with SiamCAR.

[0042] Table 1

[0043]

[0044] Table 1 shows the comparison of the present application and SiamCAR in precision, success rate, computation and parameter amount, the present application reduces the computation amount of the model by 68.3% and the parameter amount by 71%, while improving the success rate and precision of tracking in the public data set OTB100.

Claims

1. A lightweight target tracking method, characterized by, The method comprises the following steps: Step 1: obtaining a video for target tracking, defining a first frame image of the video as an initial template, copying the initial template to define a starting dynamic template, and placing the initial template and the dynamic template in a set to construct a template set; Step 2: reading a search image, inputting the template set and the search image into a preprocessing module to obtain an output result a, and the specific process is as follows: Step 2.1: in the preprocessing module, the template set uses a feature extractor t1 to extract features to obtain initial features and dynamic features; the search image is subjected to feature extraction by a feature extractor t2 identical to t1 to obtain search features; Step 2.2: a feature similarity calculator calculates the similarity of the initial features, the dynamic features and the search features respectively, and selects the highest similarity feature as the output result a; Step 3: inputting the output result a into a target prediction module to predict the center point and the bounding box of the target; The target prediction module is composed of a feature enhancer, a target box predictor and a target center predictor; The network structure of the feature enhancer is based on depth separable convolution and CBAM (Channel and Spatial Attention), and the complete structure is sequentially composed of depth separable convolution, group normalization, ReLU activation function, CBAM, group normalization and ReLU activation function; Step 4: inputting the search features into an online target predictor to obtain a prediction result b, linearly fusing b with the center point prediction result in step 3 to obtain a final center point prediction result, and completing target tracking; Step 5: inputting the search image and the prediction result b in step 4 into a template updater, and the template updater determines whether to use the search image as a new dynamic template according to an update strategy to update the dynamic template; The update strategy is specifically described as follows: first, counting from the start of the target tracking method, every T u frame selection this T u frame, the search image with the highest confidence score of the online target predictor is recorded as c, and the prediction result d of the corresponding search image c in step 3 is recorded, T u is a positive integer, representing the interval of dynamic template updating; secondly, the intersection-over-union of the rectangular prediction frame composed of the prediction result and the actual boundary frame of the target is calculated, if the intersection-over-union is greater than a set threshold TAU_R, the search image c is taken as a new dynamic template, otherwise, the dynamic template is not updated.

2. The lightweight target tracking method of claim 1, wherein, In step 1, each frame image except the first frame is sequentially used as a search image for target tracking.

3. The lightweight target tracking method of claim 2, wherein, In step 2.1, the structures of the two feature extractors t1 and t2 are based on ResNet50, and the last layer network of ResNet50 is removed.

4. The lightweight target tracking method of claim 3, wherein, The specific process of step 3 is as follows: Step 3.1: in the target prediction module, inputting a into a feature enhancer b1, and then inputting the output features of b1 into a target box predictor to obtain the bounding box of the target; Step 3.2: inputting a into a feature enhancer b2 identical to b1, and then inputting the output features of b2 into a target center predictor to obtain the center point of the target.

Images