A target tracking method and device based on a twin network and multi-prediction output alignment

Abstract

The application discloses a target tracking method and device based on a twin network and multi-prediction output alignment, and comprises the following steps: acquiring a video; pre-processing the video to extract a template image Z and a search area image X; wherein the image centered on the target extracted from the first frame of the video sequence is taken as the template image Z, and the image centered on the target position predicted in the previous frame in the subsequent frame is taken as the search area image X; inputting the template image Z and the search area image X into a pre-trained target tracking network model; and determining a target tracking result according to the output of the target tracking network model. The target tracking network model can fine-tune the target bounding box predicted by the basic tracker through a supplementary module, generate a bounding box capable of tightly wrapping the target, and then perform more accurate target region prediction.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and object tracking, and particularly relates to a target tracking method and device based on a multi-prediction output alignment of a twin network. BACKGROUND

[0002] Object tracking is a basic and challenging task in the field of computer vision, and is one of the most active research topics in the field of computer vision in recent years. The task definition of object tracking is that a video sequence can accurately track the target in each subsequent frame given only the initial frame position of the tracking target. Object tracking has a wide range of applications in the fields of automatic driving, video monitoring, ocean exploration, medical imaging, etc., and therefore is of great concern to the academic and industrial communities. Object tracking can be divided into two main branches: one is based on correlation filtering, and the other is based on deep neural networks.

[0003] Unlike correlation filter-based trackers, the offline training and online tracking method based on a twin network achieves a good balance between tracking accuracy and inference speed. The twin network tracker uses two parallel input branches (template branch and search branch) to extract the deep feature representation of the template image and the search region image. Then, the feature maps output by the two branches are matched for similarity, producing a response map (similarity map). The greater the response value at a certain position on the response map, the more likely the target is located at the corresponding position on the search region.

[0004] The most original twin network tracker SiamFC (Bertinetto L, Valmadre J, Henriques J F, et al. Fully-Convolutional Siamese Networks for Object Tracking [C] / / European Conference on Computer Vision. Springer, Cham, 2016.) uses a fully connected network for feature extraction, cross-correlation calculation and multi-scale prediction. Later, the design of SiamRPN (Bo L, Yan J, Wei W, et al. High Performance Visual Tracking with Siamese Region Proposal Network [C] / / 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2018.) introduces the region proposal network (Region Proposal Network) commonly used in target detection tasks into the twin network structure, achieving more accurate target scale estimation, and the prediction method of the RPN's double branch output also becomes popular. This divide-and-conquer prediction method converts target tracking into two sub-tasks: classification and regression. The twin network model produces a similarity map on the frame by calculating the similarity between the template and the search area (implemented by cross-correlation calculation), and additionally outputs the object bounding box prediction corresponding to each point on the map (regression). In this case, the generated bounding box corresponding to the highest confidence is considered as the minimum circumscribed rectangle of the tracked target on the current frame. SiamBAN (Chen Z, Zhong B, Li G, et al. Siamese Box Adaptive Network for Visual Tracking [C] / / 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020.) introduces the anchor-free prediction method in the target detection field into the twin network tracker, greatly reducing the parameter quantity of the tracking model while retaining the efficient inference mode of the double-branch prediction method.

[0005] However, this kind of divided boundary box prediction method has disadvantages. The prediction of confidence score and the prediction of boundary box are completely separated, and there is no information interaction between the two branches corresponding to the network. This further leads to the fact that the target boundary box corresponding to the position with the highest confidence score may not be optimal (a part of the target is not labeled by the boundary box or the boundary box drifts). SUMMARY

[0006] To solve the pain point of the inconsistency of multiple prediction outputs of the twin network tracking of the double-branch prediction structure, and to further improve the tracking performance of the tracker, the application provides a target tracking method and device based on multiple prediction output alignment of a twin network. The commonly used twin network is regarded as a basic tracker, and a supplementary module is connected after the output layer of the basic tracker, which is used to align the multiple prediction outputs of the basic tracker, and can further refine the target boundary box of the one-stage prediction to realize high-quality boundary box evaluation, so as to accurately realize the function of target tracking. A large number of experiments on four target tracking evaluation data sets OTB100, VOT2018, VOT2019 and GOT10K prove that the application has superior performance compared with recent advanced trackers.

[0007] Technical scheme: To solve the above technical problems, the technical scheme adopted by the application is:

[0008] In a first aspect, a target tracking method based on multiple prediction output alignment of a twin network is provided, comprising:

[0009] Obtaining a video;

[0010] Pretreating the video to extract a template image Z and a search region image X; wherein the image centered on the target extracted from the first frame of the video sequence is taken as the template image Z, and the image centered on the predicted target position of the previous frame in the subsequent frame is taken as the search region image X;

[0011] Inputting the template image Z and the search region image X into a pre-trained target tracking network model;

[0012] Determining a target tracking result according to the output of the target tracking network model;

[0013] The processing process of the target tracking network model comprises:

[0014] The template image Z is extracted by a first feature extraction module to obtain a template image feature Z f ; the search region image X is extracted by a second feature extraction module to obtain a search region image feature X f ;

[0015] The template image feature Z fand search region image feature X f After similarity matching by the per-depth cross-correlation module, output the classification response map R cls and regression response map R reg ;

[0016] classification response map R cls and regression response map R reg After processing by the first prediction head and the second prediction head respectively, output the single-channel foreground-background classification score map M cls and four-channel bounding box prediction map M reg , wherein the four-channel bounding box prediction map M reg indicates n sub-regions on the search region image, which, in an ideal state, contain the same target as the target region on the template image, and the sub-region bounding box is denoted as {P n , and the target region bounding box in the first frame is denoted as b.

[0017] Based on the target region bounding box b and the sub-region bounding box {P n , the first PrPool module and the second PrPool module are respectively used to extract region features from the template image feature Z f and the search region image feature X f , to obtain template region features Z r and sub-region features {X r} n ;

[0018] The per-pixel cross-correlation module performs fine-grained feature retrieval on the sub-region features {X r} n based on the template region features Z r , to obtain the response map {R me} n ;

[0019] Input {R me} n into the matching evaluation branch to perform feature compression and information integration, to obtain the bounding box evaluation score S.

[0020] Fuse the foreground-background classification score map M cls and the bounding box evaluation score S by element-wise multiplication to obtain the bounding box evaluation score map S′.

[0021] Based on the position l of the maximum value of the bounding box evaluation score map S′, determine a unique set of target bounding box coordinates P l on the four-channel bounding box prediction map M reg , i.e., the sub-region features {X r} nDetermine the unique target sub-region feature (X) r ) l ; Target sub-region features (X r ) l With template region feature Z r Perform fine-grained feature retrieval to obtain the response map R′ corresponding to the target bounding box region. me :

[0022] Utilizing refined branches to analyze the response map R′ corresponding to the target bounding box region me Channel compression, feature integration, and channel-by-channel summation are performed to obtain the compensation amount O for the bounding box position and size:

[0023] Based on the compensation amount O, the original predicted target bounding box coordinates P l After making corrections, the final target prediction box is obtained, thus achieving target tracking.

[0024] In some embodiments, template image feature Z f and search region image features X f After similarity matching via a depthwise cross-correlation module, the classification response map R is output. cls and regression response plot R reg ,include:

[0025] R cls =DW cls (Z f ,X f )

[0026] R reg =DW reg (Z f ,X f )

[0027] Where Z represents the template image, X represents the search region image, and f(·) represents feature extraction. cls and DW reg These represent the depthwise cross-correlation calculation process for the classification branch and the regression branch, respectively.

[0028] In some embodiments, the classification response graph R cls and regression response plot R reg Each passed through the first prediction head Conv cls Second prediction head Conv reg After processing, the output is a single-channel foreground / background classification score map M. cls And four-channel bounding box prediction map M reg ,include:

[0029] M cls =Conv cls (Rcls )

[0030] M reg =Conv reg (R reg )

[0031] The first prediction head Conv cls Second prediction head Conv reg .

[0032] In some embodiments, based on the target region bounding box b and the sub-region bounding box {P} n The first PrPool module and the second PrPool module are used respectively to process the template image features Z. f and search region image features X f Perform region feature extraction to obtain template region features Z of the same spatial size. r and sub-region features {X r} n ;

[0033] Z r =PrPool(Z) f b)

[0034] {X r} n =PrPool(X f ,{P} n )

[0035] The pixel-wise cross-correlation module uses template region features Z r Based on the baseline, for the sub-region feature {X r} n Perform fine-grained feature retrieval to obtain the response map {R} corresponding to the bounding box region. me} n ;

[0036] {R me} n =PW(Z) r ,{X r} n )

[0037] Where {R me} n For the response maps corresponding to n bounding box regions, PW represents the pixel-wise cross-correlation operation;

[0038] {R me} n Input the matching evaluation branch, perform feature compression and information integration, and obtain the bounding box evaluation score S;

[0039] S = MLP(Conv)me ({R me} n ))

[0040] where MLP represents a fully connected layer, Conv me represents a stacked convolutional layer; the bounding box evaluation score S is a set of scores related to the matching degree of the region features and the template features, representing the probability that the corresponding bounding box region exists foreground objects;

[0041] The foreground-background classification score map M cls and the bounding box evaluation score S are fused by element-wise multiplication, and the bounding box evaluation score map S′ is obtained;

[0042] S′=S⊙M cls

[0043] Based on the position l of the maximum value of the bounding box evaluation score map S′, a unique set of target bounding box coordinates P reg is determined on the four-channel bounding box prediction map M l , that is, the unique target sub-region feature (X r ) n is determined in the sub-region feature set {X r} l ; the target sub-region feature (X r ) l is subjected to fine-grained feature retrieval with the template region feature Z r , and a response map R′ me corresponding to the target bounding box region is obtained:

[0044] R′ me =PW(Z r ,(X r ) l )

[0045] The response map R′ me corresponding to the target bounding box region is subjected to channel compression and feature integration Conv pr and channel-by-channel summation Sum processing by using the refining branch, and a compensation amount O of the bounding box position and size is obtained:

[0046] O=Sum(Conv pr (R′ me ))

[0047] where O=(o x ,o y ,o w ,o h ) is the compensation amount of the bounding box position and size, o x ,o y is the compensation amount of the horizontal and vertical coordinates of the bounding box position, ow ,o h is a compensation amount for the width and height dimensions of the bounding box;

[0048] Based on the compensation amount O, the original predicted target bounding box coordinates P l are corrected to obtain the final target prediction box, achieving target tracking.

[0049] In some embodiments, the method for constructing the target tracking network model comprises:

[0050] The target tracking network model is a two-stage network comprising a first-stage basic tracker and a second-stage supplementary module;

[0051] The basic tracker comprises two input branches: a template branch and a search branch, which respectively take a template image and a search region image as input; the first feature extraction module of the template branch and the second feature extraction module of the search branch both use the first 4 groups of convolution layers of ResNet50, followed by a cross-depth cross-correlation module and a double-branch parallel prediction head based on anchor-free boxes: a first prediction head and a second prediction head; the basic tracker is a standard double-branch prediction twin network structure;

[0052] The supplementary module is cascaded after the basic tracker and is a twin network structure with double input and double output, using a first PrPool module and a second PrPool module for region feature extraction, followed by a pixel-by-pixel cross-correlation module and two output branches in parallel, which are a matching evaluation branch and a refining branch.

[0053] In some embodiments, the method for training the target tracking network model comprises:

[0054] The total loss function Loss1 of the first stage is:

[0055]

[0056] wherein, * represents the training label, λ1 and λ2 represent the weights for balancing the loss of each subtask, L ce , L IoU are the CE loss and the IoU loss, respectively, represent the training expected foreground-background classification score map and four-channel bounding box prediction map, respectively;

[0057] The training loss function Loss2 of the second stage is:

[0058] Loss2=λ3L ce (S,S * )+λ4L1(O,O * )

[0059] wherein,* represents a training label, L ce and L1 are CE loss and L-1 distance loss, respectively, and λ3 and λ4 are weights for balancing the loss of each subtask, S * , O * respectively represent the training expected bounding box evaluation score, the bounding box position and the compensation of the size.

[0060] The total loss function Loss of the target tracking network model total is:

[0061] Loss total = Loss1+Loss2.

[0062] In some embodiments, the target tracking network model is trained using GOT-10k, LaSOT, MS COCO, ImageNet DET, ImageNet VID and YouTube-BB training sets, and the entire training process adopts a stochastic gradient descent method to continuously iterate and optimize the entire target tracking network model until the total loss Loss total converges, obtaining the trained target tracking network model.

[0063] In some embodiments, λ1=λ2=1, λ3=λ4=0.1.

[0064] In a second aspect, the present application provides a target tracking device based on a multi-prediction output alignment of a twin network, comprising a processor and a storage medium.

[0065] The storage medium is used to store instructions.

[0066] The processor is used to operate according to the instructions to perform the steps of the method according to the first aspect.

[0067] In a third aspect, the present application provides a storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the method according to the first aspect.

[0068] The method provided by the application can effectively solve the two problems. First, the supplementary module can refine the foreground-background classification map generated by the basic tracker, weaken the multi-peak response value caused by the appearance of similar interference objects in the video, and stabilize the tracking result to prevent the boundary box from drifting. Secondly, the supplementary module can fine-tune the target boundary box predicted by the basic tracker to generate a boundary box that can tightly wrap the target, and then perform more accurate target region prediction. The method can be applied to the fields of vehicle navigation, human-computer interaction and video monitoring.

[0069] The tracking algorithm provided by the application can run at a speed far exceeding real time on a GPU platform. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 The structure diagram of the target tracking model based on the alignment of the multiple prediction outputs of the twin network in the embodiment of the application is shown in the figure.

[0071] Figure 2 The two-stage supplementary module in the embodiment of the application is shown in the figure.

[0072] Figure 3 The pixel-by-pixel cross-correlation diagram is shown in the figure.

[0073] Figure 4 The performance evaluation results of the tracker and other mainstream trackers on the UAV123 evaluation dataset in the embodiment of the application are shown in the figure.

[0074] Figure 5 The feature visualization when the tracker in the embodiment of the application tracks in three different video sequences is shown in the figure. DETAILED DESCRIPTION

[0075] In order to make the technical means, creative features, purposes and effects realized by the application easy to understand, the application will be further described below in combination with specific embodiments.

[0076] In the description of the application, if several meanings are one or more, the meaning of multiple is two or more, greater than, less than, more than, etc. are understood as not including the number, above, below, within, etc. are understood as including the number. If it is described as first, second, it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the sequence of indicated technical features.

[0077] In the description of the application, the description of the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the application. In the description, the exemplary description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any suitable manner in one or more embodiments or examples.

[0078] Embodiment 1

[0079] A target tracking method based on a twin network and multi-prediction output alignment, comprising:

[0080] Obtaining a video;

[0081] Preprocessing the video to extract a template image Z and a search region image X; wherein the image centered on the target extracted from the first frame of the video sequence is taken as the template image Z, and the image centered on the predicted target position of the previous frame in the subsequent frame is taken as the search region image X;

[0082] Inputting the template image Z and the search region image X into a pre-trained target tracking network model;

[0083] Determining a target tracking result according to the output of the target tracking network model;

[0084] The processing process of the target tracking network model comprises:

[0085] The template image Z is extracted by a first feature extraction module to obtain a template image feature Z f ;

[0086] The search region image X is extracted by a second feature extraction module to obtain a search region image feature X f ;

[0087] The template image feature Z f and the search region image feature X f are subjected to similarity matching by a depth-wise cross-correlation module to output a classification response map R cls and a regression response map R reg ;

[0088] Z f =f(Z),

[0089] X f =f(X),

[0090] R cls =DW cls (Zf ,X f ),

[0091] R reg =DW reg (Z f ,X f ).#(4)

[0092] where f(·) denotes feature extraction, DW cls and DW reg represent the cross-deep correlation calculation process of the classification branch and the regression branch, respectively;

[0093] The classification response map R cls and the regression response map R reg are processed by the first prediction head and the second prediction head, respectively, to output a single-channel foreground-background classification score map M cls and a four-channel bounding box prediction map M reg ,

[0094] M cls =Conv cls (R cls )

[0095] M reg =Conv reg (R reg ).#(5)

[0096] where the four-channel bounding box prediction map M reg indicates n sub-regions on the search region image, which ideally contain the same target as the target region on the template image, and the sub-region bounding box is denoted as {P} n , and the target region bounding box in the first frame is denoted as b;

[0097] Based on the target region bounding box b and the sub-region bounding box {P} n , the first PrPool module and the second PrPool module are used to extract region features from the template image feature Z f and the search region image feature X f , respectively, to obtain template region features Z r and sub-region features {X r} n of the same spatial size;

[0098] Z r =PrPool(Z f ,b),

[0099] {X r} n =PrPool(X f ,{P} n).#(6)

[0100] The pixel-wise cross-correlation module takes the template region feature Z r as reference, and the sub-region features {X r} n performs fine-grained feature retrieval to obtain the response map {R me} n for the bounding box region.

[0101] {R me} n = PW(Z r , {X r} n ),#(7)

[0102] where {R me} n is the response map corresponding to the n bounding box regions, and PW represents the pixel-wise cross-correlation operation.

[0103] {R me} n is input into the matching evaluation branch to perform feature compression and information integration, and the bounding box evaluation score S is obtained.

[0104] S = MLP(Conv me ({R me} n )),#(8)

[0105] where MLP represents a fully connected layer, and Conv me represents a stacked convolutional layer; the bounding box evaluation score S is a set of scores related to the matching degree of the region features and the template features, indicating the probability that the corresponding bounding box region exists foreground (target).

[0106] The foreground-background classification score map M cls and the bounding box evaluation score S are fused by element-wise multiplication to obtain the bounding box evaluation score map S′.

[0107] S′ = S ⊙ M cls ,#(9)

[0108] Based on the position l of the maximum value of the bounding box evaluation score map S′, a unique set of target bounding box coordinates P reg is determined on the four-channel bounding box prediction map M l , that is, a unique target sub-region feature (X r ) n is determined from the sub-region features {X r} l ; the target sub-region feature (X r ) l is matched with the template region feature Zr Perform fine-grained feature retrieval to obtain the response map R′ corresponding to the target bounding box region. me :

[0109] R′ me =PW(Z) r ,(X r ) l ),#(10)

[0110] Utilizing refined branches to analyze the response map R′ corresponding to the target bounding box region me Perform channel compression and feature integration (Conv) pr The compensation amount O for the bounding box position and size is obtained by performing a channel-by-channel summation (Sum) process.

[0111] O = Sum(Conv) pr (R′ me )),#(11)

[0112] Where O = (o x ,o y ,o w ,o h ) represents the compensation amount for the position and size of the bounding box, o x ,o y The compensation amounts for the horizontal and vertical coordinates of the bounding box position, o w ,o h This is the compensation amount for the width and height dimensions of the bounding box;

[0113] Based on the compensation amount O, the original predicted target bounding box coordinates P l After making corrections, the final target prediction box is obtained, thus achieving target tracking.

[0114] In some embodiments, the method for constructing the target tracking network model includes:

[0115] The target tracking network model is a two-stage network, including a basic tracker in the first stage and a supplementary module in the second stage;

[0116] The basic tracker includes two input branches: a template branch and a search branch, which take the template image and the search region image as inputs, respectively. The first feature extraction module of the template branch and the second feature extraction module of the search branch both use the first four sets of convolutional layers of ResNet50, followed by a depthwise cross-correlation module and a two-branch parallel prediction head based on an anchorless frame: a first prediction head and a second prediction head. The basic tracker is a standard two-branch prediction Siamese network structure.

[0117] The supplementary module is cascaded behind the basic tracker, and is a twin network structure with double inputs and double outputs, uses a first PrPool module and a second PrPool module to extract regional features, and is connected with a pixel-by-pixel cross-correlation module and two parallel output branches in sequence, the two output branches being a matching evaluation branch and a refinement branch.

[0118] In some embodiments, the training method of the target tracking network model comprises:

[0119] The total loss function Loss1 of the first stage is:

[0120]

[0121] wherein, * represents a training label, λ1 and λ2 represent weights for balancing the loss of each subtask and λ1 = λ2 = 1;

[0122] The training loss function Loss2 of the second stage is:

[0123] Loss2 = λ3L ce (S, S * )+ λ4L1(O, O * ).(2)

[0124] wherein, * represents a training label, L ce and L1 are respectively a CE loss and an L-1 distance loss, λ3 and λ4 are weights for balancing the loss of each subtask and λ3 = λ4 = 0.1;

[0125] The total loss function Loss total of the target tracking network model is:

[0126] Loss total = Loss1 + Loss2.(3).

[0127] In some embodiments, the target tracking network model is trained using GOT-10k, LaSOT, MS COCO, ImageNet DET, ImageNet VID and YouTube-BB training sets, and the entire training process adopts a stochastic gradient descent method to continuously iteratively optimize the entire target tracking network model until the total loss Loss total converges, and a trained target tracking network model is obtained.

[0128] The algorithm proposed in this embodiment regards a commonly used twin network as a basic tracker, and adds a supplementary module in series after the output layer of the basic tracker. The overall framework diagram of the target tracking network is as shown in Figure 1 The specific inference process comprises the following operation steps:

[0129] (1) Step one: constructing the target tracking network model.

[0130] First, the target tracking network model is constructed. The network model proposed in the present application is a two-stage network, wherein the model of the first stage (the basic tracker) is as follows Figure 1 , and the model of the second stage (the supplementary module) is as follows Figure 2 . Both input branches (the template branch and the search branch) of the basic tracker network use the first 4 groups of convolutional layers of ResNet50 as the feature extraction part, followed by a cross-correlation module and an anchor-free double-branch parallel prediction head, and the whole is a standard twin network structure. The supplementary module is also designed as a double-input branch structure, which uses the Precise RoI Pooling (PrPool) module for feature extraction, followed by a pixel-by-pixel cross-correlation layer (see Figure 3 ) and two output branches (the matching evaluation branch and the refinement branch) in parallel. Therefore, the supplementary module is also a computationally efficient twin network structure.

[0131] (2) Step two: training the network model.

[0132] The tracking model proposed in the present application is trained using GOT-10k, LaSOT, MS COCO, ImageNet DET, ImageNet VID and YouTube-BB training sets. The two input branches of the basic tracker, including the template branch and the search branch, take the template image and the search region image as input, respectively. The template image is a 127x127 pixel size image centered on the target extracted from any frame of a video sequence, while the search region is a 255x255 pixel size image centered on the target extracted from other frames of the sequence. The basic tracker network uses a double-branch prediction head to output a single-channel foreground-background classification score map M cls and a four-channel bounding box prediction map M reg . Here, the foreground-background classification task and the bounding box prediction task use the CE loss L ce and the IoU loss L IoU to calculate the loss, so the total loss function Loss1 of the first stage is:

[0133]

[0134] wherein, * represents the training label, λ1 and λ2 represent the weights used to balance the loss of each subtask and λ1 = λ2 = 1.

[0135] The M regcorresponding to n sub-regions (n = 25 x 25 = 625) on the search region image. The training input of the complement module is the features of the target region on the template image and the features within the n sub-regions on the search region image. The output of the complement module is respectively denoted as S and O, where S represents the probability of the bounding box region existing foreground (target), and O = (o x ,o y ,o w ,o h ) is the compensation amount of the bounding box position and size. Therefore, the training loss function Loss2 of the two-stage is:

[0136] Loss2 = λ3L ce (S, S * ) + λ4L1(O, O * ).(2)

[0137] wherein, * represents the training label, L ce and L1 are respectively the CE loss and L-1 distance loss, and λ3 and λ4 are the weights for balancing the loss of each sub-task and λ3 = λ4 = 0.1. The total loss function Loss total of the tracking network model is:

[0138] Loss total = Loss1 + Loss2.(3)

[0139] The entire training process takes the stochastic gradient descent method to continuously iterate and optimize the entire tracking network until the total loss Loss total converges.

[0140] (3) Step three: using the basic tracker to perform preliminary inference.

[0141] The template image in inference is the 127 x 127 pixel size image centered on the target extracted from the first frame of the video sequence (remaining unchanged in the tracking process), and the search region is the 255 x 255 pixel size image centered on the predicted target position in the previous frame. The feature extraction network extracts features from the input image, and the cross-correlation module of the basic tracker network matches the features of the two to obtain the classification response map R cls and the regression response map R reg :

[0142] Z f = f(Z),

[0143] X f = f(X),

[0144] R cls = DW cls (Z f , Xf ),

[0145] R reg = DW reg (Z f , X f ).#(4)

[0146] where Z denotes the template image, X denotes the search region image, f(·) denotes the feature extraction, DW cls and DW reg denote the cross-correlation calculation process of the classification branch and the regression branch respectively. After the response map passes through the prediction head Conv cls of the classification branch and the prediction head Conv reg of the regression branch, the output is a single-channel foreground-background classification score map M cls and a four-channel four-channel bounding box prediction map M reg :

[0147] M cls = Conv cls (R cls )

[0148] M reg = Conv reg (R reg ).#(5)

[0149] (4) Step four: extract region features and conduct fine-grained feature retrieval.

[0150] This stage is the core of the supplementary module proposed in the present application. The four-channel bounding box prediction map M reg generated in step three indicates n sub-regions (denoted as {P} n ) on the search region image, which in an ideal state contain the same target as the target region (denoted as b) on the template image. In tracking, the tracking model uses the (Precise RoIPooling) PrPool module to extract and save the template region features in the template image and the deep features of the target in these sub-regions as multi-channel feature maps of the same spatial size:

[0151] Z r = PrPool(Z f , b),

[0152] {X r} n = PrPool(X f , {P} n ).#(6)

[0153] where Z r denotes the template region features, {Xr} n represents sub-region features. In reality, Z r contains complete target features, {X r} n may contain all, part or no target features. To determine whether each predicted bounding box is accurate, the tracker uses the template features as a benchmark to perform a fine-grained feature search on the sub-region features:

[0154] {R me} n = PW(Z r , {X r} n ) (7)

[0155] where {R me} n is the response map corresponding to the n bounding box regions, and PW represents the pixel-wise cross-correlation operation. Figure 3 Specifically, the feature vector at each spatial location of the feature map Z r is treated as a cross-correlation kernel. All the cross-correlation kernels are cross-correlated on the feature map {X r} n and stacked in the channel dimension to form a feature map (response map). This method can express the correlation of each pair of feature vectors on different feature maps while maintaining the spatial dimensions of the output features unchanged, which is very suitable for fine-grained feature matching calculations.

[0156] (5) Step five: align the outputs of the multiple branches of prediction.

[0157] The response map {R me} n generated in step four is sent to the matching evaluation branch in the supplementary module for feature compression and information integration. This process can be formulated as:

[0158] S = MLP(Conv me ({R me} n )) (8)

[0159] where MLP represents a fully connected layer, and Conv me represents a stacked convolutional layer. S is a set of scores related to the matching degree of the region features and the template features, representing the probability of the existence of foreground (target) in the corresponding bounding box region. Since the activation values at each location on the classification score map saved in step one represent the probability of the region belonging to the foreground (target), the two can be fused by element-wise multiplication to obtain the bounding box evaluation score map S':

[0160] S′=S⊙M cls ,#(9)

[0161] In this way, the multi-peak response values ​​generated by the classification score map of the basic model due to cluttered backgrounds or interference can be greatly suppressed, thereby refining the classification branch output of the basic tracker, preventing the occurrence of tracking box drift, and increasing the robustness of the tracking algorithm.

[0162] (6) Step Six: Refine the coordinates of the bounding box.

[0163] Based on the position l of the maximum value of the bounding box evaluation score map S′ obtained in step five, a unique set of target bounding box coordinates P can be determined on the four-channel bounding box prediction map saved in step one. l The feature of the region within the box (X) r ) l With template feature Z r By performing fine-grained feature retrieval, we can obtain:

[0164] R′ me =PW(Z) r ,(X r ) l ),#(10)

[0165] Where R′ me For coordinate frame P l The response map corresponding to the inner region. R′ me The refined branch after supplementing the module (see) Figure 2 ), including stacked convolutional layers (Conv) pr After summing the values ​​of each channel (Sum), the result is transformed into an eigenvector of length 4.

[0166] O = Sum(Conv) pr (R′ me )),#(11)

[0167] Where O = (o x ,o y ,o w ,o h This represents the compensation amount for the bounding box position and size. x ,o y The compensation amount for the horizontal and vertical coordinates of the bounding box position, o w ,o h This is the compensation amount for the width and height dimensions of the bounding box. The coordinates P of the originally predicted target bounding box are... l After compensation correction, a more accurate target prediction box is obtained, which can achieve more accurate target tracking.

[0168] So far, the tracker model proposed in the application completes the inference of the target position and scale on a video frame. The tracker will cycle through all the steps until the last frame in the remaining video frames.

[0169] Evaluation criteria: The application evaluates the tracker on UAV123 through One-Pass-Evaluation (OPE) with two indicators of Precision and Area Under Curve (AUC) of Success Rate, evaluates the tracker on GOT-10k through Average Overlap (AO) and Success Rate (SR), and evaluates the tracker on VOT2018 and VOT2019 through Expected Average Overlap (EAO), Accuracy and Robustness.

[0170] Figure 4 The performance comparison results of the tracker of the application with other trackers on the UAV123 evaluation dataset are shown. The dataset is collected by a UAV, and the tracked objects in the video are mostly small targets, often appearing in motion states such as rotation and deformation. Therefore, the dataset is very suitable for evaluating the performance of tracking algorithms. From the performance comparison results of the tracker of the application with other trackers on the UAV123 evaluation dataset, it can be seen that the tracker model proposed in the application exceeds the other six most advanced comparison trackers in the AUC indicator of the success rate, and at the same time, it can also rank second in the precision evaluation. This shows the superiority of the tracking algorithm proposed in the application. Figure 4

[0171] Tables 1 and 2 respectively show the performance comparison results of the tracking algorithm with other methods on the VOT2018 and VOT2019 evaluation datasets, and the final ranking is based on the EAO indicator. For each evaluation indicator, the evaluation results of the top three trackers are marked in bold, underlined and italicized. On VOT2018, the performance of the algorithm proposed in the application ranks second, and it only lags behind the first-ranked tracking model by 0.001 points in the EAO indicator. On VOT2019, the tracking effect of the method proposed in the application can also rank in the top three.

[0172] Table 1

[0173]

[0174] Table 2

[0175]

[0176] Table 3

[0177] ​

[0178] Table 3 shows the performance evaluation results of the tracker of the present application and other 8 trackers on the GOT-10k large dataset. The dataset is collected in outdoor scenes, which can better show the tracking effect of the tracker in most actual application scenarios. For each evaluation index, the top three tracker evaluation results are marked in bold, underlined and italicized. The comparison results show that the tracker proposed in the present application can defeat most of the existing trackers and ranks second.

[0179] Figure 5 The feature visualization of the tracker when tracking in 3 different video sequences is shown. Column (a) is the template image, column (b) is the search area image, column (c) is the foreground-background classification score map predicted by the base tracker, column (d) is the bounding box evaluation score predicted by the supplementary module, and column (e) is the bounding box evaluation score map. Obviously, columns (c) and (d) are complementary in the expression of visual semantics, and column (e) generated after the fusion of the two can more effectively deal with the appearance of cluttered background and interference.

[0180] Embodiment 2

[0181] In a second aspect, the present embodiment provides a target tracking device based on multi-prediction output alignment of a twin network, comprising a processor and a storage medium;

[0182] The storage medium is used to store instructions.

[0183] The processor is used to operate according to the instructions to perform the steps of the method according to embodiment 1.

[0184] Embodiment 3

[0185] In a third aspect, the present embodiment provides a storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the method according to embodiment 1.

[0186] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0187] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flows and / or block diagram block or blocks. Figure 1 one or more flow or flows and / or block diagram block or blocks. Figure 1 one or more flow or flows and / or block diagram block or blocks.

[0188] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart or flows and / or block diagram block or blocks. Figure 1 one or more flow or flows and / or block diagram block or blocks. Figure 1 one or more flow or flows and / or block diagram block or blocks.

[0189] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flows and / or block diagram block or blocks. Figure 1 one or more flow or flows and / or block diagram block or blocks. Figure 1 one or more flow or flows and / or block diagram block or blocks.

[0190] Within the scope of the application, all changes or modifications that can be made to the above-described embodiments should be covered by the application.

Claims

1. A target tracking method based on Siamese networks with multi-prediction output alignment, characterized in that, The method includes: Get the video; The video is preprocessed to extract a template image Z and a search region image X; wherein the image centered on the target extracted from the first frame of the video sequence is used as the template image Z, and the image centered on the predicted target position in the previous frame in subsequent frames is used as the search region image X. Input the template image Z and the search region image X into the pre-trained target tracking network model; The target tracking result is determined based on the output of the target tracking network model. The processing steps of the target tracking network model include: The template image Z is processed by the first feature extraction module to obtain the template image features Z. f The search region image X is processed by the second feature extraction module to obtain the search region image features X. f ; Template image features Z f and search region image features X f After similarity matching via a depthwise cross-correlation module, the classification response map R is output. cls and regression response plot R reg ; Classification Response Map R cls and regression response plot R reg After being processed by the first and second prediction heads respectively, a single-channel foreground and background classification score map M is output. cls And four-channel bounding box prediction map M reg The four-channel bounding box prediction map M reg This indicates n sub-regions on the search region image, which, ideally, all contain the same target as the target region on the template image. Let the bounding boxes of these sub-regions be {P}. n Let b be the bounding box of the target region in the first frame; Based on the target region bounding box b and the sub-region bounding box {P} n The first PrPool module and the second PrPool module are used respectively to process the template image features Z. f and search region image features X f Perform region feature extraction to obtain template region features Z of the same spatial size. r and sub-region features {X r } n ; The pixel-wise cross-correlation module uses template region features Z r Based on the baseline, for the sub-region feature {X r } n Perform fine-grained feature retrieval to obtain the response map {R} corresponding to the bounding box region. me } n ; {R me } n Input the matching evaluation branch, perform feature compression and information integration, and obtain the bounding box evaluation score S; Classify the foreground and background into score charts M. cls The bounding box evaluation score S is fused with the bounding box evaluation score S′ by element-wise multiplication. Based on the location l of the maximum value in the bounding box evaluation score map S′, in the four-channel bounding box prediction map M reg A unique set of target bounding box coordinates P is determined. l That is, sub-region features {X r } n Determine the unique target sub-region feature (X) r ) l ; target sub-region features (X r ) l With template region feature Z r Perform fine-grained feature retrieval to obtain the response map R′ corresponding to the target bounding box region. me : Utilizing refined branches to analyze the response map R′ corresponding to the target bounding box region me Channel compression, feature integration, and channel-by-channel summation are performed to obtain the compensation amount O for the bounding box position and size: Based on the compensation amount O, the original predicted target bounding box coordinates P l After making corrections, the final target prediction box is obtained, thus achieving target tracking.

2. The target tracking method based on Siamese networks with multi-prediction output alignment according to claim 1, characterized in that, Template image features Z f and search region image features X f After similarity matching via a depthwise cross-correlation module, the classification response map R is output. cls and regression response plot R reg ,include: R cls =DW cls (Z f X f ) R reg =DW reg (Z f X f ) Among them, DW cls and DW reg These represent the depthwise cross-correlation calculation process for the classification branch and the regression branch, respectively.

3. The target tracking method based on Siamese networks with multi-prediction output alignment according to claim 1, characterized in that, Classification Response Map R cls and regression response plot R reg Each passed through the first prediction head Conv cls Second prediction head Conv reg After processing, the output is a single-channel foreground / background classification score map M. cls And four-channel bounding box prediction map M reg , include: M cls =Conv cls (R cls ) M reg =Conv reg (R reg )。 4. The target tracking method based on Siamese networks with multi-prediction output alignment according to claim 1, characterized in that, Based on the target region bounding box b and the sub-region bounding box {P} n The first PrPool module and the second PrPool module are used respectively to process the template image features Z. f and search region image features X f Perform region feature extraction to obtain template region features Z of the same spatial size. r and sub-region features {X r } n ; Z r =PrPool(Z f ,b) {X r } n =PrPool(X f ,{P} n ) The pixel-wise cross-correlation module uses template region features Z r Based on the baseline, for the sub-region feature {X r } n Perform fine-grained feature retrieval to obtain the response map {R} corresponding to the bounding box region. me } n ; {R me } n =PW(Z r ,{X r } n ) Where {R me } n For the response maps corresponding to n bounding box regions, PW represents the pixel-wise cross-correlation operation; {R me } n Input the matching evaluation branch, perform feature compression and information integration, and obtain the bounding box evaluation score S; S=MLP(Conv me ({R me } n )) Where MLP represents a fully connected layer, Conv me The convolutional layers are stacked; the bounding box evaluation score S is a set of scores about the degree of matching between the region features and the template features, representing the probability that a foreground object exists in the corresponding bounding box region; Classify the foreground and background into score charts M. cls The bounding box evaluation score S is fused with the bounding box evaluation score S′ by element-wise multiplication. S′=S⊙M cls Based on the location l of the maximum value in the bounding box evaluation score map S′, in the four-channel bounding box prediction map M reg A unique set of target bounding box coordinates P is determined. l That is, sub-region features {X r } n Determine the unique target sub-region feature (X) r ) l ; target sub-region features (X r ) l With template region feature Z r Perform fine-grained feature retrieval to obtain the response map R′ corresponding to the target bounding box region. me : R′ me =PW(Z r ,(X r ) l ) Utilizing refined branches to analyze the response map R′ corresponding to the target bounding box region me Perform channel compression and feature integration Conv pr The summation (Sum) for each channel yields the compensation amount O for the bounding box position and size: O=Sum(Conv pr (R′ me )) Where O = (o x ,o y ,o w ,o h ) represents the compensation amount for the position and size of the bounding box, o x ,o y The compensation amounts for the horizontal and vertical coordinates of the bounding box position, o w ,o h This is the compensation amount for the width and height dimensions of the bounding box; Based on the compensation amount O, the original predicted target bounding box coordinates P l After making corrections, the final target prediction box is obtained, thus achieving target tracking.

5. The target tracking method based on Siamese networks with multi-prediction output alignment according to claim 1, characterized in that, The method for constructing the target tracking network model includes: The target tracking network model is a two-stage network, including a basic tracker in the first stage and a supplementary module in the second stage; The basic tracker includes two input branches: a template branch and a search branch, which take the template image and the search region image as inputs, respectively. The first feature extraction module of the template branch and the second feature extraction module of the search branch both use the first four sets of convolutional layers of ResNet50, followed by a depthwise cross-correlation module and a two-branch parallel prediction head based on an anchor-free frame: a first prediction head and a second prediction head. The basic tracker is a standard two-branch prediction Siamese network structure. The supplementary module is cascaded after the basic tracker and is a twin network structure with two inputs and two outputs. It uses the first PrPool module and the second PrPool module to extract regional features, followed by a pixel-wise cross-correlation module and two parallel output branches, which are a matching evaluation branch and a refinement branch.

6. The target tracking method based on Siamese networks with multi-prediction output alignment according to claim 1, characterized in that, The training method for the target tracking network model includes: The total loss function Loss1 for the first stage is: Where * represents the training label, λ1 and λ2 represent the weights used to balance the losses of each subtask, and L ce L IoU These are CE loss and IoU loss, respectively. These represent the foreground and background classification score maps and the four-channel bounding box prediction map, respectively, representing the training expectation. The two-stage training loss function Loss2 is: Loss2=λ3L ce (S,S * )+λ4L1(O,O * ) in, * L represents the training labels. ce L1 and L2 are the CE loss and L-1 distance loss, respectively, λ3 and λ4 are the weights used to balance the losses of each subtask, and S... * O * These represent the expected bounding box evaluation score, the compensation amount for the bounding box position, and the size, respectively. The total loss function of the target tracking network model. total for: Loss total =Loss1+Loss2。 7. The target tracking method based on Siamese networks with multi-prediction output alignment according to claim 6, characterized in that, The target tracking network model was trained using the GOT-10k, LaSOT, MS COCO, ImageNet DET, ImageNet VID, and YouTube-BB training sets. The entire training process employed stochastic gradient descent to iteratively optimize the target tracking network model until the total loss was minimized. total The training converges, resulting in a well-trained target tracking network model.

8. The target tracking method based on Siamese networks with multi-prediction output alignment according to claim 6, characterized in that, λ1=λ2=1, λ3=λ4=0.

1.

9. A target tracking device based on Siamese networks with multi-prediction output alignment, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 8.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images