Anchor-free Siamese neural network-based integrated multi-target detection and tracking method

Abstract

The present invention discloses an integrated method for multi-target detection and tracking based on an anchor-free twin neural network, which mainly solves the problems of weak target surface feature recognition ability and poor network real-time performance in the existing technology. Its implementation scheme is as follows: obtain a training data set and a test data set; construct an integrated detection and tracking network based on an anchor-free twin neural network, and set its loss function Loss; generate an empty training set trajectory template set and use it and the training data set to train the constructed integrated detection and tracking network; generate an empty test set trajectory template set and input it and the test data set into the trained integrated detection and tracking network at the same time, and output the detection and tracking results on the test data set. The present invention can extract powerful discriminative features, improve the real-time performance and tracking accuracy of the network, and can be used for single-frame detection and cross-frame association of multiple targets in a video sequence, realizing accurate detection and tracking of multiple targets of interest in natural video scenes.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and further relates to an integrated multi-target detection and tracking method, which can be used for single-frame detection and cross-frame association of multiple targets in a video sequence, and realize accurate detection and tracking of multiple targets of interest in natural video scenes. Background Art

[0002] The integrated method of multi-target detection and tracking is to extract the features of the image frame from each frame of the video sequence, and use this feature to detect and track the target of interest that may exist in the current image frame. According to the features of the image frame, a model is built for target detection and target tracking. The target detection model is used to identify the location, length and width of the target. The tracking model is used to associate and match the target in the current image frame output by the detection model with the target in the previous image frame, so as to realize the detection and tracking of multiple targets in the image frame in sequence and at the same time throughout the video sequence. Existing integrated methods for multi-target detection and tracking are generally divided into two types: motion model modeling method and identity feature association matching method. Among them:

[0003] Motion modeling methods focus on extracting target motion features between adjacent image frames. By constructing a target motion feature model, they predict the target's position in the next image frame. Target detection and association are achieved by matching the predicted results with the detection model results. Common methods include optical flow, inter-frame difference, and deep learning modeling. These methods have been proven to achieve good tracking results when the number of targets is small and sparsely distributed. However, because motion modeling methods rely on a single feature and suffer from large errors in motion information extraction, they struggle to achieve good tracking results in scenarios with a large number of targets and a dense distribution.

[0004] Identity feature association matching methods focus on extracting the target's appearance information as identity features, tracking the target based on the similarity of the target's identity features in adjacent frames. Deep learning modeling methods are often used for feature modeling. Although identity feature association matching methods achieve relatively good tracking results when the appearance features between targets are highly distinguishable, they rely heavily on the strong separability of the target's appearance features, making them difficult to achieve good tracking results when the target's appearance features are scarce and there are many similar targets.

[0005] Beijing University of Posts and Telecommunications disclosed a "multi-target tracking method based on Mask R-CNN and appearance feature fusion" in its patent application document with application number 202110631768.9. It first adjusts the image size, uses the Mask R-CNN network to search for the image to be identified, and obtains the candidate area and feature map where the target is located in the image; then extracts the target features by cutting the feature map on the candidate area; and then uses the twin network to compare the similarity between multiple candidate targets to determine whether the targets belong to the same target and associate them. This method has two shortcomings: first, because this method uses independent detection networks and tracking networks for detection and tracking, and the detection uses the anchor-based Mask R-CNN network, it is impossible to perform end-to-end optimization training globally, and cannot achieve the theoretically optimal detection and tracking results; second, because the global targets are input into the twin networks separately for pairwise comparison between targets, the computational cost is greatly increased, and the real-time performance of tracking is poor.

[0006] The University of the Chinese Academy of Sciences (UCAS) disclosed in its patent application number 202010015437.8 a "UAV video multi-target tracking method based on target-scene consistency." Based on the output of the detection network, a twin neural network is constructed to calculate the confidence level of target-scene consistency and target-target similarity. The outputs of the two twin neural networks are simultaneously input into the constructed branch network to predict the target's location. Due to the presence of multiple twin neural network branches, this method results in high network complexity, difficulty in network optimization, and low real-time tracking performance. Summary of the Invention

[0007] The purpose of the present invention is to address the shortcomings of the above-mentioned existing technologies and propose an integrated method for multi-target detection and tracking based on an anchor-free twin neural network to reduce the complexity of the network and improve the real-time performance of multi-target tracking of the twin neural network; and to improve the detection and tracking accuracy by jointly optimizing the detection and tracking networks.

[0008] To achieve the above objectives, the technical solutions of the present invention include the following:

[0009] (1) Construct training and test datasets:

[0010] At least 5 optical video sequences are composed of a training dataset, and at least 1 optical video sequence is composed of a test dataset, where:

[0011] Each optical video sequence must contain at least 50 consecutive frames of images, each frame must contain at least one complete moving target, and the position of the moving target in each frame must move by at least 2 pixels compared to the position of the target in the previous frame. The length and width of each frame must not be less than 500×500.

[0012] The labels of each dataset include the center point of each target in the current frame image, the length and width of the target, the target category, and the target serial number;

[0013] (2) Constructing an integrated detection and tracking network based on an anchor-free twin neural network:

[0014] (2a) Build a parallel multi-scale feature fusion module consisting of three downsampling layers and three upsampling layers;

[0015] (2b) Build a detection branch subnetwork consisting of three convolutional layers and three activation function layers;

[0016] (2c) Select the existing ResNet50 network and SiamBAN network, and use the SiamBAN network as the tracking branch subnetwork;

[0017] (2d) Connecting the input of the parallel multi-scale feature fusion module to the third, fourth, and fifth convolutional layers of the existing ResNet50 network, and connecting the output of the parallel multi-scale feature fusion module to the tracking branch sub-network and the detection branch sub-network, respectively, to form an integrated detection and tracking network based on the anchor-free Siamese neural network;

[0018] (2e) Set the loss function of the integrated detection and tracking network based on the anchor-free Siamese neural network: Loss = L + E2, where:

[0019] E2 is the loss function used in the known SiamBAN network;

[0020] L=l heat +l size +l offset is the loss function of the detection branch sub-network, l heat 、l size 、l offset They are the target center heat map loss function, size regression loss function and target center offset loss function of the detection branch network respectively;

[0021] (3) Using the training dataset to train the integrated detection and tracking network based on the anchor-free twin neural network:

[0022] (3a) Construct an empty training set target trajectory template set and initialize it according to the label of the first frame image of the training set;

[0023] (3b) Inputting the target trajectory template set of the training set and the current image into the integrated detection and tracking network based on the anchor-free Siamese neural network to obtain the detection and tracking results;

[0024] (3c) Calculate the network loss value based on the total loss function and update the network parameters by gradient descent method;

[0025] (3d) updating the target trajectory template set;

[0026] (3e) Repeat (3b) to (3d) until the network loss function converges, and obtain the trained integrated detection and tracking network model based on the anchor-free twin neural network;

[0027] (4) Perform target detection and tracking on the images in the test set:

[0028] (4a) Input the first frame image of the test set into the trained integrated detection and tracking network based on the anchor-free twin neural network, and output the target detection result of the first frame image;

[0029] (4b) Construct an empty test set target trajectory template set, and initialize the test set target trajectory template set according to the first frame image of the test set and the detection result of (4a);

[0030] (4c) The test set and the initialized test set trajectory template set are input into the trained integrated detection and tracking network based on the anchor-free twin neural network, and the detection and tracking results of the second frame of the test set and each frame image thereafter are output.

[0031] (4d) Match the detection result and the tracking result according to the IOU size, replace the tracking result with the detection result of the matching target, and output the final detection and tracking results.

[0032] Compared with the prior art, the present invention has the following advantages:

[0033] First, since the present invention constructs an anchor-free integrated detection and tracking network, the detection and tracking branches share the image features extracted by Resnet50, which can perform multi-target tracking locally, reducing the time loss of global target matching, and using an anchor-free neural network structure, reducing the repeated extraction and calculation of features, reducing the computational loss of the network, and enhancing the real-time performance of the network.

[0034] Second, the network structure of the present invention can be trained end-to-end, that is, during the learning process, no modular or staged training is performed, and the overall goal of the task is directly optimized. This allows the various branches of the network to be jointly optimized to achieve better detection and tracking performance;

[0035] Third, since the integrated detection and tracking network based on the anchor-free twin neural network constructed by the present invention uses a parallel multi-scale feature fusion module to extract and fuse features of multiple scales of the target, this feature can effectively enhance the feature strength of the target and assist the network to achieve better detection and tracking performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1It is an implementation flow chart of the present invention;

[0037] Figure 2 Schematic diagram of the existing ResNet50 feature extraction network structure used in the present invention;

[0038] Figure 3 Schematic diagram of the structure of the parallel multi-scale feature fusion module constructed in the present invention;

[0039] Figure 4 Schematic diagram of the detection branch sub-network structure constructed in the present invention;

[0040] Figure 5 Schematic diagram of the SiamBAN tracking subnetwork structure used in the present invention;

[0041] Figure 6 It is a schematic diagram of the overall structure of the integrated detection and tracking network based on the anchor-free twin neural network constructed in the present invention. DETAILED DESCRIPTION

[0042] The embodiments and effects of the present invention are further described in detail with reference to the accompanying drawings.

[0043] like Figure 1 As shown, the specific implementation of this example is as follows:

[0044] Step 1: Build training and test datasets:

[0045] Download an existing optical video tracking dataset from the public network, and use at least 5 optical video sequences to form a training dataset and at least 1 optical video sequence to form a test dataset, where:

[0046] Each optical video sequence must contain at least 50 consecutive frames of images, each frame must contain at least one complete moving target, and the position of the moving target in each frame must move by at least 2 pixels compared to the position of the target in the previous frame. The length and width of each frame must not be less than 500×500.

[0047] The labels of each dataset include the center point of each target in the current frame image, the length and width of the target, the target category, and the target serial number.

[0048] Step 2: Build an integrated detection and tracking network based on the anchor-free twin neural network.

[0049] 2.1) Select an existing ResNet50 network as the feature extraction network for the integrated detection and tracking network based on the anchor-free twin neural network:

[0050] like Figure 2As shown in the figure, the existing ResNet50 network includes three convolutional layers, three BN layers, one activation function layer, and one maximum pooling layer. Its structure is as follows: first convolutional layer, first BN layer, activation function layer, maximum pooling layer, second convolutional layer, second BN layer, third convolutional layer, third BN layer, fourth convolutional layer, fourth BN layer, fifth convolutional layer, and fifth BN layer.

[0051] The parameter settings and calculation functions used in each layer are as follows:

[0052] The number of convolution kernels in the first to fifth convolutional layers is set to 64, 64, 128, 256, and 512, respectively, and the convolution kernel sizes are set to 7, 3, 3, 3, and 3, respectively;

[0053] The stride of the first, second, and third convolutional layers is set to 2.

[0054] The stride of the fourth and fifth convolutional layers is 1.

[0055] The convolution kernel void ratios in the fourth and fifth convolutional layers are 2 and 4 respectively;

[0056] The size of the pooling area kernel of the maximum pooling layer is set to 3×3, and the stride is set to 2;

[0057] The first to fifth BN layers all use batch normalization functions.

[0058] The activation function layer uses a linear rectification function.

[0059] The maximum pooling layer uses the regional maximum pooling function;

[0060] 2.2) Build a parallel multi-scale feature fusion module:

[0061] like Figure 3 As shown in Figure 2, the parallel multi-scale feature fusion module includes three downsampling layers and three upsampling layers, and its structural parameters are as follows:

[0062] The output ends and input ends of the first downsampling layer, the second downsampling layer, the third downsampling layer, the first upsampling layer, the second upsampling layer, and the third upsampling layer are connected in sequence, and the input end of the first downsampling layer is connected to the output end of the third upsampling layer, the input end of the second downsampling layer is connected to the output end of the second upsampling layer, and the input end of the third downsampling layer is connected to the output end of the third upsampling layer;

[0063] The three downsampling layers are composed of three parallel convolutional layers. The convolution kernel size of the first and third downsampling layers is 3×3, and the number of convolution kernels is 128, 256, and 512 respectively; the convolution kernel size of the second downsampling layer is 3, and the number of convolution kernels is 64, 128, and 256 respectively.

[0064] 2.3) Build the detection branch sub-network:

[0065] like Figure 4 As shown in Figure 2, the detection branch sub-network includes three convolutional layers and three activation function layers, and its structural parameters are as follows:

[0066] The first convolution layer is connected to the first activation function layer, the second convolution layer is connected to the second activation function layer, and the third convolution layer is connected to the third activation function layer, forming three parallel branches;

[0067] The convolution kernel size of the three convolutional layers is 3, and the number of convolution kernels is 1, 2, and 2 respectively;

[0068] The activation functions sampled in the three activation function layers are all linear rectification functions;

[0069] 2.4) Select an existing SiamBAN network as the tracking subnetwork of the integrated detection and tracking network based on the anchor-free twin neural network:

[0070] like Figure 5 As shown in FIG, the existing SiamBAN network includes three mutual correlation classification layers, three mutual correlation regression layers and two convolutional layers. Its structure is as follows: the first mutual correlation classification layer, the second mutual correlation classification layer, the third mutual correlation classification layer, and the first convolutional layer are connected in sequence; the first mutual correlation regression layer, the second mutual correlation regression layer, the third mutual correlation regression layer, and the second convolutional layer are connected in sequence to form two parallel branches.

[0071] The convolution kernel size of the first and second convolution layers are both 1, and the number of convolution kernels is 2 and 4 respectively;

[0072] 2.5) Connect the outputs of the third, fourth, and fifth convolutional layers of the ResNet50 feature extraction network to the first, second, and third convolutional layers of the first downsampling layer of the parallel multi-scale feature fusion module, respectively. Connect the outputs of the parallel multi-scale network to the inputs of the detection branch sub-network and the SiamBAN tracking branch network, respectively, to form an integrated detection and tracking network based on the anchor-free twin neural network, as shown in the figure. Figure 6 shown.

[0073] Step 3: Set the loss function Loss of the integrated detection and tracking network based on the anchor-free twin neural network.

[0074] 3.1) Set the loss function L of the detection branch sub-network:

[0075] L=l heat +l size +l offset

[0076] Among them, l heat , l size , l offsetThey are the target center heat map loss function, target size regression loss function, and target center offset loss function of the detection branch network, respectively, and are expressed as follows:

[0077]

[0078]

[0079]

[0080] Where N is the number of input training samples in each batch, p, size, and offset are the prediction results of the detection network for the target center confidence, target size, and target center offset respectively; p_gt, size_gt, and offset_gt are the true label values ​​of the target center, target size, and target center offset respectively;

[0081] 3.2) Set the loss function of the tracking branch sub-network to the loss function E2 used in the known SiamBAN network, which is expressed as follows:

[0082]

[0083] Among them, N is the number of input samples for a single training, y i is the positive and negative label of the sample, p i is the network prediction score, and IOU is the intersection-over-union ratio of the network predicted target position and the target true value;

[0084] 3.3) Add the loss function L of the detection branch sub-network and the loss function E2 of the tracking branch sub-network to obtain the loss function of the integrated detection and tracking network of the anchor-free Siamese neural network:

[0085] Loss=L+E2。

[0086] Step 4: Generate an empty training set trajectory template set and initialize it.

[0087] 4.1) Create an empty training set trajectory template set to store the template features corresponding to each trajectory;

[0088] 4.2) Select the first frame of the training video sequence as the network input and extract the output feature map of the parallel multi-scale feature fusion module;

[0089] 4.3) Based on the label of each target in the first frame of the video sequence, extract the feature block of the target area in the output feature map;

[0090] 4.4) All target feature blocks are converted into target trajectory template features through the existing technology ROI Align, and each target trajectory template feature is stored in the target trajectory training template set to complete the initialization of the target trajectory training template set.

[0091] Step 5: Train the integrated detection and tracking network based on the anchor-free twin neural network.

[0092] 5.1) Input the training set target trajectory template set and the current image into the integrated detection and tracking network based on the anchor-free Siamese neural network to obtain the detection and tracking results;

[0093] 5.2) Update the target trajectory template set using the test and tracking results output by the network;

[0094] 5.3) Calculate the network loss value based on the total loss function and update the network parameters using the gradient descent method;

[0095] 5.4) Repeat steps 5.1) to 5.3) until the network loss function converges, and obtain the trained integrated detection and tracking network model based on the anchor-free twin neural network.

[0096] Step 6: Generate an empty test set trajectory template set and initialize it.

[0097] 6.1) Create an empty test set trajectory template set to store the template features corresponding to each trajectory;

[0098] 6.2) Select the first frame of the test video sequence as the input to the trained integrated detection and tracking network based on the anchor-free Siamese neural network, and output the result of target detection in the first frame;

[0099] 6.3) Based on the detection results of each target in the first frame of the test set, extract the feature blocks of the target region in the feature map output by the parallel multi-scale fusion module;

[0100] 6.4) All target feature blocks are converted into target trajectory template features through the existing technology ROI Align, and each target trajectory template feature is stored in the target trajectory test template set to complete the target trajectory test template set initialization.

[0101] Step 7: Use the trained integrated detection and tracking network based on the anchor-free twin neural network to perform multi-target tracking.

[0102] 7.1) Input the current frame data of the test set and the test set trajectory template set simultaneously into the trained integrated detection and tracking network based on the anchor-free Siamese neural network, and output the detection and tracking results;

[0103] 7.2) Calculate the IOU value between the detection result and the tracking result for each target, and select the detection and tracking pair with the largest IOU, which is recorded as the successful matching result pair;

[0104] 7.3) Update the test set trajectory template set based on the matching results

[0105] For the successfully matched result pairs, the predicted value of the target position in the tracking result is replaced with the predicted value of the target position in the detection result. Based on the detection result, the feature blocks of the target area in the output feature map of the parallel multi-scale fusion module are extracted. These feature blocks are converted into target trajectory template features through the existing technology ROI Align, and then the content of the trajectory template set is replaced with the target trajectory template features.

[0106] For detections that do not successfully match, they are treated as new target trajectories. Based on the detection results of the new trajectories, feature blocks of the target area in the output feature map of the parallel multi-scale fusion module are extracted. These feature blocks are converted into target trajectory template features using the existing ROI Align technology, and their trajectory template features are added to the trajectory template set.

[0107] 7.4) Output detection and tracking results

[0108] The detection results of the successfully matched result pair and the detection results of the new target trajectory are output as the detection results of the current frame;

[0109] The tracking results of the successfully matched result pairs and the detection results of the new target trajectory are output as the tracking results of the current frame;

[0110] 7.5) Repeat steps 7.1) to 7.4) to output the complete detection and tracking results.

[0111] The effects of the present invention are further described below in conjunction with simulation experiments.

[0112] 1. Simulation conditions:

[0113] The hardware platform for the simulation of the present invention is: the processor is Intel(R) Core(TM) i7, the main frequency is 3.6GHz, the memory is 64G, and the graphics card is GeForce RTX 2080.

[0114] The software platforms for the simulation experiment of the present invention are: Windows 10 operating system, Pycharm 2020 software, Python 3.7 and Pytorch deep learning framework.

[0115] The simulation experiment data used in the present invention is the public dataset MOT17, which includes seven indoor and outdoor public place scenes with pedestrians. The video of each scene is divided into two segments, one for training and the other for testing.

[0116] 2. Simulation content and result analysis:

[0117] The simulation experiment of the present invention uses the three methods of the present invention and the prior art to conduct experimental verification on the MOT17 dataset. The experimental results are shown in Table 1:

[0118] Table 1 Experimental verification results of the present invention and three existing technologies on the MOT17 dataset

[0119] Algorithm indicators The present invention SiamMOT CenterTrack CTracker MOTA (%) 68.1 65.9 67.8 66.6 FPS 20 16 17.5 6.8

[0120] The three existing methods in Table 1 are from the following sources:

[0121] SiamMOT refers to the multi-target tracking algorithm proposed by Shuai B et al. in "Siammot: Siamese multi-object tracking[C] / / Pro-ceedings of the IEEE / CVF conference on computer vision and pattern recognition.2021:12372-12382.", referred to as the SiamMOT algorithm.

[0122] CenterTrack refers to the multi-target tracking algorithm proposed by Zhou X et al. in "Tracking objects as points[C] / / European Conference on Computer Vision. Springer, Cham, 2020: 474-490.", referred to as the CenterTrack algorithm.

[0123] CTracker refers to the multi-target tracking algorithm proposed by Peng J et al. in "Chained-tracker: Chaining paired attentive regression results for end-to-end joint multiple-object detection and tracking[C] / / European conference on computer vision. Springer, Cham, 2020: 145-161.", referred to as the CTracker algorithm.

[0124] In Table 1, MOTA is the tracking accuracy index, and FPS is the tracking real-time index. The calculation formula is as follows:

[0125]

[0126] FPS = the number of images tracked by the network per second

[0127] where m t is the number of error detections in the t-th frame, is the number of missed detections in the t-th frame, mme t is the number of target ID switches in the tth frame.

[0128] As can be seen from Table 1, on the MOT17 database, the MOTA of the present invention is 68.1% and the FPS is 20; the MOTA of the SiamMOT method is 65.9% and the FPS is 16; the MOTA of CenterTrack is 65.9% and the FPS is 17.5; and the MOTA of CTracker is 66.6% and the FPS is 6.8. Both the MOTA and FPS of the present invention are higher than those of the other three existing technologies, proving that the present invention can achieve better tracking results on the MOT17 dataset. This is mainly because the present invention can enhance the target feature representation capability through a multi-scale fusion module. While utilizing the powerful tracking capabilities of the twin neural network, it builds an end-to-end trainable network structure, synchronously optimizing the network's detection and tracking performance, achieving better detection and tracking performance. At the same time, the use of an anchor-free network structure can greatly reduce the network's computational loss, thereby achieving highly efficient and accurate tracking results.

Claims

1. A multi-target detection and tracking method based on anchor-free Siamese neural network, characterized in that: These include: (1) Construct training and test datasets: At least 5 optical video sequences are composed of a training dataset, and at least 1 optical video sequence is composed of a test dataset, where: Each optical video sequence must contain at least 50 consecutive frames of images, each frame must contain at least one complete moving target, and the position of the moving target in each frame must move by at least 2 pixels compared to the position of the target in the previous frame. The length and width of each frame must not be less than 500×500. The labels of each dataset include the center point of each target in the current frame image, the length and width of the target, the target category, and the target serial number; (2) Constructing an integrated detection and tracking network based on an anchor-free twin neural network: (2a) Build a parallel multi-scale feature fusion module consisting of three downsampling layers and three upsampling layers; (2b) Build a detection branch subnetwork consisting of three convolutional layers and three activation function layers; (2c) Select the existing ResNet50 network and SiamBAN network, and use the SiamBAN network as the tracking branch subnetwork; (2d) Connecting the input of the parallel multi-scale feature fusion module to the third, fourth, and fifth convolutional layers of the existing ResNet50 network, and connecting the output of the parallel multi-scale feature fusion module to the tracking branch sub-network and the detection branch sub-network, respectively, to form an integrated anchor-free twin neural network; (2e) Set the loss function of the integrated detection and tracking network based on the anchor-free Siamese neural network: Loss = L + E2, where: E2 is the loss function used in the known SiamBAN network; L=l heat +l size +l offset is the loss function of the detection branch sub-network, l heat 、l size 、l offset They are the target center heat map loss function, size regression loss function and target center offset loss function of the detection branch network respectively; (3) Using the training dataset to train the integrated detection and tracking network based on the anchor-free twin neural network: (3a) Construct an empty training set target trajectory template set and initialize it according to the label of the first frame image of the training set; (3b) Inputting the target trajectory template set of the training set and the current image into the integrated detection and tracking network based on the anchor-free Siamese neural network to obtain the detection and tracking results; (3c) Calculate the network loss value based on the total loss function and update the network parameters by gradient descent method; (3d) updating the target trajectory template set; (3e) Repeat (3b) to (3d) until the network loss function converges, and obtain the trained integrated detection and tracking network model based on the anchor-free twin neural network; (4) Perform target detection and tracking on the images in the test set: (4a) Input the first frame image of the test set into the trained integrated detection and tracking network based on the anchor-free twin neural network, and output the target detection result of the first frame image; (4b) Construct an empty test set target trajectory template set, and initialize the test set target trajectory template set according to the first frame image of the test set and the detection result of (4a); (4c) Inputting the test set and the initialized test set trajectory template set into the trained integrated detection and tracking network based on the anchor-free twin neural network, and outputting the detection and tracking results of the second frame of the test set and each frame after the second frame; (4d) Match the detection result and the tracking result according to the IOU size, replace the tracking result with the detection result of the matching target, and output the final detection and tracking results.

2. The method according to claim 1, characterized in that The structural relationship and parameters of the three up- and down-sampling layers in the parallel multi-scale feature fusion module (2a) are as follows: The output ends of the first downsampling layer, the second downsampling layer, the third downsampling layer, the first upsampling layer, the second upsampling layer, and the third upsampling layer are connected to the input ends in sequence; The input of the first downsampling layer is connected to the output of the third upsampling layer. The input of the second downsampling layer is connected to the output of the second upsampling layer. An input end of the third downsampling layer is connected to an output end of the third upsampling layer; The first downsampling layer and the third downsampling layer are composed of three parallel convolutional layers with a size of 3×3 and the number of convolution kernels being 128, 256, and 512 respectively; The second downsampling layer includes three parallel convolutional layers with a size of 3×3 and the number of convolution kernels being 64, 128, and 256 respectively; The first upsampling layer, the second upsampling layer, and the third upsampling layer are all linear interpolation layers with an upsampling factor of 2.

3. The method according to claim 1, characterized in that The structural relationship and parameters of the three convolutional layers and three activation function layers in the detection branch sub-network built in (2b) are as follows: The first convolution layer is connected to the first activation function layer, the second convolution layer is connected to the second activation function layer, and the third convolution layer is connected to the third activation function layer, forming three parallel branches; The first convolutional layer is composed of a single convolution layer with a size of 3×3 and a convolution kernel of 1; The second and third convolutional layers are composed of single-layer convolutions with a size of 3×3 and a convolution kernel of 2; The three activation function layers all use a linear rectifier function as the activation function, and the formula is: f(x)=max(0,x).

4. The method according to claim 1, wherein The loss function E2 of the SiamBAN network is known in (2e) and is expressed as follows: Among them, N is the number of input samples for a single training, y i is the positive and negative label of the sample, p i is the network prediction score, and IOU is the intersection-over-union ratio of the network predicted target position to the target true value.

5. The method according to claim 1, characterized in that The target center heat map loss function l in the detection branch network in (2e) heat , size regression loss function l size And the target center offset loss function l offset , respectively expressed as follows: Where N is the number of input training samples in each batch, p, size, and offset are the prediction results of the detection network for the target center confidence, target size, and target center offset respectively; p_gt, size_gt, and offset_gt are the true label values ​​of the target center, target size, and target center offset respectively.

6. The method according to claim 1, characterized in that In (3a), the empty training set target trajectory template set is initialized according to the label of the first frame image of the training set, which is implemented as follows: (3a1) Select the first frame image of the training set video sequence as the input of the network and extract the output feature map of the parallel multi-scale feature fusion module; (3a2) extracting the feature block of the region where the target is located in the output feature map according to the label of each target in the first frame image of the video sequence; (3a3) All target feature blocks are converted into target trajectory template features through the existing technology ROI Align, and each target trajectory template feature is stored in the target trajectory training template set to complete the initialization of the target trajectory training template set.

7. The method according to claim 1, characterized in that In (4b), the empty test set target trajectory template set is initialized according to the first frame image of the test set and the detection result of (4a), and the implementation is as follows: (4b1) Select the first frame of the input test video sequence as the input of the network, extract the feature map output by the parallel multi-scale feature fusion module, (4b2) extracting a feature block of the target region in the output feature map according to the target location in the detection result of (4a); (4b3) All target feature blocks are converted into target trajectory template features through the existing technology ROI Align, and each target trajectory template feature is stored in the target test trajectory template set to complete the initialization of the target trajectory test template set.

Images