Target tracking method and device based on inter-frame spatio-temporal information cross-channel aggregation

Abstract

The invention discloses a target tracking method and device based on inter-frame spatio-temporal information cross-channel aggregation, relates to the field of computer vision, and is used for relieving the perceptual aliasing problem of targets in an image sequence. According to the invention, the trained target tracking network is used to carry out target positioning and post-processing on the to-be-tracked input image sequence so as to realize target tracking. The target tracking network uses a continuous feature capture module to extract spatial-temporal features of the input image sequence and cross-correlation features of a search area image and a template image in the input image sequence; performing cross-channel aggregation on the spatio-temporal features by using a feature aggregation module to obtain spatio-temporal information; learning the cross-correlation features by using an attention mechanism, and decoding the spatio-temporal information according to the learned features to obtain a target feature response graph; and performing target positioning on the target feature response diagram by using a target positioning module. According to the method, the success rate of target tracking is increased, and the robustness of dealing with dynamic information is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision processing technology, and in particular to a target tracking method and device based on cross-channel aggregation of inter-frame spatiotemporal information. Background Art

[0002] Target tracking is an important research direction in the field of computer vision, and it has a wide range of applications in life. For example, in intelligent monitoring systems, it can accurately locate and track targets; in the field of vehicle autonomous driving, it can quickly and effectively perceive and respond to the surrounding environment to improve the safety of autonomous driving; in the field of drone vision, it can effectively improve the drone's ability to track specific targets and autonomous flight capabilities. However, in some application environments, targets often experience local occlusion, lighting changes, and motion deformation. In the face of these situations, the features extracted independently from images with time differences lack continuity in the time domain. As the target continues to move, this property causes the features of the target in different frames to mutate, thereby causing perceptual aliasing problems. These situations interfere with the tracking algorithm's positioning of the target, making target tracking a challenging task.

[0003] Target tracking algorithms can be mainly divided into algorithms based on spatial information and algorithms based on spatiotemporal information. Among them, algorithms based on spatial information mainly focus on the characteristics of the target in a single image frame, such as shape, color, texture, etc. Algorithms based on spatiotemporal information not only consider the characteristics of the target in a single frame, but also consider the changes of the target over time in the video sequence. They can better handle the dynamic changes of the target, have better tracking effects, and have a wider range of applications.

[0004] Current algorithms based on spatiotemporal information usually use two methods to extract spatiotemporal information: updating the target template image and modeling the relationship between multiple image features. The target image template updating method uses a dynamic template to introduce spatiotemporal information. It saves the target in the current image frame as a template and matches it with the next frame. By continuously updating the appearance features of the target, the spatiotemporal information of the features is introduced to achieve a stable tracking effect. The multi-image feature relationship modeling method extracts the features of each frame from the image sequence and models the way the target features change in different environments, so that the algorithm can accurately identify the target features in different environments and improve the robustness of tracking. Both methods extract spatial information from discrete images with time differences, and then splice them in the time dimension to obtain spatiotemporal information.

[0005] The Chinese patent document with the publication number CN116109678A discloses a target tracking method based on a contextual self-attention learning deep network. First, based on the convolutional neural network model and the contextual self-attention learning module, a feature extraction network model is constructed; then, the region proposal network is used to perform feature fusion deep convolution calculations on the extracted target template image features and the search area image features in the classification branch and the regression branch, respectively, to obtain the corresponding similarity score map, and the similarity score map corresponding to the maximum similarity score value is determined as the tracking and positioning target, and finally more accurate tracking is achieved. Although this method uses context information to improve the target representation ability, the drone tracking environment is complex, and background changes may make it difficult to model context information, thereby affecting the effect of the algorithm in dealing with the perceptual aliasing problem and leading to misjudgment of the target. Summary of the invention

[0006] The purpose of the present invention is to provide a target tracking method and device based on cross-channel aggregation of inter-frame spatiotemporal information to alleviate the perceptual aliasing problem of targets in image sequences and improve the ability to distinguish targets.

[0007] The technical solution adopted by the present invention is as follows:

[0008] A target tracking method based on cross-channel aggregation of inter-frame spatiotemporal information includes the following processes:

[0009] Using a trained target tracking network to locate the input image sequence to be tracked, and post-processing the target positioning result to achieve target tracking; the target tracking network is trained using a training set;

[0010] The target tracking network uses a continuous feature capture module to extract the spatiotemporal features of the input image sequence and the cross-correlation features of the search area image and the template image in the input image sequence; uses a feature aggregation module to perform cross-channel aggregation on the spatiotemporal features to obtain spatiotemporal information; uses an attention mechanism to learn the cross-correlation features, decodes the spatiotemporal information according to the learned features, and obtains a target feature response map; and uses a target positioning module to locate the target on the target feature response map.

[0011] Optionally, the using a target positioning module to perform target positioning on the target feature response graph includes:

[0012] Using the classification branch of the target positioning module to classify the target feature response map to obtain a classification response map;

[0013] Using the regression branch of the target positioning module to perform regression processing on the target feature response map to obtain a regression response map;

[0014] The quality perception branch of the target positioning module is used to evaluate the positioning quality of the target feature response map based on the template image features to obtain a positioning quality score.

[0015] Optionally, the post-processing of the target positioning result to achieve target tracking includes:

[0016] Applying a cosine window penalty to the classification response map to obtain an updated classification response map, reversely mapping the updated classification response map back to the coordinate dimension of the input image sequence, and finding the point with the maximum positioning quality score as the center point of the target;

[0017] According to the center point of the target, a corresponding point is found on the regression response graph, an offset of the corresponding point from the target identifier is calculated, and based on the offset, the target identifier coordinates are calculated from the center point coordinates of the target to achieve target positioning;

[0018] The response values ​​in the target feature response map are scored through the quality perception branch to obtain the spatial positioning quality score; the semantic positioning quality score is calculated by measuring the feature similarity between the positioned target identification image and the template image; and the sum of the spatial positioning quality score and the semantic positioning quality score is used as the positioning quality score.

[0019] Optionally, the continuous feature capture module includes a spatiotemporal feature extraction unit and two weight-sharing image feature extraction units; the step of extracting the spatiotemporal features of the input image sequence and the cross-correlation features of the search area image and the template image in the input image sequence by using the continuous feature capture module includes:

[0020] Extracting the spatiotemporal features of the input image sequence using a spatiotemporal feature extraction unit;

[0021] Using the two image feature extraction units to respectively extract the first image feature of the search area image in the input image sequence and the second image feature of the template image;

[0022] The first image feature and the second image feature are cross-correlated with each other at different layers to obtain feature response maps at different depths as cross-correlation features.

[0023] Optionally, the feature aggregation module includes a cascaded long-distance spatiotemporal information capture unit and a cross-channel aggregation feature unit; and the step of performing cross-channel aggregation on the spatiotemporal features using the feature aggregation module to obtain spatiotemporal information includes:

[0024] The long-distance spatiotemporal information capture unit extracts spatiotemporal data from the spatiotemporal features using a multi-layer perceptron;

[0025] The cross-channel aggregation feature unit aggregates spatiotemporal information from the spatiotemporal data using cascaded convolution and deconvolution operations.

[0026] Optionally, the using of an attention mechanism to learn the cross-correlation features, decoding the spatiotemporal information according to the learned features, and obtaining a target feature response map includes:

[0027] After concatenating the feature response maps of different depths, encoding them using an encoder including a multi-head attention operation, a normalization operation, and a feedforward propagation operation;

[0028] A decoder including a multi-head attention operation, a normalization operation and a feedforward propagation operation is used to decode the spatiotemporal information based on the encoding result of the encoder to obtain a target feature response map.

[0029] Optionally, the method for the decoder to decode the spatiotemporal information includes:

[0030] Generate a dynamic mask along the channel dimension of the spatiotemporal information:

[0031]

[0032] in, Representing spatiotemporal information The feature map of the cth channel, C is the spatiotemporal information The number of channels;

[0033] According to the threshold of the mask, the encoding result F mv Divide into blocks and get the target feature response map The block method is:

[0034]

[0035] Where Conv4(·) represents a convolution with 4 channels and a kernel size of 1×1; Conv16(·) represents a convolution with 16 channels and a kernel size of 1×1; Binary(·) represents a binarization function.

[0036] Optionally, the loss function for training the target tracking network is:

[0037] Loss=Weight cls ×loss cls +Weight loc ×loss loc +Weight st ×S L ,

[0038] Among them, loss cls is the classification loss, loss loc is the regression loss, S L is the positioning quality score, Weightcls is the weight of the classification loss, Weight loc is the weight of the regression loss, Weight st is the weight of the positioning quality score.

[0039] Optionally, when training the target tracking network, a gradient descent method is used to optimize network parameters.

[0040] The present invention also provides a target tracking device based on cross-channel aggregation of inter-frame spatiotemporal information, including a processor and a storage medium, wherein the processor runs a computer program stored in the storage medium to execute the above-mentioned target tracking method based on cross-channel aggregation of inter-frame spatiotemporal information.

[0041] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0042] The present invention establishes a continuous feature capture module, which can cross-correlate features of different depths extracted from the search area image and the template image to provide more reliable information for the model. The present invention establishes a feature aggregation module, which can aggregate spatiotemporal information across channels, which helps the model capture the diversity and complexity of spatiotemporal features and improves the robustness of the target tracking algorithm to dynamic information. The present invention constructs a target tracking network composed of a cascade of a continuous feature capture module, a feature aggregation module, an attention module and a target positioning module, which can effectively deal with the perceptual aliasing problem caused by interference such as partial occlusion of the target, rapid movement and similar targets, and improve the success rate of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0044] Figure 1 FIG. 4 is a network structure diagram of a target tracking network in one embodiment.

[0045] Figure 2 is a structural diagram of a spatiotemporal feature extraction unit in one embodiment.

[0046] Figure 3 is a structural diagram of a feature aggregation module in one embodiment. DETAILED DESCRIPTION

[0047] All features disclosed in this specification, or steps in all methods or processes disclosed, except mutually exclusive features and / or steps, can be combined in any manner.

[0048] Any feature disclosed in this specification (including any additional claims and abstract), unless otherwise stated, may be replaced by other equivalent or alternative features having similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

[0049] In order to alleviate the perceptual aliasing problem of targets in image sequences and improve the success rate of target tracking in scenarios prone to occlusion, lighting changes, etc., such as drone visual tracking, an embodiment of the present application provides a target tracking method based on cross-channel aggregation of inter-frame spatiotemporal information. The method uses a trained target tracking network to locate the target in the input image sequence to be tracked, and post-processes the target positioning results to achieve target tracking.

[0050] The target tracking network is obtained by training the constructed target tracking network model using the constructed training set.

[0051] For the constructed training set, in some embodiments, the training set is formed by acquiring the VID dataset, the GOT-10K dataset, the Lasot dataset, and the COCO dataset from public websites, and aggregating the video sequences in these datasets.

[0052] The UAV123 dataset, UAV123@10fps dataset, and DTB70 dataset are obtained from public websites, and the video sequences in these datasets are collected to form a test set.

[0053] The training set is used to train the target tracking network model, and the test set is used to perform performance tests on the trained target tracking model to verify the performance of the target tracking network.

[0054] The target tracking network uses a continuous feature capture module to extract the spatiotemporal features of the input image sequence and the cross-correlation features of the search area image and the template image in the input image sequence; uses a feature aggregation module to perform cross-channel aggregation on the spatiotemporal features to obtain spatiotemporal information; uses an attention mechanism to learn the cross-correlation features, decodes the spatiotemporal information according to the learned features, and obtains a target feature response map; and uses a target positioning module to locate the target on the target feature response map.

[0055] Therefore, if Figure 1 As shown, the target tracking network includes a cascaded continuous feature capture module, a feature aggregation module, an attention module and a target localization module.

[0056] 1. Continuous feature capture module

[0057] In some embodiments, the continuous feature capture module includes a spatiotemporal feature extraction unit, and two weight-sharing image feature extraction units.

[0058] 1) Use the spatiotemporal feature extraction unit to extract the spatiotemporal features of the input image sequence.

[0059] In some possible implementations, such as Figure 2 As shown, the spatiotemporal feature extraction unit is composed of a plurality of stacked three-dimensional convolution modules, which is intended to extract the spatiotemporal features in the input image sequence. In some specific embodiments, a two-dimensional spatial convolution module C1 and a one-dimensional temporal convolution module C2 are spliced ​​into a three-dimensional convolution module. For example, the convolution kernel size of the two-dimensional spatial convolution module C1 is 1×1, and the step size is 1; the convolution kernel size of the one-dimensional temporal convolution module C2 is 7×7, and the step size is 1; the convolution kernel size of the three-dimensional convolution module C3 is 3×3×3, and the step size is 1. The spatiotemporal feature extraction operation first obtains the intermediate feature F1 of the spatiotemporal feature extraction unit through the three-dimensional convolution module C3, and then obtains the intermediate feature F2 of the spatiotemporal feature extraction unit through the activation layer and the multi-layer feature extraction layer. F2 then passes through the activation layer plus F1 through the fully connected layer to obtain the spatiotemporal information F in the input image sequence. S The process of extracting spatiotemporal features in the spatiotemporal feature extraction unit is shown in the formula:

[0060]

[0061] Where S is the input image sequence, F S is the spatiotemporal feature in the input image sequence, F1 and F2 are the intermediate features extracted by the spatiotemporal information perception unit, TConv(·) represents one-dimensional temporal convolution, SConv(·) represents two-dimensional spatial convolution, Activate(·) represents activation function, Multlayer(·) represents multiple feature extraction layers, and FC(·) represents fully connected layer.

[0062] 2) Using two weight-sharing image feature extraction units, first image features of the search area image in the input image sequence and second image features of the template image are respectively extracted.

[0063] In some feasible implementations, the image feature extraction unit uses an AlexNet network, which is used to extract image features of the search area image or the template image. In a specific implementation, the AlexNet network can be composed of three groups of convolution modules C4, C5, and C6 with different structures in cascade, wherein the convolution kernel size of the convolution module C4 is 11×11 and the step size is 2; the convolution kernel size of the convolution module C5 is 5×5 and the step size is 2; the convolution kernel size of the convolution module C6 is 3×3 and the step size is 2.

[0064] When extracting image features, the image feature extraction unit first passes through the convolution module C4 and a pooling layer, then passes through the convolution module C5 and a pooling layer, then passes through three convolution modules C6 and a pooling layer, and finally passes through three fully connected layers to obtain the image features.

[0065] The image feature extraction unit of the above structure (or other equivalent structures) is used to extract the first image feature of the search area image in the input image sequence and the second image feature of the template image.

[0066] 3) Cross-correlating the first image feature and the second image feature at different layers to obtain feature response maps at different depths as cross-correlation features.

[0067] Image feature extraction and cross-correlation operations can be expressed as follows in some specific implementations:

[0068]

[0069] Among them, I x ,I z are the input search area image and template image, respectively, x 、F z are the corresponding first image feature and second image feature, F xi 、F zi Respectively represent the information of the i-th layer of different image features, M ri Indicates the feature response map with depth i. In this embodiment, the feature response maps of the 4th and 5th layers after cross-correlation are taken. The image features of different depths extracted from the image are cross-correlated, and the deep semantic information and shallow detail information are integrated to provide more reliable information for the target tracking network, so that the target tracking network can better understand the structure and semantics in the image.

[0070] 2. Feature Aggregation Module

[0071] In some embodiments, the feature aggregation module includes a cascaded long-range spatiotemporal information capture unit and a cross-channel aggregation feature unit.

[0072] 1) The long-distance spatiotemporal information capture unit uses a multi-layer perceptron to extract spatiotemporal data from the spatiotemporal features.

[0073] The long-distance spatiotemporal information capture unit extracts spatiotemporal data, which can be expressed as follows:

[0074] F S1 =Activate(W*F S +B), (3)

[0075] Among them, F S1 is the spatiotemporal data for establishing spatial connections, Activate(·) is the activation function, W is the mapping matrix weight, B is the bias vector, and * represents matrix multiplication.

[0076] 2) The cross-channel aggregation feature unit utilizes cascaded convolution and deconvolution operations to aggregate spatiotemporal information from the spatiotemporal data.

[0077] The cross-channel aggregation feature unit aggregates features across channels through the convolutional deconvolution network, realizes parallel perception through multi-channel convolution kernels, captures the main features of each channel, integrates and interacts the features of different channels in spatial and temporal dimensions, and obtains the spatiotemporal information after cross-channel aggregation. The aggregation operation can be expressed as follows:

[0078]

[0079] Here, Deconv(·) represents deconvolution and Conv(·) represents convolution. Aggregating spatiotemporal information across channels helps the target tracking network capture the diversity and complexity of spatiotemporal data and improves the robustness of the target tracking algorithm to dynamic information.

[0080] 3. Attention Module

[0081] In some embodiments, the attention module adopts a Transformer encoder-decoder structure to capture global contextual relationships.

[0082] Both the Transformer encoder and the Transformer decoder contain multi-head attention operations, normalization operations, and feed-forward propagation operations.

[0083] 1) The feature response maps of different depths are concatenated and encoded using the Transformer encoder.

[0084] According to the above embodiment, the first characteristic response map M r4 and the second characteristic response map M r5 The input is sent to the Transformer encoder for encoding. The correlation between each position feature and other position features is calculated by self-attention. The features at each position are weighted and summed according to the correlation to establish the connection between the information at different positions in the sequence, thereby capturing the global contextual relationship. Multiple parallel attention heads obtained by expanding the self-attention layer are used to learn different weights, so as to learn features in different representation subspaces and enhance the model's learning ability for different focus points.

[0085] In some feasible implementations, the process of calculating the weight matrix of the Transformer encoder for feature response maps of different depths is as follows:

[0086]

[0087] Among them, F mis the concatenated feature response graph, Concat(·) represents the concatenation function, M Q represents the query matrix, M K represents the bond matrix, M V represents the value matrix, W Q ,W K ,W V Represent the corresponding transformation matrices respectively.

[0088] That is, for the first feature response map M of the input r4 and the second characteristic response map M r5 , after splicing, we get F m After that, in the multi-head attention operation, F m Multiply by the query transformation matrix W Q , key transformation matrix W K , value transformation matrix W V , get the query matrix M Q , key matrix M K , value matrix M V .

[0089] Then calculate the multi-head attention score MutiAttn as follows:

[0090]

[0091] Among them, Attn is the attention score of a single attention head, Attn i represents the attention score of the ith attention head, SoftMax(·) represents the normalized exponential function, Transpose(·) represents transposition, dim(·) represents the dimension of the matrix, Liner(·) represents the linear transformation function, Concat(·) represents the concatenation function, and n represents the number of attention heads.

[0092] For the multi-head attention score MutiAttn(F m ) performs normalization and feedforward propagation operations to complete the encoding. The encoding result is F mv Each operation can be expressed by the following formula:

[0093]

[0094] Among them, LayerNorm(·) represents the normalization function, and Feedforward(·) represents the feedforward propagation function.

[0095] 2) Using a decoder including multi-head attention operation, normalization operation and feedforward propagation operation based on the encoding result F mv The spatiotemporal information Decode and obtain the target feature response map.

[0096] In some feasible implementations, the method in which the decoder decodes the spatiotemporal information includes:

[0097] Information along space and time Generate a dynamic mask Mask with the channel dimension:

[0098]

[0099] in, Representing spatiotemporal information The feature map of the cth channel, C is the spatiotemporal information The number of channels.

[0100] According to the threshold of the mask, the encoding result F mv Divide into blocks and get the target feature response map The response value exceeds the threshold F mv A large block indicates that it may contain more target information, whereas a small block indicates that it contains a large amount of irrelevant information. The block division method is:

[0101]

[0102] In the formula, Conv4(·) indicates a convolution with 4 channels and a kernel size of 1×1; Conv16(·) indicates a convolution with 16 channels and a kernel size of 1×1; Binary(·) indicates a binarization function. Using the dynamic tokens generated by the mask, the decoder can focus on the salient features of the target, minimize irrelevant information in the spatiotemporal features, and finally decode the Used to locate the target.

[0103] 4. Target Positioning Module

[0104] In some embodiments, the target positioning module responds to the target feature map Predict the target center and size. Specifically, the target localization module is used to respond to the target feature map. During the target positioning process, there are:

[0105] 1) Using the classification branch of the target positioning module to respond to the target feature map Perform classification processing and obtain a classification response map.

[0106] In some feasible implementations, the classification branch is composed of four convolutional layers of the same structure cascaded and then connected to a convolutional layer that outputs the classification result. Each of the four convolutional layers of the same structure is composed of convolution, group normalization (GroupNormalization), and ReLU activation function, and the convolutional layer that outputs the classification result is composed of convolution and SoftMax normalized exponential function.

[0107] In the classification branch, the target feature response map It is parsed and reduced in dimension, and a classification response map of size 2×H×W is output, which contains the category information and confidence of the predicted target.

[0108] 2) Use the regression branch of the target positioning module to respond to the target feature map Perform regression processing and obtain the regression response diagram.

[0109] In some feasible implementations, the regression branch is composed of four convolutional layers with the same structure cascaded and then connected to a convolutional layer that outputs the regression result. Each of the four convolutional layers with the same structure is composed of convolution, group normalization, and ReLU activation function, and the convolutional layer that outputs the regression result is composed of convolution and ReLU activation function.

[0110] The processing of the regression branch is similar to that of the classification branch, and the final output is a regression response map of size , which contains the coordinate information of all possible predicted positions.

[0111] 3) Using the quality perception branch of the target positioning module based on the second image feature F of the template image z Response plot to target feature Perform positioning quality evaluation and obtain a positioning quality score.

[0112] The second image feature F of the template image has been extracted in the continuous feature capture module. z The quality perception branch is based on the second image feature F z Response plot to target feature Conduct positioning quality evaluation.

[0113] In some feasible implementations, the positioning quality score includes a spatial positioning quality score and a semantic positioning quality score. The two scores are calculated as follows:

[0114]

[0115] Among them, S spatio is the spatial localization quality score, m is the number of response values ​​in the feature response map in the input image sequence, Max(x p ,y p ) are the horizontal and vertical coordinates of the maximum response point, x t ,y t are the horizontal and vertical coordinates of the feature response point of the t-th frame image, is the average coordinate of m response points. The spatial positioning quality score is used to determine whether the current positioning result is in a drift state. sim is the semantic localization quality score, It is the target identification image currently positioned, that is, the image of the predicted target area.

[0116] Add the two scores to get the final positioning quality score S L :

[0117] S L =S spatio +S sim , (11)

[0118] It can be seen that the target positioning result finally obtained by the target tracking network is the classification response map, regression response map and positioning quality score.

[0119] When training a target tracking network using a training set, the training is usually performed with the goal of minimizing the loss function, that is, the target tracking network is iteratively trained until the loss converges or the set number of iterations is reached.

[0120] In some embodiments, the loss function of the target tracking network includes three parts, namely, classification loss, regression loss, and positioning quality score, expressed as:

[0121] Loss=Weight cls ×loss cls +Weight loc ×loss loc +Weight st ×S L , (12)

[0122] Among them, loss cls is the classification loss, loss loc is the regression loss, S L is the positioning quality score, Weight cls is the weight of the classification loss, which is set to 1.2 in this example. loc is the weight of the regression loss, which is set to 3 in this example. st is the weight of the positioning quality score, which is set to 1 in this embodiment.

[0123] In some feasible implementations, the classification loss function loss cls Using the binary cross entropy loss function, the calculation process of the classification loss is shown as follows:

[0124]

[0125] Among them, N is the number of prediction samples, label is the predicted target label, and pred is the probability of the predicted target.

[0126] Regression loss function loss locIt consists of the intersection-over-union loss function and the center point loss function. The calculation process is shown in the following formula:

[0127] loss loc =λ3×IOU(bbox,GT)+λ4×Centerloss(bbox,GT), (14)

[0128] Wherein, IOU(·) represents the intersection-over-union loss function, Centerloss(·) represents the center point loss function, bbox represents the predicted target rectangular box, GT represents the real target rectangular box, λ3 and λ4 are the weights of the intersection-over-union loss and the center point loss, respectively, and are both set to 1.0 in this embodiment.

[0129] The calculation process of Centerloss(·) is shown in the following formula:

[0130]

[0131] Among them, bbx and bby are the horizontal and vertical coordinates of the center point of the predicted target, GTx and GTy are the horizontal and vertical coordinates of the center point of the real target, and GTw and GTh are the width and height of the target position.

[0132] Based on the constructed target tracking network and the designed loss function, the target tracking network is iteratively trained using the training set.

[0133] In some embodiments, when training the target tracking network, a gradient descent method is used to optimize network parameters.

[0134] Specifically include:

[0135] 1) Set the basic learning rate to 0.003, use the SGD optimizer, set the weight decay to 0.0001, the number of positive and negative samples is 16, the BatchSize is set to 128, and train for 70 rounds in total.

[0136] 2) Input the training set into the target tracking network model and perform forward propagation on it to obtain the predicted classification response map and regression response map.

[0137] 3) Use the loss function Loss to calculate the loss value between the predicted result and the actual result of the target tracking network, use the loss value to backpropagate the network, and calculate the gradient of each parameter

[0138] 4) Use the SGD optimizer to calculate the gradient Update the network parameters and get the currently updated parameters w k :

[0139]

[0140] Among them, w k-1 is the parameter of the model at the previous moment, α is the learning rate;

[0141] 5) Repeat the above steps 2) to 4) until the specified number of training rounds is reached to obtain a trained target tracking network.

[0142] For the input image sequence to be tracked, after it is input into the trained target tracking network, the corresponding classification response map, regression response map and positioning quality score are obtained. After post-processing, target tracking is achieved.

[0143] In some embodiments, the post-processing method for the target positioning result includes:

[0144] A cosine window penalty is applied to the classification response map to obtain an updated classification response map, and the updated classification response map is reversely mapped back to the coordinate dimension of the input image sequence, and the point with the maximum positioning quality score is found as the center point of the target.

[0145] According to the center point of the target, the corresponding point is found on the regression response graph, and the offset of the corresponding point from the target mark is calculated. Based on the offset, the target mark coordinates are calculated from the center point coordinates of the target to achieve target positioning. When locating or tracking a target, a rectangular box is usually used as a target mark to mark the position of the target. Therefore, the offset of the corresponding point from the rectangular box and the four sides of the rectangular box is usually calculated here, and then these offsets are added and subtracted from the center point coordinates of the target to obtain the predicted rectangular box coordinates. The area framed by the rectangular box is the located target.

[0146] The response values ​​in the target feature response map are scored through the quality perception branch to obtain the spatial positioning quality score; the semantic positioning quality score is calculated by measuring the feature similarity between the area image indicated by the positioned target logo (i.e., the target logo image) and the template image. The sum of the spatial positioning quality score and the semantic positioning quality score is used as the positioning quality score.

[0147] An embodiment of the present application also provides a target tracking device based on cross-channel aggregation of inter-frame spatiotemporal information, the device comprising a processor and a storage medium, the storage medium storing a computer program, the processor running the computer program in the storage medium to execute the target tracking method based on cross-channel aggregation of inter-frame spatiotemporal information in the above embodiment.

[0148] The embodiment of the present application also verifies the designed method to verify its effectiveness.

[0149] 1) Simulation experiment conditions:

[0150] The hardware platform of the simulation experiment in this embodiment is: the processor is Intel Core i9-12900KF, the main frequency is 3.1GHz, the memory is 32G, and the graphics card is NVIDIA GeForce RTX3090.

[0151] The hardware platform of the simulation experiment in this embodiment is: Ubuntu 20.04.6 LTS operating system, Python 3.8.0 language, and uses the programming framework of pytorch 1.12.1 based on CUDA 10.2 version.

[0152] The data sets used in the simulation experiment of this embodiment are the UAV123 data set, UAV123@10fps data set, and DTB70 data set obtained from public websites. The UAV123 data set is a public data set for the target tracking algorithm performance test of the UAV platform proposed in 2016. There are 123 video sequences in total, more than 110K frames, including 42 scenes such as urban roads, buildings, beaches, etc., including common problems in UAV tracking such as target scale changes, background clutter, camera movement, scale changes, and perspective changes. The UAV123@10fps data set is a data set obtained by downsampling the UAV123 data set. The features change more rapidly, and it contains 123 challenging video sequences. The DTB70 data set is a UAV perspective data set proposed in 2017. A total of 70 short video sequences are given, which contain a large number of targets of similar categories and have multiple challenging factors, such as background interference and similar targets.

[0153] The indicators for evaluating these three data sets are tracking success rate SR and tracking accuracy P:

[0154] The success rate SR is related to the overlap score. The overlap score OvrelapScore refers to the manually annotated rectangular box B gt and the rectangular box B estimated by the tracking algorithm tr The ratio of the intersection to the union of : When the overlap rate score of a frame is greater than the set threshold, the frame is considered to be tracked successfully, and the percentage of total successful frames to all frames is the success rate.

[0155] The accuracy rate P is related to the center point error. The center point error CenterError refers to the center point C of the manually marked rectangular box. gt The center point C of the rectangular box estimated by the tracking algorithm tr Euclidean distance between: CenterError = || C gt -C tr ||2, the percentage of video frames whose center position error is less than a given threshold among all frames is the accuracy rate.

[0156] 2) Simulation content and result analysis:

[0157] In simulation experiment 1, the present invention and five prior arts HiFT, SiamAPN, SiamAPN++, SiamRPN++, and DaSiamRPN are used to track the target of the test set UAV123, and the respective success rates and accuracy rates are obtained. The results are shown in Table 1.

[0158] Table 1 Comparison of success rate and accuracy of different tracking methods on the test set UAV123

[0159]

[0160] As can be seen from Table 1, compared with the existing excellent technologies, the method proposed in this application has achieved the best results in both the success rate and accuracy of tracking the target, proving that this method can effectively deal with the perceptual aliasing problem that occurs during the target tracking process.

[0161] In simulation experiment 2, the present application and five existing technologies SiamAPN++, SiamAPN, HiFT, DASiamRPN, and UpdateNet are used to track the target on the test set UAV123@10fps, and their respective success rates and accuracy rates are obtained. The results are shown in Table 2.

[0162] Table 2 Comparison of success rate and accuracy of different tracking methods on the test set UAV123@10fps

[0163]

[0164] As can be seen from Table 2, compared with the existing excellent technologies, the method proposed in this application has achieved the best results in terms of success rate and accuracy in tracking targets, proving that the smooth feature change process can introduce more information, making the algorithm have better tracking performance than the algorithm that only uses the drastically changing spatial features.

[0165] In simulation experiment 3, the present application and five existing technologies, LightTrack, SiamAPN, SiamGAT, SiamMASK, and DASiamRPN, are used to track the target on the test set DTB70, and their respective success rates and accuracy rates are obtained. The results are shown in Table 3.

[0166] Table 3 Comparison of success rate and accuracy of different tracking methods on the test set DTB70

[0167]

[0168]

[0169] As can be seen from Table 3, compared with the existing excellent technologies, the method proposed in this application has achieved the best results in terms of target tracking success rate and accuracy, which proves that the method proposed in this application is still effective in the face of a large number of similar target interferences.

[0170] The simulation results show that the target tracking algorithm based on cross-channel aggregation of inter-frame spatiotemporal information proposed in this application starts from the overall perspective of the image sequence, globally extracts the spatiotemporal information in the image sequence, and smoothes the changing process of the target features in a complex environment to alleviate the perceptual aliasing problem, suppress the interference of strong related information at other locations on the algorithm, and effectively improves the accuracy and robustness of the target tracking algorithm, thereby improving the success rate and accuracy of target tracking.

[0171] The present invention is not limited to the above-mentioned specific embodiments, but extends to any new features or any new combination disclosed in this specification, as well as any new method or process steps or any new combination disclosed.

Claims

1. A target tracking method based on cross-channel aggregation of inter-frame spatiotemporal information, characterized in that: The process includes: The trained target tracking network is used to locate the input image sequence to be tracked, and the target positioning result is post-processed to achieve target tracking; The target tracking network is trained using a training set; The target tracking network uses a continuous feature capture module to extract the spatiotemporal features of the input image sequence and the cross-correlation features of the search area image and the template image in the input image sequence; A feature aggregation module is used to perform cross-channel aggregation on the spatiotemporal features to obtain spatiotemporal information; an attention mechanism is used to learn the cross-correlation features, and the spatiotemporal information is decoded according to the learned features to obtain a target feature response map; A target positioning module is used to perform target positioning on the target feature response map.

2. The target tracking method based on cross-channel aggregation of inter-frame spatiotemporal information according to claim 1, characterized in that: The method of using a target positioning module to perform target positioning on the target feature response graph includes: Using the classification branch of the target positioning module to classify the target feature response map to obtain a classification response map; Using the regression branch of the target positioning module to perform regression processing on the target feature response map to obtain a regression response map; The quality perception branch of the target positioning module is used to evaluate the positioning quality of the target feature response map based on the template image features to obtain a positioning quality score.

3. The target tracking method based on cross-channel aggregation of inter-frame spatiotemporal information as claimed in claim 2, characterized in that: The post-processing of the target positioning result to realize target tracking includes: Applying a cosine window penalty to the classification response map to obtain an updated classification response map, reversely mapping the updated classification response map back to the coordinate dimension of the input image sequence, and finding the point with the maximum positioning quality score as the center point of the target; According to the center point of the target, a corresponding point is found on the regression response graph, an offset of the corresponding point from the target identifier is calculated, and based on the offset, the target identifier coordinates are calculated from the center point coordinates of the target to achieve target positioning; The response values ​​in the target feature response map are scored through the quality perception branch to obtain the spatial positioning quality score; the semantic positioning quality score is calculated by measuring the feature similarity between the positioned target identification image and the template image; and the sum of the spatial positioning quality score and the semantic positioning quality score is used as the positioning quality score.

4. The target tracking method based on cross-channel aggregation of inter-frame spatiotemporal information according to claim 1, characterized in that: The continuous feature capture module includes a spatiotemporal feature extraction unit and two weight-sharing image feature extraction units; the method of extracting the spatiotemporal features of the input image sequence and the cross-correlation features of the search area image and the template image in the input image sequence by using the continuous feature capture module includes: Extracting the spatiotemporal features of the input image sequence using a spatiotemporal feature extraction unit; Using the two image feature extraction units to respectively extract the first image feature of the search area image in the input image sequence and the second image feature of the template image; The first image feature and the second image feature are cross-correlated with each other at different layers to obtain feature response maps at different depths as cross-correlation features.

5. The target tracking method based on cross-channel aggregation of inter-frame spatiotemporal information as claimed in claim 1, characterized in that: The feature aggregation module includes a cascaded long-distance spatiotemporal information capture unit and a cross-channel aggregation feature unit; the use of the feature aggregation module to perform cross-channel aggregation on the spatiotemporal features to obtain spatiotemporal information includes: The long-distance spatiotemporal information capture unit extracts spatiotemporal data from the spatiotemporal features using a multi-layer perceptron; The cross-channel aggregation feature unit aggregates spatiotemporal information from the spatiotemporal data using cascaded convolution and deconvolution operations.

6. The target tracking method based on cross-channel aggregation of inter-frame spatiotemporal information as claimed in claim 4, characterized in that: The method of using the attention mechanism to learn the cross-correlation features, decoding the spatiotemporal information according to the learned features, and obtaining a target feature response map includes: After concatenating the feature response maps of different depths, encoding them using an encoder including a multi-head attention operation, a normalization operation, and a feedforward propagation operation; A decoder including a multi-head attention operation, a normalization operation and a feedforward propagation operation is used to decode the spatiotemporal information based on the encoding result of the encoder to obtain a target feature response map.

7. The target tracking method based on cross-channel aggregation of inter-frame spatiotemporal information according to claim 6, characterized in that: The method by which the decoder decodes the spatiotemporal information comprises: Generate a dynamic mask along the channel dimension of the spatiotemporal information: in, Representing spatiotemporal information The feature map of the cth channel, C is the spatiotemporal information The number of channels; According to the threshold of the mask, the encoding result F mv Divide into blocks and get the target feature response map The block method is: Where Conv4(·) represents a convolution with 4 channels and a kernel size of 1×1; Conv16(·) represents a convolution with 16 channels and a kernel size of 1×1; Binary(·) represents a binarization function.

8. The target tracking method based on cross-channel aggregation of inter-frame spatiotemporal information according to claim 2 or 3, characterized in that: The loss function for training the target tracking network is: Loss=Weight cls ×loss cls +Weight loc ×loss loc +Weight st ×S L , Among them, loss cls is the classification loss, loss loc is the regression loss, S L is the positioning quality score, Weight cls is the weight of the classification loss, Weight loc is the weight of the regression loss, Weight st is the weight of the positioning quality score.

9. The target tracking method based on cross-channel aggregation of inter-frame spatiotemporal information as claimed in claim 1, characterized in that: When training the target tracking network, the gradient descent method is used to optimize the network parameters.

10. A target tracking device based on cross-channel aggregation of inter-frame spatiotemporal information, comprising a processor and a storage medium, characterized in that: The processor runs the computer program stored in the storage medium to execute the target tracking method based on cross-channel aggregation of inter-frame spatiotemporal information as described in any one of claims 1-9.

Images