Triple attention method applied to satellite video multi-target tracking

Abstract

The invention discloses a triple attention method applied to satellite video multi-target tracking, and relates to the technical field of satellite video multi-target tracking, and the method comprises the steps: inputting a satellite video frame sequence into a triple attention module, and obtaining an enhanced feature map; inputting the enhanced feature map into a backbone network to obtain a feature map; respectively inputting the feature maps into a detection head and a tracking head; and updating and optimizing the network by using the loss function. According to the triple attention method applied to satellite video multi-target tracking, a triple attention mechanism is used, the current frame feature is enhanced from three dimensions of space, channel and time, the importance of each channel and space position in a feature map is adaptively re-calibrated, time information is fused through adjacent frame feature aggregation, and the target tracking accuracy is improved. Therefore, the representation capability and decision accuracy of the model are improved, and the method is suitable for the multi-target tracking problem in a satellite video.

Description

Technical Field

[0001] The present invention relates to the technical field of satellite video multi-target tracking, and in particular to a triple attention method applied to satellite video multi-target tracking. Background Art

[0002] Multiple Object Tracking (MOT) is an important research direction in the field of computer vision. It aims to detect and track multiple targets in a video and assign and maintain a unique identity for each target. With the rapid development of deep learning, the field of MOT has also made significant progress, but it still faces many challenges, such as target occlusion, appearance changes, and changes in the number of targets. The usual MOT paradigm is divided into detection-based tracking (DBT) and joint detection and tracking (JDT). The DBT framework first detects the target in each frame, and then matches the detection results with the existing trajectories through a data association algorithm. SORT and DeepSORT are typical representatives of the DBT paradigm. The SORT algorithm takes the detection box of the current frame as input, uses the Kalman filter to predict the possible position of the target in the next frame of the current frame, and uses the Hungarian algorithm to match the predicted box and the detection box to associate the detection results of different frames. Finally, the Kalman filter is updated with the associated detection box. However, SORT did not consider the matching of long-term lost trajectories, resulting in frequent target ID switching. Later, DeepSORT introduced cascade matching, which greatly reduced the occurrence of short trajectories and the number of ID switching by combining appearance cosine similarity and Mahalanobis distance and setting up a trajectory determination mechanism. IOU-Tracker only uses intersection-over-union (IOU) to associate the detection results of different frames. After that, VIOU-Tracktor added a visual single object tracker (SOT) as a motion model to solve the matching failure caused by the target moving too fast. Specifically, for unmatched trajectories, SOT is used to predict the position of subsequent frames and match them with the detection results. If the match is successful within a period of time, the tracking trajectory is restored, otherwise the tracking is stopped; for newly initialized tracking, SOT is used for reverse tracking. If it coincides with a certain trajectory, it is merged. At the same time, some scholars have also tried to model the multi-target tracking task as a graph planning problem, usually taking the detection boxes of adjacent frames as nodes, and using the edges between nodes to map the association between the detection boxes. For example, in the article Learning a neural solver for multiple object tracking, it is assumed that all nodes are connected, and the value of each edge (0 or 1) is predicted, converting the multi-classification problem of nodes into a binary classification problem of edges.

[0003] The JDT framework uses a shared network structure to simultaneously detect and track targets, improving the end-to-end performance of the algorithm. The JDE proposed by Wang et al. merges the detection and appearance feature extraction modules, and no longer uses a separate re-identification model to obtain target representation information. The detected target box features can be directly used for data association. The Tracktor network proposed by Bergmann et al. directly uses the bounding box regression module in Faster R-CNN to predict the tracking results, thereby removing the dedicated tracking prediction module. The CTracker network proposed by Peng et al. designs an end-to-end online multi-target tracking model, which transforms the data association problem in tracking prediction into a target pair detection problem. The MAT proposed by Han et al. uses information from multiple kinematic models to predict trajectories. CenterTrack uses CenterNet as a detector, inputs two adjacent frames of images and the previous frame heat map to predict the center point position, target box size and relative offset of the center point of the target in the next frame, obtains the predicted position of the previous frame through the offset, and associates it with the real detection box. FairMOT also uses CenterNet as a detection network and adds a ReID branch. The author believes that the ReID branch can extract more discriminative pedestrian features and can solve the target re-identification problem to a certain extent.

[0004] In addition, with the application of attention mechanisms such as TransFormer in computer vision, MOT methods based on attention mechanisms have begun to emerge, such as TransTrack and TrackFormer. These methods usually transfer the detection results of the previous frame to the current frame as the tracking query, use the self-attention mechanism of TransFormer to handle the target association problem, and use the query to realize the information transfer between video frames.

[0005] MOT tasks in remote sensing videos face unique challenges, such as long imaging distance, small target size, and complex background. SFMFMOT fully considers the problems of low-confidence bounding boxes, missed detections, false detections, and ID switching by exploring the characteristics of low-speed motion. TGraM models MOT as graph reasoning and designs a multi-task learning framework. However, the above methods still have shortcomings such as inaccurate positioning, false detections, and many missed detections for small targets commonly seen in remote sensing. Summary of the invention

[0006] The purpose of the present invention is to provide a triple attention method for multi-target tracking in satellite videos, so as to solve the problem that the existing multi-target tracking algorithms cannot accurately locate and track small moving targets in satellite videos.

[0007] To achieve the above object, the present invention provides a triple attention method for satellite video multi-target tracking, comprising the following steps:

[0008] Step 1: The satellite video frame sequence of fixed length is respectively subjected to the channel attention branch, the spatial attention branch and the temporal attention branch to obtain three independent feature maps, which are then fused to obtain an enhanced feature map;

[0009] Step 2: Input the enhanced feature map into the backbone network, and obtain a feature map containing deep semantic information after several convolutional layers and pooling layers;

[0010] Step 3: Input the feature map obtained in step 2 into the head network for detection and tracking prediction respectively;

[0011] Step 4: Optimize the overall network parameters of the tracking network, replace the L1 loss in the loss calculation with the Smooth L1 loss, and use the stochastic gradient descent method to optimize the network parameters.

[0012] Preferably, the current frame image (C, H, W represent the channel dimension and height and width of the input image respectively) Input the channel attention branch to obtain the channel attention map, including the following steps:

[0013] S11. Input the feature map X into the global maximum pooling and global average pooling modules to obtain the feature map F a , F m ,

[0014] F a =GAP(X),F m =GMP(X);

[0015] in, GAP and GMP represent global average pooling and global maximum pooling operations, respectively;

[0016] S12, F a and F m After concatenation, it is sent to the Conv layer, and the reduced dimension feature map F is obtained through the BN layer and the ReLU activation layer. c ,

[0017]

[0018] in, The number of neurons in the Conv layer is C / r, where r is the dimension reduction coefficient and is usually 4. BN represents the normalization operation, δ represents the ReLU activation function, and : represents the feature map concatenation.

[0019] S13, restore the feature map dimension through the Conv layer, and use the Sigmoid activation function for mapping to obtain the channel attention weight matrix M c , multiplied by the original image to obtain the channel attention map Y c ,

[0020] M c =σ(Conv(F c ));

[0021] Y c =X·M c T ;

[0022] in, σ represents the Sigmoid activation function, and the number of neurons in the Conv layer is C.

[0023] Preferably, the spatial attention map comprises the following steps:

[0024] S21, the image Input the horizontal pooling convolution layer and the vertical pooling convolution layer respectively to obtain the horizontal and vertical features Z h , Z w ,

[0025] Z h =Conv1(X),Z w = Conv2(X);

[0026] in, The convolution kernel sizes of Conv1 and Conv2 are (H, 1) and (1, W) respectively;

[0027] S22, the output feature maps in the horizontal and vertical directions are sequentially input into the convolution layer, normalization layer and nonlinear activation layer for feature conversion to obtain intermediate features containing horizontal and vertical spatial information t is the reduction factor;

[0028] S23, use two convolutional layers and Sigmoid function to normalize the features, and get G h and G w ,

[0029] F * =δ(BN(Conv(Z * )));

[0030] G * =σ(Conv(F * ));

[0031] in, and *h and w can be selected to represent the vertical and horizontal dimensions respectively, Conv represents the convolution layer, δ represents the ReLU activation function, σ represents the Sigmoid activation function, and BN represents the normalization layer;

[0032] S24, the output G h and Gw The weights are multiplied by the original image to get the spatial attention map

[0033] Y s =X·G h ·G w .

[0034] Preferably, the temporal attention map comprises the following steps:

[0035] S31, using 3D convolution to aggregate feature information to obtain feature map D′,

[0036] D′=Conv 3d (D);

[0037] in, Conv 3d Represents a 3D convolution operation;

[0038] S32, use global average pooling and Softmax activation to learn an attention weight W for each channel t :

[0039]

[0040] Among them, GAP 3d Represents a 3D global average pooling operation;

[0041] S33. Use W t Weight the feature map D' and obtain the time attention map through convolution operation

[0042]

[0043] Preferably, in step 1, the channel attention map Y c , spatial attention map Y s , temporal attention map Y t It is concatenated with the original feature map X, and then subjected to 1x1 convolution to obtain the fused feature map

[0044] Z=Conv 1×1 (Y c :Y s :Y t :X).

[0045] Therefore, the present invention adopts the above-mentioned triple attention method applied to satellite video multi-target tracking, which has the following beneficial effects:

[0046] (1) Channel attention and spatial attention are used for the current frame image to enhance the feature expression ability of the network from the spatial and channel aspects respectively, and the importance of each channel and spatial position in the feature map can be adaptively recalibrated.

[0047] (2) 3D convolution is used to aggregate the image features of adjacent frames into the current frame, adding time dimension information, which helps the tracking prediction branch to obtain more useful information.

[0048] (3) The feature maps output by the mixed channel, spatial, and temporal attention branches enrich the feature information, thereby improving the model’s representation ability and decision accuracy.

[0049] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of the present invention;

[0051] Figure 2 It is the overall network framework diagram of the present invention;

[0052] Figure 3 This is the network structure of the triple attention technology applied to remote sensing multi-target tracking proposed in this invention. DETAILED DESCRIPTION

[0053] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0054] Unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0055] Example

[0056] See also Figure 1-3The present invention provides a triple attention method for multi-target tracking in satellite videos. By introducing the triple attention module, the feature expression ability of the network is enhanced from the three dimensions of channel, space and time. On the one hand, the importance of each channel and spatial position in the feature map is adaptively recalibrated through channel attention and spatial attention, thereby achieving the suppression of useless background and the enhancement of target features. On the other hand, temporal attention introduces more temporal information, which is helpful for tracking prediction. In addition, the Smooth L1 loss is introduced to improve the stability of model training, and finally the accurate positioning and tracking of small moving targets in satellite videos is achieved.

[0057] A triple attention method for multi-target tracking in satellite video includes the following steps:

[0058] Step 1: Input the satellite video frame sequence of fixed length into the triple attention module, and obtain three independent feature maps through the channel attention branch, spatial attention branch and temporal attention branch respectively, and then weighted fusion is performed to obtain the enhanced feature map.

[0059] The specific process is as follows:

[0060] Channel attention: the current frame image (C, H, W represent the channel dimension, height and width of the input image respectively) Input the channel attention branch to obtain the channel attention map.

[0061] First, the feature map X is input into the global maximum pooling and global average pooling modules to obtain the feature map

[0062] F a =GAP(X),F m =GMP(X);

[0063] Among them, GAP and GMP represent global average pooling and global maximum pooling operations respectively;

[0064] Then, F a and F m After concatenation, it is sent to the Conv layer, and then passes through the BN layer and the ReLU activation layer to obtain the dimensionality reduction feature map

[0065] F c =δ(BN(Conv(F a :F m )))

[0066] Among them, the number of neurons in the Conv layer is C / r, r is the dimension reduction coefficient and is usually 4, BN represents the normalization operation, δ represents the ReLU activation function, and : represents the feature map concatenation;

[0067] Finally, the feature map dimension is restored through the Conv layer, and the Sigmoid activation function is used for mapping to obtain the channel attention weight matrix M c , multiplied by the original image to obtain the channel attention map

[0068] M c =σ(Conv(F c ));

[0069] Y c =X·M c T ;

[0070] Among them, σ represents the Sigmoid activation function, and the number of neurons in the Conv layer is C.

[0071] Spatial attention: Input the current frame image X into the spatial attention branch to obtain the spatial attention map.

[0072] First, the image Input the horizontal pooling convolution layer and the vertical pooling convolution layer respectively to obtain horizontal and vertical features.

[0073] Z h =Conv1(X),Z w = Conv2(X);

[0074] Among them, the convolution kernel sizes of Conv1 and Conv2 are (H, 1) and (1, W) respectively.

[0075] Then, the output feature maps in the horizontal and vertical directions are operated respectively, and the feature conversion is performed in the convolution layer, normalization layer and nonlinear activation layer in turn to obtain the intermediate features containing horizontal and vertical spatial information. and t is the reduction factor. Then, two convolutional layers and Sigmoid function are used to normalize the features, and we get and

[0076] F * =δ(BN(Conv(Z * )));

[0077] G * =σ(Conv(F * ));

[0078] Among them, * can be h and w, representing the vertical and horizontal dimensions respectively, Conv represents the convolutional layer, δ represents the ReLU activation function, σ represents the Sigmoid activation function, and BN represents the normalization layer.

[0079] Finally, the output G h and G w Merge into a weight matrix and multiply it with the original image to get the spatial attention map

[0080] Y s =X·G h ·G w ;

[0081] Temporal attention: The video frame sequence D = [X i-2 ,X i-1 ,X i ,X i+1 ,X i+2 ],in Input the temporal attention branch to get the temporal attention map. Specifically, for Where N = 5 represents the length of the video frame sequence. First, 3D convolution is used to aggregate feature information to obtain the feature map.

[0082] D′=Conv 3d (D);

[0083] Among them, Conv 3d Represents a 3D convolution operation.

[0084] Then, similar to channel attention, global average pooling and Softmax activation are used to learn an attention weight W for each channel. t ,

[0085] W t =σ(GAP 3d (D′));

[0086] Among them, GAP 3d Represents a 3D global average pooling operation.

[0087] Finally, use W t Weight the feature map D′ and obtain the temporal attention map through convolution operation

[0088] Y t =Conv(W t *D′);

[0089] Feature fusion: channel attention map Y c , spatial attention map Y s , temporal attention map Y t It is concatenated with the original feature map X, and then subjected to 1x1 convolution to obtain the fused feature map

[0090] Z=Conv 1×1 (Y c :Y s :Y t :X);

[0091] Step 2: Input the feature map after triple attention fusion into the backbone network, and obtain the feature map containing deep semantic information through several convolutional layers and pooling layers;

[0092] Step 3: Input the feature map obtained in the second step into the head network for detection and tracking prediction respectively;

[0093] Step 4: Optimize the overall network parameters of the tracking network. In order to improve the stability of model training, replace the L1 loss in the loss calculation with the Smooth L1 loss, and use the stochastic gradient descent method to optimize the network parameters. Specifically:

[0094] For a single input image (W represents the width of the image, H represents the height of the image, and 3 represents the three RGB color channels of the image) The overall loss function is as follows:

[0095] L=L k +L size +L loc +L off

[0096]

[0097]

[0098]

[0099]

[0100] Among them, L k represents focal loss, (C represents the total number of target categories, R is a downsampling factor with a value of 4) is the true heat map corresponding to the labeled object, represents the heat map at the (x, y) position of the cth category, N represents the total number of objects, α and β are the hyperparameters of the loss; L size The regression method is used to represent the size loss of the prediction box, for the i The i-th object, s i and Respectively represent the target box and prediction box size; L loc The position loss is further regressed to the refined central local position using the Smooth L1 loss. and p iRepresents the center point positions of the predicted box and the real box respectively; finally L off Try to minimize the displacement of the current frame t prediction relative to the previous frame The real displacement (p i t-1 -p i t ), p i t-1 and p i t Represent the center point positions of the true target box i in the t-1th frame and the tth frame respectively.

[0101] The present invention also provides a system for triple attention applied to satellite video multi-target tracking, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, a triple attention method applied to remote sensing multi-target tracking is implemented.

[0102] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the triple attention method applied to remote sensing multi-target tracking is implemented.

[0103] Therefore, the present invention adopts the above-mentioned triple attention method for multi-target tracking in satellite video, enhances the feature expression ability of the model through channel attention, spatial attention and temporal attention, and replaces L1 loss with Smooth L1, improves training stability and accelerates convergence, and finally realizes accurate detection and tracking of small-sized moving targets in remote sensing images with complex backgrounds. The system can effectively deal with challenges in small target detection, such as small target size, few features, and complex background, and shows high-accuracy performance in practical applications.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A triple attention method for multi-target tracking in satellite video, characterized in that: The following steps are involved: Step 1: The satellite video frame sequence of fixed length is respectively subjected to the channel attention branch, the spatial attention branch and the temporal attention branch to obtain three independent feature maps, which are then fused to obtain an enhanced feature map; Step 2: Input the enhanced feature map into the backbone network, and obtain a feature map containing deep semantic information after several convolutional layers and pooling layers; Step 3: Input the feature map obtained in step 2 into the head network for detection and tracking prediction respectively; Step 4: Optimize the overall network parameters of the tracking network, replace the L1 loss in the loss calculation with the Smooth L1 loss, and use the stochastic gradient descent method to optimize the network parameters.

2. The triple attention method for satellite video multi-target tracking according to claim 1, characterized in that: The current frame image C, H, and W represent the channel dimension, height, and width of the input image, respectively. The input channel attention branch obtains the channel attention map, including the following steps: S11. Input the feature map X into the global maximum pooling and global average pooling modules to obtain the feature map F a , F m , F a =GAP(X),F m =GMP(X); in, GAP and GMP represent global average pooling and global maximum pooling operations, respectively; S12, F a and F m After concatenation, it is sent to the Conv layer, and the reduced dimension feature map F is obtained through the BN layer and the ReLU activation layer. c , F c =δ(BN(Conv(F a :F m ))); in, The number of neurons in the Conv layer is C / r, where r is the dimension reduction coefficient and is usually 4. BN represents the normalization operation, δ represents the ReLU activation function, and : represents the feature map concatenation. S13, restore the feature map dimension through the Conv layer, and use the Sigmoid activation function for mapping to obtain the channel attention weight matrix M c , multiplied by the original image to obtain the channel attention map Y c , M c =σ(Conv(F c )); Y c =X·M c T ; in, σ represents the Sigmoid activation function, and the number of neurons in the Conv layer is C.

3. The triple attention method for satellite video multi-target tracking according to claim 2, characterized in that: The spatial attention map consists of the following steps: S21, the image Input the horizontal pooling convolution layer and the vertical pooling convolution layer respectively to obtain the horizontal and vertical features Z h , Z w , Z h =Conv1(X),Z w =Conv2(X); in, The convolution kernel sizes of Conv1 and Conv2 are (H, 1) and (1, W) respectively; S22, the output feature maps in the horizontal and vertical directions are sequentially input into the convolution layer, normalization layer and nonlinear activation layer for feature conversion to obtain intermediate features containing horizontal and vertical spatial information and t is the reduction factor; S23, use two convolutional layers and Sigmoid function to normalize the features, and get G h and G w , F * =δ(BN(Conv(Z * ))); G * =σ(Conv(F * )); in, and *h and w can be selected to represent the vertical and horizontal dimensions respectively, Conv represents the convolution layer, δ represents the ReLU activation function, σ represents the Sigmoid activation function, and BN represents the normalization layer; S24, the output G h and G w The weights are multiplied by the original image to get the spatial attention map Y s =X·G h ·G w 。 4. The triple attention method for satellite video multi-target tracking according to claim 3, characterized in that: The temporal attention graph consists of the following steps: S31, using 3D convolution to aggregate feature information to obtain feature map D′, D′=Conv 3d (D); in, Conv 3d Represents a 3D convolution operation; S32, use global average pooling and Softmax activation to learn an attention weight W for each channel t : W t =σ(GAP 3d (D′)); Among them, GAP 3d Represents a 3D global average pooling operation; S33. Use W t Weight the feature map D′ and obtain the temporal attention map through convolution operation Y t =Conv(W t *D′)。 5. The triple attention method for satellite video multi-target tracking according to claim 4, characterized in that: In step 1, the channel attention map Y c , spatial attention map Y s , temporal attention map Y t It is concatenated with the original feature map X, and then subjected to 1x1 convolution to obtain the fused feature map Z=Conv 1×1 (AND c :AND s :AND t :X)。

Images