Remote sensing image target tracking method based on multi-scale feature aggregation enhancement

Abstract

The present invention discloses a remote sensing image target tracking method based on multi-scale feature aggregation enhancement, the method comprising the following steps: step 1: using a feature extraction network based on an encoding-decoding structure to output a high-resolution feature map, the feature map downsampling level is 4 times that of the original image; step 2: using a multi-scale feature aggregation module to regularize and aggregate the multi-scale features in the high-resolution feature map in step 1, and output the regularized feature map; step 3: for the regularized feature map in step 2, the dependency between the spatial and channel dimensions is captured from three branches by a cross-domain attention enhancement module to enhance the feature map; step 4: predicting target position, size, offset and other information based on the enhanced feature map, and outputting the target trajectory through a matching algorithm. The method can regularize and enhance effective features at different scales, reduce false alarm source interference, and achieve target tracking in complex scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection and recognition, and relates to a tracking method for remote sensing image targets, and specifically to a remote sensing image target tracking method based on multi-scale feature aggregation enhancement. Background Art

[0002] As a basic task in computer vision, target tracking can continuously track the area of ​​interest in time and space by analyzing the position, shape, location and other information of the target in the video sequence. With the development of remote sensing imaging technology, the form of remote sensing data has gradually expanded from single-frame images to time-series videos. Compared with surveillance camera data, optical remote sensing video data has a large field of view and a wide range, which can achieve continuous monitoring of wide areas. In the civilian field, target tracking through optical remote sensing image data can monitor urban traffic flow, vehicle operation status and other information in real time, thereby providing support for urban planning and traffic management; in the military field, it can monitor the dynamics of enemy military equipment and provide intelligence support and decision-making basis.

[0003] Early tracking methods, such as correlation filtering, were mainly based on manually designed feature extraction algorithms, using color, texture, and shape information to achieve target tracking through feature matching in continuous frames. However, the manually designed features have limited representation of target features and are difficult to adapt to complex application scenarios. Deep learning-based tracking methods mostly adopt a detection-first-tracking framework, such as using target detection networks such as YOLOv5 to output the target position of a single frame, and then using re-identification methods such as DeepSORT to achieve inter-frame target matching. This type of cascaded detection and tracking method does not share the features extracted by the detection network and the re-identification network, has computational redundancy, and is difficult to meet real-time requirements. The CenterTrack network derived from the CenterNet network can simultaneously output the current frame target position and predicted trajectory, integrate the target detection and re-identification processes, and achieve a balance between the accuracy and real-time performance of target tracking.

[0004] However, most sea and land targets in remote sensing images have scale and direction variability. For example, ground vehicle targets have small geometric scales and weak texture contour characteristics, while sea ship targets have large geometric scales and relatively richer detail characteristics, and both have random direction characteristics. In addition, from a bird's-eye view, there are usually a large number of false alarm sources in remote sensing image scenes that are similar to the target's geometric contours and texture detail characteristics, which can easily cause positioning interference to target detection and tracking. In view of the above situation, it is urgent to design a targeted network structure to achieve regular enhancement of features of different scales around the difficulties in target positioning and tracking in remote sensing scenes, so as to improve target tracking accuracy. Summary of the invention

[0005] In view of the difficulties such as the variable scale and direction of the target in the remote sensing image and the interference of the false alarm source in the complex scene, the present invention provides a remote sensing image target tracking method based on multi-scale feature aggregation enhancement. The method can regularly enhance the effective features at different scales, reduce the interference of the false alarm source, and realize the target tracking in the complex scene.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] A remote sensing image target tracking method based on multi-scale feature aggregation enhancement includes the following steps:

[0008] Step 1: Use a feature extraction network based on an encoding-decoding structure to output a high-resolution feature map, and the feature map downsampling level is 4 times that of the original image;

[0009] Step 2: Use the multi-scale feature aggregation module to regularize and aggregate the multi-scale features in the high-resolution feature map in step 1, and output the regularized feature map;

[0010] Step 3: For the feature map regularized in step 2, the cross-domain attention enhancement module is used to capture the dependencies between the spatial and channel dimensions from three branches to enhance the feature map;

[0011] Step 4: Predict the target position, size, offset and other information based on the enhanced feature map, and output the target trajectory through the matching algorithm.

[0012] Compared with the prior art, the present invention has the following advantages:

[0013] (1) The present invention proposes a tracking method for remote sensing image targets, which can effectively solve the problems of weak network feature extraction capability and poor tracking performance caused by the variability of target scale and direction and interference from false alarm sources in complex scenes, and can achieve accurate and stable tracking of multiple targets under the interference of complex objects.

[0014] (2) The present invention proposes a multi-scale feature aggregation module that adaptively adjusts the receptive field. The module learns position offsets through deformable convolutions of different sizes, dynamically adjusts the receptive fields at different scales, and improves the network's feature extraction capability for remote sensing targets with varying scales and directions.

[0015] (3) The present invention proposes a cross-domain attention enhancement module that captures multi-dimensional dependencies. The module captures the dependencies between the channel dimension and the spatial dimension through three branches, generates feature enhancement weights, improves the network's perception of the target, and suppresses the interference of complex terrain scenes on effective features. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flowchart of remote sensing image target tracking based on multi-scale feature aggregation enhancement;

[0017] Figure 2 Schematic diagram of the structure of the multi-scale feature aggregation module and the cross-domain attention enhancement module;

[0018] Figure 3 The target tracking result for remote sensing images is 1;

[0019] Figure 4 The second result is the target tracking result for remote sensing images. DETAILED DESCRIPTION

[0020] The technical solution of the present invention is further described below in conjunction with the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be included in the protection scope of the present invention.

[0021] The present invention provides a remote sensing image target tracking method based on multi-scale feature aggregation enhancement, such as Figure 1 As shown, the method comprises the following steps:

[0022] Step 1: Use a feature extraction network based on an encoder-decoder structure to output a high-resolution feature map, with the feature map downsampling level being 4 times that of the original image. The specific steps are as follows:

[0023] The current frame, previous frame and previous frame heat map of the remote sensing image sequence are input. The feature extraction network based on the encoding-decoding structure realizes backbone feature extraction through downsampling, upsampling, lateral connection and other methods to generate a high-resolution feature map with a downsampling level of 4 times.

[0024] In this step, the feature extraction network based on the encoding-decoding structure that can be selected includes but is not limited to DLA-34, Hourglass-104 and other networks.

[0025] Step 2: Use the multi-scale feature aggregation module to regularize and aggregate the multi-scale features in the high-resolution feature map in step 1, and output the regularized feature map. The multi-scale feature aggregation module realizes multi-scale feature extraction by cascading deformable convolutions of different sizes, and combines 1×1 convolution to realize the aggregation of extracted features. The specific steps are as follows:

[0026] Step 2-1: Use deformable convolutions of different sizes to perform convolution operations on the backbone feature F1 output in step 1. Deformable convolution can learn position offsets and dynamically adjust the receptive field by mining training data. In order to achieve adaptive extraction of multi-scale features, deformable convolutions with convolution kernel sizes of 3×3, 6×6, and 9×9 are used to extract features at different scales, and 1×1 convolution is used to retain the information at the original resolution.

[0027] Step 2-2: Use channel concatenation and 1×1 convolution to achieve clustering and regularization of multi-scale features. The specific steps are as follows:

[0028] For the extracted multi-scale features, feature aggregation is performed through channel splicing, and then regularized through 1×1 convolution, so that the output feature F2 restores the same number of channels as the input feature map F1. The formula is written as:

[0029] F2=conv 1×1 (concat(DC 3×3 (F1),DC 6×6 (F1),DC 9×9 (F1),conv 1×1 (F1)))

[0030] Where, DC n×n (·) represents a deformable convolution with a kernel size of n×n, concat(·) is a channel concatenation operation, conv 1×1 (·) is a 1×1 convolution.

[0031] Step 3: For the feature map regularized in step 2, the cross-domain attention enhancement module captures the dependencies between spatial and channel dimensions from three branches to enhance the feature map. Cross-domain attention enhancement aggregates cross-domain information along different dimensions through pooling operations to generate a saliency map, and then obtains pixel-by-pixel weights through a broadcast mechanism and a nonlinear activation function, which are finally applied to the feature map to achieve feature enhancement. The specific steps are as follows:

[0032] Step 3-1: Capture the interdependencies along the channel dimension. Assuming that the length, width, and number of channels of the input regularized feature map F2 are recorded as (h, w, c), the first branch extracts the pixel dependencies along the channel dimension c through maximum pooling and average pooling, and uses the splicing operation to obtain an aggregated feature map of dimension (2, h, w) Then adjust it through a 7×7 convolution Capture local spatial dimension information and adjust the number of channels to obtain a saliency map of the spatial dimension The calculation formula is:

[0033] S C =conv 7×7 [MaxPool(F2),AvePool(F2)]

[0034] Where MaxPool(·) and AvePool(·) represent maximum pooling and spatial pooling respectively, conv 7×7 (·) is a 7×7 convolution.

[0035] Step 3-2: The second and third branches capture the interdependencies along the length and width dimensions respectively. The second branch first transposes the feature map along the h direction, extracts the pixel dependencies along the spatial dimension h through maximum pooling and average pooling, and then uses concatenation, 7×7 convolution, and transposition along the h direction to obtain the cross-dimensional saliency map. The third branch transposes the feature map along the w direction, extracts the pixel dependencies along the spatial dimension w through maximum pooling and average pooling, and obtains the cross-dimensional saliency map by sequentially using splicing operations, 7×7 convolutions, and transposition along the w direction.

[0036] Step 3-3: Use the broadcast mechanism to add the cross-dimensional saliency maps in different directions, and then use the Sigmoid nonlinear activation function to obtain the pixel-by-pixel weights to generate the attention map A. Finally, use the attention map A to weight the input feature map to obtain the enhanced feature map F3. The calculation formula is:

[0037] F3=F2×A=F2×Sigmoid(S H +S W +S C )

[0038] Where Sigmoid(·) is the Sigmoid nonlinear activation function.

[0039] Step 4: Predict the target position, size, offset and other information based on the enhanced feature map, and output the target trajectory through the matching algorithm. The specific steps are:

[0040] Step 4-1: According to the original CenterTrack structure, multiple 1×1 convolutions are used to output the target heat map, position prediction, length and width prediction, and motion offset.

[0041] Step 4-2: Perform target association tracking by combining the associated target trajectory and the output result of the current frame through any matching algorithm.

[0042] Here we take the greedy matching algorithm as an example to implement this step. If the target trajectory of the previous frame is recorded as Where P and O are two-dimensional vectors, representing the center point position and motion offset respectively, ID is the target number, M is the total number of trajectories associated with the previous frame, and t is the current frame. The current frame prediction point is recorded as The order is sorted according to the peak value in the heat map, where and Represent the predicted center point position and predicted motion offset respectively, and N is the number of predicted targets in the current frame. Then calculate the cost matrix The element C in the i-th row and j-th column of the matrix ij The calculation method is the target position vector in the jth associated trajectory of the previous frame The i-th target position predicted by the current frame Difference from offset The second norm of is written as:

[0043]

[0044] For all targets detected in the current frame, find the minimum C ij The corresponding unmatched track j in this case, and the threshold k is calculated at the same time. The threshold calculation method is to predict the length and width of the target The geometric mean of the target length and width (w j ,h j ). Write it as:

[0045]

[0046] If j corresponds to C ij <k, then match the predicted point i with the trajectory j. Otherwise, if all C ij >k will generate a new trajectory.

[0047] Figure 3 and Figure 4 The tracking results of vehicle targets in remote sensing time series images are given respectively, and the complex scenes shown include many false alarm sources. It can be seen from the figure that the method can accurately track all targets without generating false alarms.

Claims

1. A remote sensing image target tracking method based on multi-scale feature aggregation enhancement, characterized in that The method comprises the following steps: Step 1: Use a feature extraction network based on an encoding-decoding structure to output a high-resolution feature map, and the feature map downsampling level is 4 times that of the original image; Step 2: Use the multi-scale feature aggregation module to regularize and aggregate the multi-scale features in the high-resolution feature map in step 1, and output the regularized feature map. The specific steps are as follows: Step 2-1: Select deformable convolutions of different sizes to perform convolution operations on the backbone feature F1 output in step 1. The deformable convolution learns position offsets and dynamically adjusts the receptive field by mining the training data; Step 2-2: Aggregation and regularization of multi-scale features are achieved through channel splicing operations and 1×1 convolution; Step 3: For the feature map regularized in step 2, the cross-domain attention enhancement module is used to capture the dependencies between the spatial and channel dimensions from three branches to enhance the feature map. The specific steps are as follows: Step 3-1: Capture the interdependencies along the channel dimension. Assuming that the length, width, and number of channels of the input regularized feature map F2 are recorded as (h, w, c), the first branch extracts the pixel dependencies along the channel dimension c through maximum pooling and average pooling, and uses the splicing operation to obtain an aggregated feature map of dimension (2, h, w). Then adjust it through a 7×7 convolution Capture local spatial dimension information and adjust the number of channels to obtain a saliency map of the spatial dimension The calculation formula is: S C =conv 7×7 [MaxPool(F2),AvePool(F2)] Where MaxPool(·) and AvePool(·) represent maximum pooling and spatial pooling respectively, conv 7×7 (·) is a 7×7 convolution; Step 3-2: The second and third branches capture the interdependencies along the length and width dimensions respectively. The second branch first transposes the feature map along the h direction, extracts the pixel dependencies along the spatial dimension h through maximum pooling and average pooling, and then uses concatenation, 7×7 convolution, and transposition along the h direction to obtain the cross-dimensional saliency map. The third branch transposes the feature map along the w direction, extracts the pixel dependencies along the spatial dimension w through maximum pooling and average pooling, and obtains the cross-dimensional saliency map by sequentially using splicing operations, 7×7 convolutions, and transposition along the w direction. Step 3-3: Use the broadcast mechanism to add the cross-dimensional saliency maps in different directions, and then use the Sigmoid nonlinear activation function to obtain the pixel-by-pixel weights to generate the attention map A. Finally, use the attention map A to weight the input feature map to obtain the enhanced feature map F3. The calculation formula is: F3=F2×A=F2×Sigmoid(S H +S W +S C ) Where Sigmoid(·) is the Sigmoid nonlinear activation function; Step 4: Predict the target position, size, and offset information based on the enhanced feature map, and output the target trajectory through a matching algorithm.

2. The remote sensing image target tracking method based on multi-scale feature aggregation enhancement according to claim 1 is characterized in that The specific steps of step 1 are as follows: The current frame, previous frame and previous frame heat map of the remote sensing image sequence are input. The feature extraction network based on the encoding-decoding structure realizes backbone feature extraction through downsampling, upsampling and lateral connection to generate a high-resolution feature map with a downsampling level of 4 times.

3. The remote sensing image target tracking method based on multi-scale feature aggregation enhancement according to claim 1 is characterized in that In step 2-1, in order to achieve adaptive extraction of multi-scale features, deformable convolutions with convolution kernel sizes of 3×3, 6×6, and 9×9 are selected to extract features at different scales, and 1×1 convolution is used to retain the information at the original resolution.

4. The remote sensing image target tracking method based on multi-scale feature aggregation enhancement according to claim 1 is characterized in that The specific steps of step 2-2 are as follows: For the extracted multi-scale features, feature aggregation is performed through channel splicing, and then regularized through 1×1 convolution, so that the output feature F2 restores the same number of channels as the input feature map F1. The formula is written as: F2=conv 1×1 (concat(DC 3×3 (F1),DC 6×6 (F1),DC 9×9 (F1),conv 1×1 (F1))) Where, DC n×n (·) represents a deformable convolution with a kernel size of n×n, concat(·) is a channel concatenation operation, conv 1×1 (·) is a 1×1 convolution.

5. The remote sensing image target tracking method based on multi-scale feature aggregation enhancement according to claim 1 is characterized in that The specific steps of step 4 are: Step 4-1: According to the original CenterTrack structure, multiple 1×1 convolutions are used to output the target heat map, position prediction, length and width prediction, and motion offset; Step 4-2: Perform target association tracking by combining the associated target trajectory and the output result of the current frame through any matching algorithm.

6. The remote sensing image target tracking method based on multi-scale feature aggregation enhancement according to claim 5 is characterized in that In step 4-2, the matching algorithm adopts a greedy matching algorithm, and the specific steps are as follows: if the target trajectory of the previous frame is recorded as Where P and O are two-dimensional vectors, representing the center point position and motion offset respectively, ID is the target number, M is the total number of trajectories associated with the previous frame, and t is the current frame; the prediction point of the current frame is recorded as The order is sorted according to the peak value in the heat map, where and Represent the predicted center point position and predicted motion offset respectively, N is the number of predicted targets in the current frame; then calculate the cost matrix The element C in the i-th row and j-th column of the matrix ij The calculation method is the target position vector in the jth associated trajectory of the previous frame The i-th target position predicted by the current frame Difference from offset The second norm of is written as: For all targets detected in the current frame, find the minimum C ij The corresponding unmatched track j in this case, and the threshold k is calculated at the same time. The threshold calculation method is to predict the length and width of the target The geometric mean of the target length and width (w j ,h j )’s minimum geometric mean; writing: If j corresponds to C ij <k, then match the predicted point i with the trajectory j. Otherwise, if all C ij >k will generate a new trajectory.

Images