Unmanned aerial vehicle target tracking method and system based on cooperative attention and series channel cross-correlation
Through the method of synergistic attention and series channel intercorrelation, the problems of scale changes and occlusion and background interference of drone target tracking are solved, and more efficient target positioning and tracking performance are achieved.
Patent Information
- Application Number
- CN202410944594.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-15
AI Technical Summary
There are problems in drone target tracking with inaccurate target scale changes and occlusion, as well as feature mismatch caused by background interference and similar target interference, which affects tracking performance.
Using a method based on collaborative attention and series channel intercorrelation, templates and search features are extracted through twin networks, feature interaction is enhanced by collaborative attention modules, and more obvious features are learned through series channel intercorrelation modules, combined with anchorless tracking head network Perform target positioning.
Improves the performance and accuracy of target tracking, better identifying targets and maintaining tracking in complex scenarios, reducing the impact of background interference and similar target interference.
Smart Images

Figure CN118941985A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a method and system for tracking unmanned aerial vehicle targets based on collaborative attention and serial channel cross-correlation. Background Art
[0002] Target tracking is an important research direction in the field of computer vision. With the development of information technology, target tracking has received more and more attention and application in human-computer interaction, autonomous driving, national defense security and video surveillance. In recent years, drones have been widely used in civil and military fields due to their small size, flexibility, intelligence and autonomy. For example, they are used in atmospheric environment detection, disaster relief, enemy reconnaissance, enemy target tracking, battlefield intelligence search, etc. In particular, the most important task in the search and reconnaissance process is target tracking. Therefore, target tracking algorithms based on deep learning are increasingly used in drones. The complexity of drone visual target tracking scenes and the variability of moving targets make it difficult to extract target features and establish target models, which poses a huge challenge to target tracking performance.
[0003] Due to the rich application scenarios of target tracking technology in drones, it has become a hot research topic at home and abroad in recent years. Many excellent algorithms have emerged, and many challenging data sets have appeared. The difficulty of the data sets is gradually increasing, which also puts forward new requirements for the improvement and innovation of target tracking algorithms to cope with target tracking problems in complex and changing scenarios.
[0004] Target tracking algorithms are divided into two categories: generative target tracking algorithms and discriminative target tracking algorithms. The first category, such as particle filtering, mean shift, and Kalman filtering, all belong to the generative target tracking algorithm; the second category is the discriminative target tracking algorithm, which adds a detection link to the target tracking algorithm and pays more attention to the distinction between the tracked object and the background. A detector is obtained through offline pre-training or online learning. This detector is used to complete the task of distinguishing the background from the foreground target, and then the position of the tracked object is obtained. Therefore, it is more robust than the generative target tracking algorithm in terms of effect. In the discriminative target tracking algorithm, the research on single target tracking algorithm based on correlation filtering and deep learning is the main trend at present. Summary of the invention
[0005] The purpose of the present invention is to provide a UAV target tracking method and system based on collaborative attention and serial channel cross-correlation with strong tracking performance and high positioning accuracy. On the one hand, it can solve the problem that the target cannot be accurately located when the target changes in scale or is occluded. On the other hand, it can solve the problem of feature mismatch when the tracked target is interfered by background or similar targets.
[0006] The technical solution to achieve the purpose of the present invention is: a method for tracking a drone target based on collaborative attention and serial channel cross-correlation, comprising the following steps:
[0007] Step 1: Crop the first frame of the video to obtain the input of the template branch, and crop the subsequent frames to obtain the input of the search branch. The first frame and the subsequent frames are subjected to feature extraction through the twin network, and then feature interaction is performed through the collaborative attention module to obtain the template feature map and the search feature map.
[0008] Step 2: Input the template feature map and the search feature map obtained in step 1 into the series channel cross-correlation module for cross-correlation operation to obtain an output feature map;
[0009] Step 3: Input the output feature map obtained in step 2 into the anchor-free tracking head network, obtain the classification score of each pixel in the feature map through the classification branch, obtain the center score between each pixel and the target through the center branch, and obtain the target box information corresponding to each pixel through the regression branch;
[0010] Step 4: Multiply the classification score of each pixel in step 3 by the center score, select the pixel with the highest score, and determine the target frame information of the pixel through the regression branch, thereby determining the location of the target in the current frame;
[0011] Step 5: Repeat steps 1 to 4 until the position of the last frame in the video is tracked.
[0012] A UAV target tracking system based on collaborative attention and serial channel cross-correlation, the system is used to implement the UAV target tracking method based on collaborative attention and serial channel cross-correlation, the system includes a feature extraction module, a cross-correlation operation module, a tracking head network module, a target position determination module, and a tracking module, wherein:
[0013] The feature extraction module crops the first frame of the video to obtain the input of the template branch, crops the subsequent frame images to obtain the input of the search branch, extracts features from the first frame image and the subsequent frame images through the twin network, and then performs feature interaction through the collaborative attention module to obtain a template feature map and a search feature map;
[0014] A cross-correlation operation module inputs the template feature map and the search feature map obtained by the feature extraction module into the series channel cross-correlation module for cross-correlation operation to obtain an output feature map;
[0015] The tracking head network module inputs the output feature map obtained by the cross-correlation operation module into the anchor-free tracking head network, obtains the classification score of each pixel in the feature map through the classification branch, obtains the center score between each pixel and the target through the center branch, and obtains the target box information corresponding to each pixel through the regression branch;
[0016] The target position determination module multiplies the classification score of each pixel in the tracking head network module by the center score, selects the pixel with the highest score, and determines the target frame information of the pixel through the regression branch, thereby determining the position of the target in the current frame;
[0017] The tracking module repeats the processing of the feature extraction module, the cross-correlation operation module, the tracking head network module, and the target position determination module until the position of the last frame in the video is tracked.
[0018] A mobile terminal comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for tracking a target of an unmanned aerial vehicle based on collaborative attention and cross-correlation of serial channels is implemented.
[0019] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the method for tracking a target of an unmanned aerial vehicle based on collaborative attention and cross-correlation of serial channels.
[0020] Compared with the prior art, the present invention has the following significant advantages:
[0021] (1) The last three blocks of the pre-trained weight Resnet50 are used, and a collaborative attention module is added after each of the three blocks to deeply mine the intrinsic correlation between template features and search features, and emphasize the effectiveness of the interactive information between the two branches in strengthening the appearance expression of the model, so as to better identify the target;
[0022] (2) The cross-correlation operation is performed using the serial channel cross-correlation module to learn more obvious features, which effectively solves the problem of feature mismatch when the tracking target encounters background interference, similar target interference, etc., thereby improving the tracking performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 The flowchart of the UAV target tracking method based on collaborative attention and serial channel cross-correlation of the present invention.
[0024] Figure 2 This is a schematic diagram of the UAV target tracking method based on collaborative attention and serial channel cross-correlation of the present invention.
[0025] Figure 3 A schematic diagram of the process of feature interaction performed by the collaborative attention module in the present invention.
[0026] Figure 4 It is a schematic diagram of the process of cross-correlation of series channels in the present invention.
[0027] Figure 5 The figure is a comparison curve of the accuracy of the method of the present invention and the existing tracking algorithm in DTB70 in the embodiment.
[0028] Figure 6 The following is a comparison curve of the success rates of the method of the present invention and the existing tracking algorithm in DTB70 in the embodiment.
[0029] Figure 7 It is a schematic diagram of the tracking result in the embodiment. DETAILED DESCRIPTION
[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention is clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Based on the embodiments of the present invention, all other examples obtained by ordinary technicians in this field without creative premise work are within the scope of protection of the present invention.
[0031] The present invention provides a UAV target tracking method based on collaborative attention and serial channel cross-correlation, comprising the following steps:
[0032] Step 1: Crop the first frame of the video to obtain the input of the template branch, and crop the subsequent frames to obtain the input of the search branch. The first frame and the subsequent frames are subjected to feature extraction through the twin network, and then feature interaction is performed through the collaborative attention module to obtain the template feature map and the search feature map.
[0033] Step 2: Input the template feature map and the search feature map obtained in step 1 into the series channel cross-correlation module for cross-correlation operation to obtain an output feature map;
[0034] Step 3: Input the output feature map obtained in step 2 into the anchor-free tracking head network, obtain the classification score of each pixel in the feature map through the classification branch, obtain the center score between each pixel and the target through the center branch, and obtain the target box information corresponding to each pixel through the regression branch;
[0035] Step 4: Multiply the classification score of each pixel in step 3 by the center score, select the pixel with the highest score, and determine the target frame information of the pixel through the regression branch, thereby determining the location of the target in the current frame;
[0036] Step 5: Repeat steps 1 to 4 until the position of the last frame in the video is tracked.
[0037] As a specific example, the twin network in step 1 is a Resnet50 with shared weights. The last three Block modules of Resnet50 are used. The twin network is combined with the collaborative attention module, and a collaborative attention module is added after each Block module.
[0038] As a specific example, step 1 is as follows:
[0039] Step 1.1, crop the first frame of the video with the target as the center as the input of the template branch, and crop the subsequent frames as the input of the search branch;
[0040] The first frame of the picture is extracted through the twin network to obtain the initial template feature map Extract features from subsequent frames to obtain the initial search feature map
[0041] Where R represents the real number domain; N represents the number of samples processed at a time; W T Indicates the width of the template feature map; H T Represents the height of the template feature map; W I Indicates the width of the search feature map; H I Represents the width of the search feature map; C represents the number of channels of the feature map;
[0042] Step 1.2: The template branch is processed by a 1×1 convolution kernel k(●) and then the tensor is reshaped to obtain the first feature map The search branch is processed by a 1×1 convolution kernel g(●) and then the tensor is reshaped to obtain the second feature map The first feature map And the second feature map Perform matrix multiplication to obtain the intermediate feature map
[0043] Step 1.3: The template branch is processed by the 1×1 convolution kernel g(●) and then the tensor is reshaped to obtain the third feature map The search branch is processed by a 1×1 convolution kernel g(●) and then the tensor is reshaped to obtain the fourth feature map The third feature map obtained Fourth characteristic diagram Respectively with the intermediate feature map of step 1.2 Perform matrix multiplication operation to obtain the template branch modulation feature map and search branch modulation feature map
[0044] Step 1.4: Modulate the template branch feature map processed in step 1.3 Reshape the tensor to get the fifth feature map Searching for branch modulation signatures Reshape the tensor to get the sixth feature map
[0045] Fifth characteristic diagram Sixth characteristic diagram After passing through 1×1 convolution kernels q(●) and w(●), the channel dimension is increased to C, and the fifth feature map after the channel dimension is transformed is obtained. The sixth feature map after changing the channel dimension
[0046] Using the initial template feature map And the fifth feature map after changing the channel dimension Perform matrix addition operation to obtain the intrinsic template related feature map ψ(x; z);
[0047] Using the initial search feature map And the sixth eigenvector after transforming the channel dimension Perform matrix addition operation to obtain intrinsic search related features ψ(z; x);
[0048] Step 1.5: The intrinsic template-related features ψ(x; z) are compressed by two different operations: maximum pooling and global average pooling. Then, they are stimulated by the fully connected layer MLP. Then, the importance of each channel is normalized to a digital range of 0-1 using the Sigmoid operation. Finally, the channel parameters after Sigmoid, that is, the numbers normalized to 0-1, are multiplied by the intrinsic template-related features ψ(x; z) and the intrinsic search-related features ψ(z; x) to obtain the template feature map. and search feature graph
[0049] As a specific example, in step 2, the template feature map and the search feature map obtained in step 1 are input into the series channel cross-correlation module for cross-correlation operation to obtain an output feature map, which is as follows:
[0050] Step 2.1: Template feature map and search feature graph Perform a Conv+BN+ReLU operation respectively, and then perform the first cross-correlation operation. The formula is:
[0051]
[0052] Among them, Conv represents the convolution operation in deep learning, which filters the input data through the convolution kernel; BN represents the operation of normalizing the features; ReLU is the activation function; φ first Represents the feature map of the first cross-correlation; XCorr dw Indicates the similarity between calculated features;
[0053] Step 2.2: The first cross-correlation feature map φ obtained in step 2.1 first Perform a convolution Conv operation, and then perform two consecutive Conv+BN+ReLU operations for upsampling, with Padding set to 3 and 2 respectively, to obtain the processed features Padding refers to filling extra pixels at the edge of the input feature map in order to control the size of the output feature map during the convolution operation;
[0054] Step 2.3: The processed features obtained in step 2.2 The search feature map obtained in step 1.5 Set a weighted fusion strategy to obtain the fused search features The formula is:
[0055]
[0056] Among them, λ1 and λ2 are set to 0.8 and 0.2 respectively;
[0057] Step 2.4: The template feature map obtained in step 1.5 The template features are obtained through the ECA channel attention mechanism Then the template feature map With template features Add together to get the template features required for the second cross-correlation
[0058] Step 2.5: Perform the second cross-correlation operation, the formula is:
[0059]
[0060] φ second Feature map representing the second cross-correlation;
[0061] Step 2.6: The first cross-correlation feature map φ first And the second cross-correlation feature map φ second Add together to get the output feature map
[0062] As a specific example, the classification branch in step 3 is used to distinguish the target, and the regression branch is used to obtain the scale information of the target size;
[0063] The final response feature map obtained by the classification branch is A cls , A cls ∈R h×w×2 , response characteristic diagram A cls The size is h×w, 2 represents the response feature map A cls The number of channels, A cls Contains the probability of pixels being positive and negative samples;
[0064] The response characteristic graph finally obtained by the regression branch is A reg , A reg ∈R h×w×4 , response characteristic diagram A reg The size is h×w, 4 represents the number of channels of the feature map, A reg Contains the distance from each pixel (i, j) to each border, expressed as t (i,j) =(l,t,r,b), where l,t,r,b represents the distance from the pixel point (i,j) to the left, top, right, and bottom borders.
[0065] As a specific example, the classification branch uses the cross entropy loss function, the regression branch uses the -lnIOU loss function, and the final loss L of the entire network is expressed as:
[0066] L=λ4L cls +λ5L cen +λ6L reg
[0067] Among them, λ4, λ5 and λ6 are set to 1, 1, and 3 respectively, and L cls , L cen and L reg They represent classification loss, center loss and regression loss respectively.
[0068] As a specific example, the response feature map of the centrality branch described in step 3 is A cen , A cen ∈R h×w×1 , the response characteristic diagram is A cen The size of is h×w, 1 represents the number of channels of the response map, including the center score of each pixel. The center score is multiplied by the classification score to get the accurate target score;
[0069] The true value of the centrality C(i,j) is:
[0070]
[0071] in:
[0072]
[0073] Among them, l, t, r, b represent the feature map points P obtained by the search frame respectively. feature (i,j) corresponds to the search patch point P search (i, j) The distance between the left, top, right, and bottom boundaries of the bounding box defined in the first frame;
[0074] P feature (i,j) is the point in the current feature map, P feature (i,j) in the search patch P search There is a corresponding position in denoted as P search (i,j);P search (i,j) is a point in the search patch, P search (i,j) and feature map P feature The corresponding points in are associated;
[0075] The loss function of centrality is:
[0076]
[0077] The present invention also provides a UAV target tracking system based on collaborative attention and serial channel cross-correlation, which is used to implement the UAV target tracking method based on collaborative attention and serial channel cross-correlation. The system includes a feature extraction module, a cross-correlation operation module, a tracking head network module, a target position determination module, and a tracking module, wherein:
[0078] The feature extraction module crops the first frame of the video to obtain the input of the template branch, crops the subsequent frame images to obtain the input of the search branch, extracts features from the first frame image and the subsequent frame images through the twin network, and then performs feature interaction through the collaborative attention module to obtain a template feature map and a search feature map;
[0079] A cross-correlation operation module inputs the template feature map and the search feature map obtained by the feature extraction module into the series channel cross-correlation module for cross-correlation operation to obtain an output feature map;
[0080] The tracking head network module inputs the output feature map obtained by the cross-correlation operation module into the anchor-free tracking head network, obtains the classification score of each pixel in the feature map through the classification branch, obtains the center score between each pixel and the target through the center branch, and obtains the target box information corresponding to each pixel through the regression branch;
[0081] The target position determination module multiplies the classification score of each pixel in the tracking head network module by the center score, selects the pixel with the highest score, and determines the target frame information of the pixel through the regression branch, thereby determining the position of the target in the current frame;
[0082] The tracking module repeats the processing of the feature extraction module, the cross-correlation operation module, the tracking head network module, and the target position determination module until the position of the last frame in the video is tracked.
[0083] The present invention also provides a mobile terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for tracking unmanned aerial vehicle targets based on collaborative attention and cross-correlation of serial channels is implemented.
[0084] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the steps in the UAV target tracking method based on collaborative attention and serial channel cross-correlation are implemented.
[0085] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0086] Example
[0087] Combination Figure 1 , Figure 2 , this embodiment provides a UAV target tracking method based on collaborative attention and serial channel cross-correlation, comprising the following steps:
[0088] Step 1: Crop the first frame of the video to get the input of the template branch, and crop the subsequent frames to get the input of the search branch. The first frame and the subsequent frames are extracted through the twin network, and then the features are interacted through the collaborative attention module to obtain the template features and search features, such as Figure 3 As shown, the details are as follows:
[0089] Furthermore, the twin network is a weight-shared Resnet50, which uses the last three Block modules of Resnet50, combines the twin network with the collaborative attention module, and adds a collaborative attention module after each Block module;
[0090] Step 1.1: Extract features of the first frame using the twin network to obtain the initial template feature map Extract features from subsequent frames to obtain the initial search feature map
[0091] In the template branch, two feature vectors with channel dimension C / 2 are obtained using 1×1 convolution kernels g(●) and k(·) respectively;
[0092] In the search branch, two feature vectors with channel dimension C / 2 are obtained using 1×1 convolution kernels g(●) and k(·) respectively;
[0093] Where R represents the real number domain; N represents the number of samples processed at a time; W T Indicates the width of the template feature map; H T Represents the height of the template feature map; W I Indicates the width of the search feature map; H I Represents the width of the search feature map; C represents the number of channels of the feature map;
[0094] Step 1.2: The template branch is processed by a 1×1 convolution kernel k(·) and then the tensor is reshaped to obtain the first feature map The search branch is processed by a 1×1 convolution kernel g(●) and then the tensor is reshaped to obtain the second feature map The first feature map And the second feature map Perform matrix multiplication to obtain the intermediate feature map
[0095] Step 1.3: The template branch is processed by the 1×1 convolution kernel g(●) and then the tensor is reshaped to obtain the third feature map The search branch is processed by a 1×1 convolution kernel g(●) and then the tensor is reshaped to obtain the fourth feature map The third feature map obtained Fourth characteristic diagram Respectively with the intermediate feature map of step 1.2 Perform matrix multiplication operation to obtain the template branch modulation feature map and search branch modulation feature map
[0096] Step 1.4: Modulate the template branch feature map processed in step 1.3 Reshape the tensor to get the fifth feature map Searching for branch modulation signatures Reshape the tensor to get the sixth feature map
[0097] Fifth characteristic diagram Sixth characteristic diagram After passing through 1×1 convolution kernels q(·) and w(·), the channel dimension is increased to C, and the fifth feature map after the channel dimension is transformed is obtained. The sixth feature map after changing the channel dimension
[0098] Using the initial template feature map And the fifth feature map after changing the channel dimension Perform matrix addition operation to obtain the intrinsic template related feature map ψ(x; z);
[0099] Using the initial search feature map And the sixth eigenvector after transforming the channel dimension Perform matrix addition operation to obtain intrinsic search related features ψ(z; x);
[0100] Step 1.5: The intrinsic template-related features ψ(x; z) are compressed by two different operations: maximum pooling and global average pooling. Then, they are stimulated by the fully connected layer MLP. Then, the importance of each channel is normalized to a digital range of 0-1 using the Sigmoid operation. Finally, the channel parameters after Sigmoid, that is, the numbers normalized to 0-1, are multiplied by the intrinsic template-related features ψ(x; z) and the intrinsic search-related features ψ(z; x) to obtain the template feature map. and search feature graph
[0101] Step 2: Input the template feature map and search feature map obtained in step 1 into the series channel cross-correlation module for cross-correlation operation to obtain the output feature map, such as Figure 4 The details are as follows:
[0102] The template feature map and search feature graph Perform a Conv+BN+ReLU operation respectively, and then perform the first cross-correlation operation. The formula is:
[0103]
[0104] Among them, Conv represents the convolution operation in deep learning, which filters the input data through the convolution kernel; BN represents the operation of normalizing the features; ReLU is the activation function; φ first Represents the feature map of the first cross-correlation; XCorr dw Indicates the similarity between calculated features;
[0105] Step 2.2: The first cross-correlation feature map φ obtained in step 2.1 first Perform a convolution Conv operation, and then perform two consecutive Conv+BN+ReLU operations for upsampling, with Padding set to 3 and 2 respectively, to obtain the processed features Padding refers to filling extra pixels at the edge of the input feature map in order to control the size of the output feature map during the convolution operation;
[0106] Step 2.3: The processed features obtained in step 2.2 The search feature map obtained in step 1.5 Set a weighted fusion strategy to obtain the fused search features The formula is:
[0107]
[0108] Among them, λ1 and λ2 are set to 0.8 and 0.2 respectively;
[0109] Step 2.4: The template feature map obtained in step 1.5 The template features are obtained through the ECA channel attention mechanism Then the template feature map With template features Add together to get the template features required for the second cross-correlation
[0110] Step 2.5: Perform the second cross-correlation operation, the formula is:
[0111]
[0112] φ second Feature map representing the second cross-correlation;
[0113] Step 2.6: The first cross-correlation feature map φ first And the second cross-correlation feature map φ second Add together to get the output feature map
[0114] Step 3: Input the output feature map obtained in step 2 into the anchor-free tracking head network, obtain the classification score of each pixel in the feature map through the classification branch, obtain the center score between each pixel and the target through the center branch, and obtain the target box information corresponding to each pixel through the regression branch;
[0115] Furthermore, the tracking head network includes two branches, a classification branch and a regression branch, wherein the classification branch is used to distinguish the target, and the regression branch is used to obtain the scale information of the target size;
[0116] The final response feature map obtained by the classification branch is A cls , A cls ∈R h×w×2 , response characteristic diagram A cls The size is h×w, 2 represents the response feature map A cls The number of channels, A cls Contains the probability of pixels being positive and negative samples;
[0117] The response characteristic graph finally obtained by the regression branch is Areg , A reg ∈R h×w×4 , response characteristic diagram A reg The size is h×w, 4 represents the number of channels of the feature map, A reg Contains the distance from each pixel (i, j) to each border, expressed as t (i,j) =(l,t,r,b), where l,t,r,b represents the distance from the pixel point (i,j) to the left, top, right, and bottom borders.
[0118] Furthermore, the classification branch uses the cross entropy loss function, the regression branch uses the -lnIOU loss function, and the final loss L of the entire network is expressed as:
[0119] L=λ4L cls +λ5L cen +λ6L reg
[0120] Among them, λ4, λ5 and λ6 are set to 1, 1, and 3 respectively, and L cls , L cen and L reg They represent classification loss, center loss and regression loss respectively.
[0121] Step 4: Multiply the classification score of each pixel in step 3 by the center score, select the pixel with the highest score, and determine the target frame information of the pixel through the regression branch, thereby determining the location of the target in the current frame;
[0122] Furthermore, the response characteristic graph of the centrality branch is A cen , A cen ∈R h×w×1 , the response characteristic diagram is A cen The size of is h×w, 1 represents the number of channels of the response map, including the center score of each pixel. The center score is multiplied by the classification score to get the accurate target score;
[0123] The true value of the centrality C(i,j) is:
[0124]
[0125] in:
[0126]
[0127] Among them, l, t, r, b represent the feature map points P obtained by the search frame respectively. feature (i,j) corresponds to the search patch point P search (i, j) The distance between the left, top, right, and bottom boundaries of the bounding box defined in the first frame;
[0128] P feature (i,j) is the point in the current feature map, P feature (i,j) in the search patch P search There is a corresponding position in denoted as P search (i,j);P search (i,j) is a point in the search patch, P search (i,j) and feature map P feature The corresponding points in are associated;
[0129] The loss function of centrality is:
[0130]
[0131] Step 5: Repeat steps 1 to 4 until the position of the last frame in the video is tracked.
[0132] Furthermore, the Resnet50 is a deep learning network structure; the Block is a network fast structure unique to Resnet; the Padding is the convolution kernel padding size; the Conv is a convolution operation; the XCorr dw is a deep cross-correlation operation; the IoU is the result of dividing the overlapping part of two regions by the two set parts; the DTB70 is a data set for testing the performance of a tracker.
[0133] Furthermore, the overall network includes a feature extraction module, a collaborative attention module, a serial cross-correlation module and a tracking head network, wherein the feature extraction module is composed of two weight-sharing Resnet50s; the collaborative attention module interacts the template features with the search features to improve the performance of the features; the basic matching information of the target object in the search area is obtained in the serial cross-correlation module; the tracking head network is composed of classification and regression branches for locating and tracking the target. The twin network structure of the present invention is as follows Figure 2 shown.
[0134] This embodiment uses the DTB70 dataset to verify the tracking results of the present invention. The DTB70 dataset contains 70 fully annotated high-definition video datasets and benchmarks captured from a low-altitude aerial perspective, including aspect ratio changes, background clutter, camera motion, fast motion, full occlusion, illumination changes, low resolution, out of view, partial occlusion, similar targets, scale changes, and perspective changes. Multiple attributes can better test the performance of the tracker.
[0135] The first frame of the video is cropped into a size of 127×127×3 with the target as the center as the input of the template branch, and the subsequent frames are cropped into a size of 255×255×3 as the input of the search branch. After passing through the feature extraction network, feature extraction is performed on the first frame and subsequent frames.
[0136] The present invention uses a collaborative attention module to perform feature interaction in the last three blocks of Resnet50 to obtain the processed template features. and search features
[0137] Template Features and search features Input to the series channel cross-correlation operation, perform two cross-correlation operations, expressed as
[0138] The features after the cross-correlation operation are input into the tracking head network to perform calculations on the classification branch and the regression branch. The classification branch uses the cross entropy loss function, and the regression branch uses the -lnIOU loss function. The target box uses the coordinates (x0, y0) and (x1, y1) of the upper left and lower right corners. The distance from any point (x, y) in the search area to the four target bounding boxes is expressed as: l = x-x0, t = y-y0, r = x1-x, b = y1-y.
[0139] The -lnIOU loss function is used to calculate the gap between the true bounding box and the predicted box, thereby regressing the target box.
[0140] The tracker of the present invention is evaluated with trackers such as Ocean, SiamFC++, SiamAPN, SiamAPN++, SiamCAR, and SiamGAT. The final results are as follows: Figure 5 and Figure 6 As shown, the tracking effect is Figure 7 shown.
[0141] Depend on Figure 5 , Figure 6 , Figure 7 It can be seen that the present invention uses the collaborative attention module and the serial channel cross-correlation module to accurately extract the features of the target to obtain a more robust tracking model, which can better solve the problems of target scale change, occlusion, etc., thereby achieving accurate and robust tracking effect.
[0142] The above examples are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention is described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that obvious changes or modifications derived from the spirit of the present invention are still within the protection scope of the present invention.
Claims
1. A UAV target tracking method based on collaborative attention and cascade channel cross-correlation, characterized in that: The following steps are involved: Step 1: Crop the first frame of the video to obtain the input of the template branch, and crop the subsequent frames to obtain the input of the search branch. The first frame and the subsequent frames are subjected to feature extraction through the twin network, and then feature interaction is performed through the collaborative attention module to obtain the template feature map and the search feature map. Step 2: Input the template feature map and the search feature map obtained in step 1 into the series channel cross-correlation module for cross-correlation operation to obtain an output feature map; Step 3: Input the output feature map obtained in step 2 into the anchor-free tracking head network, obtain the classification score of each pixel in the feature map through the classification branch, obtain the center score between each pixel and the target through the center branch, and obtain the target box information corresponding to each pixel through the regression branch; Step 4: Multiply the classification score of each pixel in step 3 by the center score, select the pixel with the highest score, and determine the target frame information of the pixel through the regression branch, thereby determining the location of the target in the current frame; Step 5: Repeat steps 1 to 4 until the position of the last frame in the video is tracked.
2. The method for tracking unmanned aerial vehicle targets based on collaborative attention and serial channel cross-correlation according to claim 1, characterized in that: The twin network in step 1 is a Resnet50 with shared weights. The last three Block modules of Resnet50 are used to combine the twin network with the collaborative attention module, and a collaborative attention module is added after each Block module.
3. The method for tracking unmanned aerial vehicle targets based on collaborative attention and serial channel cross-correlation according to claim 2, characterized in that: The step 1 is specifically as follows: Step 1.1, crop the first frame of the video with the target as the center as the input of the template branch, and crop the subsequent frames as the input of the search branch; The first frame of the picture is extracted through the twin network to obtain the initial template feature map Extract features from subsequent frames to obtain the initial search feature map Where R represents the real number domain; N represents the number of samples processed at a time; W T Indicates the width of the template feature map; H T Represents the height of the template feature map; W I Indicates the width of the search feature map; H I Represents the width of the search feature map; C represents the number of channels of the feature map; Step 1.2: The template branch is processed by a 1×1 convolution kernel k(·) and then the tensor is reshaped to obtain the first feature map The search branch is processed by a 1×1 convolution kernel g(·) and then the tensor is reshaped to obtain the second feature map The first feature map And the second feature map Perform matrix multiplication to obtain the intermediate feature map Step 1.3: The template branch is processed by the 1×1 convolution kernel g(·) and then the tensor is reshaped to obtain the third feature map The search branch is processed by a 1×1 convolution kernel g(·) and then the tensor is reshaped to obtain the fourth feature map The third feature map obtained Fourth characteristic diagram Respectively with the intermediate feature map of step 1.2 Perform matrix multiplication operation to obtain the template branch modulation feature map and search branch modulation feature map Step 1.4: Modulate the template branch feature map processed in step 1.3 Reshape the tensor to get the fifth feature map Searching for branch modulation signatures Reshape the tensor to get the sixth feature map Fifth characteristic diagram Sixth characteristic diagram After passing through 1×1 convolution kernels q(·) and w(·), the channel dimension is increased to C, and the fifth feature map after the channel dimension is transformed is obtained. The sixth feature map after changing the channel dimension Using the initial template feature map And the fifth feature map after changing the channel dimension Perform matrix addition operation to obtain the intrinsic template related feature map ψ(x; z); Using the initial search feature map and the sixth eigenvector after transforming the channel dimension Perform matrix addition operation to obtain intrinsic search related features ψ(z; x); Step 1.5: The intrinsic template-related features ψ(x; z) are compressed by two different operations: maximum pooling and global average pooling. Then, they are stimulated by the fully connected layer MLP. Then, the importance of each channel is normalized to a digital range of 0-1 using the Sigmoid operation. Finally, the channel parameters after Sigmoid, that is, the numbers normalized to 0-1, are multiplied by the intrinsic template-related features ψ(x; z) and the intrinsic search-related features ψ(z; x) to obtain the template feature map. and search feature graph 4. The method for tracking unmanned aerial vehicle targets based on collaborative attention and serial channel cross-correlation according to claim 3, characterized in that: In step 2, the template feature map and the search feature map obtained in step 1 are input into the series channel cross-correlation module for cross-correlation operation to obtain an output feature map, which is as follows: Step 2.1: Template feature map and search feature graph Perform a Conv+BN+ReLU operation respectively, and then perform the first cross-correlation operation. The formula is: Among them, Conv represents the convolution operation in deep learning, which filters the input data through the convolution kernel; BN represents the operation of normalizing the features; ReLU is the activation function; φ first Represents the feature map of the first cross-correlation; XCorr dw Indicates the similarity between calculated features; Step 2.2: The first cross-correlation feature map φ obtained in step 2.1 first Perform a convolution Conv operation, and then perform two consecutive Conv+BN+ReLU operations for upsampling, with Padding set to 3 and 2 respectively, to obtain the processed features Padding refers to filling extra pixels at the edge of the input feature map in order to control the size of the output feature map during the convolution operation; Step 2.3: The processed features obtained in step 2.2 The search feature map obtained in step 1.5 Set a weighted fusion strategy to obtain the fused search features The formula is: Among them, λ1 and λ2 are set to 0.8 and 0.2 respectively; Step 2.4: The template feature map obtained in step 1.5 The template features are obtained through the ECA channel attention mechanism Then the template feature map With template features Add together to get the template features required for the second cross-correlation Step 2.5: Perform the second cross-correlation operation, the formula is: φ second Feature map representing the second cross-correlation; Step 2.6: The first cross-correlation feature map φ first And the second cross-correlation feature map φ second Add together to get the output feature map 5. The method for tracking unmanned aerial vehicle targets based on collaborative attention and serial channel cross-correlation according to claim 4, characterized in that: The classification branch in step 3 is used to distinguish the target, and the regression branch is used to obtain the scale information of the target size; The final response feature map obtained by the classification branch is A cls , A cls ∈R h×w×2 , response characteristic diagram A cls The size is h×w, 2 represents the response feature map A cls The number of channels, A cls Contains the probability of pixels being positive and negative samples; The response characteristic graph finally obtained by the regression branch is A reg , A reg ∈R h×w×4 , response characteristic diagram A reg The size is h×w, 4 represents the number of channels of the feature map, A reg Contains the distance from each pixel (i, j) to each border, expressed as t (i,j) =(l,t,r,b), where l,t,r,b represents the distance from the pixel point (i,j) to the left, top, right, and bottom borders.
6. The method for tracking unmanned aerial vehicle targets based on collaborative attention and serial channel cross-correlation according to claim 5, characterized in that: The classification branch uses the cross entropy loss function, the regression branch uses the -lnIOU loss function, and the final loss L of the entire network is expressed as: L=λ4L cls +λ5L cen +λ6L reg Among them, λ4, λ5 and λ6 are set to 1, 1, and 3 respectively, and L cls , L cen and L reg They represent classification loss, center loss and regression loss respectively.
7. The method for tracking unmanned aerial vehicle targets based on collaborative attention and serial channel cross-correlation according to claim 6, characterized in that: The response feature map of the centrality branch described in step 3 is A cen , A cen ∈R h×w×1 , the response characteristic diagram is A cen The size of is h×w, 1 represents the number of channels of the response map, including the center score of each pixel. The center score is multiplied by the classification score to get the accurate target score; The true value of the centrality C(i,j) is: in: Among them, l, t, r, b represent the feature map points P obtained by the search frame respectively. feature (i,j) corresponds to the search patch point P search (i, j) The distance between the left, top, right, and bottom boundaries of the bounding box defined in the first frame; P feature (i,j) is the point in the current feature map, P feature (i,j) in the search patch P search There is a corresponding position in denoted as P search (i,j);P search (i,j) is a point in the search patch, P search (i,j) and feature map P feature The corresponding points in are associated; The loss function of centrality is:
8. A UAV target tracking system based on collaborative attention and serial channel cross-correlation, characterized in that: The system is used to implement the UAV target tracking method based on collaborative attention and serial channel cross-correlation as described in any one of claims 1 to 7, and the system includes a feature extraction module, a cross-correlation operation module, a tracking head network module, a target position determination module, and a tracking module, wherein: The feature extraction module crops the first frame of the video to obtain the input of the template branch, crops the subsequent frame images to obtain the input of the search branch, extracts features from the first frame image and the subsequent frame images through the twin network, and then performs feature interaction through the collaborative attention module to obtain a template feature map and a search feature map; A cross-correlation operation module inputs the template feature map and the search feature map obtained by the feature extraction module into the series channel cross-correlation module for cross-correlation operation to obtain an output feature map; The tracking head network module inputs the output feature map obtained by the cross-correlation operation module into the anchor-free tracking head network, obtains the classification score of each pixel in the feature map through the classification branch, obtains the center score between each pixel and the target through the center branch, and obtains the target box information corresponding to each pixel through the regression branch; The target position determination module multiplies the classification score of each pixel in the tracking head network module by the center score, selects the pixel with the highest score, and determines the target frame information of the pixel through the regression branch, thereby determining the position of the target in the current frame; The tracking module repeats the processing of the feature extraction module, the cross-correlation operation module, the tracking head network module, and the target position determination module until the position of the last frame in the video is tracked.
9. A mobile terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the UAV target tracking method based on collaborative attention and serial channel cross-correlation as described in any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the UAV target tracking method based on collaborative attention and serial channel cross-correlation as described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Image attention visual target tracking method
CN113256677A
SAR image classification method based on attention mechanism and residual relation network
CN114372521A
Twin network tracking system and method based on space-time attention mechanism
CN114707604A
Twin network target tracking method based on deformable convolution
CN114882395A
Target tracking algorithm based on multi-graph attention mechanism
CN115578421A
