A multi-scale UAV aerial photography target tracking method and device

Abstract

The present invention discloses a multi-scale method and device for target tracking in UAV aerial photography, which relates to the technical field of image tracking and includes: obtaining an aerial photography video of a UAV, inputting the initial frame and the current frame of the aerial photography video of the UAV into the template branch and the search branch of a Siamese tracking network constructed based on the G-ResNet network, respectively outputting three groups of first weighted feature maps and second weighted feature maps from three convolutional blocks of layer2, layer3, and layer4 of the G-ResNet network, and using a plurality of anchor-free region proposal networks to perform weighted fusion on the three groups of first weighted feature maps and second weighted feature maps to obtain the target tracking result of the current frame. This method can solve the problem that none of the UAV tracking algorithms can well achieve the balance between accuracy and speed.

Description

Technical Field

[0001] The present invention relates to the technical field of image tracking, and more particularly to a multi-scale UAV aerial photography target tracking method and device. Background Art

[0002] Target tracking is used to estimate the state of the tracking target in each frame of the video sequence, and the information of this tracking target is only given in the first frame. During the tracking process of the UAV for the target, the video image is transmitted to the ground station for display through the data link system. The operator controls the stable UAV platform and the camera system to search for and reconnoiter the target through the joystick and other instructions. When the target of interest appears on the screen, the target of interest is selected. The ground computer extracts a series of features of the target and uses them as a template. The computer confirms the position information of the target of interest in the subsequent images by calculating the similarity between the template image and the subsequent images, so as to realize the continuous tracking of the target.

[0003] The main problems of the UAV tracking algorithm are divided into two aspects:

[0004] Tracking accuracy: The video field of view captured by the UAV is large and the range is wide. The size of the captured target is relatively small, the number of pixels included in the target is small, the features of the target are few and not obvious, and there is also more background information, which is prone to many similar target interferences on the tracking of the target. It is difficult for the algorithm to distinguish the background from the target and it is easy to track the wrong target; during the flight of the UAV, camera jitter and flight speed changes are likely to occur, resulting in problems such as motion blur and appearance changes, testing the small target representation and discrimination ability of the algorithm; the UAV has good maneuverability, and its flight motion usually has a higher degree of freedom and fewer constraints on flight, and it is more likely to have situations such as fast motion and large scale changes. Insufficient scale adaptation ability of the tracking algorithm will include too much background information and contaminate the target information.

[0005] Tracking speed: The shooting equipment that the UAV can carry during execution includes visible light, thermal infrared, SAR, etc. A large amount of data can be collected in one mission, and it is usually executed in the form of multiple UAVs collaborating. The amount of intelligence data to be processed also becomes larger accordingly, requiring the tracking algorithm to be able to process a large amount of data information in real time.

[0006] The traditional correlation filtering tracking algorithm is fast, but it uses manually designed features to represent the target, and its representation ability for the target is insufficient, and it is difficult to improve the tracking accuracy. Most Siamese tracking algorithms, in order to pursue tracking accuracy, use a series of cumbersome and complex operations, ignoring the requirement for tracking speed, but the tracking speed does not meet the real-time requirement and it is difficult to be deployed on the UAV platform. Currently, none of the UAV tracking algorithms can achieve a good balance between accuracy and speed. Summary of the Invention

[0007] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, a first aspect of the present invention proposes a multi-scale UAV aerial photography target tracking method, including:

[0008] Obtain the UAV aerial photography video;

[0009] Input the initial frame and the current frame of the UAV aerial photography video into the template branch and the search branch of the Siamese tracking network constructed based on the G-ResNet network, and output three groups of first weighted feature maps and second weighted feature maps respectively from the three convolutional blocks of layer2, layer3, and layer4 of the G-ResNet network. The G-ResNet network is obtained by replacing the 3×3 convolutional kernels of the residual modules of each Bottleneck in the resnet50 network with multiple convolutional layer groups of the same topological structure stacked in parallel, and adding a dual multi-scale attention module behind each Bottleneck;

[0010] Utilize multiple anchor-free region proposal networks to perform weighted fusion on the three groups of first weighted feature maps and second weighted feature maps, and track the target of the current frame according to the prediction boxes and prediction positions in the results of the weighted fusion.

[0011] Further, replacing the 3×3 convolutional kernels of the residual modules of each Bottleneck in the resnet50 network with multiple convolutional layer groups of the same topological structure stacked in parallel includes:

[0012] In layer1, divide the 3×3 convolutional kernels with 64 channels in the residual modules of 3 Bottlenecks into 32 groups of parallel stacked convolutional layer groups with 4 channels and a convolutional kernel size of 3×3 through grouped convolution;

[0013] In layer2, divide the 3×3 convolutional kernels with 128 channels in the residual modules of 4 Bottlenecks into 32 groups of parallel stacked convolutional layer groups with 8 channels and a convolutional kernel size of 3×3 through grouped convolution;

[0014] In layer3, divide the 3×3 convolutional kernels with 256 channels in the residual modules of 6 Bottlenecks into 32 groups of parallel stacked convolutional layer groups with 16 channels and a convolutional kernel size of 3×3 through grouped convolution;

[0015] In layer4, divide the 3×3 convolutional kernels with 512 channels in the residual modules of 3 Bottlenecks into 32 groups of parallel stacked convolutional layer groups with 32 channels and a convolutional kernel size of 3×3 through grouped convolution.

[0016] Further, the first weighted feature map and the second weighted feature map are output from three convolutional blocks of layer2, layer3, and layer4 of the G-ResNet network, respectively, including:

[0017] Through the dual multi-scale attention module, the first feature map and the second feature map output from the first Bottleneck in layer2, layer3, and layer4 of the template branch and the search branch are extracted respectively;

[0018] The first feature map and the second feature map are grouped respectively to obtain multiple grouped feature maps corresponding to the first feature map and the second feature map;

[0019] Each grouped feature map is decomposed into a first sub-feature map and a second sub-feature map;

[0020] The first sub-feature map and the second sub-feature map are processed by the position attention module and the channel attention module respectively to obtain a sub-feature map with position attention response and a third sub-feature map and a fourth sub-feature map with channel attention response respectively;

[0021] The third sub-feature map and the fourth sub-feature map are fused in channels to obtain a fifth sub-feature map corresponding to a grouped feature map;

[0022] Multiple fifth sub-feature maps corresponding to multiple grouped feature maps are obtained;

[0023] The multiple fifth sub-feature maps are shuffled to obtain the weighted feature maps output from the first Bottleneck of the template branch and the search branch of the first Bottleneck;

[0024] The weighted feature maps output from the first Bottleneck of the template branch and the search branch are propagated backward in sequence, and the first weighted feature map and the second weighted feature map are output from the last Bottleneck of layer2, layer3, and layer4 respectively.

[0025] Further, the expression of the position attention response includes:

[0026] ;

[0027] Wherein, represents the first sub-feature map, and IN(X k1 ) represents the completion of spatial information statistics , and respectively represent the weights and sigmoid non-linear activation function used to strengthen .

[0028] Further, the channel attention response expression includes:

[0029] ;

[0030]

[0031] where H and W respectively represent the height and width of the second sub-feature map, represents the second sub-feature map, F gap represents the global average pooling function, performs scaling and offset operations on s ; represents the sigmoid non-linear activation function.

[0032] Further, using multiple anchor-free region proposal networks to perform weighted fusion on three groups of first weighted feature maps and second weighted feature maps, including:

[0033] Set RPN modules with anchor-free strategies respectively between the layer2, layer3, and layer4 convolutional blocks of the template branch and the search branch of the G-ResNet network. The RPN modules with anchor-free strategies include a classification branch and a regression branch. The regression branch predicts the offset between the target pixel point and the true box;

[0034] Input the first weighted feature map and the second weighted feature map into the convolutional networks in the regression branch and the classification branch of the RPN module with anchor-free strategy respectively, and output a regression map and a classification map from the regression branch, and output a regression map and a classification map from the classification branch;

[0035] Perform depth cross-correlation operation on the two regression maps output by the classification branch and the regression branch to obtain a regression result;

[0036] Perform depth cross-correlation operation on the two classification maps output by the classification branch and the regression branch to obtain a classification result;

[0037] Obtain the position where the maximum value of the classification result is located as the predicted position of the target;

[0038] Obtain the predicted bounding box corresponding to the predicted position in the regression result as the predicted box of the target.

[0039] The present invention also provides a multi-scale UAV aerial photography target tracking device, including:

[0040] An acquisition module for acquiring UAV aerial photography videos;

[0041] ​​A processing module is configured to input an initial frame and a current frame of an aerial video captured by a drone into a template branch and a search branch of a Siamese tracking network constructed based on a G-ResNet network, and output three groups of first weighted feature maps and second weighted feature maps respectively from three convolutional blocks of layer2, layer3, and layer4 of the G-ResNet network. The G-ResNet network is obtained by replacing the 3×3 convolutional kernels of the residual modules of each Bottleneck in the resnet50 network with multiple convolutional layer groups of the same topological structure stacked in parallel, and adding a dual multi-scale attention module behind each Bottleneck.

[0042] An output module is configured to use multiple anchor-free region proposal networks to perform weighted fusion on the three groups of first weighted feature maps and second weighted feature maps, and track the target of the current frame according to the prediction boxes and prediction positions in the result of the weighted fusion.

[0043] The present invention also provides an electronic device, which includes a processor and a memory. At least one instruction, at least one program, a code set, or an instruction set is stored in the memory, and is loaded and executed by the processor to implement the multi-scale aerial drone target tracking method according to any one of the first aspect.

[0044] The present invention also provides a computer-readable storage medium, in which at least one instruction, at least one program, a code set, or an instruction set is stored, and is loaded and executed by the processor to implement the multi-scale aerial drone target tracking method according to any one of the first aspect.

[0045] The embodiments of the present invention provide a multi-scale aerial drone target tracking method and device. Compared with the prior art, the beneficial effects are as follows:

[0046] 1) Using the subspace learning idea of grouping-transformation-fusion, a grouped residual network G-ResNet is designed, which can extract deep semantic features and diverse features of the target, effectively cope with challenges such as appearance changes and motion blur of the target, and enhance the representation ability for small targets.

[0047] 2) A multi-scale attention module DMSAM is designed. The feature maps are grouped to extract target feature information of different scales, and then dual attention is used to extract local features of the target and establish global dependencies between the target and the background in the spatial and channel dimensions respectively. Finally, information exchange between different channels is established to enhance the scale adaptation ability and anti-interference ability of the present invention.

[0048] 3) A region proposal module AF-RPN based on an anchor-free strategy is proposed to replace the pre-defined anchor boxes, distinguish the target from the background pixel by pixel, and achieve the adaptive perception ability of the target scale. Multiple AF-RPNs are cascaded in the G-ResNet, effectively utilizing complementary detailed information and semantic information to achieve robust tracking and precise positioning of the tracked target. At the same time, the speed reaches 40.5 FPS, meeting the real-time requirement. Description of the Drawings

[0049] To more clearly illustrate the technical solutions of the present invention, the accompanying drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.

[0050] Figure 1 It is a flowchart of a multi-scale UAV aerial target tracking method provided by an embodiment of the present invention;

[0051] Figure 2 It is a network model diagram of a UAV target tracking method based on a dual multi-scale attention module provided by the present invention;

[0052] Figure 3 It is an instance diagram of replacing layer1 of a multi-scale UAV aerial target tracking method provided by an embodiment of the present invention;

[0053] Figure 4 It is a schematic diagram of the principle of DASAM of a multi-scale UAV aerial target tracking method provided by an embodiment of the present invention;

[0054] Figure 5 It is a shuffling schematic diagram of a multi-scale UAV aerial target tracking method provided by an embodiment of the present invention;

[0055] Figure 6 It is a schematic diagram of the principle of AF-RPN of a multi-scale UAV aerial target tracking method provided by an embodiment of the present invention;

[0056] Figure 7 It is a structural diagram of a multi-scale UAV aerial target tracking device provided by an embodiment of the present invention. Detailed Embodiments

[0057] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0058] This specification provides method operation steps as described in the embodiments or flowcharts, but based on routine or non-creative labor, it may include more or fewer operation steps. When actually executed in a system or server product, it can be executed sequentially or in parallel according to the methods shown in the embodiments or the drawings (for example, in an environment of parallel processors or multi-threaded processing).

[0059] Currently, target tracking algorithms are mainly divided into correlation filter-based tracking algorithms and deep learning-based tracking algorithms. Correlation filter tracking algorithms use correlation filters in the field of signal processing to calculate the similarity between the template and the search image, and use the Fourier transform to accelerate in the frequency domain, greatly reducing the amount of computation and improving the computation speed, which can reach hundreds of frames per second. However, most correlation filter algorithms use traditional feature extraction algorithms to represent the tracking target, with insufficient robustness and accuracy, and cannot effectively handle target tracking tasks in complex scenarios.

[0060] Siamese tracking algorithms have gradually become the mainstream algorithms in the field of target tracking due to their great potential in terms of accuracy and speed, and most subsequent tracking algorithms are studied based on the Siamese structure. The working principle of the Siamese tracking algorithm can be expressed as Equation (1). The Siamese tracking algorithm mainly consists of a feature extraction part , a similarity calculation part ( ) and a tracking result generation part.

[0061] (1)

[0062] In the formula, is the similarity response map; is the feature extraction part; is the cross-correlation operation; is the deviation at each position; is the identity matrix.

[0063] 1) Feature extraction part: Use a Siamese neural network to extract features. The two branches are the template branch and the search branch respectively. The template branch inputs the target image of the initial frame as the template, and the output is the template feature map , and the search branch inputs the search image of the subsequent frame, and the output is the search feature map

[0064] 2) Similarity calculation part ( ):It is used to integrate the feature information on the feature maps of two branches, calculate the similarity between the search feature map and the template feature map, and generate a similarity response map .

[0065] 3) Tracking result generation part: According to the obtained response map, predict the target position on the search image. The position with the maximum response is generally considered as the target prediction position, and then target scale estimation and bounding box regression are performed.

[0066] The process of the Siamese tracking algorithm for online tracking mainly includes the following steps:

[0067] The video sequence is input frame by frame into the feature extraction part;

[0068] If it is the first frame, the template branch extracts the target feature as the template feature;

[0069] If it is not the first frame, the search branch extracts the target feature of the current frame as the search feature;

[0070] The similarity calculation part calculates the similarity between the feature maps and generates a response map;

[0071] The tracking result generation part uses the similarity response map to predict the target position in the current frame;

[0072] Repeat steps 3 - 5 until the last frame of the video sequence.

[0073] Figure 1 It is a flowchart of a multi - scale UAV aerial photography target tracking method provided by an embodiment of the present invention. As Figure 1 shown, the method includes:

[0074] Step 101, obtain the UAV aerial photography video;

[0075] Step 102, input the initial frame and the current frame of the UAV aerial photography video into the template branch and the search branch of the Siamese tracking network constructed based on the G - ResNet network, and respectively output three groups of first weighted feature maps and second weighted feature maps from the three convolutional blocks of layer2, layer3, and layer4 of the G - ResNet network. The G - ResNet network is obtained by replacing the 3×3 convolutional kernels of the residual modules of each Bottleneck in the resnet50 network with multiple convolutional layer groups of the same topological structure stacked in parallel, and adding a dual multi - scale attention module behind each Bottleneck;

[0076] Step 103: Use multiple anchor-free region proposal networks to perform weighted fusion on three groups of first weighted feature maps and second weighted feature maps, and track the target in the current frame according to the prediction boxes and prediction positions in the results of the weighted fusion.

[0077] Figure 2 It is a network model diagram of an unmanned aerial vehicle target tracking method based on a dual multi-scale attention module provided by the present invention. As Figure 2 shown, first, a group residual network (Group Residual Network, G-ResNet) is designed, and convolutional blocks with the same topological structure are stacked in parallel to extract diverse features of the target, enhancing the representation ability for the tracked target without increasing the network depth. Secondly, in order to better screen features, two multi-scale attention (Dual Multi Scale Attention Module, DMSAM) are used to extract multi-scale feature information of the target, suppressing interference information in both the channel and spatial dimensions. In the final tracking box generation stage, multiple anchor-free region proposal networks (Anchor Free Region Proposal Network, AF-RPN) are used to adaptively perceive the scale change of the target, effectively solving the scale change problem. Experiments show that the present invention can more effectively cope with problems such as scale change, small targets, motion blur, and partial occlusion, improving the tracking effect of aerial targets, and the speed reaches 40.5 FPS, meeting the real-time requirement.

[0078] In a possible implementation manner, by replacing the 3×3 convolutional kernels of the residual modules of each Bottleneck in the resnet50 network with multiple groups of convolutional layers with the same topological structure stacked in parallel, it includes:

[0079] In layer1, through grouped convolution, the 3×3 convolutional kernels with 64 channels in the residual modules of 3 Bottlenecks are divided into 32 groups of parallel stacked convolutional layer groups with 4 channels and a convolutional kernel size of 3×3;

[0080] In layer2, through grouped convolution, the 3×3 convolutional kernels with 128 channels in the residual modules of 4 Bottlenecks are divided into 32 groups of parallel stacked convolutional layer groups with 8 channels and a convolutional kernel size of 3×3;

[0081] In layer3, through grouped convolution, the 3×3 convolutional kernels with 256 channels in the residual modules of 6 Bottlenecks are divided into 32 groups of parallel stacked convolutional layer groups with 16 channels and a convolutional kernel size of 3×3;

[0082] In layer4, through grouped convolution, the 3x3 convolutional kernels with 512 channels in the residual modules of 3 Bottlenecks are divided into 32 groups of parallel stacked convolutional layer groups with 32 channels and a convolutional kernel size of 3x3.

[0083] In the embodiments provided by the present invention, the present invention increases the cardinality on the deeper ResNet-50 to improve the network performance. Compared with increasing the number of network layers, increasing the cardinality of the network can more effectively improve the feature description ability of the network, and at the same time will not increase the number of network parameters. Based on the design idea of split-transform-merge, as Figure 3 shown, Figure 3 What is shown is a replacement instance of layer1. Considering that the in the residual module is the main part for extracting feature information, so the in the residual module is replaced with multiple parallel stacked convolutional layer groups with the same topological structure. In the convolutional process of ordinary convolution, all channels of the input feature map are required to participate in the calculation for one channel of the output feature map. In the implementation process of the parallel stacking operation, through grouped convolution, the with 64 channels is divided into 32 with 4 channels. Different convolutional layer groups can be regarded as different subspaces, and the feature information learned by each subspace has different emphases and is different from each other, that is, diverse feature information of the target is extracted.

[0084] In a possible implementation manner, three groups of first weighted feature maps and second weighted feature maps are output from the layer2, layer3, and layer4 convolutional blocks of the G-ResNet network respectively, including:

[0085] Through the dual multi-scale attention module, the first feature maps and the second feature maps output from the first Bottleneck in layer2, layer3, and layer4 of the template branch and the search branch are extracted respectively;

[0086] The first feature maps and the second feature maps are grouped respectively to obtain multiple grouped feature maps corresponding to the first feature maps and the second feature maps respectively;

[0087] Each grouped feature map is decomposed into a first sub-feature map and a second sub-feature map;

[0088] The first sub-feature map and the second sub-feature map are processed by the position attention module and the channel attention module respectively to obtain a sub-feature map with position attention response and a third sub-feature map and a fourth sub-feature map with channel attention response respectively;

[0089] Fuse the third sub-feature map and the fourth sub-feature map in the channel dimension to obtain a fifth sub-feature map corresponding to a grouped feature map;

[0090] Obtain multiple fifth sub-feature maps corresponding to multiple grouped feature maps;

[0091] Shuffle the multiple fifth sub-feature maps to obtain a weighted feature map output from the template branch and the search branch of the first Bottleneck;

[0092] Propagate the weighted feature map output from the template branch and the search branch of the first Bottleneck backward in turn, and output a first weighted feature map and a second weighted feature map from the last Bottleneck of layer2, layer3, and layer4 respectively.

[0093] In the embodiment provided by the present invention, the attention module can adaptively allocate weights and selectively filter feature map information, thereby helping the network to better focus on the target of interest and effectively making up for the deficiencies of G-ResNet. Therefore, in order to improve the discrimination ability of the present invention, a dual multi-scale attention module (DMSAM) is introduced into G-ResNet. As Figure 4 shown, in order to enable the network to learn feature information at different scales, DMSAM first extracts features at various scales and groups them; then uses the spatial and channel attention modules in parallel to adaptively capture local features and global dependencies; finally, fuses and shuffles the feature maps of all channels to strengthen the information exchange between different channels.

[0094] First, assume that the input feature map is , where represent the number of channels, height, and width of the feature map respectively. To reduce the computational cost, is divided into groups of sub-feature maps in the channel dimension. , because the sub-feature maps are divided by channels, each sub-feature map can capture specific semantic information during the training process. The sub-feature map is split into two, obtaining , one uses channel attention to capture the inter-channel relationships, and the other uses spatial attention to find the spatial relationships between features. Therefore, through the weight assignment of the attention module, the network better knows what and where to focus on is meaningful.

[0095] In a possible implementation manner, the expression of the spatial attention response includes:

[0096] (2)

[0097] Among them, represents the first sub-feature map, IN(X k1 ) represents the completion using instance normalization of the spatial information statistics , and respectively represent the weights and sigmoid non-linear activation functions used to enhance .

[0098] In the embodiments provided by the present invention, during the UAV tracking process, there are often objects similar to the tracking target, resulting in the presence of feature information of the tracking target and the feature information of similar objects on the feature map. Position attention is to enhance the discrimination ability for similar objects and give greater attention to the position where the target is located. The present invention uses Instance Normalization (IN) to complete the spatial information statistics on , and the final position attention response can be obtained from Equation (3):

[0099] (3)

[0100] In the formula: and are used to enhance the representation ability. The weight design of the position attention response for each position of the feature map effectively suppresses the interference of similar objects and clarifies what position (where) on the image the network should focus on the target.

[0101] In a possible implementation manner, the channel attention response expression includes:

[0102] (4)

[0103] (5)

[0104] Among them, H and W respectively represent the height and width of the second sub-feature map, represents the second sub-feature map, F gap represents the global average pooling function, performs scaling and offset operations on s , represents the sigmoid non-linear activation function.

[0105] On the feature map of the deep network, different channels represent different semantic information. The process of channel attention allocating weights can be regarded as the process of selecting semantic attributes for different channels. The present invention uses global average pooling (GAP) to compress Feature layer on the channel, get the result :

[0106] (6)

[0107] In order to learn the nonlinear relationship between channels, Through the sigmoid nonlinear activation function Get the weight coefficient, adaptively guide the network to select the appropriate feature map, channel attention response From formula (7), we can get:

[0108] (7)

[0109] Where: right Perform scaling and offset operations. Assign weights to the feature map according to different semantic information, with the channel where the target is located having the largest weight. During the cross-correlation operation, the responses on other channels are suppressed, making it clear what category of targets the network should focus on.

[0110] Before shuffling, attention response and Connect to get a new sub-feature map All new sub-feature maps are superimposed by channel and combined to form a feature map , as shown in formula (8). Then formula (9) uses channel shuffle (channel_shuffle) as Figure 5 As shown, the operation process First, Expand to A matrix of four dimensions, then the matrix Dimension unchanged, The dimension is transposed, and then the matrix dimension is compressed to obtain the output feature map The shuffling operation can effectively integrate the feature information on each channel and strengthen the information exchange between channels.

[0111] (8)

[0112] (9)

[0113] In DMSAM, target feature information of different scales is extracted from the grouped feature map, and then dual attention is used to extract local features of the target in the channel and spatial dimensions respectively, and to establish a global dependency relationship between the target and the background. Finally, information exchange between different channels is established to increase the distinction between target and interference information, thereby improving the scale adaptability and discrimination ability of the present invention.

[0114] In a possible implementation, multiple anchor-free region proposal networks are used to perform weighted fusion on three groups of first weighted feature maps and second weighted feature maps, including:

[0115] An RPN module with an anchor-free strategy is respectively set between the layer2, layer3, and layer4 convolutional blocks of the template branch and the search branch of the G-ResNet network. The RPN module with the anchor-free strategy includes a classification branch and a regression branch. The regression branch predicts the offset between the target pixel point and the ground truth box;

[0116] The first weighted feature map and the second weighted feature map are respectively input into the convolutional networks in the regression branch and the classification branch of the RPN module with the anchor-free strategy, and a regression map and a classification map are output from the regression branch, and a regression map and a classification map are output from the classification branch;

[0117] Perform depth cross-correlation operation on the two regression maps output by the classification branch and the regression branch to obtain a regression result;

[0118] Perform depth cross-correlation operation on the two classification maps output by the classification branch and the regression branch to obtain a classification result;

[0119] Obtain the position where the maximum value of the classification result is located as the predicted position of the target;

[0120] Obtain the predicted bounding box corresponding to the predicted position in the regression result as the predicted box of the target.

[0121] In a possible implementation, a group of anchor boxes with different scales and ratios are predefined in the RPN module of the present invention for scale estimation. The prior information of these anchor boxes is all analyzed from the video, which violates the starting point of the tracking task, and the tracking performance is very sensitive to these parameters of the anchor boxes and requires careful manual setting. Therefore, in order for the tracking algorithm to get rid of excessive dependence on the target prior information, an anchor-free strategy is used in the RPN module to complete the adaptive estimation of the target scale. Based on the RPN module (AF-RPN) with the anchor-free strategy, its bounding box regression branch No longer regresses the size (length, width, center point position) of the anchor, but predicts the offsets l, t, b between the target pixel point and the ground truth box (Ground-truth) ,r ; The previous classification branch Judges whether the target in the anchor is a positive sample by calculating the intersection over union (IoU) between the anchor and the ground truth box. Therefore, the anchor-free strategy requires a new positive and negative sample discrimination method: map the pixel points of the similarity response map back to the search image, and those falling outside the ellipse E1 are negative samples; those falling inside the ellipse E2 are positive samples, asFigure 6 as shown

[0122] (9)

[0123] wherein and are the classification result and the regression result; represents the depth cross - correlation operation; is the feature extraction network; are the width, height and number of channels of the feature map,

[0124] Then, find the maximum value on the classification result S. The position where the maximum value is located is the predicted position of the target. At the same time, there is a corresponding predicted bounding box at this position in the regression result, and it is used as the predicted box of the target.

[0125] The present invention also provides a multi - scale UAV aerial photography target tracking device 200, as Figure 7 shown, including:

[0126] An acquisition module 201, configured to acquire the UAV aerial photography video;

[0127] A processing module 202, configured to input the initial frame and the current frame of the UAV aerial photography video into the template branch and the search branch of the Siamese tracking network constructed based on the G - ResNet network, and output three groups of first weighted feature maps and second weighted feature maps respectively from the three convolutional blocks of layer2, layer3 and layer4 of the G - ResNet network. The G - ResNet network is obtained by replacing the 3×3 convolutional kernels of the residual modules of each Bottleneck in the resnet50 network with a plurality of convolutional layer groups with the same topological structure stacked in parallel, and adding a double multi - scale attention module behind each Bottleneck;

[0128] An output module 203, configured to use multiple anchor - free region proposal networks to perform weighted fusion on the three groups of first weighted feature maps and second weighted feature maps, and track the target in the current frame according to the predicted boxes and predicted positions in the result of the weighted fusion.

[0129] In another embodiment provided by the present invention, a device is also provided. The device includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the multi - scale UAV aerial photography target tracking method described in the embodiments of the present invention.

[0130] In another embodiment provided by the present invention, a computer-readable storage medium is further provided. At least one instruction, at least one program, a code set or an instruction set is stored in the storage medium. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the multi-scale UAV aerial photography target tracking method described in the embodiments of the present invention.

[0131] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes a plurality of computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that integrates a plurality of available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)).

[0132] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0133] Each embodiment in this specification is described in a related manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and reference can be made to the relevant parts of the method embodiment for the related content.

[0134] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are all included in the protection scope of the present invention.

Claims

1. A multi-scale UAV aerial photography target tracking method, characterized in that, Including: Obtaining an aerial video captured by a drone; Inputting the initial frame and the current frame of the aerial video captured by the drone into the template branch and the search branch of the Siamese tracking network constructed based on the G-ResNet network, respectively outputting three groups of first weighted feature maps and second weighted feature maps from three convolutional blocks of layer2, layer3, and layer4 of the G-ResNet network. The G-ResNet network is obtained by replacing the 3×3 convolutional kernels of the residual modules of each Bottleneck in the resnet50 network with multiple convolutional layer groups with the same topological structure stacked in parallel and adding a dual multi-scale attention module behind each Bottleneck; Using multiple anchor-free region proposal networks to perform weighted fusion on the three groups of first weighted feature maps and second weighted feature maps, and tracking the target in the current frame according to the predicted bounding boxes and predicted positions in the result of the weighted fusion; The replacing the 3×3 convolutional kernels of the residual modules of each Bottleneck in the resnet50 network with multiple convolutional layer groups with the same topological structure stacked in parallel includes: In layer1, dividing the 3×3 convolutional kernels with 64 channels in the residual modules of 3 Bottlenecks into 32 groups of convolutional layer groups with 4 channels and a convolutional kernel size of 3×3 stacked in parallel through grouped convolution; In layer2, dividing the 3×3 convolutional kernels with 128 channels in the residual modules of 4 Bottlenecks into 32 groups of convolutional layer groups with 8 channels and a convolutional kernel size of 3×3 stacked in parallel through grouped convolution; In layer3, dividing the 3×3 convolutional kernels with 256 channels in the residual modules of 6 Bottlenecks into 32 groups of convolutional layer groups with 16 channels and a convolutional kernel size of 3×3 stacked in parallel through grouped convolution; In layer4, dividing the 3×3 convolutional kernels with 512 channels in the residual modules of 3 Bottlenecks into 32 groups of convolutional layer groups with 32 channels and a convolutional kernel size of 3×3 stacked in parallel through grouped convolution; The respectively outputting three groups of first weighted feature maps and second weighted feature maps from three convolutional blocks of layer2, layer3, and layer4 of the G-ResNet network includes: Through the dual multi-scale attention module, respectively extracting the first feature maps and second feature maps output by the first Bottleneck in layer2, layer3, and layer4 of the template branch and the search branch; Respectively grouping the first feature maps and the second feature maps to obtain multiple grouped feature maps corresponding to the first feature maps and the second feature maps; Decomposing each grouped feature map into a first sub-feature map and a second sub-feature map; Using the position attention module and the channel attention module to process the first sub-feature map and the second sub-feature map respectively, and obtaining a sub-feature map with position attention response and a third sub-feature map and a fourth sub-feature map with channel attention response respectively; Channel fuse the third sub-feature map and the fourth sub-feature map to obtain a fifth sub-feature map corresponding to a grouped feature map; Obtain multiple fifth sub-feature maps corresponding to multiple grouped feature maps; Shuffle the multiple fifth sub-feature maps to obtain a weighted feature map output by the template branch and the first Bottleneck of the search branch of the first Bottleneck; Propagate the weighted feature map output by the template branch and the first Bottleneck of the search branch backward in turn, and output a first weighted feature map and a second weighted feature map from the last Bottleneck of layer2, layer3, and layer4 respectively; The weighted fusion of three groups of first weighted feature maps and second weighted feature maps by using multiple anchor-free region proposal networks includes: Set RPN modules with anchor-free strategies between the respective layer2, layer3, and layer4 convolutional blocks of the template branch and the search branch of the G-ResNet network. The RPN module with anchor-free strategy includes a classification branch and a regression branch, and the regression branch is used to predict the offset between the target pixel point and the true box; Input the first weighted feature map and the second weighted feature map into the convolutional networks in the regression branch and the classification branch of the RPN module with anchor-free strategy respectively, and output a regression map and a classification map from the regression branch, and output a regression map and a classification map from the classification branch; Perform depth cross-correlation operation on the two regression maps output by the classification branch and the regression branch to obtain a regression result; Perform depth cross-correlation operation on the two classification maps output by the classification branch and the regression branch to obtain a classification result; Obtain the position where the maximum value of the classification result is located as the predicted position of the target; Obtain the predicted bounding box corresponding to the predicted position in the regression result as the predicted box of the target.

2. The multi-scale UAV aerial photography target tracking method according to claim 1, wherein The expression of the position attention response includes: ； Among them, represents the first sub-feature map, and IN(Xk1) represents the completion of spatial information statistics , and respectively represent the weight and sigmoid non-linear activation function used to strengthen .

3. A multi-scale UAV aerial photography target tracking method according to claim 1, characterized in that, The expression of the channel attention response includes: ； ； where H and W respectively represent the height and width of the second sub-feature map, represents the second sub-feature map, Fgap represents the global average pooling function, For s perform scaling and offset operations, represents the sigmoid non-linear activation function.

4. A multi-scale UAV aerial photography target tracking device, characterized in that, Includes: An acquisition module for acquiring an aerial video of a drone; A processing module for inputting the initial frame and the current frame of the aerial video of the drone into the template branch and the search branch of the Siamese tracking network constructed based on the G-ResNet network, and outputting three groups of first weighted feature maps and second weighted feature maps from the layer2, layer3, and layer4 convolutional blocks of the G-ResNet network respectively. The G-ResNet network is obtained by replacing the 3×3 convolutional kernel of the residual module of each Bottleneck in the resnet50 network with multiple convolutional layer groups with the same topological structure stacked in parallel, and adding a dual multi-scale attention module behind each Bottleneck; An output module for using multiple anchor-free region proposal networks to perform weighted fusion on three groups of first weighted feature maps and second weighted feature maps, and tracking the target of the current frame according to the predicted box and the predicted position in the result of the weighted fusion; The replacement of the 3×3 convolutional kernel of the residual module of each Bottleneck in the resnet50 network by multiple convolutional layer groups with the same topological structure stacked in parallel includes: In layer1, through grouped convolution, the 3x3 convolutional kernels with 64 channels in the residual modules of 3 Bottlenecks are divided into 32 groups of parallelly stacked convolutional layer groups with 4 channels and a convolutional kernel size of 3x3. In layer2, through grouped convolution, the 3x3 convolutional kernels with 128 channels in the residual modules of 4 Bottlenecks are divided into 32 groups of parallelly stacked convolutional layer groups with 8 channels and a convolutional kernel size of 3x3. In layer3, through grouped convolution, the 3x3 convolutional kernels with 256 channels in the residual modules of 6 Bottlenecks are divided into 32 groups of parallelly stacked convolutional layer groups with 16 channels and a convolutional kernel size of 3x3. In layer4, through grouped convolution, the 3x3 convolutional kernels with 512 channels in the residual modules of 3 Bottlenecks are divided into 32 groups of parallelly stacked convolutional layer groups with 32 channels and a convolutional kernel size of 3x3. The three groups of first weighted feature maps and second weighted feature maps are respectively output from the three convolutional blocks of layer2, layer3, and layer4 of the G-ResNet network, including: Through the dual multi-scale attention module, the first feature maps and second feature maps output from the first Bottleneck in layer2, layer3, and layer4 of the template branch and the search branch are respectively extracted. The first feature maps and second feature maps are respectively grouped to obtain multiple grouped feature maps corresponding to the first feature maps and second feature maps. Each grouped feature map is decomposed into a first sub-feature map and a second sub-feature map. The first sub-feature map and the second sub-feature map are respectively processed by the position attention module and the channel attention module to obtain a sub-feature map with position attention response and third and fourth sub-feature maps with channel attention response. The third sub-feature map and the fourth sub-feature map are fused in channels to obtain a fifth sub-feature map corresponding to a grouped feature map. Multiple fifth sub-feature maps corresponding to multiple grouped feature maps are obtained. The multiple fifth sub-feature maps are shuffled to obtain the weighted feature maps output from the first Bottleneck of the template branch and the search branch of the first Bottleneck. The weighted feature maps output from the first Bottleneck of the template branch and the search branch are propagated backward in sequence, and the first weighted feature maps and second weighted feature maps are respectively output from the last Bottleneck of layer2, layer3, and layer4. The three groups of first weighted feature maps and second weighted feature maps are weighted and fused by using multiple anchor-free region proposal networks, including: RPN modules with anchor-free strategies are respectively set between the three convolutional blocks of layer2, layer3, and layer4 of the template branch and the search branch of the G-ResNet network. The RPN modules with anchor-free strategies include a classification branch and a regression branch, and the regression branch is used to predict the offset between the target pixel point and the true box. Input the first weighted feature map and the second weighted feature map into the convolutional networks in the regression branch and the classification branch of the anchor-free RPN module respectively, and output a regression map and a classification map from the regression branch, and output a regression map and a classification map from the classification branch; Perform depthwise cross-correlation operation on the two regression maps output by the classification branch and the regression branch to obtain a regression result; Perform depthwise cross-correlation operation on the two classification maps output by the classification branch and the regression branch to obtain a classification result; Obtain the position where the maximum value of the classification result is located as the predicted position of the target; Obtain the predicted bounding box corresponding to the predicted position in the regression result as the predicted box of the target.

5. An electronic device, characterized in that, The electronic device includes a processor and a memory. At least one instruction, at least one program, a code set or an instruction set is stored in the memory. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the multi-scale UAV aerial photography target tracking method according to any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, At least one instruction, at least one program, a code set or an instruction set is stored in the storage medium. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the multi-scale UAV aerial photography target tracking method according to any one of claims 1-3.

Images