A target tracking method for autonomous aerial refueling of UAVs based on a falcon-like visual twin network

Abstract

The present invention discloses a method for tracking a target for autonomous aerial refueling of a UAV based on a falcon-like visual twin network: 1. Multi-scale feature extraction based on the contrast sensitivity mechanism and adaptive focusing mechanism of the falcon-like vision; 2. Similarity calculation and feature fusion based on the bifoveal visual interaction mechanism of the falcon-like vision; 3. Classification and regression loss function design based on the perceptual competition and decision-making mechanism of the superior colliculus pathway of the falcon-like vision; 4. Training the falcon-like visual twin tracking network and outputting classification loss and regression loss; 5. Testing and evaluating the falcon-like visual twin tracking network. The present invention can adapt to various complex environments and changes in the size of the UAV's appearance; can efficiently fuse multi-scale spatial features and focus on the most significant areas; can maximize tracking stability and improve the success rate and safety of autonomous aerial refueling of the UAV; the training strategy is adapted to the complex autonomous aerial refueling environment of the UAV; the feature extraction and fusion are efficient, robust and generalizable, and require less training data.

Description

Technical Field

[0001] The present invention relates to a target tracking method for autonomous aerial refueling of a UAV by imitating a falcon visual twin network, and belongs to the field of computer vision. Background Art

[0002] With the widespread application of Unmanned Aerial Vehicles (UAVs) in the military field, long-term airborne and long-distance operations have become particularly important. Traditional refueling methods often require ground support or return to base, and the efficiency and flexibility of refueling are low. Autonomous Aerial Refueling (AAR) technology can make UAVs more flexible in performing missions and extend their combat time. During the close-range guidance phase of autonomous aerial refueling of UAVs, the receiving aircraft approaches the tail of the tanker at high speed. Traditional GPS navigation methods are easily affected by factors such as communication delays, enemy electromagnetic interference, and insufficient positioning accuracy in high-dynamic environments, which seriously reduces the success rate of autonomous aerial refueling and also brings great safety risks. The present invention aims to propose a visual target tracking method technology with simple principles, efficient operations, direct and flexible, to improve the success rate and safety of autonomous aerial refueling of UAVs, thereby providing support and guarantee for long-term and long-distance operations of UAVs.

[0003] Visual object tracking (VAT) is an important research area in computer vision. It involves the dynamic tracking of specific targets within video sequences. Object tracking aims to identify a specific target from the first frame of a video and accurately locate its position in each subsequent frame. Commonly used visual object tracking methods include optical flow and correlation filtering. Optical flow can quickly calculate motion information and does not rely on the specific shape or feature points of the target. It uses only brightness information for motion estimation, making it effective even when features are not obvious. However, optical flow is highly sensitive to changes in illumination and can cause tracking failure when scene lighting conditions change significantly. Correlation filtering-based object tracking uses discrete Fourier transforms to accelerate filter calculations. This results in faster execution and better adaptability to changes in the target's pose and scale. However, its tracking accuracy and stability are poor. When the target moves rapidly within the video, tracking delays or loss can occur, making it unsuitable for tracking high-speed drones during autonomous aerial refueling.

[0004] With the development of computer vision, deep learning-based Siamese network target tracking algorithms can learn effective feature representations from small amounts of data, achieving efficient feature learning and accurate anchor box prediction for targets. This provides new insights into target tracking and has garnered widespread attention. They have broad application prospects in the close-range visual guidance phase of autonomous aerial refueling for drones.

[0005] To address the challenges of visual tasks, a growing number of researchers are turning to nature for inspiration. Birds of prey, such as falcons, have highly developed visual systems, boasting exceptional visual sensitivity and exceptional wide-field perception. Even from thousands of meters in the air, they can clearly see prey on the ground. Once they locate their prey, falcons track it at high speed until they capture it. Falcons' wide field of view and sharp eyesight have garnered significant research attention in areas such as target detection, feature extraction, and tracking. Mapping the biological mechanisms underlying falcon vision into deep learning mechanisms and embedding them into a Siamese network target tracking approach can significantly improve tracking success rate and accuracy, enabling efficient, safe, and stable autonomous aerial refueling of drones.

[0006] In summary, the present invention proposes a target tracking method for autonomous aerial refueling of UAVs based on a falcon-like visual twin network, mapping the biological mechanism of falcon vision into a deep learning mechanism for autonomous aerial refueling of UAVs. The method is simple, efficient, and has good real-time performance, conforms to actual combat scenarios, and has certain reference significance. Summary of the Invention

[0007] This invention aims to provide a target tracking method for autonomous aerial refueling of unmanned aerial vehicles (UAVs) using a falcon-like twin network. This method aims to address the long-term, stable, and accurate tracking of the refueling aircraft by the receiving aircraft during close-range visual guidance during autonomous aerial refueling. By drawing on the biological mechanisms of falcon vision in nature, a target tracking mechanism based on falcon vision is established and embedded into the twin network target tracking method, providing a new solution to the problem of UAV swarm confrontation.

[0008] This paper proposes a UAV autonomous aerial refueling target tracking method that imitates the falcon visual twin network. The specific implementation steps are as follows:

[0009] Step 1: Multi-scale feature extraction of simulating the visual contrast sensitivity mechanism and adaptive focusing mechanism of the falcon The multi-scale feature extraction mechanism of the simulating visual contrast sensitivity mechanism and adaptive focusing mechanism proposed in this invention is as shown in the attached process. Figure 1 shown.

[0010] S11. Obtaining primary features of the input image through convolutional neural network

[0011] Use convolutional neural network (ResNet-50) to obtain the primary features of input template image z and search image x, and output the primary feature map and

[0012] S12. Establish a model of the visual contrast sensitivity mechanism of falcons

[0013] The falcon-like visual contrast sensitivity model first performs maximum pooling and average pooling on the channel dimension of the input feature map to emphasize the key features unique to high-density photoreceptors. Then, the feature maps after maximum pooling and average pooling are added element by element. Then, a large convolution kernel is used to Convolution operation is performed with the added feature map To enhance the local contextual relationship between features, we simulate the high-density sampling of the fovea of ​​the falcon's eye. Subsequently, we apply the activation function Sigmoid to calculate the attention weights. These weights adjust the model's sensitivity to specific features based on the contrast sensitivity function of the falcon's vision. Finally, these weights are obtained through the matrix Schur product. The input is integrated to achieve high contrast between the target and the distractors, ensuring that important features are enhanced while unimportant features are suppressed, thereby improving the model's performance in drone tracking tasks. The falcon-like visual contrast sensitivity mechanism model is expressed as follows:

[0014]

[0015] in, is the input feature map, This is the feature map output by the falcon-like visual contrast sensitivity mechanism model. They represent the maximum pooling and average pooling operators respectively, and Sigmoid is the activation function.

[0016] S13. Establishing a model of adaptive focusing mechanism of falcon vision

[0017] The distribution and structure of cones in the falcon's retina enhance the falcon's high visual resolution, while Adaptive Max Pooling similarly extracts important features to improve focus. Furthermore, the falcon's fovea achieves the highest spatial resolution under high-contrast conditions, while Adaptive Average Pooling facilitates feature aggregation and recovery across a wide range of scenarios. The falcon-like adaptive focusing mechanism model first applies Adaptive Max Pooling and Adaptive Average Pooling to the spatial dimensions to achieve precise focus on the target. Next, to mimic the falcon's ability to optimize imaging by changing the shape of the lens and ciliary muscle, a point convolution kernel is used to compress and reconstruct channel information. Specifically, the channels of the adaptively pooled feature map are first convolved with a point convolution kernel for dimensionality reduction, facilitating information extraction. An activation function is then applied to enable the network to capture more complex feature relationships. A second point convolution is then used to restore the activated feature map to its original number of channels, ensuring that the network retains important information while restoring input features. Finally, the two feature maps obtained after the point convolution are element-wise added, also using a matrix Schur product. The adaptive focusing mechanism model of the falcon-like vision is expressed as follows:

[0018]

[0019] in, This is the feature map output by the falcon-like vision adaptive focusing mechanism model. They are adaptive maximum pooling and average pooling operators respectively. They represent the point convolution used in compression and reconstruction operations respectively, and Sigmoid and ReLu are both activation functions.

[0020] S14. Obtaining multi-scale features of the image

[0021] Template feature map extracted through multi-scale features and search feature graph Respectively expressed as:

[0022]

[0023] in, and They are respectively the template feature map and search feature map output by the falcon-like visual contrast sensitivity mechanism model. and They are respectively the template feature map and search feature map output by the falcon-like vision adaptive focusing mechanism model.

[0024] Step 2: Similarity calculation and feature fusion of the bifoveal visual interaction mechanism of hawk-like vision

[0025] S21. Establish a bifoveal switching mechanism model for falcon-like vision

[0026] The similarity calculation and feature fusion mechanism process of the falcon-like bifoveal vision interaction mechanism proposed in this invention is shown in the attached figure. Figure 2 shown.

[0027] Similarity calculation and feature fusion models typically require pairing queries and keys. In this network, the query represents the feature map extracted by the fovea, which captures the detailed features of the object, while the key features come from the feature map extracted by the lateral fovea, which contains the object's location and contextual information about the environment.

[0028] Simulating the bifoveal switching mechanism of falcon vision, the similarity matrix s is expressed as:

[0029]

[0030] in, express and The inner product of the search image and the template image is nc.

[0031] S22. Establish a multi-scale spatial attention mechanism model imitating falcon vision

[0032] Since falcon vision can switch freely between two types of foveal imaging, that is, it can obtain target information from different subspaces and different positions, a multi-scale spatial attention mechanism is introduced based on the dual foveal switching mechanism. By performing h linear projections on the query and key and using different linear projection matrices to promote learning, the final similarity feature map It is obtained by concatenating multi-scale features along the number of channels, which is given by the following formula.

[0033]

[0034] Among them, a certain scale feature scale i Expressed as

[0035]

[0036] matrix and is the linear projection parameter used to combine all scales, and the Concat function represents the splicing operation along the channel dimension.

[0037] S23. Generate classification branch and regression branch

[0038] The similarity feature map calculated by the falcon-like bifoveal visual interaction mechanism is input into the head of the Siamese network, generating a classification branch and a regression branch, as shown below:

[0039]

[0040] Among them, W cls and W reg are the weight matrices for classification and regression, respectively, b cls and b reg They are bias terms for classification and regression respectively, and the Softmax function converts the input into a probability value.

[0041] Step 3: Simulate the perception competition of the superior colliculus pathway in the falcon, classify the decision-making mechanism, and design the regression loss function

[0042] S31. Establish a model of perception competition and decision-making mechanism in the superior colliculus pathway of the falcon

[0043] Falcon vision is able to quickly analyze complex hunting scenes with limited resources, that is, to locate the area of ​​interest in the highly sensitive fovea of ​​the retina to ensure that the visual system is focused on the target prey area. Studies have shown that the ability of falcon vision to efficiently filter out unimportant perceptions is attributed to the competition and decision-making mechanisms in the superior colliculus pathway of the midbrain. Studies have pointed out that when multiple visual perceptions appear in different locations of the retina, competition will arise between neural groups, and only the winning group will be selected and transmitted to higher visual centers for further processing, and guide the falcon to make perceptual decisions about multiple targets appearing on the retina. This mechanism enables the falcon to accurately filter out interference and locate prey in complex backgrounds, thereby achieving stable tracking until capture. Applying the perceptual competition and decision-making mechanism in the superior colliculus pathway of the falcon's vision to the classification loss and regression loss of the twin network can effectively enhance the anti-interference ability of the twin network, significantly improve the tracking accuracy and stability, and further improve the success rate and safety of autonomous aerial refueling of drones. The design process of the perceptual competition, decision-making mechanism of the superior colliculus pathway of the falcon is shown in the attached figure. Figure 3 shown.

[0044] S32. Classification loss function design

[0045] Let the positive sample set and the negative sample set after filtering be expressed as:

[0046] set + ={c +1 ,c +2 ,...,c +n}

[0047] set - ={c -1 ,c -2,...,c -m}

[0048] c -j (j=1,2,..,m)≥c -th

[0049] Among them, c represents the confidence score of the target, c -th is the filtering threshold, n and m are the number of samples in the positive sample set and the negative sample set respectively, c -j (j=1,2,..,m) is the overall confidence score in the negative sample set.

[0050] For the superior colliculus pathway in the midbrain of falcons, the perceptual salience S of the positive and negative sample sets on the retina is + and S - They are defined as:

[0051]

[0052] Among them, w +i and w -j is the weight of the confidence score, c +i and c -j Represent the overall confidence scores of the positive sample set and the negative sample set, respectively.

[0053] In order to retain the positive sample perception S + At the same time, suppress negative sample perception S - , weight parameter w +i and w -j Defined as:

[0054]

[0055] w -j =softmax(c -j )

[0056] Among them, the weight of the confidence score of the positive sample set ensures that the weight distribution among the positive samples is uniform.

[0057] Classification loss function based on the competition and decision-making mechanism of hawk-like visual perception Defined as:

[0058]

[0059] Here, q is the weight of positive samples, simulating the way a falcon extracts salient objects on the retina, effectively increasing the model's focus on positive samples. The parameter b controls the loss value, and a is the margin, or the distance between positive and negative samples.

[0060] S33. Regression loss function design

[0061] By simulating the perceptual competition and decision-making mechanism of the superior colliculus pathway in the falcon, we can coordinate the confidence score output by the classification branch and the bounding box output by the regression branch, aligning the classification and regression branches. Specifically, we simulate the perceptual competition and decision-making mechanism of the superior colliculus pathway in the falcon, focusing on regions of interest (positive samples) and suppressing regions of no interest (negative samples), and adopt the following strategy to design the regression loss function:

[0062]

[0063] That is to ensure that high confidence samples have accurate regression boxes, and at the same time ensure that accurate regression samples have high confidence. +i and c -j Also represents the overall confidence score in the positive sample set and the negative sample set, r +i and r -j Represent the anchor box accuracy of the positive sample set and negative sample set output by the regression branch respectively. The symbol indicates that both sides of the equation are necessary and sufficient for each other.

[0064] This paper uses the distance-penalized intersection over union (DIoU) as an indicator to measure the accuracy r of the regression bounding box. The intersection over union (IoU) calculates the ratio of the intersection and union of the predicted bounding box and the true bounding box, while the distance-penalized IoU introduces a penalty term for the distance between the predicted box and the true box on the basis of IoU, which can better handle the overlap problem of the target bounding box, thereby improving the target tracking performance, especially when the distance between the targets is close. The calculation method of the distance-penalized IoU is as follows:

[0065]

[0066] Among them, P bbox To predict the bounding box, Gt bbox is the true bounding box, E represents the Euclidean distance between the center coordinates of the target box and the center coordinates of the true box, and K represents the diagonal length of the minimum rectangle that can contain the two bounding boxes.

[0067] The regression loss function of the superior colliculus pathway perception competition and decision-making mechanism of the falcon is designed as follows:

[0068]

[0069] in, For contrast loss, the distance between positive samples is reduced while the distance between negative samples is increased, which is defined as:

[0070]

[0071] Among them, N + and N - are the number of positive sample sets and negative sample sets respectively, and p>0 controls the size of the loss function. 2 (i, j) is the minimum distance between samples, y ij is the sample similarity label (1 for similar, 0 for dissimilar), and v-margin defines the minimum distance between dissimilar samples.

[0072] The flowchart of the target tracking method for autonomous aerial refueling of UAV based on hawk-like visual twin network proposed in this invention is as follows: Figure 4 shown.

[0073] Step 4: Train the falcon-like visual twin tracking network and output classification loss and regression loss

[0074] S41. Configure the twin network operating environment

[0075] First, use Anaconda 3 to configure the Python environment. In the Conda Prompt command line window, create a new environment and install the pytorch, torchvision, torchaudio, cuda, opencv, pyyaml, yacs, tqdm, colorama, matplotlib, cython, and tensorboard packages.

[0076] Next, run the setup.py script to build the extension module using the Cython compiler in the Python environment to improve performance or use specific system resources.

[0077] S42. Deploy and initialize the twin network

[0078] First, configure the twin network training hyperparameters, including: the total number of training rounds, the learning rate for different training rounds, the weights of the classification loss function and the regression loss function, the batch size, etc.

[0079] To speed up training and improve model performance, you can usually load pre-trained weights on a large dataset. Store the pre-trained weights in the .model folder and use the model.load_state_dict() function to load the pre-trained weights.

[0080] S43. Deploy the twin network training dataset

[0081] The falcon-like visual twin network tracking method proposed in this invention uses COCO, VID, DET and YoutubeBB as training data sets. Before training, the images in the above data sets need to be cropped to adapt to the input image size required by the twin network convolutional layer.

[0082] S44. Training a falcon-like visual twin tracking network

[0083] During training, the loss of the Siamese network is the weighted sum of the classification loss and the regression loss:

[0084]

[0085] in, is the total loss of the twin network, and are the classification loss function and regression loss function that imitate the visual perception competition and decision-making mechanism of falcons, respectively. l1 and l2 are the weights of the classification loss function and regression loss function, respectively.

[0086] An appropriate weighting strategy ensures that the network achieves good performance in both classification and localization tasks, preventing one task from dominating the training process and improving the generalization of the Siamese network across a variety of UAV autonomous aerial refueling environments. Minimizing the classification loss ensures that the network can accurately identify the presence of a target, while minimizing the regression loss ensures that the network can accurately predict the target's location. When these two loss functions are ideally balanced, the network can not only accurately distinguish between targets and non-targets, but also efficiently locate the target's bounding box, thereby achieving precise target tracking and improving the success rate and safety of autonomous UAV aerial refueling.

[0087] In the training of the twin network, the AdaDelta adaptive optimization training method is used to improve the overcorrection problem in gradient descent by dynamically adjusting the learning rate of each parameter. The AdaDelta adaptive optimization method is updated based on the square of the historical gradient. It does not need to store the entire sum of squared gradients, but instead uses a sliding average to represent it. First, solve the cumulative squared gradient

[0088]

[0089] Among them, t represents the number of iterations, that is, the time step, Represents the loss function in t iterations The neural network weights θ t The gradient of the map, M represents the exponential decay average operation on the map square, and the hyperparameter β represents the decay coefficient of the gradient square, which controls the degree of historical memory of the gradient square.

[0090] Then, during the training process, the weights are updated as follows

[0091]

[0092] Where Δθ t+1 represents the update amount of the neural network weight in the t+1 iteration, represents the sliding average of the square of the update amount in the tth iteration, which is a small constant used to prevent division by zero errors. Represents the loss function in t iterations The neural network weights θ t The gradient of , M also represents the exponential decay average operation on the square of the map.

[0093] Step 5: Test and evaluate the falcon-like visual twin tracking network in real-world scenarios

[0094] During testing, the falcon-like visual twin tracking network is only provided with the ground-truth bounding box of the target in the first frame of the video sequence, and the twin network performs inference and prediction in subsequent frames of the video sequence to track the target.

[0095] S51. Output the relative offset of the predicted target bounding box coordinates relative to the initial target bounding box coordinates

[0096] During the progress of the video sequence, the fully trained falcon-like visual twin tracking network predicts the offset of the target's bounding box coordinates in the next frame relative to the initial bounding box coordinates through the aforementioned feature extraction, similarity calculation and feature fusion.

[0097] S52, calculate and output the predicted target bounding box coordinates, and use them for target object tracking in the next frame

[0098] The offset of the target bounding box coordinates in the next frame relative to the initial bounding box coordinates is added to the target bounding box in the previous frame to calculate the predicted target bounding box coordinates for target object tracking in the next frame, and steps S51 and S52 are repeated until the video sequence ends.

[0099] S53, testing the falcon-like twin tracking network in the UAV123 dataset and real drone autonomous aerial refueling scenarios

[0100] To validate the model's performance, we tested it in a real-world scenario using the UAV123 dataset. We first cleaned and formatted the dataset, including image normalization and label correction, to ensure data quality and consistency. We then loaded the test dataset into the testing framework, ensuring that each test sample included the corresponding target drone's location annotation. We then used various metrics to evaluate the tracking performance of the Siamese network.

[0101] The flowchart of the target tracking method for autonomous aerial refueling of UAV based on hawk-like visual twin network proposed in this invention is as follows: Figure 4 shown. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] Figure 1 Flowchart of the multi-scale feature extraction mechanism that imitates the contrast-sensitive mechanism and adaptive focusing mechanism of falcon vision.

[0103] Figure 2 This is a flowchart of the similarity calculation and feature fusion mechanism of the bifoveal visual interaction mechanism that imitates hawk vision.

[0104] Figure 3 Design a flowchart for the stimulus competition, classification of selection mechanism, and regression loss function of the superior colliculus pathway in the falcon.

[0105] Figure 4 Flowchart of the target tracking method for autonomous aerial refueling of UAVs using a falcon-like visual twin network.

[0106] Figure 5a Figure 5b The test results of the target tracking method for autonomous aerial refueling of UAVs using a falcon-like visual twin network.

[0107] Figure 6 Real-scene visualization results of the target tracking method for autonomous aerial refueling of UAVs using a falcon-like visual twin network. DETAILED DESCRIPTION

[0108] The following example demonstrates the effectiveness of the proposed method for autonomous aerial refueling target tracking for drones based on a falcon-like visual twin network. The simulation environment for this example is configured with Windows 11, an AMD Ryzen 77800H processor with a 3.80 GHz clock speed, 16 GB of memory, an NVIDIA RTX4060 GPU, and Pycharm 2024.1 Professional. The specific steps for this example are as follows:

[0109] Step 1: Multi-scale feature extraction imitating the falcon’s visual contrast sensitivity mechanism and adaptive focusing mechanism

[0110] S11. Obtaining primary features of the input image through convolutional neural network

[0111] Use convolutional neural network to obtain the primary features of input template image z and search image x, and output the primary feature map and

[0112] The present invention uses Resnet-50 as the neural network for extracting primary features. The core idea of ​​ResNet is to introduce skip connections, that is, in each convolution operation, in addition to the output of the current layer, the input is directly added to the output to form a residual map. This method allows the gradient to be transmitted more efficiently during back propagation, thereby alleviating the gradient vanishing problem in deep neural network training. S12. Establishing a model of the visual contrast sensitivity mechanism of hawks

[0113] The falcon-like visual contrast sensitivity mechanism model is expressed as follows:

[0114]

[0115] in, is the input feature map, This is the feature map output by the falcon-like visual contrast sensitivity mechanism model. They represent the maximum pooling and average pooling operators respectively, and Sigmoid is the activation function.

[0116] S13. Establishing a model of adaptive focusing mechanism of falcon vision

[0117] The falcon-like vision adaptive focusing mechanism model is expressed as follows:

[0118]

[0119] in, This is the feature map output by the falcon-like vision adaptive focusing mechanism model. They are adaptive maximum pooling and average pooling operators respectively. They represent the point convolution used in compression and reconstruction operations respectively, and Sigmoid and ReLu are both activation functions.

[0120] S14. Obtaining multi-scale features of the image

[0121] Template feature map extracted through multi-scale features and search feature graph Respectively expressed as:

[0122]

[0123] in, and They are respectively the template feature map and search feature map output by the falcon-like visual contrast sensitivity mechanism model. and They are respectively the template feature map and search feature map output by the falcon-like vision adaptive focusing mechanism model.

[0124] Step 2: Similarity calculation and feature fusion of the bifoveal visual interaction mechanism of hawk-like vision

[0125] S21. Establish a bifoveal switching mechanism model for falcon-like vision

[0126] Simulating the bifoveal switching mechanism of falcon vision, the similarity matrix s is expressed as:

[0127]

[0128] in, express and nc is the inner product of the search image and the template image, and the Softmax function converts the input into a probability value.

[0129] S22. Establish a multi-scale spatial attention mechanism model imitating falcon vision

[0130] By performing h linear projections on the query and key and using different linear projection matrices to facilitate learning, the final similarity feature map It is obtained by concatenating multi-scale features along the number of channels, which is given by the following formula.

[0131]

[0132] Among them, a certain scale feature scale i Expressed as

[0133]

[0134] matrix and is the linear projection parameter used to combine all scales, and the Concat function represents the splicing operation along the channel dimension.

[0135] S23. Generate classification branch and regression branch

[0136] The similarity feature map calculated by the falcon-like bifoveal visual interaction mechanism is input into the head of the Siamese network, generating a classification branch and a regression branch, as shown below:

[0137]

[0138] Among them, W cls and W reg are the weight matrices for classification and regression, respectively, b cls and b reg They are bias terms for classification and regression respectively, and the Softmax function converts the input into a probability value.

[0139] Step 3: Simulate the perception competition of the superior colliculus pathway in the falcon, classify the decision-making mechanism, and design the regression loss function

[0140] S31. Classification loss function design

[0141] Let the positive sample set and the negative sample set after filtering be expressed as:

[0142] set + ={c +1 ,c +2 ,...,c +n}

[0143] set - ={c -1 ,c -2 ,...,c -m}

[0144] c -j (j=1,2,..,m)≥c -th

[0145] Among them, c represents the confidence score of the target, c -th is the filtering threshold, n and m are the number of samples in the positive sample set and the negative sample set respectively, c -j (j=1,2,..,m) is the overall confidence score in the negative sample set.

[0146] For the superior colliculus pathway in the midbrain of falcons, the perceptual salience S of the positive and negative sample sets on the retina is + and S - They are defined as:

[0147]

[0148] Among them, w +i and w -j is the weight of the confidence score, c +i and c -j Represent the overall confidence scores of the positive sample set and the negative sample set, respectively.

[0149] In order to retain the positive sample perception S + At the same time, suppress negative sample perception S - , weight parameter w +i and w -j Defined as:

[0150]

[0151] w -j =softmax(c -j )

[0152] Among them, the weight of the confidence score of the positive sample set ensures that the weight distribution among the positive samples is uniform.

[0153] Classification loss function based on the competition and decision-making mechanism of hawk-like visual perception Defined as:

[0154]

[0155] Here, q is the weight of positive samples, simulating the way a falcon extracts salient objects on the retina, effectively increasing the model's focus on positive samples. Parameter b controls the loss value and is set to 4. And a is the margin, or the distance between positive and negative samples, and is set to 0.5. Inspired by the perceptual competition mechanism, increasing the value of q causes the model to pay more attention to misclassification of positive samples during training, thereby improving the recall of positive samples.

[0156] S32. Regression loss function design

[0157] The following strategy is used to design the regression loss function:

[0158]

[0159] That is to ensure that high confidence samples have accurate regression boxes, and at the same time ensure that accurate regression samples have high confidence. +i and c -j Also represents the overall confidence score in the positive sample set and the negative sample set, r +i and r -j Represent the anchor box accuracy of the positive sample set and negative sample set output by the regression branch respectively. The symbol indicates that both sides of the equation are necessary and sufficient for each other.

[0160] This paper uses the distance penalty intersection over union (DIoU) as an indicator to measure the regression bounding box accuracy r:

[0161]

[0162] Among them, P bbox To predict the bounding box, Gt bbox is the true bounding box, E represents the Euclidean distance between the center coordinates of the target box and the center coordinates of the true box, and K represents the diagonal length of the minimum rectangle that can contain the two bounding boxes.

[0163] The regression loss function of the superior colliculus pathway perception competition and decision-making mechanism of the falcon is designed as follows:

[0164]

[0165] in, For contrast loss, the distance between positive samples is reduced while the distance between negative samples is increased, which is defined as:

[0166]

[0167] Among them, N + and N - are the number of positive sample sets and negative sample sets respectively, and p>0 controls the size of the loss function. 2 (i, j) is the minimum distance between samples, y ij is the sample similarity label (1 for similar, 0 for dissimilar), and v-margin defines the minimum distance between dissimilar samples, which is set to 0.5.

[0168] Step 4: Train the falcon-like visual twin tracking network and output classification loss and regression loss

[0169] S41. Configure the twin network operating environment

[0170] First, use Anaconda 3 to configure the Python environment. In the Conda Prompt command line window, create a new environment and install the pytorch, torchvision, torchaudio, cuda, opencv, pyyaml, yacs, tqdm, colorama, matplotlib, cython, and tensorboard packages.

[0171] Next, run the setup.py script to build the extension module using the Cython compiler in the Python environment to improve performance or use specific system resources.

[0172] S42. Deploy and initialize the twin network

[0173] First, configure the twin network training hyperparameters. Set the total number of training rounds to 20, and the batch size to 128. For the first 10 training rounds, train only the classification and regression branches, freezing the backbone network. This means only the weights of these two branches are updated. For the next 10 training rounds, unfreeze the last three convolutional layers of the ResNet-50 backbone network and update the weights.

[0174] To speed up training and improve model performance, you can usually load pre-trained weights on a large dataset. Store the pre-trained weights in the .model folder and use the model.load_state_dict() function to load the pre-trained weights.

[0175] S43. Deploy the twin network training dataset

[0176] The falcon-like visual twin network tracking method proposed in this invention uses COCO, VID, DET and YoutubeBB as training data sets. Before training, the images in the above data sets need to be cropped to adapt to the input image size required by the twin network convolutional layer.

[0177] S44. Training a falcon-like visual twin tracking network

[0178] During training, the loss of the Siamese network is the weighted sum of the classification loss and the regression loss

[0179]

[0180] in, is the total loss of the twin network, and are respectively the classification loss function and regression loss function of the visual perception competition and decision-making mechanism of the hawk, l1 and l2 are the weights of the classification loss function and the regression loss function, which are taken as 1.1 and 0.8 respectively in the present invention.

[0181] In the training of the twin network, the AdaDelta adaptive optimization training method is used. First, the squared cumulative gradient is solved.

[0182]

[0183] Among them, t represents the number of iterations, that is, the time step, Represents the loss function in t iterations The neural network weights θ t The gradient of the map is expressed as follows: M represents the exponential decay averaging operation on the map square, and the hyperparameter β represents the decay coefficient of the gradient square, which controls the historical memory degree of the gradient square. The present invention takes it as 0.3.

[0184] Then, during the training process, the weights are updated as follows:

[0185]

[0186] Where Δθ t+1 represents the update amount of the neural network weight in the t+1 iteration, It represents the sliding average of the square of the update amount in the tth iteration, which is a small constant used to prevent division by zero errors. In this paper, it is taken as 10 -8 , Represents the loss function in t iterations The neural network weights θ t The gradient of , M also represents the exponential decay average operation on the square of the map.

[0187] Step 5: Test and evaluate the falcon-like visual twin tracking network in real-world scenarios

[0188] During testing, the falcon-like visual twin tracking network is only provided with the ground-truth bounding box of the target in the first frame of the video sequence, and the twin network performs inference and prediction in subsequent frames of the video sequence to track the target.

[0189] S51. Output the relative offset of the predicted target bounding box coordinates relative to the initial target bounding box coordinates

[0190] During the progress of the video sequence, the fully trained falcon-like visual twin tracking network predicts the offset of the target's bounding box coordinates in the next frame relative to the initial bounding box coordinates through the aforementioned feature extraction, similarity calculation and feature fusion.

[0191] S52, calculate and output the predicted target bounding box coordinates, and use them for target object tracking in the next frame

[0192] The offset of the target bounding box coordinates in the next frame relative to the initial bounding box coordinates is added to the target bounding box in the previous frame to calculate the predicted target bounding box coordinates for target object tracking in the next frame, and steps S51 and S52 are repeated until the video sequence ends.

[0193] S53, testing the falcon-like twin tracking network in the UAV123 dataset and real drone autonomous aerial refueling scenarios

[0194] In order to verify the performance of the model, the present invention is tested in the UAV123 dataset, that is, in a real scene. First, the dataset is cleaned and formatted, including image normalization, label correction, etc., to ensure the quality and consistency of the data. Then the test dataset is loaded into the test framework, and it is ensured that each test sample contains the corresponding target drone position annotation, and the tracking performance of the twin network is evaluated using different indicators. The test results of the drone autonomous aerial refueling target tracking method based on the falcon-like visual twin network are as follows: Figure 5a and Figure 5b shown.

[0195] The visualization results of the UAV autonomous aerial refueling target tracking method based on the falcon-like visual twin network in real scenes are as follows: Figure 6 As shown in the figure, it can be seen that the twin network target tracking method for autonomous aerial refueling of UAVs imitating falcon vision proposed in the present invention has a high tracking success rate and accuracy.

Claims

1. A method for tracking targets for autonomous aerial refueling of unmanned aerial vehicles (UAVs) using a falcon-like visual twin network, characterized by: The method comprises the following steps: Step 1: Multi-scale feature extraction imitating the falcon’s visual contrast sensitivity mechanism and adaptive focusing mechanism; The falcon-like visual contrast sensitivity mechanism emphasizes key features through maximum pooling and average pooling, and uses large convolution kernels to enhance local contextual relationships between features. It also integrates attention weights by calculating attention weights and performing matrix Schur product integration. The falcon-like visual adaptive focusing mechanism achieves precise focus on the target through adaptive maximum pooling and adaptive average pooling, while using point convolution kernels to compress and reconstruct channel information to optimize feature extraction. After activation function processing, the model integrates feature maps through element-by-element addition and matrix Schur product integration to retain important information and recover input features. After initially extracting input image features through a convolutional neural network, the two mechanisms are used in tandem to improve feature extraction performance in drone tracking tasks; Step 2: Similarity calculation and feature fusion based on the bifoveal visual interaction mechanism of hawk-like vision; Similarity calculation and feature fusion of bifoveal visual interaction mechanism to pair query and key; The query represents the feature map extracted by the fovea to obtain the detailed features of the target, while the key features come from the feature map extracted by the lateral fovea, which contains the location of the target and the contextual information of the environment; Step 3: Simulate the perception competition of the superior colliculus pathway in the falcon, classify the decision-making mechanism, and design the regression loss function; By imitating the perception competition and decision-making mechanism of the superior colliculus pathway in the falcon, we designed classification and regression loss functions in the twin target tracking network to ensure that positive and negative samples are correctly classified, while also ensuring that high-confidence samples have accurate regression boxes and that accurate regression samples have high classification confidence. Step 4: Train the falcon-like visual twin tracking network and output classification loss and regression loss The loss function of the falcon-like visual twin tracking network is a weighted sum of classification and regression losses. This ensures the network's performance in both classification and localization tasks, prevents one task from dominating the training process, and improves the network's generalization to various UAV autonomous aerial refueling environments. The use of adaptive optimization training methods to improve overcorrection in gradient descent; Step 5: Test and evaluate the falcon-like visual twin tracking network in real-world scenarios During testing, the falcon-like visual twin tracking network is only provided with the ground-truth bounding box of the target in the first frame of the video sequence, and the twin network performs inference and prediction in subsequent frames of the video sequence to track the target.

2. The method according to claim 1, wherein: The specific process of step one is as follows: S11. Obtain primary features of the input image through convolutional neural network: S12. Establish a falcon-like visual contrast sensitivity mechanism model: The falcon-like visual contrast sensitivity mechanism model is expressed as follows: in, is the input feature map, This is the feature map output by the falcon-like visual contrast sensitivity mechanism model. Represent the maximum pooling and average pooling operators respectively, and Sigmoid is the activation function; S13. Establish a falcon-like vision adaptive focusing mechanism model: The falcon-like vision adaptive focusing mechanism model is expressed as follows: in, This is the feature map output by the falcon-like vision adaptive focusing mechanism model. They are adaptive maximum pooling and average pooling operators respectively. Represent the point convolution used in compression and reconstruction operations respectively. Sigmoid and ReLu are both activation functions. S14. Obtaining multi-scale features of the image: Template feature map extracted through multi-scale features and search feature graph Respectively expressed as: in, and They are respectively the template feature map and search feature map output by the falcon-like visual contrast sensitivity mechanism model. and They are respectively the template feature map and search feature map output by the falcon-like vision adaptive focusing mechanism model.

3. The method according to claim 1, wherein: The specific process of step 2 is as follows: S21. Establish a bifoveal switching mechanism model simulating falcon vision: Simulating the bifoveal switching mechanism of falcon vision, the similarity matrix s is expressed as: in, express and The inner product of , nc is the number of channels of the search image and the template image; S22. Establish a multi-scale spatial attention mechanism model imitating falcon vision: The similarity feature map is obtained by concatenating multi-scale features along the number of channels, and is given by the following formula: Among them, a certain scale feature scale i Expressed as: matrix and It is the linear projection parameter used to combine all scales, and the Concat function represents the splicing operation along the channel dimension; S23. Generate classification branch and regression branch: The similarity feature map calculated by the falcon-like bifoveal visual interaction mechanism is input into the head of the Siamese network, generating a classification branch and a regression branch, as shown below: Among them, W cls and W reg are the weight matrices for classification and regression, respectively, b cls and b reg They are bias terms for classification and regression respectively, and the Softmax function converts the input into a probability value.

4. The method according to claim 1, wherein: The specific process of step three is as follows: S31. Establish a model of the perception competition and decision-making mechanism of the superior colliculus pathway in the falcon: S32. Classification loss function design: Let the positive sample set and the negative sample set after filtering be expressed as: set + ={c +1 ,c +2 ,...,c +n } set - ={c -1 ,c -2 ,...,c -m } c -j (j=1,2,..,m)≥c -th Among them, c represents the confidence score of the target, c -th is the filtering threshold, n and m are the number of samples in the positive sample set and the negative sample set respectively, c -j (j=1,2,..,m) is the overall confidence score in the negative sample set; For the superior colliculus pathway in the midbrain of falcons, the perceptual salience S of the positive and negative sample sets on the retina is + and S - They are defined as: Among them, w +i and w -j is the weight of the confidence score, c +i and c -j Represent the overall confidence scores of the positive sample set and the negative sample set respectively; In order to retain the positive sample perception S + At the same time, suppress negative sample perception S - , weight parameter w +i and w -j Defined as: w -j =softmax(c -j ) Among them, the weight of the confidence score of the positive sample set ensures that the weight distribution among the positive samples is uniform; Classification loss function based on the competition and decision-making mechanism of hawk-like visual perception Defined as: Where q is the weight of the positive sample, simulating the extraction of salient targets on the retina by a falcon, effectively enhancing the model's attention to positive samples; parameter b is used to control the loss value; and a is the margin, that is, the distance between positive and negative samples; S33. Regression loss function design: The following strategy is used to design the regression loss function: Among them, c +i and c -j Also represents the overall confidence score in the positive sample set and the negative sample set, r +i and r -j Represents the anchor box accuracy of the positive sample set and negative sample set output by the regression branch respectively; The symbol indicates that both sides of the equation are necessary and sufficient for each other.

5. The method according to claim 4, characterized in that: The distance penalty intersection-over-union ratio is used as an indicator to measure the regression bounding box accuracy r; the distance penalty intersection-over-union ratio is calculated as follows: Among them, P bbox To predict the bounding box, Gt bbox is the true bounding box, E represents the Euclidean distance between the center coordinates of the target box and the center coordinates of the true box, and K represents the diagonal length of the minimum rectangle that can contain the two bounding boxes; The regression loss function of the superior colliculus pathway perception competition and decision-making mechanism of the falcon is designed as follows: in, For contrast loss, the distance between positive samples is reduced while the distance between negative samples is increased, which is defined as: Among them, N + and N - are the number of positive sample sets and negative sample sets respectively, p>0 controls the size of the loss function; E 2 (i,j) is the minimum distance between samples, y ij is the sample similarity label, 1 is similar, 0 is dissimilar, and the v margin defines the minimum distance between dissimilar samples.

6. The method according to claim 1, wherein: The specific process of step 4 is as follows: S41. Configure the twin network operating environment: S42. Deploy and initialize the twin network: First, configure the twin network training hyperparameters, including the total number of training rounds, the learning rate for different training rounds, the weights of the classification loss function and the regression loss function, and the batch size. Then, in order to speed up training and improve model performance, load the weights pre-trained on a large dataset; store the pre-trained weights in the .model folder, and the program loads the pre-trained weights through the model.load_state_dict() function; S43. Deploy the twin network training dataset: Before this training, the images in the above dataset need to be cropped to adapt to the input image size required by the convolutional layer of the twin network; S44. Training a falcon-like visual twin tracking network: During training, the loss of the Siamese network is the weighted sum of the classification loss and the regression loss: in, is the total loss of the twin network, and are the classification loss function and regression loss function that imitate the visual perception competition and decision-making mechanism of falcons, respectively. l1 and l2 are the weights of the classification loss function and regression loss function, respectively.

7. The method according to claim 6, characterized in that: The step 4 further comprises: In the training of the twin network, the AdaDelta adaptive optimization training method is used to improve the over-correction problem in gradient descent by dynamically adjusting the learning rate of each parameter; first solve the gradient square accumulation: Among them, t represents the number of iterations, that is, the time step, Represents the loss function in t iterations For the neural network weight θ t The gradient of , M represents the exponential decay average operation on the map square, and the hyperparameter β represents the decay coefficient of the gradient square, which controls the degree of historical memory of the gradient square; Subsequently, during training, weight updates are performed as follows: Where Δθ t+1 represents the update amount of the neural network weight in the t+1 iteration, Q[Δθ t 2 ] represents the sliding average of the square of the update amount in the tth iteration, which is a small constant used to prevent division by zero errors. Represents the loss function in t iterations For the neural network weight θ t The gradient of , M also represents the exponential decay average operation on the square of the map.

8. The method according to claim 1, wherein: The specific process of step five is as follows: S51. Output the relative offset of the predicted target bounding box coordinates relative to the initial target bounding box coordinates: During the video sequence, the trained falcon-like visual twin tracking network predicts the offset of the target's bounding box coordinates in the next frame relative to the initial bounding box coordinates through the aforementioned feature extraction, similarity calculation and feature fusion; S52. Calculate and output the predicted target bounding box coordinates, and use them for target object tracking in the next frame: Add the offset of the target's bounding box coordinates in the next frame relative to the initial bounding box coordinates to the target bounding box in the previous frame to calculate the predicted target bounding box coordinates for tracking the target object in the next frame, and repeat steps S51 and S52 until the end of the video sequence; S53. Test the falcon-like twin tracking network in the UAV123 dataset and real drone autonomous aerial refueling scenarios.

Images