Target tracking method for unmanned aerial vehicle autonomous air refueling by imitating eagle-tenon vision twin network

Abstract

The invention discloses an unmanned aerial vehicle autonomous air refueling target tracking method based on an eagle-falcon-vision-imitating twin network. The method comprises the following steps: 1, multi-scale feature extraction of an eagle-falcon-vision-imitating contrast sensitivity mechanism and an adaptive focusing mechanism; 2, similarity calculation and feature fusion of an eagle-falcon-vision-imitated double-fovea-center-vision interaction mechanism are carried out; 3, simulating senecio midbrain superior hilly pathway perception competition, classifying decision-making mechanisms, and designing a regression loss function; 4, training an eagle-tenon-vision-imitating twinborn tracking network, and outputting classification loss and regression loss; and 5, testing and evaluating the eagle-falcon-vision-imitated twinborn tracking network. The method can adapt to various complex environments and shape and scale changes of the unmanned aerial vehicle; multi-scale spatial features can be efficiently fused, and the most significant region can be concerned; the tracking stability can be ensured to the maximum extent, and the autonomous aerial refueling success rate and safety of the unmanned aerial vehicle are improved; the training strategy adapts to a complex unmanned aerial vehicle autonomous air refueling environment; feature extraction and fusion are efficient, robustness and generalization are good, and training data demands are less.

Description

Technical Field

[0001] The invention discloses a target tracking method for autonomous aerial refueling of a UAV imitating a falcon visual twin network, and belongs to the field of computer vision. Background Art

[0002] With the widespread application of Unmanned Aerial Vehicle (UAV) in the military field, long-term hovering and long-distance combat have become particularly important. Traditional refueling methods often require ground support or return to the base, and the efficiency and flexibility of refueling are low. Autonomous Aerial Refueling (AAR) technology can make UAVs more flexible in performing tasks and extend their combat time. During the short-range guidance stage of autonomous aerial refueling of UAVs, the receiving aircraft approaches the tail of the tanker at high speed. The traditional GPS navigation method is easily affected by factors such as communication delays, enemy electromagnetic interference, and insufficient positioning accuracy in high-dynamic environments, which seriously reduces the success rate of autonomous aerial refueling and also brings great safety risks. The present invention aims to propose a visual target tracking method technology with simple principle, efficient operation, direct and flexible, to improve the success rate and safety of autonomous aerial refueling of UAVs, thereby providing support and guarantee for long-term and long-distance combat of UAVs.

[0003] Visual object tracking technology is an important research direction in the field of computer vision. It involves the dynamic tracking of specific targets in video sequences. Target tracking technology aims to identify a specific target from the first frame of the video and accurately locate the position of the target in each subsequent frame. Currently, the commonly used visual target tracking methods include optical flow method and correlation filtering method. The optical flow method can quickly calculate motion information and does not rely on the specific shape or feature points of the target. It only uses brightness information for motion estimation, making it effective when the features are not obvious; however, the optical flow method is very sensitive to illumination changes. When the scene illumination conditions change significantly, tracking will fail. The target tracking method based on correlation filtering uses discrete Fourier transform to accelerate the calculation of the filter. It runs faster and has good adaptability to the target's posture and scale changes. However, its tracking accuracy and stability are poor. When the tracked target moves quickly in the video, tracking delay or loss will occur. It is not suitable for tracking high-speed moving drones during autonomous aerial refueling.

[0004] With the development of computer vision, the twin network target tracking algorithm based on deep learning can learn effective feature representation from a small amount of data, realize efficient feature learning of the target and accurate anchor frame prediction, which provides a new idea for target tracking and has received widespread attention. It has broad application prospects in the close-range visual guidance stage of autonomous aerial refueling of drones.

[0005] In order to meet the challenges in visual tasks, more and more researchers are choosing to seek inspiration from nature. The visual system of birds of prey, represented by falcons, is very developed, with extremely high visual sensitivity and extremely excellent large field of view perception ability. Even at an altitude of several thousand meters, the prey on the ground can be clearly seen. And once the prey is found, the falcon will track the prey at high speed until it is captured. The falcon's wide field of vision and sharp eyes have made it a focus of research in target detection, feature extraction, and target tracking. Mapping the relevant biological mechanisms of falcon vision into deep learning mechanisms and embedding them into the twin network target tracking method can significantly improve the success rate and accuracy of tracking, and realize efficient, safe, and stable autonomous aerial refueling of drones.

[0006] In summary, the present invention proposes a target tracking method for autonomous aerial refueling of UAVs that imitates the falcon vision twin network, maps the biological mechanism of falcon vision into a deep learning mechanism, and uses it for autonomous aerial refueling of UAVs. The method is simple, efficient, and has good real-time performance, which conforms to actual combat scenarios and has certain reference significance. Summary of the invention

[0007] The purpose of the present invention is to provide a target tracking method for autonomous aerial refueling of unmanned aerial vehicles (UAVs) that simulates the vision twin network of hawks and falcons, aiming to solve the problem of long-term, stable and accurate tracking of the tanker by the receiving aircraft during the close-range visual guidance process of autonomous aerial refueling of UAVs. By drawing on the biological mechanism of hawk vision in nature, a target tracking mechanism of hawk vision is established and embedded into the twin network target tracking method, providing a new solution to the problem of UAV group confrontation.

[0008] The present invention proposes a UAV autonomous aerial refueling target tracking method imitating a falcon visual twin network, and the specific implementation steps are as follows:

[0009] Step 1: Multi-scale feature extraction of simulating the visual contrast sensitivity mechanism and adaptive focusing mechanism of hawks The multi-scale feature extraction mechanism of simulating the visual contrast sensitivity mechanism and adaptive focusing mechanism of hawks proposed in the present invention is as shown in the attached process. Figure 1 shown.

[0010] S11. Obtaining primary features of input images through convolutional neural networks

[0011] Use a convolutional neural network (ResNet-50) to obtain the primary features of the input template image z and the search image x, and output the primary feature map and

[0012] S12. Establishment of a falcon-like visual contrast sensitivity mechanism model

[0013] The falcon-like visual contrast sensitivity model first performs maximum pooling and average pooling on the channel dimension of the input feature map to emphasize the key features unique to high-density photoreceptors in the feature. Then, the feature maps after maximum pooling and average pooling are added element by element. Next, a large convolution kernel is used to Convolution operation is performed with the added feature map To enhance the local contextual relationship between features, simulate the high-density sampling of the fovea of ​​the falcon's eye. Subsequently, the activation function Sigmoid is applied to calculate the attention weights, which adjust the model's sensitivity to specific features according to the contrast sensitivity function of the falcon's vision. Finally, these weights are obtained by matrix Schur product. The input is integrated to achieve high contrast between the target and the distractor, ensuring that important features are enhanced and unimportant features are suppressed, thereby improving the performance of the model in drone tracking tasks. The falcon-like visual contrast sensitivity mechanism model is expressed as follows:

[0014]

[0015] in, is the input feature map, This is the feature map output by the falcon-like visual contrast sensitivity mechanism model. They represent the maximum pooling and average pooling operators respectively, and Sigmoid is the activation function.

[0016] S13. Establishment of a falcon-like vision adaptive focusing mechanism model

[0017] The distribution and structure of cones on the retina of the falcon enhance the high visual resolution of the falcon's eyes, while Adaptive Max Pooling also extracts important features to improve the focus on the target. In addition, the falcon's fovea achieves the highest spatial resolution under high-contrast conditions, while Adaptive Average Pooling promotes feature aggregation and recovery in various scenarios. The model of adaptive focusing mechanism of falcon vision first applies adaptive max pooling and adaptive average pooling to the spatial dimension to achieve precise focus on the target. Next, in order to simulate the falcon's ability to optimize imaging by changing the shape of the lens and ciliary muscle, a point convolution kernel is used to compress and reconstruct channel information. Specifically, the channels of the feature map after adaptive pooling are first convolved with the point convolution kernel for dimensionality reduction to facilitate information extraction. Then, the activation function is used to enable the network to capture more complex feature relationships. Subsequently, a second point convolution is used to restore the activated feature map to the original number of channels, ensuring that the network retains important information while restoring the input features. Finally, the two feature maps obtained after the point convolution are added element by element, and the matrix Schur product is also used. Integrate with the input. The falcon-like vision adaptive focusing mechanism model is expressed as follows:

[0018]

[0019] in, This is the feature map output by the falcon-like vision adaptive focusing mechanism model. They are adaptive maximum pooling and average pooling operators respectively. They represent the point convolution used in compression and reconstruction operations respectively. Sigmoid and ReLu are both activation functions.

[0020] S14. Obtaining multi-scale features of the image

[0021] Template feature map extracted through multi-scale features and search feature graph Respectively expressed as:

[0022]

[0023] in, and They are respectively the template feature map and search feature map output by the falcon-like visual contrast sensitivity mechanism model. and They are respectively the template feature map and search feature map output by the falcon-like vision adaptive focusing mechanism model.

[0024] Step 2: Similarity calculation and feature fusion of the bifoveal visual interaction mechanism of hawk-like vision

[0025] S21. Establish a dual foveal switching mechanism model for hawk-like vision

[0026] The similarity calculation and feature fusion mechanism flow of the falcon-like bifoveal visual interaction mechanism proposed in the present invention is shown in the attached figure. Figure 2 shown.

[0027] Similarity calculation and feature fusion models usually require pairing queries and keys. In this network, the query represents the feature map extracted by the fovea, which obtains the detailed features of the target, while the key features come from the feature map extracted by the lateral fovea, which contains the location of the target and the contextual information of the environment.

[0028] Simulating the dual foveal switching mechanism of falcon vision, the similarity matrix s is expressed as:

[0029]

[0030] in, express and The inner product of the search image and the template image is nc.

[0031] S22. Establish a multi-scale spatial attention mechanism model for falcon-like vision

[0032] Since falcon vision can switch freely between two types of foveal imaging, that is, it can obtain target information from different subspaces and different positions, a multi-scale spatial attention mechanism is introduced based on the dual foveal switching mechanism. By performing h linear projections on the query and key and using different linear projection matrices to promote learning, the final similarity feature map It is obtained by concatenating multi-scale features along the number of channels, which is given by the following formula.

[0033]

[0034] Among them, a certain scale feature scale i Expressed as

[0035]

[0036] matrix and It is the linear projection parameter used to combine all scales, and the Concat function represents the concatenation operation along the channel dimension.

[0037] S23. Generate classification branch and regression branch

[0038] The similarity feature map calculated by the bifoveal visual interaction mechanism of hawk-like vision is input into the head of the twin network, and the classification branch and regression branch are generated, which are expressed as follows:

[0039]

[0040] Among them, W cls and W reg are the weight matrices for classification and regression, respectively, cls and b reg They are bias terms for classification and regression respectively, and the Softmax function converts the input into a probability value.

[0041] Step 3: Simulate the perception competition of the superior colliculus pathway in the falcon, classify the decision-making mechanism, and design the regression loss function

[0042] S31. Establishment of a falcon-like model of perception competition and decision-making mechanism in the superior colliculus pathway of the midbrain

[0043] Falcon vision can quickly analyze complex hunting scenes with limited resources, that is, positioning the area of ​​interest in the highly sensitive fovea of ​​the retina to ensure that the visual system is focused on the target prey area. Studies have shown that the ability of falcon vision to efficiently filter out unimportant perceptions is attributed to the competition and decision-making mechanism in the superior colliculus pathway of the midbrain. Studies have pointed out that when multiple visual perceptions appear in different locations on the retina, competition will occur between neural groups, and only the winning group will be selected and transmitted to a higher visual center for further processing, and guide the falcon to make perceptual decisions on multiple targets appearing on the retina. This mechanism enables falcons to accurately filter out interference and locate prey in complex backgrounds, thereby achieving stable tracking until capture. Applying the perceptual competition and decision-making mechanism in the superior colliculus pathway of falcon vision to the classification loss and regression loss of the twin network can effectively enhance the anti-interference ability of the twin network, significantly improve the tracking accuracy and stability, and further improve the success rate and safety of autonomous aerial refueling of drones. The design process of the classification and regression loss function that imitates the perceptual competition and decision-making mechanism of the superior colliculus pathway of the falcon is shown in the attached figure. Figure 3 shown.

[0044] S32. Classification loss function design

[0045] Let the positive sample set and the negative sample set after filtering be expressed as:

[0046] set + ={c +1 ,c +2 ,...,c +n}

[0047] set - ={c -1 ,c -2,...,c -m}

[0048] c -j (j=1,2,..,m)≥c -th

[0049] Among them, c represents the confidence score of the target, c -th is the filtering threshold, n and m are the number of samples in the positive sample set and the negative sample set respectively, c -j (j=1,2,..,m) is the overall confidence score in the negative sample set.

[0050] For the superior colliculus pathway in the midbrain of falcons, the perceptual salience S of the positive and negative sample sets on the retina is + and S - They are defined as:

[0051]

[0052] Among them, w +i and w -j is the weight of the confidence score, c +i and c -j Represent the overall confidence scores of the positive sample set and the negative sample set respectively.

[0053] In order to retain the positive sample perception S + At the same time, suppress negative sample perception S - , weight parameter w +i and w -j Defined as:

[0054]

[0055] w -j =softmax(c -j )

[0056] Among them, the weight of the confidence score of the positive sample set ensures that the weight distribution among the positive samples is uniform.

[0057] Classification loss function based on competition and decision-making mechanism of hawk-like visual perception Defined as:

[0058]

[0059] Among them, q is the weight of the positive sample, which simulates the extraction of salient targets on the retina by a falcon, effectively enhancing the model's attention to positive samples. Parameter b is used to control the loss value. And a is the margin, that is, the distance between positive and negative samples.

[0060] S33. Regression loss function design

[0061] Based on the perceptual competition and decision-making mechanism of the superior colliculus pathway in the falcon, the confidence score output by the classification branch and the bounding box output by the regression branch can be coordinated to align the classification branch and the regression branch. Specifically, the perceptual competition and decision-making mechanism of the superior colliculus pathway in the falcon simulate the attention to the region of interest (positive samples) and the inhibition of the region of no interest (negative samples), and the following strategy is used to design the regression loss function:

[0062]

[0063] That is, it ensures that high-confidence samples have accurate regression boxes, and at the same time ensures that accurate regression samples have high confidence. +i and c -j Also represent the overall confidence scores in the positive sample set and the negative sample set, r +i and r -j They represent the anchor box accuracy of the positive sample set and the negative sample set output by the regression branch respectively. The symbol indicates that both sides of the equation are necessary and sufficient for each other.

[0064] The present invention uses the distance penalty intersection over union (DIoU) as an indicator to measure the accuracy r of the regression bounding box. The intersection over union (IoU) calculates the ratio of the intersection and union of the predicted bounding box and the true bounding box, while the distance penalty intersection over union introduces a penalty term for the distance between the predicted box and the true box on the basis of the intersection over union, which can better handle the overlap problem of the target bounding box, thereby improving the target tracking performance, especially when the distance between the targets is close. The calculation method of the distance penalty intersection over union is as follows:

[0065]

[0066] Among them, P bbox To predict the bounding box, Gt bbox is the true bounding box, E represents the Euclidean distance between the center coordinates of the target box and the center coordinates of the true box, and K represents the diagonal length of the minimum rectangle that can contain the two bounding boxes.

[0067] The regression loss function of the superior colliculus pathway perception competition and decision-making mechanism of the falcon is designed as:

[0068]

[0069] in, For the contrast loss, the distance between positive samples is reduced while the distance between negative samples is increased, which is defined as:

[0070]

[0071] Among them, N + and N - are the number of positive sample sets and negative sample sets respectively, and p>0 controls the size of the loss function. 2 (i,j) is the minimum distance between samples, y ij is the sample similarity label (1 for similarity and 0 for dissimilarity), and v margin defines the minimum distance between dissimilar samples.

[0072] The flowchart of the target tracking method for autonomous aerial refueling of UAV based on the falcon-like visual twin network proposed in this invention is as follows: Figure 4 shown.

[0073] Step 4: Train the falcon-like visual twin tracking network and output classification loss and regression loss

[0074] S41. Configure the twin network operating environment

[0075] First, use Anaconda3 to configure the Python environment. Create a new environment in the Conda Prompt command line window and install pytorch, torchvision, torchaudio, cuda, opencv, pyyaml, yacs, tqdm, colorama, matplotlib, cython, and tensorboard packages.

[0076] Next, run the setup.py script to build the extension module using the Cython compiler in the Python environment to improve performance or use specific system resources.

[0077] S42. Deploy and initialize the twin network

[0078] First, configure the twin network training hyperparameters, including: the total number of training rounds, the learning rate for different training rounds, the weights of the classification loss function and the regression loss function, the batch size, etc.

[0079] Then, in order to speed up training and improve model performance, you can usually load weights pre-trained on a large dataset. Store the pre-trained weights in the .model folder, and the program loads the pre-trained weights through the model.load_state_dict() function.

[0080] S43. Deploy the twin network training dataset

[0081] The falcon-like visual twin network tracking method proposed in the present invention uses COCO, VID, DET and YoutubeBB as training data sets. Before training, the images in the above data sets need to be cropped to adapt to the input image size required by the twin network convolutional layer.

[0082] S44. Training a falcon-like visual twin tracking network

[0083] During training, the loss of the Siamese network is the weighted sum of the classification loss and the regression loss:

[0084]

[0085] in, is the total loss of the twin network, and are respectively the classification loss function and regression loss function of the hawk-like visual perception competition and decision-making mechanism, l 1 and l 2 are the weights of the classification loss function and the regression loss function, respectively.

[0086] Appropriate weighting strategies can ensure that the network can achieve good performance in both classification and positioning tasks, avoid the dominant role of a certain task in the training process, and improve the generalization of the twin network to cope with various UAV autonomous aerial refueling environments. Minimizing the classification loss ensures that the network can accurately identify whether the target exists, while minimizing the regression loss ensures that the network can accurately predict the location of the target. When these two loss functions reach an ideal balance, the network can not only accurately distinguish between targets and non-targets, but also efficiently locate the target's bounding box, thereby achieving accurate target tracking and improving the success rate and safety of autonomous aerial refueling of UAVs.

[0087] In the training of the twin network, the AdaDelta adaptive optimization training method is used to improve the overcorrection problem in gradient descent by dynamically adjusting the learning rate of each parameter. The AdaDelta adaptive optimization method is updated based on the square of the historical gradient. It does not need to store the entire sum of squared gradients, but instead uses a sliding average to represent it. First, solve the squared cumulative gradient

[0088]

[0089] Among them, t represents the number of iterations, that is, the time step, Represents the loss function in t iterations The neural network weights θ t The gradient of the map, M represents the exponential decay average operation on the map square, and the hyperparameter β represents the decay coefficient of the gradient square, which controls the degree of historical memory of the gradient square.

[0090] Subsequently, during the training process, the weights are updated as follows

[0091]

[0092] Among them, Δθ t+1 represents the update amount of the neural network weight in the t+1 iteration, represents the sliding average of the square of the update amount in the tth iteration, which is a small constant used to prevent division by zero errors. Represents the loss function in t iterations The neural network weights θ t The gradient of , M also represents the exponential decay average operation on the square of the map.

[0093] Step 5: Test and evaluate the falcon-like visual twin tracking network in real scenarios

[0094] During testing, the falcon-like visual twin tracking network is only provided with the true bounding box value of the target in the first frame of the video sequence, and the twin network performs inference and prediction in subsequent frames of the video sequence to track the target.

[0095] S51: Output the relative offset of the predicted target bounding box coordinates relative to the initial target bounding box coordinates

[0096] During the progress of the video sequence, the fully trained falcon-like visual twin tracking network predicts the offset of the target's bounding box coordinates in the next frame relative to the initial bounding box coordinates through the aforementioned feature extraction, similarity calculation and feature fusion.

[0097] S52: Calculate and output the predicted target bounding box coordinates, and use them for target object tracking in the next frame

[0098] The offset of the target bounding box coordinates in the next frame relative to the initial bounding box coordinates is added to the target bounding box of the previous frame to calculate the predicted target bounding box coordinates for target object tracking in the next frame, and steps S51 and S52 are repeated until the video sequence ends.

[0099] S53, Testing the Falcon-like Twin Tracking Network in the UAV123 Dataset and Real UAV Autonomous Aerial Refueling Scenario

[0100] In order to verify the performance of the model, the present invention is tested in the UAV123 dataset, that is, in a real scene. First, the dataset is cleaned and formatted, including image normalization, label correction, etc., to ensure the quality and consistency of the data. Then the test dataset is loaded into the test framework, and it is ensured that each test sample contains the corresponding target drone position annotation, and different indicators are used to evaluate the tracking performance of the twin network.

[0101] The flowchart of the target tracking method for autonomous aerial refueling of UAV based on the falcon-like visual twin network proposed in this invention is as follows: Figure 4 shown. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] Figure 1 Flowchart of the multi-scale feature extraction mechanism that imitates the contrast-sensitive mechanism and adaptive focusing mechanism of falcon vision.

[0103] Figure 2 This is a flow chart of the similarity calculation and feature fusion mechanism of the bifoveal visual interaction mechanism that simulates hawk-like vision.

[0104] Figure 3 Design a flowchart for the stimulus competition, classification of selection mechanisms, and regression loss function of the superior colliculus pathway in the simulated falcon.

[0105] Figure 4 Flowchart of the target tracking method for autonomous aerial refueling of UAVs imitating the falcon visual twin network.

[0106] Figure 5a Figure 5b The test results of the target tracking method for autonomous aerial refueling of UAVs using a falcon-like visual twin network.

[0107] Figure 6 Real-scene visualization results of the target tracking method for autonomous aerial refueling of UAVs using a falcon-like visual twin network. DETAILED DESCRIPTION

[0108] The following specific examples are used to verify the effectiveness of the UAV autonomous aerial refueling target tracking method based on the falcon-like visual twin network proposed in the present invention. The simulation environment configuration of this example is Windows 11, AMD Ryzen 77800H processor, 3.80Ghz main frequency, 16G memory, NVIDIA RTX4060 GPU, and the software is Pycharm2024.1Professional version. The specific practical steps of this example are as follows:

[0109] Step 1: Multi-scale feature extraction based on the contrast sensitivity mechanism and adaptive focusing mechanism of falcon vision

[0110] S11. Obtaining primary features of input images through convolutional neural networks

[0111] Use convolutional neural network to obtain the primary features of input template image z and search image x, and output the primary feature map and

[0112] The present invention uses Resnet-50 as the neural network for extracting primary features. The core idea of ​​ResNet is to introduce skip connections, that is, in each layer of convolution operation, in addition to the output of the current layer, the input is directly added to the output to form a residual map. This method allows the gradient to be transmitted more effectively during back propagation, thereby alleviating the gradient vanishing problem in deep neural network training. S12. Establishing a model of the visual contrast sensitivity mechanism of hawks

[0113] The falcon-like visual contrast sensitivity mechanism model is expressed as follows:

[0114]

[0115] in, is the input feature map, This is the feature map output by the falcon-like visual contrast sensitivity mechanism model. They represent the maximum pooling and average pooling operators respectively, and Sigmoid is the activation function.

[0116] S13. Establishment of a falcon-like vision adaptive focusing mechanism model

[0117] The falcon-like vision adaptive focusing mechanism model is expressed as follows:

[0118]

[0119] in, This is the feature map output by the falcon-like vision adaptive focusing mechanism model. They are adaptive maximum pooling and average pooling operators respectively. They represent the point convolution used in compression and reconstruction operations respectively. Sigmoid and ReLu are both activation functions.

[0120] S14. Obtaining multi-scale features of the image

[0121] Template feature map extracted through multi-scale features and search feature graph Respectively expressed as:

[0122]

[0123] in, and They are respectively the template feature map and search feature map output by the falcon-like visual contrast sensitivity mechanism model. and They are respectively the template feature map and search feature map output by the falcon-like vision adaptive focusing mechanism model.

[0124] Step 2: Similarity calculation and feature fusion of the bifoveal visual interaction mechanism of hawk-like vision

[0125] S21. Establish a dual foveal switching mechanism model for hawk-like vision

[0126] Simulating the dual foveal switching mechanism of falcon vision, the similarity matrix s is expressed as:

[0127]

[0128] in, express and The inner product of nc is the number of channels of the search image and the template image, and the Softmax function converts the input into a probability value.

[0129] S22. Establish a multi-scale spatial attention mechanism model for falcon-like vision

[0130] By performing h linear projections on the query and key and using different linear projection matrices to facilitate learning, the final similarity feature map It is obtained by concatenating multi-scale features along the number of channels, which is given by the following formula.

[0131]

[0132] Among them, a certain scale feature scale i Expressed as

[0133]

[0134] matrix and It is the linear projection parameter used to combine all scales, and the Concat function represents the concatenation operation along the channel dimension.

[0135] S23. Generate classification branch and regression branch

[0136] The similarity feature map calculated by the bifoveal visual interaction mechanism of hawk-like vision is input into the head of the twin network, and the classification branch and regression branch are generated, which are expressed as follows:

[0137]

[0138] Among them, W cls and W reg are the weight matrices for classification and regression, respectively, cls and b reg They are bias terms for classification and regression respectively, and the Softmax function converts the input into a probability value.

[0139] Step 3: Simulate the perception competition of the superior colliculus pathway in the falcon, classify the decision-making mechanism, and design the regression loss function

[0140] S31. Classification loss function design

[0141] Let the positive sample set and the negative sample set after filtering be expressed as:

[0142] set + ={c +1 ,c +2 ,...,c +n}

[0143] set - ={c -1 ,c -2 ,...,c -m}

[0144] c -j (j=1,2,..,m)≥c -th

[0145] Among them, c represents the confidence score of the target, c -th is the filtering threshold, n and m are the number of samples in the positive sample set and the negative sample set respectively, c -j (j=1,2,..,m) is the overall confidence score in the negative sample set.

[0146] For the superior colliculus pathway in the midbrain of falcons, the perceptual salience S of the positive and negative sample sets on the retina is + and S - They are defined as:

[0147]

[0148] Among them, w +i and w -j is the weight of the confidence score, c +i and c -j Represent the overall confidence scores of the positive sample set and the negative sample set respectively.

[0149] In order to retain the positive sample perception S + At the same time, suppress negative sample perception S - , weight parameter w +i and w -j Defined as:

[0150]

[0151] w -j =softmax(c -j )

[0152] Among them, the weight of the confidence score of the positive sample set ensures that the weight distribution among the positive samples is uniform.

[0153] Classification loss function based on competition and decision-making mechanism of hawk-like visual perception Defined as:

[0154]

[0155] Among them, q is the weight of the positive sample, which simulates the extraction of salient targets on the retina by a falcon, effectively enhancing the model's attention to the positive samples. The parameter b is used to control the loss value and is set to 4. And a is the margin, that is, the distance between positive and negative samples, which is set to 0.5. Inspired by the perceptual competition mechanism, by increasing the value of q, the model pays more attention to the classification errors of positive samples during training, thereby improving the recall rate of positive samples.

[0156] S32. Regression loss function design

[0157] The following strategy is used to design the regression loss function:

[0158]

[0159] That is, it ensures that high-confidence samples have accurate regression boxes, and at the same time ensures that accurate regression samples have high confidence. +i and c -j Also represent the overall confidence scores in the positive sample set and the negative sample set, r +i and r -j They represent the anchor box accuracy of the positive sample set and the negative sample set output by the regression branch respectively. The symbol indicates that both sides of the equation are necessary and sufficient for each other.

[0160] The present invention uses the distance penalty intersection over Union (DIoU) as an indicator to measure the regression bounding box accuracy r:

[0161]

[0162] Among them, P bbox To predict the bounding box, Gt bbox is the true bounding box, E represents the Euclidean distance between the center coordinates of the target box and the center coordinates of the true box, and K represents the diagonal length of the minimum rectangle that can contain the two bounding boxes.

[0163] The regression loss function of the superior colliculus pathway perception competition and decision-making mechanism of the falcon is designed as:

[0164]

[0165] in, For the contrast loss, the distance between positive samples is reduced while the distance between negative samples is increased, which is defined as:

[0166]

[0167] Among them, N + and N - are the number of positive sample sets and negative sample sets respectively, and p>0 controls the size of the loss function. 2 (i,j) is the minimum distance between samples, y ij is the sample similarity label (1 for similarity and 0 for dissimilarity), and the v margin defines the minimum distance between dissimilar samples, which is taken as 0.5.

[0168] Step 4: Train the falcon-like visual twin tracking network and output classification loss and regression loss

[0169] S41. Configure the twin network operating environment

[0170] First, use Anaconda3 to configure the Python environment. Create a new environment in the Conda Prompt command line window and install pytorch, torchvision, torchaudio, cuda, opencv, pyyaml, yacs, tqdm, colorama, matplotlib, cython, and tensorboard packages.

[0171] Next, run the setup.py script to build the extension module using the Cython compiler in the Python environment to improve performance or use specific system resources.

[0172] S42. Deploy and initialize the twin network

[0173] First, configure the twin network training hyperparameters. The total number of training rounds is set to 20, and the batch size is set to 128. In the first 10 training rounds, only the classification branch and the regression branch are trained, and the backbone network is frozen, that is, only the weight parameters of the classification branch and the regression branch are updated. In the last 10 training rounds, the last three convolutional layers of the ResNet-50 backbone network are unfrozen to update the weight parameters.

[0174] Then, in order to speed up training and improve model performance, you can usually load weights pre-trained on a large dataset. Store the pre-trained weights in the .model folder, and the program loads the pre-trained weights through the model.load_state_dict() function.

[0175] S43. Deploy the twin network training dataset

[0176] The falcon-like visual twin network tracking method proposed in the present invention uses COCO, VID, DET and YoutubeBB as training data sets. Before training, the images in the above data sets need to be cropped to adapt to the input image size required by the twin network convolutional layer.

[0177] S44. Training a falcon-like visual twin tracking network

[0178] During training, the loss of the twin network is the weighted sum of the classification loss and the regression loss

[0179]

[0180] in, is the total loss of the twin network, and are respectively the classification loss function and regression loss function of the hawk-like visual perception competition and decision-making mechanism, l 1 and l 2 are the weights of the classification loss function and the regression loss function, which are taken as 1.1 and 0.8 respectively in the present invention.

[0181] In the training of the twin network, the AdaDelta adaptive optimization training method is used. First, the squared cumulative gradient is solved.

[0182]

[0183] Among them, t represents the number of iterations, that is, the time step, Represents the loss function in t iterations The neural network weights θ t The gradient, M represents the exponential decay average operation on the map square, and the hyperparameter β represents the attenuation coefficient of the gradient square, which controls the historical memory degree of the gradient square, and is taken as 0.3 in the present invention.

[0184] Subsequently, during the training process, the weights are updated as follows: Subsequently, during the training process, the weights are updated as follows

[0185]

[0186] Among them, Δθ t+1 represents the update amount of the neural network weight in the t+1 iteration, represents the sliding average of the square of the update amount in the tth iteration, which is a small constant used to prevent division by zero errors. In this invention, it is taken as 10 -8 , Represents the loss function in t iterations The neural network weights θ tThe gradient of , M also represents the exponential decay average operation on the square of the map.

[0187] Step 5: Test and evaluate the falcon-like visual twin tracking network in real scenarios

[0188] During testing, the falcon-like visual twin tracking network is only provided with the true bounding box value of the target in the first frame of the video sequence, and the twin network performs inference and prediction in subsequent frames of the video sequence to track the target.

[0189] S51: Output the relative offset of the predicted target bounding box coordinates relative to the initial target bounding box coordinates

[0190] During the progress of the video sequence, the fully trained falcon-like visual twin tracking network predicts the offset of the target's bounding box coordinates in the next frame relative to the initial bounding box coordinates through the aforementioned feature extraction, similarity calculation and feature fusion.

[0191] S52: Calculate and output the predicted target bounding box coordinates, and use them for target object tracking in the next frame

[0192] The offset of the target bounding box coordinates in the next frame relative to the initial bounding box coordinates is added to the target bounding box of the previous frame to calculate the predicted target bounding box coordinates for target object tracking in the next frame, and steps S51 and S52 are repeated until the video sequence ends.

[0193] S53, Testing the Falcon-like Twin Tracking Network in the UAV123 Dataset and Real UAV Autonomous Aerial Refueling Scenario

[0194] In order to verify the performance of the model, the present invention is tested in the UAV123 dataset, that is, in a real scene. First, the dataset is cleaned and formatted, including image normalization, label correction, etc., to ensure the quality and consistency of the data. Then the test dataset is loaded into the test framework, and it is ensured that each test sample contains the corresponding target drone position annotation, and different indicators are used to evaluate the tracking performance of the twin network. The test results of the drone autonomous aerial refueling target tracking method using the falcon-like visual twin network are shown in Figure 2. Figure 5a and Figure 5b shown.

[0195] The visualization results of the target tracking method for autonomous aerial refueling of UAVs based on the falcon-like visual twin network in real scenes are shown in the figure. Figure 6 It can be seen that the twin network target tracking method for autonomous aerial refueling of unmanned aerial vehicles imitating falcon vision proposed in the present invention has a high tracking success rate and accuracy.

Claims

1. A method for tracking a target for autonomous aerial refueling of a UAV by imitating a falcon visual twin network, characterized in that: The method comprises the following steps: Step 1: Multi-scale feature extraction imitating the visual contrast sensitivity mechanism and adaptive focusing mechanism of falcons; The falcon-like visual contrast sensitivity mechanism emphasizes the key parts of the features through maximum pooling and average pooling, and uses large convolution kernels to enhance the local contextual relationship between features; it calculates attention weights and performs matrix Schur product integration; the falcon-like visual adaptive focusing mechanism achieves precise focus on the target through adaptive maximum pooling and adaptive average pooling, and uses point convolution kernels to compress and reconstruct channel information to optimize feature extraction; after being processed by the activation function, the model integrates the feature map through element-by-element addition and matrix Schur product to retain important information and restore input features; After initially extracting input image features through a convolutional neural network, the two mechanisms are used in tandem to improve feature extraction performance in drone tracking tasks; Step 2: Similarity calculation and feature fusion of the bifoveal visual interaction mechanism of hawk-like vision; Similarity calculation and feature fusion of bifoveal visual interaction mechanism of falcon vision to pair query and key; The query represents the feature map extracted by the fovea to obtain the detailed features of the target, while the key features come from the feature map extracted by the lateral fovea, which contains the location of the target and the contextual information of the environment; Step 3: Simulate the perception competition of the superior colliculus pathway in the falcon, the classification of the decision-making mechanism, and the design of the regression loss function; By imitating the perception competition and decision-making mechanism of the superior colliculus pathway in the falcon, the classification loss function and regression loss function are designed in the twin target tracking network to ensure that positive and negative samples are correctly classified, while ensuring that high-confidence samples have accurate regression boxes and accurate regression samples have high classification confidence; Step 4: Train the falcon-like visual twin tracking network and output classification loss and regression loss The loss function of the falcon-like visual twin tracking network is the weighted sum of classification loss and regression loss, which ensures the performance of the network in classification and positioning tasks, avoids the dominant role of a certain task in the training process, and improves the generalization of the twin network to cope with various UAV autonomous aerial refueling environments; The use of adaptive optimization training methods to improve overcorrection in gradient descent; Step 5: Test and evaluate the falcon-like visual twin tracking network in real scenarios During testing, the falcon-like visual twin tracking network is only provided with the true bounding box value of the target in the first frame of the video sequence, and the twin network performs inference and prediction in subsequent frames of the video sequence to track the target.

2. The method according to claim 1, characterized in that: The specific process of step one is as follows: S11. Obtain primary features of the input image through a convolutional neural network: S12. Establishment of a falcon-like visual contrast sensitivity mechanism model: The falcon-like visual contrast sensitivity mechanism model is expressed as follows: in, is the input feature map, This is the feature map output by the falcon-like visual contrast sensitivity mechanism model. They represent the maximum pooling and average pooling operators respectively, and Sigmoid is the activation function; S13. Establishing a falcon-like vision adaptive focusing mechanism model: The falcon-like vision adaptive focusing mechanism model is expressed as follows: in, This is the feature map output by the falcon-like vision adaptive focusing mechanism model. They are adaptive maximum pooling and average pooling operators respectively. Represent the point convolution used in compression and reconstruction operations respectively. Sigmoid and ReLu are both activation functions. S14. Obtaining multi-scale features of the image: Template feature map extracted through multi-scale features and search feature graph Respectively expressed as: in, and They are respectively the template feature map and search feature map output by the falcon-like visual contrast sensitivity mechanism model. and They are respectively the template feature map and search feature map output by the falcon-like vision adaptive focusing mechanism model.

3. The method according to claim 1, characterized in that: The specific process of step 2 is as follows: S21. Establish a falcon-like dual foveal switching mechanism model: Simulating the dual foveal switching mechanism of falcon vision, the similarity matrix s is expressed as: in, express and The inner product of , nc is the number of channels of the search image and the template image; S22. Establish a multi-scale spatial attention mechanism model imitating falcon vision: The similarity feature map is obtained by concatenating multi-scale features along the number of channels, which is given by the following formula: Among them, a certain scale feature scale i It is expressed as: matrix and It is the linear projection parameter used to combine all scales, and the Concat function represents the concatenation operation along the channel dimension; S23. Generate classification branch and regression branch: The similarity feature map calculated by the bifoveal visual interaction mechanism of hawk-like vision is input into the head of the twin network, and the classification branch and regression branch are generated, which are expressed as follows: Among them, W cls and W reg are the weight matrices for classification and regression, respectively, cls and b reg They are bias terms for classification and regression respectively, and the Softmax function converts the input into a probability value.

4. The method according to claim 1, characterized in that: The specific process of step three is as follows: S31. Establish a model of perception competition and decision-making mechanism in the superior colliculus pathway of the falcon: S32. Classification loss function design: Let the positive sample set and the negative sample set after filtering be expressed as: set + ={c +1 ,c +2 ,...,c +n } set - ={c -1 ,c -2 ,...,c -m } c -j (j=1,2,..,m)≥c -th Among them, c represents the confidence score of the target, c -th is the filtering threshold, n and m are the number of samples in the positive sample set and the negative sample set respectively, c -j (j=1,2,..,m) is the overall confidence score in the negative sample set; For the superior colliculus pathway in the midbrain of falcons, the perceptual salience S of the positive and negative sample sets on the retina is + and S - They are defined as: Among them, w +i and w -j is the weight of the confidence score, c +i and c -j Represent the overall confidence scores of the positive sample set and the negative sample set respectively; In order to retain the positive sample perception S + At the same time, suppress negative sample perception S - , weight parameter w +i and w -j Defined as: w -j =softmax(c -j ) Among them, the weight of the confidence score of the positive sample set ensures that the weight distribution among the positive samples is uniform; Classification loss function based on competition and decision-making mechanism of hawk-like visual perception Defined as: Among them, q is the weight of the positive sample, which simulates the extraction of salient targets on the retina by a falcon, effectively enhancing the model's attention to the positive samples; parameter b is used to control the loss value; and a is the margin, that is, the distance between positive and negative samples; S33. Regression loss function design: The following strategy is used to design the regression loss function: Among them, c +i and c -j Also represent the overall confidence scores in the positive sample set and the negative sample set, r +i and r -j Respectively represent the anchor box accuracy of the positive sample set and the negative sample set output by the regression branch; The symbol indicates that both sides of the equation are necessary and sufficient for each other.

5. The method according to claim 4, characterized in that: The distance penalty intersection-over-union ratio is used as an indicator to measure the regression bounding box accuracy r; the distance penalty intersection-over-union ratio is calculated as follows: Among them, P bbox To predict the bounding box, Gt bbox is the true bounding box, E represents the Euclidean distance between the center coordinates of the target box and the center coordinates of the true box, and K represents the diagonal length of the minimum rectangle that can contain the two bounding boxes; The regression loss function of the superior colliculus pathway perception competition and decision-making mechanism of the falcon is designed as: in, For the contrast loss, the distance between positive samples is reduced while the distance between negative samples is increased, which is defined as: Among them, N + and N - are the number of positive sample sets and negative sample sets respectively, p>0 controls the size of the loss function; E 2 (i,j) is the minimum distance between samples, y ij is the sample similarity label, 1 is similar, 0 is dissimilar, and the v margin defines the minimum distance between dissimilar samples.

6. The method according to claim 1, characterized in that: The specific process of step 4 is as follows: S41. Configure the twin network operating environment: S42. Deploy and initialize the twin network: First, configure the twin network training hyperparameters, including: the total number of training rounds, the learning rate for different training rounds, the weights of the classification loss function and the regression loss function, and the batch size; Then, in order to speed up the training and improve the model performance, load the weights pre-trained on a large dataset; store the pre-trained weights in the .model folder, and the program loads the pre-trained weights through the model.load_state_dict() function; S43. Deploy the twin network training dataset: Before this training, the images in the above dataset need to be cropped to adapt to the input image size required by the convolutional layer of the twin network; S44. Training the falcon-like visual twin tracking network: During training, the loss of the Siamese network is the weighted sum of the classification loss and the regression loss: in, is the total loss of the twin network, and are the classification loss function and regression loss function that imitate the visual perception competition and decision-making mechanism of falcons, respectively. l1 and l2 are the weights of the classification loss function and regression loss function, respectively.

7. The method according to claim 6, characterized in that: The step 4 further comprises: In the training of the twin network, the AdaDelta adaptive optimization training method is used to improve the over-correction problem in gradient descent by dynamically adjusting the learning rate of each parameter; first solve the squared gradient accumulation: Among them, t represents the number of iterations, that is, the time step, Represents the loss function in t iterations The neural network weights θ t The gradient of , M represents the exponential decay average operation of the map square, and the hyperparameter β represents the decay coefficient of the gradient square, which controls the degree of historical memory of the gradient square; Subsequently, during training, weights are updated as follows: Among them, Δθ t+1 represents the update amount of the neural network weight in the t+1 iteration, Q[Δθ t 2 ] represents the sliding average of the square of the update amount in the tth iteration, which is a small constant used to prevent division by zero errors. Represents the loss function in t iterations The neural network weights θ t The gradient of , M also represents the exponential decay average operation on the square of the map.

8. The method according to claim 1, characterized in that: The specific process of step five is as follows: S51, output the relative offset of the predicted target bounding box coordinates relative to the initial target bounding box coordinates: During the video sequence, the trained falcon-like visual twin tracking network predicts the offset of the target's bounding box coordinates in the next frame relative to the initial bounding box coordinates through the aforementioned feature extraction, similarity calculation and feature fusion; S52, calculate and output the predicted target bounding box coordinates, and use them for target object tracking in the next frame: Add the offset of the target bounding box coordinates in the next frame relative to the initial bounding box coordinates to the target bounding box in the previous frame, calculate the predicted target bounding box coordinates, and use them for target object tracking in the next frame, and repeat steps S51 and S52 until the video sequence ends; S53. Test the falcon-like twin tracking network in the UAV123 dataset and the real UAV autonomous aerial refueling scenario.

Images