A lightweight infrared UAV target tracking method based on Siamese network

Abstract

The present invention discloses a lightweight infrared unmanned aerial vehicle (UAV) target tracking method based on a Siamese network, relating to the technical field of neural network-based UAV target tracking methods. The lightweight tracking network mainly consists of five components: a feature extraction network, a pixel-level feature fusion module, a dynamic template update module, a bounding box regression network, and a target classification network. The feature extraction network uses a lightweight convolutional network (FBNet) for neural network search to generate feature maps from a static template image, a dynamic template image, and a cropped search image. The linear interpolation of the static and dynamic template image feature maps is then used as the template feature representation of the tracking model. The pixel-level feature fusion module effectively fuses the template features with the search image features and outputs them to the classification and regression subnetworks. The bounding box regression network and the target classification network respectively predict the target's position and category. The dynamic template update module continuously updates the dynamic template of the tracking model, dynamically learning changes in the target's appearance and background without increasing the model's complexity. The lightweight target tracking method has good tracking robustness and real-time performance, and has strong generalization performance to drastic changes in the target's appearance and background. It is suitable for infrared UAV target tracking.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) target tracking methods based on neural networks, and in particular to a lightweight infrared UAV target tracking method based on Siamese networks. Background Art

[0002] With the development of technology, the maneuverability and controllability of aerial vehicles such as drones have been continuously enhanced. They have the characteristics of low-altitude flight, slow flight speed, and difficulty in detection. They can carry certain heavy objects and can easily break through the ground air defense firepower network, which may threaten some important ground targets. Therefore, preventing and dealing with the interference and destruction of drone targets has become a global problem in major security activities, which is prominently manifested in the difficulties of control, detection and disposal.

[0003] At present, there are various means of detecting flying targets in low-altitude airspace, mainly including radar detection, radio detection and photoelectric detection. Photoelectric detection means have the characteristics of strong anti-interference, intuitive and clear, and flexible station layout. Therefore, it can play a role in the search and detection of targets such as drones in complex urban environments. However, photoelectric detection technology is greatly affected by weather visibility, humidity, etc. At the same time, there are inevitably many interference or uncertain factors in actual scenes, including mutual occlusion between objects, background noise, sudden changes in light, etc., which makes the process of detecting and tracking moving targets in photoelectric detection extremely challenging. Therefore, it is of great significance to study target tracking algorithms.

[0004] Target tracking refers to the process of using the partial features, current position and motion trend of the target of interest obtained in the image sequence, and applying certain matching or statistical learning methods to determine the position, motion trajectory and other information of the target in the new frame. The basic framework of the single target tracking algorithm mainly includes four parts: feature model, motion model, observation model and online update mechanism. The feature model aims to use image processing technology to obtain information that can characterize the appearance characteristics of the target and serve the construction of the observation model. The features suitable for target tracking include: grayscale features, color features, directional gradient histogram features, depth features, etc.; the motion model mainly provides a set of candidate states that may appear in the current frame target based on the contextual information of the target; the role of the observation model is to predict the state of the target based on the candidate states provided by the feature model and the motion model; the online update mechanism enables the observation model to adapt to changes in the target and background and ensure that the model does not degenerate.

[0005] With the emergence of correlation filtering frameworks and deep learning frameworks, discriminant algorithms based on classification and regression models have become the mainstream methods in the current target tracking field. Among them, deep learning tracking algorithms are mainly tracking algorithms based on Siamese neural networks. Deep learning-based methods can extract higher-level semantic features and are more robust to changes in target motion. However, although the current tracking algorithms based on Siamese neural networks have achieved good results and have good real-time performance, models trained only offline are difficult to adapt to drastic changes in target appearance or background. How to efficiently combine the Siamese network model with the online update mechanism to better adapt to changes in the target and background is a key problem that the current target tracking algorithms based on Siamese networks need to solve.

[0006] The target tracking algorithm based on the Siamese network has good detection and tracking capabilities for UAV targets in complex scenes, and is highly robust to target scale changes, deformation, and complex backgrounds. The present invention uses a lightweight neural network to extract UAV target features, and adopts a pixel-level feature fusion module and a dynamic template update module to enhance the model feature expression capability, which not only improves the real-time tracking robustness of the model, but also better adapts to drastic changes in target appearance and background. The present invention has strong operability and scalability and is suitable for infrared UAV target tracking. Summary of the Invention

[0007] The purpose of the present invention is to provide a lightweight infrared UAV target tracking method based on Siamese network in order to solve the problems of low feature expression ability of the existing model and low performance in adapting to changes in the distance between the target and the background.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a lightweight infrared UAV target tracking method based on Siamese network, comprising the following steps:

[0009] Step A: The lightweight tracking network consists of five main parts: feature extraction network, pixel-level feature fusion module, dynamic template update module, bounding box regression network and target classification network. The feature extraction network uses a lightweight convolutional network FBNet with neural network search to generate feature maps from static template images, dynamic template images and cropped search images, and then uses linear interpolation of the static and dynamic template image feature maps as template feature representation. The pixel-level feature fusion module fuses the template features and the features of the search image and outputs them to the classification and regression subnetworks. The bounding box regression network and target classification network predict the position and category of the target respectively. The dynamic template update module continuously updates the dynamic template of the tracking model, which can dynamically learn the changes in the target appearance and background without increasing the complexity of the model.

[0010] As a further solution of the present invention: Step B: the bounding box regression network consists of two simple 3*3 conv-bn-relu blocks to reduce computational complexity. The classification network adopts a structure similar to the bounding box regression network. The network predicts a 16*16 score map, where each pixel represents the target confidence score of the corresponding area of the search image.

[0011] As a further solution of the present invention: Step C: The pixel-level feature fusion module introduces a pixel-by-pixel fusion module, performs pixel-by-pixel correlation operations on the target position and appearance information extracted from the search image, then connects the calculated correlation feature map with the search image features, and aggregates the results through a 1*1conv-bn-relu block, which can effectively characterize the target position and appearance information.

[0012] As a further solution of the present invention: Step D: The dynamic template updating module randomly samples a dynamic template image from the video sequence during model training to capture feature changes such as target appearance, and passes the dynamic template image through the feature extraction network to obtain the dynamic feature map F d Then, a learnable parameter w is used to map the static template feature F t Perform linear interpolation F t ′:F t ′=(1-w)F t +wF d The linear F t ′ and search image feature map F S Passed to the similarity module, the cosine similarity of the dual template and the search image embedding features is calculated. During inference, the image crop with the highest cosine value is selected for every N frames, and the dynamic template is updated with the bounding box predicted by the frame. The dual template representation module enables the model to effectively encode temporal information as well as object appearance and scale changes. The increase in model parameters and FLOPs is small or even negligible, making it an almost cost-free time module.

[0013] As a further solution of the present invention: Step E: In the training phase, firstly collect drone infrared video image datasets of various complex scenes, various drone types, various flight postures, etc., clean and annotate the infrared image datasets, and divide them into training and verification datasets, select corresponding data enhancement technology, hyperparameters, optimizers and loss functions, etc. to train, verify and save the tracking model, and convert and deploy the tracking model to the embedded platform.

[0014] As a further solution of the present invention: Step F: In the inference stage, the initial frame image of the video is input into the target detection algorithm to obtain the position of the drone target in the initial frame image of the video, and a target frame, i.e., the upper left corner pixel coordinates, the target frame width and height, is given. The drone target in this initial frame is used as the static and dynamic template input of the tracking model, and the feature representation of the template is extracted and fused. The network model tracks the drone target in subsequent infrared images, outputs the drone target frame and confidence, and calculates the similarity score of each 100 frames according to the similarity calculation model. The drone target in the frame with the highest score is updated with the dynamic template input.

[0015] Compared with the existing technology, the beneficial effects of the present invention are: the dynamic template update module continuously updates the dynamic template of the tracking model, and can dynamically learn the changes in the target appearance and background without increasing the complexity of the model. The lightweight target tracking method has good tracking robustness and real-time performance, and has strong generalization performance for drastic changes in target appearance and background, and is suitable for infrared UAV target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A network structure diagram of a tracking model for implementing the method of the present invention;

[0017] Figure 2 This is a pixel-level feature fusion module diagram for implementing the method of the present invention;

[0018] Figure 3 This is a diagram of a dynamic template update module for implementing the method of the present invention;

[0019] Figure 4 It is an algorithm flow chart of the method for implementing the present invention;

[0020] Figure 5 This is a tracking effect diagram of the implementation method of the present invention. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0022] In the description of the present invention, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the devices or components referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are only used for descriptive purposes and should not be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", and "set" should be understood in a broad sense, for example, they can be fixedly connected, detachably connected, or connected in one piece; they can be mechanically connected or electrically connected; they can be directly connected, or indirectly connected through an intermediate medium, or they can be internal connections between two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. The following describes an embodiment of the present invention based on its overall structure.

[0023] See also Figures 1 to 5 In an embodiment of the present invention, a lightweight infrared UAV target tracking method based on a Siamese network is provided. The lightweight tracking network architecture consists of five main parts: a feature extraction network, a pixel-level feature fusion module, a dynamic template update module, a bounding box regression head, and a target classification head. The feature extraction network uses a lightweight convolutional network FBNet for neural network search to generate feature maps from static template images, dynamic template images, and cropped search images. The linear interpolation of the feature maps of the static and dynamic template images is then used as the template feature representation. The pixel-level feature fusion module effectively fuses the template features and the features of the search image and outputs them to the classification and regression subnetworks. The bounding box regression head and the target classification head predict the position and category of the target, respectively.

[0024] The bounding box regression network is composed of two simple 3*3 conv-bn-relu blocks to reduce computational complexity. The classification network adopts a similar structure to the bounding box regression network. The only difference is that one kernel is used instead of four in the last convolution block. This network predicts a 16*16 score map, where each pixel represents the target confidence score of the corresponding area of the search image.

[0025] like Figure 2As shown in the figure, cross-correlation in the pixel-level feature fusion module is the core operation of fusing template features and search image features. The pixel-by-pixel fusion module is introduced to perform pixel-by-pixel correlation by extracting position and appearance information from the search image to enhance similarity information. The calculated correlation feature map is then connected with the search image features, and the result is aggregated through a 1*1conv-bn-relu block, which can effectively represent the position and appearance information of the target.

[0026] like Figure 3 As shown in Figure 2, the dynamic template update module enables the tracking model to obtain the appearance and other feature changes of the target object during the inference process without the need to perform online dynamic optimization. In addition to the main static template image and the cropped search image, a dynamic template image is randomly sampled from the video sequence during model training to capture the changes in the target appearance and other features. The dynamic template image is passed through the feature extraction network to obtain the dynamic feature map F d Then, a learnable parameter w is used to map the static template feature F t Perform linear interpolation F t ′:F t ′=(1-w)F t +wF d The linear F t ′ and search image feature map F S Pass to Figure 3 The similarity module calculates the cosine similarity of the dual template and search image embedding features. During inference, the image crop with the highest cosine value is selected for every N frames, and the dynamic template is updated with the bounding box predicted by the frame. The dual template representation module enables the model to effectively encode temporal information as well as object appearance and scale changes. The increase in model parameters and FLOPs is small or even negligible, making it an almost cost-free time module.

[0027] The algorithm flow is as follows Figure 4 As shown in the figure, it is divided into the training phase and the inference phase. In the training phase, the infrared video image datasets of various complex scenes, various drone types, and various flight postures are first collected, the infrared image datasets are cleaned and labeled, and the training and validation datasets are divided. The corresponding data enhancement technology, hyperparameters, optimizers, and loss functions are selected to train, verify, and save the tracking model, and then the tracking model is converted and deployed to the embedded platform.

[0028] In the inference stage, the initial frame image of the video is input into the target detection algorithm to obtain the position of the drone target in the initial frame image of the video, and the target box is given, that is, the pixel coordinates of the upper left corner, the width and height of the target box. The drone target in this initial box is used as the static and dynamic template input of the tracking model. The feature representation of the template is extracted and fused. The network model tracks the drone target in subsequent infrared images and outputs the drone's target box and confidence. According to the similarity calculation model, the similarity score of each 100 frames is calculated, and the drone target in the frame with the highest score is used to update the dynamic template input.

[0029] The above is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solutions and inventive concepts of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A lightweight infrared UAV target tracking method based on Siamese network, characterized in that: The following steps are involved: Step A: The lightweight tracking network consists of five main parts: a feature extraction network, a pixel-level feature fusion module, a dynamic template update module, a bounding box regression network, and a target classification network. The feature extraction network uses a lightweight convolutional network (FBNet) with neural network search to generate feature maps from static template images, dynamic template images, and cropped search images. The linear interpolation of the feature maps of the static and dynamic template images is then used as the template feature representation. The pixel-level feature fusion module fuses the template features with the features of the search image and outputs them to the classification and regression subnetworks. The bounding box regression network and the target classification network predict the location and category of the target, respectively. The dynamic template update module continuously updates the dynamic template of the tracking model, which can dynamically learn changes in the target appearance and background without increasing the complexity of the model. Step B: The bounding box regression network consists of two simple 3*3 conv-bn-relu blocks to reduce computational complexity. The classification network adopts a similar structure to the bounding box regression network. The network predicts a 16*16 score map, where each pixel represents the object confidence score of the corresponding area of the search image; Step C: Pixel-level feature fusion module introduces a pixel-by-pixel fusion module, which performs pixel-by-pixel correlation operations on the target position and appearance information extracted from the search image. The calculated correlation feature map is then connected with the search image features, and the result is aggregated through a 1*1 conv-bn-relu block, which can effectively represent the target position and appearance information; Step D: The dynamic template update module randomly samples a dynamic template image from the video sequence during model training to capture feature changes such as target appearance, and passes the dynamic template image through the feature extraction network to obtain the dynamic feature map F d Then, a learnable parameter w is used to map the static template feature F t Perform linear interpolation F t ′: F t ′=(1-w)F t +wF d The linear F t ′ and search image feature map F S The template is passed to the similarity module to calculate the cosine similarity of the embedded features of the dual template and the search image. During inference, the image crop with the highest cosine value is selected for every N frames, and the dynamic template is updated with the bounding box predicted for that frame. The dual template representation module enables the model to effectively encode temporal information as well as object appearance and scale changes. The increase in model parameters and FLOPs is small or even negligible, making it an almost cost-free time module. Step E: During the training phase, we first collect infrared video image datasets of various complex scenes, various drone types, and various flight postures. We clean and annotate the infrared image datasets, divide them into training and validation datasets, select appropriate data augmentation techniques, hyperparameters, optimizers, and loss functions, and then train, validate, and save the tracking model. The tracking model is then converted and deployed to the embedded platform. Step F: In the inference phase, the initial frame image of the video is input into the target detection algorithm to obtain the position of the drone target in the initial frame image of the video. The target frame, namely the pixel coordinates of the upper left corner, the width and height of the target frame, is given. The drone target in this initial frame is used as the static and dynamic template input of the tracking model. The feature representation of the template is extracted and fused. The network model tracks the drone target in subsequent infrared images and outputs the drone target frame and confidence score. According to the similarity calculation model, the similarity score of each 100 frames is calculated. The drone target in the frame with the highest score is used to update the dynamic template input.

Images