A method for low-altitude aerial image target tracking based on FPGA
By designing Siamese convolutional neural network and TLD tracking framework on the FPGA platform, the deep network model is optimized, and the robustness and real-time problems of low-altitude aerial image target tracking technology in complex environments is solved, and efficient and real-time target tracking is achieved. Effect.
Patent Information
- Application Number
- CN202010245717.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-03-31
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing low-altitude aerial image target tracking technology is poorly robust when facing rapid motion, complex environments and target size changes, and traditional algorithms require high-performance computer support, so it is impossible to achieve real-time tracking on low-power embedded systems.
A low-altitude aerial image target tracking method based on FPGA was designed, using Siamese convolutional neural network architecture, and the deep network model was optimized through pruning and quantization, combined with the TLD tracking framework, the hardware system and human-computer interaction interface of target tracking were achieved. , ensuring the stability and real-time tracking.
It realizes efficient and real-time target tracking on low-power embedded platforms, improves robustness, is suitable for applications of aircraft such as drones, and solves the tracking difficulty and real-time requirements of traditional methods in complex environments.
Smart Images

Figure CN111460999B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of digital image processing, and in particular relates to a method for tracking low-altitude aerial image targets based on FPGA. Background Art
[0002] With the increasing popularity of low-cost, low-altitude aerial photography platforms such as drones, target tracking using low-altitude aerial images plays an important role in related applications in drone systems. Video sequences are collected by the optical sensor of drones. Given the tracking frame of the initial frame, the position of the target in the subsequent frames is predicted using a high-performance image processing platform combined with computer vision algorithms, which can provide data support for subsequent intelligent human-machine applications (such as outdoor rescue, traffic diversion, military reconnaissance and other tasks). In recent years, with the continuous maturity of computer vision and artificial intelligence technologies, how to apply target tracking technology to intelligent drones has become a research hotspot. Target tracking is achieved through efficient video image processing algorithms and deep neural networks, which is of great significance to the development of applications such as post-disaster search and rescue, aerial mapping, and public safety maintenance based on intelligent drones.
[0003] Although the use of drones for target tracking can adapt to various dangerous and complex environments, drone platforms have problems such as fast movement speed, complex environment, and large changes in target scale in aerial images, which also increase the difficulty of target tracking and put forward higher requirements for real-time tracking. In the past few decades, some representative target tracking research results have been successfully applied to drone platforms, such as particle filtering, Cam-Shift, multi-scale KCF, etc. These methods are based on traditional methods for target feature extraction and have high requirements for feature extraction models. However, when faced with problems such as small target size, unstable image, and target occlusion, they will have poor robustness and reduced accuracy.
[0004] With the continuous development of artificial intelligence technology, a large number of target tracking algorithms based on deep learning have also emerged in recent years. Algorithms such as ECO, SiamFC, and SiamMask have made major breakthroughs in target tracking accuracy and generalization, showing excellent performance that traditional algorithms do not have. However, these methods require the use of computers with high-performance image processors (such as GPU servers) to implement. Such computers are large in size and high in power consumption and cannot be carried on low-altitude aerial imaging platforms. Therefore, low-power embedded systems have become an ideal choice for airborne computing platforms in target tracking systems. In addition, the working scenarios and avionics systems of drones have put forward high requirements on airborne systems in terms of power consumption, algorithm selection, multi-chip interaction design, and high-speed PCB design.
[0005] The performance of several commonly used architectures, including FPGA+ARM, FPGA+DSP, GPU+CPU and dedicated AI chips, is compared and analyzed in various aspects. Figure 1As shown. 1) Power consumption: Based on the analysis from the power consumption perspective, dedicated AI chips have the lowest power consumption. The Jetson series chips with GPU+CPU architecture are the embedded platforms with the highest power consumption among the three, but have the highest throughput, while the ZYNQ with FPGA+ARM architecture has the highest energy efficiency ratio (throughput / S / W), but is limited by RAM bandwidth and also has a relatively low throughput. 2) Working scenarios: The working scenarios are mainly considered from the perspective of platform temperature and external environmental interference. Under the same industrial grade conditions, the operating temperature of the ZYNQ series chips is -40℃~+100℃, and the operating temperature of the remaining three is -40℃~+85℃. Therefore, ZYNQ has stronger reliability at extreme temperatures. In addition, data interaction through external signal buses such as DSP+FPGA is easily interfered by the external environment, such as power supply noise and environmental noise, which affects the reliability of the entire system. 3) Algorithm adaptability: Among the four embedded platforms of FPGA+ARM, GPU+ARM, FPGA+DSP and dedicated AI chips, dedicated AI chips are more suitable for deep learning algorithms, but the supported network architecture is fixed when the platform is designed, and the versatility is poor. FPGA+DSP development boards are mostly used for digital signal processing algorithms, with strong real-time performance. They have a large number of DSP resources and can perform large-scale Fourier operations and multiplication and addition operations. At the same time, DSP also has a rich high-performance algorithm library provided by Texas Instruments. Jetson, as an AI computing platform for GPU-accelerated parallel processing in mobile embedded systems proposed by NVIDIA, has strong versatility both in AI algorithms and traditional algorithms. As for the ZYNQ series proposed by Xilinx, the official has developed a large amount of software for it, which enables it to use ARM to implement complex control and scheduling structures, and use FPGA to implement parallel processing of convolution pooling and other operations with high computational repeatability. However, its bandwidth determines that it is more suitable for small deep learning networks and traditional algorithms. 4) Avionics system: The avionics system of a drone is composed of a flight control computer, sensors, payloads, wireless communications, and empty batteries. At present, the maximum empty battery commonly used in the avionics system of a drone is 8000mAh*11.1V=88.8Wh, most of which is used to maintain the normal operation of the avionics system, so the power consumption requirements for the embedded platform are relatively high.5) Multi-chip interaction design and high-speed PCB (Printed Circuit Board) design: Most embedded development boards with FPGA+DSP architecture use SRIO (Serial Rapid I / O) for data interaction, ZYNQ uses a high-speed on-chip bus (AXI4) for data interaction, and Jetson uses PCIE (Peripheral Component Interconnect Express) for information interaction. The former is more difficult to develop, while the latter two are less difficult to develop. In addition, in terms of high-speed PCB design, the three are roughly the same, and all have SRIO and hyperlink interfaces for high-speed data transmission with the outside world. Therefore, the FPGA+ARM architecture with high flexibility, low power consumption, moderate development difficulty, easy multi-chip interaction and strong scalability is selected as the development platform of this patent. Summary of the invention
[0006] The purpose of the present invention is to overcome the shortcomings of the prior art, apply deep learning to embedded platforms, propose a low-altitude aerial image target tracking method based on FPGA, design a Siamese convolutional neural network architecture for target tracking, accelerate and optimize it according to the resource limitation problem of FPGA, and modularly package it as a tracker of TLD (Tracking Learning Detecting) tracking framework; build a hardware system of image acquisition, data processing and storage, target tracking module and control scheduling to ensure the stable operation of tracking; at the same time, the present invention provides a human-computer interaction interface, which is easy to operate; control by this method solves the problem that the tracking of aircraft such as drones relies on ground control and the communication is easily disturbed, and can ensure accurate real-time online tracking.
[0007] The present invention adopts the following technical solutions and implementation steps:
[0008] FPGA-based method for low-altitude aerial image target tracking
[0009] (1) Design of low-altitude aerial image target tracking method. This method mainly provides online tracking results based on the deep tracking algorithm, and uses the detector and learner of the TLD tracking framework to deal with the problem of long-term tracking lost target recovery. For low-altitude aerial image target tracking, the hardware system of the target tracking method is designed, including several main modules such as camera, data storage and processing, target tracking function and control scheduling function modularization. The specific implementation is as follows:
[0010] The camera and display are connected to the FPGA via an HDMI data cable. The HDMI input and output terminals communicate with the data storage and processing module via the AXI4 bus, and the network cable is used to connect to the Ethernet interface of the FPGA and transmit the data to the initial target tracking frame. ARM controls and schedules the tracking algorithm and sends the control signal to each module via the AXI4 bus.
[0011] According to the target tracking method based on low-altitude aerial images described in requirement (1), the image acquired by the camera is processed, and subsequent target tracking is achieved based on the provided first frame target frame. It is characterized by comprising the following steps:
[0012] (1) Build a deep network architecture for target tracking based on Siamese convolutional neural network:
[0013] The network architecture includes a SiamFC network, an inverted residual module and a feature fusion module. Since a twin network is used, the two branches share network parameters.
[0014] In this patent, the size of the convolution kernel is described by W×H, where W and H represent the width and height of the convolution kernel respectively; the size of the image is described by C×W×H, where C, W and H represent the number of channels, width and height of the image respectively.
[0015] The Siamese convolutional neural network uses AlexNet as the backbone network, and adds an inverted residual module and a feature fusion module. The inverted residual module is composed of a 1*1 convolution kernel for channel expansion, and then uses a deep convolution kernel and a 1*1 convolution kernel to output features. The last four convolution kernels of the network are replaced with the inverted residual module to reduce the amount of calculation and deepen the network. The final network structure includes 34 convolution layers and 4 pooling layers. In the feature fusion module, the 7th, 10th, and 15th layers of the twin network are upsampled to the same size, and then expanded to the same number of channels through a 1*1 convolution kernel. Finally, the features are fused by element-wise addition to obtain two feature fusion layers. Feature fusion can enhance the role of low-level features in the tracking algorithm, strengthen feature representation capabilities, and improve the accuracy of target tracking results.
[0016] In the feature extraction process, images are input into the network in pairs. Each image pair consists of the template image of the first frame and the search domain image, which are input into the corresponding feature extraction network branches. The template branch input image size is 3×127×127. In the first convolution layer, 96 convolution kernels of 11×11 with a step size of 2 are used, and then the BN layer and RELU6 activation layer are used to obtain a feature map of size 96×59×59. In the second convolution layer, 96 convolution kernels of 3×3 with a step size of 2 are used for maximum pooling (MaxPooling), and then the BN layer and RELU6 activation layer are used to obtain a feature map of size 96×29×29. In the third convolution layer, 288 convolution kernels of 1×1 with a step size of 1 are used, and then the BN layer and RELU6 activation layer are used to obtain a feature map of size 28. 8×29×29 feature map; in the fourth convolution layer, 288 convolution kernels of 5×5 size with a step size of 1 are used for depth convolution, and then the BN layer and RELU6 activation layer are used to obtain a feature map of size 288×25×25; in the fifth convolution layer, 256 convolution kernels of 1×1 size with a step size of 1 are used, and then the BN layer is used to obtain a feature map of size 256×25×25; in the sixth convolution layer, 256 convolution kernels of 3×3 size with a step size of 2 are used for maxpooling, and then the BN layer and RELU6 activation layer are used to obtain a feature map of size 256×12×12; in the seventh convolution layer, 256 convolution kernels of 3×3 size with a step size of 2 are used for maxpooling, and then the BN layer and RELU6 activation layer are used to obtain a feature map of size 256×12×12; In the convolution layer, 768 1×1 convolution kernels with a step size of 1 are used, and then the BN layer and RELU6 activation layer are used to obtain a feature map of size 768×12×12. In the eighth convolution layer, 768 3×3 convolution kernels with a step size of 1 are used for depth convolution, and then the BN layer and RELU6 activation layer are used to obtain a feature map of size 768×10×10. In the ninth convolution layer, 192 1×1 convolution kernels with a step size of 1 are used, and then the BN layer is used to obtain a feature map of size 192×25×25. In the tenth convolution layer, 576 1×1 convolution kernels with a step size of 1 are used, and then the BN layer is used to obtain a feature map of size 192×25×25. After the BN layer and the RELU6 activation layer, a feature map of size 576×10×10 is obtained; in the eleventh convolution layer, 576 convolution kernels of size 3×3 with a step size of 1 are used for depth convolution, and then the BN layer and the RELU6 activation layer are used to obtain a feature map of size 576×8×8; in the twelfth convolution layer, 192 convolution kernels of size 1×1 with a step size of 1 are used, and then the BN layer is used to obtain a feature map of size 192×8×8; in the thirteenth convolution layer, 576 convolution kernels of size 1×1 with a step size of 1 are used, and then the BN layer and the RELU6 activation layer are used to obtain a feature map of size 576×8×8;In the 14th convolution layer, 576 convolution kernels with a step size of 1 and a size of 3×3 are used for deep convolution, and then the BN layer and RELU6 activation layer are used to obtain a feature map of size 576×6×6; in the 15th convolution layer, 128 convolution kernels with a step size of 1×1 are used, and then the BN layer is used to obtain a feature map of size 128×6×6; since this network architecture is a twin network, the search branch and the template branch share the same network structure and network parameters, which will not be repeated here. The search branch input image size is 3×255×255, and after the same network structure and network parameters as the template branch, a feature map of size 128×22×22 is finally obtained. ;
[0017] In the feature fusion module of the network, features are extracted through two network branches and feature fusion is performed to obtain two feature fusion layers. The feature map of the seventh convolutional layer of feature extraction is passed through a 1×1 convolution kernel to obtain the features of 128 channels. Figure 1 The feature map of the tenth convolutional layer of feature extraction is passed through a 1×1 convolution kernel to obtain a feature map of 128 channels, and then bilinear interpolation is upsampled to the feature Figure 1 The size of the feature Figure 2 ; The feature map of the fifteenth convolutional layer of feature extraction is upsampled to feature Figure 1 The size of the feature Figure 3 ;feature Figure 1 , two 3. By adding channel-by-channel elements, the final fusion feature layer is obtained. The size of the template branch fusion feature layer is 128×12×12, and the size of the search branch fusion feature layer is 128×28×28.
[0018] Finally, the template branch fusion feature layer is regarded as a convolution kernel of size 128×12×12 with a step size of 1. A depth-wise separable convolution is performed on the search branch fusion feature layer to obtain a feature map of final size 1×17×17. The feature map is then upsampled by bilinear interpolation to 1×255×255 to obtain the tracking target position and confidence.
[0019] In the training process, the GOT10K dataset and YouTube-BB dataset are used as training sets. The dataset is first cropped. The size of each sample is cropped to 127×127 template image and 256×256 search image as the input of the network. The stochastic gradient descent (SGD) method is used to train the feature extraction network and the classification network. The weight decay is set to 0.0005, and the parameters are initialized with Gaussian distribution. The training is repeated for 50 epochs, each with 50,0000 sample pairs, the batch size is 8, and the learning rate is increased from 10 -2 Decay to 10 -8 .
[0020] (2) Deep network acceleration optimization for tracking algorithms:
[0021] The convolution operation brings a large number of multiplication and addition operations and floating-point operations. At the same time, the trained deep network model is 32-bit floating-point type, and the FPGA resources are fixed. Therefore, according to the computing power of the embedded platform, the trained deep network model is accelerated and optimized, including two steps: pruning and quantization.
[0022] ① Pruning
[0023] This patent adopts sparsity threshold pruning as a pruning strategy, sorts the weights of each layer of the convolution kernel of the model by absolute value, and sets the weights less than the threshold to 0. The threshold calculation formula is as follows:
[0024]
[0025] t∈{t0,t0+Δ t ,…,t0+nΔ t}
[0026] Where S t is the current sparsity threshold, S i = 0 is the initial sparsity, S f =0.5 is the target sparsity, Δ t =100 is the pruning frequency, t0=0 is the start time of pruning, and n=100 is the number of pruning times. Therefore, pruning is performed once every 100 steps of network training, and a total of 100 pruning times.
[0027] In addition, the EAO results of the network on the VOT2018 dataset were recorded when the network was not pruned. The EAO results of the VOT2018 dataset were tested after each pruning. When the EAO value dropped by more than 0.05 relative to the initial result, pruning was stopped. Otherwise, pruning training was continued until 100 prunings were completed to reach the target sparsity of 0.5.
[0028] ②Quantification
[0029] Quantization can reduce the amount of computation and increase resource reuse. The 32-bit floating-point network model obtained after the above pruning is converted to 8-bit fixed-point, and then the network is trained. 8-bit weights and activation functions are used in forward propagation, and floating-point numbers are used for gradient calculation in reverse gradient propagation.
[0030] (3) Online tracking framework based on low-altitude aerial image tracking method:
[0031] Combined with the TLD tracking framework, the accelerated optimized network model is used as the tracker of the tracking framework, and the TLD detector and learning module are used to complete the lost target recovery to achieve long-term online tracking.
[0032] The TLD detector is used to retrieve lost targets. It consists of three cascade classifiers: variance classifier, set classifier, and nearest neighbor classifier. First, the variance classifier is used to calculate the variance of the pixel grayscale values in the search domain, and the image blocks with a variance less than 50% of the target area are screened out. Then the passed image blocks are input into the set classifier. The set classifier has 10 basic classifiers, each with 13 pixel comparison pairs. Therefore, the pixel comparison pairs of the input image blocks are compared with each other to generate 10 13-bit binary codes. The average of the posterior probabilities corresponding to the 10 binary codes is calculated and compared with the threshold to screen out positive samples. Finally, the remaining image blocks are input into the nearest neighbor classifier, and the similarity measurement is performed to screen out the final positive samples as the detection results. The learning module evaluates the detector through the results of the tracker, and generates training samples based on the evaluation results to update the target model of the detector.
[0033] Before tracking, we first scanned the entire image according to the size and step size of the initial tracking frame information to obtain a large number of image blocks, calibrated the positive and negative samples according to the overlap rate, and generated 200 positive samples after affine transformation through small-scale transformation, offset, rotation and other operations on the positive samples, and used these positive and negative samples to train the set classifier and the nearest neighbor classifier. During the tracking process, the target tracking deep network model was used to calculate the position and confidence of the moving target, and the detector was used to calculate the detection results.
[0034] When the confidence of the tracker or the IOU (Intersection over Union) between the tracker and the detector is greater than the threshold, the tracking is considered successful and the tracking result is output. When the IOU between the tracker and the detector is less than the threshold, the positive and negative sample sets are updated according to the current frame tracking box and detection results, and the detector is updated.
[0035] When the confidence of the tracker and the IOU between the tracker and the detector are both less than the threshold, the tracking is considered to have failed, the detector is used to reinitialize the target position and scale, and the detection result is output. The positive and negative sample sets are updated according to the current frame error tracking frame and detection result, and the detector and learner are updated.
[0036] Features of the present invention:
[0037] First, the present invention uses the ZYNQ series FPGA platform to achieve low-altitude aerial image target tracking, solving the problems of current low-altitude aerial image target tracking, which relies on ground control, is easily interfered with by communication, and is difficult to achieve real-time performance over a long transmission distance. Secondly, the present invention uses a deep learning target tracking algorithm and optimizes network acceleration based on the computing power of the embedded platform. Compared with the current method of target tracking on an embedded platform based on traditional algorithms, the present invention has stronger robustness and meets the tracking requirements in practical applications.
[0038] Table 1 Network structure and parameters
[0039] (A)
[0040]
[0041] (B)
[0042]
[0043] BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 Comparative analysis results of several embedded platforms
[0045] Figure 2 The overall process of low-altitude aerial image target tracking method based on FPGA
[0046] Figure 3 Overall structure of target tracking method based on embedded platform
[0047] Figure 4 Online tracking algorithm structure
[0048] Figure 5 SiamFC Network
[0049] Figure 6 Deep network architecture for aerial image tracking methods
[0050] Figure 7 Inverted residual module
[0051] Figure 8 TLD Detector
[0052] Fig. 9 Pruning DETAILED DESCRIPTION
[0053] The following is a detailed description of the implementation examples of the present invention in conjunction with the accompanying drawings:
[0054] A method for low-altitude aerial image target tracking based on FPGA. The overall process is as follows: Figure 2 As shown in the figure, it mainly includes building a deep network architecture for target tracking based on Siamese convolutional neural network, accelerating and optimizing the deep network for tracking algorithms, and an online tracking framework based on low-altitude aerial image tracking methods. The deep network architecture for target tracking based on Siamese convolutional neural network is shown in the figure. Figure 6 Figure 7 As shown in the attached figure, the deep network acceleration optimization for the tracking algorithm is shown in the attached figure. Fig. 9As shown. Based on the method of this patent, first, the deep network architecture for low-altitude aerial image target tracking is built, including the AlexNet backbone network, the inverted residual module and the feature fusion module; then the network model of the target tracking algorithm is improved to be suitable for hardware through pruning, quantization and other methods; finally, the accelerated optimized network model is used as the tracker of the TLD tracking framework, combined with the detector of the TLD tracking framework and the lost target retrieval function of the learning module, to achieve long-term online tracking.
[0055] The present invention adopts the following technical solutions and implementation steps:
[0056] (1) Design of low-altitude aerial image target tracking method. This method mainly designs the hardware system of the target tracking method by providing online tracking results based on the target tracking framework of Siamese convolutional neural network, such as Figure 3 The structural diagram of the tracking method is given, including several main functional modules such as camera, data storage and processing, target tracking function and control scheduling function modularization.
[0057] The camera and display are connected to the FPGA via an HDMI data cable. The HDMI input and output terminals communicate with the data storage and processing module via the AXI4 bus, and the network cable is used to connect to the Ethernet interface of the FPGA and transmit the data to the initial target tracking frame. ARM controls and schedules the tracking algorithm and sends the control signal to each module via the AXI4 bus.
[0058] (2) In the design of low-altitude aerial image target tracking method, an online tracking framework of low-altitude aerial image tracking method is designed and embedded into the hardware system to provide tracking results online. Figure 4 The online tracking framework is given in and the workflow of the tracking framework is briefly introduced.
[0059] The present invention adopts the following technical solutions and implementation steps:
[0060] (1) Build a deep network architecture for target tracking based on Siamese convolutional neural network. The specific steps are as follows:
[0061] As attached Figure 6 As shown in the figure, the network architecture consists of three parts: SiamFC network, inverted residual module and feature fusion module. Since the twin network is adopted, the two branches share network parameters.
[0062] SiamFC Network: Figure 5The network structure is shown in the figure. The main branch is the AlexNet network. The two branches share the same network parameters. To ensure translation invariance, no padding operation is performed. The parameters are shown in (A) of Table 1. They include 5 convolutional layers and 2 pooling layers, respectively. The pooling layer uses a 3×3 convolution kernel with a step size of 2 to implement the pooling operation. First, the first frame target is expanded to a 127×127 template image and a 256×256 search image. The two images are then sent to the corresponding branches. Finally, the 6×6×128 feature map generated by the template branch (upper) acts as a convolution kernel to convolve the 22×22×128 feature map generated by the search branch (lower). The obtained feature map is upsampled to the original search image size to obtain the final target tracking coordinates and the confidence value of the corresponding position.
[0063] Inverted residual module: attached Figure 7 The figure shows the network structure diagram, which replaces a W×H convolution with t (t=3) 1×1 convolutions (left), 1 W×H deep convolution (middle) and 1 1×1 convolution (right). The middle and right parts are collectively called depthwise separable convolutions. This structural design deepens the number of network layers, improves the nonlinear changes of the model, enhances the generalization ability of the model, and reduces the amount of computation.
[0064] Feature fusion module: as attached Figure 6 As shown in the figure, the inverted residual module (IR) replaces the four convolution kernels of the SiamFC network, and the 7th, 10th, and 15th layers of the two network branches are upsampled to the same size, the template branch is 12×12, and the search branch is 28×28, and then expanded to the same 128 channels through a 1×1 convolution kernel. Finally, the features are fused by adding channel-by-channel elements to obtain two feature fusion layers with sizes of 128×12×12 and 128×28×28. Such a multi-feature fusion module can enhance the role of low-level features in the tracking algorithm and use the powerful learning ability of convolutional neural networks to learn the association between different features.
[0065] Therefore, the final deep network structure is as shown in the attached figure. Figure 6As shown in (B) of Table 1, in the feature extraction process, the image pairs are input into the network. Each image pair consists of the template image of the first frame and the search domain image, which are respectively input into the corresponding feature extraction network branches. The template branch input image size is 3×127×127. In the first convolution layer, after 96 convolution kernels of 11×11 with a step size of 2, the BN layer and the RELU6 activation layer are passed to obtain a feature map of size 96×59×59; in the second convolution layer, after 96 convolution kernels of 3×3 with a step size of 2 are used for maximum pooling (MaxPooling), after the BN layer and the RELU6 activation layer are passed, the feature map of size 96×29×29 is obtained; in the third convolution layer, after 288 convolution kernels of 1×1 with a step size of 1, the BN layer and the RELU6 activation layer are passed to obtain a feature map of size 28. 8×29×29 feature map; in the fourth convolution layer, 288 convolution kernels of 5×5 size with a step size of 1 are used for depth convolution, and then the BN layer and RELU6 activation layer are used to obtain a feature map of size 288×25×25; in the fifth convolution layer, 256 convolution kernels of 1×1 size with a step size of 1 are used, and then the BN layer is used to obtain a feature map of size 256×25×25; in the sixth convolution layer, 256 convolution kernels of 3×3 size with a step size of 2 are used for maxpooling, and then the BN layer and RELU6 activation layer are used to obtain a feature map of size 256×12×12; in the seventh convolution layer, 256 convolution kernels of 3×3 size with a step size of 2 are used for maxpooling, and then the BN layer and RELU6 activation layer are used to obtain a feature map of size 256×12×12; In the convolution layer, 768 1×1 convolution kernels with a step size of 1 are used, and then the BN layer and RELU6 activation layer are used to obtain a feature map of size 768×12×12. In the eighth convolution layer, 768 3×3 convolution kernels with a step size of 1 are used for depth convolution, and then the BN layer and RELU6 activation layer are used to obtain a feature map of size 768×10×10. In the ninth convolution layer, 192 1×1 convolution kernels with a step size of 1 are used, and then the BN layer is used to obtain a feature map of size 192×25×25. In the tenth convolution layer, 576 1×1 convolution kernels with a step size of 1 are used, and then the BN layer is used to obtain a feature map of size 192×25×25. After the BN layer and the RELU6 activation layer, a feature map of size 576×10×10 is obtained; in the eleventh convolution layer, 576 convolution kernels of size 3×3 with a step size of 1 are used for depth convolution, and then the BN layer and the RELU6 activation layer are used to obtain a feature map of size 576×8×8; in the twelfth convolution layer, 192 convolution kernels of size 1×1 with a step size of 1 are used, and then the BN layer is used to obtain a feature map of size 192×8×8; in the thirteenth convolution layer, 576 convolution kernels of size 1×1 with a step size of 1 are used, and then the BN layer and the RELU6 activation layer are used to obtain a feature map of size 576×8×8;In the 14th convolution layer, 576 convolution kernels with a step size of 1 and a size of 3×3 are used for deep convolution, and then the BN layer and RELU6 activation layer are used to obtain a feature map of size 576×6×6; in the 15th convolution layer, 128 convolution kernels with a step size of 1×1 are used, and then the BN layer is used to obtain a feature map of size 128×6×6; since this network architecture is a twin network, the search branch and the template branch share the same network structure and network parameters, which will not be repeated here. The search branch input image size is 3×255×255, and after the same network structure and network parameters as the template branch, a feature map of size 128×22×22 is finally obtained. ;
[0066] In the feature fusion module of the network, features are extracted through two network branches and feature fusion is performed to obtain two feature fusion layers. The feature maps of the seventh convolutional layer of feature extraction are 768×12×12 and 768×28×28, and the feature maps of size 128×12×12 and 128×28×28 are obtained after the convolution kernel of size 1×1. Figure 1 ; The 576×10×10 and 576×26×26 feature maps of the tenth convolutional layer of feature extraction are passed through a 1×1 convolution kernel to obtain a 128-channel feature map, which is then upsampled to feature maps using bilinear interpolation. Figure 1 The size of the feature Figure 2 ; The 128×6×6 and 128×22×22 feature maps of the fifteenth convolutional layer of feature extraction are upsampled to feature Figure 1 The size of the feature Figure 3 ;feature Figure 1 , two 3. By adding channel-by-channel elements, the final fusion feature layer is obtained. The size of the template branch fusion feature layer is 128×12×12, and the size of the search branch fusion feature layer is 128×28×28.
[0067] Finally, the template branch fusion feature layer is regarded as a convolution kernel of size 128×12×12 with a step size of 1. A depth-wise separable convolution is performed on the search branch fusion feature layer to obtain a feature map of size 1×17×17. The feature map is then upsampled by bilinear interpolation to obtain a feature map of size 1×255×255 to obtain the tracking target position and confidence.
[0068] In the training process, the GOT10K dataset and YouTube-BB dataset are used as training sets. The dataset is first cropped. The size of each sample is cropped to 127×127 template image and 256×256 search image as the input of the network. The stochastic gradient descent (SGD) method is used to train the feature extraction network and the classification network. The weight decay is set to 0.0005, and the parameters are initialized with Gaussian distribution. The training is repeated for 50 epochs, each with 50,0000 sample pairs, the batch size is 8, and the learning rate is increased from 10 -2 Decay to 10 -8 The loss function formula used is as follows:
[0069]
[0070] Where v is the single response value output by the network, y is the actual value, D is the final feature map, and u is the point in the feature map.
[0071] (2) Deep network acceleration optimization for tracking algorithms. The specific steps are as follows:
[0072] Deep network calculations bring about a large number of multiplication and addition operations and floating-point operations. At the same time, the trained deep network model is a 32-bit floating-point type, and FPGA resources are fixed. Therefore, based on the computing power of the embedded platform, the trained deep network model is accelerated and optimized, including two steps: pruning and quantization.
[0073] ① Pruning
[0074] This patent adopts sparsity threshold pruning as a pruning strategy, sorts the weights of each layer of the convolution kernel of the model by absolute value, and sets the weights less than the threshold to 0. The threshold calculation formula is as follows:
[0075]
[0076] t∈{t0,t0+Δ t ,…,t0+nΔ t}
[0077] Where S t is the current sparsity threshold, S i = 0 is the initial sparsity, S f =0.5 is the target sparsity, Δ t =100 is the pruning frequency, t0=0 is the start time of pruning, and n=100 is the number of pruning times. Therefore, pruning is performed once every 100 steps of network training, and a total of 100 pruning times.
[0078] Before pruning, the EAO results of the network model on the VOT2018 dataset were recorded, and the EAO results of the VOT2018 dataset were tested after each pruning. When the EAO value dropped by more than 0.05 relative to the initial result, pruning was stopped, otherwise pruning training was continued until 100 prunings were completed to reach the target sparsity of 0.5.
[0079] ②Quantification
[0080] Quantization is to convert the 32-bit floating-point network model obtained after the above pruning to 8-bit fixed-point. The original 32-bit floating-point value range is -3.4×10^38~+3.4×10^38, and the value range after 8-bit fixed-point conversion is -128~+127. Therefore, quantization can be regarded as a mapping process. The mapping formula is as follows:
[0081] x fp32 =scale×x int (4)
[0082]
[0083] |max| is the maximum value of 32-bit weight, x int It is a quantized 8-bit fixed-point value.
[0084] Then, the network is trained, using 8-bit weights and activation functions in forward propagation and floating-point numbers for gradient calculation in reverse gradient propagation. This operation can reduce the amount of calculation, increase resource reuse, and improve computing efficiency.
[0085] (3) Online tracking framework based on low-altitude aerial image tracking method:
[0086] The present invention adopts the TLD tracking framework as the main framework. As an effective tracking framework, the TLD framework has been widely used in the industry and has good universality. It mainly includes a detector, a tracker and a learning module. The present invention directly adopts the structure and threshold of the detector and the learning module, uses the detector and the learning module to complete the lost target recovery to achieve long-term online tracking, and uses the accelerated optimized network model as the tracker of the tracking framework. The online tracking process is as follows: Figure 2 shown.
[0087] The TLD detector consists of three cascade classifiers: variance classifier, ensemble classifier and nearest neighbor classifier. Figure 8 As its basic structure, the ensemble classifier and the nearest neighbor classifier are updated by the positive and negative samples generated by the tracking results of each frame.
[0088] First, the variance of the pixel grayscale values in the search domain is calculated by the variance classifier, and the image blocks with a variance less than 50% of the target area are screened out. The calculation formula is as follows:
[0089] D(p)=E(p 2 )-E 2 (p) (5)
[0090] Among them, p is the location of the image block, and E is the expectation.
[0091] Then the image block that has passed is input into the ensemble classifier. The ensemble classifier has 10 basic classifiers, each with 13 pixel comparison pairs. Therefore, after the pixel comparison pairs of the input image block are compared with each other, 10 13-bit binary codes are generated. The average of the posterior probabilities corresponding to the 10 binary codes is calculated and compared with the threshold (initial value is 0.6) to filter out positive samples. The posterior probability formula is as follows:
[0092]
[0093] #P and #N represent the number of positive and negative samples corresponding to the binary code during training and updating.
[0094] Finally, the remaining image blocks are input into the nearest neighbor classifier, and the similarity measurement is performed to select the final positive samples as the detection results. The formula is:
[0095] S r (p,M)>θ NN (7)
[0096] Where M is the sample set, p is the new sample, S r is the relative similarity of the new sample, θ NN is the threshold, and its initial value is 0.65.
[0097] Before tracking, the TLD learning module first scans the current frame through 21 scale scanning windows according to the initial tracking frame to obtain a large number of image blocks. The scaling ratio of the scanning window scale is 1.2, the scale transformation is 0.16151~6.19174, the horizontal step coefficient is 10% of the width of the initial target bounding box, the vertical step coefficient is 10% of the height of the initial target bounding box, and the size of the smallest bounding box is 20 pixels. The generated samples are calibrated according to the overlap rate. The 10 samples with the smallest overlap rate are positive samples, and the rest are negative samples. Each positive sample is randomly scaled and offset within the range of ±1% for 20 times, and rotated within the range of ±10% to generate 200 affine transformed positive samples to balance the positive and negative samples. These positive and negative samples are used to form a positive and negative sample set M to train the set classifier and the nearest neighbor classifier, and the sample set is updated by the positive samples judged as negative samples and the negative samples judged as positive samples during the training process.
[0098] During the tracking process, the position and confidence of the moving target in the current frame are calculated using the deep network model through the tracking box position of the previous frame, and a large number of image blocks are generated using the learning module. The detector calculates the detection results through these image blocks.
[0099] When the confidence of the tracker is greater than 0.4 or the IOU between the tracker and the detector is greater than 0.3, the tracking is considered successful and the tracking result is output. At the same time, since the tracker uses a deep model, it has higher reliability. Therefore, when the tracking is successful and the IOU between the tracker and the detector is less than 0.3, the positive and negative sample sets are updated according to the current frame tracking frame and detection results, and the detector is updated.
[0100] When the confidence of the tracker is less than 0.4 and the IOU of the tracking box between the tracker and the detector is less than 0.3, it means that the target is lost and the tracking is judged to have failed. The detector is used to reinitialize the target position and scale and output the detection result. The positive and negative sample sets are updated according to the current frame error tracking box and detection result, and the detector and learner are updated.
Claims
1. A method for low-altitude aerial image target tracking based on FPGA. Features: The camera and display are connected to the FPGA via an HDMI data cable. The HDMI input and output terminals communicate with the data storage and processing module via the AXI4 bus. The network cable is used to connect to the Ethernet interface of the FPGA and transmit the initial target tracking frame. The ARM controls and schedules the tracking algorithm and sends the control signal to each module via the AXI4 bus. The image acquired by the camera is processed and subsequent target tracking is achieved according to the provided first frame target frame; the following steps are included: (1) Build a deep network architecture for target tracking based on Siamese convolutional neural network: The network architecture includes SiamFC network, inverted residual module and feature fusion module. Since the twin network is adopted, the two branches share network parameters; The size of the convolution kernel is described by W×H, where W and H represent the width and height of the convolution kernel respectively; the size of the image is described by C×W×H, where C, W, and H represent the number of channels, width, and height of the image respectively; The Siamese convolutional neural network uses AlexNet as the backbone network, and adds an inverted residual module and a feature fusion module; the inverted residual module is composed of a 1*1 convolution kernel for channel expansion, and then uses a deep convolution kernel and a 1*1 convolution kernel to output features. The last four convolution kernels of the network are replaced with the inverted residual module to reduce the amount of calculation and deepen the network. The final network structure includes 34 convolution layers and 4 pooling layers; in the feature fusion module, the 7th, 10th, and 15th layers of the twin network are upsampled to the same size, and then expanded to the same number of channels through a 1*1 convolution kernel. Finally, the features are fused by adding channel-by-channel elements to obtain two feature fusion layers; In the feature extraction process, images are input into the network in pairs. Each image pair consists of a template image of the first frame and a search domain image, which are input into the corresponding feature extraction network branches respectively. The template branch input image size is 3×127×127. In the first convolution layer, 96 convolution kernels of 11×11 with a step size of 2 are used, and then the BN layer and the RELU6 activation layer are used to obtain a feature map of 96×59×59. In the second convolution layer, 96 convolution kernels of 3×3 with a step size of 2 are used for maximum pooling, and then the BN layer and the RELU6 activation layer are used to obtain a feature map of 96×29×29. In the third convolution layer, the template branch input image size is 3×127×127. In the first convolution layer, the convolution kernels of 11×11 with a step size of 2 are used for maximum pooling, and then the convolution kernels of 3×3 ...3×3 with a step size of 3×3 with a step size of 3×3 with a step size of 3×3 with a step size of 3×3 with a step size of 3×3 with a step size of 3×3 with a step size of 3×3 with a step size of 3×3 with a step size of 3×3 with a step size of In the first convolution layer, 288 convolution kernels of 1×1 with a step size of 1 are used, and then the BN layer and the RELU6 activation layer are used to obtain a feature map of size 288×29×29; in the fourth convolution layer, 288 convolution kernels of 5×5 with a step size of 1 are used for depth convolution, and then the BN layer and the RELU6 activation layer are used to obtain a feature map of size 288×25×25; in the fifth convolution layer, 256 convolution kernels of 1×1 with a step size of 1 are used, and then the BN layer is used to obtain a feature map of size 256×25×25; in the sixth convolution layer, 256 convolution kernels of 3×3 with a step size of 2 are used for maxpooling, and then the BN layer and RELU6 activation layer are used. LU6 activation layer, a feature map of size 256×12×12 is obtained; in the seventh convolution layer, after 768 1×1 convolution kernels with a step size of 1, and then through the BN layer and RELU6 activation layer, a feature map of size 768×12×12 is obtained; in the eighth convolution layer, after 768 3×3 convolution kernels with a step size of 1 are used for depth convolution, and then through the BN layer and RELU6 activation layer, a feature map of size 768×10×10 is obtained; in the ninth convolution layer, after 192 1×1 convolution kernels with a step size of 1, and then through the BN layer, a feature map of size 192×25×25 is obtained; in the tenth convolution layer, after 576 steps of 1, and then pass through the BN layer and the RELU6 activation layer to obtain a feature map of size 576×10×10; in the eleventh convolution layer, after 576 convolution kernels of size 3×3 with a step size of 1 for depth convolution, and then pass through the BN layer and the RELU6 activation layer, a feature map of size 576×8×8 is obtained; in the twelfth convolution layer, after 192 convolution kernels of size 1×1 with a step size of 1, and then pass through the BN layer, a feature map of size 192×8×8 is obtained; in the thirteenth convolution layer, after 576 convolution kernels of size 1×1 with a step size of 1, and then pass through the BN layer and the RELU6 activation layer, a feature map of size 576×8×8 is obtained;In the fourteenth convolution layer, 576 convolution kernels of 3×3 size with a step size of 1 are used for depth convolution, and then the BN layer and RELU6 activation layer are used to obtain a feature map of size 576×6×6; in the fifteenth convolution layer, 128 convolution kernels of 1×1 size with a step size of 1 are used, and then the BN layer is used to obtain a feature map of size 128×6×6; since this network architecture is a twin network, the search branch and the template branch share the same network structure and network parameters. The search branch input image size is 3×255×255, and after the same network structure and network parameters as the template branch, a feature map of size 128×22×22 is finally obtained; In the feature fusion module of the network, features are extracted through two network branches, and feature fusion is performed to obtain two feature fusion layers; the feature map of the seventh convolutional layer of feature extraction is subjected to a 1×1 convolution kernel to obtain a feature map of 128 channels; the feature map of the tenth convolutional layer of feature extraction is subjected to a 1×1 convolution kernel to obtain a feature map of 128 channels, and then bilinearly interpolated upsampled to the size of feature map one to obtain feature map two; the feature map of the fifteenth convolutional layer of feature extraction is subjected to bilinear interpolation upsampled to the size of feature map one to obtain feature map three; feature maps one, two, and three are added channel by channel element to obtain the final fusion feature layer, the size of the template branch fusion feature layer is 128×12×12, and the size of the search branch fusion feature layer is 128×28×28; Finally, the template branch fusion feature layer is regarded as a convolution kernel of size 128×12×12 with a step size of 1. A depth-wise separable convolution is performed on the search branch fusion feature layer to obtain a feature map of size 1×17×17. The feature map is then upsampled by bilinear interpolation to 1×255×255 to obtain the tracking target position and confidence. During the training process, the dataset was first cropped. The size of each sample was cropped to a 127×127 template image and a 256×256 search image as the network input. The stochastic gradient descent method was used to train the feature extraction network and the classification network. The weight decay was set to 0.0005 and the parameters were initialized with Gaussian distribution. The training was repeated for 50 epochs, with 500,000 sample pairs in each epoch. The batch size was 8 and the learning rate was increased from 10 -2 Decay to 10 -8 ; (2) Deep network acceleration optimization for tracking algorithms: Accelerate and optimize the trained deep network model, including pruning and quantization; ① Pruning The sparsity threshold pruning is used as the pruning strategy. The weights of each layer of the convolution kernel of the model are sorted by absolute value, and the weights less than the threshold are set to 0. The threshold calculation formula is as follows: Where S t is the current sparsity threshold, S i = 0 is the initial sparsity, S f =0.5 is the target sparsity, Δ t =100 is the pruning frequency, t 0 =0 is the start time of pruning, n=100 is the number of pruning times; therefore, pruning is performed once every 100 steps of network training, for a total of 100 times; In addition, the EAO results of the network on the VOT2018 dataset were recorded when the network was not pruned. The EAO results of the VOT2018 dataset were tested after each pruning. When the EAO value dropped by more than 0.05 relative to the initial result, pruning was stopped. Otherwise, pruning training was continued until 100 prunings were completed to reach the target sparsity of 0.
5. ②Quantification The 32-bit floating-point network model obtained after the above pruning is converted to 8-bit fixed-point, and then the network is trained. 8-bit weights and activation functions are used in forward propagation, and floating-point numbers are used for gradient calculation in reverse gradient propagation. (3) Online tracking framework based on low-altitude aerial image tracking method: Combined with the TLD tracking framework, the accelerated optimized network model is used as the tracker of the tracking framework, and the TLD detector and learning module are used to complete the lost target recovery to achieve long-term online tracking; The TLD detector is used to retrieve lost targets. It consists of three cascade classifiers: variance classifier, set classifier, and nearest neighbor classifier. First, the variance classifier is used to calculate the variance of the pixel grayscale values in the search domain, and the image blocks with a variance less than 50% of the target area are screened out. Then, the passed image blocks are input into the set classifier. The set classifier has 10 basic classifiers, each of which has 13 pixel comparison pairs. Therefore, after the pixel comparison pairs of the input image blocks are compared in pairs, 10 13-bit binary codes are generated. The average of the posterior probabilities corresponding to the 10 binary codes is calculated and compared with the threshold to screen out positive samples. Finally, the remaining image blocks are input into the nearest neighbor classifier, and the similarity measurement is performed to screen out the final positive samples as the detection results. The learning module evaluates the detector through the results of the tracker, and generates training samples based on the evaluation results to update the target model of the detector. Before tracking, these positive and negative samples are used to train the set classifier and the nearest neighbor classifier; during the tracking process, the target tracking deep network model is used to calculate the position and confidence of the moving target, and the detector is used to calculate the detection result; When the confidence of the tracker or the IOU of the tracker and the detector is greater than the threshold, the tracking is determined to be successful and the tracking result is output; when the IOU of the tracker and the detector is less than the threshold, the positive and negative sample sets are updated according to the current frame tracking frame and detection results, and the detector is updated; When the confidence of the tracker and the IOU between the tracker and the detector are both less than the threshold, the tracking is judged to have failed, the detector is used to reinitialize the target position and scale, and the detection result is output; the positive and negative sample sets are updated according to the current frame error tracking box and detection result, and the detector and learner are updated.
