UAV High-Speed Detection Method Based on Siamese Tracking Network and Abnormal Scheduler

Abstract

The present invention discloses a high-speed detection method for unmanned aerial vehicles based on a Siamese tracking network and an anomaly scheduler. The method includes the following steps: 1. Collect unmanned aerial vehicle flight videos, and obtain an unmanned aerial vehicle video detection data set after data annotation and data augmentation; 2. For the one-stage detection network branch, design and construct a Siamese tracking network branch that shares partial weights to perform high-speed tracking on the detected unmanned aerial vehicle; 3. Design a background suppression mechanism based on a channel mixture descriptor to improve the accuracy of tracking unmanned aerial vehicles in complex backgrounds; 4. Design a scheduling network branch based on outlier detection, call tracking to perform high-speed interpolation on the detection results, and automatically re-call detection when tracking fails based on tracking quality assessment and anomaly status assessment; 5. Train the detection, tracking, and scheduling branches in sequence on the unmanned aerial vehicle video detection data set, and during inference, achieve low-computing-power and high-speed unmanned aerial vehicle detection through the cooperation of detection and tracking.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and particularly relates to a high-speed and high-precision vision detection method for unmanned aerial vehicle (UAV) targets based on a deep neural network. Specifically, it is a high-speed detection method for UAVs based on a Siamese tracking network and an anomaly scheduler, which can be used to monitor invading UAVs in real time in a complex environment. Background Art

[0002] In recent years, with the gradual opening of the low-altitude airspace and the booming development of information technology, UAVs have been widely used in various industries such as aerial photography, agriculture and forestry, logistics, security, etc. At the same time, along with the prosperous development of the UAV market and the explosion in the number of UAVs, while UAVs bring convenience to life and production, they also bring many risks in terms of safety and privacy. Small UAVs have characteristics such as high mobility and high concealment, and there are not many operators with professional flight skills and qualities. Incidents of UAVs flying in dangerous areas without filing or approval occur frequently, bringing relatively large hidden dangers to public safety, aviation safety, personal property and privacy safety. The government has introduced various interim measures and management regulations to supervise the flight activities of UAVs, and at the same time, relevant UAV enterprises have also restricted the dangerous flight behaviors of UAVs from a technical perspective. However, due to the wide audience and large number of UAVs, the phenomenon of using self-made UAVs to fly or using technical means to break through no-fly zones occurs from time to time, and the phenomena of "illegal flight" and "unregulated flight" of UAVs still emerge in an endless stream. To avoid the safety threats brought by UAVs invading specific areas, it is very necessary to detect UAVs through reliable technical means.

[0003] Currently, UAV detection technologies are mainly divided into typical technical means such as radar detection technology, radio frequency detection technology, audio detection technology, and vision detection technology. Among them, radar detection technology has poor detection effects on small UAVs with low flight altitudes and small radar cross-sections; radio frequency detection technology is difficult to detect UAV communication signals under complex electromagnetic interference in the battlefield environment; audio detection technology will also fail due to noise interference from artillery and gunfire in the battlefield environment. In contrast, vision detection technology has received extensive attention from the academic community due to its low cost, high medium- and short-range detection accuracy, and the ability to provide rich and intuitive video information. UAVs have high mobility in the urban environment with a complex background, so a vision-based UAV detection system needs to minimize the reaction time and improve the detection frequency. However, current vision detection technologies often rely on neural networks with a large number of parameters and cannot achieve high-speed detection of UAVs on small computing platforms. Therefore, there is an urgent need to design a high-speed and reliable UAV detection method.

[0004] It should be noted that the information disclosed in the above background art section is only used to enhance the understanding of the background of the present disclosure, and thus may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0005] In view of this, the present invention provides a high-speed drone detection method based on a Siamese tracking network and an anomaly scheduler. By designing a Siamese tracking network that shares some weights with the detection network, high-speed tracking interpolation is performed on the detection results provided by the detection network to assist in accelerating the detection, and the anomaly scheduler adaptively switches to call detection and tracking based on tracking quality evaluation and anomaly status evaluation, so as to achieve high-speed, reliable, and low-computing-power-demand drone detection.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or be learned in part through the practice of the present disclosure.

[0007] According to one aspect of the embodiments of the present disclosure, a high-speed drone detection method based on a Siamese tracking network and an anomaly scheduler is provided, including five steps. Step 1: Use an optoelectronic sensor to record multiple groups of videos of flying drones in different backgrounds to form a video set, label the rectangular bounding boxes of the targets in each frame, and design a brightness jitter algorithm and a double blurring algorithm for environmental light changes and rainy and foggy weather to perform data augmentation on the video set, and finally obtain a drone video detection data set. Step 2: For the one-stage detection network branch, design and construct a Siamese tracking network branch based on a residual network and convolutional cross-correlation calculation to perform high-speed tracking on the detected drones. The Siamese tracking network branch shares some feature extraction layers with the detection network branch. Step 3: Design a background suppression mechanism based on a channel mixture descriptor in the Siamese tracking network branch, construct a channel mixture descriptor that effectively expresses the global information of the channel feature map to accurately distinguish the background channel, and suppress the background channel of the feature and enhance the foreground channel of the feature based on the channel attention mechanism to improve the accuracy of tracking drones in complex backgrounds. Step 4: Design an anomaly scheduler based on outlier detection, call the tracking operation to perform high-speed interpolation on the detection results, and evaluate the target tracking quality and tracking anomaly status through outlier detection based on normalizing flow. Adaptively switch the detection and tracking states according to the tracking quality evaluation and anomaly status evaluation results, and while accelerating the detection, improve the overall detection accuracy in the detection and tracking cooperation framework. Step 5: In the training stage, train the detection network branch and the Siamese tracking network branch on the drone video detection data set respectively, and use the positive samples corresponding to the Siamese tracking network branch in this data set to train the anomaly scheduler based on outlier detection. In the inference stage, the anomaly scheduler based on outlier detection adaptively switches detection and tracking to achieve low-computing-power and high-speed detection of drones, and obtain high-speed and reliable detection results for drones. We will introduce each step in detail in turn below.

[0008] In Step 1, a brightness jitter algorithm and a double blurring algorithm are designed for environmental light changes and rainy and foggy weather. The brightness jitter algorithm randomly changes the image brightness and performs jitter on the red and blue channels to simulate the change in image brightness caused by the environmental light changing over time.

[0009] Specifically, the brightness jitter algorithm first performs random brightness jitter on the image. Assume that p xyc ∈ [0, 255] is a pixel point on the input RGB image I ∈ R 3×H×W where H and W are the width and length of the image, x ∈ {1, 2,..., W} and y ∈ {1, 2,..., H} are the planar coordinates of the pixel point on the image, and c ∈ {r, g, b} are the three channels of the input RGB image. For a, b ∈ R, define a V b and a ∧ b to represent the maximum and minimum values of a and b respectively, then the random jitter of the image brightness is shown in formula (1):

[0010]

[0011] where p′ xyc represents the output pixel point, g ~ U(t gl , t gh ) and γ ~ U(t γl , t γh ) correspond to the gain and gamma parameters for changing the image brightness, and U represents a uniform distribution. t gl , t gh , t γl , t γh are parameters that control the distributions of g and γ.

[0012] Then, for the change in the color temperature of the environmental light from early morning to dusk, the brightness jitter algorithm randomly adds and subtracts the pixel intensities of the red and blue channels in opposite directions. Assume that p′ xyr , p′ xyg , p′ xyb represent the pixels of the red, green, and blue channels respectively, then the random jitter of the image color temperature is shown in formula (2):

[0013]

[0014] where δ ~ U(t δl , t δh ) represents the random variable for adding and subtracting the intensities of the red and blue channels. An increase in δ will increase the color temperature of the picture, and t δl and t δl control the distribution of δ.

[0015] The double blurring algorithm randomly adds a fog mask to the image and performs Gaussian filtering blurring to simulate the blurring effect of the edges of the drone in rainy and foggy weather. Specifically, the double blurring algorithm calculates the weighted average of the input image and a white mask of the same size, and uses a Gaussian filter kernel to convolve the image to weaken the edge texture features of the image, as shown in formula (3):

[0016] I out = α·G(I in ) + 255·(1 - α)·1 3×H×W , #(3)

[0017] where I in , I out ∈R 3×H×W represent the input and output images respectively, H and W are the width and length of the image, α~U(t αl , t αh ) represents the weight of the mask weighted average, U represents the uniform distribution, 1 3×H×W represents the all - one matrix as the white mask, G(·) represents the Gaussian filter function, t αl and t αh are used to control the degree of blurring of the drone edge.

[0018] In step two, a Siamese tracking network branch based on residual network and convolutional cross - correlation calculation is designed and constructed. Based on the detection results provided by the one - stage detection network, high - speed tracking interpolation results are provided to assist in accelerating the detection. The Siamese tracking network branch shares some feature extraction layers with the detection network branch and provides real - time tracking results of the target between two detections.

[0019] Specifically, after the detection network gives the image region z corresponding to the target in the previous frame F t-1 of the drone monitoring screen, the Siamese tracking network needs to find the target region in a larger search region x corresponding to z in the current frame F t . To this end, the Siamese tracking network branch first uses the feature extraction layer to extract feature maps for z and x respectively, and adds the features extracted from z and x along the channel dimension to obtain the tracking feature maps f z and f x . After feature extraction, taking f x as the convolution object and f z as the convolution kernel, a convolution operation is performed to calculate the feature map correlation, and finally the tracking score map f = f x *f z + bI is obtained. Where * is the convolution operator, the range adjustment factor b is an arbitrary real number, I is the unit matrix of the same size as the result of f x *f z , and the position region of the target of the drone corresponds to the maximum confidence of the tracking score map f.

[0020] For the design of the feature extraction layer, considering the reuse of some network layers with the detection network while maintaining the fully convolutional nature of the Siamese tracking network, a fully convolutional feature extraction network is constructed, which includes a starting layer and three residual modules, with a total of 16 layers. The first two residual modules are shared with the detection network. The starting layer consists of a convolutional layer with a kernel size of 7×7, a padding margin of 1, and a stride of 1. The three residual modules contain 15 convolutional layers. The k-th residual module contains a downsampling layer and k residual layers. Each downsampling layer uses a convolutional layer with a kernel size of 3×3, a padding margin of 1, and a stride of 2. Each residual layer extracts features using a convolutional layer with a kernel size of 3×3, a padding margin of 0, and a stride of 1 and a convolutional layer with a kernel size of 1×1, a padding margin of 0, and a stride of 1, and performs cross-layer connection on the output feature map of this residual layer and the input feature map of this residual layer. After the cross-layer connection, the outermost circle of pixels of the concatenated feature map is cropped, that is, the cropping inner margin is equal to 1. Thus, a feature extraction layer shared with part of the detection network is constructed for the tracking-assisted Siamese network.

[0021] In step three, to address the problem of easy tracking drift in complex backgrounds, for the tracking feature map f x channel mixing descriptors that describe the global information of each channel are extracted, and the channel attention mechanism is used to calculate the scores of different channels according to the channel mixing descriptors, and the original tracking feature map f x is weighted to obtain the background-suppressed tracking feature map f x ′, giving a smaller weight to the background channel to suppress the background.

[0022] Specifically, first, channel mixing descriptors that describe the global information are extracted from the tracking feature map f x For a feature map f x ∈R c×h×w with the number of channels c, height h, and width w, the mean, maximum value, and average variance of each channel feature map are calculated respectively as the global description vector of each channel. The mean and maximum value of all pixels of a certain channel feature map are used as the mean and maximum value of this channel. After dividing the channel feature map into 3×3 grids, the average value of the variances of all pixels in each grid is used as the average variance of this channel. The means, maximum values, and average variances of different channels are concatenated in channel order to obtain the global description vector f pooling ∈R C×3×1 ; then the channel attention mechanism is applied, and f pooling is calculated based on a 1×1 convolutional layer and a non-linear activation layer, and the foreground score f x of the feature map f score ∈R C×1×1 After multiplying all the channel feature maps by the corresponding foreground scores, they are added up to obtain the tracking feature map f' with enhanced foreground channel attention. x And it is convolved with the tracking feature map f z to finally obtain the tracking score map. Thus, the interference of the background channel to the tracking score map is suppressed, and the UAV tracking accuracy based on the tracking score map is improved.

[0023] Step 4 addresses the detection and tracking scheduling problem of the designed twin tracking network-assisted acceleration detection. An anomaly scheduler based on outlier detection is adopted to divide the serial scheduling of detection and tracking into two steps. First, after the detection network detects the UAV target, the result is assigned to the twin tracking network, and the twin tracking network performs subsequent tracking. Then, when the tracking quality deteriorates and drift occurs, the tracking is terminated and the detection is called again. The key lies in the tracking quality assessment and anomaly state assessment. In Step 4, the assessment of the tracking quality is regarded as an outlier detection problem of the tracking feature map. The outlier detection network branch based on the normalizing flow maps the feature map f' of the positive samples in the training set x to the standard normal distribution, so as to calculate the maximum likelihood score of the mapped distribution of the input feature map f' x and the standard normal distribution during the inference stage, intelligently evaluate the current tracking quality, and adaptively switch the detection and tracking states accordingly to achieve the balance between accuracy and speed.

[0024] Specifically, for the set composed of the positive samples of the tracking feature map f x ', let all the tracking feature maps f x ' in the set be characterized by the high-dimensional continuous random variable x. It is necessary to fit a distribution mapping function f through the normalizing flow so that the mapped distribution z = f(x) conforms to the standard normal distribution, that is, z ~ N(0, 1), where the distribution mapping function f needs to be a surjective and first-order differentiable; assume x ~ p x (x), z ~ p z (z), and the channel dimensions of both x and z are D. According to the theorem given by Poirier in 1995, p z (z) can be expressed as shown in formula (4):

[0025]

[0026] Since z = f(x) is known, let p z (z) be as shown in formula (5):

[0027]

[0028] Substituting it into formula (4) gives p x (x) as shown in formula (6):

[0029]

[0030] Thus, on the tracking feature map f′ x On the positive sample set, the maximum likelihood estimate value of the distribution z corresponding to x under the action of the distribution mapping function f following the standard normal distribution can be calculated as shown in formula (7):

[0031]

[0032] Using formula (7) as the loss function, a suitable distribution mapping function f can be trained, and the maximum likelihood score of the tracking feature map x can be evaluated in the inference stage. However, the value of the third term in formula (7) has a high computational complexity or is even difficult to calculate in the network. To overcome this problem, a normalizing flow is used to quickly estimate the quality of the tracking feature map f x ′.

[0033] The input of the normalizing flow network is the tracking feature map f x ′. The network is composed of multiple identical normalizing flow units connected in series. Taking a certain normalizing flow unit as an example, assuming the input feature map of this unit is y, first make an arbitrary division on the channels of the feature map y, divide it into two groups of feature maps, let them be y1 with the number of channels d and y2 with the number of channels D - d respectively, and copy y1 to h1; then use the mapping function m to extract features from y1, where the mapping function m is designed as two layers of 3×3 convolutional layers to obtain m(y1); add m(y1) and y2 to get h2, and then concatenate h1 and h2 in the channel dimension to finally obtain the feature map h after normalizing flow processing; thus can be simplified to a lower triangular matrix with a diagonal of 1, and further make be 0, as shown in formula (8):

[0034]

[0035] where is a d-dimensional identity matrix, is a (D - d)-dimensional identity matrix, is a (D - d)×d-dimensional all-zero matrix. Connecting the normalizing flow units in series can obtain a normalizing flow outlier detection network with a powerful distribution mapping ability, and the of the distribution mapping function f of the entire normalizing flow network can be calculated by the chain rule and is 0; the finally obtained normalizing flow outlier detection network calculates the maximum likelihood score of the input tracking feature map f x and uses it as the quality evaluation score for tracking, so as to realize the intelligent evaluation of the tracking quality and re-detection when the tracking drifts, and improve the overall detection accuracy without sacrificing the overall detection speed.

[0036] Step 5 is the implementation of the training strategy in the training phase and the implementation of the inference strategy in the inference phase. In the training phase, the detection network branch and the siamese tracking network branch are respectively trained on the UAV video detection dataset, and the positive samples corresponding to the siamese tracking network branch in this dataset are used to train the outlier detection-based anomaly scheduler. In the inference phase, the outlier detection-based anomaly scheduler adaptively switches to call detection and tracking to achieve low computing power requirements and high-speed detection of UAVs.

[0037] Specifically, in the training phase, the airspace target UAV video detection dataset is divided into a video training set and a video test set according to the ratio of training set: test set = K:1. The videos in the video training set are cut into pictures, and the pictures with UAVs are aggregated into a picture training set. The detection network branch is trained on the picture training set using the CloU loss function, and the siamese tracking network branch is trained using the binary cross-entropy loss function. For the trained siamese tracking network branch, the tracking results on the video training set are obtained, and the tracking feature maps corresponding to the positive samples in the tracking results are aggregated into an outlier detection training set to train the outlier detection-based anomaly scheduler.

[0038] In the inference phase, the first frame of the video is detected by the detection network branch, and the detection results are handed over to the siamese tracking network branch for tracking. The tracking results are evaluated by the outlier detection network branch based on the normalizing flow, that is, the maximum likelihood score of the input tracking feature map is calculated according to formula (7) and used as the evaluation score. Then, the outlier detection-based anomaly scheduler adaptively switches the detection and tracking states according to the evaluation results, improving the overall detection accuracy in the detection and tracking cooperation framework while achieving high-speed detection, and achieving a balance between speed and accuracy while significantly reducing the computing power requirements.

[0039] The beneficial effects of the present invention are as follows:

[0040] 1. Aiming at the problem of the urgent need for high-speed detection of highly maneuverable UAVs, a siamese tracking network branch based on the residual network and convolutional cross-correlation calculation is designed, which shares part of the weights with the detection network branch, performs high-speed interpolation on the detection results of the detection network branch, assists the detection network to speed up, improves the detection speed of the detection method for highly maneuverable UAVs, and reduces the computing power overhead.

[0041] 2. Aiming at the problem of easy tracking drift in complex backgrounds, a background suppression mechanism based on channel mixture descriptors is introduced in the siamese tracking network. A channel mixture descriptor that effectively expresses the global information of the channel feature map is designed to accurately discriminate the background channel. Based on the channel attention mechanism, the background channel is suppressed and the foreground channel is enhanced, improving the accuracy of interpolating the UAV detection results in complex backgrounds and enhancing the reliability of the detection method.

[0042] 3. Regarding the problem of the detection and tracking cooperation strategy, an abnormal scheduler based on outlier detection is designed. The quality of target tracking and the abnormal state of tracking are evaluated through outlier detection based on the standardized flow. The detection and tracking states are adaptively switched according to the results of tracking quality evaluation and abnormal state evaluation, so as to improve the overall detection accuracy under the detection and tracking cooperation framework.

[0043] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. Brief Description of the Drawings

[0044] The drawings herein are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts. In the drawings:

[0045] Figure 1 Shows the flowchart of the UAV high-speed detection method based on the twin tracking network and the abnormal scheduler;

[0046] Figure 2 Shows the schematic diagrams of the brightness jitter algorithm and the double fuzzy algorithm;

[0047] Figure 3 Shows the tracking principle diagram of the twin tracking network;

[0048] Figure 4 Shows the flowchart of the tracking quality evaluation and abnormal state evaluation of the outlier detection network based on the standardized flow. Detailed Embodiments

[0049] Now, the exemplary embodiments will be described more fully with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art.

[0050] Figure 1The flowchart of the UAV high-speed detection method based on the Siamese tracking network and the anomaly scheduler is shown. According to one aspect of the embodiments of the present disclosure, a UAV high-speed detection method based on the Siamese tracking network and the anomaly scheduler is provided, including five steps. Step one: An optoelectronic sensor is used to record multiple groups of videos of flying UAVs in different backgrounds to form a video set. The rectangular bounding box of each frame target is labeled, and a brightness jitter algorithm and a double blur algorithm are designed for environmental light changes and rain and fog weather to perform data augmentation on the video set, and finally a UAV video detection data set is obtained. Step two: For the one-stage detection network branch, a Siamese tracking network branch based on the residual network and convolutional cross-correlation calculation is designed and constructed to perform high-speed tracking on the detected UAVs. The Siamese tracking network branch shares some feature extraction layers with the detection network branch. Step three: A background suppression mechanism based on the channel mixing descriptor is designed in the Siamese tracking network branch. A channel mixing descriptor that effectively expresses the global information of the channel feature map is constructed to accurately distinguish the background channel. Based on the channel attention mechanism, the background channel of the feature is suppressed and the foreground channel of the feature is enhanced to improve the accuracy of tracking UAVs in complex backgrounds. Step four: An anomaly scheduler based on outlier detection is designed to call the tracking operation to perform high-speed interpolation on the detection results, and the quality of target tracking and the tracking anomaly state are evaluated through outlier detection based on the normalizing flow. The detection and tracking states are adaptively switched according to the tracking quality evaluation and anomaly state evaluation results, while accelerating the detection, the overall detection accuracy under the detection and tracking cooperation framework is improved. Step five: In the training stage, the detection network branch and the Siamese tracking network branch are respectively trained on the UAV video detection data set, and the positive samples corresponding to the Siamese tracking network branch in this data set are used to train the anomaly scheduler based on outlier detection. In the inference stage, the anomaly scheduler based on outlier detection adaptively switches detection and tracking to achieve low-computing-power and high-speed detection of UAVs. Next, we will introduce each step in detail in turn.

[0051] In step one, for environmental light changes and rain and fog weather, a brightness jitter algorithm and a double blur algorithm are designed, and the algorithm schematic diagrams are shown by Figure 2 As shown. Among them, the brightness jitter algorithm randomly changes the image brightness and performs jitter on the red and blue channels to simulate the change of image brightness caused by the environmental light changing over time; while the double blur algorithm randomly adds a fog mask to the image and performs Gaussian filter blurring to simulate the blurring effect of the UAV edge in rain and fog weather.

[0052] Specifically, the brightness jitter algorithm first performs random brightness jitter on the image. Assume that p xyc ∈[0, 255] is the input RGB image I∈R 3×H×WPixels on it, where H and W are the width and length of the image, x ∈ {1, 2,..., W} and y ∈ {1, 2,..., H} are the planar coordinates of the pixel on the image, and c ∈ {r, g, b} are the three channels of the input RGB image. For a, b ∈ R, define aVb and a∧b to represent the maximum and minimum values of a and b respectively. Then the stochastic dithering of the image brightness is shown in formula (1):

[0053]

[0054] where p′ xyc represents the output pixel, g ∼ U(t gl , t gh ) and γ ∼ U(t γl , t γh ) correspond to the gain and gamma parameters for changing the image brightness, and U(·) represents the uniform distribution. t gl , t gh , t γl , t γh are the parameters controlling the distributions of g and γ, and the specific values are set to 0.5, 1.5, 0.6, and 1.4 respectively.

[0055] Then, the brightness dithering algorithm randomly adds and subtracts the pixel intensities of the red and blue channels in the opposite direction for the ambient light color temperature change from early morning to dusk. Assume p′ xyr , p′ xyg , p′ xyb represent the pixels of the red, green, and blue channels respectively. Then the stochastic dithering of the image color temperature is shown in formula (2):

[0056]

[0057] where δ ∼ U(t δl , t δh ) represents the random variable for adding and subtracting the intensities of the red and blue channels. An increase in δ will increase the color temperature of the picture, and t δl and t δl control the distribution of δ, and the specific values are set to -20 and 20 respectively.

[0058] The double blur algorithm randomly adds a fog mask to the picture and performs Gaussian filtering blur processing to simulate the blur effect of the edges of the UAV in rainy and foggy weather. Specifically, the double blur algorithm calculates the weighted average of the input image and an equal-sized white mask, and uses a Gaussian filter kernel to convolve the image to weaken the edge texture features of the image, as shown in formula (3):

[0059] I out =α·G(I in )+255·(1-α)·1 3×H×W , #(3)

[0060] where I in and I out ∈R 3×H×W represent the input and output images respectively, H and W are the width and length of the image, α~U(t αl , t αh ) represents the weight of the masked weighted average, U represents the uniform distribution, 1 3×H×W represents the all - ones matrix as the white mask, G(·) represents the Gaussian filtering function, t αl and t αh are used to control the degree of edge blur of the UAV.

[0061] In the second step, YOLOX is used as the one - stage detection network, and a Siamese tracking network branch based on the residual network and convolutional cross - correlation calculation is designed and constructed. Based on the detection results provided by the one - stage detection network, high - speed tracking interpolation results are provided to assist in accelerating the detection. The Siamese tracking network branch shares some feature extraction layers with the detection network branch and provides real - time tracking results of the target between two detections.

[0062] Specifically, after the detection network gives the image region z corresponding to the target in the previous frame F t-1 in the UAV monitoring screen, the Siamese tracking network needs to find the target region in a larger search region x corresponding to z in the current frame F t . To this end, the Siamese tracking network branch first uses the feature extraction layer to extract feature maps for z and x respectively, and adds the features extracted from z and x along the channel dimension to obtain the tracking feature map f z and f x . After feature extraction, taking f x as the convolution object and f z as the convolution kernel, a convolution operation is performed to calculate the feature map correlation, and finally the tracking score map f = f x *f z +bI is obtained, where * is the convolution operator, the range adjustment factor b is an arbitrary real number, I is the identity matrix of the same size as the result of f x *f z , and the position area of the UAV target corresponds to the maximum confidence of the tracking score map f. The UAV tracking process of the Siamese tracking network is shown by Figure 3 .

[0063] For the design of the feature extraction layer, considering the reuse of some network layers with the detection network while maintaining the fully convolutional nature of the Siamese tracking network, a DarkNet-16-CIR fully convolutional feature extraction network is constructed, which includes a starting layer and three residual modules, with a total of 16 layers. The first two residual modules are shared with the detection network. The starting layer consists of a convolutional layer with a kernel size of 7×7, a padding margin of 1, and a stride of 1. The three residual modules contain 15 convolutional layers. The k-th residual module contains a downsampling layer and k residual layers. Each downsampling layer uses a convolutional layer with a kernel size of 3×3, a padding margin of 1, and a stride of 2. Each residual layer uses a convolutional layer with a kernel size of 3×3, a padding margin of 0, and a stride of 1 and a convolutional layer with a kernel size of 1×1, a padding margin of 0, and a stride of 1 to extract features, and the output feature map of this residual layer is cross-connected with the input feature map of this residual layer. After the cross-connection, the outermost circle of pixels of the concatenated feature map is cropped, that is, the cropping inner margin is equal to 1. Thus, a feature extraction layer shared with part of the detection network is constructed for the tracking-assisted Siamese network.

[0064] In step three, aiming at the problem that the tracked UAV target is prone to drift in complex backgrounds, a background suppression mechanism based on channel mixture descriptors is adopted to extract the channel mixture descriptors that describe the global information of each channel of the tracking feature map f x . According to the channel mixture descriptors, the channel attention mechanism is used to calculate the scores of different channels, and the original tracking feature map f x is weighted to obtain the background-suppressed tracking feature map f x ′, and a smaller weight is given to the background channel to suppress the background.

[0065] Specifically, first, the channel mixture descriptors that describe the global information are extracted from the tracking feature map f x . For the feature map f x ∈R c×h×w with the number of channels c, height h, and width w, the mean, maximum value, and average variance of each channel feature map are calculated respectively as the global description vectors of each channel. The mean and maximum value of all pixels of a certain channel feature map are used as the mean and maximum value of this channel. After dividing the channel feature map into 3×3 grids, the average value of the variances of all pixels in each grid is used as the average variance of this channel. The means, maximum values, and average variances of different channels are concatenated in the channel order to obtain the global description vector f pooling ∈R c×3×1 . Then, the channel attention mechanism is applied to calculate f pooling based on a 1×1 convolutional layer and a non-linear activation layer. After passing through the sigmoid layer, the foreground score f x of the feature map f score ∈Rc ×1×1 After multiplying all the channel feature maps by the corresponding foreground scores and adding them up, the tracking feature map f' with enhanced foreground channel attention is obtained. x , and convolve it with the tracking feature map f z to finally obtain the tracking score map. Thus, the interference of the background channel to the tracking score map is suppressed, and the UAV tracking accuracy based on the tracking score map is improved.

[0066] In step four, for the detection and tracking scheduling problem of the designed Siamese tracking network-assisted acceleration detection, an outlier-based anomaly scheduler is used to divide the serial scheduling of detection and tracking into two steps. First, after the detection network detects the UAV target, the result is assigned to the Siamese tracking network, and the subsequent tracking is performed by the Siamese tracking network. Then, when the tracking quality deteriorates and drifts occur, the tracking is terminated and the detection is called again. The key lies in the tracking quality evaluation and the abnormal state evaluation. In step four, the evaluation of the tracking quality is regarded as an outlier detection problem of the tracking feature map. The outlier detection network branch based on the normalizing flow maps the feature map f' x of the positive samples in the training set to the standard normal distribution, so as to calculate the maximum likelihood score of the mapping distribution of the input feature map f' x and the standard normal distribution in the inference stage, intelligently evaluate the current tracking quality, and adaptively switch the detection and tracking states accordingly to achieve the balance between accuracy and speed.

[0067] Specifically, the outlier detection network based on the normalizing flow evaluates the tracking quality according to the tracking feature map, and the process is shown by Figure 4 . For the set composed of the positive samples of the tracking feature map f x ', assuming that all the tracking feature maps f x ' in the set are characterized by the high-dimensional continuous random variable x, a distribution mapping function f needs to be fitted through the normalizing flow so that the mapped distribution z = f(x) conforms to the standard normal distribution, that is, z ∼ N(0, 1), where the distribution mapping function f needs to be a surjective and first-order differentiable; assuming x ∼ p x (x), z ∼ p z (z), and the channel dimensions of both x and z are D. According to the theorem given by Poirier in 1995, p z (z) can be expressed as shown in formula (4):

[0068]

[0069] Since z = f(x) is known, let p z (z) be as shown in formula (5):

[0070]

[0071] Substituting it into formula (4) gives px (x) As shown in Equation (6):

[0072]

[0073] Thus, on the positive sample set of the tracking feature map f′ x the maximum likelihood estimate value of the distribution z corresponding to x under the action of the distribution mapping function f, which follows a standard normal distribution, is as shown in Equation (7):

[0074]

[0075] Using Equation (7) as the loss function, a suitable distribution mapping function f can be trained, and the maximum likelihood score of the tracking feature map x can be evaluated in the inference stage. However, the value of the third term in Equation (7) has a high computational complexity or is even difficult to calculate in the network. To overcome this problem, a normalizing flow is used to quickly estimate the quality of the tracking feature map f x ′.

[0076] The input of the normalizing flow network is the tracking feature map f x ′. The network is composed of multiple identical normalizing flow units connected in series. Taking a certain normalizing flow unit as an example, assuming the input feature map of this unit is y, first make an arbitrary division on the channels of the feature map y, dividing it into two groups of feature maps. Let's assume they are y1 with d channels and y2 with D - d channels. y1 is copied as h1; then the mapping function m is used to extract features from y1, where the mapping function m is designed as two layers of 3×3 convolutional layers to obtain m(y1); add m(y1) and y2 to get h2, and then concatenate h1 and h2 in the channel dimension to finally obtain the feature map h after normalizing flow processing; thus it can be simplified to a lower triangular matrix with a diagonal of 1, and further make be 0, as shown in Equation (8):

[0077]

[0078] where is a d-dimensional identity matrix, is a (D - d)-dimensional identity matrix, is a (D - d)×d-dimensional all-zero matrix. By connecting the normalizing flow units in series, a normalizing flow outlier detection network with a powerful distribution mapping ability can be obtained, and the of the distribution mapping function f of the entire normalizing flow network can be calculated by the chain rule and is 0; the finally obtained normalizing flow outlier detection network calculates the input tracking feature map f according to Equation (7) xThe maximum likelihood score is used as the quality evaluation score for tracking, so as to intelligently evaluate the tracking quality and perform re-detection when the tracking drifts, improve the overall detection accuracy without sacrificing the overall detection speed, and enhance the detection reliability.

[0079] Step 5 is the implementation of the training strategy in the training stage and the implementation of the inference strategy in the inference stage. In the training stage, the detection network branch and the Siamese tracking network branch are respectively trained on the UAV video detection dataset, and the positive samples corresponding to the Siamese tracking network branch in this dataset are used to train the outlier detection-based anomaly scheduler. In the inference stage, the outlier detection-based anomaly scheduler adaptively switches to call detection and tracking to achieve low computing power requirements and high-speed detection of UAVs.

[0080] Specifically, in the training stage, the UAV video detection dataset is divided into a video training set and a video test set according to the ratio of training set: test set = K:1. The videos in the video training set are cut into pictures, and the pictures with UAVs are aggregated into a picture training set. The detection network branch is trained on the picture training set using the CloU loss function, and the Siamese tracking network branch is trained using the binary cross-entropy loss function. For the trained Siamese tracking network branch, its tracking results on the video training set are obtained, and the tracking feature maps corresponding to the positive samples in the tracking results are aggregated into an outlier detection training set to train the outlier detection-based anomaly scheduler.

[0081] In the inference stage, the first frame of the video is detected by the detection network branch, and the detection results are handed over to the Siamese tracking network branch for tracking. The tracking results are evaluated by the outlier detection network branch based on the normalizing flow, that is, the maximum likelihood score of the input tracking feature map is calculated according to formula (7) and used as the evaluation score. Then, the outlier detection-based anomaly scheduler adaptively switches the detection and tracking states according to the evaluation results, improves the overall detection accuracy in the detection and tracking cooperation framework while achieving high-speed detection, and balances the speed and accuracy while significantly reducing the computing power requirements.

[0082] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure, which follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed by the present disclosure. The specification and embodiments are only considered exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0083] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. A high-speed detection method for drones based on a Siamese tracking network and an anomaly scheduler, characterized in that, It includes the following steps: Step 1: Use an optoelectronic sensor to record multiple groups of videos of an unmanned aerial vehicle (UAV) in flight under different backgrounds to form a video set. Mark the rectangular bounding boxes of the targets in each frame, and design a brightness jitter algorithm and a double blur algorithm for environmental light changes and rainy and foggy weather to perform data augmentation on the video set, and finally obtain a UAV video detection data set; Step 2: For the one-stage detection network branch, design and construct a Siamese tracking network branch based on a residual network and convolutional cross-correlation calculation to perform high-speed tracking on the detected UAV. The Siamese tracking network branch shares some feature extraction layers with the detection network branch; Step 3: Design a background suppression mechanism based on a channel mixture descriptor in the Siamese tracking network branch. Construct a channel mixture descriptor that effectively expresses the global information of the channel feature map to accurately distinguish the background channel, and use a channel attention mechanism to suppress the background channel of the feature and enhance the foreground channel of the feature to improve the accuracy of tracking the UAV in a complex background; Step 4: Design an outlier detection-based anomaly scheduler, call the tracking operation to perform high-speed interpolation on the detection results, and evaluate the target tracking quality and tracking anomaly status through outlier detection based on normalizing flows. Adaptively switch the detection and tracking states according to the tracking quality evaluation and anomaly status evaluation results, and while accelerating the detection, improve the overall detection accuracy under the detection and tracking cooperation framework; Step 5: In the training stage, train the detection network branch and the Siamese tracking network branch on the UAV video detection data set respectively, and use the positive samples corresponding to the Siamese tracking network branch in this data set to train the outlier detection-based anomaly scheduler. In the inference stage, the outlier detection-based anomaly scheduler adaptively switches between detection and tracking to achieve low-computing-power and high-speed detection of the UAV.

2. The UAV high-speed detection method based on the twin tracking network and the anomaly scheduler according to claim 1, wherein, In the brightness jitter algorithm in Step 1, the brightness of the picture is randomly changed and the red and blue channels are jittered to simulate the change in the picture brightness caused by the environmental light changing over time; Specifically, the brightness jitter algorithm first performs random brightness jitter on the image. Assume that p xyc ∈ [0, 255] is a pixel point on the input RGB image I ∈ R 3×H×W . Here, H and W are the width and length of the image, x ∈ {1, 2,..., W} and y ∈ {1, 2,..., H} are the planar coordinates of the pixel point on the image, and c ∈ {r, g, b} are the three channels of the input RGB image. For a, b ∈ R, define a ∨ b and a ∧ b to represent the maximum and minimum values of a and b respectively. Then the random jitter of the image brightness is shown in Equation (1): where p' xyc represents the output pixel, g ~ U(t gl , t gh ) and γ ~ U(t γl , t γh ) correspond to the gain and gamma parameters for changing the image brightness, U represents a uniform distribution, and t gl , t gh , t γl , t γh are parameters for controlling the distributions of g and γ; Then, for the change in the color temperature of the ambient light from early morning to dusk, the brightness dithering algorithm randomly adds or subtracts the pixel intensities of the red and blue channels in opposite directions. Assuming p′ xyr , p′ xyg , p′ xyb represent the pixels of the red, green, and blue channels respectively, the random dithering of the image color temperature is shown in Equation (2): where δ~U(t δl , t δh ) represents a random variable for adding and subtracting intensities to the red and blue channels. An increase in δ increases the color temperature of the image, and t δl and t δl control the distribution of δ.

3. The high-speed detection method for unmanned aerial vehicles based on the twin tracking network and the anomaly scheduler according to claim 1, wherein In the double blur algorithm in Step 1, a fog mask is randomly added to the picture, and Gaussian filtering is performed for blurring to simulate the blurring effect of the UAV edge in rainy and foggy weather; Specifically, the double blur algorithm calculates the weighted average of the input image and an equal-sized white mask, and convolves the image with a Gaussian filter kernel to weaken the edge texture features of the image, as shown in formula (3); I out = α·G(I in ) + 255·(1 - α)·1 3×H×W ,#(3) where I in , I out ∈R 3×H×W represent the input and output images respectively, H and W are the width and length of the image, α ~ U(t αl , t αh ) represents the weight of the masked weighted average, U represents the uniform distribution, 1 3×H×W represents the all - one matrix as the white mask, G(·) represents the Gaussian filtering function, t αl and t αh are used to control the degree of edge blurring of the drone.

4. The UAV high-speed detection method based on the twin tracking network and the anomaly scheduler according to claim 1, characterized in that, In Step 2, design and construct a Siamese tracking network branch based on a residual network and convolutional cross-correlation calculation, so as to provide high-speed tracking-assisted detection for the one-stage detection network, share some feature extraction layers with the detection network, and provide the result of real-time tracking of the target between two detections; Specifically, after detecting the image region z corresponding to the target in the given drone monitoring frame F of the detection network t-1 in the current frame F, the Siamese tracking network needs to find the target region in a larger search region x corresponding to z. To this end, the Siamese tracking network branch first uses the feature extraction layer to extract feature maps for z and x respectively, and adds the features extracted from z and x along the channel dimension to obtain the tracking feature map f t After that, taking f z as the convolution object and f x as the convolution kernel, a convolution operation is performed to calculate the feature map correlation. Finally, the tracking score map f = f x *f z + bI is obtained, where * is the convolution operator, the range adjustment factor b is an arbitrary real number, and I is the identity matrix of the same size as the result of f x *f z . The position region of the target of the drone corresponds to the position with the maximum confidence of the tracking score map f x *f z .

5. The UAV high-speed detection method based on the twin tracking network and the anomaly scheduler according to claim 1, wherein, For the design of the feature extraction layer, considering the reuse of some network layers with the detection network while maintaining the fully convolutional nature of the Siamese tracking network, a fully convolutional feature extraction network is constructed, which includes a starting layer and three residual modules, with a total of 16 layers. The first two residual modules are shared with the detection network. The starting layer consists of a convolutional layer with a kernel size of 7×7, a padding of 1, and a stride of 1. The three residual modules contain 15 convolutional layers. The k-th residual module contains a downsampling layer and k residual layers. Each downsampling layer uses a convolutional layer with a kernel size of 3×3, a padding of 1, and a stride of 2. Each residual layer extracts features using a convolutional layer with a kernel size of 3×3, a padding of 0, and a stride of 1 and a convolutional layer with a kernel size of 1×1, a padding of 0, and a stride of 1, and performs cross-layer connection between the output feature map and the input feature map of the residual layer. After the cross-layer connection, the outermost circle of pixels of the concatenated feature map is cropped, that is, the cropping inner margin is equal to 1. Thus, a feature extraction layer shared with part of the detection network is constructed for the tracking-assisted Siamese network.

6. The method for high-speed detection of unmanned aerial vehicles based on a twin tracking network and an anomaly scheduler according to claim 4, characterized in that, In the third step, the background suppression mechanism based on the channel mixture descriptor addresses the problem of easy drift in tracking in complex backgrounds. For the tracking feature map f x of each channel, a channel mixture descriptor that describes the global information of the channel is extracted. According to the channel mixture descriptor, a channel attention mechanism is used to calculate the scores of different channels, and the scores are weighted with the original tracking feature map f x to obtain the background-suppressed tracking feature map f x ′, and a smaller weight is given to the background channel to suppress the background; Specifically, first, for the tracking feature map f x Extract the channel mixing descriptor that describes the global information. For the feature map f with the number of channels c, height h, and width w x ∈R c×h×w , calculate the mean, maximum value, and average variance of each channel feature map respectively as the global description vector of each channel; among them, the mean and maximum value of all pixels of a certain channel feature map are used as the mean and maximum value of this channel. After dividing the channel feature map into 3×3 grids, the average value of the variances of all pixels in each grid is used as the average variance of this channel; concatenate the means, maximum values, and average variances of different channels in the channel order to obtain the global description vector f pooling ∈R c×3×1 ; Then apply the channel attention mechanism, and calculate f pooling based on the 1×1 convolutional layer and the non-linear activation layer. After passing through the sigmoid layer, output the foreground score f x of the feature map f score ∈R c×1×1 ; After multiplying all channel feature maps by their corresponding foreground scores and summing them up, the tracking feature map f' with enhanced foreground channel attention is obtained. x , and it is convolved with the tracking feature map f z to finally obtain the tracking score map. Thus, the interference of the background channel on the tracking score map is suppressed, and the UAV tracking accuracy based on the tracking score map is improved.

7. The UAV high-speed detection method based on the twin tracking network and the anomaly scheduler according to claim 6, wherein, The outlier detection-based anomaly scheduler in step 4 divides the serial scheduling of detection and tracking into two steps. First, after the detection network detects the UAV target, the result is assigned to the Siamese tracking network for subsequent tracking. Then, when the tracking quality deteriorates and drift occurs, the tracking is terminated and the detection is called again. where The key lies in tracking quality assessment and abnormal state assessment. In step four, the assessment of tracking quality is regarded as an outlier detection problem of the tracking feature map. The outlier detection network branch based on normalizing flow maps the feature map f′ of the positive samples in the training set x to the standard normal distribution, so as to calculate the maximum likelihood score of the mapping distribution of the input feature map f′ x and the standard normal distribution in the inference stage, intelligently evaluate the current tracking quality, and accordingly adaptively switch the detection and tracking states to achieve the balance between accuracy and speed.

8. The UAV high-speed detection method based on the twin tracking network and the anomaly scheduler according to claim 7, wherein Step 4 is specifically as follows: For the tracking feature map f x ′, the set composed of positive samples. Let all the tracking feature maps in the set be f x ′ be characterized by the high-dimensional continuous random variable x. It is necessary to fit a distribution mapping function f through the normalizing flow so that the mapped distribution z = f(x) conforms to the standard normal distribution, that is, z ∼ N(0, 1), where the distribution mapping function f needs to be a surjective and first-order differentiable; assume x ∼ p x (x), z ∼ p z (z), and the channel dimensions of both x and z are D, and p z (z) can be expressed as shown in formula (4): Since it is known that z = f(x), let p z (z) be as shown in formula (5): Substituting into Equation (4) gives p x (x) is as shown in Equation (6): Thus, on the feature map f′ x on the positive sample set, the maximum likelihood estimate value of the distribution z corresponding to x under the action of the distribution mapping function f following the standard normal distribution can be calculated as shown in formula (7): Using Equation (7) as the loss function, a suitable distribution mapping function f can be trained, and the maximum likelihood score of the feature map f' can be evaluated during the inference phase. However, the value of the third term in Equation (7) has a high computational complexity in the network and is even difficult to compute. To overcome this problem, normalizing flow is used to quickly estimate the quality of the tracking feature map f'. x However, the value of the third term in Equation (7) has a high computational complexity in the network and is even difficult to compute. To overcome this problem, normalizing flow is used to quickly estimate the quality of the tracking feature map f'. x ​ The input of the normalized flow network is the tracking feature map f x ′. The network is composed of multiple identical normalized flow units connected in series. Taking a certain normalized flow unit as an example, assuming the input feature map of this unit is y, first make an arbitrary division on the channels of the feature map y, dividing it into two groups of feature maps, let's say y1 with the number of channels d and y2 with the number of channels D - d. y1 is copied as h1; then use the mapping function m to extract features from y1, where the mapping function m is designed as two layers of 3×3 convolutional layers to obtain m(y1); add m(y1) and y2 to get h2, and then concatenate h1 and h2 in the channel dimension to finally obtain the feature map h after the normalized flow processing; thus can be simplified to a lower triangular matrix with a diagonal of 1, and further make be 0, as shown in Equation (8): Among them is a d-dimensional identity matrix, is a (D - d)-dimensional identity matrix, is a (D - d)×d-dimensional all-zero matrix. By connecting the normalizing flow units in series, a normalizing flow outlier detection network with powerful distribution mapping capabilities can be obtained. Moreover, the of the distribution mapping function f of the entire normalizing flow network can be calculated by the chain rule and is 0; the finally obtained normalizing flow outlier detection network calculates the maximum likelihood score of the input tracking feature map f x ′ and uses it as the quality evaluation score for tracking, thereby realizing the intelligent evaluation of the tracking quality and re-detection when the tracking drifts, and improving the overall detection accuracy without sacrificing the overall detection speed.

9. The high-speed detection method for unmanned aerial vehicles based on a twin tracking network and an anomaly scheduler according to claim 1, characterized in that, In step 5, during the training phase, the aerial target UAV video detection data set is divided into a video training set and a video test set according to the ratio of training set: test set = K:

1. The videos in the video training set are cut into pictures, and the pictures with UAVs are aggregated into a picture training set. The CIoU loss function is used to train the detection network branch on the picture training set, and the binary cross-entropy loss function is used to train the Siamese tracking network branch. For the trained Siamese tracking network branch, the tracking results on the video training set are obtained, and the background-suppressed tracking feature maps corresponding to the positive samples in the tracking results are aggregated into an outlier detection training set to train the outlier detection-based anomaly scheduler.

10. The high-speed detection method of an unmanned aerial vehicle based on a twin tracking network and an anomaly scheduler according to claim 1, wherein In step 5, during the inference phase, the first frame of the video is detected by the detection network branch, and the detection result is handed over to the Siamese tracking network branch for tracking. The tracking result is evaluated by the outlier detection network branch based on normalizing flow, that is, the maximum likelihood score of the input tracking feature map is calculated and used as the evaluation score. The outlier detection-based anomaly scheduler adaptively switches the detection and tracking states according to the evaluation result, improving the overall detection accuracy under the detection and tracking cooperation framework, and achieving a balance between speed and accuracy while significantly reducing the computing power requirements.

Images