Biological vision inspired unmanned aerial vehicle video image target detection method

Abstract

The invention discloses an unmanned aerial vehicle video image target detection method inspired by biological vision, which is characterized in that an existing detection algorithm is improved from two aspects of enlarging a sensing range and focusing a target through a double-branch target detection algorithm for decoupling classification and regression tasks. A spatial self-correction module is introduced to simulate an eagle eye to acquire a mechanism of a wide view field, and a network sensing range is expanded and a sampling position is adaptively corrected through an up-sampling branch in self-correction convolution and deformable convolution; an inner convolution module is introduced to simulate the convergence effect of an eagle eye on a target, and a higher weight is generated for a target region of interest in a mode of dynamically generating a convolution kernel, so that interference caused by a complex background of an aerial image is reduced.

Description

Technical Field

[0001] The invention belongs to the field of unmanned aerial vehicle images, and in particular relates to a biological vision-inspired unmanned aerial vehicle video image target detection method. Background Art

[0002] With the rapid development and widespread application of drone technology, drone video image detection technology plays an increasingly important role in various fields (such as monitoring, search and rescue, agriculture, urban planning, etc.). However, due to the changing shooting angles, non-fixed heights and complex shooting environments (such as light changes, cluttered backgrounds, etc.) of drones, object detection in video images faces many challenges.

[0003] Existing drone video image detection algorithms mainly rely on deep learning techniques such as convolutional neural networks (CNNs) and region proposal networks (RPNs). Although these methods have made significant progress in target detection tasks, they still have some shortcomings:

[0004] 1. Limited receptive range: Traditional convolutional neural networks have a limited receptive range when processing high-resolution aerial images, making it difficult to capture global information, resulting in limited accuracy in target detection.

[0005] 2. Serious background interference: Aerial images usually contain complex background information, and these interference factors will significantly affect the performance of target detection. Existing methods often have difficulty in effectively distinguishing between targets and backgrounds, resulting in frequent false detections and missed detections.

[0006] 3. High computational complexity: High-precision target detection algorithms usually require a lot of computing resources and time, which is an important limitation for UAV applications with high real-time requirements. Summary of the invention

[0007] The purpose of the present invention is to disclose a biological vision-inspired target detection method for drone video images to capture more global information and generate higher weights for target regions of interest, thereby effectively reducing the interference caused by complex backgrounds and improving detection accuracy.

[0008] The object of the present invention is to achieve a biological vision-inspired drone video image target detection method by the following technical means, comprising the following steps:

[0009] PVT network, input the image to be detected into the non-convolutional Transformer network, namely PVT network, and extract four feature maps of different resolutions from C2 to C5 in layers;

[0010] The feature maps extracted and output by the FPN network and the PVT network are input into the FPN feature fusion connected from top to bottom to obtain 5 feature maps P2 to P6;

[0011] Extract positioning features from C2 to C5, and extract classification features from P2 to P6;

[0012] The classification features are sent to the bounding box regression branch of the RetinaNet detector head. After the classification features and positioning features are fused, they are sent to the classification branch of the RetinaNet detector head. The two are stacked through convolutional layers to obtain the bounding box detection results, that is, all objects that may belong to the target in the image are marked with a bounding box, and each marked target is classified.

[0013] The positioning features are extracted from C2 to C5 through the self-convolution correction module.

[0014] The self-convolution correction module is:

[0015] The input feature map is split into two parts in the channel dimension, namely X1 and X2. Then, the original convolution kernel is split into four parts, namely K1, K2, K3, and K4. K2, K3, and K4 are used to process the feature map X1 to obtain the output Y1, and K1 is used to process the feature map X2 to obtain the output Y2.

[0016] For feature map X1, we first downsample it by a factor of 2 using an average pooling operation, and then restore it to its original size by upsampling it using a conventional convolution operation. The specific process is shown in formula (4.1):

[0017] shown.

[0018] X1'=Upsample(AvgPool(X1)*K2)(4.1)

[0019] In the formula, Upsample represents bilinear interpolation upsampling, and * represents convolution operation;

[0020] Then, the original feature map X1 is connected to X1' through a residual connection structure, and after Sigmoid activation, it is multiplied element by element with the feature map obtained by Conv K3 convolution of the feature map X1. Finally, a 3×3 convolution with K4 as the convolution kernel is added to obtain the calculation result of the upper half branch. The calculation process is shown in formula (4.2).

[0021] Y1=(X1*K3·σ(X1+X′1))*K4 (4.2)

[0022] For feature map X2, first obtain the sampling position offsets offset for input feature X2, and use the offset to replace the square sampling position of conventional convolution, then use K1 as the deformable convolution kernel to perform deformable convolution operation on X2, so as to obtain the output Y2 of the lower half branch. The calculation of deformable convolution is as shown in formula (4.3);

[0023]

[0024] In the formula, x is the input feature map, y is the output feature map, w is the convolution kernel parameter, p0 is the output position, and p k ∈{(-1,-1),(-1,0),…,(0,1),(1,1)} represents the pixel neighborhood of the sampling center, Δp k represents the learnable offset;

[0025] Finally, the feature maps Y1 and Y2 output by the two branches of the self-correction convolution are cascaded to obtain the final result Y of the module.

[0026] The classification features are extracted from P2 to P6 through the inner convolution module.

[0027] The inner convolution module process is to obtain the unique convolution kernel parameters of each position and perform the inner convolution operation. The feature map with input dimensions (H, W, C) is first subjected to a convolution linear transformation operation with a kernel size of 1×1 to obtain a feature map with dimensions (H, W, K).

[0028] ×K) weight map, K represents the kernel size of the inner convolution;

[0029] Calculate the convolution kernel weight for each pixel position in the weight map, that is, generate a convolution kernel H for each position of the input image i,j , H i,j =φ(X i,j )=W1σ(W0X i,j ) (4.4)

[0030] In the formula, X i,j represents the value of the input feature map at (i, j), W0 and W1 represent the parameters of two 1×1 convolution kernels, and σ represents the BN layer and nonlinear activation function;

[0031] For a generated inner convolution kernel H i,j , which acts on a specific spatial location The output for each position is as follows,

[0032]

[0033] In the formula, Δ K is the neighborhood offset set of the convolution center position, K is the size of the generated convolution kernel, and k is the channel number.

[0034] The classification features are sent to the border regression branch of the RetinaNet detector head and are subjected to bounding box regression through four stacked convolutional layers; after the classification features and the positioning features are fused, they are sent to the classification branch of the RetinaNet detector head and are subjected to classification through four stacked convolutional layers;

[0035] Get the bounding box detection result.

[0036] The bounding box regression branch uses GIoU loss for back propagation.

[0037]

[0038] L reg =1-GIOU(4.8)

[0039] In the formula, A and B represent the predicted box and the real box respectively, and C is the minimum bounding box of the real box and the predicted box.

[0040] The classification branch uses Focal loss for back propagation.

[0041] The calculation method is as follows,

[0042]

[0043] In the formula, y′ represents the classification confidence of the output, α and γ are the adjustment factors added by Focal loss to solve the imbalance of positive and negative samples, α is 0.25, and γ is 2.

[0044] The downsampling multiples of C2-C5 relative to the original image are 4, 8, 16, and 32 respectively; the downsampling multiples of P2-P6 relative to the original image are 4, 8, 16, 32, and 64 respectively.

[0045] The final loss function is the sum of classification and regression losses, and the calculation formula is shown in (4.10).

[0046]

[0047] Where N pos Represents the number of positive samples in the image, L cls is the classification branch loss function, L reg is the loss function of the bounding box regression branch.

[0048] The beneficial effects of the present invention are: based on a deep neural network, the working mechanism of an eagle eye in capturing a target is imitated, an existing target detection algorithm is improved, and a detection network is designed that can simultaneously imitate the eagle eye's large-scale perception ability and small-target focusing ability, aiming to enhance the network's receptive field while focusing on the salient position of the image.

[0049] The improved self-correction convolution module is used to adaptively expand the receptive field, while the intra-channel convolution module is used to focus on the target. In addition, a dual-branch bounding box prediction method is proposed, which uses different feature maps and different feature processing methods in classification and localization tasks to separate the differences in feature requirements of the two tasks. This chapter uses the Pyramid Vision Transformer (PVT) as the backbone network for feature extraction. The self-attention module of PVT has good local feature perception ability and can adapt to dense target scenes in drone images. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a structural diagram of a biological vision-inspired method for detecting targets in drone video images;

[0051] Figure 2 This is the structure diagram of the dual-branch bounding box prediction module;

[0052] Figure 3 This is the structure diagram of the self-correction convolution module;

[0053] Figure 4 This is the structure diagram of the inner convolution module;

[0054] Figure 5 It is the feature pyramid heat map comparison;

[0055] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. DETAILED DESCRIPTION

[0056] [Example 1]

[0057] A biological vision-inspired target detection method for drone video images, comprising the following steps:

[0058] PVT network, input the image to be detected into the non-convolutional Transformer network, namely PVT network, and extract four feature maps of different resolutions from C2 to C5 in layers;

[0059] The feature maps extracted and output by the FPN network and the PVT network are input into the FPN (feature pyramid) feature fusion connected from top to bottom to obtain 5 feature maps P2 to P6;

[0060] Extract positioning features from C2 to C5, and extract classification features from P2 to P6;

[0061] The classification features are sent to the bounding box regression branch of the RetinaNet detector head. After the classification features and positioning features are fused, they are sent to the classification branch of the RetinaNet detector head. The two are stacked through convolutional layers to obtain the bounding box detection results, that is, all objects that may belong to the target in the image are marked with a bounding box, and each marked target is classified.

[0062] Through the bounding box regression branch, the bounding box is regressed to obtain the bounding box regression result, and each object in the image that may be a target is selected; instead of identifying a specific target from the image.

[0063] Through the classification branch, the classification confidence is obtained, that is, the classification of each marked target is obtained.

[0064] Finally, the target's bounding box and the corresponding classification are fused together, NMS filters the overlapping boxes, and outputs the final result.

[0065] For example, in an image, three objects are marked with a common bounding box. The first object is classified as a table; the second object is classified as a chair; and the third object is classified as a cup.

[0066] This method mainly includes PVT and dual-branch prediction network. The dual-branch bounding box prediction network consists of a spatial domain self-correction convolution module and a channel domain convolution module. Other processing processes (such as FPN, NMS, classification and regression loss functions, etc.) are the same as the single-stage RetinaNet algorithm settings.

[0067] The downsampling multiples of C2-C5 relative to the original image are 4, 8, 16, and 32 respectively; the downsampling multiples of P2-P6 relative to the original image are 4, 8, 16, 32, and 64 respectively.

[0068] Backbone network: The currently popular non-convolutional Transformer backbone network, namely the PVT network, is used to hierarchically extract feature maps of four different resolutions, C2 to C5, with downsampling multiples of 4, 8, 16, and 32 relative to the original image respectively.

[0069] Like most single-stage algorithms, this chapter sets a top-down FPN after the backbone network for feature fusion, and obtains 5 feature maps P2~P6 which are downsampled to 4, 8, 16, 32, and 64 relative to the original image.

[0070] Dual-branch bounding box prediction module: The C2-C5 layer feature maps obtained by the backbone and the P2-P6 layer feature maps obtained by FPN are used in the final positioning and classification tasks respectively. C2-C5 are unified to the same number of channels of 256, and C2-C5 are sent to the self-correction convolution module to extract positioning features; P2-P6 are sent to the inner convolution module to extract classification features. The classification features are sent to the bounding box regression branch of the RetinaNet detector head. After the classification features and positioning features are fused, they are sent to the classification branch of the RetinaNet detector head. The bounding box detection results are obtained through stacked convolution layers, and the non-maximum suppression algorithm is used to eliminate overlapping boxes (non-maximum suppression is a default process for target detection).

[0071] In order to make the detector have the ability to focus and expand the receptive field, the inner convolution module and the self-correction convolution module are combined to design a dual-branch bounding box prediction method that decouples classification and regression. Different feature map acquisition methods and different feature processing methods are used in classification and positioning tasks. Specifically, the self-correction convolution module that can expand the receptive field is used to strengthen the positioning feature, and the inner convolution module with focusing effect is used to strengthen the classification feature.

[0072] like Figure 1 and Figure 2 As shown in the figure, the feature maps obtained by the backbone network are denoted as C2~C5, and the feature maps after FPN are denoted as P2~P6. Previous algorithms usually use P2~P6 to uniformly send them to the classification and bounding box regression subnetworks for prediction, while this method uses different schemes for prediction on the classification and regression branches. First, feature pyramids from different sources are used for the classification and regression tasks. Among them, the feature map of the classification task is taken from the top-down constructed FPN, and the rich semantic information brought by it is beneficial to the classification task. The feature map of the positioning task is taken from the PVT network, which aims to fully retain the fine-grained features such as the boundary and texture of the target. In addition, the processing methods on the classification and regression branches are also different. For the regression task that needs to fully obtain the target boundary information, the self-correction convolution module is used to obtain the adaptive receptive field of the target, and for the classification task that needs to have focusing ability, the inner convolution module with strong focusing ability is used to perceive the local dependency of pixels.

[0073] The classification features are sent to the border regression branch of the RetinaNet detector head, and bounding box regression is performed through 4 stacked convolutional layers; after the classification features and the positioning features are fused, they are sent to the classification branch of the RetinaNet detector head, and classification is performed through 4 stacked convolutional layers; and the bounding box detection result is obtained.

[0074] In the bounding box regression branch, the feature maps C2~C5 obtained by the PVT network are first sent to the self-correction convolution module to obtain an adaptive receptive field, and then pass through 4 stacked convolution layers for bounding box regression. The feature map of the backbone network is closer to the original image information, has rich fine-grained features including target boundaries and details, and is more suitable for target positioning tasks. In the classification branch, the feature maps P2~P6 obtained by FPN are sent to the inner convolution module for target focusing, and then pass through 4 stacked convolution layers for classification. The top-down connection path of FPN can bring deep semantic information to the shallow layer, so it is more suitable for classification. At the same time, this chapter also integrates the feature map obtained by the self-correction convolution module into the classification branch, and adds it to the feature map of the classification branch element by element, so that the feature information of the additional receptive range provided by the self-correction convolution module is also transmitted to the classification branch.

[0075] The bounding box regression branch uses GIoU loss for back propagation.

[0076]

[0077] L reg =1-GIOU(4.8)

[0078] In the formula, A and B represent the predicted box and the real box respectively, and C is the minimum bounding box of the real box and the predicted box.

[0079] The classification branch uses Focal loss for back propagation.

[0080] The calculation method is as follows,

[0081]

[0082] In the formula, y′ represents the classification confidence of the output, α and γ are the adjustment factors added by Focal loss to solve the imbalance of positive and negative samples, α is 0.25, and γ is 2.

[0083] The final loss function is the sum of classification and regression losses, and the calculation formula is shown in (4.10).

[0084]

[0085] Where N pos Represents the number of positive samples in the image, L cls is the classification branch loss function, L reg is the loss function of the bounding box regression branch.

[0086] During training, GIoU loss and Focal loss are used for back propagation of regression and classification branches respectively.

[0087] The calculation formula of GIoU loss is shown in (4.7).

[0088]

[0089] L reg =1-GIOU(4.8)

[0090] In the formula, A and B represent the predicted box and the true box respectively, and C is the minimum bounding box of the true box and the predicted box. The focal loss calculation formula used in the classification task is shown in (4.9).

[0091]

[0092] In the formula, y′ represents the classification confidence of the output, α and γ are the adjustment factors added by Focal loss to solve the imbalance of positive and negative samples. In this chapter, α is set to 0.25 and γ is set to 2.

[0093] The final loss function is the sum of classification and regression losses, and the calculation formula is shown in (4.10).

[0094]

[0095] Where N pos Represents the number of positive samples in the image. The specific training process and algorithm framework are shown in Table 4.1.

[0096] Table 4.1 Object detection method process inspired by biological vision

[0097]

[0098] The positioning features are extracted from C2 to C5 through the self-convolution correction module.

[0099] The self-convolution correction module is:

[0100] The input feature map is split into two parts in the channel dimension, namely X1 and X2. Then, the original convolution kernel is split into four parts, namely K1, K2, K3, and K4. K2, K3, and K4 are used to process the feature map X1 to obtain the output Y1, and K1 is used to process the feature map X2 to obtain the output Y2.

[0101] For the feature map X1, we first use the average pooling operation to downsample it by a factor of 2, and then perform a conventional convolution operation to upsample it to its original size. The specific process is shown in formula (4.1).

[0102] X1'=Upsample(AvgPool(X1)*K2)(4.1)

[0103] In the formula, Upsample represents bilinear interpolation upsampling, and * represents convolution operation;

[0104] Then, the original feature map X1 is connected to X1' through a residual connection structure, and after Sigmoid activation, it is multiplied element by element with the feature map obtained by Conv K3 convolution of the feature map X1. Finally, a 3×3 convolution with K4 as the convolution kernel is added to obtain the calculation result of the upper half branch. The calculation process is shown in formula (4.2).

[0105] Y1=(X1*K3·σ(X1+X′1))*K4 (4.2)

[0106] For feature map X2, first obtain the sampling position offsets offset for input feature X2, and use the offset to replace the square sampling position of conventional convolution, then use K1 as the deformable convolution kernel to perform deformable convolution operation on X2, so as to obtain the output Y2 of the lower half branch. The calculation of deformable convolution is as shown in formula (4.3);

[0107]

[0108] In the formula, x is the input feature map, y is the output feature map, w is the convolution kernel parameter, p0 is the output position, and p k ∈{(-1,-1),(-1,0),…,(0,1),(1,1)} represents the pixel neighborhood of the sampling center, Δp k represents the learnable offset;

[0109] Finally, the feature maps Y1 and Y2 output by the two branches of the self-correction convolution are cascaded to obtain the final result Y of the module.

[0110] The eagle, the top hunter in the sky, has a strong field of view capture ability. It can quickly and accurately identify ground prey with the help of the two foveas in its eyes. This ability to obtain a strong receptive field is also important for target detection tasks, especially drone target detection in high-altitude bird's-eye view scenes. In computer vision tasks, it is usually necessary to establish long-range dependencies to expand the receptive field, and focus on the key point position according to the attention mechanism of the local pixel area. In order to capture the interdependencies between different positions and channels on the feature map, a large number of channel, spatial and other types of attention mechanisms have emerged. For example, SENet obtains a strong response on the beneficial channel based on the correlation between channels, and the non-local module establishes long-range dependencies between pixels to enhance feature representation. In this method, a self-correction convolution module is designed in the regression subnetwork of target detection to expand the receptive field and adjust the receptive field range. The self-correction convolution module first reduces the size of the feature map by downsampling, and then restores it to its original size by upsampling, thereby expanding the receptive field of each spatial position, and then adaptively adjusts the receptive field of the feature map through deformable convolution.

[0111] The structure of the self-corrected convolution module is as follows: Figure 3 As shown. The self-correcting convolution module decomposes the entire feature transformation process into multiple parallel branches, and finally cascades the results of each branch to obtain the output. The self-correcting convolution module divides the convolution kernel parameters in the convolution so that different branches are responsible for different functions. This operation similar to grouped convolution ensures that the overall parameter quantity remains unchanged. Specifically, the self-correcting convolution module splits the input feature map into two parts in the channel dimension, namely X1 and X2. Then, the original convolution kernel is split into four parts, namely K1, K2, K3, and K4. K2, K3, and K4 are used to process the feature map X1 to obtain the output Y1, and K1 is used to process the feature map X2 to obtain the output Y2.

[0112] For the feature map X1, we first use the average pooling operation to downsample it by a factor of 2, and then perform a conventional convolution operation to upsample it to its original size. The specific process is shown in formula (4.1).

[0113] X1'=Upsample(AvgPool(X1)*K2)(4.1)

[0114] In the formula, Upsample represents bilinear interpolation upsampling, and * represents convolution operation. According to the characteristics of convolutional neural networks, the larger the downsampling step of the feature map relative to the original image, the larger the receptive range. Therefore, the receptive field of the feature map obtained in this step will also be doubled. Then, the original feature map X1 is connected to X1' through a residual connection structure, which can maintain the original favorable information and alleviate the gradient descent problem to a certain extent. After Sigmoid activation, the feature map obtained by the Conv K3 convolution with the feature map X1 is element-by-element multiplied, so that the expanded additional receptive field can be applied to the original feature map, thereby guiding the feature transformation process in the original feature space. Finally, a 3×3 convolution with K4 as the convolution kernel is added to obtain the calculation result of the upper half branch. The calculation process is shown in formula (4.2).

[0115] Y1=(X1*K3·σ(X1+X′1))*K4 (4.2)

[0116] Considering that the receptive field enhanced by downsampling is still a regular square grid, it cannot adaptively adjust the receptive field according to the target scale. Therefore, this chapter integrates deformable convolution into the self-correction convolution module so that the network can adaptively adjust the receptive field of the feature map according to the input feature map. Figure 4As shown in the figure, a deformable convolution module is set in the lower branch of the self-correcting convolution module. First, the sampling position offset for the input feature X2 is obtained, and the square sampling position of the conventional convolution is replaced by the offset. Then, K1 is used as the deformable convolution kernel to perform a deformable convolution operation on X2, so as to obtain the output Y2 of the lower branch. The calculation of the deformable convolution is shown in Formula (4.3).

[0117]

[0118] In the formula, x is the input feature map, y is the output feature map, w is the convolution kernel parameter, p0 is the output position, and p k ∈{(-1,-1),(-1,0),…,(0,1),(1,1)} represents the pixel neighborhood of the sampling center, Δp k Represents a learnable offset. The deformable convolution enables the network to add a learnable offset to each pixel, thereby adaptively adjusting the convolution kernel sampling position according to the target and realizing self-correction of the receptive field. Finally, the Y1 and Y2 feature maps output by the two branches of the self-correction convolution are cascaded to obtain the final result Y of the module.

[0119] The self-correcting convolution module is equivalent to splitting the original convolution kernel into four parts, so the number of parameters of each convolution kernel is one-fourth of the original, which is the same as the number of parameters of conventional convolution but does not bring additional computational overhead. In addition, the self-correcting convolution divides the original convolution operation into two parallel paths, the upper part is used to expand the receptive field, and the lower part is used for adaptive sampling, which achieves the expansion of the receptive field while dynamically adjusting it according to the input features.

[0120] The classification features are extracted from P2 to P6 through the inner convolution module.

[0121] The inner convolution module process is to obtain the unique convolution kernel parameters of each position and perform the inner convolution operation. The feature map with input dimensions (H, W, C) is first subjected to a convolution linear transformation operation with a kernel size of 1×1 to obtain a weight map with a size of (H, W, K×K), where K represents the kernel size of the inner convolution.

[0122] Calculate the convolution kernel weight for each pixel position in the weight map, that is, generate a convolution kernel H for each position of the input image i,j , H i,j =φ(X i,j )=W1σ(W0X i,j ) (4.4)

[0123] In the formula, X i,j represents the value of the input feature map at (i, j), W0 and W1 represent the parameters of two 1×1 convolution kernels, and σ represents the BN layer and nonlinear activation function;

[0124] For a generated inner convolution kernel H i,j , which acts on a specific spatial location The output for each position is as follows,

[0125]

[0126] In the formula, Δ K is the neighborhood offset set of the convolution center position, K is the size of the generated convolution kernel, and k is the channel number.

[0127] When observing prey, eagles are usually able to capture areas with stronger contrast with surrounding areas, so their visual system can quickly locate salient areas containing targets, which is inspiring for computer vision tasks. The conventional convolution layer establishes a window for each pixel, and calculates the local weighted sum within the window range through specific convolution parameters, which can get a stronger response in the area containing the target. However, the sliding window strategy of the conventional convolution layer uses the same convolution kernel at each position, and cannot determine different convolution parameters based on pixel differences, and cannot achieve the super-strong focusing ability of the eagle's eyes. This method uses an inner convolution network to improve the algorithm's ability to focus on pixels, and uses a dynamic convolution kernel that depends on the input pixel at each spatial pixel position to simulate the eagle's eyes' super-strong focusing ability on the target.

[0128] Different channels of the conventional convolution layer are responsible for collecting different encoding information, and the convolution kernel parameters are not shared on the channel but use the same convolution parameters in space. The inner convolution module of this method adopts a completely different approach, that is, each pixel position has a different weight in space, but has the same weight in the channel. Therefore, the inner convolution can adaptively assign different convolution kernels to each spatial position for operation. The pixel position containing the target can obtain a more discriminative response after the inner convolution operation, giving priority to the visual elements with the largest amount of information in the spatial domain, thereby achieving rapid focusing on the target pixel position.

[0129] The calculation process of inner convolution is as follows Figure 4 As shown in Figure 1, it mainly includes two steps: obtaining the unique convolution kernel parameters for each position and performing the inner convolution operation. For an input dimension of (H, W,

[0130] C), first undergoes a convolution linear transformation operation with a kernel size of 1×1 to obtain a weight map of size (H, W, K×K), where K represents the kernel size of the inner convolution and is set to 3 in this chapter. Each pixel position in the weight map can be regarded as a K×K vector, and the vector is expanded to be the convolution kernel weight of this pixel position, which is equivalent to generating a convolution kernel H for each position of the input image. i,j, and share weights on different channels. The calculation process is shown in formula (4.4). Finally, the generated convolution kernel is used to perform inner convolution operation at each pixel position.

[0131] H i,j =φ(X i,j )=W1σ(W0X i,j )(4.4)

[0132] Where, X i,j represents the value of the input feature map at (i, j), W0 and W1 represent the parameters of two 1×1 convolution kernels, and σ represents the BN layer and nonlinear activation function.

[0133] Compared with conventional convolution, inner convolution is also calculated by weighted sum of kernel and pixel points. The difference is that inner convolution shares the same kernel in different channels, and each pixel position generates a kernel unique to that position, that is, inner convolution conforms to channel invariance and spatial specificity. In the implementation process, this chapter expands the inner convolution kernel to C times, where C is the number of channels of the input feature map, and then uses depthwise separable convolution to obtain the final result. For a generated inner convolution kernel H i,j , which acts on a specific spatial location The output at each position is shown in formula (4.5).

[0134]

[0135] In the formula, Δ K is the neighborhood offset set of the convolution center position, K is the size of the generated convolution kernel, and k is the channel number.

[0136] Inner convolution generates different responses for different pixels by dynamically generating convolution kernels, which can be used to simulate the process of focusing on the target in an eagle's eye. Therefore, this chapter uses the inner convolution module in the classification branch of the detector head to make each spatial position of the feature map more discriminative, so as to achieve the purpose of focusing on the area of ​​interest for accurate classification.

[0137] Experimental data

[0138] In order to verify the influence of the dual-branch bounding box prediction method, self-corrected convolution module and inner convolution module on the target detection accuracy, this chapter tests the accuracy of the dual-branch prediction method, self-corrected convolution, inner convolution and the combination of dual-branch, self-corrected convolution and inner convolution on the UAVDT dataset. For fair comparison, all experiments use FPN and RetinaNet as baselines. The ablation experiment results are shown in Table 4.4.

[0139] Table 4.4 Ablation experiment results

[0140]

[0141] As shown in Table 4.4, the dual-branch prediction module can bring a 1.3 gain in AP index to the baseline, and AP 50 and AP 75 In the experiment of combining the dual-branch bounding box prediction method, the self-correction convolution module and the inner convolution module, this chapter embeds the self-correction convolution module and the inner convolution module into the regression and classification branches respectively. The experimental results show that the combination of the three can bring huge benefits to the detection accuracy, with an overall AP improvement of 2.1 and AP 50 Increased AP by 2.8 75 The improvement is 1.7, which confirms that the self-correction convolution module and the inner convolution module provide better spatial location information and classification features for the target, and verifies that the combination of the three can jointly promote the network's learning ability for classification and regression tasks.

[0142] Comparative experiments with other algorithms

[0143] This chapter compares the proposed method with 11 state-of-the-art target detection methods, including two-stage methods, single-stage methods, anchor-free methods and Transformer-based methods. The detection results are shown in the following table.

[0144] Table 4.5 Performance comparison of different methods on the XUPTDT dataset

[0145]

[0146] As shown in Table 4.5, the detection performance of this method on the XUPTDT dataset is better than most existing algorithms, with an AP of 72.8 and a detection speed of 56.3FPS. 50 ,AP 75 and AP l This method achieves the best detection performance. 50 and AP 75 To a certain extent, it shows the algorithm's ability to distinguish foreground and background, proving that this method is effective in simulating eagle eyes to focus on objects. l The improvement shows that the self-corrected convolution module in this method can provide a larger receptive field for large-scale targets, thereby improving the detection performance of large objects. The AP of this method is 0.4 lower than that of CascadeR-CNN, the most accurate two-stage algorithm. The reason is that this method only performs classification and regression in a single stage and lacks the multi-stage refinement process in Cascade R-CNN. However, the detection speed of this method is 17.1FPS higher than that of Cascade R-CNN, which greatly improves the detection speed with a small loss of accuracy. In addition, the algorithm in this chapter has an AP of 1.s and AP m The reason is that the self-correction convolution module and the inner convolution module in this method do not optimize the multi-scale feature fusion stage, but AP s and AP m It still achieves better performance than most algorithms. The experimental results show that the method of simulating eagle eyes to locate and focus targets is effective.

[0147] In order to further verify the effectiveness of the algorithm proposed in this chapter, the performance of different target detections was verified on the UAVDT dataset, and the results are shown in Table 4.6.

[0148] Table 4.6 Performance comparison of different methods on the UAVDT dataset

[0149]

[0150] As shown in Table 4.6, this method also has excellent detection performance on the UAVDT dataset, proving that the proposed method has strong generalization performance. Compared with the RFEPNet proposed in Chapter 3, the AP 50 and AP 75 Both of them surpass RFEPNet. Our method achieves better detection performance at a higher IoU threshold, which proves that our method can better distinguish foreground objects from background. m In terms of performance, this method is slightly inferior to RFEPNet. The reason is that the cross-level context fusion module and residual feature enhancement module used in RFEPNet bring more discriminative features to small targets and targets with large scale changes, while this method focuses on solving the problems of acquiring and focusing the receptive field of the target.

[0151] like Figure 5 As shown, in order to verify the effect of the method proposed in this application in expanding the receptive field and focusing on the area of ​​interest, a heat map analysis is performed on the feature map after passing through the two modules.

[0152] Figure 5 (a)~ Figure 5 (e) is the heat map of the 5-layer feature map of the feature pyramid before the double branch. Figure 5 (f)~ Figure 5 (j) is the feature pyramid heat map used for regression after the self-correction convolution module. Figure 5 (k)~ Figure 5 (o) represents the feature pyramid heat map used for classification after passing through the inner convolution module. Figure 5 It can be seen that although the original feature pyramid can generate strong responses for target areas to a certain extent, the activation values ​​for some small targets are small, the background interference of some similar targets cannot be distinguished well, and the target activation area is also small. Figure 5 (f)~ Figure 5 (j) It can be seen that after the self-correction convolution module including deformable convolution, the network receptive field is significantly expanded, and activation areas of different sizes are generated for targets of different scales. Deformable convolution can offset the receptive field, thereby adaptively obtaining the position of the target and expanding the receptive range. Figure 5 (k)~ Figure 5 (o) It can be seen that the contrast between the foreground target and the background in the heat map after the inner convolution module is greater, and the activation response value near the target area is greater than that of the background area, which is beneficial to the classification task. The inner convolution module plays a significant role mainly because the inner convolution module generates a non-shared convolution kernel for each position, allowing the network to efficiently learn the difference between the foreground and the background and generate a discriminative feature map.

[0153] In summary, this method proposes a dual-branch target detection algorithm that decouples classification and regression tasks, and improves the existing detection algorithm from two aspects: expanding the perception range and focusing on the target. The spatial self-correction module is introduced to simulate the mechanism of eagle eyes to obtain a wide field of view. The upsampling branch and deformable convolution in the self-correction convolution are used to expand the network perception range and adaptively correct the sampling position; the inner convolution module is introduced to simulate the convergence effect of eagle eyes on the target, and a higher weight is generated for the target area of ​​interest by dynamically generating convolution kernels, thereby reducing the interference caused by the complex background of aerial images. On the self-built drone aerial image dataset XUPTDT, the average accuracy of this method reached 72.8, and the detection speed reached 56.3FPS, surpassing other target detection algorithms.

Claims

1. A biological vision-inspired target detection method for drone video images, characterized in that: The following steps are involved: PVT network, input the image to be detected into the non-convolutional Transformer network, namely PVT network, and extract four feature maps of different resolutions from C2 to C5 in layers; The feature maps extracted and output by the FPN network and the PVT network are input into the FPN feature fusion connected from top to bottom to obtain 5 feature maps P2 to P6; Extract positioning features from C2 to C5, and extract classification features from P2 to P6; The classification features are sent to the bounding box regression branch of the RetinaNet detector head. After the classification features and positioning features are fused, they are sent to the classification branch of the RetinaNet detector head. The two are stacked through convolutional layers to obtain the bounding box detection results, that is, all objects that may belong to the target in the image are marked with a bounding box, and each marked target is classified.

2. The method for detecting targets in drone video images inspired by biological vision according to claim 1, characterized in that: The positioning features are extracted from C2 to C5 through the self-convolution correction module.

3. The method for detecting target in drone video images inspired by biological vision according to claim 2, characterized in that: The self-convolution correction module is: The input feature map is split into two parts in the channel dimension, namely X1 and X2. Then, the original convolution kernel is split into four parts, namely K1, K2, K3, and K4. K2, K3, and K4 are used to process the feature map X1 to obtain the output Y1, and K1 is used to process the feature map X2 to obtain the output Y2. For the feature map X1, we first use the average pooling operation to downsample it by a factor of 2, and then perform a conventional convolution operation to upsample it to its original size. The specific process is shown in formula (4.1). X1'=Upsample(AvgPool(X1)*K2)(4.1) In the formula, Upsample represents bilinear interpolation upsampling, and * represents convolution operation; Then, the original feature map X1 is connected to X1' through a residual connection structure, and after Sigmoid activation, it is multiplied element by element with the feature map obtained by Conv K3 convolution of the feature map X1. Finally, a 3×3 convolution with K4 as the convolution kernel is added to obtain the calculation result of the upper half branch. The calculation process is shown in formula (4.2). Y1=(X1*K3·σ(X1+X1'))*K4 (4.2) For feature map X2, first obtain the sampling position offsets offset for input feature X2, and use the offset to replace the square sampling position of conventional convolution, then use K1 as the deformable convolution kernel to perform deformable convolution operation on X2, so as to obtain the output Y2 of the lower half branch. The calculation of deformable convolution is as shown in formula (4.3); In the formula, x is the input feature map, y is the output feature map, w is the convolution kernel parameter, p0 is the output position, and p k ∈{(-1,-1),(-1,0),…,(0,1),(1,1)} represents the pixel neighborhood of the sampling center, Δp k represents the learnable offset; Finally, the feature maps Y1 and Y2 output by the two branches of the self-correction convolution are cascaded to obtain the final result Y of the module.

4. The method for detecting targets in drone video images inspired by biological vision according to claim 1, characterized in that: The classification features are extracted from P2 to P6 through the inner convolution module.

5. The method for detecting target in drone video images inspired by biological vision according to claim 4, characterized in that: The inner convolution module process is to obtain the unique convolution kernel parameters of each position and perform the inner convolution operation. The feature map with input dimensions (H, W, C) is first subjected to a convolution linear transformation operation with a kernel size of 1×1 to obtain a weight map with a size of (H, W, K×K), where K represents the kernel size of the inner convolution. Calculate the convolution kernel weight for each pixel position in the weight map, that is, generate a convolution kernel H for each position of the input image i,j , H i,j =φ(X i,j )=W1σ(W0X i,j ) (4.4) Where, X i,j represents the value of the input feature map at (i, j), W0 and W1 represent the parameters of two 1×1 convolution kernels, and σ represents the BN layer and nonlinear activation function; For a generated inner convolution kernel H i,j , which acts on a specific spatial location The output for each position is as follows, In the formula, Δ K is the neighborhood offset set of the convolution center position, K is the size of the generated convolution kernel, and k is the channel number.

6. The method for detecting targets in drone video images inspired by biological vision according to claim 1, characterized in that: The classification features are fed into the bounding box regression branch of the RetinaNet detector head and are subjected to bounding box regression through four stacked convolutional layers. After the classification features and positioning features are fused, they are sent to the classification branch of the RetinaNet detector head and classified through 4 stacked convolutional layers; Get the bounding box detection result.

7. A biological vision-inspired drone video image target detection method according to claim 1 or 6, characterized in that: The bounding box regression branch uses GIoU loss for back propagation. L reg =1-GIOU (4.8) In the formula, A and B represent the predicted box and the real box respectively, and C is the minimum bounding box of the real box and the predicted box.

8. The method for detecting target in drone video images inspired by biological vision according to claim 1 or 6, characterized in that: The classification branch uses Focal loss for back propagation. The calculation method is as follows, In the formula, y′ represents the classification confidence of the output, α and γ are the adjustment factors added by Focalloss to solve the imbalance of positive and negative samples, α is 0.25, and γ is 2.

9. The method for detecting target in drone video images inspired by biological vision according to claim 1, characterized in that: The downsampling multiples of C2-C5 relative to the original image are 4, 8, 16, and 32 respectively; the downsampling multiples of P2-P6 relative to the original image are 4, 8, 16, 32, and 64 respectively.

10. The method for detecting target in drone video images inspired by biological vision according to claim 1, characterized in that: The final loss function is the sum of classification and regression losses, and the calculation formula is shown in (4.10). Where N pos Represents the number of positive samples in the image, L cls is the classification branch loss function, L reg is the loss function of the bounding box regression branch.

Images