Unmanned aerial vehicle target detection method based on spiking neural network

Abstract

The invention discloses an unmanned aerial vehicle target detection method based on a pulse neural network, and the method reduces the delay through a directly trained pulse neural network, enhances the extraction of spatial-temporal features through a pulse multi-scale attention mechanism, and improves the accuracy of target detection. The method comprises the steps of obtaining a to-be-detected image of an unmanned aerial vehicle, and performing pulse coding on the to-be-detected image; inputting the pulse signal into a pulse residual backbone network formed by pulse residual basic blocks for feature extraction, and generating a multi-scale and multi-level feature map; inputting the feature map into a pulse multi-scale attention mechanism for processing; and inputting the processed feature map into a detection head, carrying out target detection, and outputting a result. According to the method, richer image feature information can be extracted, the method has the advantages of efficient calculation performance and low energy consumption, and low delay is realized through a relatively small time step length.

Description

Technical Field

[0001] The invention belongs to the technical field of neural network and is a UAV target detection method based on pulse neural network. Background Art

[0002] Unmanned aerial vehicles (UAVs) have been widely used in civil and military fields due to their high cost-effectiveness, flexibility and efficiency, thanks to the decline in manufacturing costs and the advancement of flight control technology. UAV target detection has important applications in power line inspection, plant growth disaster analysis, emergency rescue, public safety monitoring, traffic management and other fields. However, since UAVs usually fly at high altitudes, the images they capture have a wide field of view and the background is complex and changeable, which increases the difficulty of target detection.

[0003] The limited energy of drones is mainly used to maintain flight, so their efficiency is greatly limited when performing image recognition and processing. Although drone target detection methods based on traditional neural networks (ANNs) can optimize calculations through model pruning and parameter quantization, they still rely on a large number of floating-point operations, which not only consumes a lot of computing resources, but also significantly increases energy consumption, which has certain limitations for drone platforms with limited resources.

[0004] As the third generation of artificial neural network, the spiking neural network (SNN) shows extremely high energy efficiency due to its event-driven mechanism and efficient computing characteristics. SNN simulates the working mode of biological brain, uses binary pulse signals (0 or 1) instead of traditional continuous real values ​​for calculation, and triggers sparse addition only when the pulse is emitted, otherwise it remains idle, thereby greatly reducing the computational complexity and energy consumption. However, in the process of implementing the present invention, it is found that the existing spiking neural network still has the problem of insufficient accuracy in drone target detection. Summary of the invention

[0005] The present invention provides a method for detecting drone targets based on a spiking neural network, aiming to solve the problem of low accuracy of current spiking neural networks in detecting drone targets. The method significantly improves target detection performance and effectively reduces detection delay.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for detecting unmanned aerial vehicle targets based on a spiking neural network comprises the following steps:

[0008] Step 1: Obtain the image of the drone to be detected and pulse encode it;

[0009] Step 2: Input the pulse signal into the pulse residual backbone network composed of the pulse residual basic block to extract features and obtain feature maps of different scales and levels;

[0010] Step 3: Input the feature map obtained in step 2 into the pulse multi-scale attention mechanism for processing;

[0011] Step 4: Input the feature map processed in step 3 into the detection head, perform target detection and output the detection result.

[0012] As a preferred solution, the pulse residual basic block includes a residual path and a short-circuit path; the pulse basic block includes LIF neurons, convolution (Conv) and batch normalization (BN);

[0013] The LIF neuron converts the neuron floating point inputs at different time steps into binary pulse representations (0 or 1) through nonlinear operations;

[0014] The convolutional layer extracts spatial features, while the BN layer stabilizes the training process by normalizing the output;

[0015] O block =BN(Conv(LIF(X block )))

[0016] Where: X block Represents the floating point input, O block represents the output of the impulse basic block; in the residual path, two impulse basic blocks are used consecutively, while in the shortcut path, a max pooling layer is applied for downsampling to reduce parameters; the impulse-based output from the shortcut path is then concatenated with the output of the residual path after passing through another impulse basic block.

[0017] As a preferred solution, the step 1 is specifically as follows:

[0018] Pulse encoding is performed on the input UAV image to be detected to obtain an image pulse signal; wherein the input image to be detected is a visible light image or an infrared image, and the format of the image to be detected is RGB format.

[0019] As a preferred solution, the step 2 is specifically as follows:

[0020] The impulse residual backbone network consists of impulse residual basic blocks, including N layers of impulse basic blocks, where N is an integer greater than 2; feature maps of different scales and hierarchical structures are extracted from the image impulse signal; the time step of the impulse residual basic block is 2.

[0021] As a preferred solution, the step three is specifically as follows:

[0022] The feature maps obtained in step 2 are fed into the pulse multi-scale attention mechanism for feature enhancement; the pulse multi-scale attention mechanism uses feature grouping to divide the features into multiple sets across different resources, ensuring that each group is well distributed in space; then the feature maps of these groups are processed in parallel through 1×1 and 3×3 attention branches;

[0023] X=[X 0 ,X 1 ,X i ,…,X G ],X i ∈R T×(C / / G)×H×W

[0024] Where: The input feature tensor X is expressed as X = [X 0 ,X 1 ,X i ,…,X G ], where: X i is the i-th subgroup feature, satisfying X i ∈R T×(C / / G)×H×W ; R is a matrix, indicating the dimension of the parameter tensor; T represents the number of time steps, C is the total number of channels, G represents the number of groups, H and W are the height and width of the feature map respectively; by setting G<<C, the subgroup convolution takes advantage of the parallel computing power of modern hardware to process simultaneously across different groups;

[0025] P=GN(σ(f PW (AvgPool H (X S )))⊙σ(f PW (AvgPool W (X S )))⊙X S )

[0026] Where: σ represents the Sigmod function, ⊙ is the element-wise dot product, GN is the group normalization, AvgPool H Represents the global average pooling operation along the height dimension H, AvgPool W represents the global average pooling operation along the width dimension W; P is the point-wise convolution f applied by combining the global average pooling results of the two spatial dimensions (H, W). PW The output is calculated from; where: AvgPool H (X S )∈R T×(C / / G)×1×W and AvgPool H (X S )∈R T×(C / / G)×H×1 Respectively represent the global average pooling operation along the height and width dimensions; X S ∈R T×(C / / G)×H×W Represents the input feature map;

[0027] In the 1×1 attention branch Att 1×1 In the 1×1 convolution operation, no dimensionality reduction is required, so channel feature integration and spatial attention calculation are performed;

[0028]

[0029] in: represents matrix multiplication, AvgPool(P)∈R T×(C / / G)×1×1 represents the global average pooling of P, thus producing a compact feature map that captures the global context; in the 3×3 attention branch Att 3×3 In the example, a 3×3 convolutional layer f is stacked. 3×3 , used to capture multi-scale feature representation and local context information:

[0030]

[0031] Where: AvgPool(f 3×3 (X S )) is the intermediate feature map after 3×3 convolution and subsequent pooling; two attention maps Att 1×1 and Att 3×3 The shape is R T×1×H×W , which enables them to effectively capture spatial dependencies while maintaining inter-channel interactions;

[0032] Att S =σ(Att 1×1 +Att 3×3 )

[0033] Fusion of multi-scale attention maps Att S The final output feature map O is obtained by summing the outputs of the 1×1 and 3×3 branches, performing Sigmoid activation, and then multiplying it element-wise with the input feature map X. att :

[0034] O att =X⊙Att S .

[0035] As a preferred solution, the step 4 is specifically as follows:

[0036] On the feature map obtained in step 3, the last membrane potential of the neuron is used as input, which is a five-dimensional feature map T×B×C×H×W. By averaging the T dimension, a four-dimensional feature map B×C×H×W is obtained. T represents the number of time steps, B represents the batch size, C represents the total number of channels, and H and W represent the height and width of the feature map respectively; then multiple anchor boxes of different scales are used for prediction; each anchor box needs to calculate the matching degree with the target and predict the category and position of the target; after predicting the category and position of each anchor box, the non-maximum suppression algorithm (NMS) is used to remove overlapping detection results, and only the results with the highest confidence are retained; finally, the bounding box and category of the target are obtained based on the prediction results, and the detection results are output.

[0037] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0038] Aiming at the problem of high energy consumption of current drones based on traditional neural networks, the present invention designs a drone target detection algorithm based on direct training of pulse neural networks. With a time step of only 2, it is possible to maintain high detection accuracy at a low latency. By introducing a pulse multi-scale attention mechanism that includes time and channel dimensions, it not only improves the feature extraction capability of the network, but also significantly reduces the pulse emission rate of the detection head, thereby effectively reducing energy consumption and improving overall performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is the pulse residual backbone network part of the present invention.

[0040] Figure 2 It is the basic block structure of the pulse residual in the model of the present invention.

[0041] Figure 3 It is the pulse multi-scale attention mechanism structure of the present invention.

[0042] Figure 4 It is a step diagram of the detection method in the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described below in conjunction with the embodiments.

[0044] The present invention provides a UAV target detection method based on a pulse neural network, which includes pulse coding of UAV images, extracting features through a pulse residual backbone network, and then enhancing features through a pulse multi-scale attention mechanism, and finally outputting target detection results. Figure 1-4 The invention is further described in the embodiments.

[0045] A method for detecting unmanned aerial vehicle targets based on a spiking neural network comprises the following steps:

[0046] Step 1: Obtain the image of the drone to be detected and pulse encode it;

[0047] The step 1 is specifically as follows:

[0048] Pulse encoding is performed on the input UAV image to be detected to obtain an image pulse signal; wherein the input image to be detected is a visible light image or an infrared image, and the format of the image to be detected is RGB format.

[0049] Step 2: Input the pulse signal into the pulse residual backbone network composed of the pulse residual basic block to extract features and obtain feature maps of different scales and levels;

[0050] The impulse residual basic block includes a residual path and a short-circuit path; the impulse basic block includes LIF neurons, convolution (Conv) and batch normalization (BN);

[0051] The LIF neuron converts the neuron floating point inputs at different time steps into binary pulse representations (0 or 1) through nonlinear operations;

[0052] The convolutional layer extracts spatial features, while the BN layer stabilizes the training process by normalizing the output;

[0053] O block =BN(Conv(LIF(X block )))

[0054] Where: X block Represents the floating point input, O block represents the output of the impulse basic block; in the residual path, two impulse basic blocks are used consecutively, while in the shortcut path, a max pooling layer is applied for downsampling to reduce parameters; the impulse-based output from the shortcut path is then concatenated with the output of the residual path after passing through another impulse basic block.

[0055] The step 2 is specifically as follows:

[0056] The pulse residual backbone network is composed of pulse residual basic blocks, including N layers of pulse basic blocks, where N is greater than 2.

[0057] An integer; feature maps of different scales and hierarchical structures are extracted from the image impulse signal; the time step of the impulse residual basic block is 2.

[0058] Step 3: Input the feature map obtained in step 2 into the pulse multi-scale attention mechanism for processing;

[0059] The step three is specifically as follows:

[0060] The feature maps obtained in step 2 are fed into the pulse multi-scale attention mechanism for feature enhancement; the pulse multi-scale attention mechanism uses feature grouping to divide the features into multiple sets across different resources, ensuring that each group is well distributed in space; then the feature maps of these groups are processed in parallel through 1×1 and 3×3 attention branches;

[0061] X=[X 0 ,X 1 ,X i ,…,X G ],X i ∈R T×(C / / G)×H×W

[0062] Where: The input feature tensor X is expressed as X = [X 0 ,X 1 ,X i ,…,X G ], where: X i is the i-th subgroup feature, satisfying X i ∈R T×(C / / G)×H×W ; R is a matrix, indicating the dimension of the parameter tensor; T represents the number of time steps, C is the total number of channels, G represents the number of groups, H and W are the height and width of the feature map respectively; by setting G<<C, the subgroup convolution takes advantage of the parallel computing power of modern hardware to process simultaneously across different groups;

[0063] P=GN(σ(f PW (AvgPool H (X S )))⊙σ(f PW (AvgPool W (X S )))⊙X S )

[0064] Where: σ represents the Sigmod function, ⊙ is the element-wise dot product, GN is the group normalization, AvgPool H Represents the global average pooling operation along the height dimension H, AvgPool W represents the global average pooling operation along the width dimension W; P is the point-wise convolution f applied by combining the global average pooling results of the two spatial dimensions (H, W). PW The output is calculated from; where: AvgPool H (X S )∈R T×(C / / G)×1×W and AvgPool H (X S )∈R T×(C / / G)×H×1 Respectively represent the global average pooling operation along the height and width dimensions; X S ∈R T×(C / / G)×H×WRepresents the input feature map;

[0065] In the 1×1 attention branch Att 1×1 In the 1×1 convolution operation, no dimensionality reduction is required, so channel feature integration and spatial attention calculation are performed;

[0066]

[0067] in: represents matrix multiplication, AvgPool(P)∈R T×(C / / G)×1×1 represents the global average pooling of P, thus producing a compact feature map that captures the global context; in the 3×3 attention branch Att 3×3 In the example, a 3×3 convolutional layer f is stacked. 3×3 , used to capture multi-scale feature representation and local context information:

[0068]

[0069] Where: AvgPool(f 3×3 (X S )) is the intermediate feature map after 3×3 convolution and subsequent pooling; two attention maps Att 1×1 and Att 3×3 The shape is R T×1×H×W , which enables them to effectively capture spatial dependencies while maintaining inter-channel interactions;

[0070] Att S =σ(Att 1×1 +Att 3×3 )

[0071] Fusion of multi-scale attention maps Att S The final output feature map O is obtained by summing the outputs of the 1×1 and 3×3 branches, performing Sigmoid activation, and then multiplying it element-wise with the input feature map X. att :

[0072] O att =X⊙Att S .

[0073] Step 4: Input the feature map processed in step 3 into the detection head, perform target detection and output the detection result.

[0074] The step 4 is specifically as follows: on the feature map obtained in step 3, the last membrane potential of the neuron is used as input, which is a five-dimensional feature map T×B×C×H×W. By averaging the T dimension, a four-dimensional feature map B×C×H×W is obtained. Wherein T represents the number of time steps, B is the batch size, C is the total number of channels, and H and W represent the height and width of the feature map respectively; then multiple anchor boxes of different scales are used for prediction; each anchor box needs to calculate the matching degree with the target, and predict the category and position of the target; after predicting the category and position of each anchor box, the non-maximum suppression algorithm (NMS) is used to remove overlapping detection results, and only the result with the highest confidence is retained; finally, the bounding box and category of the target are obtained according to the prediction results, and the detection result is output.

[0075] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A UAV target detection method based on pulse neural network, characterized in that: The following steps are involved: Step 1: Obtain the image of the drone to be detected and pulse encode it; Step 2: Input the pulse signal into the pulse residual backbone network composed of the pulse residual basic block to extract features and obtain feature maps of different scales and levels; Step 3: Input the feature map obtained in step 2 into the pulse multi-scale attention mechanism for processing; Step 4: Input the feature map processed in step 3 into the detection head, perform target detection and output the detection result.

2. The method for detecting unmanned aerial vehicle targets based on a pulse neural network according to claim 1, characterized in that: The pulse residual basic block includes a residual path and a short-circuit path; the pulse basic block includes a LIF neuron, a convolution Conv and a batch normalization BN; The LIF neuron converts the neuron floating point inputs at different time steps into binary pulses representing 0 or 1 through nonlinear operations; The convolutional layer extracts spatial features, while the BN layer stabilizes the training process by normalizing the output; O block =BN(Conv(LIF(X block ))) Where: X block Represents the floating point input, O block represents the output of the impulse basic block; in the residual path, two impulse basic blocks are used consecutively, while in the shortcut path, a max pooling layer is applied for downsampling to reduce parameters; the impulse-based output from the shortcut path is then concatenated with the output of the residual path after passing through another impulse basic block.

3. The method for detecting unmanned aerial vehicle targets based on a pulse neural network according to claim 1, characterized in that: The step 1 is specifically as follows: Pulse encoding is performed on the input UAV image to be detected to obtain an image pulse signal; wherein the input image to be detected is a visible light image or an infrared image, and the format of the image to be detected is RGB format.

4. The method for detecting unmanned aerial vehicle targets based on a pulse neural network according to claim 1, characterized in that: The step 2 is specifically as follows: The impulse residual backbone network consists of impulse residual basic blocks, including N layers of impulse basic blocks, where N is an integer greater than 2; feature maps of different scales and hierarchical structures are extracted from the image impulse signal; the time step of the impulse residual basic block is 2.

5. The method for detecting unmanned aerial vehicle targets based on a pulse neural network according to claim 1, characterized in that: The step three is specifically as follows: The feature maps obtained in step 2 are fed into the pulse multi-scale attention mechanism for feature enhancement; the pulse multi-scale attention mechanism uses feature grouping to divide the features into multiple sets across different resources, ensuring that each group is well distributed in space; then the feature maps of these groups are processed in parallel through 1×1 and 3×3 attention branches; X=[X0,X1,X i ,…,X G ],X i ∈R T×(C / / G)×H×W Among them: The input feature tensor X is expressed as X = [X0, X1, X i ,…,X G ], where: X i is the i-th subgroup feature, satisfying X i ∈R T×(c / / G)×H×W ; R is a matrix, indicating the dimension of the parameter tensor; T represents the number of time steps, C is the total number of channels, G represents the number of groups, H and W are the height and width of the feature map respectively; by setting G<<C, the subgroup convolution takes advantage of the parallel computing power of modern hardware to process simultaneously across different groups; P=GN(σ(f PW (AvgPool H (X S )))⊙σ(f PW (AvgPool W (X S )))⊙X S ) Where: σ represents the Sigmod function, ⊙ is the element-wise dot product, GN is the group normalization, AvgPool H Represents the global average pooling operation along the height dimension H, AvgPool W represents the global average pooling operation along the width dimension W; P is the point-wise convolution f applied by combining the global average pooling results of the two spatial dimensions (H, W). PW The output is calculated from; where: AvgPool H (X S )∈R T×(C / / G)×1×W and AvgPool H (X S )∈R T×(C / / G)×H×1 Respectively represent the global average pooling operation along the height and width dimensions; X S ∈R T×(G / / G)×H×W Represents the input feature map; In the 1×1 attention branch AAt 1×1 In the 1×1 convolution operation, no dimensionality reduction is required, so channel feature integration and spatial attention calculation are performed; in: represents matrix multiplication, AvgPool(P)∈R T×(C / / G)×1×1 represents the global average pooling of P, thus producing a compact feature map that captures the global context; in the 3×3 attention branch Att 3×3 In the example, a 3×3 convolutional layer f is stacked. 3 ×3 , used to capture multi-scale feature representation and local context information: Where: AvgPool(f 3×3 (X S )) is the intermediate feature map after 3×3 convolution and subsequent pooling; two attention maps Att 1×1 and Att 3×3 The shape is R T×1×H×W , To S =σ(To 1×1 +To 3×3 ) Fusion of multi-scale attention maps Att S The final output feature map O is obtained by summing the outputs of the 1×1 and 3×3 branches, performing Sigmoid activation, and then multiplying it element-wise with the input feature map X. att : THE att =X⊙Att S 。 6. The target detection method based on cross-modal fusion of unmanned aerial vehicles according to claim 1 is characterized in that: The step 4 is specifically as follows: On the feature map obtained in step 3, the last membrane potential of the neuron is used as input, which is a five-dimensional feature map T×B×C×H×W. By averaging the T dimension, a four-dimensional feature map B×C×H×W is obtained; where: T represents the number of time steps, B is the batch size, C is the total number of channels, H and W represent the height and width of the feature map respectively; then multiple anchor boxes of different scales are used for prediction; each anchor box needs to calculate the degree of match with the target, and predict the category and position of the target; after predicting the category and position of each anchor box, the non-maximum suppression algorithm NMS is used to remove overlapping detection results, and only the results with the highest confidence are retained; finally, the bounding box and category of the target are obtained based on the prediction results, and the detection results are output.

Images