Flying object target detection method and device based on pulse neural network improvement

Abstract

The invention discloses an improved flying object target detection method and device based on a pulse neural network, relates to the technical field of target detection, and mainly aims to solve the problem of low target detection accuracy of a flying object in the prior art. The method mainly comprises the steps of obtaining a target detection image and event data, and generating an image frame sequence based on event driving according to the target detection image and the event data. The image frame sequence is converted into pulse features through a pulse feature extraction layer, feature extraction is carried out on the pulse features step by step through feature extraction sub-networks, and image enhancement features corresponding to the feature extraction sub-networks are obtained; and performing multi-scale feature fusion on the plurality of image enhancement features through the neck network to obtain a fusion feature corresponding to each feature fusion sub-network, and performing target identification on the respective corresponding fusion feature through each target detection layer to obtain a flying object target detection result. The method is mainly used for flying object target detection.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a flying object target detection method and device based on an improved pulse neural network. Background Art

[0002] Object detection is one of the core tasks in the field of computer vision, which aims to automatically identify and locate the target objects of interest, such as people, vehicles, animals, etc., from images or videos. Its application range is wide, covering security monitoring, drone flight driving, autonomous driving, medical image analysis, human-computer interaction, smart retail, augmented reality and many other fields.

[0003] Existing target detection methods mainly rely on image data acquired by frame-based cameras that capture frames at a fixed rate. Although frame-based cameras provide dense intensity information, they often have problems with limited dynamic range and relatively low frame rate. In application scenarios such as low-light conditions and fast-moving objects, the captured images have a lot of motion blur, which in turn affects the richness and accuracy of features in the target detection process, resulting in low target detection accuracy for flying objects. Summary of the invention

[0004] In view of this, the present invention provides a flying object target detection method and device based on an improved pulse neural network, the main purpose of which is to solve the problem of low accuracy of existing target detection methods for flying objects.

[0005] According to one aspect of the present invention, there is provided a flying object target detection method based on an improved spiking neural network, comprising: Target recognition is performed by using a target detection model improved based on a pulse neural network, wherein the target detection model includes a backbone network for pulse conversion and pulse feature extraction, a neck network for multi-scale feature fusion, and a head network for multi-scale target recognition, the backbone network includes a pulse feature extraction layer and multiple feature extraction subnetworks based on a lightweight attention mechanism, the neck network includes multiple feature fusion subnetworks of different scales, and the head network includes multiple target detection layers of different scales. The method includes: Acquire a target detection image and event data of the target detection image, and generate an event-driven image frame sequence based on the target detection image and the event data, wherein the target detection image includes an object that moves rapidly relative to an image acquisition device; The image frame sequence is converted into pulse features through the pulse feature extraction layer, and the pulse features are gradually extracted through each feature extraction sub-network to obtain the image enhancement features corresponding to each feature extraction sub-network; multi-scale feature fusion is performed on multiple image enhancement features through the neck network to obtain the fusion features corresponding to each feature fusion sub-network, and target recognition is performed on the corresponding fusion features through each target detection layer to obtain the flying object target detection result.

[0006] Furthermore, the backbone network sequentially includes a pulse feature extraction layer, a first feature extraction subnetwork, a second feature extraction subnetwork, a third feature extraction subnetwork and a fast spatial pyramid pooling subnetwork; The step of gradually extracting the pulse features through each of the feature extraction sub-networks to obtain image enhancement features corresponding to each feature extraction sub-network includes: The pulse feature is input into the first feature extraction subnetwork to gradually perform feature extraction through each feature extraction subnetwork and the fast spatial pyramid pooling subnetwork, and the output of the first feature extraction subnetwork is obtained as the second image enhancement feature, the output of the second feature extraction subnetwork is obtained as the second image enhancement feature, the output of the third feature extraction subnetwork is obtained as the third image enhancement feature, and the output of the fast spatial pyramid pooling subnetwork is obtained as the fourth image enhancement feature.

[0007] Furthermore, the pulse feature extraction layer includes a neuron sublayer, a convolution sublayer and a batch normalization sublayer, and the image frame sequence is converted into a pulse feature through the pulse feature extraction layer, including: Extracting spatial features and temporal features of the image frame sequence through a neuron sublayer, fusing the spatial features and the temporal features to generate a membrane potential function, and generating a pulse signal according to a membrane potential threshold and the membrane potential function; The pulse signal is subjected to graph feature extraction through the convolution sublayer to obtain a pulse signal feature graph, and the temporal dependency of the pulse signal feature graph is captured through a batch normalization sublayer to obtain a pulse feature.

[0008] Furthermore, each of the feature extraction subnetworks includes a feature extraction layer that introduces a receptive field spatial attention mechanism and a residual connection layer that introduces a channel and spatial attention mechanism, the feature extraction layer includes a first convolution branch and a second convolution branch, and the feature extraction subnetwork processes the input features, including: Performing cross-channel feature interaction and combination processing on the input features through the first convolution branch to obtain an attention map; The first convolution branch is used to capture local spatial information and feature representations of different scales in the input feature to obtain a receptive field spatial feature, and the attention map and the receptive field spatial feature are fused to obtain a first image enhancement feature, wherein the number of convolution kernels of the second convolution branch is greater than the number of convolution kernels of the first convolution branch; The first image enhancement feature is subjected to feature extraction based on channel space fusion attention through the residual connection layer to obtain an image enhancement feature.

[0009] Furthermore, each residual connection layer includes two pulse feature extraction sublayers and a lightweight attention sublayer in sequence, the lightweight attention sublayer includes a channel attention branch and a spatial feature branch, and the image enhancement feature is obtained by performing feature extraction based on channel space fusion attention on the first image enhancement feature through the residual connection layer, including: Stepwise feature extraction of the first image enhancement feature is performed through two pulse feature extraction sub-layers to obtain a second image enhancement feature; A global spatial descriptor of the second image enhancement feature is obtained through a channel attention branch, and a channel attention map is calculated based on the global spatial descriptor, and spatial feature information of the second image enhancement feature is expanded through a spatial feature branch to generate cross-channel average pooling features and maximum pooling features, and cross-channel spatial features are obtained by concatenating and convolving the average pooling features and the maximum pooling features, and the channel attention map and the cross-channel spatial features are fused to obtain a third image enhancement feature; The first image enhancement feature and the third image enhancement feature are fused to obtain an image enhancement feature.

[0010] Further, the neck network includes a first feature fusion subnetwork, a second feature fusion subnetwork, a third feature fusion subnetwork, a fourth feature fusion subnetwork, a fifth feature fusion subnetwork, a sixth feature fusion subnetwork and a seventh feature fusion subnetwork in sequence, and the fusion features include a first fusion feature, a second fusion feature, a third fusion feature and a fourth fusion feature with decreasing granularity in sequence; The multi-scale feature fusion of multiple image enhancement features through the neck network includes: Inputting a plurality of image enhancement features into the neck network, and extracting the output of the first feature fusion subnetwork as the first fusion feature, extracting the output of the third feature fusion subnetwork as the second feature fusion feature, extracting the output of the fifth feature fusion subnetwork as the third fusion feature, and extracting the output of the seventh fusion subnetwork as the fourth fusion feature; Among them, each feature fusion sub-network includes a residual connection layer, and the third feature fusion sub-network and the seventh feature fusion sub-network also include a pulse feature extraction layer for feature fusion.

[0011] Furthermore, the head network includes a small target detection layer, a first target detection layer, a second target detection layer and a third target detection layer, and each target detection layer performs target recognition on the corresponding fusion features to obtain a flying object target detection result, including: The small target detection layer performs target recognition based on spatiotemporal features on the first fusion feature to obtain a small target detection result, specifically including: performing an average pooling operation on the first fusion feature to obtain an average pooling feature, performing a maximum pooling operation on the first fusion feature to obtain a maximum pooling feature, performing weighted fusion of the average pooling operation and the maximum pooling feature through a shared multi-layer perceptron to obtain a temporal attention map, extracting a global spatial feature of the first fusion feature, and fusing the temporal attention map and the global spatial feature to generate a spatiotemporal fusion feature map, performing flying object positioning and classification processing according to the spatiotemporal fusion feature map, and obtaining a small target detection result; Performing target recognition on the second fused feature through the first target detection layer to obtain a first target detection result, performing target recognition on the third fused feature through the second target detection layer to obtain a second target detection result, and performing target recognition on the fourth fused feature through the third target detection layer to obtain a third target detection result; A positioning frame is determined based on the small target detection result, the first target detection result, the second target detection result and the third target detection result, and a flying object target detection result is generated according to the positioning frame.

[0012] Further, the acquiring of the target detection image and the event data of the target detection image, and generating an event-driven image frame sequence based on the target detection image and the event data, includes: According to the timestamp corresponding to each pixel in the event data, an event set corresponding to different timestamps is constructed, wherein the event set includes the position coordinates, timestamps and polarities of the pixels corresponding to different timestamps; Converting the event data into a frame sequence according to a timestamp-based functional relationship between a pulse pattern tensor and an event set; The frame sequence is spliced ​​with the target detection image to obtain an event-driven image frame sequence.

[0013] Furthermore, before performing target recognition by using the target detection model improved based on the spiking neural network, the method further includes: Acquire a training data set, wherein the training data set includes event-driven image frame sequence samples marked with calibration frames; Constructing an initial target detection model, wherein the initial target detection model includes a backbone network, a neck network and a head network; the backbone network includes a pulse feature extraction layer and multiple feature extraction subnetworks based on a lightweight attention mechanism, each feature extraction subnetwork sequentially includes a feature extraction layer that introduces a receptive field spatial attention mechanism and a residual connection layer that introduces a channel and spatial attention mechanism; the neck network sequentially includes a first feature fusion subnetwork, a second feature fusion subnetwork, a third feature fusion subnetwork, a fourth feature fusion subnetwork, a fifth feature fusion subnetwork, a sixth feature fusion subnetwork and a seventh feature fusion subnetwork, and the head network includes a small target detection layer, a first target detection layer, a second target detection layer and a third target detection layer; The event data is converted into frame sequence samples using the initial target detection model to perform target detection processing to obtain a prediction frame; Calculating a first distance between the calibration frame and the upper left corner point of the prediction frame, and a second distance between the calibration frame and the upper right corner point of the prediction frame; Constructing a loss function according to the first distance, the second distance and an attention-weighted loss term; The initial target detection model is trained using the loss function to obtain a trained target detection model.

[0014] According to another aspect of the present invention, there is provided a flying object target detection device based on an improved pulse neural network, comprising: an acquisition module and a target detection module, wherein the target detection module performs target recognition through a target detection model based on an improved pulse neural network, wherein the target detection model comprises a backbone network for pulse conversion and pulse feature extraction, a neck network for multi-scale feature fusion, and a head network for multi-scale target recognition. The target detection model comprises a backbone network for pulse conversion and pulse feature extraction, a neck network for multi-scale feature fusion, and a head network for multi-scale target recognition, wherein the backbone network comprises a pulse feature extraction layer and a plurality of feature extraction sub-networks based on a lightweight attention mechanism, wherein the neck network comprises a plurality of feature fusion sub-networks of different scales, and the head network comprises a plurality of target detection layers of different scales; The acquisition module is used to acquire a target detection image and event data of the target detection image, and generate an event-driven image frame sequence based on the target detection image and the event data, wherein the target detection image includes an object that moves rapidly relative to the image acquisition device; The target detection module is used to convert the image frame sequence into pulse features through the pulse feature extraction layer, and gradually extract the pulse features through each feature extraction sub-network to obtain the image enhancement features corresponding to each feature extraction sub-network; perform multi-scale feature fusion on multiple image enhancement features through the neck network to obtain the fusion features corresponding to each feature fusion sub-network, and perform target recognition on the corresponding fusion features through each target detection layer to obtain the flying object target detection result.

[0015] By means of the above technical solution, the technical solution provided by the embodiment of the present invention has at least the following advantages: The present invention provides a flying object target detection method and device improved based on a pulse neural network. The embodiment of the present invention obtains a target detection image and event data of the target detection image, and generates an event-driven image frame sequence based on the target detection image and the event data, wherein the target detection image includes an object that moves rapidly relative to an image acquisition device; the image frame sequence is converted into a pulse feature through a pulse feature extraction layer in a target detection model improved based on a pulse neural network, and the pulse feature is gradually extracted through each feature extraction subnetwork to obtain an image enhancement feature corresponding to each feature extraction subnetwork; a plurality of image enhancement features are multi-scale feature fused through a neck network to obtain a fusion feature corresponding to each feature fusion subnetwork, and target recognition is performed on the corresponding fusion features through each target detection layer to obtain a flying object target detection result, thereby realizing target detection based on image data and event data, processing event data, converting the event-driven image frame sequence into a pulse signal, and performing convolution feature extraction on the pulse signal, thereby solving the problem that the existing convolution neural network cannot perform feature extraction on event data, and while realizing the processing of event-driven image data, the amount of data calculation is greatly reduced. In addition, feature extraction based on the image frame sequence that combines image static data and event data can effectively capture dynamic targets and reduce static background interference while supplementing high-latitude features, thereby improving target recognition accuracy in low-light environments and images of high-speed moving objects.

[0016] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings: Figure 1 A flow chart of a flying object target detection method based on an improved spiking neural network provided by an embodiment of the present invention is shown; Figure 2 A schematic diagram of an improved target detection model architecture based on a spiking neural network provided by an embodiment of the present invention is shown; Figure 3 A schematic diagram of the architecture of a pulse feature extraction layer provided by an embodiment of the present invention is shown; Figure 4 A schematic diagram showing the principle of neuron pulse emission provided by an embodiment of the present invention is shown; Figure 5 A schematic diagram of the architecture of a residual connection layer based on lightweight attention provided by an embodiment of the present invention is shown; Figure 6 A schematic diagram of a processing process of a lightweight attention sublayer provided by an embodiment of the present invention is shown; Figure 7 A schematic diagram of the architecture of a small target detection layer provided by an embodiment of the present invention is shown; Figure 8 A schematic diagram of a processing process of a spatiotemporal fusion feature extraction sublayer provided by an embodiment of the present invention is shown; Fig. 9 A block diagram of a flying object target detection device improved based on a pulse neural network provided in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0018] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0019] In view of the problem that the existing target detection methods have low accuracy in detecting flying objects, the embodiment of the present invention provides a flying object target detection method based on an improved pulse neural network, such as Figure 1 As shown, the method includes: 101. Acquire a target detection image and event data of the target detection image, and generate an event-driven image frame sequence based on the target detection image and the event data.

[0020] In an embodiment of the present invention, the image to be detected includes an object that moves rapidly relative to the image acquisition device. For example, in application scenarios such as unmanned aerial vehicles, intelligent traffic monitoring, and security patrols, in order to achieve flight safety monitoring and obstacle avoidance, images are collected based on the image acquisition device configured by the flight equipment. Among them, the image acquisition device is a camera based on a dynamic visual sensor, also known as an event camera. The event camera captures the scene asynchronously through pixel-level illumination changes, provides lower latency and higher dynamic range, and can capture images with less motion blur. However, the data generated by the event camera includes two parts: the image to be detected and the event data corresponding to each pixel in the image to be detected. In order to make full use of these two parts of data, it is necessary to convert the event data, and integrate the converted event data and the image to be detected to obtain an image frame sequence based on event driving. It includes the position coordinates, timestamp and polarity of the current pixel. The polarity represents the increase or decrease in brightness, which can be represented by +1 / -1.

[0021] 102. Target recognition is performed through an improved target detection model based on pulse neural network.

[0022] The target detection model includes a backbone network for pulse conversion and pulse feature extraction, a neck network for multi-scale feature fusion, and a head network for multi-scale target recognition. The backbone network includes a pulse feature extraction layer and multiple feature extraction subnetworks based on a lightweight attention mechanism, the neck network includes multiple feature fusion subnetworks of different scales, and the head network includes multiple target detection layers of different scales.

[0023] In the embodiment of the present invention, based on the pulse neurons in the pulse feature extraction layer, the image frame sequence containing event data is converted into a pulse signal, and the pulse signal is convolutionally extracted to obtain the pulse feature, and the image frame sequence is converted into a pulse feature through the pulse feature extraction layer. Then, different feature extraction subnetworks are connected in sequence, and after the pulse feature is input into the first feature extraction subnetwork, each feature extraction subnetwork gradually performs feature extraction, and the output feature of the previous feature extraction subnetwork is used as the input of the current feature extraction subnetwork, and the output of the current feature extraction subnetwork is used as the input of the next feature extraction subnetwork. The pulse feature is gradually extracted through each feature extraction subnetwork to obtain the image enhancement feature corresponding to each feature extraction subnetwork. Among them, the structure of each feature extraction subnetwork is the same, and they are all feature extraction subnetworks based on the lightweight attention mechanism. This feature extraction subnetwork includes a feature extraction layer that introduces the receptive field space attention mechanism and a residual connection layer that introduces the channel and space attention mechanism. The neck network performs multi-scale feature fusion on multiple image enhancement features to obtain the fusion feature corresponding to each feature fusion subnetwork, and each target detection layer performs target recognition on the corresponding fusion features to obtain the flying object target detection result.

[0024] It should be noted that by converting the event-driven image frame sequence into a pulse signal and performing convolution feature extraction on the pulse signal, the problem that the existing convolutional neural network cannot extract features from event data is solved. While realizing the processing of event-driven image data, the amount of data calculation is also greatly reduced. In addition, feature extraction based on the image frame sequence combining image static data and event data can effectively capture dynamic targets and reduce static background interference while supplementing high-dimensional features, thereby improving the target recognition accuracy in low-light environments and high-speed moving object images.

[0025] In one embodiment of the present invention, for further explanation and limitation, the step of gradually extracting the pulse features through each feature extraction subnetwork to obtain the image enhancement features corresponding to each feature extraction subnetwork includes: The pulse feature is input into the first feature extraction subnetwork to gradually perform feature extraction through each feature extraction subnetwork and the fast spatial pyramid pooling subnetwork, and the output of the first feature extraction subnetwork is obtained as the second image enhancement feature, the output of the second feature extraction subnetwork is obtained as the second image enhancement feature, the output of the third feature extraction subnetwork is obtained as the third image enhancement feature, and the output of the fast spatial pyramid pooling subnetwork is obtained as the fourth image enhancement feature.

[0026] In the embodiment of the present invention, the backbone network includes a pulse feature extraction layer, a first feature extraction subnetwork, a second feature extraction subnetwork, a third feature extraction subnetwork and a fast spatial pyramid pooling subnetwork. Figure 2 As shown, after the event-driven image frame sequence is input into the backbone network, the pulse feature extraction layer first performs pulse conversion and pulse feature extraction, and the pulse feature output by the pulse feature extraction layer is used as the input feature of the first feature extraction subnetwork to start the step-by-step feature extraction by multiple feature extraction subnetworks and the fast spatial pyramid pooling subnetwork. Specifically, the output of the first feature extraction subnetwork is used as the input feature of the second feature extraction subnetwork, the output of the second feature extraction subnetwork is used as the input feature of the third feature extraction subnetwork, and the output of the third feature extraction subnetwork is used as the input feature of the fast spatial pyramid pooling subnetwork. The output of each feature extraction subnetwork is used as the input feature of the next feature extraction subnetwork and also as the output feature of the backbone network, that is, the first feature extraction subnetwork outputs the second image enhancement feature, the second feature extraction subnetwork outputs the second image enhancement feature, the third feature extraction subnetwork outputs the third image enhancement feature, and the fast spatial pyramid pooling subnetwork outputs the fourth image enhancement feature. It should be noted that the fast spatial pyramid pooling subnetwork includes the fourth feature extraction subnetwork and the fast spatial pyramid pooling layer.

[0027] In one embodiment of the present invention, for further explanation and limitation, the step of converting the image frame sequence into pulse features through the pulse feature extraction layer includes: Extracting spatial features and temporal features of the image frame sequence through a neuron sublayer, fusing the spatial features and the temporal features to generate a membrane potential function, and generating a pulse signal according to a membrane potential threshold and the membrane potential function; The pulse signal is subjected to graph feature extraction through the convolution sublayer to obtain a pulse signal feature graph, and the temporal dependency of the pulse signal feature graph is captured through a batch normalization sublayer to obtain a pulse feature.

[0028] In the embodiment of the present invention, Figure 3 As shown in the figure, the pulse feature extraction layer includes a neuron sublayer, a convolution sublayer, and a batch normalization sublayer (Threshold-dependent batch normalization, TDBN). Based on the simulation of the complex spatiotemporal dynamic characteristics of biological neurons and the trade-off of mathematical forms, neurons can adopt a (leaky integrate-and-fire) neuron model. Figure 4As shown in Figure 1, the neuron processing process includes: first, the input data is gradually extracted through multi-layer perceptron (MLP), convolution, normalization, and average pooling to extract spatial features from the image frame sequence. , and the spatial features of the current moment And time input (time features of the previous layer at the current moment) Integrated into membrane potential , generates a spatial pulse tensor for the next layer (n+1 layer) of neurons and generates a new neuron state at the next moment (t+1). Greater than or equal to threshold When , the pulse tensor will be output , the membrane potential gradually decays to the resting potential , get the time characteristics of the current time t , at the same time, the pulse tensor The input is the n+1 layer of neurons. If the membrane potential Less than threshold , then at time t, the neurons in the nth layer do not emit pulses, , and the time characteristics Gradually decay, waiting for the next input accumulation. Through the above process, the pulse domain features of the image frame sequence including event data are refined and converted into pulses, and pulse-based convolution feature extraction is performed to realize the processing of event data and improve the computational density of the neural network. In order to solve the problem that the pulse cannot be differentiated in back propagation, the above pulse emission process is expressed by the proxy gradient formula: ; Among them, i represents the i-th neuron in the n-th layer of neurons, represents the pulse tensor output by the i-th neuron, represents the membrane potential output by the ith neuron, This ensures that the integral of the gradient is 1 and determines the steepness of the curve. represents the pulse generating function.

[0029] After performing pulse-based convolution feature extraction and obtaining the pulse signal feature map, the batch normalization sublayer processes the pulse signal feature map, optimizes the feature representation by modeling the temporal dependency, and considers both the time domain and the spatial domain, thereby improving the accuracy of the extracted features. The batch normalization sublayer processes the pulse signal feature map as follows: ; ; in, represents the membrane potential generated by the i-th neuron in the n+1-th neuron layer at the t+1 time step, represents the membrane potential generated by the i-th neuron in the n+1-th neuron layer at time step t, represents the pulse tensor generated by the i-th neuron in the n+1-th neuron layer at time step t, represents the neuronal leakage attenuation factor, It represents the total input of the i-th neuron at time step t+1, which is usually the weighted sum of the output pulses of the neurons in the previous layer. represents the mean of each channel under the mini-batch sequence input, Represents the variation value of each channel under the small batch sequence input, , represents a tiny constant to avoid division by zero, and represents the trainable parameters, represents a threshold-dependent hyperparameter, represents the synaptic weight between the i-th and j-th neurons.

[0030] In one embodiment of the present invention, for further explanation and limitation, the feature extraction subnetwork processes the input features, including: Performing cross-channel feature interaction and combination processing on the input features through the first convolution branch to obtain an attention map; Capturing local spatial information and feature representations of different scales in the input features through the first convolution branch to obtain a receptive field spatial feature, and fusing the attention map and the receptive field spatial feature to obtain a first image enhancement feature; The first image enhancement feature is subjected to feature extraction based on channel space fusion attention through the residual connection layer to obtain an image enhancement feature.

[0031] In an embodiment of the present invention, each of the feature extraction subnetworks includes a feature extraction layer that introduces a receptive field spatial attention mechanism and a residual connection layer that introduces a channel and spatial attention mechanism. The feature extraction layer that introduces the receptive field spatial attention mechanism includes a first convolution branch and a second convolution branch. The number of convolution kernels of the second convolution branch is greater than the number of convolution kernels of the first convolution branch. The first convolution branch can be a convolution branch with a convolution kernel of 1×1, and the second convolution branch can be a convolution branch with a convolution kernel of In the first convolution branch, in order to reduce the complexity of the network structure and the number of parameters, first, the global feature information is obtained through the average pooling layer, and then the channel information is interacted through the 1×1 convolution operation. The Softmax function is used to emphasize the importance of different feature information in space, and the corresponding weights are obtained to obtain the attention map. Set the attention map to It can be expressed as: ; in, represents average pooling, Represents a convolution with a kernel size of 1×1.

[0032] In the second convolution branch, different receptive field space feature structures and sliding steps are formed according to different sizes of convolution kernels. The receptive field space is dynamically adjusted through the convolution kernel slider, and the corresponding weights are generated by interactive fusion with the attention information. The receptive field space features are prioritized to dynamically adjust the receptive field space. The size of the convolution operation is grouped, the feature information of the receptive field space is expanded, and the number of feature information channels is increased to times to ensure that adjacent feature information does not overlap, generating the same Corresponding receptive field spatial features , which can be expressed as: ; in, Indicates that the convolution kernel size is The convolution of represents normalization, represents the rectified linear unit activation function.

[0033] Combined with attention map and receptive field spatial features ,Right now × As the first image enhancement feature, dynamically adjusting the receptive field space can solve the computational redundancy problem caused by repeated feature extraction and traditional convolution kernel parameter sharing, thereby reducing the amount of calculation and improving data processing efficiency.

[0034] In one embodiment of the present invention, for further explanation and limitation, the first image enhancement feature is subjected to feature extraction based on channel space fusion attention through the residual connection layer to obtain the image enhancement feature, including: Stepwise feature extraction of the first image enhancement feature is performed through two pulse feature extraction sub-layers to obtain a second image enhancement feature; A global spatial descriptor of the second image enhancement feature is obtained through a channel attention branch, and a channel attention map is calculated based on the global spatial descriptor, and spatial feature information of the second image enhancement feature is expanded through a spatial feature branch to generate cross-channel average pooling features and maximum pooling features, and cross-channel spatial features are obtained by concatenating and convolving the average pooling features and the maximum pooling features, and the channel attention map and the cross-channel spatial features are fused to obtain a third image enhancement feature; The first image enhancement feature and the third image enhancement feature are fused to obtain an image enhancement feature.

[0035] In the embodiment of the present invention, Figure 5 As shown in Figure 1, each residual connection layer includes two pulse feature extraction sublayers and a lightweight attention sublayer in sequence. The pulse feature extraction sublayer has the same structure combination as the pulse feature extraction layer in the backbone network. After two layers of pulse feature extraction, the second image enhancement feature is obtained, which then needs to pass through the lightweight attention sublayer to obtain the third image enhancement feature expressed as , and then link to the previous layer (n-1 layer) Moment Get image enhancement features , this process is expressed as Among them, the lightweight attention sublayer includes the channel attention branch and the spatial feature branch. Figure 6 The extraction process of the channel attention branch includes: the input features pass through the global average pooling layer with a size of (avgpool) obtains the global space descriptor and the maximum pooling result size with the input feature is Perform linear processing, Rectified Linear Unit (ReLU) function and Sigmoid activation function together, and perform shared weighting of the two layers to obtain a one-dimensional channel attention map It can be expressed as: ; in, , Indicates the number of channels after convolution and other operations, represents the average pooling result, represents the maximum pooling result, represents the sigmoid function, Represents the weight of the i-th layer under the current number of channels, represents the weight of the jth layer under the current number of channels, that is and are the shared MLP weights at different time steps, is the channel dimension reduction factor. Figure 6 As shown in the figure, the feature extraction process of the spatial feature branch includes content expansion of the spatial feature information through grouped convolution (the feature size is ), after normalization and linear correction (feature size is ), and then after tensor shape adjustment (feature size is ), after the above normalization operation, average pooling and maximum pooling are performed to aggregate the channel information of the feature map to generate two 2D feature maps represented as . Each feature map represents the average pooling feature and the maximum pooling feature across channels. The two feature maps are then connected and convolution is used to generate cross-channel spatial features. ( ). Among them, the above representation of feature size is the number of channels, is the number of convolution kernels, is the feature map size width × height, and the convolution kernel is The cross-channel spatial features can be expressed as: ; in, , , represents the sigmoid function, Represents the image height after the maximum pooling operation, Represents the width of the image after the maximum pooling operation.

[0036] In one embodiment of the present invention, for further explanation and limitation, multi-scale feature fusion is performed on multiple image enhancement features through the neck network, including: Multiple image enhancement features are input into the neck network, and the output of the first feature fusion subnetwork is extracted as the first fusion feature, the output of the third feature fusion subnetwork is extracted as the second feature fusion feature, the output of the fifth feature fusion subnetwork is extracted as the third fusion feature, and the output of the seventh fusion subnetwork is extracted as the fourth fusion feature.

[0037] In the embodiment of the present invention, Figure 2 As shown, the neck network includes the first feature fusion subnetwork, the second feature fusion subnetwork, the third feature fusion subnetwork, the fourth feature fusion subnetwork, the fifth feature fusion subnetwork, the sixth feature fusion subnetwork and the seventh feature fusion subnetwork in sequence, and the fusion features include the first fusion feature, the second fusion feature, the third fusion feature and the fourth fusion feature with decreasing granularity in sequence. Among them, each feature fusion subnetwork includes a residual connection layer, and the third feature fusion subnetwork and the seventh feature fusion subnetwork also include a pulse feature extraction layer for feature fusion. In addition, each feature fusion subnetwork also includes a splicing sublayer and an upsampling sublayer. Among them, the larger the granularity, the larger the image scale of the corresponding feature and the larger the spatial receptive field. Conversely, the smaller the granularity, the smaller the image scale of the corresponding feature and the smaller the spatial receptive field.

[0038] In one embodiment of the present invention, for further explanation and limitation, the small target detection layer performs target recognition based on spatiotemporal features on the first fusion feature to obtain a small target detection result; Performing target recognition on the second fused feature through the first target detection layer to obtain a first target detection result, performing target recognition on the third fused feature through the second target detection layer to obtain a second target detection result, and performing target recognition on the fourth fused feature through the third target detection layer to obtain a third target detection result; A positioning frame is determined based on the small target detection result, the first target detection result, the second target detection result and the third target detection result, and a flying object target detection result is generated according to the positioning frame.

[0039] In the embodiment of the present invention, Figure 2 As shown in FIG, the head network includes a small target detection layer, a first target detection layer, a second target detection layer, and a third target detection layer. The architecture of the small target detection layer is as follows: Figure 7 As shown in the figure, it includes a classification branch and a bounding box branch. The two branches have the same structure. From the input end, they are the pulse feature extraction sublayer, the spatiotemporal fusion feature sublayer and the convolution sublayer. During the training process, the corresponding loss functions are constructed based on the output of the convolution sublayer. The process of the spatiotemporal fusion feature sublayer is shown in the figure. Figure 8 As shown, it specifically includes: performing an average pooling operation on the first fusion feature to obtain an average pooling feature (the feature size is ), by performing the maximum pooling operation on the first fusion feature, the maximum pooling feature is obtained (the feature size is ), the average pooling operation and the maximum pooling feature are weighted fused by sharing a multi-layer perceptron to obtain the temporal attention map, and the global spatial feature of the first fusion feature is extracted based on the grouped convolution (the feature size is ), and fuse the temporal attention map with the global spatial features to generate a spatiotemporal fusion feature map. The flying object positioning and classification processing are performed based on the spatiotemporal fusion feature map to obtain the small target detection result. In order to enhance the detection of small targets, the internal features of the network in the time dimension are refined by mining the cross-time step relationship of the feature blocks, thereby achieving both performance gain and energy consumption reduction. First, the spatial channel information of the feature blocks is aggregated at each time step through average pooling and maximum pooling operations to generate two different temporal context descriptors, representing the average pooling feature and the maximum pooling feature respectively; then, the average pooling and maximum pooling features are converted into a temporal attention map through a shared MLP. , detect the time characteristics of flying targets. Temporal attention map It can be expressed as: ; in, , , represents average pooling, represents the result of the maximum pooling layer, Represents the number of time steps after operations such as convolution, represents the sigmoid function, and is the weight of the linear layer i and j in the shared MLP, represents the temporal dimensionality reduction factor used to control the computational burden of the multilayer perceptron.

[0040] Then, the global spatial enhancement feature is extracted by the first fusion feature, and a global spatial feature map is generated based on grouped convolution, normalization and activation function, and then the temporal attention map is fused. Generate spatiotemporal fusion feature map.

[0041] In one embodiment of the present invention, for further explanation and limitation, the step of acquiring a target detection image and event data of the target detection image, and generating an event-driven image frame sequence based on the target detection image and the event data, includes: According to the timestamp corresponding to each pixel in the event data, construct an event set corresponding to different timestamps; Converting the event data into a frame sequence according to a timestamp-based functional relationship between a pulse pattern tensor and an event set; The frame sequence is spliced ​​with the target detection image to obtain an event-driven image frame sequence.

[0042] In the embodiment of the present invention, the event set includes the position coordinates, timestamps and polarities of the pixels corresponding to different timestamps. The spatial resolution of the image is set to , the pulse pattern tensor Equal to timestamp The event set ,in, Indicates pixels, ( ， ) indicates the The position coordinates of the pixels, Indicates the polarity of the event. Based on the timestamp-based functional relationship between the pulse pattern tensor and the event set, the event data is converted into a frame form to obtain a frame sequence. The functional relationship is expressed as follows: ; Indicates the frame form of the input event data converted to at time t, where t∈{1,2,…,T} is the time step, is the element-wise addition function, Represents the temporal resolution factor, which determines how many consecutive events are included in each frame.

[0043] In one embodiment of the present invention, for further explanation and limitation, before performing target recognition by using the target detection model improved based on the spiking neural network, the method further includes: Get the training dataset; Build an initial target detection model; The event data is converted into frame sequence samples using the initial target detection model to perform target detection processing to obtain a prediction frame; Calculating a first distance between the calibration frame and the upper left corner point of the prediction frame, and a second distance between the calibration frame and the upper right corner point of the prediction frame; Constructing a loss function according to the first distance, the second distance and an attention-weighted loss term; The initial target detection model is trained using the loss function to obtain a trained target detection model.

[0044] In an embodiment of the present invention, the training data set includes event-driven image frame sequence samples marked with calibration boxes. The initial target detection model includes a backbone network, a neck network and a head network; the backbone network includes a pulse feature extraction layer and multiple feature extraction subnetworks based on a lightweight attention mechanism, each feature extraction subnetwork includes a feature extraction layer that introduces a receptive field spatial attention mechanism and a residual connection layer that introduces a channel and spatial attention mechanism; the neck network includes a first feature fusion subnetwork, a second feature fusion subnetwork, a third feature fusion subnetwork, a fourth feature fusion subnetwork, a fifth feature fusion subnetwork, a sixth feature fusion subnetwork and a seventh feature fusion subnetwork, and the head network includes a small target detection layer, a first target detection layer, a second target detection layer and a third target detection layer. Setting is the distance between the upper left corner of the predicted box and the calibrated box; is the distance between the lower right corner of the predicted box and the calibrated box, the loss can be expressed as: ; in, , is the width of the prediction box, is the height of the prediction box, is the width of the calibration box, is the height of the calibration frame, Represents pixels The width of Represents pixels height.

[0045] Based on the above loss, the attention is weighted and the loss function is obtained, which is expressed as: ;in, Represents the weight coefficient, which is a hyperparameter. Indicates The attention weight of the prediction box is calculated by the attention mechanism. Represents the total number of target boxes detected in the current image (or feature map). Based on the above loss function, the initial target detection layer is trained to dynamically adjust the geometric error weight, alleviate the positioning deviation in complex scenes, and at the same time, improve the bounding box regression accuracy, solve the positioning deviation in complex scenes, and thus improve the robustness of the model. In the specific application example, the model training parameter configuration can select Adam for the optimizer, 150 training rounds, 16 batch size, input image size / pixel: 640×640, and initial learning rate 0.01.

[0046] After verification by actual ablation experiments and comparative experiments, and using the YOLOv8 original network model as an experimental comparison: the above-mentioned target detection model based on pulse improvement has the most obvious performance improvement. Compared with the YOLOv8 original network model, the average accuracy of the algorithm increased by 5.3 percentage points, the precision increased by 4.8 percentage points, the recall rate increased by 4.9 percentage points, and the total floating-point operations decreased by 0.75×109 when the threshold was 0.5. It not only improves the detection accuracy, but also achieves the lightweight of the model.

[0047] The present invention provides a flying object target detection method improved based on a pulse neural network. The embodiment of the present invention obtains a target detection image and event data of the target detection image, and generates an event-driven image frame sequence according to the target detection image and the event data, wherein the target detection image includes an object that moves rapidly relative to an image acquisition device; the image frame sequence is converted into a pulse feature through a pulse feature extraction layer in a target detection model improved based on a pulse neural network, and the pulse feature is gradually extracted through each feature extraction subnetwork to obtain an image enhancement feature corresponding to each feature extraction subnetwork; a plurality of image enhancement features are multi-scale feature fused through a neck network to obtain a fusion feature corresponding to each feature fusion subnetwork, and target recognition is performed on the corresponding fusion features through each target detection layer to obtain a flying object target detection result, thereby realizing target detection based on image data and event data, processing event data, converting the event-driven image frame sequence into a pulse signal, and performing convolution feature extraction on the pulse signal, thereby solving the problem that the existing convolution neural network cannot perform feature extraction on event data, and while realizing the processing of event-driven image data, the amount of data calculation is greatly reduced. In addition, feature extraction based on the image frame sequence that combines image static data and event data can effectively capture dynamic targets and reduce static background interference while supplementing high-latitude features, thereby improving target recognition accuracy in low-light environments and images of high-speed moving objects.

[0048] Furthermore, as a response to the above Figure 1 The embodiment of the present invention provides a flying object target detection device based on an improved pulse neural network, such as Fig. 9 As shown, the device includes: an acquisition module 21 and a target detection module 22, wherein the target detection module 22 performs target recognition through a target detection model improved based on a pulse neural network, wherein the target detection model includes a backbone network for pulse conversion and pulse feature extraction, a neck network for multi-scale feature fusion, and a head network for multi-scale target recognition. Wherein, the target detection model includes a backbone network for pulse conversion and pulse feature extraction, a neck network for multi-scale feature fusion, and a head network for multi-scale target recognition, wherein the backbone network includes a pulse feature extraction layer and multiple feature extraction sub-networks based on a lightweight attention mechanism, wherein the neck network includes multiple feature fusion sub-networks of different scales, and the head network includes multiple target detection layers of different scales; The acquisition module 21 is used to acquire a target detection image and event data of the target detection image, and generate an event-driven image frame sequence based on the target detection image and the event data, wherein the target detection image includes an object that moves rapidly relative to the image acquisition device; The target detection module 22 is used to convert the image frame sequence into pulse features through the pulse feature extraction layer, and gradually extract the pulse features through each feature extraction sub-network to obtain the image enhancement features corresponding to each feature extraction sub-network; perform multi-scale feature fusion on multiple image enhancement features through the neck network to obtain the fusion features corresponding to each feature fusion sub-network, and perform target recognition on the corresponding fusion features through each target detection layer to obtain the flying object target detection result.

[0049] Furthermore, the backbone network includes a pulse feature extraction layer, a first feature extraction subnetwork, a second feature extraction subnetwork, a third feature extraction subnetwork and a fast spatial pyramid pooling subnetwork in sequence; the target detection module includes: The first feature extraction unit is used to input the pulse feature into the first feature extraction subnetwork through the second feature extraction subnetwork, so as to gradually perform feature extraction through each feature extraction subnetwork and the fast spatial pyramid pooling subnetwork, and obtain the output of the first feature extraction subnetwork as the second image enhancement feature, the output of the second feature extraction subnetwork as the second image enhancement feature, the output of the third feature extraction subnetwork as the third image enhancement feature, and the output of the fast spatial pyramid pooling subnetwork as the fourth image enhancement feature.

[0050] Furthermore, the pulse feature extraction layer includes a neuron sublayer, a convolution sublayer and a batch normalization sublayer, and the image frame sequence is converted into a pulse feature through the pulse feature extraction layer. The target detection module includes: A generating unit, configured to extract spatial features and temporal features of the image frame sequence through a neuron sublayer, fuse the spatial features and the temporal features to generate a membrane potential function, and generate a pulse signal according to a membrane potential threshold and the membrane potential function; The third feature extraction unit is used to extract graph features of the pulse signal through the convolution sublayer to obtain a pulse signal feature graph, and to capture the temporal dependency of the pulse signal feature graph through a batch normalization sublayer to obtain a pulse feature.

[0051] Furthermore, the target detection module 22 includes: A first processing unit, configured to perform cross-channel feature interaction and combination processing on the input features through the first convolution branch to obtain an attention map; A first fusion unit is used to capture local spatial information and feature representations of different scales in the input feature through the first convolution branch to obtain a receptive field spatial feature, and fuse the attention map and the receptive field spatial feature to obtain a first image enhancement feature, wherein the number of convolution kernels of the second convolution branch is greater than the number of convolution kernels of the first convolution branch; The fourth feature extraction unit is used to perform feature extraction based on channel space fusion attention on the first image enhancement feature through the residual connection layer to obtain an image enhancement feature.

[0052] Furthermore, in a specific application scenario, the fourth feature extraction unit is specifically used to gradually extract features from the first image enhancement feature through two pulse feature extraction sublayers to obtain a second image enhancement feature; obtain a global spatial descriptor of the second image enhancement feature through a channel attention branch, and calculate a channel attention map based on the global spatial descriptor, expand the spatial feature information of the second image enhancement feature through a spatial feature branch, generate cross-channel average pooling features and maximum pooling features, obtain cross-channel spatial features by concatenating and convolving the average pooling features and the maximum pooling features, and obtain a third image enhancement feature by fusing the channel attention map and the cross-channel spatial features; and obtain an image enhancement feature by fusing the first image enhancement feature and the third image enhancement feature.

[0053] Furthermore, the target detection module 22 includes: A fifth feature extraction unit is used to input multiple image enhancement features into the neck network, and extract the output of the first feature fusion subnetwork as the first fusion feature, extract the output of the third feature fusion subnetwork as the second feature fusion feature, extract the output of the fifth feature fusion subnetwork as the third fusion feature, and extract the output of the seventh fusion subnetwork as the fourth fusion feature; wherein each feature fusion subnetwork includes a residual connection layer, and the third feature fusion subnetwork and the seventh feature fusion subnetwork also include a pulse feature extraction layer for feature fusion.

[0054] Furthermore, the target detection module 22 includes: a sixth feature extraction unit, configured to perform target recognition based on spatiotemporal features on the first fused features through the small target detection layer to obtain a small target detection result, specifically comprising: performing an average pooling operation on the first fused features to obtain an average pooling feature, performing a maximum pooling operation on the first fused features to obtain a maximum pooling feature, performing weighted fusion of the average pooling operation and the maximum pooling feature through a shared multi-layer perceptron to obtain a temporal attention map, extracting a global spatial feature of the first fused feature, and fusing the temporal attention map with the global spatial feature to generate a spatiotemporal fusion feature map, performing flying object positioning and classification processing according to the spatiotemporal fusion feature map to obtain a small target detection result; A second fusion unit is used to perform target recognition on the second fused feature through the first target detection layer to obtain a first target detection result, perform target recognition on the third fused feature through the second target detection layer to obtain a second target detection result, and perform target recognition on the fourth fused feature through the third target detection layer to obtain a third target detection result; A determination unit is used to determine a positioning frame based on the small target detection result, the first target detection result, the second target detection result and the third target detection result, and generate a flying object target detection result according to the positioning frame.

[0055] Furthermore, the acquisition module 21 includes: A construction unit, configured to construct an event set corresponding to different timestamps according to the timestamp corresponding to each pixel in the event data, wherein the event set includes position coordinates, timestamps and polarities of the pixels corresponding to the different timestamps; A conversion unit, configured to convert the event data into a frame sequence according to a timestamp-based functional relationship between the pulse pattern tensor and the event set; The splicing unit is used for the third fusion to splice the frame sequence with the target detection image to obtain an event-driven image frame sequence.

[0056] Furthermore, the device also includes: The acquisition module 21 is further used to acquire a training data set, wherein the training data set includes event-driven image frame sequence samples marked with calibration frames; A first construction module is used to construct an initial target detection model, wherein the initial target detection model includes a backbone network, a neck network and a head network; the backbone network includes a pulse feature extraction layer and multiple feature extraction subnetworks based on a lightweight attention mechanism, each feature extraction subnetwork sequentially includes a feature extraction layer that introduces a receptive field spatial attention mechanism and a residual connection layer that introduces a channel and spatial attention mechanism; the neck network sequentially includes a first feature fusion subnetwork, a second feature fusion subnetwork, a third feature fusion subnetwork, a fourth feature fusion subnetwork, a fifth feature fusion subnetwork, a sixth feature fusion subnetwork and a seventh feature fusion subnetwork, and the head network includes a small target detection layer, a first target detection layer, a second target detection layer and a third target detection layer; A second construction module is used to use the initial target detection model to convert the event data into frame sequence samples for target detection processing to obtain a prediction box; calculate a first distance between the calibration box and the upper left corner point of the prediction box, and a second distance between the calibration box and the upper right corner point of the prediction box; and construct a loss function based on the first distance, the second distance and the attention weighted loss term; The training module is used to train the initial target detection model using the loss function to obtain a trained target detection model.

[0057] The present invention provides a flying object target detection device improved based on a pulse neural network. The embodiment of the present invention obtains a target detection image and event data of the target detection image, and generates an event-driven image frame sequence according to the target detection image and the event data, wherein the target detection image includes an object that moves rapidly relative to an image acquisition device; the image frame sequence is converted into a pulse feature through a pulse feature extraction layer in a target detection model improved based on a pulse neural network, and the pulse feature is gradually extracted through each feature extraction subnetwork to obtain an image enhancement feature corresponding to each feature extraction subnetwork; a plurality of image enhancement features are multi-scale feature fused through a neck network to obtain a fusion feature corresponding to each feature fusion subnetwork, and target recognition is performed on the corresponding fusion features through each target detection layer to obtain a flying object target detection result, thereby realizing target detection based on image data and event data, processing event data, converting the event-driven image frame sequence into a pulse signal, and performing convolution feature extraction on the pulse signal, thereby solving the problem that the existing convolution neural network cannot perform feature extraction on event data, and while realizing the processing of event-driven image data, the amount of data calculation is greatly reduced. In addition, feature extraction based on the image frame sequence that combines image static data and event data can effectively capture dynamic targets and reduce static background interference while supplementing high-latitude features, thereby improving target recognition accuracy in low-light environments and images of high-speed moving objects.

[0058] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0059] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A flying object detection method based on an improved pulse neural network, characterized in that: Target recognition is performed by using a target detection model improved based on a pulse neural network, wherein the target detection model includes a backbone network for pulse conversion and pulse feature extraction, a neck network for multi-scale feature fusion, and a head network for multi-scale target recognition, the backbone network includes a pulse feature extraction layer and multiple feature extraction subnetworks based on a lightweight attention mechanism, the neck network includes multiple feature fusion subnetworks of different scales, and the head network includes multiple target detection layers of different scales. The method includes: Acquire a target detection image and event data of the target detection image, and generate an event-driven image frame sequence based on the target detection image and the event data, wherein the target detection image includes an object that moves rapidly relative to an image acquisition device; The image frame sequence is converted into pulse features through the pulse feature extraction layer, and the pulse features are gradually extracted through each feature extraction sub-network to obtain the image enhancement features corresponding to each feature extraction sub-network; multi-scale feature fusion is performed on multiple image enhancement features through the neck network to obtain the fusion features corresponding to each feature fusion sub-network, and target recognition is performed on the corresponding fusion features through each target detection layer to obtain the flying object target detection result.

2. The method according to claim 1, characterized in that The backbone network sequentially includes a pulse feature extraction layer, a first feature extraction subnetwork, a second feature extraction subnetwork, a third feature extraction subnetwork and a fast spatial pyramid pooling subnetwork, and the pulse features are gradually extracted through each of the feature extraction subnetworks to obtain image enhancement features corresponding to each feature extraction subnetwork, including: The pulse feature is input into the first feature extraction subnetwork to gradually perform feature extraction through each feature extraction subnetwork and the fast spatial pyramid pooling subnetwork, and the output of the first feature extraction subnetwork is obtained as the second image enhancement feature, the output of the second feature extraction subnetwork is obtained as the second image enhancement feature, the output of the third feature extraction subnetwork is obtained as the third image enhancement feature, and the output of the fast spatial pyramid pooling subnetwork is obtained as the fourth image enhancement feature.

3. The method according to claim 1, characterized in that The pulse feature extraction layer includes a neuron sublayer, a convolution sublayer and a batch normalization sublayer, and the image frame sequence is converted into a pulse feature through the pulse feature extraction layer, including: Extracting spatial features and temporal features of the image frame sequence through a neuron sublayer, generating a membrane potential function based on the fusion of the spatial features and the temporal features, and generating a pulse signal based on a membrane potential threshold and the membrane potential function; The pulse signal is subjected to graph feature extraction through the convolution sublayer to obtain a pulse signal feature graph, and the temporal dependency of the pulse signal feature graph is captured through a batch normalization sublayer to obtain a pulse feature.

4. The method according to claim 2, characterized in that: Each of the feature extraction subnetworks includes a feature extraction layer that introduces a receptive field spatial attention mechanism and a residual connection layer that introduces a channel and spatial attention mechanism. The feature extraction layer includes a first convolution branch and a second convolution branch. The feature extraction subnetwork processes the input features, including: Performing cross-channel feature interaction and combination processing on the input features through the first convolution branch to obtain an attention map; The first convolution branch is used to capture local spatial information and feature representations of different scales in the input feature to obtain a receptive field spatial feature, and the attention map and the receptive field spatial feature are fused to obtain a first image enhancement feature, wherein the number of convolution kernels of the second convolution branch is greater than the number of convolution kernels of the first convolution branch; The first image enhancement feature is subjected to feature extraction based on channel space fusion attention through the residual connection layer to obtain an image enhancement feature.

5. The method according to claim 4, characterized in that Each residual connection layer includes two pulse feature extraction sublayers and a lightweight attention sublayer in sequence, and the lightweight attention sublayer includes a channel attention branch and a spatial feature branch. The residual connection layer performs feature extraction based on channel space fusion attention on the first image enhancement feature to obtain an image enhancement feature, including: Stepwise feature extraction of the first image enhancement feature is performed through two pulse feature extraction sub-layers to obtain a second image enhancement feature; A global spatial descriptor of the second image enhancement feature is obtained through a channel attention branch, and a channel attention map is calculated based on the global spatial descriptor, and spatial feature information of the second image enhancement feature is expanded through a spatial feature branch to generate cross-channel average pooling features and maximum pooling features, and cross-channel spatial features are obtained by concatenating and convolving the average pooling features and the maximum pooling features, and the channel attention map and the cross-channel spatial features are fused to obtain a third image enhancement feature; The first image enhancement feature and the third image enhancement feature are fused to obtain an image enhancement feature.

6. The method according to claim 1, characterized in that The neck network includes a first feature fusion subnetwork, a second feature fusion subnetwork, a third feature fusion subnetwork, a fourth feature fusion subnetwork, a fifth feature fusion subnetwork, a sixth feature fusion subnetwork and a seventh feature fusion subnetwork in sequence, and the fusion features include a first fusion feature, a second fusion feature, a third fusion feature and a fourth fusion feature with decreasing granularity in sequence; The multi-scale feature fusion of multiple image enhancement features through the neck network includes: Inputting a plurality of image enhancement features into the neck network, and extracting the output of the first feature fusion subnetwork as the first fusion feature, extracting the output of the third feature fusion subnetwork as the second feature fusion feature, extracting the output of the fifth feature fusion subnetwork as the third fusion feature, and extracting the output of the seventh fusion subnetwork as the fourth fusion feature; Among them, each feature fusion sub-network includes a residual connection layer that introduces a channel and spatial attention mechanism, and the third feature fusion sub-network and the seventh feature fusion sub-network also include a pulse feature extraction layer for feature fusion.

7. The method according to claim 5, characterized in that The head network includes a small target detection layer, a first target detection layer, a second target detection layer and a third target detection layer. Each target detection layer performs target recognition on the corresponding fusion features to obtain a flying object target detection result, including: The small target detection layer performs target recognition based on spatiotemporal features on the first fusion feature to obtain a small target detection result, specifically including: performing an average pooling operation on the first fusion feature to obtain an average pooling feature, performing a maximum pooling operation on the first fusion feature to obtain a maximum pooling feature, performing weighted fusion of the average pooling operation and the maximum pooling feature through a shared multi-layer perceptron to obtain a temporal attention map, extracting a global spatial feature of the first fusion feature, and fusing the temporal attention map and the global spatial feature to generate a spatiotemporal fusion feature map, performing flying object positioning and classification processing according to the spatiotemporal fusion feature map, and obtaining a small target detection result; Performing target recognition on the second fused feature through the first target detection layer to obtain a first target detection result, performing target recognition on the third fused feature through the second target detection layer to obtain a second target detection result, and performing target recognition on the fourth fused feature through the third target detection layer to obtain a third target detection result; A positioning frame is determined based on the small target detection result, the first target detection result, the second target detection result and the third target detection result, and a flying object target detection result is generated according to the positioning frame.

8. The method according to claim 1, characterized in that The acquiring of the target detection image and the event data of the target detection image, and generating an event-driven image frame sequence based on the target detection image and the event data, comprises: According to the timestamp corresponding to each pixel in the event data, an event set corresponding to different timestamps is constructed, wherein the event set includes the position coordinates, timestamps and polarities of the pixels corresponding to different timestamps; Converting the event data into a frame sequence according to a timestamp-based functional relationship between a pulse pattern tensor and an event set; The frame sequence is spliced ​​with the target detection image to obtain an event-driven image frame sequence.

9. The method according to claim 1, characterized in that: Before performing target recognition by using the target detection model improved based on the pulse neural network, the method further includes: Acquire a training data set, wherein the training data set includes event-driven image frame sequence samples marked with calibration frames; Constructing an initial target detection model, wherein the initial target detection model includes a backbone network, a neck network and a head network; the backbone network includes a pulse feature extraction layer and multiple feature extraction subnetworks based on a lightweight attention mechanism, each feature extraction subnetwork sequentially includes a feature extraction layer that introduces a receptive field spatial attention mechanism and a residual connection layer that introduces a channel and spatial attention mechanism; the neck network sequentially includes a first feature fusion subnetwork, a second feature fusion subnetwork, a third feature fusion subnetwork, a fourth feature fusion subnetwork, a fifth feature fusion subnetwork, a sixth feature fusion subnetwork and a seventh feature fusion subnetwork, and the head network includes a small target detection layer, a first target detection layer, a second target detection layer and a third target detection layer; The event data is converted into frame sequence samples using the initial target detection model to perform target detection processing to obtain a prediction frame; Calculating a first distance between the calibration frame and the upper left corner point of the prediction frame, and a second distance between the calibration frame and the upper right corner point of the prediction frame; Constructing a loss function according to the first distance, the second distance and an attention-weighted loss term; The initial target detection model is trained using the loss function to obtain a trained target detection model.

10. A flying object target detection device based on improved pulse neural network, characterized in that: The device comprises an acquisition module and a target detection module, wherein the target detection module performs target recognition through a target detection model improved based on a pulse neural network, wherein the target detection model comprises a backbone network for pulse conversion and pulse feature extraction, a neck network for multi-scale feature fusion, and a head network for multi-scale target recognition, wherein the backbone network comprises a pulse feature extraction layer and a plurality of feature extraction subnetworks based on a lightweight attention mechanism, the neck network comprises a plurality of feature fusion subnetworks of different scales, and the head network comprises a plurality of target detection layers of different scales; The acquisition module is used to acquire a target detection image and event data of the target detection image, and generate an event-driven image frame sequence based on the target detection image and the event data, wherein the target detection image includes an object that moves rapidly relative to the image acquisition device; The target detection module is used to convert the image frame sequence into pulse features through the pulse feature extraction layer, and gradually extract the pulse features through each feature extraction sub-network to obtain the image enhancement features corresponding to each feature extraction sub-network; perform multi-scale feature fusion on multiple image enhancement features through the neck network to obtain the fusion features corresponding to each feature fusion sub-network, and perform target recognition on the corresponding fusion features through each target detection layer to obtain the flying object target detection result.

Images