Underwater target detection method based on spiking neural network

Abstract

The invention relates to an underwater target detection method based on a pulse neural network, which can realize underwater target detection with high precision and low power consumption. By fusing the cross-stage partial network and the YOLO architecture, the problem of pulse degradation is effectively solved, and the feature extraction capability of the model is enhanced; in order to solve the problem of underwater noise interference, the pulse-based underwater image denoising method is designed, only integer addition is used in the method, a pulse neural network structure can be seamlessly embedded, and the quality of a feature map is enhanced; in order to solve the problem that a traditional normalization method is low in precision in the spiking neural network, separate batch normalization is provided, by independently normalizing a feature map in multiple time steps and optimizing a residual structure, the time dynamic state of the SNN can be effectively captured, and the detection precision is improved. The network shows excellent performance in underwater target detection, and compared with an artificial neural network of the same scale, the network has higher performance and lower energy consumption.

Description

Technical Field

[0001] The present invention relates to the fields of computer technology and underwater detection technology, and in particular to a technology for using artificial neural networks for underwater object detection, and specifically to an underwater target detection method based on a pulse neural network. Background Art

[0002] Underwater object detection is crucial in marine science and industrial safety inspections. Existing underwater target detection methods are mainly based on artificial neural networks (ANNs). Several ANN models optimized for underwater object detection have been proposed, such as YOLOv9s-UI, which is disclosed in the paper "Optimization and application of improved yolov9s-ui for underwater object detection, Applied Sciences", which outperforms the baseline model in accuracy. However, the introduction of attention mechanisms often increases computational complexity, making these models unsuitable for resource-constrained devices such as autonomous underwater vehicles (AUVs). To address this issue, lightweight models have been introduced, such as the models disclosed in the papers "Yolo nano underwater: A fast and compact object detector for embedded device" and "An underwater light-weight object detector for edge computing", which reduce computational complexity by reducing parameters and computational requirements. However, these simplifications often reduce detection accuracy compared to contemporary general-purpose object detection models.

[0003] Spiking neural networks (SNNs) offer a promising approach for achieving low computation and high accuracy. Recent research has explored the application of SNNs to object detection, such as Spiking-YOLO (published in the paper "Spiking-yolo: Spiking neural network for energy-efficient object detection"), EMS-YOLO (published in the paper "Deep directly trained spiking neural networks for object detection"), and SpikeYOLO (published in the paper "Integer-valued training and spike-driven inference spiking neural network for high-performance and energy-efficient object detection"). However, these works primarily focus on general object detection and lack optimization for underwater conditions, including image preprocessing and noise suppression, which limits their performance in underwater environments. Summary of the Invention

[0004] The purpose of this invention is to solve the technical problems that existing target detection mainly focuses on general object detection, and these technologies have poor performance and low detection accuracy when applied to underwater object detection, and propose an underwater target detection method based on pulse neural network.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A method for underwater target detection based on a spiking neural network comprises the following steps:

[0007] Step 1: Input the RGB image into the pulse encoder and output the binary pulse feature map;

[0008] Step 2: Send the feature map output in step 1 to the impulse denoiser to perform image denoising and output a cleaner feature map;

[0009] Step 3: The denoised feature map from step 2 is fed into the model backbone, where multi-scale features are extracted through the SU-Block1 feature extraction module and the SpikeSPP pulse pyramid pooling module, outputting a larger number of feature maps with richer information.

[0010] Step 4: The feature map output in step 3 is fed into the neck part of the model, and feature fusion is performed through the SU-Block2 feature fusion module, the feature channel splicing layer, the upsampling layer, and the downsampling layer. The feature information from each layer is spliced and fused and then output;

[0011] Step 5: Send the feature map output in step 4 to the pulse detection head and output the final detection result;

[0012] Through the above steps, underwater target detection based on pulse neural network is realized.

[0013] In step 1, the spike encoder consists of a 2D convolutional layer, a separable batch normalization layer, and an IF spike neuron. A convolution operation is performed to convert the RGB image of the input network into a binary spike feature map. The resulting feature map is replicated four times as the output of the spike encoder at four time steps.

[0014] In step 2, the pulse denoiser is implemented as follows:

[0015] 2-1) In order to find missing pixels (negative noise) and isolated pixels (positive noise) in the feature map, three convolution filters are defined:

[0016]

[0017] Among them, K1 and K2 are used to detect negative noise, and K3 is used to detect positive noise;

[0018] 2-2) Define the judgment and operation rules. If the convolution result of the current pixel in the feature map and the filter is equal to 4, the current pixel value is inverted. Otherwise, no processing is performed. This process can be described by the following formula:

[0019]

[0020] Among them, I in (x, y) is the pixel value at coordinate (x, y) in the input feature map, I in (x+i, y+j) is the pixel value at the coordinate (x+i, y+j) in the input feature map, I conv (x, y) is the convolution result, I out (x, y) is the output, K n (n=1,2,3) represents the nth convolution kernel used, K n (i+1, j+1) represents the convolution kernel K n The value at coordinate (i+1,j+1);

[0021] 2-3) Perform pixel-by-pixel convolution on the feature map output by the pulse encoder using the K1 kernel according to the rules in 2-2), fill in missing pixels, and output the processed feature map;

[0022] 2-4) Perform pixel-by-pixel convolution on the feature map output in 2-3) using the K2 kernel according to the rules in 2-2), fill in the missing pixels again, and output the processed feature map;

[0023] 2-5) Perform pixel-by-pixel convolution on the feature map outputted in 2-4) using the K3 kernel according to the rules in 2-2) to remove isolated pixels. The processed feature map is then outputted as the final output of the pulse denoiser.

[0024] In step 3, the model backbone network consists of three SU-Block1 feature extraction modules and a SpikeSPP pulse pyramid pooling module arranged in sequence, specifically including the following steps:

[0025] 3-1) Downsample the feature map output by the pulse denoiser and use it as the input of the first SU-Block1 feature extraction module to extract preliminary features;

[0026] 3-2) Using the output of the first SU-Block1 feature extraction module as the input of the second SU-Block1 feature extraction module to extract features again;

[0027] 3-3) Using the output of the second SU-Block1 feature extraction module as the input of the third SU-Block1 feature extraction module, extracting features for the final time;

[0028] 3-4) The output of the third SU-Block1 feature extraction module is sent to SpikeSPP for multi-level pooling, and the multi-scale feature map is output to the model neck network.

[0029] The processing flow of the SU-Block1 feature extraction module is as follows:

[0030] S1) Copy the feature map input to the SU-Block1 feature extraction module into two copies, namely feature map F1 and feature map F2;

[0031] S2) wherein the feature map F1 undergoes a first 2D convolution with a kernel size of 3×3, undergoes separable batch normalization, and then undergoes feature channel segmentation and is equally divided into two parts along the channel dimension, namely, feature map F3 and feature map F4; wherein the feature map F3 remains unchanged, and the feature map F4 is sent to the IF pulse neuron, and its output feature map undergoes a second 2D convolution and separable batch normalization again to obtain feature map F5; the feature map F2 undergoes a third 2D convolution with a kernel size of 1×1, and undergoes separable batch normalization, and the output feature map is element-wise added to the feature map F5 to obtain feature map F6;

[0032] S3) performing feature channel splicing on the feature map F6 and the feature map F3, sending the result to the IF pulse neuron, and outputting the final extracted feature map;

[0033] The processing flow of SU-Block1 can be expressed by the formula:

[0034]

[0035] where X l 1 and X l 2 represents the two feature maps obtained after the lth layer segmentation, X l out represents the output of the lth layer, IF(·) represents the IF neuron; Conv(·) represents the convolutional layer, Split(·) is an operation that divides the input into two equal parts along the channel dimension, and Concat(·) is an operation that concatenates the input along the channel dimension.

[0036] The implementation steps of the separable batch normalization method are as follows:

[0037] S1) concatenate the input feature maps of multiple time steps along the channel dimension and remove the time dimension;

[0038] S2) Calculate the mean of the concatenated feature map and variance Where N is the batch size, x t,k represents the feature map of the kth channel at time step t, μ t,k represents the mean value of the kth channel at time step t, σ 2 t,k represents the variance of the kth channel at time step t:

[0039] S3) Calculate the normalized data element by element

[0040] Among them, x t,k represents the feature map of the kth channel at time step t, n is a positive integer representing the number of SeBN modules connected to neurons after the current module, and ε is a small constant to avoid division by zero; t,k and β t,k is a trainable parameter, y t,k is the normalized output; μ t,k and σ 2 t,k They are x t,k The mean and variance of y t,k is the normalized feature map of the kth channel at time step t;

[0041] S4) Re-add the time dimension to the normalized feature map and evenly divide it into the time dimension along the channel dimension according to the number of time steps, so as to keep consistent with the input data shape.

[0042] The processing flow of the SpikeSPP pulse pyramid pooling module is as follows:

[0043] S1) performs a 2D convolution on the input feature map and performs separable batch normalization to obtain the output feature map F7;

[0044] S2) performing maximum pooling on the feature map F7 to obtain a feature map F8;

[0045] S3) performing maximum pooling on the feature map F8 to obtain a feature map F9;

[0046] S4) Concatenate feature maps F7, F8, and F9 along the feature channel, and sequentially pass through IF spike neurons, 2D convolution, separable batch normalization, and IF spike neurons to output the final feature map.

[0047] The processing flow of the SpikeSPP pulse pyramid pooling module can be expressed by the formula:

[0048]

[0049] where X l 1. X l 2. X l 3 represent the feature maps after 2D convolution, first pooling, and second pooling in the first layer, respectively. l out Represents the output of the lth layer, IF(·) represents the IF neuron; Conv(·) represents the convolutional layer, MaxPool(·) represents the maximum pooling layer, and Concat(·) is the concatenation operation on the input along the channel dimension.

[0050] In step 4, the model neck network contains two SU-Block2 feature fusion modules, two feature channel splicing layers, an upsampling layer, and a downsampling layer, specifically including the following steps:

[0051] 4-1) Upsample the feature map output by SpikeSPP, perform feature channel splicing with the output of the second SU-Block1 feature extraction module, and output it to the first SU-Block2 feature fusion module for processing;

[0052] 4-2) Downsample the feature map output by the first SU-Block2 feature fusion module, perform feature channel splicing with the output of SpikeSPP, and output it to the second SU-Block2 feature fusion module for processing;

[0053] 4-3) The output results of the first SU-Block2 feature fusion module and the second SU-Block2 feature fusion module are sent to the pulse detection head, the detection results are output, and the target detection is completed.

[0054] The processing flow of the SU-Block2 feature fusion module is as follows:

[0055] S1) The feature map of the input SU-Block2 is subjected to a 2D convolution with a kernel size of 3×3, and is subjected to separate batch normalization. The feature channel is then split and divided into two equal parts along the channel dimension, namely feature map F10 and feature map F11. Feature map F10 remains unchanged, and feature map F11 is sent to the IF spike neuron. The output feature map is again subjected to 2D convolution and separate batch normalization to obtain feature map F12;

[0056] S2) The feature map F10 and the feature map F12 are spliced together through feature channels, and the result is fed into the IF pulse neuron to output the final fused feature map.

[0057] The processing flow of the SU-Block2 feature fusion module can be expressed by the formula:

[0058]

[0059] where X l 1 and X l 2 represents the two feature maps obtained after the lth layer segmentation, X l out represents the output of layer l, IF(·) represents the IF neuron, Conv(·) represents the convolutional layer, Split(·) is an operation that splits the input into two equal parts along the channel dimension, and Concat(·) is an operation that concatenates the input along the channel dimension.

[0060] Compared with the prior art, the present invention has the following technical effects:

[0061] 1) This paper implements the first underwater target detection method based on spiking neural networks—SU-YOLO. Leveraging the lightweight and energy-efficient nature of spiking neural networks, this method achieves both high accuracy and low energy consumption. A redesigned series of network modules integrates cross-stage partial networks with a spiking architecture, effectively addressing the spike degradation issue commonly seen in existing spiking neural networks.

[0062] 2) This paper proposes an image denoising method suitable for spiking neural networks to address the severe noise problem in underwater images. Unlike existing RGB or grayscale image denoising methods, this method denoises binary spiking feature maps and uses only integer addition. The purely binary computation allows it to be seamlessly embedded in various spiking neural network architectures, improving feature map quality with extremely low computational overhead.

[0063] 3) This paper proposes a normalization method specifically for spiking neural networks. By independently normalizing feature maps across multiple time steps and optimizing the residual structure, it can better exploit the temporal dynamics of spiking neural networks. Its performance surpasses existing normalization methods for artificial neural networks and spiking neural networks, and it has general applicability to other spiking neural network models.

[0064] 4) Experiments were conducted using the proposed model on the URPC2019 and UDD underwater target detection public datasets. The results showed that SU-YOLO achieved an average precision of 78.8% on the URPC2019 dataset and 58.2% on the UDD dataset, with 6.97M parameters and 2.98mJ of energy consumption. It outperformed mainstream spiking neural network models and some lightweight artificial neural network models in both detection accuracy and computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The present invention will be further described below with reference to the accompanying drawings and examples:

[0066] Figure 1 Schematic diagram of the SU-YOLO model and its module structure in the present invention;

[0067] Figure 2 for Figure 1 Schematic diagram of the pulse denoiser processing flow;

[0068] Figure 3 for Figure 1 Flowchart of the split batch normalization method. DETAILED DESCRIPTION

[0069] A method for underwater target detection based on a spiking neural network comprises the following steps:

[0070] Step 1: Input the RGB image into the pulse encoder 1 and output the binary pulse feature map;

[0071] Step 2: Send the feature map output in step 1 to the pulse denoiser 2 to perform image denoising and output a cleaner feature map;

[0072] Step 3: The denoised feature map from step 2 is fed into the model backbone, where multi-scale features are extracted through the SU-Block1 feature extraction module and the SpikeSPP pulse pyramid pooling module, outputting a larger number of feature maps with richer information.

[0073] Step 4: The feature map output in step 3 is fed into the neck part of the model, and feature fusion is performed through the SU-Block2 feature fusion module, the feature channel splicing layer, the upsampling layer, and the downsampling layer. The feature information from each layer is spliced and fused and then output;

[0074] Step 5: Send the feature map output in step 4 to the pulse detection head and output the final detection result;

[0075] Through the above steps, underwater target detection based on pulse neural network is realized.

[0076] In step 1, the spike encoder consists of a 2D convolutional layer, a separable batch normalization layer, and an IF spike neuron. A convolution operation is performed to convert the RGB image of the input network into a binary spike feature map. The resulting feature map is replicated four times as the output of the spike encoder at four time steps.

[0077] In step 2, the pulse denoiser 2 is implemented as follows:

[0078] 2-1) In order to find missing pixels (negative noise) and isolated pixels (positive noise) in the feature map, three convolution filters are defined:

[0079]

[0080] Among them, K1 and K2 are used to detect negative noise, and K3 is used to detect positive noise;

[0081] 2-2) Define the judgment and operation rules. If the convolution result of the current pixel in the feature map and the filter is equal to 4, the current pixel value is inverted. Otherwise, no processing is performed. This process can be described by the following formula:

[0082]

[0083] Among them, I in (x, y) is the pixel value at coordinate (x, y) in the input feature map, I in (x+i, y+j) is the pixel value at the coordinate (x+i, y+j) in the input feature map, I conv (x, y) is the convolution result, I out (x, y) is the output, K n (n=1,2,3) represents the nth convolution kernel used, K n (i+1, j+1) represents the convolution kernel Kn The value at coordinate (i+1,j+1);

[0084] 2-3) Perform pixel-by-pixel convolution on the feature map output by the pulse encoder using the K1 kernel according to the rules in 2-2), fill in missing pixels, and output the processed feature map;

[0085] 2-4) Perform pixel-by-pixel convolution on the feature map output in 2-3) using the K2 kernel according to the rules in 2-2), fill in the missing pixels again, and output the processed feature map;

[0086] 2-5) Perform pixel-by-pixel convolution on the feature map outputted in 2-4) using the K3 kernel according to the rules in 2-2) to remove isolated pixels. The processed feature map is then outputted as the final output of the pulse denoiser.

[0087] In step 3, the model backbone network consists of three SU-Block1 feature extraction modules and a SpikeSPP pulse pyramid pooling module arranged in sequence, specifically including the following steps:

[0088] 3-1) Downsampling 3 the feature map output by the pulse denoiser 2 as the input of the first SU-Block 1 feature extraction module 4 to extract preliminary features;

[0089] 3-2) The output of the first SU-Block 1 feature extraction module 4 is used as the input of the second SU-Block 1 feature extraction module 5 to extract features again;

[0090] 3-3) The output of the second SU-Block 1 feature extraction module 5 is used as the input of the third SU-Block 1 feature extraction module 6 to extract features for the final time;

[0091] 3-4) The output of the third SU-Block1 feature extraction module 6 is sent to SpikeSPP for multi-level pooling, and a multi-scale feature map is output to the model neck network.

[0092] The processing flow of the SU-Block1 feature extraction module is as follows:

[0093] S1) Copy the feature map input to the SU-Block1 feature extraction module into two copies, namely feature map F1 and feature map F2;

[0094] S2) wherein the feature map F1 undergoes a first 2D convolution 24 with a kernel size of 3×3, undergoes a separate batch normalization 25, and then undergoes a feature channel segmentation 28, and is divided equally into two parts along the channel dimension, namely, feature map F3 and feature map F4; wherein the feature map F3 remains unchanged, and the feature map F4 is sent to the IF pulse neuron 29, and its output feature map undergoes a second 2D convolution 30 and a separate batch normalization 31 again to obtain a feature map F5; the feature map F2 undergoes a third 2D convolution 26 with a kernel size of 1×1, and undergoes a separate batch normalization 27, and the output feature map is element-wise added to the feature map F5 to obtain a feature map F6;

[0095] S3) performing feature channel splicing 32 on the feature map F6 and the feature map F3, sending the result to the IF pulse neuron 33, and outputting the final extracted feature map;

[0096] The processing flow of SU-Block1 can be expressed by the formula:

[0097]

[0098] where X l 1 and X l 2 represents the two feature maps obtained after the lth layer segmentation, X l out represents the output of the lth layer, IF(·) represents the IF neuron; Conv(·) represents the convolutional layer, Split(·) is an operation that divides the input into two equal parts along the channel dimension, and Concat(·) is an operation that concatenates the input along the channel dimension.

[0099] The implementation steps of the separable batch normalization method are as follows:

[0100] S1) concatenate the input feature maps of multiple time steps along the channel dimension and remove the time dimension;

[0101] S2) Calculate the mean of the concatenated feature map and variance Where N is the batch size, x t,k represents the feature map of the kth channel at time step t, μ t,k represents the mean value of the kth channel at time step t, σ 2 t,k represents the variance of the kth channel at time step t:

[0102] S3) Calculate the normalized data element by element

[0103] Among them, x t,krepresents the feature map of the kth channel at time step t, n is a positive integer representing the number of SeBN modules connected to neurons after the current module, and ε is a small constant to avoid division by zero; t,k and β t,k is a trainable parameter, y t,k is the normalized output; μ t,k and σ 2 t,k They are x t,k The mean and variance of y t,k is the normalized feature map of the kth channel at time step t;

[0104] S4) Re-add the time dimension to the normalized feature map and evenly divide it into the time dimension along the channel dimension according to the number of time steps, so as to keep consistent with the input data shape.

[0105] The processing flow of the SpikeSPP pulse pyramid pooling module is as follows:

[0106] S1) performs a 2D convolution 15 on the input feature map and performs separable batch normalization 16 to obtain an output feature map F7;

[0107] S2) performing maximum pooling 17 on the feature map F7 to obtain a feature map F8;

[0108] S3) performing maximum pooling 18 on the feature map F8 to obtain a feature map F9;

[0109] S4) The feature maps F7, F8, and F9 are concatenated 19 along the feature channel, and sequentially pass through IF pulse neurons 20, 2D convolution 21, separable batch normalization 22, and IF pulse neurons 23 to output the final feature map.

[0110] The processing flow of the SpikeSPP pulse pyramid pooling module can be expressed by the formula:

[0111]

[0112] where X l 1. X l 2. X l 3 represent the feature maps after 2D convolution, first pooling, and second pooling in the first layer, respectively. l out Represents the output of the lth layer, IF(·) represents the IF neuron; Conv(·) represents the convolutional layer, MaxPool(·) represents the maximum pooling layer, and Concat(·) is the concatenation operation on the input along the channel dimension.

[0113] In step 4, the model neck network contains two SU-Block2 feature fusion modules, two feature channel splicing layers, an upsampling layer, and a downsampling layer, specifically including the following steps:

[0114] 4-1) Upsampling the feature map output by SpikeSPP 8, performing feature channel splicing 9 with the output of the second SU-Block 1 feature extraction module 5, and outputting it to the first SU-Block 2 feature fusion module 10 for processing;

[0115] 4-2) Downsampling 11 the feature map output by the first SU-Block2 feature fusion module 10, performing feature channel splicing 12 with the output of SpikeSPP, and outputting it to the second SU-Block2 feature fusion module 13 for processing;

[0116] 4-3) The output results of the first SU-Block2 feature fusion module 10 and the second SU-Block2 feature fusion module 13 are sent to the pulse detection head, and the detection results are output to complete the target detection.

[0117] The processing flow of the SU-Block2 feature fusion module is as follows:

[0118] S1) The feature map of the input SU-Block2 is subjected to a 2D convolution 34 with a kernel size of 3×3, and is subjected to a separate batch normalization 35. Then, the feature channel is segmented 36 and the feature map is divided into two equal parts along the channel dimension, namely, feature map F10 and feature map F11. Feature map F10 remains unchanged, and feature map F11 is sent to the IF pulse neuron 37. The output feature map is again subjected to a 2D convolution 38 and a separate batch normalization 39 to obtain feature map F12;

[0119] S2) The feature map F10 and the feature map F12 are subjected to feature channel splicing 40, and are sent to the IF pulse neuron 41 to output the final fused feature map.

[0120] The processing flow of the SU-Block2 feature fusion module can be expressed by the formula:

[0121]

[0122] where X l 1 and X l 2 represents the two feature maps obtained after the lth layer segmentation, X l out represents the output of layer l, IF(·) represents the IF neuron, Conv(·) represents the convolutional layer, Split(·) is an operation that splits the input into two equal parts along the channel dimension, and Concat(·) is an operation that concatenates the input along the channel dimension.

[0123] In order to facilitate those skilled in the art to better understand the present invention, further explanation is given as follows:

[0124] The pulse denoiser in the present invention (such as Figure 2 As shown in Figure 2, an image denoising method designed specifically for SNN is used to solve the serious noise problem of underwater images and improve detection accuracy. The method uses three convolution filters, which are defined as follows:

[0125]

[0126] The input is a pulse-coded binary feature map, which is sequentially passed through K 1、 K2 and K3 perform convolution operation. If the convolution result is equal to 4, the current pixel is inverted. This process is described as follows:

[0127]

[0128] Among them, I in (x, y) is the pixel value at coordinate (x, y) in the input feature map, I conv (x, y) is the convolution result, I out (x, y) is the output. K n (n=1, 2, 3) represents the nth convolution kernel used.

[0129] The main operations of this method are filling missing pixels (negative noise) and removing isolated pixels (positive noise). Kernels K1 and K2 are used to fill missing pixels. When the central pixel is 0 and the surrounding pixels are 1, the convolution result is 4, indicating that the pixel is negative noise, which is then changed to 1. Kernel K3 is used to remove isolated pixels. If the central pixel is 1 and the surrounding pixels are 0, the convolution result is 4, indicating that the pixel is positive noise, which is then changed to 0.

[0130] The pulse denoiser module runs only during inference and has extremely low computational overhead. For a 320×320 image input, the computational cost is less than 0.1GSOPs, the energy consumption is less than 0.01mJ, and the mean average precision (mAP) on the UDD dataset can be improved by 3.5%. This approach retains the full pulse characteristics of the network. Both the input and output feature maps are binary, so that the multiplication operation can be optimized to sparse addition like traditional convolution. Importantly, this process only requires integer addition calculations, which reduces energy consumption by 9 times and 37 times compared to floating-point addition and multiplication, respectively. Thanks to pure pulse computation, the pulse denoiser can be easily integrated into various pulse neural network architectures.

[0131] The SU-Block in the present invention (such as Figure 1The basic residual block used in the model includes two variants: SU-Block1 feature extraction module and SU-Block2 feature fusion module. SU-Block1 is used for feature extraction, downsampling, and increasing the number of input feature maps. It is described as follows:

[0132]

[0133] where X l out Denotes the output of layer l. IF(·) denotes an IF neuron. Conv(·) denotes a convolutional layer. Split(·) is an operation that splits the input into two equal parts along the channel dimension. Concat(·) concatenates the input along the channel dimension.

[0134] In SU-Block 1, the feature map undergoes a 3×3 convolution and is then divided into two groups. The first group remains unchanged, while the second group undergoes an additional 3×3 convolution to further extract features. These two groups are then concatenated, activated, and output. Because the second group lacks a residual path, a shortcut connection is added to preserve gradient information.

[0135] SU-Block2 is used for the neck, but does not perform downsampling or additional residual connections, which makes it more efficient in feature fusion. It is described as follows:

[0136]

[0137] The SpikeSPP pulse pyramid pooling module in the present invention (such as Figure 1 As shown in Figure 2). SpikeSPP is a simplified version of Spatial Pyramid Pooling (SPP). The description of SpikeSPP is as follows:

[0138]

[0139] where X l out Represents the output of layer l. IF(·) represents an IF neuron. Conv(·) represents a convolutional layer. MaxPool(·) represents a maximum pooling layer. Concat(·) concatenates the inputs along the channel dimension.

[0140] SPP provides multi-level pooling, which helps extract multi-scale feature information. However, the pooling operation in SPP is performed after the activation function, resulting in information loss in the SNN. Due to the binary nature of spikes, if activation precedes pooling, many pixels will have the same value of 1, resulting in information loss, especially in underwater object detection where salt and pepper noise is dense. Pooling before activation helps preserve information; therefore, SpikeSPP adjusts the pooling order to better distinguish between noise and object edges. In addition, experimental results show that the third MaxPool layer in SPP contributes little information; therefore, it is removed to reduce the computational load. The features are then passed to the neurons, which output spikes at an appropriate firing rate.

[0141] The separated batch normalization in the present invention (such as Figure 3 As shown). A batch normalization method designed specifically for spiking neural networks. In a spiking neural network, each spiking neuron charges and pulses independently at each time step; therefore, the spike distribution may vary across time steps. Existing normalization methods mix feature maps at different time steps to force them to form a uniform distribution. This approach effectively loses a latent variable and reduces the expressive power of the network. Other methods ignore the summation structure in residual networks, which can cause neurons to overcharge and lead to excessively high spike firing rates in the output. To address this limitation, separated batch normalization (SeBN) is proposed. This method normalizes feature maps independently at different time steps and is optimized for integration with the residual structure, enhancing the learning ability of the network. SeBN is defined as:

[0142]

[0143] Among them, x t,k represents the feature map of the k-th channel at time step t, n is a positive integer representing the number of SeBN modules connected to the neuron after the current module, and ε is a small constant to avoid division by zero. t,k and β t,k is a trainable parameter, y t,k is the normalized output. μ t,k and σ 2 t,k They are x t,k The mean and variance of , defined as:

[0144]

[0145] Where N represents the batch size.

[0146] The feature maps from T time steps are concatenated along the channel dimension and then subjected to a SeBN operation to normalize them along the same dimension. The normalized feature maps are then averaged and re-output in T time steps. In the normalization calculation, n adjusts the distribution of the input data from N(0,1) to N(0,1 / n) to ensure that the summed residual structure maintains the distribution of N(0,1) to prevent over-activation of neurons. During training, the running mean and variance of each batch on T×C are updated using a moving average to calculate μ run,t,k and σ 2 run,t,k , which are used during inference.

[0147] The pulse encoder in this invention consists of a convolution-batch normalization-IF pulse neuron, which is used to convert the floating-point image input to the network into binary pulses. This method is the mainstream encoding method of existing spiking neural networks.

[0148] The pulse detection head in this invention is used to fuse and output detection results. The detection head adopts the decoupling head in YOLOv9 and is modified for pulse-based input and output. Separate convolutional layers calculate category and bounding box information. The pulse detection head uses multi-level feature fusion and accepts input from two SU-Block2s to capture richer features. The membrane potential threshold of the last IF neuron in the pulse detection head is set to infinity, and all floating-point values are accumulated at each time step to ensure more accurate predicted coordinates. The parameter n of SeBN is set to the time step T, which narrows the range of single-step output values and stabilizes the accumulated membrane potential, thereby improving gradient stability. The membrane potential of these neurons is read as the final network output.

[0149] In addition: the present invention uses the GELAN network of YOLOv9 as the benchmark model, replaces all batch normalization layers in the network with a separate batch normalization method; the present invention uses a pulse encoder to replace the first convolutional layer of the network to realize the encoding process of floating-point images to binary pulses; the present invention adds an additional pulse denoiser to the second layer of the network for pulse feature map denoising to enhance image quality; the present invention replaces all RepNCSPELAN4 modules in the network backbone layer with SU-Block1 for feature extraction and downsampling; the present invention replaces the original SPPELAN module with SpikeSPP to realize multi-scale feature extraction that is more suitable for pulse neural networks; the present invention replaces all RepNCSPELAN4 modules at the neck of the network with SU-Block2 for feature fusion; the present invention uses IF pulse neurons to replace the activation function in the network detection head to obtain a fully pulsed underwater target detection model SU-YOLO; the present invention performs performance evaluation on the constructed model and applies it to actual scenarios to realize an underwater target detection method based on pulse neural networks.

[0150] Example:

[0151] Table 1 Comparative experiments of SU-YOLO and mainstream models

[0152]

[0153] Experimental results on URPC2019 and UDD. mAP0.5 represents the mean average precision at an IoU threshold of 0.5, and mAP@0.5:0.95 covers IoU thresholds from 0.5 to 0.95. For SNNs, the computational load includes FLOPs (from encoding) and SOPs. SpikeYOLO's time is calculated as timesteps × virtual timesteps. mAP values marked with * are from the ANN before conversion. The conversion from ANN to SNN is assumed to be lossless at T = 3500.

[0154] Experimental results show that directly trained spiking neural networks (SNNs) outperform spiking neural networks obtained by converting artificial neural networks (ANNs) to spiking neural networks when using fewer time steps. Compared to existing directly trained spiking neural networks, SU-YOLO achieves higher mean average precision (mAP) with the same number of time steps and fewer parameters. Furthermore, the model demonstrates comparable performance when compared to lightweight ANNs.

[0155] EMS-YOLO is the current benchmark for object detection in SNNs. At URPC2019, EMS-YOLO achieved a mAP@0.5 of 0.672 using the lightest version of ResNet10 with an energy consumption of 2.73 mJ. As the network size increases, EMS-YOLO achieves an even higher mAP@0.5 of 0.735 using ResNet34. SpikeYOLO performs even better, reaching an mAP@0.5 of 0.784 with an energy consumption of 8.64 mJ when using one time step and four virtual time steps. Our method surpasses these results, achieving an mAP@0.5 of 0.788 using only 6.97M parameters and a computational load of 2.90 GSOPs. Furthermore, with an energy consumption as low as 2.98 mJ, it significantly outperforms other SNNs for underwater object detection.

[0156] The experiments also compared the model with several ANN networks. Although SU-YOLO's mean average precision (mAP) was slightly lower than that of larger, more advanced ANNs like YOLOv7, the model's energy consumption was only 5% of YOLOv7. For lightweight ANNs with similar energy consumption, SU-YOLO showed higher performance.

[0157] To verify the generalization ability of SU-YOLO, additional experiments were conducted on UDD. The results show that the model still performs best compared to other methods.

[0158] Table 2 Ablation experiments of each module in SU-YOLO

[0159]

[0160] Ablation experiments of each module in SU-YOLO on URPC2019 and UDD. BL, SD, SS, SB2, and SB1 represent the baseline model, Spike Denoiser, Spike Spatial Pyramid Pooling (SpikeSPP), SU-Block2, and SU-Block1, respectively.

[0161] Experiments evaluated the effectiveness of the improved modules in SU-YOLO. Maintaining the same network structure, the experiment progressively replaced the SpikeDenoiser modules with the original YOLOv9 modules as a baseline. As shown in Table 3-2, adding each module reduced parameter size while improving accuracy. When all improvements were applied, mAP@0.5 increased by 4.5% and mAP@0.5.95 increased by 3.3%. Furthermore, when the SpikeDenoiser module was added to the baseline model, it independently achieved a 0.9% improvement in mAP@0.5 performance.

[0162] Table 3 Separate batch normalization comparison test

[0163]

[0164] Detection results of BN, tdBN, TEBN and SeBN on the URPC2019 dataset and Pascal VOC 2012 dataset.

[0165] SeBN, an innovative normalization method for spiking neural networks, was compared with the traditional BN method. Models using traditional BN performed relatively poorly. tdBN and TEBN showed some improvement over BN; however, TEBN underperformed tdBN in the EMS-YOLO model, likely due to its lack of consideration of the residual structure in element-wise summation, which makes extensive use of such residual blocks. In SU-YOLO, TEBN and tdBN performed similarly due to the less extensive use of element-wise summation residual structures. Replacing these two methods with SeBN further improved detection accuracy. The experiments also tested the effectiveness of SeBN on EMS-YOLO, finding that SeBN performed best. To further validate SeBN, experiments were conducted on PascalVOC. SeBN also demonstrated excellent performance for general object detection, achieving the highest mean average precision (mAP).

Claims

1. A method for underwater target detection based on a pulse neural network, characterized in that: The following steps are involved: Step 1: Input the RGB image into the pulse encoder (1) and output the binary pulse feature map; Step 2: Send the feature map output in step 1 to the pulse denoiser (2) to perform image denoising and output a cleaner feature map; Step 3: The denoised feature map from step 2 is fed into the model backbone, where multi-scale features are extracted through the SU-Block1 feature extraction module and the SpikeSPP pulse pyramid pooling module, outputting a larger number of feature maps with richer information. Step 4: The feature map output in step 3 is fed into the neck part of the model, and feature fusion is performed through the SU-Block2 feature fusion module, the feature channel splicing layer, the upsampling layer, and the downsampling layer. The feature information from each layer is spliced and fused and then output; Step 5: Send the feature map output in step 4 to the pulse detection head and output the final detection result; Through the above steps, underwater target detection based on pulse neural network is realized.

2. The method according to claim 1, characterized in that In step 2, the pulse denoiser (2) is implemented as follows: 2-1) In order to find missing pixels and isolated pixels in the feature map, three convolution filters are defined Among them, K1 and K2 are used to detect negative noise, and K3 is used to detect positive noise; 2-2) Define the judgment and operation rules. If the convolution result of the current pixel in the feature map and the filter is equal to 4, the current pixel value is inverted. Otherwise, no processing is performed. This process can be described by the following formula: Among them, I in (x, y) is the pixel value at coordinate (x, y) in the input feature map, I in (x+i, y+j) is the pixel value at the coordinate (x+i, y+j) in the input feature map, I conv (x, y) is the convolution result, I out (x, y) is the output, K n (n=1,2,3) represents the nth convolution kernel used, K n (i+1, j+1) represents the convolution kernel K n The value at coordinate (i+1,j+1); 2-3) Perform pixel-by-pixel convolution on the feature map output by the pulse encoder using the K1 kernel according to the rules in 2-2), fill in missing pixels, and output the processed feature map; 2-4) Perform pixel-by-pixel convolution on the feature map output in 2-3) using the K2 kernel according to the rules in 2-2), fill in the missing pixels again, and output the processed feature map; 2-5) Perform convolution operations pixel by pixel on the feature map outputted by 2-4) using the K3 kernel according to the rules in 2-2) to remove isolated pixels. The processed feature map is then outputted, which is the final output of the pulse denoiser.

3. The method according to claim 1, characterized in that In step 3, the model backbone network consists of three SU-Block1 feature extraction modules and a SpikeSPP pulse pyramid pooling module arranged in sequence, specifically including the following steps: 3-1) Downsampling (3) the feature map output by the pulse denoiser (2) and using it as the input of the first SU-Block1 feature extraction module (4) to extract preliminary features; 3-2) The output of the first SU-Block1 feature extraction module (4) is used as the input of the second SU-Block1 feature extraction module (5) to extract features again; 3-3) The output of the second SU-Block1 feature extraction module (5) is used as the input of the third SU-Block1 feature extraction module (6) to extract features for the last time; 3-4) The output of the third SU-Block1 feature extraction module (6) is sent to SpikeSPP (7) for multi-level pooling, and the multi-scale feature map is output to the model neck network.

4. The method according to claim 3, characterized in that The processing flow of the SU-Block1 feature extraction module is as follows: S1) Copy the feature map input to the SU-Block1 feature extraction module into two copies, namely feature map F1 and feature map F2; S2) wherein, The feature map F1 undergoes the first 2D convolution (24) with a kernel size of 3×3, and undergoes separable batch normalization (25), and then undergoes feature channel segmentation (28), and is divided into two parts along the channel dimension, namely feature map F3 and feature map F4; wherein the feature map F3 remains unchanged, and the feature map F4 is sent to the IF pulse neuron (29), and its output feature map undergoes the second 2D convolution (30) and separable batch normalization (31) again to obtain feature map F5; the feature map F2 undergoes the third 2D convolution (26) with a kernel size of 1×1, and undergoes separable batch normalization (27), and the output feature map is added element-by-element to the feature map F5 to obtain feature map F6; S3) performing feature channel concatenation (32) on the feature map F6 and the feature map F3, sending the concatenated feature maps to the IF pulse neuron (33), and outputting the final extracted feature map; The processing flow of SU-Block1 can be expressed by the formula: where X l 1 and X l 2 represents the two feature maps obtained after the lth layer segmentation, X l out represents the output of the lth layer, IF(·) represents the IF neuron; Conv(·) represents the convolutional layer, Split(·) is an operation that divides the input into two equal parts along the channel dimension, and Concat(·) is an operation that concatenates the input along the channel dimension.

5. The method according to any one of claims 1 to 4, characterized in that The implementation steps of the separable batch normalization method are as follows: S1) concatenate the input feature maps of multiple time steps along the channel dimension and remove the time dimension; S2) Calculate the mean of the concatenated feature map and variance Where N is the batch size, x t,k represents the feature map of the kth channel at time step t, μ t,k represents the mean value of the kth channel at time step t, σ 2 t,k represents the variance of the kth channel at time step t: S3) Calculate the normalized data element by element Among them, x t,k represents the feature map of the kth channel at time step t, n is a positive integer representing the number of SeBN modules connected to neurons after the current module, and ε is a small constant to avoid division by zero; t,k and β t,k is a trainable parameter, y t,k is the normalized output; μ t,k and σ 2 t,k They are x t,k The mean and variance of y t,k is the normalized feature map of the kth channel at time step t; S4) Re-add the time dimension to the normalized feature map and evenly divide it into the time dimension along the channel dimension according to the number of time steps, so as to keep consistent with the input data shape.

6. The method according to claim 1 or 3, characterized in that The processing flow of the SpikeSPP pulse pyramid pooling module is as follows: S1) performs a 2D convolution (15) on the input feature map and performs a separate batch normalization (16) to obtain the output feature map F7; S2) performing maximum pooling (17) on the feature map F7 to obtain the feature map F8; S3) performing maximum pooling (18) on the feature map F8 to obtain the feature map F9; S4) The feature maps F7, F8, and F9 are concatenated along the feature channel (19), and sequentially passed through IF pulse neurons (20), 2D convolution (21), separable batch normalization (22), and IF pulse neurons (23) to output the final feature map.

7. The method according to claim 6, characterized in that The processing flow of the SpikeSPP pulse pyramid pooling module can be expressed by the formula: where X l 1. X l 2. X l 3 represent the feature maps after 2D convolution, first pooling, and second pooling in the first layer, respectively. l out Represents the output of the lth layer, IF(·) represents the IF neuron; Conv(·) represents the convolutional layer, MaxPool(·) represents the maximum pooling layer, and Concat(·) is the concatenation operation on the input along the channel dimension.

8. The method according to claim 1, characterized in that In step 4, the model neck network contains two SU-Block2 feature fusion modules, two feature channel splicing layers, an upsampling layer, and a downsampling layer, specifically including the following steps: 4-1) Upsampling the feature map output by SpikeSPP (7) (8), performing feature channel splicing (9) with the output of the second SU-Block1 feature extraction module (5), and outputting it to the first SU-Block2 feature fusion module (10) for processing; 4-2) Downsampling (11) the feature map output by the first SU-Block2 feature fusion module (10), performing feature channel splicing (12) with the output of SpikeSPP (7), and outputting it to the second SU-Block2 feature fusion module (13) for processing; 4-3) The output results of the first SU-Block2 feature fusion module (10) and the second SU-Block2 feature fusion module (13) are sent to the pulse detection head, and the detection results are output to complete the target detection.

9. The method according to claim 8, characterized in that The processing flow of the SU-Block2 feature fusion module is as follows: S1) The feature map of the input SU-Block2 is subjected to 2D convolution (34) with a kernel size of 3×3, and is subjected to separate batch normalization (35). Then, the feature channel segmentation (36) is performed and the feature map is divided into two parts along the channel dimension, namely, feature map F10 and feature map F11. Feature map F10 remains unchanged, and feature map F11 is sent to the IF pulse neuron (37). The output feature map is again subjected to 2D convolution (38) and separate batch normalization (39) to obtain feature map F12; S2) The feature map F10 and the feature map F12 are spliced together in feature channels (40), and sent to the IF pulse neuron (41), and the final fused feature map is output.

10. The method according to claim 9, characterized in that The processing flow of the SU-Block2 feature fusion module can be expressed by the formula: where X l 1 and X l 2 represents the two feature maps obtained after the lth layer segmentation, X l out represents the output of layer l, IF(·) represents the IF neuron, Conv(·) represents the convolutional layer, Split(·) is an operation that splits the input into two equal parts along the channel dimension, and Concat(·) is an operation that concatenates the input along the channel dimension.

Images