Pulse neural network conversion method and device for autonomous detection of underwater sonar

Abstract

The present application provides a kind of pulse neural network conversion method and device for underwater sonar autonomous detection, the method comprises the following steps: step S1: modify YOLOv3-tiny model based on deep neural network model (DNN), obtain the YOLOv3-tiny model after modification;Step S2: load underwater sonar image data set, train the YOLOv3-tiny model after modification;Step S3: for underwater target detection, the trained YOLOv3-tiny model is converted into YOLOv3-SNN model based on pulse neural network (SNN), and the YOLOv3-SNN model includes input layer, hidden layer and output layer, wherein input layer uses real number coding, hidden layer uses the two-state pulse neural coding of active state and resting state, and output layer uses membrane voltage decoding;Step S4: using the YOLOv3-SNN model based on pulse neural network (SNN), is directly used for underwater sonar small target detection task.

Description

Technical Field

[0001] This invention belongs to the field of image target detection applications, and particularly relates to a pulse neural network conversion method and device for autonomous underwater sonar detection. Background Technology

[0002] Unmanned surface vessels (USVs), as unmanned surface ships, are characterized by their small size, high speed, strong maneuverability, and modularity, making them suitable for dangerous missions and tasks unsuitable for manned vessels. With the development of new technologies such as big data, artificial intelligence, and computer vision, USVs equipped with sonar devices can autonomously detect and identify underwater targets, and are now widely used in both civilian and military fields. Currently, target detection algorithms based on deep neural networks (DNNs) have demonstrated excellent performance in various complex unmanned application scenarios. However, a drawback is that these models require high-performance graphics cards for deployment, and the power consumption of high-performance graphics cards far exceeds that of ordinary computing devices. Many computers or embedded chips have limited computing power and cannot meet the high-precision floating-point operation requirements of DNNs like graphics cards; furthermore, the energy resources of USVs and small devices are limited, making it difficult to perform all-weather autonomous underwater detection even with high-performance graphics cards.

[0003] In recent years, spiking neural network (SNN) technology, inspired by the biological brain, has attracted widespread attention because it can operate with extremely low power on neuromorphic chips, solving the challenges of applying DNNs in resource-constrained systems. SNNs are hailed as the third generation of neural networks due to their high biomimeticity and low power consumption. They transmit information between neurons using discrete binary "0-1" signal pulse sequences; therefore, at the same model complexity, SNNs have lower accuracy than DNNs. In SNNs, spiking neurons output a pulse signal only if their membrane voltage exceeds a certain threshold after accumulating input. This pulse addition method eliminates the need for multiplication in forward operations and allows for hardware deployment via electrical signal superposition, thus achieving low-power operation.

[0004] Directly training deep SNN models faces difficulties such as non-differentiability, resulting in a significant gap in accuracy compared to DNNs. Therefore, transformation-based methods are gaining popularity. These methods convert trained DNN models and their parameters into SNN representations, sacrificing model accuracy to achieve deep SNNs. Rueckauer et al. proposed a comprehensive transformation scheme, enabling the VGG-SNN model to achieve high accuracy in image classification tasks. Kim et al. also proposed a series of spiking neural network encoding methods to improve their accuracy in image classification. However, current SNN research has limitations, with most focusing on image classification tasks, which are relatively simple and feature limited network models. Developing low-power spiking neural networks for autonomous underwater sonar detection to achieve target detection is a promising area of ​​research. Summary of the Invention

[0005] In view of this, the present invention provides a pulse neural network conversion method for autonomous underwater sonar detection. By taking advantage of the low power consumption characteristics of its network binary transmission, the method solves the difficulty of applying deep neural networks in resource-constrained underwater autonomous detection systems at the algorithm level. The method includes: step S1: modifying the YOLOv3-tiny model based on the deep neural network model (DNN) to obtain the modified YOLOv3-tiny model.

[0006] Step S2: Load the underwater sonar image dataset and train the modified YOLOv3-tiny model;

[0007] Step S3: To perform underwater target detection, the trained YOLOv3-tiny model is converted into a YOLOv3-SNN model based on a spiking neural network (SNN). The YOLOv3-SNN model includes an input layer, a hidden layer, and an output layer. The input layer uses real number encoding, the hidden layer uses two-state spiking neural encoding (active state and resting state), and the output layer uses membrane voltage decoding.

[0008] Step S4: Using a two-state encoding scheme, the generated YOLOv3-SNN model based on spiking neural network (SNN) is directly used for underwater sonar small target detection tasks.

[0009] Specifically, the modification of the YOLOv3-tiny model based on the deep neural network (DNN) model in step S1 includes: replacing the max pooling layer in the YOLOv3-tiny model with a 3×3 convolutional layer of equal stride to simulate the function of max pooling; replacing the activation function that outputs negative values ​​with the ReLU function; replacing the upsampling layer with a deconvolutional layer; and merging the parameters of the batch normalization (BN) layer into the convolutional layer.

[0010] Specifically, in step S2, during the training process, multi-scale training is performed for each iteration, scaling the short side length of the input image to a random value in the range of 320 to 640, and scaling the image resolution to a uniform size of 416×416 when testing the image.

[0011] Specifically, in step S3, in the YOLOv3-SNN model, the spiking neuron uses an integral firing model, where the spiking neuron retains a portion of its membrane voltage after firing a pulse, and is likely to fire another pulse at the next moment; the calculation formula is shown below:

[0012]

[0013] In the formula, the spiking neuron i in the l-th layer inputs its data at time t. Accumulated to its membrane voltage superior; Indicates whether to output a pulse; the value is 1 or 0. Input The calculation method and The generation method is as follows:

[0014]

[0015]

[0016] Where w l and b 1 These are the parameters of the l-th layer, V th (t) represents the voltage threshold, and U(·) represents the unit step function, used to determine whether the membrane voltage exceeds the threshold, as shown in the following formula:

[0017]

[0018] Specifically, in step S3, in the input layer encoding method, the YOLOv3-SNN model directly skips the input layer and adopts a real-valued encoding scheme, applying a constant voltage input to each neuron in the first layer network at each time step. The calculation formula is as follows:

[0019]

[0020] Finally, in order to obtain the target bounding box coordinates, the YOLOv3-SNN model designed an output layer voltage decoding method, namely, letting... The output neurons are prevented from firing pulses, and the membrane voltage accumulated over all time steps is directly used as the real value after decoding.

[0021] Specifically, in step S3, the hidden layer employs a two-state spiking neural encoding, consisting of an active state and a resting state. Based on the frequency of the emitted pulses, neurons switch between these two states during information transmission; its specific manifestation is V. th (t) changes dynamically over time and is specific to each neuron, rather than a global variable, as shown in the following equation:

[0022]

[0023] Where α represents a larger active state threshold, and β represents a smaller resting state threshold; the initial voltage threshold for each neuron. A neuron enters an active state when it fires two consecutive pulses; in this state, the threshold... Get bigger, if The added membrane voltage will be enhanced, and if Membrane voltage is suppressed; the active state is a process that simulates long-term enhancement and long-term inhibition; and once the neuron does not fire a pulse at the current moment, it becomes a resting state, waiting for the next activation.

[0024] Specifically, the YOLOv3-SNN model uses a channel-based parameter normalization method to apply normalization constraints to the model parameters. When calculating the maximum value of the output of each layer, the normalization is refined to each channel. This allows for differentiated calculation of the normalization factor for each convolutional kernel in the convolutional layer, avoiding excessively low firing rates in some channels. The calculation expression is as follows:

[0025] and

[0026] Where j represents the first... l The number of output channels of the convolutional layer, where i represents the number of input channels, i.e., the number of output channels of the (l-1)th layer; the 99th largest value among the output values ​​of each layer is used as the normalization factor λ for that layer to appropriately improve the firing rate of neurons. The firing rate is calculated as follows:

[0027] Where T is the total time step simulated by the SNN, and N is the number of pulses emitted during this period.

[0028] The present invention also proposes a pulse neural network conversion device for autonomous underwater sonar detection, comprising: a deep neural network model modification module, used to modify the YOLOv3-tiny model based on a deep neural network model (DNN) to obtain a modified YOLOv3-tiny model;

[0029] The model training module is used to load the underwater sonar image dataset and train the modified YOLOv3-tiny model.

[0030] The spiking neural network model conversion module is used to convert the trained YOLOv3-tiny model into a YOLOv3-SNN model based on spiking neural networks (SNN) for underwater target detection. The YOLOv3-SNN model includes an input layer, a hidden layer, and an output layer. The input layer uses real number encoding, the hidden layer uses two-state spiking neural encoding (active state and resting state), and the output layer uses membrane voltage decoding.

[0031] The target detection module is used to directly apply the generated YOLOv3-SNN model based on the spiking neural network (SNN) to the underwater sonar small target detection task through a two-state encoding scheme.

[0032] Beneficial effects:

[0033] (1) In this invention, a spiking neural network is applied to the autonomous underwater sonar detection mission. By taking advantage of the low power consumption characteristics of its network binary transmission, the difficulty of applying deep neural networks in resource-constrained underwater autonomous detection systems is solved at the algorithm level.

[0034] (2) In this invention, a conversion scheme specifically designed for target detection tasks is designed, and a multi-scale spiking neural network target detection model is generated for target detection in sonar images;

[0035] (3) Inspired by the biological neurological phenomena of long-term enhancement and long-term inhibition, this invention proposes a two-state encoding scheme for the hidden layer to accelerate the efficiency of pulse information transmission and shorten the model runtime.

[0036] (4) The YOLOv3-SNN model described in this invention uses a channel-based parameter normalization method to perform normalization constraints on the model parameters. This channel-based parameter normalization method improves the low firing rate of deep neurons in the SNN, thereby enhancing the model's detection accuracy. To avoid the negative impact of maximum outliers;

[0037] (5) In this invention, YOLOv3-SNN uses an integral firing model in spiking neurons and adopts a “soft reset” method. The soft reset method allows spiking neurons to retain a portion of the membrane voltage after firing a pulse, and they are likely to fire a pulse again at the next moment. Theoretically, it can achieve higher detection accuracy than the hard reset method and requires less time step. Attached Figure Description

[0038] Figure 1 This is a flowchart of a pulse neural network for autonomous underwater sonar detection in this invention;

[0039] Figure 2This is a model framework diagram of the improved YOLOv3-tiny in this invention;

[0040] Figure 3 This is a schematic diagram illustrating the use of YOLOv3-SNN to process sonar images in this invention. Detailed Implementation

[0041] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0042] This invention provides, for example Figure 1 As shown, the specific implementation steps of the pulse neural network conversion method for autonomous underwater sonar detection of the present invention are as follows:

[0043] Step S1: Modify some modules in the YOLOv3-tiny model that cannot or are inconvenient to be represented by SNN. The modified model framework diagram is as follows. Figure 2 As shown.

[0044] First, regarding max pooling layers, using max pooling in SNNs increases the model's computation time from O(T) to O(nT), where T is the simulation time step and n is the number of max pooling layers. Therefore, in classification tasks, the original DNN network typically uses average pooling layers, which achieve similar results to max pooling layers. However, in advanced vision tasks such as object detection, it's usually necessary to capture the most prominent features in the image. Replacing max pooling with average pooling in the detection model would significantly reduce detection accuracy. Therefore, DNN models for object detection almost exclusively use max pooling. Thus, to balance detection accuracy and runtime, the max pooling layers in the YOLOv3-tiny model are replaced with 3×3 convolutional layers of equal stride to simulate the function of max pooling.

[0045] Then, the Leaky_ReLU activation function, which produces negative values ​​in the YOLOv3-tiny model, is replaced with ReLU. This is because SNNs only transmit "0-1" pulses; if negative values ​​were to be transmitted, the number of neuron synapses would need to be doubled to represent a negative 1 pulse, which is counterproductive.

[0046] Meanwhile, to achieve end-to-end computation of the SNN model, the double upsampling operation is replaced with a 3×3 deconvolution layer. The deconvolution layer is computed in the same way as the convolution layer, so it can be converted into an SNN.

[0047] In addition, by merging the parameters of the batch normalization (BN) layer into the convolutional layer and not treating the BN layer as a separate layer, the computational efficiency of the binary pulse data in each network layer is improved.

[0048] Step S2: Load the underwater sonar image dataset and train the modified YOLOv3-tiny model.

[0049] To improve the robustness of the model, multi-scale training is performed for each iteration during training. The shorter side length of the input image is scaled to a random value in the range of 320 to 640. When testing the image, the image resolution is uniformly scaled to 416×416.

[0050] The model training cycle is 300, with 8 images input per iteration. The initial learning rate is 0.01, which is decreased throughout the training process using a cosine function.

[0051] Step S3: Design an SNN conversion scheme for object detection to convert the trained YOLOv3-tiny model into a YOLOv3-SNN.

[0052] First, to avoid significant information loss, YOLOv3-SNN uses an integral firing model in its spiking neurons and employs a "soft reset" method. The soft reset method allows the spiking neuron to retain a portion of its membrane voltage after firing a pulse, making it more likely to fire again in the next time step. Theoretically, it can achieve higher detection accuracy than the hard reset method and requires fewer time steps. The calculation formula is shown below:

[0053]

[0054] In the formula, the spiking neuron i in the l-th layer inputs its data at time t. Accumulated to its membrane voltage superior. Indicates whether to output a pulse; the value is 1 or 0. Input The calculation method and The generation method is as follows:

[0055]

[0056]

[0057] Where w l and b l These are the parameters of the l-th layer, V th (t) represents the voltage threshold, which is typically set to a constant in basic conversion methods, but it can be designed as a function that varies with time. U(·) represents the unit step function, used to determine whether the membrane voltage exceeds the threshold, as shown in the following equation:

[0058]

[0059] YOLOv3-SNN uses a channel-based parameter normalization method to apply normalization constraints to the model parameters. When calculating the maximum output value for each layer, it refines the normalization to each channel. This allows for differentiated calculation of the normalization factor for each convolutional kernel in the convolutional layers, preventing some channels from having excessively low firing rates. The calculation expression is as follows:

[0060] and

[0061] Where j represents the number of output channels in the l-th convolutional layer, and i represents the number of input channels, i.e., the number of output channels in the (l-1)-th layer. Channel-based parameter normalization improves the low firing rate of deep neurons in SNNs, thus enhancing model detection accuracy. To avoid the negative impact of outlier values, the 99th largest value among the output values ​​of each layer is used as the normalization factor λ for that layer to appropriately increase the firing rate of neurons. The firing rate is calculated as follows:

[0062]

[0063] Where T is the total time step simulated by the SNN, and N is the number of pulses emitted during this period (at most one pulse is emitted at each moment). Too high an ignition rate will increase the model power, while too low an ignition rate will cause information transmission to be obstructed. Therefore, a moderate ignition rate is more suitable, which is also the calculation principle of rate coding.

[0064] Then, in the input layer encoding method, YOLOv3-SNN skips the input layer and adopts a real-valued encoding scheme, applying a constant voltage input to each neuron in the first layer at each time step. The calculation formula is as follows:

[0065]

[0066] Finally, to obtain the target bounding box coordinates, YOLOv3-SNN designed an output layer voltage decoding method, namely, letting... The output neurons are prevented from firing pulses, and the membrane voltage accumulated over all time steps is directly used as the real value after decoding.

[0067] Step S4: Using a two-state encoding scheme, an end-to-end spiking neural network is finally generated, which can be directly used for underwater sonar small target detection tasks. The process of processing sonar images is as follows: Figure 3 As shown.

[0068] There are many biological neurological phenomena, and many of these can serve as the basis for encoding SNN neurons. This method draws on the phenomenon of long-term potentiation (LTP), where repeated high-frequency stimulation of the former can enhance the postsynaptic membrane potential of the latter over a prolonged period, while the opposite phenomenon is called long-term inhibition (LTI). This enables a two-state encoding scheme, assigning SNN neurons two states: an active state and a resting state. The neuron switches between these two states during information transmission based on the frequency of the emitted pulses. Its specific manifestation is V... th (t) changes dynamically over time and is specific to each neuron, rather than a global variable, as shown in the following equation:

[0069]

[0070] Where α represents a larger active state threshold and β represents a smaller resting state threshold. The initial voltage threshold for each neuron. A neuron enters an active state when it fires two consecutive pulses; in this state, the threshold... Get bigger, if The added membrane voltage will be enhanced, and if Membrane voltage is suppressed. The active state is a process that simulates long-term enhancement and long-term inhibition. Once the neuron does not fire a pulse at the current moment, it enters a resting state, waiting for the next activation.

[0071] Ultimately, the entire YOLOv3-SNN adopts a hybrid encoding mechanism, with real-number encoding used in the input layer, two-state encoding used in the hidden layer, and membrane voltage decoding used in the output layer.

[0072] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0073] It will be apparent to those skilled in the art that the embodiments of the present invention are not limited to the details of the exemplary embodiments described above, and that the embodiments of the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the embodiments of the present invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the embodiments of the present invention is defined by the appended claims rather than the foregoing description. Therefore, all variations falling within the meaning and scope of equivalents of the claims are intended to be encompassed within the embodiments of the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units, modules, or devices recited in the system, apparatus, or terminal claims may also be implemented by the same unit, module, or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention and are not intended to limit them. Although the embodiments of the present invention have been described in detail with reference to the above preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the embodiments of the present invention should not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A pulse neural network conversion method for autonomous underwater sonar detection, characterized in that, include: Step S1: Modify the YOLOv3-tiny model based on the deep neural network model (DNN) to obtain the modified YOLOv3-tiny model; Step S2: Load the underwater sonar image dataset and train the modified YOLOv3-tiny model; Step S3: To perform underwater target detection, the trained YOLOv3-tiny model is converted into a YOLOv3-SNN model based on a spiking neural network (SNN). The YOLOv3-SNN model includes an input layer, a hidden layer, and an output layer. The input layer uses real number encoding, the hidden layer uses two-state spiking neural encoding (active state and resting state), and the output layer uses membrane voltage decoding. Step S4: Using a two-state encoding scheme, the generated YOLOv3-SNN model based on a spiking neural network (SNN) is directly applied to the underwater sonar small target detection task; in step S3, the hidden layer adopts a two-state spiking neural encoding with active and resting states. The neurons switch between the two states during information transmission based on the frequency of the emitted pulses; its specific manifestation is as follows: It changes dynamically over time, and is specific to each neuron, rather than a global variable, as shown in the following equation: in Indicates the active state threshold. Represents the resting state threshold; the initial voltage threshold for each neuron. A neuron enters an active state when it fires two consecutive pulses; in this state, the threshold... Get bigger, if The added membrane voltage will be enhanced, and if Membrane voltage is suppressed; the active state is a process that simulates long-term enhancement and long-term inhibition; and once the neuron does not fire a pulse at the current moment, it becomes a resting state, waiting for the next activation.

2. The pulse neural network conversion method for autonomous underwater sonar detection as described in claim 1, characterized in that, The modification of the YOLOv3-tiny model based on the deep neural network (DNN) model in step S1 specifically includes: replacing the max pooling layer in the YOLOv3-tiny model with a 3×3 convolutional layer with equal strides to simulate the function of max pooling; replacing the activation function that outputs negative values ​​with the ReLU function; replacing the upsampling layer operation with a deconvolutional layer; and merging the parameters of the batch normalization (BN) layer into the convolutional layer.

3. The pulse neural network conversion method for autonomous underwater sonar detection as described in claim 1, characterized in that, In step S2, during the training process, multi-scale training is performed for each iteration, scaling the short side length of the input image to a random value in the range of 320 to 640, and scaling the image resolution to 416×416 when testing the image.

4. The pulse neural network conversion method for autonomous underwater sonar detection as described in claim 1, characterized in that, In step S3, the YOLOv3-SNN model uses an integral firing model for spiking neurons, where the spiking neuron retains a portion of the membrane voltage after firing a pulse, and fires another pulse at the next moment. The calculation formula is as follows: In the formula Layer of spiking neurons exist Always input it Accumulated to its membrane voltage superior; Indicates whether to output a pulse, with a value of 1 or 0; Input The calculation method and The generation method is as follows: in and It is the first Layer parameters, For voltage threshold, The unit step function, used to determine whether the membrane voltage exceeds a threshold, is shown in the following equation: 。 5. The pulse neural network conversion method for autonomous underwater sonar detection as described in claim 4, characterized in that, In step S3, in the input layer encoding method, the YOLOv3-SNN model directly skips the input layer and adopts a real-valued encoding scheme, applying a constant voltage input to each neuron in the first layer network at each time step, as calculated by the following formula: Finally, in order to obtain the target bounding box coordinates, the YOLOv3-SNN model designed an output layer voltage decoding method, namely, letting... The output neurons are prohibited from firing pulses, and the membrane voltage accumulated over all time steps is directly used as the real value after decoding.

6. The pulse neural network conversion method for autonomous underwater sonar detection as described in claim 1 or 2, characterized in that, The YOLOv3-SNN model uses a channel-based parameter normalization method to normalize the model parameters. When calculating the maximum value of the output of each layer, it is refined to each channel. This allows for differentiated calculation of the normalization factor for each convolutional kernel in the convolutional layer, avoiding excessively low firing rates in some channels. The calculation expression is as follows: and in Representing the The number of output channels of a convolutional layer. Represents the number of input channels, i.e., the first... Number of output channels of the layer; Use the 99th largest value among the output values ​​of each layer as the normalization factor for that layer. To appropriately increase the ignition rate of neurons, the ignition rate is calculated as follows: firing rate ;in The total time step simulated by the SNN. This represents the number of pulses emitted during this period.

7. A pulse neural network conversion device for autonomous underwater sonar detection, characterized in that, include: The Deep Neural Network Model Modification Module is used to modify the YOLOv3-tiny model based on the Deep Neural Network (DNN) model to obtain the modified YOLOv3-tiny model. The model training module is used to load the underwater sonar image dataset and train the modified YOLOv3-tiny model. The spiking neural network model conversion module is used to convert the trained YOLOv3-tiny model into a YOLOv3-SNN model based on spiking neural networks (SNN) for underwater target detection. The YOLOv3-SNN model includes an input layer, a hidden layer, and an output layer. The input layer uses real number encoding, the hidden layer uses two-state spiking neural encoding (active state and resting state), and the output layer uses membrane voltage decoding. The target detection module is used to directly apply the generated YOLOv3-SNN model based on a spiking neural network (SNN) to underwater sonar small target detection tasks using a two-state encoding scheme. In the spiking neural network model conversion module, the hidden layer employs a two-state spiking neural encoding with active and resting states. The neurons switch between these two states during information transmission based on the frequency of the emitted pulses. Its specific implementation is as follows: It changes dynamically over time, and is specific to each neuron, rather than a global variable, as shown in the following equation: in Indicates the active state threshold. Represents the resting state threshold; the initial voltage threshold for each neuron. A neuron enters an active state when it fires two consecutive pulses; in this state, the threshold... Get bigger, if The added membrane voltage will be enhanced, and if Membrane voltage is suppressed; the active state is a process that simulates long-term enhancement and long-term inhibition; and once the neuron does not fire a pulse at the current moment, it becomes a resting state, waiting for the next activation.

Images