Low-delay SNN hyperspectral image classification method of brain-mind-like development mechanism

Abstract

The invention relates to the technical field of remote sensing information processing, and provides a brain-mind development mechanism-like low-delay SNN hyperspectral image classification method, which comprises the following steps of: 1, constructing a hyperspectral image classification model SNN-ATC based on an attention mechanism and a dual-channel SNN; step 2, simulating brain nerve plasticity and a mental development mechanism to prune the hyperspectral image classification model SNN-ATC to obtain a hyperspectral image classification model SNN-BMC; step 3, training the hyperspectral image classification model SNN-BMC by using a knowledge distillation method; and step 4, realizing hyperspectral image classification based on the trained hyperspectral image classification model SNN-BMC. According to the method, the real-time performance of classification can be improved on the premise of ensuring the performance of the network model.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing information processing, and particularly to a low-latency SNN hyperspectral image classification method for the mechanism of brain-like mental development. Background Art

[0002] Hyperspectral images are multi-dimensional data containing multiple bands. They measure the spectral characteristics of an observed target within dozens to hundreds of narrow and continuous bands through a spectral imager on an aircraft or satellite, and convert the spectral measurement values of each band into the digital bits of image pixels, thereby obtaining the spectral response image of the target in all bands. By analyzing the spectral curves of each pixel, information such as the material composition and content of the target can be quantitatively analyzed. Compared with traditional visible light images, hyperspectral images can provide richer spectral information and can more accurately identify the spectral characteristics of objects, and are used for monitoring and analysis in fields such as agriculture, mineral resources, geological exploration, and vegetation coverage.

[0003] Hyperspectral image classification is widely used in remote sensing image analysis. For the hyperspectral image classification task, researchers usually use a convolutional neural network (CNN) for modeling. Based on 3D-CNN, spatial features and spectral features can be extracted from hyperspectral images simultaneously, thereby achieving a relatively high classification accuracy. However, this method requires a large amount of matrix calculations, resulting in high computational complexity and energy consumption. On the other hand, although the method based on a spiking neural network (SNN) can effectively reduce energy consumption, in order to ensure the model performance, usually a larger time step needs to be adopted for the network, which limits its inference speed and is difficult to meet the application scenarios with high real-time requirements.

[0004] To improve the inference speed of the spiking neural network in hyperspectral images, by simulating the brain's neural plasticity and mental development mechanism, calculate the activity level of each synapse and its contribution to the network output, and prune the synapses with activity levels lower than the threshold θ. In addition, the knowledge distillation technique is used to compress the time dimension of the student network, and its time step is dynamically adjusted during the training process to ensure a lower time step while maintaining high performance. Summary of the Invention

[0005] Aiming at the problem that the existing hyperspectral image classification technology cannot simultaneously take into account the performance of the network model and the inference speed, the present invention proposes a low-latency SNN hyperspectral image classification method for the mechanism of brain-like mental development. By simulating the brain's neural plasticity and mental development mechanism, prune the synapses with activity levels lower than the threshold θ, and use the knowledge distillation technique to compress the time dimension of the student network, and dynamically adjust its time step during the training process to ensure a lower time step while maintaining high performance.

[0006] A low-latency SNN hyperspectral image classification method for a brain-like mental development mechanism provided by the present invention includes:

[0007] Step 1: Construct a hyperspectral image classification model SNN-ATC based on the attention mechanism and the dual-channel SNN;

[0008] Step 2: Prune the hyperspectral image classification model SNN-ATC by simulating the brain neuroplasticity and mental development mechanism to obtain a hyperspectral image classification model SNN-BMC;

[0009] Step 3: Train the hyperspectral image classification model SNN-BMC using the knowledge distillation method;

[0010] Step 4: Implement hyperspectral image classification based on the trained hyperspectral image classification model SNN-BMC.

[0011] Further, the specific steps of Step 1 include:

[0012] Step 1.1: Construct a dual-channel SNN, which includes a convolutional layer, a dual-channel module, a channel mixing layer, a convolutional layer, an average pooling layer, a convolutional layer, and an average pooling layer connected in sequence; the first channel module in the dual-channel module includes two 3×3 convolutional layers, and the second channel module includes a 3×3 depthwise separable convolutional layer and a 1×1 partial convolutional layer;

[0013] Step 1.2: Add an attention SSE module and a fully connected layer after the last average pooling layer in the dual-channel SNN to obtain the hyperspectral image classification model SNN-ATC.

[0014] Further, the neuron type of the dual-channel SNN is an improved LIF neuron, and the membrane potential formula of the improved LIF neuron is as follows:

[0015]

[0016] Where, represents the membrane potential at the current time step, represents the membrane potential that did not exceed the threshold in the previous time step t-1, τ′ m represents the membrane potential time constant, which is used to describe the decay rate of the neuron membrane potential, w ij represents the synaptic weight between neuron i and neuron j, represents the flag indicating whether neuron j fires a pulse at time step t.

[0017] Furthermore, the attention SSE module obtains channel attention weights through a global average pooling layer, a channel mixing layer, a fully connected Relu activation layer, and a fully connected Sigmoid activation layer, and multiplies the channel attention weights by the input feature map of the attention SSE module for information integration.

[0018] Furthermore, step 2 specifically includes:

[0019] Step 2.1: Make a small perturbation to the synaptic weights of the spiking neural network to obtain the sensitivity of the synapses, as shown in formula (2):

[0020] ΔAcc(s ij ) = f(w ij + Δx) - f(w ij ) (2)

[0021] where ΔAcc(s ij ) represents the sensitivity of synapse s ij , f(·) represents the evaluation function of network performance, and Δx represents the small perturbation added to the synaptic weight w ij ;

[0022] Step 2.2: Calculate the energy consumption based on the firing frequency of the synapses. The higher the firing frequency of the synapses, the greater the energy consumption, and the calculation is performed through formula (3):

[0023] E(s ij ) = λf ij (3)

[0024] where E(s ij ) represents the network energy consumption of synapse s ij , f ij represents the firing frequency of synapse s ij , and λ is a constant representing the energy consumption coefficient per unit firing frequency;

[0025] Step 2.3: Analyze based on the sensitivity of the synapses and the impact on the network energy consumption of the hyperspectral image classification model SNN-ATC to obtain the importance score of the synapses; the calculation is performed through formula (4):

[0026] S(s ij ) = uΔAcc(s ij ) - (1 - u)E(s ij ) (4)

[0027] where S(s ij ) represents the importance score of synapse s ij , and u represents the balance coefficient;

[0028] Step 2.4: Compare the importance score S of the synapse with the set threshold θ to formulate a pruning decision rule: if S < θ, then prune the synapse; if S ≥ θ, then retain the synapse. As shown in formula (5), the hyperspectral classification model SNN - BMC is finally obtained;

[0029]

[0030] where θ represents the set pruning threshold.

[0031] Further, the specific steps of step 3 include:

[0032] Step 3.1: Collect hyperspectral images and annotate hard labels to construct a hyperspectral dataset, where the hard label is the classification label of the hyperspectral image;

[0033] Step 3.2: Input the hyperspectral dataset into the hyperspectral image classification model SNN - ATC, and the generated class probabilities are used as the soft labels of the hyperspectral dataset;

[0034] Step 3.3: The hyperspectral image classification model SNN - BMC is trained under the guidance of the soft labels generated by the hyperspectral image classification model SNN - ATC and the hard labels of the hyperspectral dataset. The training process is as shown in formula (6):

[0035]

[0036] where L all represents the comprehensive loss function of the training process, L ce represents the cross - entropy loss learned by the hyperspectral image classification model SNN - BMC from the hard labels, L kd represents the KL - divergence loss between the hyperspectral image classification model SNN - ATC and the hyperspectral image classification model SNN - BMC, q s and q s represent controlling the smoothness of the output with temperature T, y true represents the true label, and α represents the weight controlling L ce and L kd .

[0037] Further, the student network dynamically adjusts the time step according to the loss value, specifically including:

[0038] When the loss value exceeds the threshold, increase the time step; when the loss value is low, decrease the time step. The time step adjustment formula is as shown in formula (7):

[0039]

[0040] where T (t+1)represents the time step of the (t + 1)-th iteration, L t , L min and L max respectively represent the current loss value, the minimum loss value, and the maximum loss value. α and β represent the amplitude of increase and decrease for controlling the time step.

[0041] The beneficial effects of the present invention are as follows:

[0042] The present invention provides a low-latency SNN hyperspectral image classification method for a brain-like mental development mechanism. By simulating the brain neuroplasticity and mental development mechanism to streamline the network, a lightweight network is constructed. At the same time, knowledge distillation technology is used to dynamically adjust the time step of the student network, significantly reducing its time step while maintaining the high performance of the student network. This method not only meets the accuracy requirements of hyperspectral image processing but also overcomes the challenges of fast response and resource efficiency, providing a better solution for applications with strong real-time requirements. Description of the Drawings

[0043] Figure 1 is a schematic flowchart of a low-latency SNN hyperspectral image classification method for a brain-like mental development mechanism provided by an embodiment of the present invention;

[0044] Figure 2 is a schematic structural diagram of the classification model SNN-ATC provided by an embodiment of the present invention;

[0045] Figure 3 is a schematic diagram of the knowledge distillation method provided by an embodiment of the present invention;

[0046] Figure 4 is a schematic diagram of hyperspectral soft labels and hard labels provided by an embodiment of the present invention. Detailed Embodiments

[0047] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0048] Brain neuroplasticity: Brain neuroplasticity refers to the ability of the brain to change its structure and function according to experience, learning, and environmental stimuli.

[0049] Mental development mechanism: The mental development mechanism involves how the brain forms and changes cognitive functions through learning and experience.

[0050] Such as Figure 1As shown, a low-latency SNN hyperspectral image classification method for a brain-like mental development mechanism provided by an embodiment of the present invention includes:

[0051] Step 1: Construct a hyperspectral image classification model SNN-ATC based on the attention mechanism and the dual-channel SNN;

[0052] Step 2: Prune the hyperspectral image classification model SNN-ATC by simulating the brain neuroplasticity and mental development mechanism to obtain the hyperspectral image classification model SNN-BMC;

[0053] Step 3: Train the hyperspectral image classification model SNN-BMC using the knowledge distillation method;

[0054] Step 4: Implement hyperspectral image classification based on the trained hyperspectral image classification model SNN-BMC.

[0055] The hyperspectral image classification method provided by the present invention aims to optimize the network structure and shorten the time step, and uses the low-latency SNN-BMC of the brain-like mental development mechanism to implement hyperspectral image classification. Through the pruning strategy and low-latency optimization, SNN-BMC effectively improves the model inference speed and computational efficiency, while maintaining the accuracy of the classification performance.

[0056] Specifically, Step 1 specifically includes:

[0057] Step 1.1: Construct a dual-channel SNN, which includes a convolutional layer, a dual-channel module, a channel mixing layer, a convolutional layer, an average pooling layer, a convolutional layer, and an average pooling layer connected in sequence; the first channel module in the dual-channel module includes two 3×3 convolutional layers, and the second channel module includes a 3×3 depthwise separable convolutional layer and a 1×1 partial convolutional layer;

[0058] Step 1.2: Add an attention SSE (ShuffleSqueeze-and-Excitation) module and a fully connected layer after the last average pooling layer in the dual-channel SNN, as Figure 2 shown, to obtain the hyperspectral image classification model SNN-ATC. Among them, the attention SSE module obtains the channel attention weight through a global average pooling layer, a channel mixing layer, a fully connected Relu activation layer, and a fully connected Sigmoid activation layer, and multiplies the channel attention weight by the input feature map of the attention SSE module for information integration.

[0059] Furthermore, the neuron type of the dual-channel SNN is an improved LIF neuron, which can effectively accumulate more spike signals within a shorter time step, improving the network response speed and processing ability. The membrane potential formula of the improved LIF neuron is as follows:

[0060]

[0061] Among them, represents the membrane potential at the current time step, represents the membrane potential that did not exceed the threshold in the previous time step t-1, τ′ m represents the membrane potential time constant, which is used to describe the decay rate of the neuron membrane potential, w ij represents the synaptic weight between neuron i and neuron j, represents the flag indicating whether neuron j fires a pulse at time step t. During the forward propagation process, the membrane potential is jointly composed of the membrane potential that did not exceed the threshold in the previous time step t-1 and the output pulse of the presynaptic neuron in the current time step t. At the same time, these two parts achieve leakage of the membrane potential according to a certain proportion.

[0062] Specifically, step 2 specifically includes:

[0063] Step 2.1: Make a small perturbation to the synapses of the spiking neural network to obtain the sensitivity of the synapses, as shown in formula (2):

[0064] ΔAcc(s ij ) = f(w ij +Δx)-f(w ij ) (2)

[0065] Among them, ΔAcc(s ij ) represents the sensitivity of synapse s ij , f(·) represents the evaluation function of the network performance, Δx represents the small perturbation added to the synaptic weight w ij ; usually a very small value (such as 0.01); if the network performance drops significantly, and the higher the activity level, it indicates that the synapse is more important for the task, otherwise it may be unimportant.

[0066] Step 2.2: Calculate the energy consumption based on the activity frequency of the synapses. The higher the activity frequency of the synapses, the greater the energy consumption, and the calculation is carried out through formula (3):

[0067] E(s ij ) = λf ij (3)

[0068] Among them, E(s ij ) represents the network energy consumption of synapse s ij , f ij represents the activity frequency of synapse s ij ; λ is a constant, representing the energy consumption coefficient per unit activity frequency; among them, the activity frequency can be estimated through the firing frequency of neurons in the two-channel spiking neural network SSN.

[0069] Step 2.3: Analyze based on the sensitivity of the synapses and the impact on the network energy consumption of the hyperspectral image classification model SNN-ATC to obtain the importance score of the synapses; calculate through formula (4):

[0070] S(s ij ) = uΔAcc(s ij ) - (1 - u)E(s ij ) (4)

[0071] Among them, S(s ij ) represents the importance score of synapse s ij , and u represents the balance coefficient;

[0072] Step 2.4: Compare the importance score S of the synapses with the set threshold θ to formulate a pruning decision rule: if S < θ, then prune the synapse, if S ≥ θ, then retain the synapse, as shown in formula (5), and finally obtain the hyperspectral classification model SNN-BMC;

[0073]

[0074] Among them, θ represents the set pruning threshold.

[0075] Specifically, as Figure 3 shown, Step 3 specifically includes:

[0076] Step 3.1: Collect hyperspectral images and label hard labels to construct a hyperspectral dataset, where the hard labels are hyperspectral image classification labels;

[0077] Step 3.2: Input the hyperspectral dataset into the hyperspectral image classification model SNN-ATC, and generate the class probabilities as the soft labels of the hyperspectral dataset;

[0078] Step 3.3: As Figure 4 shown, the hyperspectral image classification model SNN-BMC is trained under the guidance of the soft labels generated by the hyperspectral image classification model SNN-ATC and the hard labels of the hyperspectral dataset, and the supervised learning process is as shown in formula (6):

[0079]

[0080] Among them, L all represents the comprehensive loss function of the training process, L ce represents the cross-entropy loss learned by the hyperspectral image classification model SNN-BMC in the hard labels, L kd represents the KL divergence loss between the hyperspectral image classification model SNN-ATC and the hyperspectral image classification model SNN-BMC, q s and qs represents the smoothness of the output controlled by temperature T, y true represents the true label, and α represents the weight for controlling L ce and L kd weights.

[0081] Specifically, the knowledge distillation method is used to train the classification model SNN - BMC. The unpruned network (SNN - ATC) is used as the teacher network, while the pruned network (SNN - BMC) is used as the student network. Through knowledge distillation, the student network learns under the guidance of the soft labels generated by the teacher network and the hard labels of the hyperspectral dataset, thereby optimizing the training and achieving efficient and accurate knowledge transfer.

[0082] Furthermore, the student network dynamically adjusts the time step according to the loss value, specifically including:

[0083] When the loss value exceeds the threshold, increase the time step; when the loss value is low, decrease the time step; the time step adjustment formula is shown in formula (7):

[0084]

[0085] where, T (t+1) represents the time step of the (t + 1)-th iteration, L t , L min and L max represent the current loss value, the minimum loss value, and the maximum loss value respectively, and α and β represent the increase and decrease amplitudes for controlling the time step.

[0086] Specifically, the teacher network sets a relatively high time step, which is beneficial for the model to capture complex time - dependent relationships, thereby obtaining higher accuracy; the student network dynamically adjusts the time step according to the loss value. When the loss value is high, increase the time step; when the loss value is low, decrease the time step. Through this dynamic adjustment, the training process can gradually reduce the time step of the student network, thereby achieving a better training effect.

[0087] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A low-latency SNN hyperspectral image classification method based on brain-like mental development mechanism, characterized in that: include: Step 1: Construct a hyperspectral image classification model SNN-ATC based on the attention mechanism and dual-channel SNN; Step 2: Pruning the hyperspectral image classification model SNN-ATC by simulating brain neural plasticity and mental development mechanism to obtain the hyperspectral image classification model SNN-BMC; Step 3: Using the knowledge distillation method to train the hyperspectral image classification model SNN-BMC; Step 4: Implement hyperspectral image classification based on the trained hyperspectral image classification model SNN-BMC.

2. According to the low-latency SNN hyperspectral image classification method of the brain-like mental development mechanism of claim 1, it is characterized in that: The step 1 specifically includes: Step 1.1: construct a dual-channel SNN, wherein the dual-channel SNN includes a convolutional layer, a dual-channel module, a channel mixing layer, a convolutional layer, an average pooling layer, a convolutional layer, and an average pooling layer connected in sequence; the first channel module in the dual-channel module includes two 3×3 convolutional layers, and the second channel module includes a 3×3 depth-separable convolutional layer and a 1×1 partial convolutional layer; Step 1.2: Add an attention SSE module and a fully connected layer after the last average pooling layer in the dual-channel SNN to obtain the hyperspectral image classification model SNN-ATC.

3. The low-latency SNN hyperspectral image classification method of the brain-like mental development mechanism according to claim 2 is characterized in that: The neuron type of the dual-channel SNN is an improved LIF neuron, wherein the improved LIF neuron membrane potential formula is as follows: in, represents the membrane potential at the current time step, represents the membrane potential that did not exceed the threshold in the previous time step t-1, τ′ m represents the membrane potential time constant, which is used to describe the decay rate of the neuronal membrane potential. ij represents the synaptic weight between neuron i and neuron j, A flag indicating whether neuron j fires a spike at time step t.

4. The low-latency SNN hyperspectral image classification method of the brain-like mental development mechanism according to claim 2 is characterized in that: The attention SSE module obtains the channel attention weight through the global average pooling layer, the channel mixing layer, the fully connected Relu activation layer and the fully connected Sigmoid activation layer, and multiplies the channel attention weight with the input feature map of the attention SSE module to integrate information.

5. The low-latency SNN hyperspectral image classification method of the brain-like mental development mechanism according to claim 1 is characterized in that: The step 2 specifically includes: Step 2.1: Perform a small perturbation on the synaptic weights of the spiking neural network to obtain the synaptic sensitivity, as shown in formula (2): ΔAcc(s ij )=f(w ij +Δx)-f(w ij ) (2) Among them, ΔAcc(s ij ) represents synaptic s ij , f(·) represents the evaluation function of network performance, and Δx represents the sensitivity to synaptic weight w ij Increased small disturbances; Step 2.2: Calculate the energy consumption based on the activity frequency of the synapse. The higher the activity frequency of the synapse, the greater the energy consumption. Calculate using formula (3): E(s ij )=λf ij (3) Among them, E(s ij ) represents synaptic s ij The network energy consumption, f ij Synapses ij activity frequency, λ is a constant, which represents the energy consumption coefficient per unit activity frequency; Step 2.3: Based on the sensitivity of the synapse and the impact on the network energy consumption of the hyperspectral image classification model SNN-ATC, the importance score of the synapse is obtained; calculated by formula (4): S(s ij )=uΔAcc(s ij )-(1-u)E(s ij ) (4) Among them, S(s ij ) represents synaptic s ij The importance score of , u represents the balance coefficient; Step 2.4: According to the comparison between the importance score S of the synapse and the set threshold θ, a pruning decision rule is formulated: if S<θ, the synapse is pruned; if S≥θ, the synapse is retained, as shown in formula (5), and finally the hyperspectral classification model SNN-BMC is obtained; Among them, θ represents the set pruning threshold.

6. The low-latency SNN hyperspectral image classification method of the brain-like mental development mechanism according to claim 1 is characterized in that: The step 3 specifically includes: Step 3.1: Collect hyperspectral images and annotate hard labels to construct a hyperspectral dataset, wherein the hard labels are classification labels of the hyperspectral images; Step 3.2: Input the hyperspectral dataset into the hyperspectral image classification model SNN-ATC, and generate the category probability as the soft label of the hyperspectral dataset; Step 3.3: The hyperspectral image classification model SNN-BMC is trained under the guidance of the soft labels generated by the hyperspectral image classification model SNN-ATC and the hard labels of the hyperspectral dataset. The training process is shown in formula (6): Among them, L all Represents the comprehensive loss function of the training process, L ce represents the cross entropy loss of the hyperspectral image classification model SNN-BMC learned in hard labels, L kd represents the KL divergence loss of the hyperspectral image classification model SNN-ATC and the hyperspectral image classification model SNN-BMC, q s and q s Indicates the smoothness of the output controlled by temperature T, y true represents the true label, α represents the control L ce and L kd The weight of .

7. The low-latency SNN hyperspectral image classification method of brain-like mental development mechanism according to claim 6 is characterized in that: The student network dynamically adjusts the time step by the loss value, specifically including: When the loss value exceeds the threshold, the time step is increased; when the loss value reaches the bottom, the time step is reduced; the time step adjustment formula is shown in formula (7): Among them, T (t+1) represents the time step of the t+1th iteration, L t , L min and L max They represent the current loss value, minimum loss value, and maximum loss value respectively, and α and β represent the increase or decrease of the control time step.

Images