Image classification method and device, equipment, medium and product

Abstract

The invention discloses an image classification method and device, equipment, a medium and a product, and relates to the field of image classification, and the method comprises the steps: obtaining to-be-classified image data; according to the to-be-classified image data, performing classification by using a classification model to obtain a classification category; the classification model is a trained pulse neural network; the spiking neural network comprises a plurality of neural layers; each neural layer comprises a plurality of neurons; each neuron comprises an emission threshold value and an input resistance; the training process of the classification model specifically comprises the following steps: taking sample image data as the input of a spiking neural network, taking a sample classification category as the output of the spiking neural network, and determining a total loss function of the spiking neural network according to cross entropy loss and adaptive sparse loss; and optimizing the synaptic weight, the emission threshold and the input resistance of the spiking neural network by using a back propagation algorithm to obtain a classification model. The method can improve the classification precision.

Description

Technical Field

[0001] The present application relates to the field of image classification, and in particular to an image classification method, apparatus, device, medium and product. Background Art

[0002] Spiking Neural Networks (SNNs), as third-generation neural networks, have recently attracted widespread attention in the fields of artificial intelligence and neuromorphic computing due to their biological interpretability and low energy consumption. SNNs mimic the spiking mechanism of biological neurons, using discrete binary pulse signals (spikes) to transmit information. Compared to traditional artificial neural networks (ANNs), SNNs offer potential advantages in temporal information processing and energy efficiency. However, current SNNs used in image processing still suffer from low accuracy. Summary of the Invention

[0003] The purpose of this application is to provide an image classification method, device, equipment, medium and product that can improve classification accuracy.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides an image classification method, comprising:

[0006] Obtain image data to be classified;

[0007] Classify the image data to be classified using a classification model to obtain a classification category; the classification model is a trained spiking neural network; the spiking neural network includes multiple neural layers; each neural layer includes multiple neurons; each neuron includes an emission threshold and an input resistance;

[0008] The training process of the classification model specifically includes:

[0009] The sample image data is used as the input of the spiking neural network, the sample classification category is used as the output of the spiking neural network, the total loss function of the spiking neural network is determined by the cross entropy loss and the adaptive sparsity loss, and the synaptic weights, emission threshold and input resistance of the spiking neural network are optimized using the back propagation algorithm to obtain a classification model.

[0010] In one embodiment, the classification model is used to classify the image data to obtain the classification category, specifically including:

[0011] determining a membrane potential according to the graphic data to be classified, a synaptic weight, and an input resistance;

[0012] The membrane potential is compared to the firing threshold to determine the classification category.

[0013] In one embodiment, the triangular proxy gradients are used to update synaptic weights, firing thresholds, and input resistances in the back-propagation algorithm.

[0014] In one embodiment, the expression of the adaptive sparse loss is:

[0015]

[0016] Among them, L S is the adaptive sparse loss, ρ is the target activation rate, l is the number of neural layers, and L is the total loss function. is the average activation rate of neurons in layer l, α l is the dynamic coefficient.

[0017] In one embodiment, the expression of the dynamic coefficient is:

[0018]

[0019] Among them, ρ threshold is the threshold hyperparameter.

[0020] In one embodiment, the expression of the membrane potential is:

[0021]

[0022] in, is the membrane potential, τ is the decay constant, is the pulse output of the i-th neuron in the n-th layer, R n is the input resistance, is the synaptic input current of the i-th neuron in the n-th layer, and t is the time step.

[0023] In a second aspect, the present application provides an image classification device, comprising:

[0024] An acquisition module, used to acquire image data to be classified;

[0025] A classification module, configured to classify the image data to be classified using a classification model to obtain a classification category; the classification model is a trained spiking neural network; the spiking neural network comprises a plurality of neural layers; each neural layer comprises a plurality of neurons; each neuron comprises an emission threshold and an input resistance;

[0026] The training process of the classification model specifically includes:

[0027] The sample image data is used as the input of the spiking neural network, the sample classification category is used as the output of the spiking neural network, the total loss function of the spiking neural network is determined by the cross entropy loss and the adaptive sparsity loss, and the synaptic weights, emission threshold and input resistance of the spiking neural network are optimized using the back propagation algorithm to obtain a classification model.

[0028] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image classification method.

[0029] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the image classification method when executed by a processor.

[0030] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which implements the image classification method when executed by a processor.

[0031] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0032] The present application provides an image classification method, apparatus, device, medium and product, which classifies the image data to be classified using a classification model to obtain a classification category; the classification model is a trained pulse neural network; the pulse neural network includes multiple neural layers; each neural layer includes multiple neurons; each neuron includes an emission threshold and an input resistance; the training process of the classification model specifically includes: using sample image data as the input of the pulse neural network, using the sample classification category as the output of the pulse neural network, and optimizing the synaptic weights, emission thresholds and input resistance of the pulse neural network based on the total loss function of the pulse neural network determined by the cross entropy loss and the adaptive sparse loss, to obtain a classification model. The emission threshold and input resistance are set in the neurons of the pulse neural network, and combined with the adaptive sparse loss, so as to achieve high precision, sparsity and low energy consumption of the neural network, so that the classification model obtained also has the advantages of high precision, sparsity and low energy consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1This is a diagram of an application environment of an image classification method in one embodiment of the present application;

[0035] Figure 2 A flowchart of an image classification method provided in one embodiment of the present application;

[0036] Figure 3 A schematic diagram of functional modules of an image classification device provided in another embodiment of the present application;

[0037] Figure 4 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0039] In the prior art, commonly used neuron models include the Leaky-Integrate-and-Fire (LIF) model and its variants. These models generate pulse signals by comparing the integral of the membrane potential with a threshold.

[0040] In terms of training methods, traditional SNNs mainly adopt the following two strategies:

[0041] 1. ANN-to-SNN conversion-based approach: First, the model is trained on an ANN, then the weights are mapped to an SNN, using spikes to approximate continuous activation values. This approach has achieved some success in static tasks (such as image classification), but the conversion process introduces significant accuracy loss, and performs poorly in time series tasks (such as DVS data).

[0042] 2. Direct training: This method uses a backpropagation algorithm (such as Backpropagation Through Time, BPTT) combined with a surrogate gradient to directly optimize SNN parameters. In recent years, direct training methods based on the LIF model have achieved high accuracy (e.g., 95.03%) on datasets such as CIFAR-10, significantly outperforming conversion methods.

[0043] However, despite the progress of SNNs in theory and application, existing solutions still have the following technical problems:

[0044] It's difficult to balance accuracy and sparsity: Traditional LIF models use fixed firing thresholds and input resistances, which limits the neurons' ability to respond to input. To improve accuracy, the network often needs to increase its firing rate, which reduces sparsity and, in turn, increases computational complexity and energy consumption. For example, the average activation rate of existing S-SNN models is typically between 0.3 and 0.5, far exceeding the ideal sparsity level.

[0045] Insufficient dynamic adaptability: The parameters of existing models (such as emission thresholds) are usually fixed during training and inference and cannot be dynamically adjusted according to the characteristics of the input data. This limits the robustness and generalization ability of SNNs when processing diverse tasks (such as static images and dynamic neuromorphic data).

[0046] High energy consumption: Due to the lack of explicit constraints on pulse output, traditional SNNs on neuromorphic hardware consume a lot of energy, making it difficult to meet the needs of low-power scenarios such as edge computing. For example, the energy consumption of traditional LIF-SNNs is often more than 50% higher than theoretical expectations.

[0047] Against the above background, the research and development of this application originated from an in-depth analysis of the performance bottlenecks of SNNs and an exploration inspired by biological neural systems. Biological neurons are highly heterogeneous and adaptable, and their firing thresholds and synaptic input response capabilities can change dynamically according to external stimuli. However, the fixed parameter design of the traditional LIF model cannot fully simulate this characteristic, resulting in limited performance of SNNs in complex tasks. To this end, the R&D team proposed an improvement idea: by introducing a VTR-LIF neuron model with variable firing threshold (Variable Firing Threshold) and variable input resistance (Variable Input Resistance), and combining it with a direct training method, the expressive power and efficiency of SNNs are enhanced.

[0048] While existing direct training methods can optimize synaptic weights, they are inadequate for adjusting intrinsic neuronal parameters (such as thresholds). Furthermore, sparsity, a key characteristic of SNNs' low energy consumption, lacks effective control methods. The VTR-LIF model-based spiking neural network training method and output regulation method proposed in this application address the shortcomings of existing technologies and solve the following problems:

[0049] The balance between accuracy and sparsity: Through the variable parameters of the VTR-LIF model and the sparse constraints of the ASL function, this application controls the average activation rate to around 0.1 while maintaining high accuracy (such as 96.25% on CIFAR-10), significantly improving sparsity.

[0050] The problem of insufficient dynamic adaptability: By utilizing the dynamic calculation of variable emission threshold and input resistance and the joint optimization mechanism, this application enables the network to adaptively adjust parameters, thereby enhancing its robustness to diverse tasks (such as DVS-CIFAR10).

[0051] High energy consumption problem: Through sparsity optimization and parameter adjustment, this application reduces the amount of pulse activity and reduces energy consumption by about , which is more suitable for neuromorphic hardware and edge computing applications.

[0052] In summary, based on the existing technology, this application overcomes the limitations of traditional SNNs in accuracy, efficiency and adaptability through an innovative VTR-LIF model and training strategy, and provides a new solution for the further development of spiking neural networks.

[0053] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0054] The image classification method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, or integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the image data to be classified to the server 104. After the server 104 receives the image data to be classified, the server 104 uses the classification model to classify the image data to be classified to obtain a classification category. The server 104 can feed back the obtained classification category to the terminal 102. In addition, in some embodiments, the image classification method can also be implemented separately by the server 104 or the terminal 102. For example, the terminal 102 can directly perform image classification on the image data to be classified, or the server 104 can obtain the image data to be classified from the data storage system and perform image classification on the image data to be classified.

[0055] Terminal 102 may include, but is not limited to, various desktop computers, laptops, smartphones, tablet computers, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Server 104 may be implemented as a standalone server or a server cluster consisting of multiple servers, or may be a cloud server.

[0056] In an exemplary embodiment, Figure 2As shown, an image classification method is provided. The method is executed by a computer device. Specifically, it can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate the process, including the following steps 201 to 202.

[0057] Step 201: Obtain image data to be classified.

[0058] Step 202: Classify the image data to be classified using a classification model to obtain a classification category; the classification model is a trained spiking neural network; the spiking neural network includes multiple neural layers; each neural layer includes multiple neurons; each neuron includes an emission threshold and an input resistance.

[0059] The training process of the classification model specifically includes:

[0060] The spiking neural network uses sample image data as input and the sample classification category as output. Based on the total loss function of the spiking neural network determined by cross-entropy loss and adaptive sparsity loss, the backpropagation algorithm is used to optimize the synaptic weights, firing threshold, and input resistance of the spiking neural network to obtain a classification model. In practical applications, the firing threshold and input resistance are both variable parameters.

[0061] By implementing the above steps 201 to 202, setting the emission threshold and input resistance in the neurons of the pulse neural network, and combining with the adaptive sparse loss, high precision, sparsity and low energy consumption of the neural network can be achieved, and the resulting classification model also has the advantages of high precision, sparsity and low energy consumption.

[0062] In an exemplary embodiment, the classification model is used to perform classification based on the image data to be classified to obtain a classification category, specifically including: determining the membrane potential based on the graphic data to be classified, the synaptic weight and the input resistance; and comparing the membrane potential with the emission threshold to determine the classification category.

[0063] In one exemplary embodiment, the triangular proxy gradients are used to update synaptic weights, firing thresholds, and input resistances in the back-propagation algorithm.

[0064] In an exemplary embodiment, the adaptive sparse loss is expressed as:

[0065]

[0066] Among them, L S is the adaptive sparse loss, ρ is the target activation rate, l is the number of neural layers, and L is the total loss function. is the average activation rate of neurons in layer l, α l is the dynamic coefficient.

[0067] In an exemplary embodiment, the expression of the dynamic coefficient is:

[0068]

[0069] Among them, ρ threshold is the threshold hyperparameter.

[0070] In an exemplary embodiment, the expression of the membrane potential is:

[0071]

[0072] in, is the membrane potential, τ is the decay constant, is the pulse output of the i-th neuron in the n-th layer, R n is the input resistance, is the synaptic input current of the i-th neuron in the n-th layer, and t is the time step, which determines the time step of the spiking neuron.

[0073] In an exemplary embodiment, the present application also provides a specific process of a pulse neural network, which is as follows.

[0074] This method directly trains spiking neural networks (SNNs) using the VTR-LIF neuron model, jointly optimizing synaptic weights, variable firing thresholds, and input resistances via the Backpropagation Through Time (BPTT) algorithm. This method dynamically adjusts neuron parameters during training, improving the network's temporal memory and sparsity. It has applications in image classification (e.g., CIFAR-10, CIFAR-100, Tiny-ImageNet) and neuromorphic data processing (e.g., DVS-CIFAR10). By using an adaptive sparse loss (ASL) function to adjust the output distribution of VTR-LIF neurons and dynamically constraining neuron activation rates using the Kullback-Leibler Divergence (KL-D) function, the method ensures both sparsity and energy efficiency in SNNs. This method, combined with the cross-entropy loss, balances task performance and network energy consumption through hyperparameters. This method can be applied to optimize the inference output of spiking neural networks, reducing energy consumption while maintaining high accuracy.

[0075] The learnable parameters of the VTR-LIF model (emission threshold and input resistance) and their synergy with the ASL function are the core innovations, which can significantly improve the performance of SNNs (such as 96.25% accuracy on CIFAR-10, 81.03% on CIFAR-100, etc.).

[0076] A spiking neural network training method and output adjustment method based on the VTR-LIF model is executed by a computer device and is used to train spiking neural networks (SNNs). By introducing a VTR-LIF neuron model with a variable firing threshold (Variable Firing Threshold) and a variable input resistance (Variable Input Resistance) (hereinafter referred to as VTR-LIF neuron), and combining it with an adaptive sparse loss function (Adaptive Sparse Loss, ASL), the method achieves high precision, sparsity, and low energy consumption of the neural network. The method includes the following steps:

[0077] Step 1: Initialize the parameters of the spiking neural network

[0078] A spiking neural network is constructed in a computer device. The network comprises multiple neural layers, each layer consisting of multiple VTR-LIF neurons. The spiking neural network framework adopts an existing framework, and the neurons therein are VTR-LIF neurons, that is, neurons with a variable firing threshold and variable input resistance.

[0079] Initialize the synaptic weights (w) of the network, the firing threshold (θ) of each layer of VTR-LIF neurons n ) and input resistance (Input Resistance, R n ), where n represents the nth neural layer. It provides initial parameters for subsequent training to ensure that the network can receive input data and generate initial pulse signals.

[0080] Step 2: Dynamically calculate membrane potential and generate pulse signals through VTR-LIF neurons

[0081] The computer device calculates the membrane potential (Membrane Potential, u) of each layer of VTR-LIF neurons at each time step (TimeStep, t) based on the input data (such as image data or neuromorphic data). t ), and generates a binary pulse signal (Spike Output, o t), by dynamically adjusting the input resistance and firing threshold, the heterogeneity and adaptability of neurons are enhanced, thus improving the network's ability to process time information. The specific calculation process is as follows:

[0082] 1. Calculate the synaptic input current of the i-th neuron in the n-th layer. ): The synaptic input current is the input data. Specifically, the first layer of neurons receives external input (input data), which is multi-time dimension data, not just the first time step. The data received by each layer of neurons is multi-time step, and the output data is also multi-time step data.

[0083]

[0084] in, is the pulse output of the jth neuron in the n-1th layer. is the synaptic weight between the jth neuron in the n-1th layer and the ith neuron in the nth layer. The initial value is set by a random distribution. I(n-1) is the synaptic input current in the n-1th layer.

[0085] 2. Update membrane potential

[0086]

[0087] Among them, τ is the decay constant (Decay Constant), which is a known hyperparameter and its initial value is usually set to 0.5; R n is the input resistance of the nth layer. n is the input resistance of the nth layer of neurons. The initial value is a hyperparameter and can be set to 0.75, 1, or 1.25, etc.

[0088] Generate pulse signal

[0089]

[0090] Among them, θ n is the firing threshold of the n-th layer of neurons. The initial value is a hyperparameter and can be set to 0.75, 1, or 1.25, etc. It can be automatically optimized during the training process.

[0091] Step 3: Jointly optimize network parameters and adjust pulse output distribution

[0092] The computer device uses the backpropagation through time (BPTT) algorithm to jointly optimize the synaptic weight w and the firing threshold θ. n and input resistance Rn And adjust the neuron pulse output distribution through the adaptive sparse loss function (ASL). The specific steps are as follows:

[0093] 1. Calculate the total loss function (L):

[0094] L=(1-λ)L CE +λL S

[0095] L CE is the cross-entropy loss (Cross-Entropy Loss): Among them, O is the pulse sequence of the output layer (predicted value), Y is the target label, and C is the number of categories, all of which are known quantities.

[0096] O can be understood as the output pulse sequence of the last layer of neurons. Can be viewed as a vector O n The component vector (nth layer) represents the pulse output of the jth neuron in the nth layer. The relationship between the two is the output of each layer of neurons (hidden layer) O n All the information will be passed to the next layer of neurons until the (output layer) neurons output the predicted value O. Image classification is to determine which category the input image belongs to, for example, whether it is a cat or a dog. The image size is (H*W*C: height*width*number of channels), where C is the number of classification categories of the image (how many categories there are in total).

[0097] The specific prediction method is: first convert the image from a tensor of H*W*C dimensions to an SNN pulse input signal (this step has been integrated into the spiking neural network. The input to the spiking neural network is the image feature tensor, which does not need to be converted into a pulse signal sequence before input). Then the spiking neural network generates an output pulse sequence O based on the input signal. O is the predicted value (the output pulse sequence of the neural network), with a dimension of t*C. The output pulse O with the largest probability value is selected. i As the predicted value (activation rate ρ can be used as a classification indicator, the one with the most outputs of 1 in t time steps and the most activation times) is selected for classification, and its index i (i∈C) corresponds to the classification category.

[0098] Y is the target label, that is, the true value, y i L is the Y vector, index i indicates whether the category of category i is true (one-hot encoding, only the true category is 1 and other categories are 0). The method to determine whether the prediction is correct is to compare the corresponding item indexes of Y and O to see if they are consistent (for example, the index corresponding to the predicted value of O is 4, and the index corresponding to the true value of Y is 5. If they are inconsistent, the prediction fails). CEis the cross entropy loss function, which is used to evaluate the performance of the model. It can be simply understood as the more successful the prediction, the better the L CE The smaller the value, the smaller the loss function. The purpose of updating the model's prominent weights through the back-propagation algorithm is to make the loss function's loss value as small as possible.

[0099] Among them, L S is the adaptive sparse loss (ASL):

[0100]

[0101] Where ρ is the target activation rate (a known hyperparameter, such as 0.1). is the dynamic coefficient, ρ threshold is the threshold for constraining the activation rate, which is a threshold hyperparameter (e.g., 0.2); λ is the balancing factor (a known hyperparameter, e.g., 0.5). is the average activation rate.

[0102] is the average firing rate of neurons in the lth layer, and the calculation formula is:

[0103]

[0104] Where T is the time step, because the spike neuron is iterated with time step t, it may output a spike at each time step. l is the number of layers, N l is the total number of neurons. This formula is used to calculate the average activation rate of neurons in a certain layer, that is, the probability that the neurons in this layer output 1 (neuron output 1 indicates activation).

[0105] Compute gradients and update parameters:

[0106] For w, θ n 、R n The gradients are:

[0107]

[0108] in, and Calculated by the chain rule.

[0109]

[0110]

[0111]

[0112] This application uses a triangular surrogate gradient to solve the problem of non-differentiable pulse signals:

[0113]

[0114] Here, γ is a hyperparameter (e.g. 1).

[0115] As shown in the above derivation formula, both contain However, this term is non-differentiable (non-differentiable), so a triangle proxy gradient is chosen to approximate the gradient.

[0116] 3. Update w, θ n 、R n To minimize the loss function L.

[0117] Parameter weight update method: This part involves the backpropagation algorithm. The weights w, R, θ (learnable parameter values, weights, meaning the same thing) are updated according to the calculated gradient as follows:

[0118]

[0119]

[0120]

[0121] Where η is the learning rate, a manually set hyperparameter. The weights are updated using the gradients calculated above.

[0122] When the loss value (error value) of the loss function is less than the expected loss value, it means that the synaptic weights of the current pulse neural network do not need to be updated, so there is no need to change the synaptic weights, and the initial synaptic weights can be used.

[0123] This application designs a new spiking neuron VTRLIF and a sparsity loss function ASL. This application introduces more neurons with learnable parameters into the spiking neural network model and uses specialized computing equipment for parameter training to obtain a new model with improved classification accuracy. At the same time, this application uses a sparsity loss function to constrain the number / distribution of neuron output pulses in the model during training (the more neuron pulses output, the higher the energy consumption), achieving high energy efficiency.

[0124] Set the target activation rate and use KL divergence as the loss function L s To measure the difference between the target activation rate and the actual activation rate, the sparsity constraint is implemented by minimizing the difference, and the distribution of neuron pulse output is adjusted.

[0125] The KL divergence (Kullback-Leibler Divergence), also known as relative entropy, is an asymmetric measure of the difference between two probability distributions. It is commonly used in information theory, machine learning, and statistics, and is particularly useful for comparing a model distribution with the true distribution.

[0126] Adaptive adjustment of neuron parameters is achieved through joint optimization, and the ASL function is used to constrain the pulse output of high activation rate layers to improve the sparsity and energy efficiency of the network.

[0127] Problems similar to the following can be solved

[0128] 1. Low-power image classification on edge devices

[0129] Scenario: Smart cameras monitor wildlife in the wild (such as identifying bears and deer). The devices are battery-powered and need to run for a long time.

[0130] Problem: Traditional CNNs consume high power (tens to hundreds of watts) when running on GPUs or CPUs, making them unsuitable for resource-constrained edge devices and resulting in short battery life.

[0131] SNN’s solution:

[0132] SNNs are used to encode image pixels into spike trains, extract features through spike convolution layers, and ultimately classify animal categories using spike counts. Deployed on neuromorphic chips such as Intel Loihi, the sparse computing nature of SNNs reduces power consumption to milliwatts (for example, Loihi consumes approximately 1mW when processing MNIST, while traditional GPUs may require 50W+). On small subsets similar to ImageNet, SNNs achieve accuracy exceeding 90% while extending device runtime by several times.

[0133] Problem solved: High energy consumption and resource limitations, suitable for scenarios without external power supply.

[0134] 2. Real-time dynamic visual processing (such as autonomous driving)

[0135] Scenario: Autonomous vehicles use event-based cameras to detect pedestrians or obstacles on the road.

[0136] Problem: Traditional CNNs have a large amount of computation and high latency (tens of milliseconds per frame) when processing frame data, making it difficult to cope with high-speed scenarios. They are also sensitive to static frames and make insufficient use of dynamic information.

[0137] SNN’s solution:

[0138] Method: Event cameras only record pixel changes and output a sparse pulse stream, which is naturally compatible with SNNs. SNNs receive the pulse stream and classify objects in real time (e.g., "pedestrian" corresponds to a high-frequency pulse) through temporal encoding and pulse convolution.

[0139] Results: SNNs achieve classification accuracy of 85%-90% on DVS (Dynamic Vision Sensor) datasets (such as N-Caltech101) with latency as low as microseconds, making them more suitable for high-speed decision-making compared to the millisecond latency of CNNs.

[0140] Problem solved: Insufficient processing of high latency and dynamic data, improving real-time and robustness.

[0141] 3. Sparse Data Classification of Biomedical Signals

[0142] Scenario: Portable electroencephalogram (EEG) devices detect epileptic seizures and need to analyze sparse EEG signals with low power consumption.

[0143] Problem: Traditional ANNs are inefficient in processing sparse signals, require intensive computing, and have difficulty meeting real-time and energy consumption requirements on portable devices.

[0144] SNN’s solution:

[0145] Methods: EEG signals are encoded into pulse trains (such as voltage peak-triggered pulses). SNN captures epileptic characteristics (such as abnormal high-frequency pulses) through temporal dynamics and outputs classification results (normal / epileptic).

[0146] Results: Studies have shown (e.g., a 2020 Nature Communications paper) that SNNs can achieve 95% accuracy in EEG classification tasks while consuming more than 10 times less power than traditional methods.

[0147] Problem solved: Efficient processing of sparse signals with low power consumption, suitable for wearable medical devices.

[0148] 4. Real-time gesture recognition in robot control

[0149] Scenario: A service robot uses a camera to recognize user gestures (such as "stop" and "go forward") and responds quickly.

[0150] Problem: Traditional CNNs require a large number of floating-point operations, and latency and power consumption limit the performance of robots on low-computing hardware.

[0151] SNN’s solution:

[0152] Method: A depth camera is used to capture gesture images, which are converted into pulse streams. SNN is used to classify gestures based on pulse frequency. STDP (Spike-Timing-Dependent Plasticity) is used for online learning.

[0153] Results: SNN achieves 90% accuracy on gesture datasets (such as the DVS Gesture Dataset), consumes only 1 / 50 of the power of traditional methods, and reduces response time to less than 10ms.

[0154] Problems solved: High latency and computational complexity, enhancing the robot's real-time interaction capabilities

[0155] Compared with the existing technology, the pulse neural network training method and output adjustment method based on the VTR-LIF model provided in this application have significant advantages in classification accuracy, computational efficiency, sparsity, energy consumption and robustness.

[0156] 1. Higher Classification Accuracy

[0157] This application achieves significantly better classification accuracy than traditional spiking neural networks (SNNs) and some existing improved models on multiple benchmark datasets. For example, on the CIFAR-10 dataset, the accuracy reaches 96.25%, on the CIFAR-100 dataset, it reaches 81.03%, on the Tiny-ImageNet dataset, it reaches 63.81%, and on the DVS-CIFAR10 dataset, it reaches 88.10%, all of which exceed traditional techniques (such as the fixed-parameter LIF-SNN, which has an accuracy of 95.03% on CIFAR-10).

[0158] Using a novel neuron model to train spiking neural networks improves classification accuracy. The VTRLIF model (spiking neuron model) introduces learnable neuron activation thresholds and learnable resistors, replacing the previously unlearnable ones. It also introduces the ASL loss function, which significantly improves accuracy.

[0159] Dynamic calculation of membrane potential and generation of spike signals by VTR-LIF neurons: by introducing a variable firing threshold θ n and variable input resistor R n , this application dynamically regulates membrane potential This enhances the heterogeneity of neurons and their ability to process temporal information. This dynamicity enables the network to more accurately capture the spatiotemporal characteristics of input data, thereby improving classification accuracy.

[0160] Jointly optimize network parameters and adjust pulse output distribution: use the back propagation through time (BPTT) algorithm to simultaneously optimize synaptic weights w and firing thresholds θ n and input resistance R n , which enables adaptive adjustment of neuron parameters during training. Compared with the traditional LIF model with fixed parameters, this application shows higher accuracy in complex tasks.

[0161] Explanation: Traditional SNNs typically use fixed emission thresholds and input resistances, which limit the dynamic response of neurons to inputs. However, this application enhances the network's expressiveness through learnable parameters, enabling better fitting of complex patterns, especially when processing high-dimensional data (such as CIFAR-100 and Tiny-ImageNet).

[0162] 2. Higher Computational Efficiency and Sparsity

[0163] This application significantly improves the sparsity of pulse output while maintaining high accuracy, and the average activation rate It can be controlled at a low level (e.g. around 0.1), reducing the spike activity by about 50%-70% compared to traditional SNNs (e.g., the average activation rate of S-SNN is about 0.3-0.5). This reduces the amount of computation, making it more suitable for efficient inference.

[0164] Jointly optimize network parameters and adjust spike output distribution: Dynamically constrain the spike output of high-activation layers through an adaptive sparse loss function. The dynamic coefficient further ensures that only layers exceeding the threshold are penalized, thus achieving a balance between accuracy and sparsity.

[0165] Explanation: Traditional SNNs lack explicit control over activation rates, leading to excessive spike activity and increased computational burden. This application utilizes the KL divergence of the ASL function to optimize the output distribution. This, combined with the dynamic adjustment capabilities of the VTR-LIF model, enables the network to maintain high performance even under sparse conditions.

[0166] 3. Lower Energy Consumption

[0167] Due to the improved pulse sparsity, the energy consumption of the spiking neural network in the inference phase of this application is significantly reduced. Compared with the traditional LIF model (which consumes about 1.5-2 times the energy of this application), this application reduces energy consumption by about 30%-50% under the same task, making it particularly suitable for resource-constrained edge computing scenarios.

[0168] Jointly optimize network parameters and adjust pulse output distribution: By optimizing the emission threshold θ nand input resistance R n , combined with ASL function to reduce pulse activity Reduces the energy consumption per pulse trigger.

[0169] Explanation of the reason: The energy consumption of pulse neural networks mainly comes from pulse generation and transmission. This application reduces the number of pulses through sparsity optimization and dynamically adjusts parameters to avoid redundant calculations. Compared with traditional fixed parameter models, it has significant energy consumption advantages.

[0170] 4. Stronger Robustness and Adaptability

[0171] This application shows excellent performance on different types of datasets (static images such as CIFAR-10 and dynamic neuromorphic data such as DVS-CIFAR10), and can still maintain high accuracy in noisy environments, with better robustness than traditional SNNs.

[0172] Dynamic calculation of membrane potential and generation of spike signals by VTR-LIF neurons: variable firing threshold θ n and input resistance R n It enables neurons to adaptively adjust according to the characteristics of the input data, avoiding the overfitting or underfitting problems caused by fixed parameters in traditional LIF models.

[0173] Reason explanation: Traditional SNNs have fixed parameters and cannot flexibly respond to changes in data distribution. However, the learnable parameters and dynamic adjustment mechanism of this application enable it to adapt to diverse task requirements.

[0174] The core advantages of this application over the existing technology are:

[0175] High accuracy: derived from the dynamic calculation of the VTR-LIF model in step 2 and the joint optimization in step 3.

[0176] High efficiency and low energy consumption: derived from the sparsity constraints and parameter optimization of the ASL function in step 3.

[0177] Strong adaptability: derived from the variable parameter design of the dynamic calculation mechanism in step 2.

[0178] These advantages are achieved through innovations in the method (such as variable parameters of the VTR-LIF model and dynamic adjustment of the ASL loss function), which gives this application significant practical value and competitive advantages in the field of pulse neural network training and application.

[0179] Training method based on traditional LIF model combined with sparse regularization

[0180] Solution Description: Based on the traditional Leaky-Integrate-and-Fire (LIF) model, synaptic weights are optimized through direct training methods (such as BPTT combined with proxy gradients). At the same time, L1 or L2 regularization terms are added to the loss function to constrain the distribution of neuron pulse outputs, thereby improving sparsity.

[0181] Feasibility analysis:

[0182] Advantages: L1 regularization can reduce pulse output to a certain extent, improve sparsity, reduce energy consumption, and is simple to implement.

[0183] limitation:

[0184] Limited accuracy: The LIF model with fixed parameters lacks dynamic adjustment capabilities and is difficult to achieve the high accuracy of this application (such as 96.25% on CIFAR-10). The accuracy usually remains around 95%.

[0185] Insufficient sparsity control: L1 regularization imposes global constraints on pulse distribution and cannot dynamically adjust specific high-activation rate layers like the ASL function in this application, making it difficult to balance sparsity and accuracy.

[0186] Weak adaptability: Fixed thresholds and resistors cannot adapt to diverse tasks (such as DVS-CIFAR10) and are not robust enough.

[0187] Conclusion: This solution can improve sparsity to a certain extent, but it cannot fully achieve the comprehensive goals of this application in terms of accuracy, adaptability, and energy efficiency.

[0188] 2. ANN-to-SNN conversion and pulse coding optimization method

[0189] Solution description: First train the model on a traditional artificial neural network (ANN) and then convert it to an SNN. By optimizing the pulse encoding method of the input data (such as rate encoding or time encoding) to reduce the pulse activity, the efficiency and sparsity can be improved.

[0190] Feasibility analysis:

[0191] Advantages: ANN training is mature, the high precision of the pre-trained model can be utilized after conversion, and coding optimization can reduce some pulse activities.

[0192] limitation:

[0193] Accuracy loss: The conversion process inevitably introduces errors, especially in time series tasks (such as DVS-CIFAR10, where the accuracy is usually less than 88%), which cannot match the direct training accuracy of this application.

[0194] Limited sparsity: Coding optimization only affects the input layer and cannot dynamically constrain the pulse distribution of the entire network like the ASL function in step 3 of this application.

[0195] Energy consumption and efficiency issues: The converted SNN still requires more time steps to approximate the ANN output, resulting in higher energy consumption than the sparse direct training method of this application.

[0196] Conclusion: This solution is suitable for static tasks, but it cannot replace this application in terms of dynamic tasks and energy consumption optimization.

[0197] 3. Training method based on the Dynamic Threshold LIF model

[0198] Solution Description: Dynamic Threshold is introduced into the traditional LIF model. The threshold is adaptively adjusted according to the membrane potential history or input intensity. The weights and thresholds are optimized through BPTT to improve network performance.

[0199] Feasibility analysis:

[0200] Advantages: Dynamic thresholding can enhance the adaptability of neurons, potentially improving accuracy and sparsity, and is close to the variable threshold design in step 2 of this application.

[0201] limitation:

[0202] Lack of input resistance adjustment: Only adjusting the threshold while ignoring the input resistance (such as Rn R^nRn in step 2 of this application) limits the dynamic response ability of the neuron to the input current, and the accuracy and robustness are lower than those of this application.

[0203] Insufficient sparsity control: Lacking explicit sparsity constraints like the ASL function (step 3), the average activation rate is difficult to maintain around 0.1.

[0204] Optimization complexity: Threshold adjustment rules (such as α\alphaα) need to be manually designed, which is difficult to optimize jointly with weights, and the training efficiency is lower than this application.

[0205] Conclusion: This solution has certain similarities with the present application in terms of dynamics, but lacks the synergistic effect of input resistance and sparse loss, and cannot fully realize the multi-objective optimization of the present application.

[0206] Based on the same inventive concept, embodiments of the present application also provide an image classification device for implementing the aforementioned image classification method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations in one or more of the following image classification device embodiments can be found in the above-described limitations on the image classification method and will not be further elaborated here.

[0207] like Figure 3 As shown, in an exemplary embodiment, an image classification device is provided, comprising:

[0208] The acquisition module is used to obtain image data to be classified.

[0209] A classification module is used to classify the image data to be classified using a classification model to obtain a classification category; the classification model is a trained pulse neural network; the pulse neural network includes multiple neural layers; each neural layer includes multiple neurons; each neuron includes an emission threshold and an input resistance.

[0210] The training process of the classification model specifically includes:

[0211] The sample image data is used as the input of the spiking neural network, the sample classification category is used as the output of the spiking neural network, the total loss function of the spiking neural network is determined by the cross entropy loss and the adaptive sparsity loss, and the synaptic weights, emission threshold and input resistance of the spiking neural network are optimized using the back propagation algorithm to obtain a classification model.

[0212] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store image classification data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image classification method is implemented.

[0213] Those skilled in the art will understand that Figure 4The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application and does not constitute a limitation on the computer device to which the solution of the present application is applied. A specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the above-mentioned method embodiments when executing the computer program.

[0214] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the above-mentioned method embodiments when executed by a processor.

[0215] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the above method embodiments are implemented.

[0216] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0217] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0218] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0219] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0220] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0221] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. An image classification method, characterized in that: The image classification method comprises: Obtain image data to be classified; Classify the image data to be classified using a classification model to obtain a classification category; the classification model is a trained spiking neural network; the spiking neural network includes multiple neural layers; each neural layer includes multiple neurons; each neuron includes an emission threshold and an input resistance; The training process of the classification model specifically includes: The sample image data is used as the input of the spiking neural network, the sample classification category is used as the output of the spiking neural network, the total loss function of the spiking neural network is determined by the cross entropy loss and the adaptive sparsity loss, and the synaptic weights, emission threshold and input resistance of the spiking neural network are optimized using the back propagation algorithm to obtain a classification model.

2. The image classification method according to claim 1, characterized in that Classification is performed using a classification model according to the image data to be classified to obtain classification categories, specifically including: determining a membrane potential according to the graphic data to be classified, a synaptic weight, and an input resistance; The membrane potential is compared to the firing threshold to determine the classification category.

3. The image classification method according to claim 1, wherein: In the back-propagation algorithm, the triangle proxy gradient is used to update the synaptic weights, firing thresholds, and input resistances.

4. The image classification method according to claim 1, wherein: The expression of the adaptive sparse loss is: Among them, L S is the adaptive sparse loss, ρ is the target activation rate, l is the number of neural layers, and L is the total loss function. is the average activation rate of neurons in layer l, α l is the dynamic coefficient.

5. The image classification method according to claim 4, characterized in that The expression of the dynamic coefficient is: Among them, ρ threshold is the threshold hyperparameter.

6. The image classification method according to claim 2, characterized in that The expression of the membrane potential is: in, is the membrane potential, τ is the decay constant, is the pulse output of the i-th neuron in the n-th layer, R n is the input resistance, is the synaptic input current of the i-th neuron in the n-th layer, and t is the time step.

7. An image classification device, characterized in that: The image classification device comprises: An acquisition module, used to acquire image data to be classified; A classification module, configured to classify the image data to be classified using a classification model to obtain a classification category; the classification model is a trained spiking neural network; the spiking neural network comprises a plurality of neural layers; each neural layer comprises a plurality of neurons; each neuron comprises an emission threshold and an input resistance; The training process of the classification model specifically includes: The sample image data is used as the input of the spiking neural network, the sample classification category is used as the output of the spiking neural network, the total loss function of the spiking neural network is determined by the cross entropy loss and the adaptive sparsity loss, and the synaptic weights, emission threshold and input resistance of the spiking neural network are optimized using the back propagation algorithm to obtain a classification model.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image classification method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image classification method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the image classification method according to any one of claims 1 to 6 is implemented.

Images