A steady-state unbiased sparse topology-based pulse neural network compression method

Abstract

The application discloses a kind of based on steady-state unbiased sparse topology's pulse neural network compression method, belong to pulse neural network compression field, this method is aimed at the defect that static N:M structured sparse easily causes gradient expectation deviation, neuron irreversible silence, adopts gradient perception dynamic mask strategy, in combination with shadow gradient accumulation mechanism generates N:M sparse topology mask in line with hardware constraint;Utilize multiplication type mask update to complete scale invariant competition, eliminate model initialization biasing.Meanwhile, introduce firing rate steady-state plasticity, scale synaptic weight globally in long time sequence dimension, stabilize neuron pulse firing level, avoid synaptic sparse to cause neuron silence failure.The application is optimized by alternately iterating mask and synaptic weight, realizes hardware adaptation structured sparse under the premise of guaranteeing network precision, effectively reduces the operation power consumption of pulse neural network, adapts to the low-power deployment needs of various general neural network accelerators.

Description

Technical Field

[0001] This invention belongs to the field of spiking neural network compression, and more specifically, relates to a spiking neural network compression method based on steady-state unbiased sparse topology. Background Technology

[0002] Spiking neural networks transmit information through discrete pulse events, and their inherent spatiotemporal sparsity provides great potential for low-power image processing computation.

[0003] However, practical hardware acceleration faces a structural mismatch between dense weight matrices and event-driven computation. N:M fine-grained structured sparsity requires retaining exactly N non-zero elements in a continuous set of M weights. This pattern can be directly mapped to the sparse tensor kernel of general-purpose neural network accelerators, resulting in significant speedup. However, directly applying static N:M structured sparsity to spiking neural networks induces two key pathologies. First, gradient expectation bias. The non-commutativity between the static mask and the surrogate gradient operator leads to a systematic bias in gradient estimation, causing some parameters to permanently lack effective learning signals. Second, irreversible neuronal silencing. The structural reduction in the number of synaptic afferents may make it difficult for the neuronal membrane potential to accumulate to the firing threshold. Once the neuron stops firing, the surrogate gradients of all its afferent weights become zero, and the static mask cannot restore this activity.

[0004] Existing spiking neural network pruning methods mostly employ static criteria based on amplitude or heuristic probability rules, failing to address the aforementioned dynamic adaptability issues. Therefore, there is an urgent need for a spiking neural network compression method that can simultaneously optimize the topology and maintain neuronal functional viability under hardware-friendly constraints. Summary of the Invention

[0005] To address the above-mentioned deficiencies or improvement needs of existing technologies, this invention provides a spiking neural network compression method based on steady-state unbiased sparse topology, which enables efficient deployment of N:M structured sparse spiking neural networks on general-purpose neural network accelerators.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a compression method based on a steady-state unbiased sparse topology spiking neural network is provided, wherein the spiking neural network is used for image processing, and the method includes: During the training process of a spiking neural network, the dynamic mask matrix and weight matrix of each network layer are updated alternately. After training, the latest weight matrix of each network layer is multiplied element by element with the latest dynamic mask matrix to obtain the corresponding sparse weight matrix. The update process of the dynamic mask matrix includes: Generate the current dynamic mask matrix based on the current weight matrix and gradient matrix; correct the current gradient matrix using the current shadow gradient buffer matrix; update the current shadow gradient buffer matrix using the current dynamic mask matrix; update the scoring matrix using the corrected gradient matrix, thereby updating the dynamic mask matrix. The update process of the weight matrix includes: Calculate the deviation between the actual average distribution rate and the preset target distribution rate, and update the current weight matrix based on the deviation.

[0007] According to a second aspect of the present invention, a hardware deployment method for a spiking neural network is provided, comprising: The compressed spiking neural network obtained by the spiking neural network compression method as described in the first aspect is deployed to a general neural network accelerator; wherein the compressed spiking neural network is characterized by the sparse weight matrix of each network layer.

[0008] According to a third aspect of the present invention, an electronic device is provided, comprising: a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in the first or second aspect.

[0009] According to a fourth aspect of the invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to perform the method as described in the first or second aspect.

[0010] According to a fifth aspect of the invention, a computer program product is provided, comprising a computer program or instructions that, when executed by a processor, implement the method as described in the first or second aspect.

[0011] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: 1. The method provided by this invention eliminates the gradient expectation bias in N:M sparse spiking neural networks by using gradient-aware dynamic masks and shadow gradient accumulation, thus ensuring the unbiasedness of sparse topology search.

[0012] 2. The method provided by this invention eliminates the initialization advantage bias through the multiplicative mask update rule, so that the topology evolution can reflect the global task-related features rather than the early weight distribution.

[0013] 3. The method provided by this invention uses the firing rate steady-state plasticity mechanism to provide a formal survival guarantee for sparse neurons, theoretically ensuring that the network still maintains its functional vitality under extreme sparsity ratios.

[0014] 4. The method provided by this invention is fully compatible with hardware-friendly N:M structured sparse patterns and can directly utilize the sparse tensor kernel of a general neural network accelerator to achieve a significant reduction in the number of floating-point operations and energy savings.

[0015] In summary, the method provided by this invention is geared towards modern high-performance chips. By performing a hybrid pruning-quantization reshaping of the spiking neural network, the performance loss of the reshaping spiking neural network model does not exceed 1%, enabling efficient deployment of N:M structured sparse spiking neural networks on general-purpose neural network accelerators. Attached Figure Description

[0016] Figure 1 This is one of the flowcharts for a spiking neural network compression method based on steady-state unbiased sparse topology provided in an embodiment of the present invention.

[0017] Figure 2 The second flowchart of the spiking neural network compression method based on steady-state unbiased sparse topology provided in the embodiments of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0019] This invention provides a compression method based on a steady-state unbiased sparse topology spiking neural network. The spiking neural network is used for image processing, including basic visual feature extraction, image recognition, target detection, and target tracking. Figures 1 to 2 As shown, the method includes: During the training process of a spiking neural network, the dynamic mask matrix and weight matrix of each network layer are updated alternately. After training, the latest weight matrix of each network layer is multiplied element by element with the latest dynamic mask matrix to obtain the corresponding sparse weight matrix. The update process of the dynamic mask matrix of each network layer includes: Generate the current dynamic mask matrix based on the current weight matrix and gradient matrix; correct the current gradient matrix using the current shadow gradient buffer matrix; update the current shadow gradient buffer matrix using the current dynamic mask matrix; update the scoring matrix using the corrected gradient matrix, thereby updating the dynamic mask matrix. The update process of the weight matrix of each network layer includes: Calculate the deviation between the actual average distribution rate and the preset target distribution rate, and update the current weight matrix based on the deviation.

[0020] Preferably, the steps of generating the current dynamic mask matrix based on the current weight matrix and gradient matrix; correcting the current gradient matrix using the current shadow gradient buffer matrix; updating the current shadow gradient buffer matrix using the current dynamic mask matrix; and updating the scoring matrix using the corrected gradient matrix to update the dynamic mask matrix include: (1) Let W be the current weight matrix and gradient matrix respectively. t G t Generate the current dynamic mask matrix Mask t Let H be the current shadow gradient buffer matrix. t Using H t Correction G t Obtain the corrected gradient matrix t Use Mask t Update H t The updated shadow gradient buffer matrix H is obtained. t+1 This is used to adjust G during the next update of the dynamic mask matrix (i.e., during the (t+1)th update). t+1 Perform correction; (2) Let Q be the current rating matrix. t ,use t Update Q t The updated rating matrix Q is obtained. t+1 Using Q t+1 Calculate the updated dynamic mask matrix Mask t+1 ; Among them, Q t+1 =(1-α)·Q t +α·(| t | / t ), where α is the momentum renewal rate. t =mean(| t|), mean(·) represents calculating the global mean, and t is the index of the update time step of the dynamic mask matrix.

[0021] Specifically, the update process of the dynamic mask matrix for each network layer is as follows: S1. For each network layer in the spiking neural network, a gradient-aware dynamic mask generation strategy is adopted, combined with the shadow gradient accumulation mechanism, to generate an initial N:M sparse topological mask for the weight matrix that satisfies the hardware acceleration constraint. The N:M sparse constraint requires that exactly N non-zero weights are retained in M ​​consecutive weights.

[0022] The gradient-aware dynamic mask generation strategy is as follows: the selection of the dynamic mask matrix is ​​based on both the absolute magnitude of the weights |W| and the instantaneous gradient magnitude of the weights |G|, that is: Mask t =Mask t (W t G t )=select N,M (|W t |+λ·|G t |)(1) Where t is the update time step index of the dynamic mask matrix, and W t G represents the weight matrix at step t (which can be understood as the size of the weight matrix for any network layer being equal to the number of input neurons multiplied by the number of output neurons in that network layer). t This represents the weight gradient matrix at step t, i.e., the loss function with respect to W. t The matrix composed of the gradients of each weight in the Mask. t Let represent the dynamic mask matrix at step t; λ is the weighting coefficient, which is a hyperparameter used to balance the current synaptic efficacy and potential learning ability. select N,M (·) indicates setting the element in the mask matrix corresponding to the N largest weights out of every M consecutive weights to 1, and setting the corresponding elements in other positions to 0. That is, setting the matrix |W t |+λ·|G t | Divide M consecutive elements into non-overlapping groups. Within each group, select the top N weights with the largest weights, set the corresponding elements in the dynamic mask matrix to 1, and set the corresponding elements in other positions to 0.

[0023] The shadow gradient accumulation mechanism is as follows: historical gradient information corresponding to weights temporarily masked by the current dynamic mask is accumulated through a shadow buffer. The update recursive relationship of this shadow buffer is as follows: H t+1 =ρ·H t +(1–Mask t)⊙G t (2) Among them, H t Let H1 be the shadow gradient buffer matrix at step t, with its initial value set to zero. ρ is the decay factor that controls the historical gradient time window and satisfies 0 < ρ < 1. ⊙ represents element-wise multiplication.

[0024] This mechanism fuses the gradient information accumulated in the shadow buffer with the current gradient to generate a corrected gradient estimate: t = G t + H t (3) in, t This is the corrected gradient, used for subsequent mask update decisions.

[0025] S2, the sparse topological mask generated in step S1 is iteratively updated using a multiplicative mask update rule. This update rule uses normalized gradient information to perform scale-invariant competition on the mask score in order to eliminate the initialization advantage bias in the sparse topological evolution process.

[0026] The multiplicative mask update rule is as follows: First, maintain a scoring matrix Q, and the iterative update of this scoring matrix adopts the following normalized multiplicative form: Q t+1 =(1-α)·Q t +α·(| t | / t (4) Where α is the momentum renewal rate and satisfies 0 < α < 1, t Let be the corrected gradient at step t. t = mean(| t |) represents the global mean of the absolute values ​​of the corrected gradients within the current layer. The initial value of Q, Q1, is set to the zero matrix.

[0027] At each mask update, based on the updated score matrix Q t+1 Re-execute select N,M The (·) operation generates a new binary dynamic mask matrix, i.e., Mask. t+1 =select N,M (Q t+1 ).

[0028] Preferably, the step of calculating the deviation between the actual average distribution rate and the preset target distribution rate, and updating the current weight matrix based on the deviation, includes: A1, let W denote the current weight matrix and the actual average distribution rate, respectively. n r a (n) After the spiking neural network completes n forward propagations, r is updated. a (n) The updated average distribution rate r was obtained. a (n+1) Using r a (n+1) Update W n The updated weight matrix W is obtained. n+1 ; Where, r a (n+1) = c1·r a (n) + c2·MEAN(s (n) ), s (n) Let W be the pulse tensor input to the spiking neural network at the nth time step. MEAN(·) represents averaging along the neuron dimension. c1 and c2 are weight coefficients, and c1 + c2 = 1. n+1 =W n (1+η·(r t -r a (n+1) )), r t Let η be the preset target distribution rate, η be the steady-state coupling constant, n be the update time step index of the weight matrix, and r be the value of the target distribution rate. a (n) The initial value is r a (1) =0.

[0029] Specifically, the update process for the weight matrix of each network layer is as follows: A firing rate steady-state plasticity mechanism is introduced, which performs global multiplicative scaling on the synaptic weights of neurons on a slow timescale to adjust the average firing rate of the neuron cluster to a preset target firing rate level, thus preventing irreversible neuronal silencing caused by a reduction in the number of synaptic afferents.

[0030] The steady-state plasticity mechanism of the firing rate is as follows: The actual average firing rate r of each layer in the network is monitored in real time using an exponential moving average. a Its update method is as follows: r a (n+1) = c1·r a (n) + c2·mean(s (n) (5) Where n is the index of the update time step of the weight matrix (i.e., the pulse time step of the forward propagation of the spiking neural network), and s (n) Let c1 be the pulse tensor input to the spiking neural network at the nth time step, and mean(·) be the average value calculated along the neuron dimension. The coefficients c1 and c2 satisfy a convex combination relationship of c1 + c2 = 1.

[0031] That is, after the spiking neural network completes the nth forward propagation, r is updated. a (n) The updated average distribution rate r was obtained. a (n+1) Then, the actual average distribution rate r was calculated. a (n+1) Compared with the preset target distribution rate r t The deviation between them is used to perform global multiplicative scaling on the synaptic weights (i.e., the weight matrix) of the neurons: W n+1 =W n (1+η·(r t -r a (n+1) ))(6) Where η is the steady-state coupling constant and satisfies η>0, r t For the preset target distribution rate, r a For the updated international average distribution rate, this multiplicative scaling operation preserves the relative information structure carried by the weight distribution.

[0032] During the training of the spiking neural network, the iterative updates of the mask matrix and the weight matrix are performed alternately. The iterative update of the mask matrix is ​​to dynamically optimize the sparse connection structure of the network based on steps S1 and S2. The iterative update of the weight matrix is ​​to adjust the synaptic weight values ​​based on step A1 combined with the surrogate gradient of backpropagation. After training is completed, the latest weight matrix of each network layer is multiplied element-wise with the latest dynamic mask matrix to obtain the corresponding sparse weight matrix, thus completing the compression of the spiking neural network. This sparse weight matrix is ​​used to characterize the compressed spiking neural network.

[0033] That is: W sparse t+1 =W t+1 ⊙Mask t+1 (7) Understandably, during the training phase, the input to the spiking neural network is a pulse sequence generated by encoding the sample images in the training set; during the inference phase, the input to the spiking neural network is a pulse sequence generated by encoding the image to be processed.

[0034] Preferably, t = Kn, where K is a positive integer.

[0035] It is understandable that the update time step index n of the weight matrix and the update time step index t of the dynamic mask matrix can be the same, i.e., t=n. In other words, every time the weight matrix is ​​updated, the dynamic mask matrix is ​​updated simultaneously.

[0036] Considering that frequent updates to the dynamic mask matrix can lead to significant inference time delays and energy consumption, the updates of the weight matrix and the dynamic mask matrix can be performed on different time scales. The update time scale of the weight matrix is ​​set to be smaller than that of the dynamic mask matrix, i.e., t > n and t = Kn. That is, after every Kn-th weight matrix update, the nth dynamic mask matrix update is performed simultaneously. The weight matrix is ​​updated after each mini-batch training, while the mask matrix is ​​updated once after every K training mini-batches. Furthermore, the update amplitude of the firing rate steady-state adjustment is limited by the steady-state coupling constant η to maintain training stability.

[0037] The method provided by the present invention will be further illustrated below with a specific example.

[0038] Taking the spurious residual network on the CIFAR-10 dataset as an example, the network uses a leaky integral firing neuron model with a time step of 4, a membrane potential decay factor β=0.5, and a firing threshold θ=0.5. The N:M sparsity constraint is set to 2:4, meaning that the two connections with the largest weights are retained out of every four consecutive weights.

[0039] During training, a surrogate gradient method is used for backpropagation. The surrogate gradient approximation function is: s / v ≈ σ(5v)·(1 - σ(5v)) Where σ(·) is the sigmoid function, s is the pulse output, and v is the membrane potential.

[0040] Before each forward propagation, gradient-aware dynamic mask generation is performed, with λ=0.1. The shadow gradient buffer has a decay factor ρ=0.9, and the mask is updated every 200 mini-batches. The momentum update rate α of the scoring matrix is ​​0.1. The target firing rate r of the firing rate steady-state plasticity mechanism is... t Set to 0.05, steady-state coupling constant η = 0.01, moving average coefficients c1 = 0.99, c2 = 0.01.

[0041] Training was performed using the Adam optimizer with an initial learning rate of 0.001 and a training period of 200 epochs. Experimental results show that, under a 2:4 sparsity constraint, the method of this invention maintains classification accuracy comparable to that of dense networks, while theoretically reducing the number of floating-point operations by about 50%, and without exhibiting complete neuron quiescence.

[0042] This invention provides a hardware deployment method for a spiking neural network, comprising: The compressed spiking neural network obtained by the spiking neural network compression method described in any of the above embodiments is deployed to a general neural network accelerator; wherein the compressed spiking neural network is characterized by the sparse weight matrix of each network layer.

[0043] It is understood that the general-purpose neural network accelerator can be a high-performance chip.

[0044] This invention provides an electronic device, including: a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any of the above embodiments.

[0045] This invention provides a computer-readable storage medium storing computer instructions that cause a processor to perform the method described in any of the above embodiments.

[0046] This invention provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the method described in any of the above embodiments.

[0047] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A compression method based on a steady-state unbiased sparse topology spiking neural network, wherein the spiking neural network is used for image processing and is compatible with hardware-friendly N:M structured sparse patterns, characterized in that... The method includes: During the training process of a spiking neural network, the dynamic mask matrix and weight matrix of each network layer are updated alternately. After training, the latest weight matrix of each network layer is multiplied element by element with the latest dynamic mask matrix to obtain the corresponding sparse weight matrix. The update process of the dynamic mask matrix includes: Generate the current dynamic mask matrix based on the current weight matrix and gradient matrix; correct the current gradient matrix using the current shadow gradient buffer matrix; update the current shadow gradient buffer matrix using the current dynamic mask matrix; update the scoring matrix using the corrected gradient matrix, thereby updating the dynamic mask matrix. The update process of the weight matrix includes: Calculate the deviation between the actual average distribution rate and the preset target distribution rate, and update the current weight matrix based on the deviation; The process of generating the current dynamic mask matrix based on the current weight matrix and gradient matrix; correcting the current gradient matrix using the current shadow gradient buffer matrix; updating the current shadow gradient buffer matrix using the current dynamic mask matrix; and updating the scoring matrix using the corrected gradient matrix to update the dynamic mask matrix includes: (1) Let W be the current weight matrix and gradient matrix respectively. t G t Generate the current dynamic mask matrix Mask t Let H be the current shadow gradient buffer matrix. t Using H t Correction G t Obtain the corrected gradient matrix t Use Mask t Update H t The updated shadow gradient buffer matrix H is obtained. t+1 ; wherein Mask t = select N,M (|W t |+λ·|G t |), λ is a weight coefficient, select N,M (·) means that the elements corresponding to the positions of the first N largest weights in each continuous M weights in the mask matrix are set to 1, and the elements corresponding to the other positions are set to 0; H t+1 = ρ·H t +(1–Mask t )⊙G t , ρ is an attenuation factor, and ⊙ represents element-wise multiplication; (2) Let Q be the current rating matrix. t ,use t Update Q t The updated rating matrix Q is obtained. t+1 Using Q t+1 Calculate the updated dynamic mask matrix Mask t+1 ; Among them, Q t+1 =(1-α)·Q t +α·(| t | / t ), where α is the momentum renewal rate. t =mean(| t |), mean(·) represents calculating the global mean, and t is the index of the update time step of the dynamic mask matrix.

2. The method as described in claim 1, characterized in that, The step of calculating the deviation between the actual average distribution rate and the preset target distribution rate, and updating the current weight matrix based on the deviation, includes: Let W denote the current weight matrix and the actual average distribution rate, respectively. n r a (n) After the spiking neural network completes the nth forward propagation, r is updated. a (n) The updated actual average distribution rate r is obtained. a (n+1) Using r a (n+1) Update W n The updated weight matrix W is obtained. n+1 ; Where, r a (n+1) = c1·r a (n) + c2·MEAN(s (n) ), s (n) Let W be the pulse tensor input to the spiking neural network at the nth time step. MEAN(·) represents averaging along the neuron dimension. c1 and c2 are weight coefficients, and c1 + c2 = 1. n+1 =W n (1+η·(r t -r a (n+1) )), r t The preset target distribution rate is η, the steady-state coupling constant is n, and the update time step index of the weight matrix is ​​n.

3. The method as described in claim 2, characterized in that, t = Kn, where K is a positive integer.

4. A hardware deployment method for a spiking neural network, characterized in that, include: The compressed spiking neural network obtained by the spiking neural network compression method as described in any one of claims 1-3 is deployed to a general neural network accelerator; wherein the compressed spiking neural network is characterized by the sparse weight matrix of each network layer.

5. An electronic device, characterized in that, include: Computer-readable storage media and processors; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any one of claims 1-3 or the method as described in claim 4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a processor to perform the method as described in any one of claims 1-3 or the method as described in claim 4.

7. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1-3 or the method as described in claim 4.

Images