A lightweight pulse neural network method and system suitable for embedded devices

Abstract

The present invention belongs to the field of computer vision and machine learning technology, and in particular to a pulse neural network lightweight method and system suitable for embedded devices. The pulse neural network lightweight method suitable for embedded devices of the present invention, by utilizing knowledge distillation technology, transfers the knowledge learned by a complex artificial neural network to a hybrid pulse neural network, thereby realizing the miniaturization and lightweighting of the pulse neural network. The present invention greatly improves the model inference speed through parameter clustering and TensorRT conversion of the distilled pulse neural network model, and realizes the efficient operation of the pulse neural network. The pulse neural network lightweight system suitable for embedded devices disclosed by the present invention, by deploying the lightweight pulse neural network on the embedded device, realizes real-time image acquisition, real-time inference and outputs the inference results through the display module in real time, thereby realizing the real-time operation of the pulse neural network on the embedded device.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and machine learning, and in particular relates to a spiking neural network lightweight method and system suitable for embedded devices. Background Art

[0002] With the development of artificial intelligence (AI) technology, the intelligent capabilities of embedded devices are gaining increasing attention. However, due to hardware resource and power constraints, deploying complex, full-scale neural networks on these devices is nearly impossible. Furthermore, embedded devices often interact directly with users, so the real-time performance of neural network deployment on these devices directly impacts the user experience. The Nvidia Jetson series is Nvidia's flexible and scalable hardware for embedded AI scenarios. Each Nvidia Jetson module features a complete GPU, CPU, memory, power management, and expansion interfaces, allowing for easy integration into diverse hardware. Leveraging Nvidia's extensive experience in GPU design, Nvidia Jetson devices possess the powerful computing power required for inference of neural network models in embedded devices, along with the TensorRT inference runtime library for accelerated computation for embedded device deployment.

[0003] Spiking neural networks, inspired by the biological brain, use discrete spike trains of neurons to transmit information. Because spike signals have rich spatiotemporal characteristics, they can represent a rich array of information. Furthermore, the discrete nature of spike signals enables spiking neural networks to achieve powerful computational capabilities with extremely low power consumption. Consequently, spiking neural networks are considered to hold great potential as next-generation neural networks. However, due to limitations in the learning mechanisms of spiking neural networks and the neuron size of neuromorphic hardware, running hybrid spiking neural networks on traditional embedded hardware has become a new option. Specifically, it is possible to run a hybrid spiking neural network with a partial artificial neural network on traditional embedded hardware. The artificial neural network portion primarily performs pulse encoding of continuous data, while the core of the network and its information transmission method remain in the form of spikes.

[0004] Knowledge distillation is a major method for network lightweighting. Its core idea is to first train a large, high-performance teacher network with complete knowledge, and then use the teacher network to guide the learning of a smaller, lightweight student network. Knowledge distillation effectively leverages the powerful fitting capabilities of the large network, transferring the knowledge learned from the large network to the smaller network through distillation, thereby achieving superior performance in the smaller network.

[0005] Deploying high-performance spiking neural networks on embedded devices to achieve real-time information acquisition and real-time display of image classification results is a challenging task. To achieve this, it is necessary to adopt lightweight models and inference acceleration for spiking neural networks. Summary of the Invention

[0006] In order to solve the problems of the prior art, the present invention provides a pulse neural network lightweight method and system suitable for embedded devices.

[0007] The technical solution of the present invention is:

[0008] A lightweight method for a spiking neural network suitable for embedded devices includes the following steps:

[0009] S1. Build the teacher network: The teacher network uses ResNet-34 as the basic network structure, with an input size of 32*32*3 and an RGB image.

[0010] Construct a student network: The student network is a pulse convolutional neural network with VGG as the basic network structure. In the vertical direction, it includes the first convolution layer, the first batch normalization layer, the first maximum pooling layer, the first pulse neuron layer, the second convolution layer, the second pulse neuron layer, the third convolution layer, the second batch normalization layer, the second maximum pooling layer, the third pulse neuron layer, the fourth convolution layer, the fourth pulse neuron layer, the fifth convolution layer, the third batch normalization layer, the third maximum pooling layer, the fifth pulse neuron layer, the sixth convolution layer, the sixth pulse neuron layer, the first average pooling layer and the first fully connected layer; all convolution layers of the student network are two-dimensional convolution layers with a convolution kernel size of 3*3, a stride of 1, and a padding of 1; the size of the maximum pooling layer and the average pooling layer of the student network is 2*2; the input of the student network is a 32*32*3 RGB image input, and the output is a pulse sequence output;

[0011] S2. Use the Cifar-10 dataset to train the teacher network to obtain a trained teacher network;

[0012] S3, initialize the parameters of the student network and perform 10 rounds of pre-training using the Cifar-10 dataset;

[0013] S4. Use the Cifar-10 dataset and the trained teacher network model to adjust and train the student network. The adjustment training uses the knowledge learned from the teacher network to guide the student network training process. Specifically:

[0014] The distillation loss function of the student network during the adjustment training process is defined as:

[0015] L=αL S+βL H

[0016] Among them, α and β are the training hyperparameters set during the training process, L S and L H They are soft label classification loss and hard label classification loss respectively;

[0017] The soft label classification loss is:

[0018]

[0019] Where N is the total number of samples, T is the hyperparameter for adjusting training, and is defined as the distillation temperature; represents the output of the teacher network for the jth sample at the distillation temperature T; represents the output of the student network for the jth sample at the distillation temperature T;

[0020] The hard label classification loss is:

[0021]

[0022] Among them, c j represents the true label of the jth sample; when calculating the hard label loss, the distillation temperature T is equal to 1;

[0023] During the training process, the same sample is input, and the student network and the teacher network each obtain an output. The loss is calculated based on the difference between the two. The student network updates its weights based on the distillation loss of the student network.

[0024] S5. Freeze and adjust the parameters of the student network model obtained after training, cluster the model parameters of each layer, and represent the parameters of each layer with 8 cluster centers to obtain a cluster model;

[0025] S6. Convert the clustering model to a TensorRT model. After conversion, a lightweight model is obtained for direct deployment in embedded devices.

[0026] Furthermore, when training the teacher network in S2, the Cifar-10 data is first preprocessed, and the dataset is randomly flipped, randomly cropped, and the data is normalized; the training process uses the SGD optimization algorithm, and the number of training iterations is 300 rounds.

[0027] Furthermore, the 10-round pre-training method for the student network in S3 is based on the back-propagation algorithm with gradient substitution, and the substitution function used is the Sigmoid function.

[0028] Furthermore, the clustering method used in S5 is K clustering, and the process is as follows:

[0029] First, obtain the parameters of each layer in the student model;

[0030] Secondly, the k-means method in the sklearn machine learning library is called to cluster the obtained model parameters, the clustering class is set to 8, and 8 cluster centers and the mapping between each parameter and the cluster center are obtained;

[0031] Next, each parameter is replaced by the cluster center to obtain the clustered parameters;

[0032] Finally, the clustering model is obtained by replacing the student model parameters of each layer with the clustered parameters.

[0033] A lightweight pulse neural network system suitable for embedded devices, comprising:

[0034] The image acquisition module, which includes a general-purpose USB camera, acquires color RGB images and transmits them to the Nvidia Jetson embedded device via the USB interface. After conversion, they serve as input to the network inference module.

[0035] The network lightweight module trains the teacher model, pre-trains the student model and distills the knowledge learned in the teacher model into the student model, and is responsible for completing the lightweight operation before network deployment; the teacher model uses ResNet-34 as the basic network structure, the input size is 32*32*3, and the input is an RGB image; the student model is a pulse convolutional neural network with VGG as the basic network structure, which includes the first convolution layer, the first batch of normalization layers, the first maximum pooling layer, the first pulse neuron layer, the second convolution layer, the second pulse neuron layer, the third convolution layer, the second batch of normalization layers, the second maximum pooling layer, the third pulse neuron layer, the fourth convolution layer, the fourth pulse neuron layer, the fifth convolution layer, the third batch of normalization layers, the third maximum pooling layer, the fifth pulse neuron layer layer, the sixth convolutional layer, the sixth spike neuron layer, the first average pooling layer and the first fully connected layer; all convolutional layers of the student network are two-dimensional convolutional layers, with a convolution kernel size of 3*3, a stride of 1, and a padding of 1; the maximum pooling layer and the average pooling layer of the student network are 2*2 in size; the input of the student network is a 32*32*3 RGB image input, and the output is a pulse sequence output; the method for training the teacher model is to train the teacher network using the Cifar-10 dataset to obtain a trained teacher network; the pre-trained student model is to initialize the parameters of the student network and perform 10 rounds of pre-training using the Cifar-10 dataset; the specific method of distilling the knowledge learned in the teacher model into the student model is: define and adjust the distillation loss function of the student network during training as:

[0036] L=αL S+βL H

[0037] Among them, α and β are the training hyperparameters set during the training process, L S and L H They are soft label classification loss and hard label classification loss respectively;

[0038] The soft label classification loss is:

[0039]

[0040] Where N is the total number of samples, T is the hyperparameter for adjusting training, and is defined as the distillation temperature; represents the output of the teacher network for the jth sample at the distillation temperature T; represents the output of the student network for the jth sample at the distillation temperature T;

[0041] The hard label classification loss is:

[0042]

[0043] Among them, c j represents the true label of the jth sample; when calculating the hard label loss, the distillation temperature T is equal to 1;

[0044] During the training process, the same sample is input, and the student network and the teacher network each obtain an output. The loss is calculated based on the difference between the two. The student network updates its weights based on the distillation loss of the student network.

[0045] The parameters of the student network model obtained after freeze adjustment training are clustered, and the model parameters of each layer are represented by 8 cluster centers to obtain a cluster model; the cluster model is converted into a TensorRT model, and after conversion, a lightweight model is obtained for direct deployment in embedded devices.

[0046] The network inference module deploys the student model converted by TensorRT and is used to implement real-time computing and output results in the Nvidia Jetson embedded device;

[0047] The display module displays the real-time image data collected by the image acquisition module and overlays the inference results output by the network inference module.

[0048] Beneficial effects of the present invention: The pulse neural network lightweight method suitable for embedded devices of the present invention transfers the knowledge learned by a complex artificial neural network to a hybrid pulse neural network by utilizing knowledge distillation technology, thereby realizing the miniaturization and lightweighting of the pulse neural network. The present invention greatly improves the model inference speed through parameter clustering and TensorRT conversion of the distilled pulse neural network model, thereby realizing the efficient operation of the pulse neural network. The pulse neural network lightweight system suitable for embedded devices disclosed by the present invention realizes real-time image acquisition, real-time inference and outputs the inference results through the display module in real time by deploying the lightweight pulse neural network on the embedded device, thereby realizing the real-time operation of the pulse neural network on the embedded device. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is the basic structure of the ResNet-34 teacher network;

[0050] Figure 2 is the network structure diagram of the student network;

[0051] Figure 3 Schematic diagram of knowledge distillation during the training phase of adjusting the student network;

[0052] Figure 4 This is a module connection diagram of a spiking neural network lightweight system suitable for embedded devices;

[0053] Figure 5 This is an operation flow chart of a pulse neural network lightweight method and system applicable to embedded devices disclosed by the present invention. DETAILED DESCRIPTION

[0054] The technical solution of the present invention is described in detail below with reference to the accompanying drawings:

[0055] Implementing high-performance, real-time neural networks in embedded devices often requires targeted hardware optimization or custom development. However, such hardware often lacks versatility and is costly to develop. Deploying spiking neural networks into embedded devices often consumes significant hardware resources and cannot achieve real-time inference performance. Due to limited hardware resources, spiking neural networks deployed in embedded devices often have simple structures, few layers, and poor performance.

[0056] To achieve the above objectives, the present invention proposes a lightweight spiking neural network method and system suitable for embedded devices, which includes the following two parts:

[0057] The first part is a lightweight approach to spiking neural networks suitable for embedded devices, including:

[0058] Build a ResNet-34 artificial neural network as the teacher network and train it for 300 iterations;

[0059] Build a hybrid spiking neural network with 6 convolutional layers and perform 10 rounds of pre-training;

[0060] Construct a distillation loss function and determine its hyperparameters through experimental methods;

[0061] The student network is trained using the Cifar-10 dataset and the teacher model to obtain a student model with performance close to that of the teacher model.

[0062] Freeze the student model parameters, cluster the model parameters of each layer, and represent the parameters of each layer with 8 cluster centers to obtain the cluster model;

[0063] Convert the clustering model to a TensorRT model.

[0064] The second part is a lightweight pulse neural network system suitable for embedded devices, including:

[0065] The image acquisition module includes a general-purpose USB camera that acquires color RGB images and transmits them to the Nvidia Jetson embedded device via the USB interface as input for the converted student model;

[0066] The network lightweight module trains the teacher model, pre-trains the student model, and distills the knowledge learned from the teacher model into the student model, completing lightweight operations before network deployment;

[0067] The network inference module deploys the student model converted by TensorRT, which can realize real-time calculation and output results in the Nvidia Jetson embedded device;

[0068] The display module displays the real-time image data collected by the image acquisition module and overlays the inference results output by the network inference module.

[0069] In order to more clearly illustrate the implementation method of a spiking neural network lightweight system suitable for embedded devices disclosed in the present invention, the specific implementation steps of the present invention will be described below with reference to the accompanying drawings:

[0070] Step 1, such as Figure 1 As shown in the figure, based on the commonly used network structure ResNet in the field of image classification in artificial neural networks, a ResNet-34 teacher network is constructed, and the network input is a 32*32*3 RGB image;

[0071] Step 2: Construct a hybrid pulse neural network with 6 convolutional layers, defined as a student network. The network structure and input and output are as follows: Figure 2 As shown in the figure, the basic structure of the student network is VGG, the input of the student network is 32*32*3, and the student network contains a spike neuron layer;

[0072] Step 3: Perform random flipping, random cropping, and data normalization on the Cifar-10 dataset.

[0073] Step 4: Use the processed Cifar-10 dataset to train the teacher network for 300 rounds of iterations to obtain a teacher model with good performance;

[0074] Step 5: Pre-train the student network using the back-propagation algorithm based on gradient substitution and the Cifar-10 dataset for 10 rounds.

[0075] Step 6: Construct the distillation loss function of the student network and select appropriate α, β and distillation temperature T by experimenting with different parameter combinations;

[0076] Step 7: Use the teacher model and the constructed distillation loss function to adjust the pre-trained student network. The knowledge distillation process of the training process is adjusted as follows: Figure 3 As shown in the figure, the parameters of the student network are adjusted according to the distillation loss. The adjustment algorithm is the back propagation algorithm based on the surrogate gradient. The number of iterations is 100 to obtain the student model.

[0077] Step 8: Freeze the student model parameters and cluster the parameters of each layer in the student model using the K clustering method. The parameters of each layer are clustered into 8 categories, and the 8 cluster centers are used to replace the original parameters to obtain the cluster model.

[0078] Step 9: Use the Nvidia TensorRT runtime library in the network lightweight module to convert the clustering model into a TensorRT model;

[0079] Step 10: Deploy the TensorRT model to the network inference module. The image acquisition module is connected to the network inference module via a USB interface. The network inference module uses the image input as the input of the TensorRT model. The TensorRT model performs inference in the network inference module to obtain the image classification results. The classification results of the network inference module are displayed and output through the display module. The connection relationship between the modules is as follows: Figure 4 shown.

[0080] like Figure 5The figure shows a flow chart for fully implementing a pulse neural network lightweight method and system suitable for embedded devices disclosed in the present invention, specifically involving a pulse neural network lightweight method suitable for embedded devices proposed by the present invention and the entire process of its deployment in the corresponding system.

Claims

1. A spiking neural network lightweight method suitable for embedded devices, characterized in that: The following steps are involved: S1. Build the teacher network: The teacher network uses ResNet-34 as the basic network structure, with an input size of 32*32*3 and an RGB image. Construct a student network: The student network is a pulse convolutional neural network with VGG as the basic network structure. In the vertical direction, it includes the first convolution layer, the first batch normalization layer, the first maximum pooling layer, the first pulse neuron layer, the second convolution layer, the second pulse neuron layer, the third convolution layer, the second batch normalization layer, the second maximum pooling layer, the third pulse neuron layer, the fourth convolution layer, the fourth pulse neuron layer, the fifth convolution layer, the third batch normalization layer, the third maximum pooling layer, the fifth pulse neuron layer, the sixth convolution layer, the sixth pulse neuron layer, the first average pooling layer and the first fully connected layer; all convolution layers of the student network are two-dimensional convolution layers with a convolution kernel size of 3*3, a stride of 1, and a padding of 1; the size of the maximum pooling layer and the average pooling layer of the student network is 2*2; the input of the student network is a 32*32*3 RGB image input, and the output is a pulse sequence output; S2. Use the Cifar-10 dataset to train the teacher network to obtain a trained teacher network; S3, initialize the parameters of the student network and perform 10 rounds of pre-training using the Cifar-10 dataset; S4. Use the Cifar-10 dataset and the trained teacher network model to adjust and train the student network. The adjustment training uses the knowledge learned from the teacher network to guide the student network training process. Specifically: The distillation loss function of the student network during the adjustment training process is defined as: L=αL S +βL H Among them, α and β are the training hyperparameters set during the training process, L S and L H They are soft label classification loss and hard label classification loss respectively; The soft label classification loss is: Where N is the total number of samples, T is the hyperparameter for adjusting training, and is defined as the distillation temperature; represents the output of the teacher network for the jth sample at the distillation temperature T; represents the output of the student network for the jth sample at the distillation temperature T; The hard label classification loss is: Among them, c j represents the true label of the jth sample; when calculating the hard label loss, the distillation temperature T is equal to 1; During the training process, the same sample is input, and the student network and the teacher network each obtain an output. The loss is calculated based on the difference between the two. The student network updates its weights based on the distillation loss of the student network. S5. Freeze and adjust the parameters of the student network model obtained after training, cluster the model parameters of each layer, and represent the parameters of each layer with 8 cluster centers to obtain a cluster model; S6. Convert the clustering model to a TensorRT model. After conversion, a lightweight model is obtained for direct deployment in embedded devices.

2. The method for lightweighting a pulse neural network suitable for embedded devices according to claim 1, characterized in that: When training the teacher network in S2, the Cifar-10 data is first preprocessed, the dataset is randomly flipped, randomly cropped, and the data is normalized; the training process uses the SGD optimization algorithm for training, and the number of training iterations is 300 rounds.

3. The method for lightweighting a pulse neural network suitable for embedded devices according to claim 2, characterized in that: The 10-round pre-training method for the student network in S3 is a back-propagation algorithm based on gradient substitution, and the substitution function used is the Sigmoid function.

4. The method for lightweighting a spiking neural network suitable for embedded devices according to claim 3, characterized in that: The clustering method used in S5 is K clustering, and the process is as follows: First, obtain the parameters of each layer in the student model; Secondly, the k-means method in the sklearn machine learning library is called to cluster the obtained model parameters, the clustering class is set to 8, and 8 cluster centers and the mapping between each parameter and the cluster center are obtained; Next, each parameter is replaced by the cluster center to obtain the clustered parameters; Finally, the clustering model is obtained by replacing the student model parameters of each layer with the clustered parameters.

5. A spiking neural network lightweight system suitable for embedded devices, characterized in that: include: The image acquisition module, which includes a general-purpose USB camera, acquires color RGB images and transmits them to the Nvidia Jetson embedded device via the USB interface. After conversion, they serve as input to the network inference module. The network lightweight module trains the teacher model, pre-trains the student model and distills the knowledge learned in the teacher model into the student model, and is responsible for completing the lightweight operation before network deployment; the teacher model uses ResNet-34 as the basic network structure, the input size is 32*32*3, and the input is an RGB image; the student model is a pulse convolutional neural network with VGG as the basic network structure, which includes the first convolution layer, the first batch of normalization layers, the first maximum pooling layer, the first pulse neuron layer, the second convolution layer, the second pulse neuron layer, the third convolution layer, the second batch of normalization layers, the second maximum pooling layer, the third pulse neuron layer, the fourth convolution layer, the fourth pulse neuron layer, the fifth convolution layer, the third batch of normalization layers, the third maximum pooling layer, the fifth pulse neuron layer layer, the sixth convolutional layer, the sixth spike neuron layer, the first average pooling layer and the first fully connected layer; all convolutional layers of the student network are two-dimensional convolutional layers, with a convolution kernel size of 3*3, a stride of 1, and a padding of 1; the maximum pooling layer and the average pooling layer of the student network are 2*2 in size; the input of the student network is a 32*32*3 RGB image input, and the output is a pulse sequence output; the method for training the teacher model is to train the teacher network using the Cifar-10 dataset to obtain a trained teacher network; the pre-trained student model is to initialize the parameters of the student network and perform 10 rounds of pre-training using the Cifar-10 dataset; the specific method of distilling the knowledge learned in the teacher model into the student model is: define and adjust the distillation loss function of the student network during training as: L=αL S +θL H Among them, α and β are the training hyperparameters set during the training process, L S and L H They are soft label classification loss and hard label classification loss respectively; The soft label classification loss is: Where N is the total number of samples, T is the hyperparameter for adjusting training, and is defined as the distillation temperature; represents the output of the teacher network for the jth sample at the distillation temperature T; represents the output of the student network for the jth sample at the distillation temperature T; The hard label classification loss is: Among them, c j represents the true label of the jth sample; when calculating the hard label loss, the distillation temperature T is equal to 1; During the training process, the same sample is input, and the student network and the teacher network each obtain an output. The loss is calculated based on the difference between the two. The student network updates its weights based on the distillation loss of the student network. Freeze and adjust the parameters of the student network model obtained after training, cluster the model parameters of each layer, and represent the parameters of each layer with 8 cluster centers to obtain a cluster model; convert the cluster model into a TensorRT model, and obtain a lightweight model for direct deployment in embedded devices; The network inference module deploys the student model converted by TensorRT and is used to implement real-time computing and output results in the Nvidia Jetson embedded device; The display module displays the real-time image data collected by the image acquisition module and overlays the inference results output by the network inference module.

Images