Spiking neural network generation method, image recognition device, and computer system
By modifying the structure and learning conditions of deep neural networks to generate SNNs, the method addresses the accuracy deterioration issue in converting CNNs to SNNs, achieving high recognition accuracy and low power consumption for image recognition tasks.
Patent Information
- Application Number
- PCT/JP2024/015164
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-17
- Filing Date
- 2024-04-16
- Publication Date
- 2024-11-21
AI Technical Summary
Converting multi-layer convolutional neural networks (CNNs) to spiking neural networks (SNNs) results in significant deterioration of recognition accuracy due to input attenuation in each convolutional layer, leading to loss of original image information and reduced performance.
A method for generating SNNs by modifying the deep neural network structure to satisfy specific neuron activation rates and learning conditions, involving the synthesis of outputs and inputs across convolutional layers, and using software tools like LAVA or Nengo to convert learned DNNs into SNNs, ensuring high recognition accuracy.
The approach generates SNNs that maintain recognition accuracy comparable to CNNs, enabling efficient image recognition with reduced power consumption, suitable for applications like surveillance systems, by optimizing neuron activation rates and layer connections.
Smart Images

Figure JP2024015164_21112024_PF_FP_ABST
Abstract
Description
Spiking neural network generation method, image recognition device, and computer system Incorporation by Reference
[0001] This application claims priority from Japanese Patent Application No. 2023-81386, filed on May 17, 2023, the contents of which are incorporated herein by reference.
[0002] The present invention relates to a method for generating spiking neural networks that mimic brain processing.
[0003] In recent years, the use of AI, particularly deep learning, has progressed in various applications including image recognition and voice recognition, and is being implemented in society in areas such as video surveillance and chatbots. Examples of applications of deep learning include video surveillance, for example, the detection and tracking of specific individuals such as suspicious or missing persons in public places such as train stations, airports, and streets, and the detection of specific behaviors of workers at work sites, such as dangerous behavior.
[0004] For image learning / inference, VGGnet (see, for example, Non-Patent Document 1) and Residual Network (ResNet) (see, for example, Non-Patent Document 2), which are types of convolutional neural networks (CNNs), are used because they can improve the recognition accuracy of complex images. In order to achieve high recognition accuracy when there are dozens or more types of objects to be recognized in an image, these neural networks are configured by connecting dozens to 100 or more convolutional layers. When performing image recognition using these CNNs, the computational processing load is large, so processing must be performed on a GPU (Graphics Processing Unit) server, which poses the problem of high power consumption.
[0005] To reduce power consumption, research is being conducted into converting CNNs into spiking neural networks (SNNs) that mimic brain processing, enabling inference to be performed on low-power brain-like devices (see, for example, Non-Patent Document 3).
[0006] “Very deep convolutional networks for large-scale image recognition”, K. Simonyan and A. Zisserman, in ICLR, 2015. “Deep Residual Learning for Image Recognition”, Kaiming He, et. Al., CVPR, 2016, Volume 1, pp 770-78”Benchmarking Keyword Spotting Efficiency on Neuromorphic Hardware”, P. Blouw, et al., in NICE '19, March 2019.
[0007] Converting a CNN to an SNN is advantageous in that it allows the use of CNNs that have proven success in applications such as image recognition, and is a major SNN construction method. Conventional CNNs with proven success in applications have many convolutional layers in their networks, ranging from tens to over 100. However, converting a CNN with many convolutional layers to an SNN poses the challenge of significantly degrading the recognition accuracy of the SNN compared to the CNN itself. This degradation in recognition accuracy is due to input attenuation in the convolutional layers that make up the CNN. When multiple convolutional layers are stacked, input attenuation in each convolutional layer results in the loss of information about the original image (input) in the final stage, resulting in a degradation in recognition accuracy. Because of this degradation in recognition accuracy when an SNN is converted from a multi-layer CNN, it has been impossible to utilize multi-layer CNNs that have proven success in applications.
[0008] Therefore, an object of the present invention is to generate an SNN whose recognition accuracy does not deteriorate compared to the recognition accuracy of a CNN when converting a multi-layer CNN into an SNN.
[0009] A representative example of the invention disclosed in the present application is as follows: A method for generating a spiking neural network executed by a computer having a processing device and a memory connected to the processing device, the method comprising: a first step in which the processing device receives first definition information defining a deep neural network including a plurality of connected convolutional layers and stores the information in the memory; a second step in which the processing device receives a condition that an activation rate of a neuron in the last convolutional layer must satisfy and stores the condition in the memory; a third step in which the processing device modifies the structure of the deep neural network so that an output of at least one of the convolutional layers and an input to the deep neural network are combined; a fourth step in which the processing device executes a training process using the deep neural network whose structure has been modified so that the condition is satisfied; and a fifth step in which the processing device converts the trained deep neural network to generate second definition information defining a spiking neural network and stores the second definition information in the memory.
[0010] According to the present invention, it is possible to generate an SNN that suppresses deterioration of recognition accuracy. Problems, configurations, and effects other than those described above will become clear from the following description of the embodiments.
[0011] FIG. 1 is a diagram illustrating structural features of an SNN according to a first embodiment. FIG. 2 is a diagram illustrating an example of the structure of an SNN according to a first embodiment. FIG. 3 is a diagram illustrating an example of the structure of an SNN according to a first embodiment. FIG. 4 is a flowchart illustrating an example of a method for generating an SNN according to a first embodiment. FIG. 5 is a diagram illustrating an example of a brain-type computer according to a first embodiment. FIG. 6 is a diagram illustrating an example of a utilization of a brain-type computer on which an SNN according to a first embodiment is implemented. FIG. 7 is a flowchart illustrating an example of image recognition processing executed by a brain-type computer on which an SNN according to a first embodiment is implemented. FIG. 8 is a diagram illustrating an example of a surveillance camera system according to a first embodiment. FIG. 9 is a diagram illustrating another example of the surveillance camera system according to a first embodiment.
[0012] An embodiment of the present invention will be described below with reference to the drawings, although the present invention is not limited to the embodiment.
[0013] In Example 1, a method for generating an SNN with high accuracy in recognition processing using an image containing a recognition target will be described. More specifically, in this embodiment, a method for realizing a system that satisfies the recognition accuracy required for a system that recognizes people, etc. in images obtained from a surveillance camera, etc. will be described.
[0014] First, the structural features of the spiking neural network (SNN) of this embodiment will be described. Fig. 1 is a diagram showing the structural features of the SNN of Example 1. Fig. 1 shows a part of the SNN 1, specifically showing the input to the convolutional layer, the output from the convolutional layer, and the connections between the convolutional layers.
[0015] The SNN1 is a spiking neural network that mimics brain processing for image recognition. The SNN1 has a structure in which multiple convolutional layers 10 are connected in series. A first input 50-1 to the SNN1 is input to the first convolutional layer 10-1. A first output 60-1 from the convolutional layer 10-1 is combined with a second input 50-2 branched from the first input 50-1, and input to the convolutional layer 10-2 as a third input 50-3. A second output 60-2 from the convolutional layer 10-2 is combined with the second input 50-2, and input to the next convolutional layer 50 (not shown) as a fourth input 50-4.
[0016] The SNN1 is generated by converting a trained DNN using, for example, LAVA from OSS (Open Source Software) or Nengo from Applied Brain Research.
[0017] Similar to the SNN1, the DNN has a structure in which multiple layers are connected in series.
[0018] FIG. 2 is a diagram illustrating an example of the structure of the SNN according to the first embodiment.
[0019] The SNN2 is composed of ten convolutional layers 10 and two fully connected layers 20. The multiple convolutional layers 10 are connected in series, and fully connected layers 20-1 and 20-2 are connected in series in the next stage after the final convolutional layer 10-10.
[0020] The first input 50-1 to the SNN2 is input to the first convolutional layer 10-1. The first output 60-1 of the convolutional layer 10-1 is combined with the second input 50-2, which is branched from the first input 50-1, and input to the convolutional layer 10-2 as the third input 50-3. The second output 60-2 of the convolutional layer 10-2 is combined with the second input 50-2 and input to the next convolutional layer 10 as the fourth input 50-4. The tenth output 60-10 of the final convolutional layer 10-10 is combined with the second input 50-2 and input to the fully connected layer 20-1 as the twelfth input 50-12. The final fully connected layer 20-2 outputs an output 70 as the final output of the SNN2.
[0021] FIG. 3 is a diagram illustrating an example of the structure of the SNN according to the first embodiment.
[0022] The SNN3 is composed of ten convolutional layers 11 and two fully connected layers 21. The multiple convolutional layers 11 are connected in series, and fully connected layers 21-1 and 21-2 are connected in series in the next stage after the final convolutional layer 11-10.
[0023] The first input 51-1 to the SNN3 is input to the first convolutional layer 11-1. The first output 61-1 of the convolutional layer 11-1 is combined with the second input 51-2, which is branched from the first input 51-1, and input to the convolutional layer 11-2 as the third input 51-3. The second output 61-2 of the convolutional layer 11-2 is combined with the second input 51-2 and input to the next convolutional layer 11 as the fourth input 51-4. The ninth output 61-9 of the ninth convolutional layer 11-9 is combined with the second input 51-2 and input to the final convolutional layer 11-10 as the eleventh input 51-11. The tenth output 61-10 of the final convolutional layer 11-10 is input to the fully connected layer 21-1. The final fully connected layer 21-2 outputs an output 71 as the final output of the SNN3.
[0024] The difference between SNN2 and SNN3 is the input to the first fully connected layers 20-1 and 21-1. Since the recognition accuracy of SNN2 and SNN3 varies depending on the recognition target, it is advisable to select an SNN that can achieve high recognition accuracy depending on the recognition target.
[0025] In SNN2, the outputs 60 of all convolutional layers 10 are combined with the second input 50-2, but in SNN3, the outputs 61 of the ninth convolutional layer 11-9, one layer before the final convolutional layer 11-10, are combined with the second input 51-2. The number of outputs to be combined with the second input can be adjusted, and it is advisable to select a configuration that provides the highest recognition accuracy.
[0026] Next, a method for generating an SNN will be described. Fig. 4 is a flowchart illustrating an example of a method for generating an SNN according to the first embodiment. Fig. 5 is a diagram illustrating an example of a brain-type computer according to the first embodiment.
[0027] To determine the configuration, a predetermined recognition accuracy is set based on the requirements of the image recognition system. Furthermore, a predetermined neuron activation rate is set based on the requirements of the predetermined recognition accuracy. Based on these two conditions, an SNN is generated.
[0028] The processing described below is executed by a brain-type computer 510 as shown in Fig. 5. The brain-type computer 510 includes a processing unit 511, a main memory device 512, an auxiliary memory device 513, a brain-type device 514, and a chipset 515.
[0029] The processing unit 511 is, for example, a CPU, and controls the entire brain-type computer 510. The brain-type device 514 is a dedicated device for executing the SNN.
[0030] The main memory device 512 is, for example, a volatile semiconductor memory, and temporarily stores software and various data loaded from the auxiliary memory device 513. The functions of the brain-type computer 510 are realized when the processing device 511 executes the programs stored in the main memory device 512. The auxiliary memory device 513 is, for example, a large-capacity non-volatile storage device such as a hard disk drive or SSD (Solid State Drive). The auxiliary memory device 513 stores software, various data used by the software, processing results of the brain-type device 514, etc.
[0031] The main storage device 512 and the auxiliary storage device 513 may be collectively referred to as a storage device.
[0032] Although not shown, the brain-type computer 510 is connected to terminals such as a monitor and a remote terminal, which allow the administrator to give operational instructions to the brain-type computer 510 and display the image recognition results.
[0033] The functions of the brain-type computer 510 (e.g., processing unit) may be realized using software, hardware such as a dedicated circuit, or a combination of software and hardware. Furthermore, one function of the brain-type computer 510 may be divided into multiple functions, or multiple functions may be combined into one function. Furthermore, some of the functions of the brain-type computer 510 may be provided as separate functions, or may be included in other functions. Furthermore, some of the functions of the brain-type computer 510 may be realized by another computer that can communicate with the brain-type computer 510.
[0034] The brain-type computer 510 receives information such as definition information of the DNN, recognition accuracy determined based on the requirements of the image recognition system, activation rates of neurons in the final convolutional layer determined based on the recognition accuracy, and information about the convolutional layer that synthesizes the branched inputs (second inputs), and then starts the processing described below.
[0035] When the processing device 511 starts the SNN generation process, it first sets the DNN (step S401).
[0036] Here, the number of convolutional layers to be connected and the number of fully connected layers to be connected are set. Also, as explained in Figures 2 and 3, the output from the final convolutional layer to the fully connected layer and the combination of the output of a specified convolutional layer and the second input are set. These settings are determined based on the characteristics of the image group, such as the type of image to be recognized.
[0037] Next, the processing device 511 sets learning parameters (step S402).
[0038] For example, the batch size, the number of epochs, the size of the filter, which is a matrix of weights multiplied by the input to the neurons constituting the convolution layer, and the number of filters are set as learning parameters. Parameters related to the activation rate of neurons are also set as learning parameters.
[0039] Next, the processing device 511 trains the DNN using the training teacher image under the set training parameters (step S403). The present invention is not limited to the training method of the DNN.
[0040] The processing device 511 determines whether the learning is completed (step S404).
[0041] Specifically, the processing device 511 determines whether the recognition accuracy of the DNN is equal to or greater than a first threshold value. If the recognition accuracy of the DNN is equal to or greater than the first threshold value, the processing device 511 determines that learning is complete.
[0042] If learning is not complete, the processing device 511 determines whether the activation rate of the neurons in the final convolutional layer is equal to or greater than a second threshold (step S405).
[0043] If the activation rate is lower than the second threshold, the processing device 511 returns to step S402, changes the learning parameters, and executes the same processing. If the activation rate is equal to or higher than the second threshold, the processing device 511 returns to step S401, changes the configuration of the DNN, and executes the same processing. The DNN and learning parameters may be set again by the user, or may be set automatically by the brain-type computer 510 based on the information initially set.
[0044] If it is determined in step S404 that the learning is complete, the processing device 511 converts the learned DNN into an SNN using a software tool (step S406), and then ends the process.
[0045] When performing inference, the processing device 511 uses a software tool to implement an SNN in the brain-like device 514 and performs inference using the brain-like device 514.
[0046] According to the above procedure, it is possible to generate an SNN that satisfies the recognition accuracy determined by the system requirements.
[0047] FIG. 6 is a diagram showing an example of utilization of the brain-type computer 510 that implements the SNN of the first embodiment.
[0048] 6 shows an image recognition device 500 consisting of a brain-type computer 510 and multiple cameras 520. The brain-type computer 510 is connected to the cameras 520 via a chipset 515. The cameras 520 capture images of the location where they are installed and transmit the image data in a predetermined format to the brain-type computer 510.
[0049] The processing device 511 executes software stored in the main memory device 512 to extract predetermined frame images from the video data input from the camera 520, and inputs the frame images to the brain-like device 514 to perform inference processing, and recognizes people and other objects appearing in the images based on the processing results. The brain-like device 514 processes the frame images using an SNN.
[0050] The video data transmitted from the camera 520, the frame images extracted from the video data, and the processing results by the brain-type device 514 are stored in at least one of the main memory device 512 and the auxiliary memory device 513.
[0051] FIG. 7 is a flowchart illustrating an example of image recognition processing executed by the brain-type computer 510 that implements the SNN of the first embodiment.
[0052] The processing device 511 acquires an image from the camera 520 and stores it in the main storage device 512 (step S701).
[0053] The processing device 511 extracts frame images from the video and extracts an inferred part for each frame image (step S702). For example, a part including a recognition target such as a person is extracted.
[0054] The processing device 511 generates an inference image by converting the extracted inference region into a predetermined image size for input to the brain-type device 514, and stores the image in the main storage device 512 (step S703). The inference image may also be stored in the auxiliary storage device 513 for confirmation.
[0055] The processing device 511 reads the inference image from the main storage device 512 and inputs it to the brain-type device 514 to execute inference processing (step S704). The brain-type device 514 executes inference processing on the inference image using the SNN generated by the processing shown in Fig. 2. The brain-type device 514 stores the results of the inference processing in the main storage device 512.
[0056] The processing device 511 reads the results of the inference process from the main memory device 512 and displays them on a monitor (not shown) connected to the brain-type computer 510 (step S705).
[0057] The processing device 511 determines whether to end the image recognition (step S706), for example, whether an instruction to end the image recognition process has been received from the administrator.
[0058] If the image recognition is to be continued, the processing device 511 returns to step S701 and executes the same processing. If the image recognition processing is to be ended, the processing device 511 ends the image recognition processing.
[0059] In this way, SNN processing can use a brain-type device with low power consumption, making it possible to realize a low-power image recognition device.
[0060] FIG. 8 is a diagram illustrating an example of a surveillance camera system according to the first embodiment.
[0061] The surveillance camera system 800 is composed of multiple cameras 520, a management device 810 that manages the cameras 520, and a network 820. The network 820 is composed of one or more network switches 830. The management device 810 is connected to the multiple cameras 520 via the network 820. The management device 810 is, for example, a general-purpose computer.
[0062] A brain-type device 514 (not shown) equipped with an SNN is connected to a processing device that controls the entire camera 520. If there is sufficient power to supply to the camera 520, a miniaturized brain-type computer 510 may be connected to the camera 520.
[0063] 8, there is no need to install a GPU in the management device 810 to execute image recognition processing, which reduces the power consumption of the system.
[0064] FIG. 9 is a diagram illustrating another example of the surveillance camera system according to the first embodiment.
[0065] In the surveillance camera system 800 shown in Fig. 8, the number of brain-type devices 514 that can be installed in the camera 520 or the brain-type computer 510 connected to the camera 520 depends on the power supply and is limited to one or two. Therefore, it may not be possible to implement an SNN that achieves the required recognition accuracy. In response to this, a system configuration such as that shown in Fig. 9 can be adopted.
[0066] In the surveillance camera system 900, a brain-type computer 510 is connected to multiple cameras 520 via a network 820.
[0067] Since there is no limit to the number of brain-type devices 514 that can be installed in the brain-type computer 510, it is possible to implement an SNN of the minimum necessary scale to achieve the required recognition accuracy.
[0068] The image recognition process executed by the surveillance camera system 900 shown in FIG. 9 consumes less power than image recognition processes using a GPU or the like, and therefore the power consumption of the system can be reduced.
[0069] As described above, according to the first embodiment, an SNN with high recognition accuracy can be generated. Furthermore, the SNN can be implemented using a predetermined number of brain-like devices 514 so as to achieve a predetermined recognition accuracy. This reduces the power consumption of the inference system, making it applicable to environments with limited power supply.
[0070] The present invention can be widely applied to systems that perform image recognition of people, etc., such as recognizing specific people in public places such as stations, airports, and streets, or recognizing specific actions of workers at work sites.
[0071] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments are provided to explain the present invention in detail, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, some of the configurations of each embodiment can be added to, deleted from, or replaced with other configurations.
[0072] Furthermore, some or all of the above-described configurations, functions, processing units, processing means, etc. may be implemented in hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that implements the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiments, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media for providing such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs, optical disks, magneto-optical disks, CD-Rs, magnetic tape, non-volatile memory cards, and ROMs.
[0073] Furthermore, the program code that realizes the functions described in this embodiment can be implemented in a wide range of programming or scripting languages, such as assembler, C / C++, perl, Shell, PHP, Python, and Java.
[0074] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed via a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read and execute the program code stored in the storage means or the storage medium.
[0075] In the above-described embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines are necessarily shown in the product. All components may be interconnected.
Claims
1. A method for generating a spiking neural network executed by a computer having a processing device and a memory connected to the processing device, comprising: a first step in which the processing device accepts first definition information defining a deep neural network including a plurality of connected convolution layers and stores the information in the memory; a second step in which the processing device accepts a condition that an activation rate of a neuron in the last convolution layer must satisfy and stores the information in the memory; a third step in which the processing device modifies the structure of the deep neural network so that an output of at least one of the convolution layers and an input to the deep neural network are combined; a fourth step in which the processing device executes a learning process using the deep neural network whose structure has been modified so as to satisfy the condition; and a fifth step in which the processing device converts the trained deep neural network to generate second definition information defining a spiking neural network and stores the information in the memory.
2. A method for generating a spiking neural network as described in claim 1, wherein the deep neural network defined by the first definition information includes at least one fully connected layer that processes the output of the final convolutional layer, and the third step includes a step in which the processing device changes the structure of the deep neural network so that the output of at least one of the convolutional layers other than the final convolutional layer is combined with the input to the deep neural network, and so that the output of the final convolutional layer is combined with the input to the deep neural network.
3. A method for generating a spiking neural network as described in claim 1, wherein the deep neural network defined by the first definition information includes at least one fully connected layer that processes the output of the last convolutional layer, and the third step includes a step in which the processing device changes the structure of the deep neural network so that the output of at least one of the convolutional layers other than the last convolutional layer is combined with the input to the deep neural network.
4. An image recognition apparatus comprising a memory device for storing images and a brain-type device for executing image recognition processing, wherein the brain-type device executes the image recognition processing on the images using the spiking neural network described in claim 1.
5. A computer system comprising: an edge device that acquires an image and executes image recognition processing using the image; and a management computer that acquires the results of the image recognition processing of the edge device, wherein the edge device executes the image recognition processing on the image using the spiking neural network described in claim 1.
Citation Information
Patent Citations
Abnormality detection system and method thereof
JP2022177892A
Spiking neural network configuration method, image recognition apparatus, and monitoring camera system
JP2023036463A
