Spiking neural network generation method, image recognition device, and computer system

By setting neuron activation rates and structurally modifying SNNs, the method maintains recognition accuracy while converting CNNs to SNNs, facilitating low-power image recognition.

JP2024165300AInactive Publication Date: 2024-11-28HITACHI LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023081386
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-17
Publication Date
2024-11-28
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Converting multilayer CNNs to SNNs results in significant deterioration of recognition accuracy due to input attenuation in convolutional layers, making them unsuitable for applications requiring high accuracy.

Method used

A method for generating SNNs by setting activation rates of neurons in the last convolutional layer and modifying the neural network structure to ensure recognition accuracy is maintained, using a brain-like computer to convert a trained DNN into an SNN.

Benefits of technology

Generates SNNs with preserved recognition accuracy, enabling low-power image recognition suitable for environments with limited power supply.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024165300000001_ABST
    Figure 2024165300000001_ABST
Patent Text Reader

Abstract

To generate an SNN (spiking neural network) with high recognition accuracy from a DNN (deep neural network).SOLUTION: A processing device receives first definition information defining a DNN including a plurality of connected convolution layers and a condition to be satisfied by an activation rate of neurons in a final convolution layer, modifies a structure of the DNN so that the output of at least one convolution layer and the input to the DNN are combined, executes a learning process using the DNN with the modified structure so as to satisfy the condition, and converts the trained DNN to generate second definition information defining an SNN.SELECTED DRAWING: Figure 4
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a method for generating spiking neural networks that mimic brain processing. [Background technology]

[0002] In recent years, the use of AI, especially deep learning, has progressed in various applications including image recognition and voice recognition, and social implementation is progressing in video surveillance, chatbots, etc. Examples of applications of deep learning include video surveillance, for example, detection and tracking of specific individuals such as suspicious or missing persons in public places such as stations, airports, and streets, and detection of specific behaviors of workers at work sites, such as dangerous behavior.

[0003] For image learning / inference, VGGnet (see, for example, Non-Patent Document 1) and Residual Network (ResNet) (see, for example, Non-Patent Document 2), which are types of convolutional neural networks (CNN), are used because they can improve the recognition accuracy of complex images. In order to achieve high recognition accuracy when there are dozens or more types of objects to be recognized appearing in an image, these neural networks are configured by connecting dozens to more than 100 convolutional layers. When performing image recognition using these CNNs, the computational processing load is large, so processing must be performed on a GPU (Graphics Processing Unit) server, which poses the problem of high power consumption.

[0004] In order to reduce power consumption, research is being conducted on a technique of converting CNN into a spiking neural network (SNN) that mimics brain processing and performs inference in a low-power brain-like device (see, for example, Non-Patent Document 3). [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] “Very deep convolutional networks for large-scale image recognition”, K. Simonyan and A. Zisserman, in ICLR, 2015. [Non-Patent Document 2] “Deep Residual Learning for Image Recognition”, Kaiming He, et. Al., CVPR, 2016, Volume 1, pp 770-78 [Non-Patent Document 3] “Benchmarking Keyword Spotting Efficiency on Neuromorphic Hardware”, P. Blouw, et al., in NICE '19, March 2019. Summary of the Invention [Problem to be solved by the invention]

[0006] The method of converting CNN to SNN has the advantage of being able to use CNNs that have a proven track record in applications such as image recognition, and is the main method of constructing SNNs. Conventional CNNs that have a proven track record in applications have many convolutional layers in the network, ranging from tens to over 100 layers. However, when a CNN with a large number of convolutional layers is converted to an SNN, there is an issue that the recognition accuracy of the SNN is significantly degraded compared to that of the CNN. The reason for the degradation in recognition accuracy is the attenuation of the input in the convolutional layers that make up the CNN. When many convolutional layers are stacked, the attenuation of the input in each convolutional layer causes the information of the original image (input) to be lost in the final stage, resulting in a degradation in recognition accuracy. As a result of the degradation in recognition accuracy in an SNN converted from a multi-layer CNN, it was not possible to use a multi-layer CNN that has a proven track record in applications in an SNN.

[0007] Therefore, an object of the present invention is to generate an SNN whose recognition accuracy does not deteriorate compared to the recognition accuracy of a CNN when converting a multi-layer CNN into an SNN. [Means for solving the problem]

[0008] A representative example of the invention disclosed in the present application is as follows: That is, a method for generating a spiking neural network executed by a computer having a processing device and a memory connected to the processing device includes a first step in which the processing device receives first definition information defining a deep neural network including a plurality of connected convolution layers and stores the information in the memory, a second step in which the processing device receives a condition that an activation rate of a neuron in the last convolution layer should satisfy and stores the information in the memory, a third step in which the processing device changes the structure of the deep neural network so that an output of at least one of the convolution layers and an input to the deep neural network are combined, a fourth step in which the processing device executes a learning process using the deep neural network whose structure has been changed so as to satisfy the condition, and a fifth step in which the processing device converts the learned deep neural network to generate second definition information defining a spiking neural network and stores the information in the memory. Effect of the Invention

[0009] According to the present invention, it is possible to generate an SNN in which the deterioration of recognition accuracy is suppressed. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a diagram showing the structural features of the SNN of the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of the structure of an SNN according to a first embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of the structure of an SNN according to a first embodiment. [Figure 4] 1 is a flowchart illustrating an example of a method for generating an SNN according to a first embodiment. [Diagram 5]FIG. 1 is a diagram showing an example of a brain-type computer according to a first embodiment. [Figure 6] FIG. 1 is a diagram showing an example of utilization of a brain-type computer that implements the SNN of the first embodiment. [Figure 7] 1 is a flowchart for explaining an example of image recognition processing executed by a brain-type computer in which the SNN of the first embodiment is implemented. [Figure 8] FIG. 1 is a diagram illustrating an example of a surveillance camera system according to a first embodiment. [Figure 9] FIG. 2 is a diagram illustrating another example of the surveillance camera system according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS EXAMPLES

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings, however, the present invention is not limited to the embodiment.

[0012] In Example 1, a method for generating an SNN with high accuracy in recognition processing using an image including a recognition target will be described. More specifically, in this embodiment, a method for realizing a system that satisfies the recognition accuracy required for a system that recognizes people, etc. in an image obtained from a surveillance camera, etc. will be described.

[0013] First, the structural features of the spiking neural network (SNN) of this embodiment will be described. Fig. 1 is a diagram showing the structural features of the SNN of embodiment 1. Fig. 1 shows a part of the SNN1, specifically, the input to the convolution layer, the output from the convolution layer, and the way of connections between the convolution layers.

[0014] The SNN1 is a spiking neural network that mimics brain processing for image recognition. The SNN1 has a structure in which multiple convolutional layers 10 are connected in series. A first input 50-1 to the SNN1 is input to the first convolutional layer 10-1. A first output 60-1 of the convolutional layer 10-1 is combined with a second input 50-2 that is a branch of the first input 50-1, and input to the convolutional layer 10-2 as a third input 50-3. A second output 60-2 of the convolutional layer 10-2 is combined with the second input 50-2, and input to the next convolutional layer 50 (not shown) as a fourth input 50-4.

[0015] The SNN1 is generated by converting a trained DNN. For the conversion, for example, LAVA, an open source software, or Nengo, an Applied Brain Research company, can be used.

[0016] Like SNN1, DNN has a structure in which multiple layers are connected in series.

[0017] FIG. 2 is a diagram illustrating an example of the structure of the SNN according to the first embodiment.

[0018] The SNN2 is composed of ten convolutional layers 10 and two fully connected layers 20. The multiple convolutional layers 10 are connected in series, and the fully connected layers 20-1 and 20-2 are connected in series in the next stage to the last convolutional layer 10-10.

[0019] The first input 50-1 to the SNN2 is input to the first convolutional layer 10-1. The first output 60-1 of the convolutional layer 10-1 is combined with the second input 50-2, which is a branch of the first input 50-1, and input to the convolutional layer 10-2 as the third input 50-3. The second output 60-2 of the convolutional layer 10-2 is combined with the second input 50-2 and input to the next convolutional layer 10 as the fourth input 50-4. The tenth output 60-10 of the last convolutional layer 10-10 is combined with the second input 50-2 and input to the fully connected layer 20-1 as the twelfth input 50-12. The fully connected layer 20-2 of the final stage outputs an output 70 as the final output of the SNN2.

[0020] FIG. 3 is a diagram illustrating an example of the structure of the SNN according to the first embodiment.

[0021] The SNN3 is composed of ten convolutional layers 11 and two fully connected layers 21. The multiple convolutional layers 11 are connected in series, and fully connected layers 21-1 and 21-2 are connected in series in the next stage to the last convolutional layer 11-10.

[0022] The first input 51-1 to the SNN3 is input to the first convolutional layer 11-1. The first output 61-1 of the convolutional layer 11-1 is combined with the second input 51-2 obtained by branching the first input 51-1, and is input to the convolutional layer 11-2 as the third input 51-3. The second output 61-2 of the convolutional layer 11-2 is combined with the second input 51-2, and is input to the next convolutional layer 11 as the fourth input 51-4. The ninth output 61-9 of the ninth convolutional layer 11-9 is combined with the second input 51-2, and is input to the last convolutional layer 11-10 as the eleventh input 51-11. The tenth output 61-10 of the last convolutional layer 11-10 is input to the fully connected layer 21-1. The fully connected layer 21-2 of the final stage outputs an output 71 as the final output of the SNN3.

[0023] The difference between SNN2 and SNN3 is the difference in the input to the first fully connected layers 20-1 and 21-1. Since the recognition accuracy of SNN2 and SNN3 varies depending on the recognition target, it is advisable to select an SNN that can achieve high recognition accuracy depending on the recognition target.

[0024] In SNN2, the outputs 60 of all convolutional layers 10 are combined with the second input 50-2, whereas in SNN3, the outputs 61 of the ninth convolutional layer 11-9, one layer before the final convolutional layer 11-10, are combined with the second input 51-2. It is possible to adjust how many outputs are combined with the second input, and it is advisable to select a configuration that provides the highest recognition accuracy.

[0025] Next, a method for generating an SNN will be described. Fig. 4 is a flowchart for explaining an example of a method for generating an SNN according to the first embodiment. Fig. 5 is a diagram showing an example of a brain-type computer according to the first embodiment.

[0026] To determine the configuration, a certain recognition accuracy is set based on the requirements of the system that performs image recognition.Furthermore, a certain neuron activation rate is set based on the requirements of the certain recognition accuracy.The SNN is generated based on these two conditions.

[0027] The processing described below is executed by a brain-type computer 510 as shown in Fig. 5. The brain-type computer 510 includes a processing unit 511, a main memory device 512, an auxiliary memory device 513, a brain-type device 514, and a chipset 515.

[0028] The processing unit 511 is, for example, a CPU, and controls the entire brain-type computer 510. The brain-type device 514 is a dedicated device for executing the SNN.

[0029] The main memory 512 is, for example, a volatile semiconductor memory, and temporarily stores software and various data loaded from the auxiliary memory 513. The processing device 511 executes the programs stored in the main memory 512, thereby realizing the functions of the brain-type computer 510. The auxiliary memory 513 is, for example, a large-capacity non-volatile storage device such as a hard disk drive or SSD (Solid State Drive). The auxiliary memory 513 stores software, various data used by the software, processing results of the brain-type device 514, and the like.

[0030] The main memory device 512 and the auxiliary memory device 513 may be collectively referred to as a memory device.

[0031] Although not shown in the figure, the brain-type computer 510 is connected to terminals such as a monitor and a remote terminal, allowing an administrator to give operational instructions to the brain-type computer 510 and display the image recognition results, etc.

[0032] The functions of the brain-type computer 510 (e.g., a processing unit) may be realized by using software, by hardware such as a dedicated circuit, or by a combination of software and hardware. One function of the brain-type computer 510 may be divided into multiple functions, or multiple functions may be integrated into one function. Some of the functions of the brain-type computer 510 may be provided as a separate function, or may be included in another function. Some of the functions of the brain-type computer 510 may be realized by another computer capable of communicating with the brain-type computer 510.

[0033] The brain-type computer 510 receives information on the definition of the DNN, the recognition accuracy determined based on the requirements of the image recognition system, the activity rate of the neurons in the final convolutional layer determined based on the recognition accuracy, and information on the convolutional layer that synthesizes the branched inputs (second inputs), and starts the processing described below.

[0034] When the processing device 511 starts the process of generating an SNN, it first sets the DNN (step S401).

[0035] Here, the number of convolutional layers to be connected and the number of fully connected layers to be connected are set. Also, as explained in Figures 2 and 3, the output from the final convolutional layer to the fully connected layer and the combination of the output of a specified convolutional layer and the second input are set. These settings are determined based on the characteristics of the image group, such as the type of image to be recognized.

[0036] Next, the processor 511 sets learning parameters (step S402).

[0037] For example, the batch size, the number of epochs, and the size and number of filters, which are weight matrices multiplied by the input to the neurons constituting the convolution layer, are set as learning parameters. In addition, parameters related to the activation rate of neurons are also set as learning parameters.

[0038] Next, the processing device 511 trains the DNN using the training teacher image under the set training parameters (step S403). The present invention is not limited to the training method of the DNN.

[0039] The processor 511 determines whether the learning is completed or not (step S404).

[0040] Specifically, the processing device 511 determines whether the recognition accuracy of the DNN is equal to or greater than a first threshold. If the recognition accuracy of the DNN is equal to or greater than the first threshold, the processing device 511 determines that the learning is completed.

[0041] If the learning is not completed, the processing device 511 determines whether the activation rate of the neuron in the final convolutional layer is equal to or greater than a second threshold (step S405).

[0042] If the activation rate is lower than the second threshold, the processing device 511 returns to step S402, changes the learning parameters, and executes the same process. If the activation rate is equal to or higher than the second threshold, the processing device 511 returns to step S401, changes the configuration of the DNN, and executes the same process. The setting of the DNN and the learning parameters may be performed again by the user, or may be automatically performed by the brain-type computer 510 based on the information initially set.

[0043] If it is determined in step S404 that the learning is completed, the processing device 511 converts the learned DNN into an SNN using a software tool (step S406), and then ends the process.

[0044] When performing inference, the processing device 511 uses a software tool to implement the SNN in the brain-type device 514 and performs inference using the brain-type device 514.

[0045] According to the above procedure, it is possible to generate an SNN that satisfies the recognition accuracy determined by the system requirements.

[0046] FIG. 6 is a diagram showing an example of utilization of a brain-type computer 510 that implements the SNN of the first embodiment.

[0047] 6 shows an image recognition device 500 consisting of a brain-type computer 510 and multiple cameras 520. The brain-type computer 510 is connected to the cameras 520 via a chipset 515. The cameras 520 capture images of the location where they are installed and transmit the image data in a specified format to the brain-type computer 510.

[0048] The processing device 511 executes software stored in the main memory device 512, extracts predetermined frame images from the video data input from the camera 520, inputs the frame images to the brain type device 514 to execute inference processing, and recognizes people and the like appearing in the images based on the processing results. The brain type device 514 processes the frame images using an SNN.

[0049] The video data transmitted from the camera 520, the frame images extracted from the video data, and the processing results by the brain-type device 514 are stored in at least one of the main memory device 512 and the auxiliary memory device 513.

[0050] FIG. 7 is a flowchart illustrating an example of image recognition processing executed by the brain-type computer 510 implementing the SNN of the first embodiment.

[0051] The processing device 511 acquires an image from the camera 520 and stores it in the main storage device 512 (step S701).

[0052] The processing device 511 extracts frame images from the video and extracts an inference part for each frame image (step S702). For example, a part including a recognition target such as a person is extracted.

[0053] The processing device 511 generates an inference image by converting the extracted inference part into a predetermined image size for input to the brain type device 514, and stores the image in the main storage device 512 (step S703). The inference image may be stored in the auxiliary storage device 513 for confirmation.

[0054] The processing device 511 reads out the image for inference from the main memory device 512 and inputs it to the brain type device 514 to execute inference processing (step S704). The brain type device 514 executes inference processing on the image for inference using the SNN generated by the processing shown in Fig. 2. The brain type device 514 stores the result of the inference processing in the main memory device 512.

[0055] The processing device 511 reads out the results of the inference processing from the main storage device 512, and displays them on a monitor (not shown) connected to the brain-type computer 510 (step S705).

[0056] The processor 511 determines whether or not to end the image recognition (step S706) For example, the processor 511 determines whether or not an instruction to end the image recognition process has been received from the administrator.

[0057] If the image recognition is to be continued, the processing device 511 returns to step S701 and executes the same process. If the image recognition process is to be ended, the processing device 511 ends the image recognition process.

[0058] In this way, SNN processing can be performed using low-power brain-type devices, making it possible to realize a low-power image recognition device.

[0059] FIG. 8 is a diagram illustrating an example of a surveillance camera system according to a first embodiment.

[0060] The surveillance camera system 800 is composed of multiple cameras 520, a management device 810 that manages the cameras 520, and a network 820. The network 820 is composed of one or more network switches 830. The management device 810 is connected to the multiple cameras 520 via the network 820. The management device 810 is, for example, a general-purpose computer.

[0061] A brain-type device 514 equipped with an SNN is connected to a processing device that controls the entire camera 520 (not shown). If there is sufficient power to be supplied to the camera 520, a miniaturized brain-type computer 510 may be connected to the camera 520.

[0062] 8, there is no need to mount a GPU on the management device 810 to execute image recognition processing, and therefore the power consumption of the system can be reduced.

[0063] FIG. 9 is a diagram illustrating another example of the surveillance camera system according to the first embodiment.

[0064] In the surveillance camera system 800 shown in Fig. 8, the number of brain-type devices 514 that can be mounted on the camera 520 or the brain-type computer 510 connected to the camera 520 is limited to one or two, depending on the power supply. Therefore, it may not be possible to implement an SNN that achieves the required recognition accuracy. To address this issue, a system configuration such as that shown in Fig. 9 can be adopted.

[0065] In the surveillance camera system 900, a brain-type computer 510 is connected to multiple cameras 520 via a network 820.

[0066] Since there is no limit to the number of brain-type devices 514 that can be installed in the brain-type computer 510, it is possible to implement an SNN of the minimum necessary scale to achieve the required recognition accuracy.

[0067] The image recognition process executed by the surveillance camera system 900 shown in FIG. 9 consumes less power than image recognition process using a GPU or the like, and therefore the power consumption of the system can be reduced.

[0068] As described above, according to the first embodiment, an SNN with high recognition accuracy can be generated. In addition, the SNN can be implemented using a predetermined number of brain-type devices 514 so as to satisfy a predetermined recognition accuracy. This can reduce the power consumption of the inference system, making it applicable to an environment with limited power supply.

[0069] The present invention can be widely applied to systems that perform image recognition of people, etc., such as recognizing a specific person in public places such as stations, airports, and streets, or recognizing a specific behavior of a worker at a workplace.

[0070] The present invention is not limited to the above-mentioned embodiment, but includes various modified examples. For example, the above-mentioned embodiment describes the configuration in detail to easily explain the present invention, and the present invention is not necessarily limited to the configuration including all the described configurations. Also, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0071] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that realizes the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-mentioned embodiments, and the program code itself and the storage medium storing it constitute the present invention. Examples of storage media for supplying such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs, optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc.

[0072] Furthermore, the program code for realizing the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, Python, Java (registered trademark), etc.

[0073] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed over a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read out and execute the program code stored in the storage means or storage medium.

[0074] In the above-mentioned embodiment, the control lines and information lines are shown as those considered necessary for the explanation, and not all the control lines and information lines are shown in the product. All the components may be connected to each other. [Explanation of symbols]

[0075] 1, 2, 3 Spiking Neural Networks 510 Brain-type computer 511 Processing equipment 512 Main storage 513 Auxiliary storage device 514 Brain Device 515 Chipset 520 Camera 800, 900 Surveillance Camera System 810 Management device 820 Network 830 Network Switch

Claims

1. 1. A method for generating a spiking neural network executed by a computer having a processing unit and a memory coupled to the processing unit, comprising: A first step in which the processing device accepts and stores in the memory first definition information defining a deep neural network including a plurality of concatenated convolutional layers; a second step in which the processing device receives a condition that the activation rate of the neurons of the final convolutional layer must satisfy and stores the condition in the memory; a third step in which the processing device modifies a structure of the deep neural network such that an output of at least one of the convolution layers and an input to the deep neural network are combined; A fourth step in which the processing device executes a learning process using the deep neural network whose structure has been changed so as to satisfy the condition; a fifth step in which the processing device converts the trained deep neural network to generate second definition information defining a spiking neural network and stores the second definition information in the memory; A method for generating a spiking neural network, comprising:

2. 2. A method for generating a spiking neural network according to claim 1, comprising: The deep neural network defined by the first definition information includes at least one fully connected layer that processes an output of the last convolutional layer, The third step is characterized in that the processing device changes the structure of the deep neural network so that an output of at least one of the convolutional layers other than the final convolutional layer is combined with an input to the deep neural network, and so that an output of the final convolutional layer is combined with an input to the deep neural network.

3. 2. A method for generating a spiking neural network according to claim 1, comprising: The deep neural network defined by the first definition information includes at least one fully connected layer that processes an output of the last convolutional layer, The third step includes a step in which the processing device modifies a structure of the deep neural network so that an output of at least one of the convolutional layers other than the final convolutional layer and an input to the deep neural network are combined.

4. An image recognition device comprising a storage device for storing images and a brain-type device for executing image recognition processing, 2. An image recognition apparatus, comprising: a brain-type device that performs the image recognition process using the spiking neural network according to claim 1 on the image.

5. 1. A computer system comprising: An edge device that acquires an image and executes image recognition processing using the image; a management computer that acquires a result of the image recognition process of the edge device, 2. A computer system comprising: an edge device that executes the image recognition process using the spiking neural network according to claim 1 on the image.