Processing system, processing method and processing program
The system addresses the lack of confidence calculation in SNNs by converting input data into spike train data and identifying invalid determinations for relearning, thereby improving recognition accuracy in SNN-based systems.
Patent Information
- Application Number
- JP2023078198
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-10
- Publication Date
- 2024-11-21
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Spiking neural networks (SNNs) cannot calculate confidence levels like deep neural networks (DNNs), making it impossible to automatically detect input data with low confidence for relearning and improving recognition accuracy.
A processing system that converts input data into spike train data using a spike converter and utilizes a spiking neural network (SNN) device to output inference results, accompanied by a determination unit to identify invalid determinations for relearning data storage.
Enables easy and appropriate identification of data for relearning, enhancing recognition accuracy in SNNs by incorporating relearning data to improve inference performance.
Smart Images

Figure 2024162569000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a processing system or the like that performs inference using machine learning. [Background technology]
[0002] Conventional surveillance camera systems detect abnormal or dangerous conditions by visually checking the captured video. On the other hand, recent surveillance camera systems have the ability to detect objects such as people and their movements that are worthy of attention in the video, and infer the attributes of those objects to detect abnormal conditions in real time, providing high functionality that does not require visual confirmation.
[0003] In general, neural networks, a type of technology known as machine learning, are used for inference. A neural network is constructed using a mathematical model of the neurons in the brain and the neural network that connects them. The mathematical model is calculated, for example, by a Graphic Processing Unit (GPU) for high-speed calculations.
[0004] In recent years, deep learning, which constructs a deep neural network (DNN) as one large network, has also been used to achieve more complex machine learning. Deep learning requires higher processing performance, making it difficult to achieve high-throughput processing with low power consumption.
[0005] To address this issue, a machine learning method using a brain-type device capable of low-power operation has been researched, which converts the input data into a spiking neural network (SNN) that mimics DNN and processes the data (see, for example, Non-Patent Document 1). Non-Patent Document 2 also discloses a brain-type device that uses spike trains as input data and processes the data with a brain-type array that fires only under specific conditions, consuming power only when the data fires, and that can achieve 2 to 48 times lower power consumption than a mathematical model method using a GPU. The output of this method is either 0 or 1, indicating whether or not the data fires, and there is no gradation from 0 to 1.
[0006] In general machine learning, learning data consisting of pairs of input data and data attributes is input, and a learning process (learning process) is executed using a GPU and DNN. By learning using a large amount of learning data in the learning process, it is possible to improve the inference accuracy when new data is input in the subsequent inference process.
[0007] Next, consider the social implementation of inference devices in surveillance camera systems and the like. As mentioned above, machine learning can be categorized into a pre-learning process and an inference process after social implementation. After social implementation, continuous improvement of recognition accuracy, such as adapting to environmental changes, is required. For this purpose, a technology called active learning is used. For example, Non-Patent Document 3 discloses that in addition to recognition accuracy, the certainty level is calculated, and input data with low certainty is used for re-learning, thereby improving recognition accuracy.
[0008] There are two main types of methods for calculating the confidence level: a technique for correcting the recognition accuracy at the time of prediction so that it directly becomes the confidence level, such as the Temperature Scaling method disclosed in Non-Patent Document 4, and a technique for calculating the confidence level from the distance between the inference data and the pre-trained data at the time of inference, such as the Trust Score method disclosed in Non-Patent Document 5. These confidence level calculations use arithmetic operations using a mathematical model. [Prior art documents] [Non-patent literature]
[0009] [Non-Patent Document 1] “Benchmarking Keyword Spotting Efficiency on Neuromorphic Hardware”, P. Blouw, et al., in NICE '19, March 2019. [Non-Patent Document 2] “Loihi A Neuromorphic Manycore Processor with On-Chip Learning ”, M. Davies, et. Al., in IEEE Micro 38, 82-99 (2018) [Non-Patent Document 3] “Confidence-Based Active Learning”, Mingkun Li, et. Al., IEEE PAMI 28, No.8, 1251-1261 (2006) [Non-Patent Document 4] “On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning”, Bolin Gao, et. Al., in arXiv preprint arXiv:1704.00805. (2017) [Non-Patent Document 5] “To Trust Or Not To Trust A Classifier”, Heinrich Jiang, et. Al., in NeurIPS2018. (2018) Summary of the Invention [Problem to be solved by the invention]
[0010] In the calculation of the inference result and the confidence level by the mathematical model using DNN, the mathematical model is calculated by arithmetic operation by software. On the other hand, the output of SNN is only whether it fires or not (outputs the value 1 or 0), so it is not possible to calculate the confidence level in the same way as the mathematical model using DNN.
[0011] Therefore, when using an SNN, it is not possible to automatically detect low-confidence input data after implementation in society, and it is not possible to learn the data to be re-trained. As a result, active learning is not possible, and it is not possible to improve the recognition accuracy of the SNN.
[0012] The present invention has been made in consideration of the above circumstances, and an object of the present invention is to provide a technique that can easily and appropriately grasp data that can be used for re-learning in inference using a spiking neural network. [Means for solving the problem]
[0013] In order to achieve the above object, a processing system according to one aspect is a processing system that performs inference processing based on input data, and includes: a spike conversion unit that converts the input data into spike train data; an SNN device that inputs the spike train data and outputs an inference result that estimates which of a plurality of classes the input data belongs to using a spiking neural network; and a judgment unit that judges whether the inference result is an indeterminate judgment that no class was inferred, or that multiple classes were inferred. Effect of the Invention
[0014] According to the present invention, in inference using a spiking neural network, data available for re-learning can be easily and appropriately grasped. [Brief description of the drawings]
[0015] [Figure 1] FIG. 1 is a diagram for explaining an overview of multi-class inference. [Diagram 2]FIG. 2 is a diagram for explaining an overview of multi-label inference. [Diagram 3] FIG. 3 is a functional configuration diagram of the surveillance camera system according to an embodiment. [Figure 4] FIG. 4 is a diagram illustrating an inference result according to an embodiment. [Diagram 5] FIG. 5 is a flowchart of a determination process according to an embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a hardware configuration of the surveillance camera system according to an embodiment. [Figure 7] FIG. 7 is a diagram illustrating another example of the hardware configuration of the surveillance camera system according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0016] The following embodiments will be described with reference to the drawings. Note that the following embodiments do not limit the scope of the invention, and not all of the elements and combinations thereof described in the embodiments are necessarily essential to the solution of the invention.
[0017] First, the inference content in the processing system according to this embodiment will be described.
[0018] FIG. 1 is a diagram for explaining an overview of multi-class inference.
[0019] Multi-class inference is an inference that classifies input data into one of multiple classes. Figure 1 shows an example of selecting one of three classes (a, b, c) as the class of the input data. In the example of Figure 1, the input data is classified into one class, b. In multi-class inference, a so-called one-hot output is expected, in which one class corresponding to the input data is output.
[0020] FIG. 2 is a diagram for explaining an overview of multi-label inference.
[0021] Multi-label inference is an inference that classifies input data into one of multiple classes for each of multiple categories. Figure 2 shows an example of classifying input data into one class from three classes (a, b, c) for the category indicating type, and into one class from three classes (blank, patterned, black) for the category indicating pattern. In the example of Figure 2, the type of the input data is classified into class b, and the pattern is classified into patterned. With multi-label inference, a so-called multi-hot output is expected, in which one class is selected for each category of the input data and output.
[0022] Next, a surveillance camera system will be described as an example of a processing system.
[0023] FIG. 3 is a functional configuration diagram of the surveillance camera system according to an embodiment.
[0024] The surveillance camera system 1 includes a learning process unit 10 that executes a process (learning process) for creating a trained SNN (spiking neural network) 140, and an inference process unit 20 that executes a process (inference process) for performing inference using the SNN.
[0025] The learning process unit 10 includes a learning dataset 100, a learning unit 110, a trained DNN (deep neural network) 120, a conversion unit 130, and a trained SNN 140.
[0026] The training dataset 100 is a dataset that stores training data for training a DNN, and includes multiple pieces of teacher data that are pairs of training data 101 to be input to the DNN for training and attribute data 102 that includes attributes (correct answers) corresponding to the training data 101.
[0027] The learning unit 110 receives training data from the learning dataset 100, performs machine learning on the DNN, and outputs a trained DNN 120.
[0028] The conversion unit 130 performs a process of converting the trained DNN 120 into a trained SNN 140 that outputs an inference result similar to that of the trained DNN 120. A known technique can be used to convert a DNN into an SNN. The conversion unit 130 also transmits the trained SNN 140 to the inference unit 220 to set the trained SNN 140.
[0029] Here, the SNN update unit is made up of the learning unit 110 and the conversion unit 130, and the setting unit is made up of the conversion unit 130.
[0030] The inference processor 20 includes actual data 200 , a spike converter 210 , an inference unit 220 , a decision unit 230 , a memory controller 240 , and re-learning data 250 .
[0031] The real data 200 is data to be inferred (real data: input data). The real data 200 is, for example, image data of a monitoring target area captured by a camera 320 (see FIG. 6).
[0032] Spike conversion unit 210 receives real data 200 and converts the received real data 200 into time-series spike train data for input to an SNN. Known techniques can be used to convert real data 200 into spike train data.
[0033] The inference unit 220 is set to the trained SNN 140 generated by the learning process unit 10, and performs inference processing on the input spike train data using the trained SNN, outputting an inference result 221. Here, the inference processing by the inference unit 220 is, for example, processing for estimating which of a plurality of classes the actual data 200 corresponding to the spike train data belongs to.
[0034] The judgment unit 230 judges whether the inference result 221 is valid or not, and outputs the judgment result. Here, the judgment unit 230 judges that the inference result 221 is valid when it indicates one class, and judges that it is invalid when the estimated class in the inference result 221 is indefinite, specifically, when it indicates multiple classes or when it indicates no class.
[0035] When the judgment result by the judgment unit 230 indicates that the inference result 221 is invalid, the storage control unit 240 stores the actual data corresponding to this inference result 221 as data to be used for re-learning (re-learned data 250). The storage control unit 240 may add the re-learned data 250 to the learning dataset 100 so that it is used in the subsequent learning process in the learning process unit 10. In this case, attribute data for the re-learned data may be set by the user. By executing the learning process using such re-learned data 250, the recognition accuracy of the trained SNN 140 can be improved.
[0036] Next, a specific example of the inference result 221 by the inference unit 220 will be described.
[0037] FIG. 4 is a diagram illustrating an inference result according to an embodiment.
[0038] FIG. 4 shows an example of a case where the inference unit 220 performs multi-class inference in which one class is selected from the three classes shown in FIG.
[0039] The inference unit 220 selects one of the three classes using the trained SNN, so the output (inference result 221) is 3-bit information in which each bit indicates whether or not the corresponding class. In this inference result 221, the bit corresponding to the selected class is set to 1, and the bit corresponding to the unselected class is set to 0.
[0040] Here, the inference results 221 can be classified into three types: One-hit, in which only one class is selected; Zero-hit, in which no class is selected; and Multi-hit, in which multiple classes are selected.
[0041] One-hit is an inference result 221 that is either "001", "010", or "100", and corresponds to a correct judgment in which the class was correctly determined, or an incorrect judgment in which the class was incorrectly determined. Note that since no attribute (correct answer) is assigned to the real data 200, it is not possible to determine from the inference result 221 whether the judgment is correct or incorrect.
[0042] Zero-hit corresponds to an inference result 221 of "000", in which no class is selected and no single class can be determined, resulting in an indeterminate judgment.
[0043] Multi-hit corresponds to an inference result 221 being any one of "011", "101", "110", and "111", where multiple classes are selected and it is not possible to determine one class, resulting in an indeterminate judgment.
[0044] Here, in the multi-class inference in the inference unit 220, it is expected that one class will be selected, and indeterminate judgments such as zero-hit and multi-hit are not valid judgments. In the multi-label inference shown in Fig. 2, the inference result 221 has the above-mentioned 3-bit information for each category.
[0045] Next, the processing operation of the determination process by the determination unit 230 will be described.
[0046] Fig. 5 is a flowchart of a determination process according to an embodiment of the present invention, in which the inference unit 220 performs inference (multi-class inference, multi-label inference) for one or more categories.
[0047] The judgment unit 230 performs category division for the inference result 221 (S10), and performs the following process on one of the unprocessed categories as the processing target. Here, a category is a group in which one class is selected from multiple classes, and for example, in the case of the multi-class inference shown in Fig. 1, there is one category, and in the case of the multi-label inference shown in Fig. 2, there are two categories, namely, type and pattern.
[0048] The judgment unit 230 judges whether the inference result for the category to be processed (current category) is a zero-hit (S11), and if it is a zero-hit (S11: Yes), outputs a judgment result that the inference result is invalid (indeterminate judgment).
[0049] On the other hand, if the current category is not a zero-hit (S11: No), the judgment unit 230 judges whether the inference result for the current category is multi-hot or not (S12), and if it is multi-hit (S12: Yes), it outputs a judgment result that the inference result is invalid (indeterminate judgment).
[0050] On the other hand, if the current category is not multi-hit (S12: No), the judgment unit 230 judges whether the current category is the final category or not (S13), and if it is the final category (S13: Yes), outputs a judgment result that the inference result is valid.
[0051] On the other hand, if it is not the final category (S13: No), the decision unit 230 sets the processing target to the next category (S14), and the process proceeds to step S11.
[0052] As described above, the surveillance camera system 1 can properly recognize real data 200 for which the inference result is invalid, i.e., real data for which the inference result is an indeterminate judgment, and can store this real data as re-learned data. Therefore, by adding this re-learned data to the training data set 100 and executing the training process again, a trained SNN 140 with higher inference accuracy can be generated, and by using this trained SNN 140 in the inference unit 220, the inference accuracy can be improved in the inference process.
[0053] Next, an example of a hardware configuration that constitutes the surveillance camera system 1 will be described.
[0054] FIG. 6 is a diagram illustrating an example of a hardware configuration of the surveillance camera system according to an embodiment.
[0055] The surveillance camera system 1A includes an inference device 310 and a camera 320.
[0056] The camera 320 captures images of the monitored area and inputs them to the inference device 310.
[0057] The inference device 310 is, for example, a computer such as a PC (Personal Computer) or a general-purpose server, and includes a processor 330, a chipset 340, a network interface 350, a main memory device 360, an SNN device 370, an auxiliary memory device 380, and a GPU 390.
[0058] The chipset 340 connects the processor 330, the network interface 350, the SNN device 370, the auxiliary storage device 380, the GPU 390, and the camera 320 so that they can communicate with one another.
[0059] The network interface 350 communicates with other devices over a network.
[0060] The processor 330 is, for example, a CPU (Central Processing Unit), and executes various processes according to programs stored in the main storage device 360 and / or the auxiliary storage device 380 .
[0061] The main memory device 360 is, for example, a RAM (RANDOM ACCESS MEMORY), and stores the programs executed by the processor 330 and other necessary information.
[0062] The auxiliary storage device 380 is, for example, a hard disk or flash memory, and stores the program (processing program) executed by the processor 330, data used by the processor 330, the DNN training dataset 100, the trained DNN 120, the trained SNN 140, the actual data 200 used for inference, the re-training data 250, etc.
[0063] The SNN device 370 is sometimes called a brain-type device, for example, and receives spike data as input, executes inference processing using an SNN, and outputs the inference results.
[0064] The GPU 390 is, for example, a processor suitable for executing a specific process, for example, suitable for executing processes performed in parallel. In this embodiment, the GPU 390 executes the process by the DNN according to the instruction of the processor 330.
[0065] Here, the learning unit 110 is mainly constituted by the processor 330 and the GPU 390 working together, the conversion unit 130, the spike conversion unit 210, the judgment unit 230, and the memory control unit 240 are mainly constituted by the processor 330 executing a processing program, and the inference unit 220 is mainly constituted by the SNN device 370.
[0066] Next, another example of the hardware configuration that constitutes the surveillance camera system 1 will be described.
[0067] FIG. 7 is a diagram illustrating another example of the hardware configuration of the surveillance camera system according to an embodiment.
[0068] The surveillance camera system 1B includes a camera 320, an inference device 410, and a learning device 510. The inference device 410 and the learning device 510 are connected via a network.
[0069] The inference device 410 is a device constituting the inference process unit 20, and is, for example, a computer such as a PC or a general-purpose server, and includes a processor 430, a chipset 440, a network interface 450, a main memory device 460, an SNN device 470, and an auxiliary memory device 480. Each component is similar to the components of the same name in the inference device 300 in FIG.
[0070] In this example, the spike conversion unit 210, the judgment unit 230, and the memory control unit 240 are mainly configured by the processor 430 executing a processing program, and the inference unit 220 is mainly configured by the SNN device 470. Here, the memory control unit 240 corresponds to the transmission unit, and transmits the re-learning data 250 to the learning device 510 via the network.
[0071] Learning device 510 is a device constituting learning process unit 10, and is, for example, a computer such as a PC or a general-purpose server, and includes a processor 530, a chipset 540, a network interface 550, a main memory device 560, an auxiliary memory device 580, and a GPU 590. Each component is similar to the components of the same name in inference device 300 in FIG.
[0072] In this example, the learning unit 110 is mainly configured by the processor 530 and the GPU 590 working together, and the conversion unit 130 is mainly configured by the processor 530 executing a processing program.
[0073] In the surveillance camera system 1B, the re-learning data 250 is transmitted from the inference device 410 to the learning device 510 via the network, and the trained SNN 140 is transmitted from the learning device 510 to the inference device 410 via the network and set.
[0074] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit of the present invention. For example, the present invention is not necessarily limited to an embodiment having all of the configurations described above.
[0075] For example, in the above embodiment, when the inference result is judged to be invalid, the storage control unit 240 stores the corresponding input data as re-learning data 250. However, the present invention is not limited to this. For example, all actual data 200 may be stored at all times, and when the inference result is judged to be invalid, timestamp data indicating the corresponding input data may be stored, and the input data to be used as re-learning data may be identified based on the timestamp. In short, it is sufficient to store the corresponding input data or data that can identify the input data (collectively referred to as input data identification information).
[0076] In the above embodiment, when the judgment result by the judgment unit 230 indicates that the inference result is invalid, the storage control unit 240 may display (output) actual data corresponding to this inference result and an alert indicating that the inference result is invalid on, for example, a display device (not shown) or another device. Here, the storage control unit 240 corresponds to an alert output unit.
[0077] In addition, in the above embodiment, a DNN is trained using the training dataset 100, and a trained SNN is generated by converting the trained DNN, but the method of generating a trained SNN is not limited to this, and an SNN may be generated or trained without converting from a DNN.
[0078] In the above-described embodiments, a part or all of the processing performed by the processor may be performed by a dedicated hardware circuit. The programs in the above-described embodiments may be installed from a program source. The program source may be a program distribution server or a recording medium (e.g., a portable recording medium). [Explanation of symbols]
[0079] Reference Signs List 1, 1A, 1B... Surveillance camera system, 10... Learning process, 20... Inference process, 110... Learning unit, 130... Conversion unit, 210... Spike conversion unit, 220... Inference unit, 230... Judgment unit, 240... Memory control unit, 310... Inference device, 320... Camera, 330... Processor, 340... Chipset, 350... Network interface, 360... Main memory device, 370... Device for SNN, 380... Auxiliary memory device, 390... GPU
Claims
1. A processing system that performs inference processing based on input data, a spike converter for converting input data into spike train data; an SNN device that receives the spike train data and outputs an inference result that estimates which of a plurality of classes the input data belongs to using a spiking neural network; A determination unit that determines whether the inference result is an indeterminate determination that no class has been inferred or a plurality of classes have been inferred; A processing system having:
2. The method further includes a storage control unit that stores input data identification information capable of identifying the input data on which the inference result is based when the inference result is determined to be the indefinite determination. The processing system of claim 1 .
3. The present invention further includes an SNN update unit that updates the spiking neural network using input data specified by the input data specification information. The processing system of claim 2 .
4. The SNN update unit is A learning unit that learns a deep neural network that estimates which of a plurality of classes the input data belongs to based on learning data including the input data identified by the input data identification information; A conversion unit that converts the trained deep neural network into a corresponding spiking neural network; A setting unit that sets the converted spiking neural network in the SNN device. The processing system of claim 3 .
5. The processing system includes: An inference device and a learning device connected via a network, The inference device comprises: The spike conversion unit, the SNN device, the determination unit, the memory control unit, and a transmission unit that transmits input data identified by the input data identification information to the learning device, The learning device includes: The SNN update unit is provided The processing system of claim 4 .
6. The device further includes an alert output unit that outputs an alert when the determination unit determines that the determination is indefinite. The processing system of claim 1 .
7. A processing method by a processing system which performs inference processing based on input data, converting the input data into spike train data; inputting the spike train data, and outputting an inference result obtained by estimating which of a plurality of classes the input data belongs to using a spiking neural network; Determine whether the inference result is an indeterminate determination that no class was inferred or multiple classes were inferred. Processing methods.
8. A processing program for causing a computer to execute an inference process based on input data, The computer includes: an SNN device that receives spike train data corresponding to the input data, and outputs an inference result obtained by estimating which of a plurality of classes the input data belongs to using a spiking neural network; The computer includes: converting the input data into spike train data; inputting the spike train data into the SNN device; determining whether the inference result is an indeterminate determination that no class was inferred or multiple classes were inferred; Processing program.
