Processing system, processing method, and processing program
The proposed processing system addresses the challenge of low confidence detection in SNNs by converting input data and assessing inference results, allowing for relearning and improving recognition accuracy in surveillance camera systems.
Patent Information
- Application Number
- PCT/JP2024/016207
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-10
- Filing Date
- 2024-04-25
- Publication Date
- 2024-11-14
AI Technical Summary
Spiking neural networks (SNNs) used in surveillance camera systems for real-time object detection and inference cannot automatically detect input data with low confidence levels, preventing active learning and subsequent improvement of recognition accuracy due to their binary output nature, which hinders the identification of data for relearning.
A processing system that includes a spike converter to convert input data into spike train data, an SNN device for inference, and a determination unit to assess the validity of the inference results, allowing for the identification of data that requires relearning by classifying input data into valid or indeterminate categories, thereby enabling the improvement of recognition accuracy.
Enables the easy and appropriate identification of data for relearning in SNNs, enhancing the inference accuracy by retraining data that initially produced indeterminate results, thus improving the overall recognition performance of the surveillance camera systems.
Smart Images

Figure JP2024016207_14112024_PF_FP_ABST
Abstract
Description
Processing system, processing method, and processing program
[0001] The present invention relates to a processing system or the like that performs inference using machine learning.
[0002] While conventional surveillance camera systems detect abnormal or dangerous conditions by visually checking the captured video, recent surveillance camera systems have the ability to detect objects of interest, such as people and their movements, infer the attributes of those objects, and detect abnormal conditions in real time, providing high functionality that eliminates the need for visual confirmation.
[0003] Common inference techniques use neural networks, a type of machine learning technology. A neural network is constructed using a mathematical model of the neurons in the brain and the neural network that connects them. The mathematical model is calculated using, for example, a GPU (Graphical Processing Unit) for high-speed calculations.
[0004] In recent years, deep learning, which builds a deep neural network (DNN) into a single large network, has been used to achieve more complex machine learning. However, deep learning requires higher processing performance, making it difficult to achieve high-throughput processing with low power consumption.
[0005] To address this issue, research has been conducted into machine learning methods that convert DNNs into spiking neural networks (SNNs) that mimic DNNs and process them using low-power brain-like devices (see, for example, Non-Patent Document 1). Non-Patent Document 2 discloses a brain-like device that uses spike trains as input data and processes them using a brain-like array that fires only under specific conditions, thereby consuming power only when the spikes fire. It is disclosed that this device can achieve 2 to 48 times lower power consumption compared to mathematical modeling methods using GPUs. The output of this method is either 0 or 1, depending on whether the spikes fire or not; there is no gradation between 0 and 1.
[0006] In general machine learning, training data consisting of pairs of input data and data attributes is input, and a learning process (learning process) is executed using a GPU and a DNN. By learning using a large amount of training data in the learning process, it is possible to improve the inference accuracy when new data is input in a subsequent inference process.
[0007] Next, let's consider the social implementation of inference devices in surveillance camera systems, etc. As mentioned above, machine learning can be divided into a pre-learning process and an inference process after social implementation. After social implementation, continuous improvement of recognition accuracy, such as adapting to environmental changes, is required. To address this, a technology called active learning is used. For example, Non-Patent Document 3 discloses that in addition to recognition accuracy, the certainty factor is calculated and input data with low certainty is subjected to re-learning, thereby improving recognition accuracy.
[0008] There are two main types of methods for calculating the confidence level: a technique for correcting the recognition accuracy at the time of prediction so that it directly becomes the confidence level, such as the Temperature Scaling method disclosed in Non-Patent Document 4, and a technique for calculating the confidence level from the distance between the inference data and the pre-trained data at the time of inference, such as the Trust Score method disclosed in Non-Patent Document 5. These confidence level calculations use arithmetic operations using mathematical models.
[0009] “Benchmarking Keyword Spotting Efficiency on Neuromorphic Hardware”, P. Blouw, et al., in NICE '19, March 2019. “Loihi A Neuromorphic Manycore Processor with On-Chip Learning ”, M. Davies, et. Al., in IEEE Micro 38, 82-99 (2018) “Confidence-Based Active Learning”, Mingkun Li, et. Al., IEEE PAMI 28, No.8, 1251-1261 (2006)“On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning”, Bolin Gao, et. Al., in arXiv preprint arXiv:1704.00805. (2017)“To Trust Or Not To Trust A Classifier”, Heinrich Jiang, et. Al., in NeurIPS2018. (2018)
[0010] In the calculation of the inference result and the confidence level based on the mathematical model using DNN, the mathematical model is calculated by performing arithmetic operations using software. On the other hand, the output of SNN is only whether it fires or not (outputting the value 1 or 0), so it is not possible to calculate the confidence level in the same way as the mathematical model using DNN.
[0011] Therefore, when using an SNN, it is not possible to automatically detect low-confidence input data after implementation in society, and it is not possible to acquire data to be re-learned. As a result, active learning is not possible, and the recognition accuracy of the SNN cannot be improved.
[0012] The present invention has been made in consideration of the above circumstances, and its purpose is to provide a technology that can easily and appropriately grasp data that can be used for relearning in inference using a spiking neural network.
[0013] In order to achieve the above object, a processing system according to one aspect is a processing system that performs inference processing based on input data, and includes: a spike conversion unit that converts the input data into spike train data; an SNN device that inputs the spike train data and outputs an inference result that estimates which of a plurality of classes the input data belongs to using a spiking neural network; and a judgment unit that judges whether the inference result is an indeterminate judgment that no class was estimated, or that a plurality of classes were estimated.
[0014] According to the present invention, in inference using a spiking neural network, data available for re-learning can be easily and appropriately grasped.
[0015] FIG. 1 is a diagram illustrating an overview of multi-class inference. FIG. 2 is a diagram illustrating an overview of multi-label inference. FIG. 3 is a functional configuration diagram of a surveillance camera system according to an embodiment. FIG. 4 is a diagram illustrating an inference result according to an embodiment. FIG. 5 is a flowchart of a determination process according to an embodiment. FIG. 6 is a diagram illustrating an example of the hardware configuration of a surveillance camera system according to an embodiment. FIG. 7 is a diagram illustrating another example of the hardware configuration of a surveillance camera system according to an embodiment.
[0016] The following description of the embodiments will be given with reference to the drawings. Note that the embodiments described below do not limit the scope of the invention as claimed, and not all of the elements and combinations thereof described in the embodiments are necessarily essential to the solution of the invention.
[0017] First, the inference content in the processing system according to this embodiment will be described.
[0018] FIG. 1 is a diagram for explaining an overview of multi-class inference.
[0019] Multi-class inference is an inference that classifies input data into one of multiple classes. Figure 1 shows an example in which one of three classes (a, b, c) is selected as the class of the input data. In the example of Figure 1, the input data is classified into one class, b. In multi-class inference, a so-called one-hot output is expected, in which one class corresponding to the input data is output.
[0020] FIG. 2 is a diagram for explaining an overview of multi-label inference.
[0021] Multi-label inference is an inference that classifies input data into one of multiple classes for each of multiple categories. FIG. 2 shows an example in which input data is classified into one of three classes (a, b, c) for the category indicating type, and into one of three classes (blank, patterned, black) for the category indicating pattern. In the example of FIG. 2, the input data is classified into class b for type, and into class b for pattern. Multi-label inference is expected to produce a so-called multi-hot output, in which one class is selected and output for each category of input data.
[0022] Next, a monitoring camera system will be described as an example of a processing system.
[0023] FIG. 3 is a functional configuration diagram of a surveillance camera system according to an embodiment.
[0024] The surveillance camera system 1 includes a learning process unit 10 that executes a process (learning process) for creating a trained SNN (spiking neural network) 140, and an inference process unit 20 that executes a process (inference process) for performing inference using the SNN.
[0025] The learning process unit 10 includes a learning dataset 100 , a learning unit 110 , a trained DNN (deep neural network) 120 , a conversion unit 130 , and a trained SNN 140 .
[0026] The training dataset 100 is a dataset that stores training data for training a DNN, and includes multiple sets of training data, each set consisting of training data 101 to be input to the DNN for training and attribute data 102 containing attributes (correct answers) corresponding to the training data 101.
[0027] The learning unit 110 receives training data from the training dataset 100, performs machine learning on the DNN, and outputs a trained DNN 120.
[0028] The conversion unit 130 performs a process of converting the trained DNN 120 into a trained SNN 140 that outputs an inference result similar to that of the trained DNN 120. Note that a known technique can be used to convert a DNN into an SNN. The conversion unit 130 also transmits the trained SNN 140 to the inference unit 220 to set the trained SNN 140.
[0029] Here, the SNN update unit is made up of the learning unit 110 and the conversion unit 130, and the setting unit is made up of the conversion unit 130.
[0030] The inference processor 20 includes actual data 200 , a spike converter 210 , an inference unit 220 , a judgment unit 230 , a memory control unit 240 , and re-learning data 250 .
[0031] The actual data 200 is data to be inferred (actual data: input data). The actual data 200 is, for example, image data of a monitoring target area captured by a camera 320 (see FIG. 6).
[0032] Spike conversion section 210 receives actual data 200 and converts the received actual data 200 into time-series spike train data to be input to the SNN. Known techniques can be used to convert actual data 200 into spike train data.
[0033] The inference unit 220 is configured with the trained SNN 140 generated by the learning process unit 10, and performs inference processing on input spike train data using the trained SNN, outputting an inference result 221. Here, the inference processing by the inference unit 220 is, for example, processing to estimate which of a plurality of classes the actual data 200 corresponding to the spike train data belongs to.
[0034] The determination unit 230 determines whether the inference result 221 is valid and outputs the determination result. Here, the determination unit 230 determines that the inference result 221 is valid if it indicates one class, and determines that the inference result 221 is invalid if the estimated class in the inference result 221 is indefinite, specifically, if it indicates multiple classes or does not indicate any class.
[0035] If the judgment result by the judgment unit 230 indicates that the inference result 221 is invalid, the storage control unit 240 stores the actual data corresponding to this inference result 221 as data to be used for re-learning (re-learning data 250). Note that the storage control unit 240 may add the re-learning data 250 to the training dataset 100 so that it may be used in a subsequent training process in the training process unit 10. In this case, attribute data for the re-learning data may be set by the user. By executing the training process using such re-learning data 250, the recognition accuracy of the trained SNN 140 can be improved.
[0036] Next, a specific example of the inference result 221 by the inference unit 220 will be described.
[0037] FIG. 4 is a diagram illustrating an inference result according to an embodiment.
[0038] FIG. 4 shows an example of a case where the inference unit 220 performs multi-class inference, selecting one class from the three classes shown in FIG.
[0039] The inference unit 220 selects one class from the three classes using the trained SNN, and the output (inference result 221) is three-bit information, each bit indicating whether or not the corresponding class is a certain class. In this inference result 221, the bit corresponding to the selected class is set to 1, and the bit corresponding to the unselected class is set to 0.
[0040] Here, the inference result 221 can be classified into three types: One-hit, in which only one class is selected; Zero-hit, in which no class is selected; and Multi-hit, in which multiple classes are selected.
[0041] One-hit means that the inference result 221 is either "001", "010", or "100", and corresponds to a correct determination where the class was correctly determined, or an incorrect determination where the class was incorrectly determined. Note that, since no attribute (correct answer) is assigned to the actual data 200, it is not possible to determine from the inference result 221 whether the determination is correct or incorrect.
[0042] A zero-hit corresponds to an indeterminate judgment in which the inference result 221 is "000", no class is selected, and one class cannot be determined.
[0043] Multi-hit corresponds to an indeterminate judgment in which the inference result 221 is one of "011", "101", "110", or "111", and multiple classes are selected, and it is not possible to determine one class.
[0044] Here, in the multi-class inference in the inference unit 220, it is expected that one class will be selected, and indeterminate judgments such as zero-hit and multi-hit are not valid judgments. Note that in the multi-label inference shown in Figure 2, the inference result 221 has the above-mentioned 3-bit information for each category.
[0045] Next, the processing operation of the determination process by the determination unit 230 will be described.
[0046] 5 is a flowchart of a determination process according to an embodiment of the present invention, which illustrates an example in which the inference unit 220 performs inference (multi-class inference, multi-label inference) for one or more categories.
[0047] The judgment unit 230 divides the inference result 221 into categories (S10), and performs subsequent processing on one of the unprocessed categories. Here, a category is a group in which one class is selected from multiple classes. For example, in the case of the multi-class inference shown in Figure 1, there is one category, and in the case of the multi-label inference shown in Figure 2, there are two categories: type and pattern.
[0048] The judgment unit 230 judges whether the inference result for the category to be processed (current category) is a zero-hit (S11), and if it is a zero-hit (S11: Yes), it outputs a judgment result that the inference result is invalid (an indeterminate judgment).
[0049] On the other hand, if the current category is not a zero-hit (S11: No), the judgment unit 230 judges whether the inference result for the current category is multi-hit or not (S12), and if it is multi-hit (S12: Yes), it outputs a judgment result that the inference result is invalid (an indeterminate judgment).
[0050] On the other hand, if the current category is not multi-hit (S12: No), the judgment unit 230 determines whether the current category is the final category (S13), and if it is the final category (S13: Yes), outputs a judgment result that the inference result is valid.
[0051] On the other hand, if the category is not the final category (S13: No), the determining unit 230 sets the processing target to the next category (S14), and the process proceeds to step S11.
[0052] As described above, the surveillance camera system 1 can appropriately recognize real data 200 for which the inference result is invalid, i.e., real data for which the inference result is determined to be indeterminate, and can store this real data as re-training data. Therefore, by adding this re-training data to the training dataset 100 and executing the training process again, it is possible to generate a trained SNN 140 with higher inference accuracy, and by using this trained SNN 140 in the inference unit 220, it is possible to improve the inference accuracy in the inference process.
[0053] Next, an example of the hardware configuration of the surveillance camera system 1 will be described.
[0054] FIG. 6 is a diagram illustrating an example of a hardware configuration of a surveillance camera system according to an embodiment.
[0055] The surveillance camera system 1A includes an inference device 310 and a camera 320.
[0056] The camera 320 captures images of the monitored area and inputs them to the inference device 310 .
[0057] The inference device 310 is, for example, a computer such as a PC (Personal Computer) or a general-purpose server, and includes a processor 330, a chipset 340, a network interface 350, a main memory device 360, an SNN device 370, an auxiliary memory device 380, and a GPU 390.
[0058] The chipset 340 connects the processor 330, the network interface 350, the SNN device 370, the auxiliary storage device 380, the GPU 390, and the camera 320 so that they can communicate with each other.
[0059] The network interface 350 communicates with other devices over a network.
[0060] The processor 330 is, for example, a CPU (Central Processing Unit), and executes various processes in accordance with programs stored in the main storage device 360 and / or the auxiliary storage device 380 .
[0061] The main memory device 360 is, for example, a RAM (random access memory), and stores the programs executed by the processor 330 and necessary information.
[0062] The auxiliary storage device 380 is, for example, a hard disk or flash memory, and stores the program (processing program) executed by the processor 330, data used by the processor 330, the DNN training dataset 100, the trained DNN 120, the trained SNN 140, the actual data 200 used for inference, the re-training data 250, etc.
[0063] The SNN device 370 is sometimes called a brain-type device, for example, and receives spike data as input, executes inference processing using an SNN, and outputs the result of the inference.
[0064] The GPU 390 is a processor suitable for executing specific processes, for example, processes that are performed in parallel. In this embodiment, the GPU 390 executes the DNN process in accordance with instructions from the processor 330.
[0065] Here, the learning unit 110 is mainly constituted by the processor 330 and the GPU 390 working together, the conversion unit 130, the spike conversion unit 210, the judgment unit 230, and the memory control unit 240 are mainly constituted by the processor 330 executing a processing program, and the inference unit 220 is mainly constituted by the SNN device 370.
[0066] Next, another example of the hardware configuration that constitutes the surveillance camera system 1 will be described.
[0067] FIG. 7 is a diagram illustrating another example of the hardware configuration of the surveillance camera system according to an embodiment.
[0068] The surveillance camera system 1B includes a camera 320, an inference device 410, and a learning device 510. The inference device 410 and the learning device 510 are connected via a network.
[0069] The inference device 410 is a device that constitutes the inference process unit 20, and is, for example, a computer such as a PC or a general-purpose server, and includes a processor 430, a chipset 440, a network interface 450, a main memory device 460, an SNN device 470, and an auxiliary memory device 480. Each component is similar to the components of the same name in the inference device 300 in FIG.
[0070] In this example, the spike conversion unit 210, the judgment unit 230, and the memory control unit 240 are mainly configured by the processor 430 executing processing programs, and the inference unit 220 is mainly configured by the SNN device 470. Here, the memory control unit 240 corresponds to the transmission unit, and transmits the re-learning data 250 to the learning device 510 via a network.
[0071] The learning device 510 is a device that constitutes the learning processor 10, and is, for example, a computer such as a PC or a general-purpose server, and includes a processor 530, a chipset 540, a network interface 550, a main memory device 560, an auxiliary memory device 580, and a GPU 590. Each component is similar to the components of the same name in the inference device 300 in FIG. 6 .
[0072] In this example, the learning unit 110 is mainly configured by the processor 530 and the GPU 590 working together, and the conversion unit 130 is mainly configured by the processor 530 executing a processing program.
[0073] In the surveillance camera system 1B, the re-learning data 250 is transmitted from the inference device 410 to the learning device 510 via the network, and the trained SNN 140 is transmitted from the learning device 510 to the inference device 410 via the network and set.
[0074] The present invention is not limited to the above-described embodiment, and can be appropriately modified and implemented without departing from the spirit of the present invention. For example, the present invention is not necessarily limited to an embodiment having all of the configurations described above.
[0075] For example, in the above embodiment, when the inference result is determined to be invalid, the storage control unit 240 stores the corresponding input data as re-learning data 250. However, the present invention is not limited to this. For example, all actual data 200 may be stored at all times, and when the inference result is determined to be invalid, timestamp data indicating the corresponding input data may be stored, and the input data to be used as re-learning data may be identified based on the timestamp. In short, it is sufficient to store the corresponding input data or data that can identify the input data (collectively referred to as input data identification information).
[0076] In the above embodiment, when the determination result by the determination unit 230 indicates that the inference result is invalid, the storage control unit 240 may be configured to display (output) the actual data corresponding to this inference result and an alert indicating that the inference result is invalid, for example, on a display device (not shown) or another device. Here, the storage control unit 240 corresponds to an alert output unit.
[0077] In addition, in the above embodiment, a DNN is trained using the training dataset 100, and a trained SNN is generated by converting the trained DNN, but the method of generating a trained SNN is not limited to this, and an SNN may be generated or trained without converting from a DNN.
[0078] In addition, in the above-described embodiments, some or all of the processing performed by the processor may be performed by a dedicated hardware circuit. Furthermore, the programs in the above-described embodiments may be installed from a program source. The program source may be a program distribution server or a recording medium (e.g., a portable recording medium).
[0079] 1, 1A, 1B... surveillance camera system, 10... learning process, 20... inference process, 110... learning unit, 130... conversion unit, 210... spike conversion unit, 220... inference unit, 230... judgment unit, 240... memory control unit, 310... inference device, 320... camera, 330... processor, 340... chipset, 350... network interface, 360... main memory device, 370... SNN device, 380... auxiliary memory device, 390... GPU
Claims
1. A processing system for performing inference processing based on input data, comprising: a spike conversion unit which converts the input data into spike train data; an SNN device which inputs the spike train data and outputs an inference result which infers which of a plurality of classes the input data belongs to using a spiking neural network; and a judgment unit which judges whether the inference result is an indeterminate judgment that no class was inferred, or that multiple classes were inferred.
2. The processing system according to claim 1, further comprising a memory control unit that stores input data identification information capable of identifying the input data on which the inference result is based when the inference result is determined to be an indeterminate judgment.
3. The processing system according to claim 2, further comprising an SNN update unit that updates the spiking neural network using input data specified by the input data specification information.
4. The processing system of claim 3, wherein the SNN update unit includes: a learning unit that learns a deep neural network that estimates which of a plurality of classes input data belongs to based on learning data including input data identified by the input data identification information; a conversion unit that converts the trained deep neural network into a corresponding spiking neural network; and a setting unit that sets the converted spiking neural network in the SNN device.
5. The processing system according to claim 4, further comprising an inference device and a learning device connected via a network, the inference device comprising the spike conversion unit, the SNN device, the judgment unit, the memory control unit, and a transmission unit that transmits input data identified by the input data identification information to the learning device, and the learning device comprising the SNN update unit.
6. The processing system according to claim 1, further comprising an alert output unit that outputs an alert when the determination unit determines that the determination is indeterminate.
7. A processing method by a processing system that performs inference processing based on input data, comprising the steps of: converting the input data into spike train data; inputting the spike train data; outputting an inference result that infers which of a plurality of classes the input data belongs to using a spiking neural network; and determining whether the inference result is an indeterminate judgment that no class was inferred or that a plurality of classes were inferred.
8. A processing program to be executed by a computer performing inference processing based on input data, the computer comprising an SNN device that inputs spike train data corresponding to the input data and outputs an inference result that infers which of a plurality of classes the input data belongs to using a spiking neural network, the processing program causing the computer to convert the input data into spike train data, input the spike train data to the SNN device, and determine whether the inference result is an indeterminate judgment that no class was inferred or that a plurality of classes were inferred.
Citation Information
Patent Citations
Method of converting neural network and recognition apparatus using the same
US20180121802A1
Feedback signaling to facilitate data classification functionality of a spiking neural network
WO2019125421A1
