Pooling method in event-driven type chip, chip and electronic device

Abstract

The application discloses a pooling method in an event-driven type chip, a chip and electronic equipment. In order to solve the technical problem of high power consumption of the pooling operation in the event-driven type chip, in the application, a feature map contains a plurality of to-be-pooled targets, the to-be-pooled targets are a set of pulse events, and the plurality of to-be-pooled targets at least include a first to-be-pooled target; all pulse events contained in the first to-be-pooled target are projected to a first pulse neuron. The application takes the pulse events in the to-be-pooled target as the technical means and projects all the pulse events to the same neuron, thereby solving the technical problems of high power consumption of the pooling operation in the event-driven type chip, complex chip design and the like, and achieving the technical effect of low-power pooling. The application is suitable for the field of brain-like chips.

Description

Technical Field

[0001] This invention relates to a pooling method, a chip, and an electronic device, specifically to a pooling method, a chip, and an electronic device in an event-driven chip. Background Technology

[0002] Pooling, essentially sampling, is a form of compression of feature maps to speed up computation and reduce data processing. Common pooling methods include average pooling and max pooling. This is widely used not only in Artificial Neural Networks (ANNs) but also in the newer Spiking Neural Networks (SNNs). Regardless of whether it's an ANN or SNN, besides computer simulation (which consumes extremely high resources), efficient operation of these networks requires the design of corresponding ANN and SNN chips / processors.

[0003] Since ANN chips are typically based on conventional von Neumann architecture computing platforms, they can easily perform various numerical calculations. However, event-driven chips disrupt the traditional in-memory computing architecture and do not have the concept of computer programs. Therefore, designing such new chips requires redesigning the pooling scheme, especially the pooling circuit implementation, according to the characteristics of event-driven computing. Existing technology 1 is a max-pooling scheme in event-driven chips. Although its power consumption is quite low, it should be noted that the maximum value operation is a non-linear operation, which consumes relatively more resources and still has room for improvement.

[0004] Prior art 1: CN113673681B.

[0005] Multiplication and division are common in traditional computing platforms. However, one of the core purposes of SNN chips is to solve the "memory wall" problem caused by the separation of storage and computation in traditional computing platforms. They are AI chips that pursue extreme low power consumption. Multiplication and division are relatively resource-intensive computational operations, so performing multiplication or division in SNN chips is a luxury. Such computation would weaken the power consumption advantage of SNN chips, especially when such operations need to be widely used in the chip.

[0006] The purpose of this invention is to disclose a pooling method, chip, and electronic device applied to event-driven chips, which have the advantage of lower power consumption. Summary of the Invention

[0007] To solve or alleviate some or all of the above-mentioned technical problems, the present invention is achieved through the following technical solution: A pooling method in an event-driven chip, wherein a feature map contains a plurality of targets to be pooled, the targets to be pooled being a set of pulse events; the plurality of targets to be pooled includes at least a first target to be pooled; and all pulse events contained in the first target to be pooled are projected to a first spiking neuron.

[0008] In one embodiment, the target to be pooled comes from two or more spiking neurons / pixels.

[0009] In one embodiment, for each of the plurality of targets to be pooled, all of its pulse events are projected to the corresponding spiking neuron.

[0010] In one embodiment, all the spiking events contained in the first target to be pooled originate from different spiking neurons / pixels; for spiking events from the same source, the synaptic weights they rely on when projected onto the first spiking neuron are the same. For spiking events from different sources, the synaptic weights they rely on when projected onto the first spiking neuron are independent of each other.

[0011] In one embodiment, the event-driven chip contains several spiking neurons, including the first spiking neuron, forming a first pooling layer; several pooling targets contained in the feature map correspond one-to-one with several spiking neurons in the first pooling layer; all spiking events contained in each pooling target in the feature map are projected to the corresponding spiking neuron.

[0012] In one type of embodiment, the event-driven chip includes two or more pooling layers.

[0013] In one type of embodiment, the feature map is derived from an event camera or a convolution operation.

[0014] In one type of embodiment, the event-driven chip is an event camera or an SNN chip.

[0015] In one embodiment, at least a portion of the synaptic weights on the event-driven chip are obtained by quantizing the synaptic weights obtained by dividing the synaptic weights trained by the training device by the size of the target to be pooled.

[0016] In one embodiment, the event-driven chip contains several spiking neurons, including the first spiking neuron, forming a first pooling layer; the at least partial synaptic weights are the synaptic weights applied when projecting spiking events onto the spiking neurons in the first pooling layer.

[0017] In one embodiment, all pulse events contained in the first target to be pooled are weighted according to the first synaptic weight matrix and projected to the first spiking neuron; the synaptic weights in the first synaptic weight matrix are obtained by dividing the corresponding synaptic weights obtained by training by the training device by the size of the first target to be pooled.

[0018] A chip, which is an event-driven chip, includes a pooling layer composed of a plurality of spiking neurons; the plurality of spiking neurons includes a first spiking neuron; the chip applies a pooling method as described in any of the preceding event-driven chips.

[0019] In one type of embodiment, the chip is a neuromorphic chip.

[0020] An electronic device includes a chip as described in any of the preceding claims, the chip receiving input information from a sensor, performing inference operations on the input information through the chip to obtain an inference result, and the electronic device also making a corresponding response based on the inference result.

[0021] Some or all of the embodiments of the present invention have the following beneficial technical effects: 1) Low power consumption; pooling can be achieved without introducing complex operations such as multiplication / division into the chip. 2) The solution is simple and easy to implement, reducing the complexity of chip design.

[0022] Further beneficial effects will be described in the preferred embodiments.

[0023] The technical solutions / features disclosed above are intended to summarize the technical solutions and features described in the Detailed Embodiments section, and therefore the scope of the description may not be entirely the same. However, these new technical solutions disclosed in this section are also part of the numerous technical solutions disclosed in this invention document. The technical features disclosed in this section, together with the technical features disclosed in the subsequent Detailed Embodiments section and some contents in the drawings not explicitly described in the specification, disclose more technical solutions in a reasonable combination.

[0024] The technical solution formed by combining all the technical features disclosed at any position in this invention is used to support the summary of the technical solution, the modification of the patent document, and the disclosure of the technical solution. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the positional relationship of the feature maps; Figure 2 This is a schematic diagram of feature map pooling in this invention; Figure 3 This is a pooling diagram of the target to be pooled; Figure 4 This is a schematic diagram illustrating the deployment of synaptic weights onto a chip; Figure 5 This is a weighted diagram of the synaptic weight matrix during the pooling process; Figure 6 This is a diagram illustrating compression and quantization. Detailed Implementation

[0026] Since it is impossible to exhaustively describe all alternative solutions, the key points of the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Other technical solutions and details not disclosed in detail below generally belong to technical objectives or features that can be achieved by conventional means in the art, and due to space limitations, they will not be described in detail here.

[0027] Unless it refers to division, the " / " in any position in this invention represents logical "OR". The serial numbers "first", "second", etc., in any position in this invention are merely descriptive distinguishing marks and do not imply an absolute temporal or spatial order, nor do they imply that terms prefixed with such serial numbers necessarily refer to different things than the same terms prefixed with other modifiers.

[0028] This invention describes various key points used to combine into various specific embodiments, which will be incorporated into various methods and products. In this invention, even if a key point is described only when introducing a method / product solution, it means that the corresponding product / method solution also explicitly includes that technical feature.

[0029] The description of the existence or inclusion of a step, module, or feature at any location in this invention does not imply that such existence is exclusive or unique. Those skilled in the art can obtain other embodiments by supplementing the technical solutions disclosed in this invention with other technical means. The embodiments disclosed in this invention are generally for the purpose of disclosing preferred embodiments, but this does not imply that opposite embodiments of the preferred embodiments are excluded by this invention. As long as such opposite embodiments at least solve one of the technical problems of this invention, they are intended to be covered by this invention. Based on the key points described in the specific embodiments of this invention, those skilled in the art can substitute, delete, add, combine, or change the order of certain technical features to obtain a technical solution that still follows the concept of this invention. These solutions that do not depart from the technical concept of this invention are also within the protection scope of this invention.

[0030] Event-driven chips can be SNN processors (also called SNN chips) or event cameras. Event cameras, also known as dynamic vision sensors, have pixels that operate independently. When light changes exceed a preset range, a spike event is triggered, unlike traditional frame image sensors which use frames. Events can be transmitted within the chip via a routing system. During transmission, events typically record information such as the coordinates, polarity, and timestamp of their occurrence; however, this invention does not limit the specific content of this information. In this invention, an SNN processor / chip refers to a chip that includes dedicated circuitry for running the SNN network. The chip itself may also contain other modules, such as an x86 core.

[0031] The first spiking neuron, the first target to be pooled, etc., described in this invention can be regarded as a generalized description of several equivalent components. That is, when describing the first spiking neuron, the first target to be pooled, etc., all equivalent components can follow the same operating mode. However, the scheme also allows for some exceptions, allowing some non-equivalent components (seemingly equal in status, but not equal due to different operating modes) to perform different operating modes.

[0032] refer to Figure 1 The feature map matrix (also known as the event matrix, or simply feature map) is a set of events that express spatiotemporal information. It can be obtained from an event camera or from a convolutional layer in an SNN processor. This invention does not limit the specific source of the feature map.

[0033] refer to Figure 2 This diagram illustrates the pooling operation performed on feature map 10 according to the present invention. Feature map 10 includes multiple pooling targets 101, 201, 901, etc., each of which is a set of events. A pooling layer (first pooling layer) corresponds to this feature map, which includes a corresponding number (9 in the example) of spiking neurons 102, 202, 902 (referred to as neurons). Spiking events in pooling target 101 are projected to spiking neuron 102, spiking events in another pooling target 201 are projected to spiking neuron 202, spiking events in another pooling target 901 are projected to spiking neuron 902, and so on.

[0034] refer to Figure 3This diagram illustrates how spike events in a target 101 (the first target to be pooled) are pooled. For example, the target 101 is a 3×3 target, corresponding to (at least) 9 spiking neurons / pixels. Typically, these spikes do not occur simultaneously in the time domain, but are discretely distributed at different times. This differs significantly from pooling in traditional ANNs, where pooling requires all values ​​in the feature map (or its target to be pooled) to be simultaneously and completely acquired at the same time.

[0035] For example, within a certain time segment, four pulses are emitted at the coordinate (2, 3) of the target to be pooled in the corresponding feature map, two pulses are emitted at the coordinate (3, 2), one pulse is emitted at each of the other three locations, and no pulses are emitted at the remaining locations. When compressing the target to be pooled in the feature map, all pulses at the above nine locations are weighted according to the first synaptic weight matrix and projected to the spiking neuron 102 (the first spiking neuron). In other words, in this invention, the neuron 102 in the pooling layer receives all the pulse events in the corresponding target to be pooled. After a pulse is emitted at any location, it is directly delivered to the corresponding spiking neuron without any delay. This pooling method is extremely simple, without additional multiplication or division operations, making it extremely simple and convenient to design chips, and the chip has low power consumption, giving it a power advantage.

[0036] refer to Figure 4 This diagram illustrates the process from training the SNN to deploying it on an SNN chip. The SNN can be simulated and trained on a training device (such as a high-performance GPU) using methods based on surrogate gradients. Driven by sample data, various network configuration parameters of the SNN network (such as synaptic weights and / or time constants) are optimized. The optimized network configuration parameters are then deployed / mapped into the chip. The SNN network on the chip (including hardware-implemented spiking neurons and synapses) receives input data, performs inference, and outputs the inference results, enabling high-precision information processing such as keyword recognition, anomaly detection, and gesture recognition.

[0037] Normally, trained synaptic weights can be directly mapped into the chip. However, for chips using the pooling method described above, preferably: when training the SNN on the training device, average pooling is used; but before mapping, the trained synaptic weights are divided by the size of the target to be pooled before being deployed into the chip. For example, Figure 3If the size of the target to be pooled is 9, and the trained synaptic weights are assumed to be W, then the actual synaptic weights mapped to the chip are W / 9 (before quantization). Thus, the pooling scheme in the chip achieves average pooling without using division, and it does not require waiting for all the pulse events in the target to be pooled to be presented during pooling, i.e., it does not require designing a time slice window, thus avoiding excessive system latency.

[0038] refer to Figure 5 This diagram illustrates how pulse events contained in the target to be pooled 101 are projected onto spiking neurons 102 via synaptic weights in the kernel (also known as the synaptic weight matrix). The four pulses at coordinates (2, 3) are all projected onto spiking neurons 102 with a weighted synaptic weight of 0.1. A pulse at coordinates (2, 2) is projected onto spiking neurons 102 with a weighted synaptic weight of 0.8, and so on at other positions. The aforementioned synaptic weights W / 9 (which generally require quantization) actually deployed in the chip can be nine times the weight in the training device, i.e., W: each element in the former matrix is ​​1 / 9 of each element in the latter matrix. For example, the aforementioned synaptic weight value of 0.1 corresponds to a synaptic weight value of 0.9 in the training device. It should be noted that the numbers here are merely examples for ease of understanding and do not represent any real-world case. They are not intended to limit this application. In actual use, quantization may be required, such as quantizing 0.1 to 127.

[0039] refer to Figure 6 This diagram illustrates compression and quantization. When training the network, the training device uses average pooling to obtain synaptic weights W. However, when mapping to the chip, compressed synaptic weights W / 9 are used (this is just an example; the general formula requires dividing by the size of the target to be pooled). All synaptic values ​​have a distribution range, and clearly the latter is denser on the horizontal axis than the former. Preferably, the compressed synaptic weights W / 9 are quantized and deployed to the chip, thereby obtaining a corresponding spiking neural network that can be used for inference within the chip. The quantization essentially maps synaptic weights from one distribution range to another to adapt to the physical requirements of the chip. This invention does not limit the specific method of quantization.

[0040] Preferably, the synaptic weights in the first synaptic weight matrix are quantized by dividing the corresponding synaptic weights obtained through training by the training device by the size of the first target to be pooled. The same principle applies to other spiking neurons in the pooling layer.

[0041] Theoretically, the accumulation of synaptic current to the neuron's membrane voltage often involves multiplication, such as w*x, where w is the synaptic weight and x is the input. However, to eliminate this multiplication, x can be set to 1; and the neuron's equivalent resistance R is also set to 1, and the synaptic current I... syn The result of multiplying by R is I. syn The neuron's own membrane voltage V. Therefore, during the actual accumulation phase of membrane voltage, the neuron's membrane voltage V... mem The updated value can also be obtained through simple addition: V mem +I syn This allows for a more thorough elimination of multiplication and / or division operations. For example, the synaptic weights on the chip are 8 bits wide, while the membrane voltage is 16 bits wide.

[0042] Furthermore, this invention also discloses an electronic device comprising the aforementioned chip (such as an SNN chip, also known as a neuromorphic chip). This chip is equipped with an SNN network and, by receiving input information from sensors, performs inference on the input information with extremely low power consumption. For example, an event camera (sensor) and the chip can be incorporated into a toy (including a power module). If a face is detected, the chip outputs the inference result to a subsequent system, such as issuing a preset sound or waving the toy's robotic arm.

[0043] Although the invention has been described with reference to specific features and embodiments, various modifications, combinations, and substitutions can be made therein without departing from the invention. The scope of protection of this invention is not limited to the specific embodiments of processes, machines, manufactures, material compositions, apparatuses, methods, and steps described in the specification, and these methods and modules may also be implemented in one or more related, interdependent, cooperative, or upstream / downstream products or methods.

[0044] Therefore, the specification and drawings should be simply regarded as a description of some embodiments of the technical solutions defined by the appended claims, and thus the appended claims should be interpreted in accordance with the principle of the greatest reasonable interpretation, and are intended to cover as much as possible all modifications, variations, combinations or equivalents within the scope of the invention, while avoiding unreasonable interpretations.

[0045] To achieve better technical effects or for the needs of certain applications, those skilled in the art may make further improvements to the technical solution based on this invention. However, even if such improvements / designs are inventive and / or progressive, as long as they rely on the technical concept of this invention and cover the technical features defined in the claims, the technical solution should also fall within the protection scope of this invention.

[0046] The technical features mentioned in the appended claims may have alternative technical features, or the order of certain technical processes or material organization may be rearranged. Those skilled in the art, upon learning of this invention, will readily conceive of these alternative means, or alter the order of the technical processes or material organization, and then employ substantially the same means to solve substantially the same technical problems and achieve substantially the same technical effects. Therefore, even if the claims explicitly define the aforementioned means and / or order, these modifications, alterations, and substitutions should all fall within the scope of protection of the claims based on the principle of equivalents.

[0047] The method steps or modules described in the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application or design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered outside the scope of protection claimed by this invention.

Claims

1. A pooling method in an event-driven chip, characterized in that: The event-driven chip contains several spiking neurons, including a first spiking neuron. The feature map contains several targets to be pooled; The target to be pooled is a set of impulse events; The plurality of targets to be pooled includes at least a first target to be pooled; All pulse events contained in the first target to be pooled are projected to the first spiking neuron; as well as, At least some of the synaptic weights on the event-driven type chip are obtained by quantizing the synaptic weights obtained by training the synaptic weights through a training device and dividing them by the size of the target to be pooled, and then deploying the at least some of the synaptic weights into the event-driven type chip.

2. The pooling method in an event-driven chip according to claim 1, characterized in that: The event-driven chip contains several spiking neurons, including the first spiking neuron, which together form the first pooling layer. The feature map contains several targets to be pooled, which correspond one-to-one with several spiking neurons in the first pooling layer. All the pulse events contained in each target to be pooled in the feature map are projected to the corresponding spiking neuron.

3. The pooling method in an event-driven chip according to claim 1, characterized in that: All the spiking events contained in the first target to be pooled come from different spiking neurons or pixels; for spiking events from the same source, the synaptic weights they rely on are the same when projected onto the first spiking neuron.

4. The pooling method in an event-driven chip according to claim 1, characterized in that: The event-driven type chip is an event camera or an SNN chip.

5. The pooling method in an event-driven chip according to any one of claims 1-4, characterized in that: The event-driven chip contains several spiking neurons, including the first spiking neuron, which together form the first pooling layer. The at least partial synaptic weights are the synaptic weights applied when projecting spiking events onto spiking neurons in the first pooling layer.

6. The pooling method in an event-driven chip according to claim 5, characterized in that: For all the pulse events contained in the first target to be pooled, they are weighted according to the first synaptic weight matrix and projected to the first spiking neuron; The synaptic weights in the first synaptic weight matrix are obtained by quantizing the corresponding synaptic weights obtained through training by the training device by dividing the size of the first target to be pooled.

7. The pooling method in an event-driven chip according to any one of claims 1-4, 6, characterized in that: The target to be pooled comes from two or more spiking neurons / pixels.

8. The pooling method in an event-driven chip according to any one of claims 1-4, 6, characterized in that: For each of the several targets to be pooled, all of its pulse events are projected to the corresponding spiking neuron.

9. The pooling method in an event-driven chip according to any one of claims 1-4, 6, characterized in that: All the pulse events contained in the first target to be pooled come from different spiking neurons or pixels; for pulse events from different sources, the synaptic weights on which they depend when projected onto the first spiking neuron are independent of each other.

10. The pooling method in an event-driven chip according to any one of claims 1-4, 6, characterized in that: The event-driven type chip includes two or more pooling layers.

11. A chip, the chip being an event-driven chip, and the chip including a pooling layer composed of a plurality of spiking neurons; wherein the plurality of spiking neurons includes a first spiking neuron; characterized in that: The chip uses the pooling method in event-driven type chips as described in any one of claims 1-10.

12. The chip according to claim 11, wherein the chip is a neuromorphic chip.

13. An electronic device, comprising: The electronic device includes a chip as described in any one of claims 11-12, the chip receiving input information from a sensor, performing inference operations on the input information to obtain an inference result, and the electronic device also making a corresponding response based on the inference result.

Images