NEURON OF A NEURAL NETWORK, ASSOCIATED DEVICE, SYSTEM, PLATFORM AND METHOD FOR ANALYSIS OF IMAGES

Abstract

Neuron of a neural network, associated device, system, platform and image analysis method The invention relates to an electronic circuit (36) physically implementing a neuron of a neural network, the electronic circuit (36) comprising: - a receiver (38) capable of receiving from a plurality of inputs real numbers corresponding to activations of the neuron sent by neurons of a neuron layer lower than the neuron layer comprising the neuron of the electronic circuit (36), - a processing unit (40) capable of applying a transfer function to the inputs to obtain a binary value, and - a transmitter (42) capable of sending the binary values obtained in the form of pulses. Figure for the abstract: Figure 2

Description

Title of the invention: Neuron of a neural network, associated device, system, platform and method for analyzing images

[0001] The present invention relates to an electronic circuit physically implementing a neuron of a neural network. The present invention also relates to a device, an embedded system and a platform comprising the circuit. The present invention also relates to an image analysis method using the device.

[0002] In the field of image analysis, one difficulty is analyzing a large volume of images in which many images are irrelevant. It is therefore necessary to be able to quickly eliminate a set of irrelevant images in order to examine only the most relevant images.

[0003] For this, it is known to use automatic image classification. Such classification is often carried out by a neural network and more particularly a convolutional neural network.

[0004] A convolutional neural network is also called a convolutional neural network.

[0005] The convolutional neural network is sometimes referred to as CNN, the abbreviation CNN referring to the English term “Convolutional Neural Network”.

[0006] However, physical implementations of convolutional neural networks involve significant use of hardware resources and therefore high energy consumption. As a result, such physical implementations are incompatible with embedded systems that require limited weight, size and energy power.

[0007] There is therefore a need for a physical implementation of a neural network or part of the neural network which is compatible with the constraints of use in an embedded system.

[0008] The description relates to an electronic circuit physically implementing a neuron of a neural network, the electronic circuit comprising a receiver capable of receiving from a plurality of inputs real numbers corresponding to activations of the neuron sent by neurons of a layer of neurons lower than the layer of neurons comprising the neuron of the electronic circuit, a processing unit capable of applying a transfer function to the inputs to obtain a binary value, and a transmitter capable of sending the binary values obtained in the form of pulses.

[0009] To achieve the development of such a neural network, the applicant studied the different physical implementations for selecting the impulse neuron as shown.

[0010] The impulse neuron is a bio-inspired neuron model that encodes information in the form of impulses. Such a model allows for drastically lowering the intensity of material resources and energy.

[0011] The use of impulse neurons also makes it possible to obtain a network of spiking neurons compatible with real-time processing and on-chip learning, including online and unsupervised.

[0012] The applicant then encountered the difficulty that the data provided as input to the network must be in pulse format to be usable by the pulse neural network.

[0013] However, the use of event cameras (more often referred to as DVS referring to the English term for "Dynamic Vision Sensor") is not completely satisfactory. In particular, the use of event cameras providing a continuous stream of events representing movement is not compatible with processing of still images, due to their event-based nature.

[0014] On the other hand, the use of conventional visual sensors involves the use of a module dedicated to transcoding the images obtained by the visual sensors. Such a module consumes energy and hardware resources, which makes the entire physical implementation of the neural network incompatible with embedded use.

[0015] Also, faced with this second problem, the applicant developed a neuron serving as an interface between a formal world (continuous variables) and an impulse world (binary variables). Such a neuron makes it possible to obtain a hybrid neural network architecture involving a classical (formal) part upstream and an impulse part downstream.

[0016] Such an architecture makes it possible to benefit from the aforementioned advantages of a pulse neural network while using a standard visual sensor and avoiding the specific transcoding module.

[0017] Furthermore, the hybrid architecture is more efficient than a pulse architecture in that the formal convolution stage is more efficient than the pulse convolution stage.

[0018] According to particular embodiments, the electronic circuit comprises one or more of the following characteristics, taken in isolation or in all technically possible combinations:

[0019] - the processing unit is capable of generating binary values at regular intervals.

[0020] - the transfer function includes the application of a sum weighted by weights, the weights being stored in the processing unit.

[0021] - the transfer function includes the application of a thresholding function.

[0022] - the electronic circuit is a programmable logic circuit.

[0023] The present description also relates to a device physically implementing a neural network comprising neurons, at least one neuron being an electronic circuit.

[0024] According to particular embodiments, the electronic circuit comprises one or more of the following characteristics, taken in isolation or in all technically possible combinations:

[0025] - the neural network is a convolutional neural network.

[0026] - the neural network comprises a formal convolution stage, a stage of neurons each physically implemented by an electronic circuit and an impulse classification stage.

[0027] Furthermore, the description also relates to an embedded system comprising a device as previously described.

[0028] The present description also relates to a platform comprising an embedded system as previously described.

[0029] Furthermore, the description also relates to a method of analyzing images comprising the use of a device as previously described.

[0030] Other characteristics and advantages of the invention will appear on reading the following description of embodiments of the invention, given by way of example only and with reference to the drawings which are: [0031 ] - [fig. 1 ] [fig. 1 ], a schematic representation of a platform comprising a electronic circuit,

[0032] - [fig.2][fig.2], a schematic representation of an example of an electronic circuit, And

[0033] - [fig.3][fig.3], a schematic representation of the operation of the example of electronic circuit of [fig.2].

[0034] A platform 10 is shown in [fig. 1].

[0035] The platform 10 is either a mobile platform, in particular a train, an aerial platform (airplane, helicopter or satellite), or a fixed installation (for example a building surveillance system).

[0036] The platform 10 comprises an embedded system 12 comprising a device 14 physically implementing a neural network.

[0037] The device 14 is an in situ programmable gate array. Such a device 14 is often referred to as an FPGA, which refers to the English term for “field-programmable gate array”.

[0038] The device 14 is a set of logic components configurable to physically implement a set of desired functions.

[0039] In the example described, the FPGA is configured according to a language description VHDL. The abbreviation VHDL stands for Very High Speed Integrated Circuits Hardware Description Language. VHDL is a language for describing a system's architecture at the RTL level. The abbreviation RTL stands for Register Transfer Level. Using a hardware synthesis tool, the VHDL description is interpreted and synthesized on the FPGA. The FPGA will then be configured to mimic the behavior described in the VHDL code. In addition, Verilog and OpenCL could also have been used to perform the hardware description of the system.

[0040] Alternatively, the device 14 is a programmable logic device (PLD) or programmable logic arrays (PLA).

[0041] According to yet another variant, the device 14 is an integrated circuit, in particular an ASIC circuit. The abbreviation ASIC refers to the English term for “Application Specific Integrated Circuit” which literally means “integrated circuit specific to an application”.

[0042] In the proposed example, the neural network is a convolutional neural network.

[0043] The neural network is a set of layers of neurons, each layer performing a function.

[0044] Thus, according to the proposed example, all the neurons of the same layer perform the same function (that of the layer) on incident data which are generally different data. The function operated by the layer also depends on the interconnection policy of the neurons of the layer considered with the neurons of the previous layer.

[0045] As visible in [fig.l], the device 14 comprises a convolution stage 16, an interfacing stage 18 and a classification stage 20.

[0046] In the example described, the convolution stage 16 is part of the formal domain while the classification stage 20 is part of the impulse domain.

[0047] The device 14 is hybrid in the sense that the device 14 operates in the two preceding domains since the convolution stage 16 is formal while the classification stage 20 is impulsive.

[0048] It should be noted that each stage 16, 18 or 20 corresponds, from the point of view of physical implementation, to an electronic circuit which is part of the device 14. It is the set of electronic circuits carrying out the functions of each stage 16, 18 or 20 which forms the device 14.

[0049] The convolution stage 16 is a stage implementing convolution and compression or “pooling” functions according to the usual terminology.

[0050] Each function is implemented by at least one respective layer.

[0051] According to the proposed example, the convolution stage 16 successively comprises a first convolution layer 22, a first pooling layer 24, a second convolution layer 26, a second pooling layer 28 and a third convolution layer 30.

[0052] As previously indicated, the convolution stage 16 is formal.

[0053] This means that the convolution stage 16 operates in the formal domain, that is, the neurons of the convolution stage 16 operate on inputs in the form of real numbers (continuous variables) to transform them into other real numbers. In other words, the neurons of the convolution stage 16 operate on real activations.

[0054] The convolution stage 16 is thus formed of formal neurons 32 organized according to layers 34.

[0055] For illustration purposes, two formal neurons 32 are represented in each layer 34, it being understood that the number of formal neurons 32 is most generally much larger and not necessarily the same between each layer 34.

[0056] A formal neuron 32 receives as input activations coming from neurons of a lower layer and weighting coefficients corresponding to the synaptic weight between the formal neuron 32 and the neurons having emitted the received activations.

[0057] The formal neuron 32 applies a first transfer function to the received activations to obtain an output activation.

[0058] According to the proposed example, the first transfer function is composed of a first activation function and a first weighted accumulation function.

[0059] For example, the first activation function is a hyperbolic tangent or a sigmoid.

[0060] With the first weighted accumulation function, the formal neuron 32 implements a weighting of the received activations and an integration of the activations thus weighted.

[0061] According to the proposed example, the first weighted integration function is a sum weighted by the synaptic weights between the formal neuron 32 and the neurons having emitted the received activations.

[0062] Mathematically, such an operation is written:

[0063] v (O - / . L • n w..V. (0 - j ' 7 J 1 \ *"* ; = 0 l] ■ i ' s !

[0064] Where:

[0065] y j1 is the output activation of formal neuron j of layer 1, with j and 1 two integers,

[0066] / j is the first activation function, and

[0067] wy the weight of the synaptic connection between neurons i of layer 1-1 and formal neurons j of layer 1, with i an integer.

[0068] It is noted that such a formal neuron 32 operates synchronously. This implies that all activations are modified simultaneously at regular intervals.

[0069] The interface stage 18 is a stage making it possible to create an interface between the convolution stage 16 and the classification stage 20.

[0070] More precisely, the interfacing stage 18 is a stage providing the interface between the formal domain and the impulse domain.

[0071] In this sense, the interface stage 18 is a hybrid stage.

[0072] As visible in [fig.l], the interface stage 18 comprises several “hybrid” neurons 36.

[0073] The “hybrid” interfacing stage 18 can be physically implemented by one or more electronic circuits 36 shown in [fig.2].

[0074] The case of an implementation by a single electronic circuit 36 corresponds to the case of a multiplexed implementation, whereas the case of an implementation by several electronic circuits 36 corresponds to the case of a parallel implementation. The case where the “hybrid” interfacing stage 18 is physically implemented by as many electronic circuits 36 as it has neurons corresponds to the case of a completely parallel implementation.

[0075] The electronic circuit 36 is a part of the device 14.

[0076] The electronic circuit 36 comprises a receiver 38, a processing unit 40 and a transmitter 42.

[0077] The receiver 38 is capable of receiving real values as input.

[0078] The real numbers correspond to activations sent by neurons of the neuron layer lower than the neuron layer comprising the hybrid neuron emulated by the electronic circuit 36.

[0079] In this case, this means that the real numbers come from the outputs of the convolution stage 16.

[0080] The processing unit 40 comprises a memory 44 and a calculation unit 46.

[0081] Memory 44 is suitable for storing values which are the synaptic weights relating to the incoming synapses of hybrid neuron 36.

[0082] The calculation unit 46 is capable of applying a second transfer function to the inputs received by the receiver 38 to obtain a binary value.

[0083] According to the proposed example, the second transfer function is composed of a second activation function and a second weighted accumulation function.

[0084] With the second weighted accumulation function, the calculation unit 46 performs a weighting and accumulation of the activations received by the receiver 38, by means of the synaptic weights stored in the memory 44. The result of the pon-

[0085]

[0086]

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096]

[0097]

[0098] deration-accumulation is called membrane potential. The computing unit 46 then applies the second activation function to the value of the membrane potential. The second activation function is a threshold function: when the membrane potential reaches a certain threshold 0, the transmitter 42 emits an output pulse, and the calculation unit 46 decrements the membrane potential by the value 0. The output of the neuron as generated by transmitter 42 is a binary signal whose value is 1 if a pulse is emitted, and zero otherwise. Mathematically, using the previous notations, the operation of such a hybrid neuron 36 is written: y = 1 sipl\ti) + E ,=0 a j J 0 otherwise In the previous equation, pl ( t - 1 ) corresponds to the initial value of the potential of J membrane before the weighted accumulation of continuous incoming activations 1(t). If the weighted accumulation of incoming activations causes the membrane potential to exceed the threshold 0, the binary output y1(t) goes to 1, and to 0 otherwise. In the same " .i time, if the membrane potential exceeds threshold 8, the membrane potential is decremented by 8; the membrane potential is unchanged in the opposite case. Alternatively, other functions can be used for the interface stage 18. The hybrid neuron 36 thus makes it possible to obtain a set of binary values as an output depending on the activations received as input. As explained previously, the outputs of the interfacing stage 18 are connected to at least one input of the classification stage 20. The classification stage 20 is a stage used to classify the data extracted by the convolution stage 16. The classification stage 20 is a fully connected stage, that is, each neuron 48 of the classification stage 20 is connected to the other neurons. Such a property is often called “fully connected” in reference to the corresponding English term. According to the proposed example, the classification stage 20 operates in the impulse domain, that is to say with binary variables. The classification stage 20 comprises, according to the example of [fig.2], impulse neurons 48 arranged in the form of several successive layers 50. A 48-spiking neuron receives pulses of constant amplitude and length as input. Such pulses are weighted by the weights of the synaptic connections and then accumulated. When the accumulated value exceeds a threshold 0, a pulse is output, and the accumulator is decremented by 0.

[0099] Mathematically, using the previous notations, the operation of such an impulse neuron 48 is written:

[0100] y! (t) =1 If 0 otherwise

[0101] In the previous equation, p1 ( t - 1 ) corresponds to the initial value of the potential of j membrane before weighting and integration of incoming activations y'*1 ( t ) binaries. If the weighting and integration of incoming activations leads the potential of membrane to exceed the threshold 0, the binary output y1 ( t ) goes to 1, and to 0 otherwise. In j At the same time, if the membrane potential exceeds the threshold 0, the membrane potential is decremented by 8; the membrane potential is unchanged in the opposite case.

[0102] The operation of the device 14 physically implementing the neural network is now described with reference to an image analysis method.

[0103] The analysis method comprises the reception of conventional images from a conventional image sensor (in the non-event sense).

[0104] The analysis method then comprises a step of processing the images using the neural network. This makes it possible to obtain as output a mapping of characteristics also referred to by the English term “feature maps”, the mapping associating the images with characteristics.

[0105] Convolution stage 16 implements the convolution and pooling functions.

[0106] Because the convolution stage 16 operates formally, the convolution and pooling functions are implemented efficiently.

[0107] The interface stage 18 makes it possible to convert the outputs of the convolution stage 16 in the formal domain into outputs in the impulse domain.

[0108] More precisely, in the example described, the operation of the interfacing stage 18 can be represented in functional form according to the schematic representation of [fig.3],

[0109] In this example, the hybrid neuron 36 comprises a first input 52, an internal memory 54, a memory interface 56, a neuron counter 58, an input counter 59, a comparator 60, a subtractor 62, a multiplication and accumulation unit 64 and two outputs 66 and 68.

[0110] The hybrid neuron 36 receives on the first input 52 continuous activations coming from a neuron of the last layer (layer L1 in the following) of the convolution stage 16.

[0111] More specifically, hybrid neuron 36 receives inputs from several neurons of layer L1 of convolution stage 16 successively.

[0112] The neuron counter 58 makes it possible to know the address of the neuron currently being processed (in the L layer).

[0113] The input counter 59 makes it possible to know the origin address of the activation received on the first input 52. The origin address is the address of the neuron from which the activation comes.

[0114] Both addresses are transmitted to memory interface 56.

[0115] Furthermore, the memory interface 56 recovers from the memory 44 the weight cor responding to the synaptic connection associated with the addresses.

[0116] The memory interface 56 transmits said weight to the multiplication and accumulation unit 64.

[0117] The multiplication and accumulation unit 64 then multiplies the activation weight (binary) received as input. The result of this multiplication is then accumulated in an accumulator of the multiplication and accumulation unit 64. The value of the accumulator corresponds to the membrane potential.

[0118] The multiplication and accumulation unit 64 transmits the value thus calculated to the comparator 60 which compares its value to the activation threshold of the hybrid neuron 36.

[0119] When the activation threshold is exceeded, the comparator 60 emits a pulse on the first output 66. The emitted pulse is represented by a binary signal.

[0120] Furthermore, also when the activation threshold is exceeded, the comparator 60 emits on the second output 68 the value of the neuron counter 58. The signal emitted on the second output 68 is the address of the neuron (of the L layer) currently being processed.

[0121] On the other hand, also when the activation threshold is exceeded, the comparator 60 transmits the membrane potential to the subtractor 62. The subtractor 62 decrements the membrane potential by the threshold value. A residual value of the membrane potential is thus obtained. The subtractor 62 then transmits the residual value of the membrane potential to the multiplication and accumulation unit 64. The accumulator of the multiplication and accumulation unit 64 thus takes the residual value of the membrane potential.

[0122] When no signal is emitted on one of the two outputs 66 or 68, the comparator 60 transmits the unchanged membrane potential to the multiplication and accumulation unit 64 without passing through the subtractor 62. The accumulator of the multiplication and accumulation unit 64 thus takes the unchanged value of the membrane potential.

[0123] The operation is repeated for each input coming from a neuron of the last layer of the LL convolution stage. Simultaneously, the input counter is incremented.

[0124] The signals obtained on outputs 66 and 68 are then sent to the input of the stage of classification 20.

[0125] The classification stage 20 then implements a classification.

[0126] Thus, a layer of neurons produced with the electronic circuit 36 is used to propagate the information in the network while operating the transcoding of the information from the formal domain to the impulse domain.

[0127] In this sense, the electronic circuit 36 is a hybrid neuron since the neuron has a formal input and a pulse output.

[0128] The use of such an electronic circuit 36 therefore makes it possible to integrate the transcoding into a layer of the neural network and not into a separate module which would be exclusively dedicated to the transcoding operation.

[0129] Thus, in the example presented, the electronic circuit 36 is economical in hardware resources and in calculation, which allows the use of the electronic circuit 36 in an embedded system 12.

[0130] Furthermore, the implementation of the interfacing stage 18 is multiplexed: a single physical neuron sequentially supports a set of logical neurons. Such an implementation saves hardware and energy resources, and adapts to the multiplexed implementation usually chosen for the convolution stage 16.

[0131] Such a hybrid neuron 36 thus constitutes a basic building block making it possible to obtain a physical implementation of a neural network which is compatible with the constraints of use in an embedded system 12.

Claims

Claims

1. Electronic circuit (36) physically implementing a neuron of a neural network, the electronic circuit (36) comprising: - a receiver (38) capable of receiving from a plurality of inputs real numbers corresponding to activations of the neuron sent by neurons of a layer of neurons lower than the layer of neurons comprising the neuron of the electronic circuit (36), - a processing unit (40) capable of applying a transfer function to the inputs to obtain a binary value, and - a transmitter (42) capable of sending the binary values obtained in the form of pulses.

2. Electronic circuit according to claim 1, in which the processing unit (40) is capable of generating binary values at regular intervals.

3. Electronic circuit according to claim 1 or 2, in which the transfer function comprises the application of a sum weighted by weights, the weights being stored in the processing unit (40).

4. Electronic circuit according to any one of claims 1 to 3, in which the transfer function comprises the application of a thresholding function.

5. An electronic circuit according to any one of claims 1 to 4, wherein the electronic circuit (36) is a programmable logic circuit.

6. Device (14) physically implementing a neural network comprising neurons, at least one neuron being an electronic circuit (36) according to any one of claims 1 to 5.

7. The device of claim 6, wherein the neural network is a convolutional neural network.

8. Device according to claim 6 or 7, in which the neural network comprises a formal convolution stage (16), a stage of neurons each physically implemented by an electronic circuit (36) according to any one of claims 1 to 5 and an impulse classification stage (20).

9. Embedded system (12) comprising a device (14) according to one any of claims 6 to 8.

10. Platform (10) comprising an on-board system (12) according to claim 9.

11. A method of analyzing images comprising using a device (14) according to any one of claims 6 to 8.

Images