Multimodal neuronal structures
By using a multimodal neuron structure, the problem that existing neurons can only receive a single pulse code form is solved, realizing flexible modular design and efficient pulse sensor adaptation, thereby improving the adaptability and hardware efficiency of the neural network.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INST OF SEMICONDUCTORS - CHINESE ACAD OF SCI
- Filing Date
- 2024-04-24
- Publication Date
- 2026-05-15
AI Technical Summary
Existing neurons can only receive a specific pulse code, which limits the flexibility of modular design.
Employing a multimodal neuron structure, including a reference voltage generation module and a pulse generation module, it controls the on/off state of different charge packet counter sub-circuits through a mode signal input circuit. It can receive frequency-coded pulse sequences or time-coded pulse sequences and transmit target pulses by generating different reference voltages.
A modular design of the neuron structure was achieved, which can be adapted to various types of pulse sensors, improving flexibility and adaptability, and reducing the complexity and resource consumption of hardware design.
Smart Images
Figure CN118468949B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial neural network technology, and in particular to a multimodal neuron structure. Background Technology
[0002] Leveraging biomimetic technology, the semiconductor industry has made significant progress in silicon-based imaging devices and image processing. Silicon-based imaging chips have expanded the imaging environment beyond visible light to the infrared and ultraviolet bands. Meanwhile, the imitation of the brain's neurobiological systems has brought us an important structure in artificial intelligence—the spike neural network (SNN). SNNs use short pulses to encode and transmit information.
[0003] Spike coding techniques used in spike neural networks can be broadly categorized into two types: rate coding and temporal coding. All specialized coding schemes can be classified into these two types by answering the question of whether the exact timing and order of spikes are crucial to the information being submitted. However, rate coding is too slow to account for rapid responses in biology; while temporal coding has proven to have higher information capacity, faster reaction times, and higher transmission speeds, it involves more complex architectures and lacks practical training methods. Image processing systems built using spike neural networks typically employ only a single spike coding method in their hardware implementation. This is hardware-friendly, but it limits the neural network to receiving image information at a single scale. For neural network systems trained using prior information and backpropagation, the self-evolutionary nature endows them with the ability to process information at different scales; a single coding method restricts the adaptive scale advantage of using neural networks for image processing.
[0004] Therefore, solving the problem that existing neurons can only receive a specific pulse code form, which limits modular design, has become an urgent task. Summary of the Invention
[0005] In view of the above problems, embodiments of this disclosure provide a multimodal neuron structure to solve the problem that existing neurons can only receive a specific pulse code form, which limits modular design.
[0006] This disclosure provides a multimodal neuron structure, including: a reference voltage generation module, comprising a first reference voltage generation module and a second reference voltage generation module, wherein the first reference voltage generation module generates a first reference voltage and the second reference voltage generation module generates a second reference voltage; and a pulse generation module connected to the reference voltage generation module, wherein the input terminal of the pulse generation module is used to receive a frequency-coded pulse sequence or a time-coded pulse sequence, and the pulse generation module includes a first charge packet counter subcircuit, a second charge packet counter subcircuit, and a mode signal input circuit; the first charge packet counter subcircuit is used to receive the frequency-coded pulse sequence, and when the voltage corresponding to the accumulated charge of the frequency-coded pulse sequence reaches the first reference voltage, a target pulse is emitted; the second charge packet counter subcircuit is used to receive the time-coded pulse sequence, and when the voltage corresponding to the accumulated charge of the time-coded pulse sequence reaches the second reference voltage, a target pulse is emitted; the mode signal input circuit is used to input a first mode signal when receiving the frequency-coded pulse sequence to control the first charge packet counter subcircuit to conduct, and to input a second mode signal when receiving the time-coded pulse sequence to control the second charge packet counter subcircuit to conduct.
[0007] Optionally, the multimodal neuron structure includes: a first transistor located at the input port of the first charge packet counter sub-circuit, wherein when the frequency-coded pulse sequence is received, the first transistor controls the first charge packet counter sub-circuit to conduct according to the first mode signal; and a second transistor located at the input port of the second charge packet counter sub-circuit, wherein when the time-coded pulse sequence is received, the second transistor controls the second charge packet counter sub-circuit to conduct according to the second mode signal.
[0008] Optionally, the first charge packet counter sub-circuit includes: a first integrating capacitor, which accumulates the charge input by the frequency pulse sequence when the first charge packet counter sub-circuit is in operation; and a third transistor, connected to the first integrating capacitor, which is used to turn on the third transistor when the first charge packet counter sub-circuit emits a target pulse, by inputting a first reset signal to the third transistor, thereby clearing the charge on the first integrating capacitor.
[0009] Optionally, the second charge packet counter sub-circuit includes: a second integrating capacitor containing a preset amount of charge, one end of which is grounded; when the second charge packet counter sub-circuit starts to conduct, the preset amount of charge gradually decreases over time; when the second integrating capacitor receives the time-coded pulse sequence, the preset amount of charge stops decreasing; when the voltage value corresponding to the remaining charge on the second integrating capacitor reaches the second reference voltage, a target pulse is emitted; and a fourth transistor connected to the second integrating capacitor, used to turn on the fourth transistor when the pulse generation module emits the target pulse, by inputting a second reset signal to the fourth transistor, thereby clearing the charge on the second integrating capacitor.
[0010] Optionally, the mode signal input circuit includes a NAND gate and a fifth transistor. The first input terminal of the NAND gate is connected to the transmitting terminal of the pulse generation module that transmits the target pulse. The second input terminal is used to input a first mode signal or a second mode signal. The output terminal is connected to the fifth transistor, and the fifth transistor is connected to the first charge packet counter sub-circuit. When the NAND gate inputs the first mode signal, it controls the first charge packet counter sub-circuit to conduct. When the NAND gate inputs the second mode signal, it controls the second charge packet counter sub-circuit to conduct.
[0011] Optionally, the pulse generation module further includes: a comparator, whose first input terminal is connected to the first charge packet counter sub-circuit, whose second input terminal is connected to the second charge packet counter sub-circuit, and whose third input terminal is connected to the reference voltage generation module; when the first charge packet counter sub-circuit is turned on, the comparator compares the voltage corresponding to the charge accumulated in the frequency-coded pulse sequence with the first reference voltage; when the voltage corresponding to the charge accumulated in the frequency-coded pulse sequence reaches the first reference voltage, the comparator output terminal emits a target pulse; when the second charge packet counter sub-circuit is turned on, the comparator compares the voltage corresponding to the charge accumulated in the time-coded pulse sequence with the second reference voltage; when the voltage corresponding to the charge accumulated in the time-coded pulse sequence reaches the second reference voltage, the comparator output terminal emits a target pulse.
[0012] Optionally, the first reference voltage is a constant DC voltage.
[0013] Optionally, the second reference voltage generation module includes: a first variable resistor, a second variable resistor, a third variable resistor, and a fourth variable resistor, wherein a first terminal of the first variable resistor, a first terminal of the second variable resistor, a first terminal of the third variable resistor, and a first terminal of the fourth variable resistor are connected; a first transistor switch, a second transistor switch, a third transistor switch, a fourth transistor switch, a fifth transistor switch, a sixth transistor switch, and a seventh transistor switch, wherein a first terminal of the first transistor switch is connected to a second terminal of the first variable resistor, a first terminal of the second transistor switch is connected to a second terminal of the second variable resistor, and a second terminal of the second transistor switch is connected to a first terminal of the third transistor switch. The third terminal is connected to the third terminal of the first transistor switch; the fourth transistor switch is connected to the second terminal of the third variable resistor; the first terminal of the fifth transistor switch is connected to the second terminal of the fourth transistor switch; the first terminal of the sixth transistor switch is connected to the second terminal of the fourth variable resistor; and the first terminal of the seventh transistor switch is connected to the second terminal of the sixth transistor switch. A DC current source has its second terminal connected to both the third and third terminals of the first transistor switch. A capacitor has its first terminal connected to the second terminal of the third transistor switch, and its second terminal grounded. An adder has its input terminal connected to the second terminals of the third, fifth, and seventh transistor switches, and its output terminal connected to the pulse generation module.
[0014] Optionally, the second reference voltage is a piecewise linear ramp voltage.
[0015] Optionally, the first reference voltage generation module is turned on when the pulse generation module receives the frequency-coded pulse sequence, and the second reference voltage generation module is turned on when the pulse generation module receives the time-coded pulse sequence.
[0016] The above-described at least one technical solution adopted in the embodiments of this disclosure can achieve the following beneficial effects:
[0017] 1. This disclosure provides a multimodal neuron structure comprising a reference voltage generation module and a pulse generation module. Different mode signals are input through a mode signal input circuit to control the on / off state of a first charge packet counter subcircuit and a second charge packet counter subcircuit. The pulse generation module can receive encoded pulse sequences or time-coded pulse sequences. The reference voltage generation module generates a first reference voltage or a second reference voltage based on the time-coded pulse sequence received by the pulse generation module, in order to transmit a target pulse. This multimodal neuron structure adopts a modular design, enabling it to receive different types of pulse codes and adapt to various types of pulse sensors, eliminating the need to determine the pulse coding method during the initial system design.
[0018] 2. This disclosure uses multiple variable resistors and multiple transistor arrays to generate a piecewise linear ramp as the reference voltage of the comparator, which has high configurability of the neuron cell membrane threshold voltage and high linearity of input and output.
[0019] 3. The circuit structure disclosed herein is simple, occupies a small area, and has high reusability, making it suitable for constructing spiking neural network models of various sizes. Attached Figure Description
[0020] To gain a more complete understanding of this disclosure and its advantages, reference will now be made to the following description taken in conjunction with the accompanying drawings, in which:
[0021] Figure 1 This illustration schematically shows a neuronal structure of a conventional received frequency encoded pulse sequence provided in an embodiment of the present disclosure;
[0022] Figure 2 This illustration schematically shows a neuronal structure of a conventional received time-coded pulse sequence provided in an embodiment of the present disclosure;
[0023] Figure 3 This illustration schematically shows a multimodal neuron structure provided in an embodiment of the present disclosure;
[0024] Figure 4 A schematic diagram of a pulse generation module provided in an embodiment of this disclosure is shown.
[0025] Figure 5 A schematic diagram of a second reference voltage module provided in an embodiment of this disclosure is shown.
[0026] Figure 6 The illustration schematically shows a diagram of a common ramp wave as a reference voltage provided in an embodiment of the present disclosure;
[0027] Figure 7 This illustration schematically shows the linearity of a common ramp reference voltage and an input time-coded pulse sequence provided in an embodiment of the present disclosure;
[0028] Figure 8 The illustration schematically shows a diagram of a piecewise linear ramp wave as a reference voltage provided in an embodiment of the present disclosure;
[0029] Figure 9 The diagram illustrates the linearity of the piecewise linear ramp reference voltage and the input time-coded pulse sequence provided in the embodiments of this disclosure. Detailed Implementation
[0030] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.
[0031] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.
[0032] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.
[0033] Spike neural networks consist of neurons that describe action potential generation as a non-differential equation in mathematics, thereby approximating the observed behavior of biological systems. Spike encoding techniques used in spike neural networks can be broadly categorized into two types: rate encoding and temporal encoding. Rate encoding embeds information into the instantaneous or average spike generation rate of a single neuron or a group of neurons, producing a value describing neuronal activity that is identical to the activation value of a typical non-spike artificial neuron. Temporal encoding methods encode information using precise time intervals between spikes, including absolute time relative to a global reference, relative time intervals between spikes emitted by different neurons, or the order in which neurons generate specific spikes.
[0034] Figure 1 The illustration shows a schematic diagram of a neuronal structure of a conventional received frequency encoded pulse sequence provided in an embodiment of the present disclosure.
[0035] See Figure 1 As shown, Figure 1 This is an integral-fire spiking neuron circuit structure for image processing. A photodiode (PD) converts light intensity into an electrical signal. A spike encoding circuit connected to the PD encodes the electrical signal into a frequency-coded pulse sequence received by the spiking neuron. Multiple frequency-coded pulse sequences generated by the PDs are input into the integral capacitor within the spiking neuron, accumulated on the first capacitor C1, and a comparator detects whether the accumulated pulses reach a threshold voltage V. refOnce the neuron meets the ignition condition, it fires a pulse. After the pulse is fired, an RST reset signal is used to reset the first capacitor C1, reducing the voltage V across the first capacitor C1 that has not yet reached the threshold voltage. ref The charge is cleared, awaiting the next integration process. The first capacitor C1 inside the neuron structure is small, allowing for rapid integration to the threshold voltage and subsequent clearing to zero.
[0036] Figure 1 The illustrated neuron structure accumulates the charge of the encoded pulse signal sequence and uses a comparator to determine whether the accumulated voltage value reaches the cell membrane threshold. If it does, a pulse is transmitted to the postsynapse. Multiple neurons form a layer of the spiking neural network. Each neuron in the layer receives the pulse signal sequence transmitted from the previous layer through its synapses, but the weights between the synapses are different. Multiple layers form a spiking neural network model that uses the pulse sequence and weight information to perform nonlinear operations on the image information, ultimately obtaining the output result.
[0037] For systems that solely employ pulse-time coding, the information carrier is the number of pulses within a fixed time step, similar to digital coding, ensuring high accuracy. However, in the sensor's readout circuit, each pixel needs to generate a large number of pulse sequences throughout the entire time frame. Especially for large background images without a target, this design consumes significant power, generates redundant coding that does not aid in target recognition, and may even increase resource consumption within subsequent image processing cores, such as the number of neurons in the input layer of a neural network.
[0038] Figure 2 The illustration shows a schematic diagram of a neuronal structure of a conventional time-coded pulse sequence provided in an embodiment of the present disclosure.
[0039] See Figure 2 As shown, the photodiode PD converts light intensity into an electrical signal. The spike encoding circuit connected to the photodiode PD encodes the electrical signal into a time-coded pulse sequence received by the spike neuron. The time-coded pulse sequences generated by multiple photodiodes PD are input into the second capacitor C2 within the spike neuron, accumulated on the second capacitor C2, and detected by a comparator whether the accumulated pulses reach the threshold voltage V. ref Once the neuron meets the ignition condition, it fires a pulse. After the pulse is fired, the second capacitor C2 is reset using an RST reset signal, thus reducing the voltage V across the second capacitor C2 that has not yet reached the threshold voltage. ref The charge is cleared, awaiting the next integration process. The second capacitor C2 has a large capacitance value, allowing its charge to slowly increase over time until it receives pulses arriving at different times, at which point it abruptly changes. This capacitance covers the entire counting cycle.
[0040] For systems employing pulse-time coding alone, although time coding generates fewer pulses compared to frequency coding, and can encode information into the time difference between pulses or the duration of the pulses, the readout circuit needs to control the precise timing of pulse generation to ensure that subsequent networks receive sufficiently accurate information to improve the accuracy of the final task (such as detection or recognition). Analog circuit designs that require precise timing are often complex, and delays in the information transmission path can be affected by the manufacturing process. For multi-pixel area array sensors, this not only increases hardware resource consumption and design complexity but also makes it difficult to maintain consistent time offsets during manufacturing.
[0041] Given that existing neuronal structures can only receive one type of pulse coding, pulse coding artificial vision systems need to determine the pulse coding method of the entire system during the initial design process, which is not conducive to the modular design of the system.
[0042] The multimodal neuron structure disclosed herein adopts a modular design approach, in which the pulse image sensor and the pulse image processing neural network are designed and implemented separately. The pulse neuron structure can receive different types of pulse codes.
[0043] Figure 3 The illustration shows a schematic diagram of a multimodal neuron structure provided in an embodiment of the present disclosure.
[0044] See Figure 3 As shown, this embodiment of the present disclosure provides a multimodal neuron structure, including a reference voltage generation module and a pulse generation module. The reference voltage generation module includes a first reference voltage generation module and a second reference voltage generation module. The first reference voltage generation module generates a first reference voltage, and the second reference voltage generation module generates a second reference voltage. The pulse generation module is connected to the reference voltage generation module. The input terminal of the pulse generation module is used to receive a frequency-coded pulse sequence or a time-coded pulse sequence. The pulse generation module includes a first charge packet counter sub-circuit, a second charge packet counter sub-circuit, and a mode signal input circuit.
[0045] The first charge packet counter sub-circuit is used to receive a frequency-coded pulse sequence. When the voltage corresponding to the accumulated charge in the frequency-coded pulse sequence reaches a first reference voltage, a target pulse is emitted. The second charge packet counter sub-circuit is used to receive a time-coded pulse sequence. When the voltage corresponding to the accumulated charge in the time-coded pulse sequence reaches a second reference voltage, a target pulse is emitted. The mode signal input circuit is used to input a first mode signal when receiving a frequency-coded pulse sequence to control the first charge packet counter sub-circuit to turn on, and to input a second mode signal when receiving a time-coded pulse sequence to control the second charge packet counter sub-circuit to turn on.
[0046] Specifically, see Figure 3 As shown, the photodiodes (PDs) in the image sensor array convert light intensity into electrical signals. These electrical signals are then converted by an encoding circuit into pulse sequences carrying information. The carrier of the light intensity information can be the frequency of the pulses in the pulse sequence (i.e., frequency encoding), or the time of pulse generation or the duration of the pulses (i.e., time encoding). These two parts constitute the imaging sensor array module of the pulse imaging system. The configurable multimodal neuron structure is an image processing module composed of a spiking neural network. This multimodal neuron structure is connected to the imaging sensor array module and can receive both frequency-encoded and time-encoded pulse sequences.
[0047] Figure 4 A schematic diagram of a pulse generation module provided in an embodiment of this disclosure is shown.
[0048] See Figure 4 As shown, in an optional embodiment, the multimodal neuron structure further includes a first transistor Q1 and a second transistor Q2. The first transistor Q1 is located at the input port of the first charge packet counter sub-circuit. When a frequency-coded pulse sequence is received, the first transistor Q1 controls the first charge packet counter sub-circuit to turn on according to the first mode signal. The second transistor Q2 is located at the input port of the second charge packet counter sub-circuit. When a time-coded pulse sequence is received, the second transistor Q2 controls the second charge packet counter sub-circuit to turn on according to the second mode signal.
[0049] Specifically, the first charge packet counter subcircuit is connected to the second charge packet counter subcircuit. The first transistor Q1 controls the conduction of the first charge packet counter subcircuit, and the second transistor Q2 controls the conduction of the second charge packet counter subcircuit. The first charge packet counter subcircuit is used to receive frequency-coded pulse sequences, and the second charge packet counter subcircuit is used to receive time-coded pulse sequences. Thus, the pulse generation module can configure the on / off state of the first or second charge packet counter subcircuit according to the received frequency-coded or time-coded pulse sequences, without needing to determine the form of the pulse sequence encoding in advance, thereby improving flexibility.
[0050] As an optional embodiment, the first charge packet counter sub-circuit includes a first integrating capacitor C. int1 When the first charge packet counter subcircuit is open, the third transistor Q3 accumulates the charge input by the frequency pulse sequence; the third transistor Q3 and the first integrating capacitor C int1 The connection is used so that when the first charge packet counter subcircuit emits a target pulse, the first reset signal is input to the third transistor Q3, the third transistor Q3 is turned on, and the capacitor C on the first integrating capacitor is cleared. int1 The charge.
[0051] Specifically, the third transistor Q3 and the first integrating capacitor C int1 When the first charge packet counter sub-circuit emits a target pulse, the first reset signal turns on the third transistor Q3, clearing the first integrating capacitor C. int1 The residual charge that has not reached the reference voltage waits for the first integrating capacitor C. int1 The next charge accumulation process begins.
[0052] As an optional embodiment, the second charge packet counter sub-circuit includes a second integrating capacitor C. int2 With the fourth transistor Q4, the second integrating capacitor C int2 Containing a preset amount of charge, one end of which is grounded, when the second charge packet counter subcircuit starts to conduct, the preset amount of charge gradually decreases over time, and the second integrating capacitor C... int2 When the time-coded pulse sequence is received, the preset amount of charge stops decreasing, and when the second integrating capacitor C... int2 When the voltage value corresponding to the remaining charge reaches the second reference voltage, the target pulse is emitted; the fourth transistor Q4 and the second integrating capacitor C int2 The connection is used so that when the pulse generation module emits the target pulse, the second reset signal is input to the fourth transistor Q4, the fourth transistor Q4 is turned on, and the second integrating capacitor C is cleared. int2 The charge on it.
[0053] Specifically, the fourth transistor Q4 and the second integrating capacitor C int2 When the pulse generation module transmits the target pulse, the second reset signal turns on the fourth transistor Q4, clearing the second integrating capacitor C. int4 The residual charge that has not reached the reference voltage waits for the second integrating capacitor C. int2 The next charge accumulation process begins.
[0054] As an optional embodiment, the mode signal input circuit includes a NAND gate and a fifth transistor Q5. The first input terminal of the NAND gate is connected to the transmitting terminal of the pulse generation module for transmitting the target pulse, the second input terminal is used to input a first mode signal or a second mode signal, and the output terminal is connected to the fifth transistor Q5. The fifth transistor Q5 is connected to the first charge packet counter sub-circuit. When the NAND gate receives the first mode signal, it controls the first charge packet counter sub-circuit to conduct; when the NAND gate receives the second mode signal, it controls the second charge packet counter sub-circuit to conduct.
[0055] In one illustrative embodiment, the operating mode of the multimodal neuron is configured according to the first mode signal and the second mode signal of the NAND gate. When the first mode signal 1'b0 is input to the NAND gate, the first charge packet counter sub-circuit is turned on, and when the second mode signal 1'b1 is input to the NAND gate, the second charge packet counter sub-circuit is turned on.
[0056] As an optional embodiment, the pulse generation module further includes a comparator. The first input terminal of the comparator is connected to the first charge packet counter sub-circuit, its second input terminal is connected to the second charge packet counter sub-circuit, and its third input terminal is connected to the reference voltage generation module. When the first charge packet counter sub-circuit is turned on, the comparator compares the voltage corresponding to the charge accumulated in the frequency-coded pulse sequence with the first reference voltage. When the voltage corresponding to the charge accumulated in the frequency-coded pulse sequence reaches the first reference voltage, the comparator output terminal emits the target pulse. When the second charge packet counter sub-circuit is turned on, the comparator compares the voltage corresponding to the charge accumulated in the time-coded pulse sequence with the second reference voltage. When the voltage corresponding to the charge accumulated in the time-coded pulse sequence reaches the second reference voltage, the comparator output terminal emits the target pulse.
[0057] As an optional embodiment, the first reference voltage is a constant DC voltage.
[0058] As an optional embodiment, the first reference voltage generation module is turned on when the pulse generation module receives the frequency-coded pulse sequence, and the second reference voltage generation module is turned on when the pulse generation module receives the time-coded pulse sequence.
[0059] Specifically, when the pulse generation module receives the frequency-coded pulse sequence, the first reference voltage module is turned on, and the first reference voltage is a constant DC voltage. Upon receiving the frequency-coded pulse sequence, the constant DC voltage serves as the reference voltage, enabling the transmission of the target pulse.
[0060] Figure 5 A schematic diagram of a second reference voltage module provided in an embodiment of this disclosure is shown.
[0061] See Figure 5 As shown, in one optional embodiment, the second reference voltage generation module includes a first variable resistor R1, a second variable resistor R2, a third variable resistor R3, a fourth variable resistor R4, a first transistor switch K1, a second transistor switch K2, a third transistor switch K3, a fourth transistor switch K4, a fifth transistor switch K5, a sixth transistor switch K6, a seventh transistor switch K7, a DC current source, and a capacitor C. REFThe adder is connected to the first terminal of the first variable resistor R1, the first terminal of the second variable resistor R2, the first terminal of the third variable resistor R3, and the first terminal of the fourth variable resistor R4; the first terminal of the first transistor switch K1 is connected to the second terminal of the first variable resistor R1, the first terminal of the second transistor switch K2 is connected to the second terminal of the second variable resistor R2, the second terminal of the second transistor switch K2 is connected to the first terminal of the third transistor switch K3, the third terminal of the second transistor switch K2 is connected to the third terminal of the first transistor switch K1, the fourth transistor switch K4 is connected to the second terminal of the third variable resistor R3, the first terminal of the fifth transistor switch K5 is connected to the second terminal of the fourth transistor switch K4, the first terminal of the sixth transistor switch K6 is connected to the second terminal of the fourth variable resistor R4, and the first terminal of the seventh transistor switch K7 is connected to the second terminal of the sixth transistor switch K6; the second terminal of the DC current source is connected to the third terminal of the first transistor switch K1; capacitor C REF The first terminal is connected to the second terminal of the third transistor switch K3, and the second terminal is grounded; the input terminal of the adder is connected to the second terminals of the third transistor switch K3, the fifth transistor switch K5, and the seventh transistor switch K7, and its output terminal is connected to the pulse generation module.
[0062] As an optional embodiment, the second reference voltage is a piecewise linear ramp voltage.
[0063] Specifically, when the pulse generation module receives the time-coded pulse sequence, the intersection of the reference voltage and current is non-linear in time for different electrical signal inputs. This affects the temporal non-linearity of the time-coded pulse sequence and also affects the interpretability of the time-coded pulse sequence received by the multimodal neuron structure.
[0064] Figure 6 The illustration shows a schematic diagram of an embodiment of the present disclosure using a common ramp wave as a reference voltage.
[0065] Figure 7 The diagram illustrates the linearity of a common ramp reference voltage and an input time-coded pulse sequence provided in an embodiment of the present disclosure.
[0066] See Figure 6 As shown, the red line represents the integral wave, and the blue line represents the reference wave. In time-pulse coding mode, a ramp is used as the comparator reference voltage. The start point of the ramp is the beginning time of the counting cycle, and the end point is the end time of the counting cycle. The end level is the power supply voltage. (Reference) Figure 7 As shown, the red line represents the actual time-coded spike delay time, and the blue line represents the linear fitting line. When a normal ramp is used as the reference voltage, the linearity between the time-coded pulse sequence and the input current is low.
[0067] Figure 8The illustration schematically shows a diagram of a piecewise linear ramp wave as a reference voltage provided in an embodiment of the present disclosure;
[0068] Figure 9 The diagram illustrates the linearity of the piecewise linear ramp reference voltage and the input time-coded pulse sequence provided in the embodiments of this disclosure.
[0069] See Figure 8 As shown, the red line represents the integral wave, and the blue line represents the reference wave. The slopes of the segmented ramp wave in the blue line are not the same. To further improve the aforementioned nonlinearity problem, a segmented linear ramp wave is introduced into the reference voltage. Through multiple variable resistors and multiple transistor switches in the second reference voltage module, the time of a single integration cycle is divided into slices, and a slope-enhancing current is introduced according to each time slice. Three variable resistors are used to store the parameters required to configure the current slope. (Refer to...) Figure 9 As shown, the nonlinearity factor improved from 0.099 to 0.0045, and the linearity was significantly improved.
[0070] In one illustrative embodiment, the circuit structure that generates the second reference voltage signal is a shared circuit for the entire area array and does not occupy the readout circuit area of a single pixel.
[0071] In summary, the multimodal neuron structure provided in this disclosure controls the on / off state of the first and second charge packet counter subcircuits by inputting signals of different modes through a mode signal input circuit. The pulse generation module can receive coded pulse sequences or time-coded pulse sequences, and the reference voltage generation module generates a first or second reference voltage based on the time-coded pulse sequence received by the pulse generation module, in order to transmit the target pulse. This disclosure can receive different types of pulse codes, is compatible with various types of pulse sensors, and has a simple circuit structure with a small footprint.
[0072] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.
[0073] Although this disclosure has been shown and described with reference to specific exemplary embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made to this disclosure without departing from the spirit and scope of the disclosure as defined by the appended claims and their equivalents. Therefore, the scope of this disclosure should not be limited to the above embodiments, but should be defined not only by the appended claims, but also by their equivalents.
Claims
1. A multimodal neuron structure, characterized in that, include: A reference voltage generation module includes a first reference voltage generation module and a second reference voltage generation module, wherein the first reference voltage generation module generates a first reference voltage and the second reference voltage generation module generates a second reference voltage; A pulse generation module is connected to the reference voltage generation module. The input terminal of the pulse generation module is used to receive a frequency-coded pulse sequence or a time-coded pulse sequence. The pulse generation module includes a first charge packet counter sub-circuit, a second charge packet counter sub-circuit, and a mode signal input circuit. The first charge packet counter sub-circuit is used to receive the frequency-coded pulse sequence, and when the voltage corresponding to the accumulated charge in the frequency-coded pulse sequence reaches the first reference voltage, it transmits the target pulse; The second charge packet counter sub-circuit is used to receive the time-coded pulse sequence, and when the voltage corresponding to the accumulated charge in the time-coded pulse sequence reaches the second reference voltage, it transmits the target pulse; The mode signal input circuit is used to input a first mode signal when receiving the frequency-coded pulse sequence to control the first charge packet counter sub-circuit to turn on, and to input a second mode signal when receiving the time-coded pulse sequence to control the second charge packet counter sub-circuit to turn on.
2. The multimodal neuron structure according to claim 1, characterized in that, include: A first transistor is located at the input port of the first charge packet counter sub-circuit. When the frequency-encoded pulse sequence is received, the first transistor controls the first charge packet counter sub-circuit to turn on according to the first mode signal. The second transistor is located at the input port of the second charge packet counter sub-circuit. When the time-coded pulse sequence is received, the second transistor controls the second charge packet counter sub-circuit to turn on according to the second mode signal.
3. The multimodal neuron structure according to claim 1, characterized in that, The first charge packet counter sub-circuit includes: The first integrating capacitor accumulates the charge input by the frequency pulse sequence when the first charge packet counter sub-circuit is in operation; The third transistor, connected to the first integrating capacitor, is used to turn on the third transistor when the first charge packet counter subcircuit emits a target pulse, and clears the charge on the first integrating capacitor.
4. The multimodal neuron structure according to claim 1, characterized in that, The second charge packet counter sub-circuit includes: The second integrating capacitor contains a preset amount of charge, one end of which is grounded. When the second charge packet counter sub-circuit starts to conduct, the preset amount of charge gradually decreases over time. When the second integrating capacitor receives the time-coded pulse sequence, the preset amount of charge stops decreasing. When the voltage value corresponding to the remaining charge on the second integrating capacitor reaches the second reference voltage, the target pulse is emitted. The fourth transistor, connected to the second integrating capacitor, is used to turn on the fourth transistor when the pulse generation module emits a target pulse, and clear the charge on the second integrating capacitor.
5. The multimodal neuron structure according to claim 1, characterized in that, The mode signal input circuit includes a NAND gate and a fifth transistor. The first input terminal of the NAND gate is connected to the transmitting terminal of the pulse generation module to transmit the target pulse, the second input terminal is used to input the first mode signal or the second mode signal, and the output terminal is connected to the fifth transistor. The fifth transistor is connected to the first charge packet counter sub-circuit. When the NAND gate receives the first mode signal, it controls the first charge packet counter sub-circuit to turn on. When the NAND gate receives the second mode signal, it controls the second charge packet counter sub-circuit to turn on.
6. The multimodal neuron structure according to claim 1, characterized in that, The pulse generation module further includes: The comparator has its first input terminal connected to the first charge packet counter sub-circuit, its second input terminal connected to the second charge packet counter sub-circuit, and its third input terminal connected to the reference voltage generation module. When the first charge packet counter sub-circuit is turned on, the comparator compares the voltage corresponding to the charge accumulated in the frequency-coded pulse sequence with the first reference voltage. When the voltage corresponding to the charge accumulated in the frequency-coded pulse sequence reaches the first reference voltage, the comparator output terminal emits a target pulse. When the second charge packet counter sub-circuit is turned on, the comparator compares the voltage corresponding to the charge accumulated in the time-coded pulse sequence with the second reference voltage. When the voltage corresponding to the charge accumulated in the time-coded pulse sequence reaches the second reference voltage, the comparator output terminal emits the target pulse.
7. The multimodal neuron structure according to claim 1, characterized in that, The first reference voltage is a constant DC voltage.
8. The multimodal neuron structure according to claim 1, characterized in that, The second reference voltage generation module includes: A first variable resistor, a second variable resistor, a third variable resistor, and a fourth variable resistor, wherein the first end of the first variable resistor, the first end of the second variable resistor, the first end of the third variable resistor, and the first end of the fourth variable resistor are connected together; A first transistor switch, a second transistor switch, a third transistor switch, a fourth transistor switch, a fifth transistor switch, a sixth transistor switch, and a seventh transistor switch, wherein the first terminal of the first transistor switch is connected to the second terminal of the first variable resistor, the first terminal of the second transistor switch is connected to the second terminal of the second variable resistor, the second terminal of the second transistor switch is connected to the first terminal of the third transistor switch, the third terminal of the second transistor switch is connected to the third terminal of the first transistor switch, the fourth transistor switch is connected to the second terminal of the third variable resistor, the first terminal of the fifth transistor switch is connected to the second terminal of the fourth transistor switch, the first terminal of the sixth transistor switch is connected to the second terminal of the fourth variable resistor, and the first terminal of the seventh transistor switch is connected to the second terminal of the sixth transistor switch. A DC current source, wherein the second terminal of the DC current source is connected to the third terminal and the third terminal of the first transistor switch; The capacitor has its first terminal connected to the second terminal of the third transistor switch, and its second terminal is grounded. The adder has its input terminal connected to the second terminals of the third, fifth, and seventh transistor switches, and its output terminal connected to the pulse generation module.
9. The multimodal neuron structure according to claim 1, characterized in that, The second reference voltage is a piecewise linear ramp voltage.
10. The multimodal neuron structure according to claim 1, characterized in that, The first reference voltage generation module is turned on when the pulse generation module receives the frequency-coded pulse sequence, and the second reference voltage generation module is turned on when the pulse generation module receives the time-coded pulse sequence.