Neural network acceleration system and method based on optoelectronic in-memory computing cells

Abstract

The application discloses a neural network acceleration system based on an optoelectronic memory and computing unit and a method thereof. The system comprises an optoelectronic memory and computing unit array module, an array driving module, an optical input module, a readout module and a processing module, wherein the optoelectronic memory and computing unit array module is connected with the array driving module and the readout module, the array driving module is located before the optoelectronic memory and computing unit array module, the readout module is located after the optoelectronic memory and computing unit array module, the processing module is arranged between the array driving module and the optoelectronic memory and computing unit array module or between the readout module and the optoelectronic memory and computing unit array module. The application provides a multifunctional, compatible and low-power neural network acceleration system based on the optoelectronic memory and computing unit, and can realize reset, optical input, excitation input, readout and processing operations of the optoelectronic memory and computing unit, and solves the system-level deployment problem of the optoelectronic memory and computing unit.

Description

TECHNICAL FIELD

[0001] The application provides a multifunctional, good compatibility and low-power neural network acceleration system based on an optoelectronic memory and computing unit, which can realize reset, optical input, excitation input, readout and processing operations of the optoelectronic memory and computing unit, and solves the system-level deployment problem of the optoelectronic memory and computing unit. BACKGROUND

[0002] In view of the weak operation ability, large volume and high power consumption of the fast real-time processing system in real-time detection and real-time classification and identification of the perception system, a new computing architecture is needed to realize high operation ability, small volume and low power consumption for deployment on various terminals. The memory and computing integration technology successfully overcomes the data transfer power consumption problem caused by the separation of memory units and computing units in the traditional von Neumann architecture. The optoelectronic computing unit, as a typical memory and computing integration technology (such as patent No. ZL 201811398206.9), has the advantages of good process compatibility, easy super large scale integration and high single device storage bit width, but due to the use of analog domain computing, the optoelectronic memory and computing unit is prone to problems such as single function, complex readout circuit and complex configuration in actual application. SUMMARY

[0003] In view of the above technical problems of the existing optoelectronic computing unit system in the neural network acceleration algorithm, the application provides a multifunctional, good compatibility and low-power neural network acceleration system based on an optoelectronic memory and computing unit and a method thereof, which can realize reset, optical input, excitation input, readout and processing operations of the optoelectronic memory and computing unit, and solves the system-level deployment problem of the optoelectronic memory and computing unit.

[0004] The technical scheme of the system of the application is as follows:

[0005] The neural network acceleration system based on the optoelectronic memory and computing unit comprises:

[0006] The optoelectronic memory and computing unit array module is composed of a plurality of optoelectronic memory and computing units, which is used for processing input signals in a computing mode, writing weights of the optoelectronic memory and computing unit in an optical input and erasing mode, and realizing nonvolatile storage;

[0007] The array driving module comprises an input control circuit and a high-voltage driving circuit; the input control circuit is used for controlling the working state of the high-voltage driving circuit; the high-voltage driving circuit is connected with each port of the optoelectronic memory and computing unit array module, and is used for providing various high voltages required in different working modes;

[0008] The light input module comprises a light emitting unit array, an auxiliary optical element, a light sensing circuit and a programming calibration circuit; the light emitting unit array is used to provide a light signal required by a programming mode; the auxiliary optical element is used to assist in realizing an optical coupling relationship between the light emitting unit array and a photovoltaic storage and calculation unit array module; the light sensing circuit is used to collect a light signal of the light emitting unit array; and the programming calibration circuit is used to read out a programmed photovoltaic storage and calculation unit and compare it with an expected value programming value to complete a calibration function.

[0009] The readout module comprises a clamping circuit and a format conversion circuit; the clamping circuit is used to provide a calculation voltage required by the photovoltaic storage and calculation unit array module in a calculation mode; and the format conversion circuit is used to convert a current signal of the photovoltaic storage and calculation unit into a data form that can be recognized and processed by a subsequent circuit.

[0010] The processing module comprises a non-ideal factor correction circuit, a shift accumulation circuit, a nonlinear activation function circuit, a negative number generation circuit and a pooling circuit; the non-ideal factor correction circuit is used to compensate for a temperature drift phenomenon and a current offset phenomenon; the shift accumulation circuit is used to realize a shift accumulation operation; the nonlinear activation function circuit is used to realize an activation operation; the negative number generation circuit is used to convert a negative number weight or a negative number input into a positive number realized in the photovoltaic storage and calculation unit array for processing; and the pooling circuit is used to realize a pooling operation.

[0011] The photovoltaic storage and calculation unit array module is connected with an array driving module and a readout module; the array driving module is arranged before the photovoltaic storage and calculation unit array module; the readout module is arranged after the photovoltaic storage and calculation unit array module; and the processing module is arranged between the array driving module and the photovoltaic storage and calculation unit array module, or between the readout module and the photovoltaic storage and calculation unit array module.

[0012] Further, the excitation signal input mode of the array driving module is a digital input by bit or a time dimension regulation or analog voltage signal using pulse width modulation.

[0013] Further, the clamping circuit comprises a circuit structure amplifier capable of realizing negative feedback.

[0014] Further, in the format conversion circuit, a transimpedance amplifier composed of an amplifier and a resistor or a capacitive transimpedance amplifier composed of an amplifier and a capacitor are included to realize current-voltage conversion; and a successive approximation type analog-to-digital converter, a single slope analog-to-digital converter or a mixed topology thereof are adopted to realize analog signal and digital signal conversion.

[0015] Further, the non-ideal factor calibration circuit compensates by changing the calculation voltage loaded on the photoelectric storage and calculation unit array module through a lookup table method, or by introducing a reference column to perform current subtraction.

[0016] Further, the shift accumulation circuit realizes charge sharing through capacitors with different capacitance values according to weight values.

[0017] Further, the non-linear activation function circuit realizes the activation function through a comparator.

[0018] Further, the negative number generation circuit realizes the conversion from negative number to positive number by performing current subtraction in the analog domain.

[0019] Further, the pooling circuit realizes maximum pooling through a multi-input comparator.

[0020] The application also provides a working method of the above-mentioned neural network acceleration system, which comprises the following steps:

[0021] (1) Reset: reset the weight values of the photoelectric storage and calculation unit to the initial values through the array driving module;

[0022] (2) Light input: program the weight values of the photoelectric storage and calculation unit to the desired values through the light input module and the array driving module;

[0023] (3) Excitation input: load the signal to be processed into the photoelectric storage and calculation unit array module through the array driving module, so that the photoelectric storage and calculation unit generates a current response;

[0024] (4) Readout: read out the current signal generated by the photoelectric storage and calculation unit and convert it into a data form that can be recognized and processed by the subsequent circuit through the readout module;

[0025] (5) Processing: process the signal output by the readout module through the processing module, or process the excitation input signal before the excitation input.

[0026] The neural network acceleration system of the application proposes other auxiliary modules cooperating with the array on the basis of the photoelectric storage and calculation unit array, so that the array can work efficiently in a complex working environment. Since the photoelectric storage and calculation unit needs to realize accurate light input, the use of photoelectric hybrid control can make the light input more accurate. After the excitation input is completed, the processing module can cooperate with the readout module to complete efficient readout, and through non-ideal factor correction, shift accumulation, non-linear activation and other operations, the neural network acceleration system has better compatibility and lower power consumption. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 The structure block diagram of the neural network acceleration system based on the photoelectric storage and calculation unit of the application is shown in the figure;

[0028] Figure 2 Fig. 2 is a schematic diagram of the coupling of the array driving module and the optoelectronic memory and calculation unit array module;

[0029] Figure 3 Fig. 3 is a schematic diagram of a circuit of the array driving module;

[0030] Figure 4 Fig. 4 is a schematic diagram of the coupling of the light input module and the optoelectronic memory and calculation unit array module;

[0031] Figure 5 Fig. 5 is a schematic diagram of the coupling of the readout module and the optoelectronic memory and calculation unit array module;

[0032] Figure 6 Fig. 6 is a schematic diagram of a circuit of the readout module. DETAILED DESCRIPTION

[0033] Figure 1 Fig. 1 is a structural block diagram of an embodiment of a neural network acceleration system based on an optoelectronic memory and calculation unit, the neural network acceleration system comprising:

[0034] An array driving module 1 comprising an input control circuit and a high-voltage driving circuit, the input control circuit being configured to control the working state of the high-voltage driving circuit, and the high-voltage driving circuit being connected to each port of the optoelectronic memory and calculation unit array and configured to provide various high voltages required in different working modes and accurately control the duration of the high voltages;

[0035] A processing module 2 comprising a non-ideal factor correction circuit, a shift and accumulation circuit, a nonlinear activation function circuit, a negative number generation circuit and a pooling circuit, the non-ideal factor correction circuit being configured to compensate for the current offset caused by the device temperature drift and the metal line voltage division, thereby ensuring the accuracy of the calculation results, the shift and accumulation circuit being configured to implement the shift and accumulation operation, the nonlinear activation function circuit being configured to implement the activation operation, the negative number generation circuit being configured to convert the negative number weight or negative number input required in the algorithm into a positive number that can be easily implemented in the optoelectronic memory and calculation unit array for processing, and the pooling circuit being configured to implement the pooling operation;

[0036] An optoelectronic memory and calculation unit array module 3 comprising a plurality of optoelectronic memory and calculation units, which can process the input signal in the calculation mode, write the weight of the optoelectronic memory and calculation unit in the light input and erasing mode, and realize the nonvolatile storage;

[0037] A readout module 4 comprising a clamping circuit and a format conversion circuit, the clamping circuit being configured to provide the calculation voltage required by the optoelectronic memory and calculation unit array in the calculation mode, and the format conversion circuit being configured to convert the current signal of the optoelectronic memory and calculation unit into a data form that can be recognized and processed by the subsequent circuit;

[0038] The light input module 5 comprises a light emitting unit array, an auxiliary optical element, a light sensing circuit and a programming calibration circuit, the light emitting unit array is used for providing a light signal required by a programmed mode, the auxiliary optical element is used for assisting to realize an optical coupling relationship between the light emitting unit array and the photoelectric storage and calculation integrated array, the light sensing circuit is used for collecting a light signal of the light emitting unit array, facilitating the system to regulate and control the light emitting unit array, and the programming calibration circuit is used for reading out the programmed photoelectric storage and calculation unit and comparing with an expected value programming value, so as to complete a calibration function.

[0039] The weight of the photoelectric storage and calculation unit is adjustable, which is equivalent to storing a variable analog weight data in each photoelectric storage and calculation unit, and the photoelectric storage and calculation unit array can map an analog weight matrix; when the photoelectric storage and calculation unit is excited, a current is induced, the current size is equal to the excitation multiplied by the weight, and then the coupling relationship between the array units can realize the convergence of the same row or column current, and then various operations can be completed by using the photoelectric storage and calculation unit array.

[0040] The neural network acceleration system provided by the application receives an external control signal through the array driving module 1 to complete different mode selection; in the light input mode, each photoelectric storage and calculation unit is input to an expected input weight value through external control; in the reset mode, each photoelectric storage and calculation unit is erased to an expected erase weight value through external control; and in the calculation mode, an excitation signal is sent to the array through external control.

[0041] The neural network acceleration system provided by the application also completes the light input operation through the light input module 5 cooperating with the array driving module; the weight data can be reflected on the electrical signal or the optical signal, or the common control of the electrical signal and the optical signal.

[0042] The neural network acceleration system provided by the application also completes the calculation operation through the readout module 4 cooperating with the array driving module; the array driving module can provide the calculation voltage, or the readout module can provide the calculation voltage.

[0043] The neural network acceleration system provided by the application also completes the calculation operation through the processing module 2 cooperating with the readout module and the array driving module; the processing module 2 can be between the array driving module 1 and the photoelectric storage and calculation unit array module 3, or between the readout module 4 and the photoelectric storage and calculation unit array module 3.

[0044] Figure 2Figure 1 is a schematic diagram of the coupling of the array driving module 1 and the optoelectronic memory and calculation unit array module 3. The optoelectronic memory and calculation unit array module 3 has M rows and N columns. The word line WL driving circuit 201, the bit line BL driving circuit 202, and the source line SL driving circuit 203 constitute the array driving module 1. Each WL, BL, and SL of the optoelectronic memory and calculation unit array is configured with an independent driving circuit. The driving circuit and the array can also be multiplexed through a switch to save the area of the driving circuit. The BL driving circuit and the SL driving circuit have the same position, and their functions can be replaced with each other. The directions of the BL and the SL can be parallel or vertical. The excitation signal can be input from any one end of the driving circuit. The excitation input mode can be digital input by bit, or time-dimension control using pulse width modulation or analog voltage signal. The following embodiment is described by taking the excitation input from the WL driving circuit as an example.

[0045] Figure 3 Figure 2 is a schematic diagram of the circuit of the array driving module. The WL driving circuit 201 is composed of a digital register 301 and a level shift circuit 302. The excitation signal is input into the digital register through an external controller. The level shift circuit converts the logic of the digital signal into a high-voltage signal for operating the optoelectronic memory and calculation unit array module. If a multi-bit excitation signal needs to be input, it can be input bit by bit in time division to achieve the effect of multi-bit excitation input. The level shift circuit 302 can be replaced by a PWM generation circuit and a DAC circuit to achieve excitation signal input.

[0046] Figure 4 Figure 3 is a schematic diagram of the coupling of the light input module and the optoelectronic memory and calculation unit array module. The light-emitting array 401 provides the light signal required by the light input of the optoelectronic memory and calculation unit array module. The light input can be achieved by direct projection or with the help of a lens. According to the weight configuration, the light intensity at different positions is achieved to realize the weight write operation of the optoelectronic memory and calculation unit. The light input control device 402 provides the control signal of the light-emitting array to realize the automatic write and calibration function of the weight.

[0047] Figure 5 Figure 4 is a schematic diagram of the coupling of the readout module and the optoelectronic memory and calculation unit array module. The clamp circuit 501 and the format conversion circuit 502 constitute the readout module. The calculation voltage required in the calculation mode is provided by the clamp circuit 501. The current excitation generated by the optoelectronic memory and calculation unit array is converted by the format conversion circuit and sent to the subsequent module for processing.

[0048] Figure 6A circuit schematic diagram of the readout module, the clamping circuit 501 is composed of an operational amplifier and a MOS tube, the current of the photoelectric storage and calculation unit array is copied to the current-voltage (IV) conversion circuit 601 through the current mirror after being obtained, and then the voltage signal is converted and quantized through the digital-to-analog conversion (ADC) 602, the clamping circuit 501 can also be realized by using various structures capable of realizing negative feedback, and if the ADC 602 is a current input type, the IV conversion circuit 601 can be omitted. The clamping circuit 501 and the IV conversion circuit 601 can also be integrated in the same module to optimize the area and power consumption.

[0049] As shown in Figure 1 The processing module 2 can be located between the array driving module 1 and the photoelectric storage and calculation unit array module 3, and the processing module 2 located at this position is a non-ideal factor correction circuit, which can compensate for current drift caused by temperature change and IR drop by adjusting the calculation voltage in the calculation mode, so that the calculation result is more accurate. The temperature drift model and the IR drop model can be obtained by fitting the measured data. The processing module 2 can also be located between the photoelectric storage and calculation unit array module 3 and the readout module 4, and the processing module 2 located at this position is a non-ideal factor correction circuit. The processing module 2 located at this position is a shift accumulation circuit, a nonlinear activation function circuit, a negative number generation circuit, a pooling circuit or a combination of several of them. The shift accumulation circuit is used to realize the weighted accumulation operation, the nonlinear activation function circuit is used to realize the activation function function, the negative number generation circuit is used to realize the negative weight or negative incentive required in the algorithm, so that the algorithm has stronger universality, and the pooling circuit is used to realize the pooling function. The processing module 2 can also be located after the readout module 4, and the processing module 2 located at this position can be the accumulation circuit, the nonlinear activation function circuit, the negative number generation circuit, the pooling circuit or the necessary processing circuit required in the subsequent algorithm.

[0050] The embodiment provides a working method of a neural network acceleration system based on a photoelectric storage and calculation unit:

[0051] (1) Reset: reset the weight of the photoelectric storage and calculation unit array 3 to the initial value through the array driving module 1, add-9V voltage to all WL, add 5V voltage to all BL, add 5V voltage to all SL, and add 5V voltage to the substrate. After continuing for several seconds, the weight of all photoelectric storage and calculation units is reset to the initial value.

[0052] (2) Light input: the weight of the photoelectric storage and calculation unit array 3 is programmed to the desired value through the light input module 5 and the array driving module 1, the selected WL is added with a voltage of 5.5V, the unselected WL is added with a voltage of 0V, the selected BL is added with a voltage of 0V, the unselected BL is added with a voltage of-2V, the selected SL is added with a voltage of 0V, the unselected SL is added with a voltage of-2V, the substrate is added with a voltage of-2V, while the light-emitting unit array is controlled to project uniform light, and the weight writing is performed by controlling the light input time of each photoelectric storage and calculation unit;

[0053] (3) Excitation input: the signal to be processed is loaded into the photoelectric storage and calculation unit array 3 through the array driving module 1, the selected WL is added with a voltage of 3.2V, the unselected WL is added with a voltage of 0V, the photoelectric storage and calculation unit array 3 generates a current response according to the excitation input signal and the weight of the corresponding photoelectric storage and calculation unit, and the excitation signal input with a total bit width of 8 bits is completed by using the method of time-sharing bit-by-bit digital input;

[0054] (4) Readout: the current signal generated by the photoelectric storage and calculation unit array 3 is read out and converted into a data form that can be recognized and processed by the subsequent circuit through the readout module 4, the clamping voltage is 0.2V, and the method of time-sharing bit-by-bit readout is also used;

[0055] (5) Processing: the signals output by the readout module 4 are shifted and accumulated by the processing module 2 to obtain the final calculation result.

Claims

1. A photonic-electronic in-memory computing unit based neural network acceleration system, characterized in that, The system comprises: The optoelectronic memory and calculation unit array module is composed of a plurality of optoelectronic memory and calculation units, is used for processing input signals in a calculation mode, is used for writing weights of the optoelectronic memory and calculation units in an optical input and erasing mode, and realizes nonvolatile storage; The array driving module comprises an input control circuit and a high-voltage driving circuit; the input control circuit is used for controlling the working state of the high-voltage driving circuit; the high-voltage driving circuit is connected with each port of the optoelectronic memory and calculation unit array module, and is used for providing various high voltages required in different working modes; The light input module comprises a light-emitting unit array, auxiliary optical elements, a light-sensing circuit and a programming calibration circuit; the light-emitting unit array is used for providing optical signals required in a programming mode; the auxiliary optical elements are used for assisting to realize an optical coupling relationship between the light-emitting unit array and the optoelectronic memory and calculation unit array module; the light-sensing circuit is used for collecting optical signals of the light-emitting unit array; and the programming calibration circuit is used for reading out programmed optoelectronic memory and calculation units and comparing them with expected values, thereby completing a calibration function; The readout module comprises a clamping circuit and a format conversion circuit; the clamping circuit is used for providing calculation voltages required by the optoelectronic memory and calculation unit array module in a calculation mode; and the format conversion circuit is used for converting current signals of the optoelectronic memory and calculation units into a data form that can be recognized and processed by a subsequent circuit. The processing module comprises a non-ideal factor correction circuit, a shift and accumulation circuit, a nonlinear activation function circuit, a negative number generation circuit and a pooling circuit; the non-ideal factor correction circuit is used for compensating for temperature drift and current offset phenomena; the shift and accumulation circuit is used for realizing a shift and accumulation operation; the nonlinear activation function circuit is used for realizing an activation operation; the negative number generation circuit is used for converting negative weight values or negative input into positive numbers for processing in the optoelectronic memory and calculation unit array; and the pooling circuit is used for realizing a pooling operation. The optoelectronic memory and calculation unit array module is connected with the array driving module and the readout module; the array driving module is arranged before the optoelectronic memory and calculation unit array module; the readout module is arranged after the optoelectronic memory and calculation unit array module; and the processing module is arranged between the array driving module and the optoelectronic memory and calculation unit array module, or between the readout module and the optoelectronic memory and calculation unit array module.

2. The photonic-electronic memory cell based neural network acceleration system of claim 1, wherein, The excitation signal input mode of the array driving module is digital input by bit or time-dimension regulation and control or analog voltage signal using pulse width modulation.

3. The photonic-electronic memory cell based neural network acceleration system of claim 1, wherein, The clamping circuit comprises a circuit structure amplifier capable of realizing negative feedback.

4. The photonic-electronic memory cell based neural network acceleration system of claim 1, wherein, In the format conversion circuit, a transimpedance amplifier composed of an amplifier and a resistor or a capacitance transimpedance amplifier composed of an amplifier and a capacitance is included to realize conversion between current and voltage; and a successive approximation type analog-to-digital converter, a single slope analog-to-digital converter or a mixed topology thereof is adopted to realize conversion between analog signals and digital signals.

5. The photonic-electronic memory cell based neural network acceleration system of claim 1, wherein, The non-ideal factor correction circuit compensates by changing calculation voltages loaded on the optoelectronic memory and calculation unit array module through a lookup table method, or compensates by introducing a reference column for current subtraction.

6. The photonic-electronic memory cell based neural network acceleration system of claim 1, wherein, The shift accumulation circuit realizes charge sharing through capacitors with different capacitances distributed by weight values.

7. The photonic crossbar-based neural network acceleration system of claim 1, wherein, The nonlinear activation function circuit realizes the activation function through a comparator.

8. The photonic-electronic memory cell based neural network acceleration system of claim 1, wherein, The negative number generation circuit realizes the conversion from negative number to positive number through current subtraction in the analog domain.

9. The photonic crossbar-based neural network acceleration system of claim 1, wherein, The pooling circuit realizes maximum pooling through a multi-input comparator.

10. A method of operating a photonic-electronic memory computing cell based neural network acceleration system as claimed in claim 1, wherein, The method comprises the following steps: (1) Reset: reset the weight values of the optoelectronic storage and calculation unit to the initial values through the array driving module; (2) Light input: program the weight values of the optoelectronic storage and calculation unit to the desired values through the light input module and the array driving module; (3) Excitation input: load the signal to be processed into the optoelectronic storage and calculation unit array module through the array driving module, so that the optoelectronic storage and calculation unit generates a current response; (4) Readout: read out the current signal generated by the optoelectronic storage and calculation unit and convert it into a data form that can be recognized and processed by the subsequent circuit through the readout module; (5) Processing: process the signal output by the readout module through the processing module, or process the excitation input signal before the excitation input.

Images