A sensor-storage-computing integrated image recognition system and method based on RRAM

Abstract

The present invention discloses an RRAM-based sensing, storage and computing integrated image recognition system and method, which relate to the field of image recognition technology. An image recognition model is constructed using RRAM and FPGA, and the image recognition model is used to recognize an image to be recognized collected by an image sensor to obtain an image recognition result. The RRAM is used to complete the operation of a first network layer in the image recognition model, and the first network layer is a network layer that completes multiplication and accumulation operations. The FPGA is used to complete the operation of a second network layer in the image recognition model, and the second network layer is a network layer other than the first network layer in the image recognition model. Thus, part of the operations involved in the image recognition process are completed in a memory, and the problem that a large amount of data needs to be frequently transferred between a computing unit and a memory is avoided, and the problems of low computing efficiency caused by a memory wall, high power consumption caused by frequent memory access and bus operations, etc. are solved. The system is suitable for end-side devices.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to an RRAM-based sensing, storage and computing integrated image recognition system and method. Background Art

[0002] At present, image recognition technology has been widely used in all walks of life, and image recognition systems based on image recognition technology have also been deployed in various end-side devices, such as access control gates, monitoring equipment, and electronic wearable devices. These end-side devices often do not require extreme computing power, but are very sensitive to power consumption. The image recognition algorithms used in image recognition systems are mainly based on neural networks, especially convolutional neural networks (CNN). After a long period of development, the network depth of CNN has gradually increased, and the amount of data has reached millions to hundreds of millions. The operation form is mainly vector and matrix multiplication, and its essence is multiplication and accumulation.

[0003] However, most current image recognition systems still follow the von Neumann architecture's idea of ​​separation of memory and computing in hardware architecture. The computing system is composed of computing units such as CPU, GPU or dedicated AI chip and external memory. They cannot avoid the problem of large amounts of data needing to be frequently transferred between computing units and memory. On the one hand, the computing efficiency is severely limited due to the mismatch between the computing unit and memory speed, coupled with the limitation of bus bandwidth; on the other hand, frequent memory accesses and bus operations lead to a significant increase in power consumption.

[0004] Based on this, there is an urgent need for an image recognition system and method that integrates sensing, storage and computing and is suitable for end-side devices. Summary of the invention

[0005] The purpose of the present invention is to provide an RRAM-based integrated sensing, storage and computing image recognition system and method, which solves the problems of low computing efficiency caused by memory walls, high power consumption caused by frequent memory accesses and bus operations, and is suitable for terminal devices.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] An RRAM-based sensing, storage and computing integrated image recognition system, comprising: an image sensor, RRAM and FPGA;

[0008] The image sensor is used to collect the image to be identified;

[0009] The RRAM and the FPGA can build an image recognition model, and the image recognition model is used to recognize the image to be recognized to obtain an image recognition result; wherein the RRAM is used to complete the operation of the first network layer in the image recognition model, and the first network layer is a network layer that completes multiplication and addition operations; the FPGA is used to complete the operation of the second network layer in the image recognition model, and the second network layer is a network layer other than the first network layer in the image recognition model.

[0010] In some embodiments, the first network layer includes a convolutional layer and a fully connected layer; the second network layer includes a pooling layer and an activation function layer.

[0011] In some embodiments, the FPGA is also used to preprocess the image to be identified according to the data format required by the first network layer to obtain preprocessed data, and transfer the preprocessed data to the RRAM; wherein the preprocessing includes format adjustment.

[0012] In some embodiments, the FPGA is also used to pulse encode the preprocessed data to obtain a pulse encoding signal, generate an input pulse control signal based on the pulse encoding signal, and control the RRAM to generate an input pulse based on the input pulse control signal; wherein the input pulse control signal includes a first signal for controlling the pulse amplitude of a word line, a second signal for controlling the pulse amplitude of a bit line, a third signal for controlling the pulse width of a word line, and a fourth signal for controlling the pulse width of a bit line.

[0013] In some embodiments, the storage unit in the i-th row and j-th column of the RRAM is deployed with a first weight value, and the storage unit in the i+1-th row and j-th column is deployed with a second weight value, and the difference between the first weight value and the second weight value is equal to the value of the j-th weight of the first network layer to complete the deployment of the weight of the first network layer; wherein the first weight value and the second weight value are both greater than or equal to 0;

[0014] The RRAM is used to complete the operation of the first network layer based on the weight of the first network layer and the input pulse.

[0015] In some embodiments, the FPGA is also used to send a first control signal and configuration data to the image sensor, where the first control signal is used to control the image sensor, the first control signal includes a clock signal, an external frame synchronization signal and a reset signal, and the configuration data is used to configure the image sensor.

[0016] In some embodiments, the FPGA is further used to send a second control signal to the RRAM, the second control signal is used to control the RRAM, the second control signal includes a working state control signal, a read / write trigger signal, a read output enable signal and an address signal; the address signal is used to select an array, a word line, a bit line and a source line in the RRAM.

[0017] In some embodiments, the FPGA is also used to perform at least one of compression, temporary storage, and transmission timing control on the image to be identified.

[0018] A method for image recognition based on RRAM with integrated sensing, storage and computing, controlling the above-mentioned image recognition system based on RRAM with integrated sensing, storage and computing to work, comprising:

[0019] An image recognition model is constructed using RRAM and FPGA, and the image recognition model is used to recognize an image to be recognized acquired by an image sensor to obtain an image recognition result; wherein the RRAM is used to complete the operation of the first network layer in the image recognition model, and the first network layer is a network layer that completes multiplication and accumulation operations; and the FPGA is used to complete the operation of the second network layer in the image recognition model, and the second network layer is a network layer other than the first network layer in the image recognition model.

[0020] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0021] The present invention provides an RRAM-based sensing, storage and computing integrated image recognition system and method. An image recognition model is constructed with RRAM and FPGA. The image recognition model is used to recognize an image to be recognized collected by an image sensor to obtain an image recognition result. The RRAM is used to complete the operation of the first network layer in the image recognition model, and the first network layer is a network layer that completes multiplication and accumulation operations. The FPGA is used to complete the operation of the second network layer in the image recognition model, and the second network layer is other network layers in the image recognition model except the first network layer. In this way, part of the operations involved in the image recognition process are completed in the memory, avoiding the problem that a large amount of data needs to be frequently transferred between a computing unit and a memory, solving the problems of low computing efficiency caused by a memory wall, high power consumption caused by frequent memory access and bus operations, and the like. The system is suitable for end-side devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A schematic diagram of the top-level architecture of the sensing, storage and computing integrated image recognition system provided in Example 1 of the present invention;

[0024] Figure 2 A schematic diagram of the hardware design portion of the FPGA provided in Embodiment 1 of the present invention;

[0025] Figure 3 A schematic diagram of the RRAM chip architecture provided in Example 1 of the present invention;

[0026] Figure 4 A schematic diagram of the principle of deploying weights on RRAM and implementing multiplication and accumulation provided in Example 1 of the present invention;

[0027] Figure 5 A schematic diagram of the process of running a convolutional neural network when implementing video image recognition provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] The purpose of the present invention is to provide an RRAM-based integrated sensing, storage and computing image recognition system and method, which solves the problems of low computing efficiency caused by memory walls, high power consumption caused by frequent memory accesses and bus operations, and is suitable for terminal devices.

[0030] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] Example 1

[0032] RRAM (Resistive Random Access Memory) is a storage array composed of variable resistor devices. It has lower latency among many new types of memories, and as a non-volatile memory (NVM), it has good retention and durability. In addition, based on Kirchhoff's law, RRAM can quickly complete multiplication and accumulation operations in the analog domain, that is, complete in-memory calculations, fundamentally solving the memory wall problem caused by the von Neumann architecture. At the same time, calculations in the analog domain have very low power consumption, so RRAM has received widespread attention in the fields of in-memory computing and neural network acceleration, and has good application prospects in end-side devices.

[0033] Based on this, this embodiment designs a sensing-storage-computing integrated architecture that combines an image sensor with RRAM. The sensing-storage-computing integrated architecture refers to an architecture in which the image sensor and a memory with in-memory computing function (i.e., RRAM) are tightly coupled. The image data collected by the image sensor can be directly sent to the memory for in-situ computing after simple processing, without the need to be transferred between the memory and the processor chip responsible for computing (i.e., the computing unit). Based on the sensing-storage-computing integrated architecture, a sensing-storage-computing integrated image recognition system is further constructed, that is, an image recognition system that can achieve sensing-storage-computing integration is constructed. The entire image recognition system does not require a CPU, GPU or dedicated AI chip, which solves the problems of low computing efficiency caused by memory walls and high power consumption caused by frequent memory accesses and bus operations.

[0034] like Figure 1 As shown, the sensing, storage and computing integrated image recognition system based on RRAM in this embodiment includes: image sensor, RRAM and FPGA in hardware.

[0035] The image sensor is used to collect images to be recognized.

[0036] RRAM and FPGA can build an image recognition model, which is used to identify the image to be recognized and obtain the image recognition result. Among them, RRAM is used to complete the operation of the first network layer in the image recognition model, and the first network layer is the network layer that completes the multiplication and accumulation operation. FPGA is used to complete the operation of the second network layer in the image recognition model, and the second network layer is the network layer other than the first network layer in the image recognition model.

[0037] The image sensor of this embodiment is used to collect image data (i.e., image signals), and the collected image signals are converted into digital image signals through ADC (Analog-to-digital converter), and the digital image signals are sent to FPGA. For image sensors with built-in ADC, there is no need to use an additional ADC, otherwise, an additional ADC needs to be set between the image sensor and the FPGA. The image sensor can be an infrared image sensor, a visible light image sensor, and other types of image sensors. The image data of this embodiment can be multiple frames of video image data that constitute a video, and thus video image recognition can be completed.

[0038] The image recognition model of this embodiment can be any neural network model, such as CNN, which generally includes a convolution layer, a pooling layer, an activation function layer and a fully connected layer. The specific network architecture can be determined according to user needs. The image recognition model is used to identify the image to be identified and obtain the image recognition result. Image recognition refers to the technology of using computers to process, analyze and understand images to identify targets and objects of various different patterns. It is a practical application of deep learning algorithms. As an example, the image recognition of this embodiment can be used to identify faces in the image to be identified, and can also be used to identify an object in the image to be identified.

[0039] The image recognition model of this embodiment is constructed by RRAM and FPGA. According to the nature of the operations of each network layer in the image recognition model, each network layer in the image recognition model is divided into two categories. One category is the first network layer that completes the multiplication and accumulation operations. The first network layer may include a convolution layer and a fully connected layer. The other category is the second network layer that completes other operations. The second network layer may include a pooling layer and an activation function layer. The second network layer is the other network layer in the image recognition model except the first network layer. The first network layer is deployed in RRAM, and the operation of the first network layer in the image recognition model is completed by RRAM. The second network layer is deployed in FPGA, and the operation of the second network layer in the image recognition model is completed by FPGA. Through the interaction process between RRAM and FPGA, the image recognition model completes the recognition process of the image to be recognized and obtains the image recognition result.

[0040] In this embodiment, RRAM can be a 1T1R array structure composed of transistors and resistive switching devices. RRAM is the storage center and computing core of the image recognition system. The weights of the trained neural network (i.e., the image recognition model) will be stored in RRAM in a certain way, and then the image data will be input in a certain way to realize the multiplication and accumulation operations of vector and matrix multiplication. That is, the weights of the first network layer in the trained neural network are stored in RRAM. As the computing unit of the entire image recognition system, it realizes the function of in-memory computing, that is, completes the operation of the first network layer. At the same time, as a non-volatile memory, RRAM can also store weight data for a long time after power failure.

[0041] In this embodiment, the FPGA can use any model on the market that meets the logic resource requirements, wherein the logic resources are determined according to the image recognition system's requirements for the functions undertaken by the FPGA. The FPGA is the control center of the entire image recognition system. On the one hand, it provides control signals and configuration data for the image sensor and RRAM, and on the other hand, it processes the image data and controls the flow of the image data in the entire image recognition system. Specifically, the functions undertaken by the FPGA include: (1) implementing the control signal and configuration data generation logic according to the chip manual of the image sensor and RRAM, and providing control signals and configuration data for the image sensor and RRAM; (2) as a bridge for data transmission between the image sensor and RRAM, compressing the image data in the FPGA, adjusting the data transmission format, etc., using the internal storage resources of the FPGA to temporarily store the image data, and controlling the transmission timing of the image data between the image sensor and RRAM; (3) pulse coding control of the image data and outputting it to the RRAM; (4) implementing some calculations required by the neural network (i.e., image recognition model) algorithm, such as pooling, activation function, and other calculations that RRAM is not good at.

[0042] The image data collected by the image sensor is first transmitted to the FPGA. The FPGA can pre-process the original image data according to the data format required by the first network layer, and then send it to the RRAM. If necessary, the image data can be temporarily stored using the internal storage resources of the FPGA. At the same time, the transmission timing of the image data can be controlled, and the image data can be recognized in the order in which the image data is collected. During the RRAM calculation process, if the image recognition model involves more complex data processing, the data can also be sent back to the FPGA. After the FPGA has processed it (that is, after the FPGA has completed the operations required by the second network layer), it will be sent to the RRAM for the next step of calculation. The recognition process of the image recognition model is completed through the interaction between RRAM and FPGA.

[0043] like Figure 2As shown, it is a schematic diagram of the hardware design part of the FPGA. It should be noted that the functions implemented in the FPGA can be customized and developed according to the specifications of the image sensor and RRAM used.

[0044] The image sensor in this embodiment can be an external synchronous image sensor, so the control signals required by it need to be provided by FPGA, such as clock signal SCL, external frame synchronization signal FS_L and reset signal RST_L; FPGA can also configure the registers in the image sensor through the serial bus protocol, such as through the SDA signal, using the SPI protocol to input configuration data to the image sensor; the image sensor has an internal ADC, which can directly output digital image data. For this type of image sensor, Figure 2 The left part of is the image sensor control part, which is used to provide the control signal and configuration data required by the image sensor and receive the image data from the image sensor. The FPGA is used to send a first control signal and configuration data to the image sensor. The first control signal is used to control the image sensor. The first control signal includes a clock signal, an external frame synchronization signal and a reset signal. The clock signal is a signal quantity with a fixed period and is independent of operation. It is the basis of the timing logic and determines when the state in the logic unit is updated. The reset signal can restore the operation of the image sensor to the initial state after power-on. The configuration data is used to configure the image sensor. Specifically, for the image sensor used in this embodiment, the clock signal SCL is determined according to the chip manual of the image sensor, and the external frame synchronization signal FS_L is generated under the control of a group of column, row, and frame counters in the FPGA. When the image sensor is a 388×292 image sensor, the column counter counts to 388 as a row, and the row counter counts to 292 as a frame. Whenever a frame is counted, the FPGA sends an external frame synchronization signal to the image sensor to drive the internal logic of the image sensor to work in the correct timing. The SPI interface is used to send register configuration data to the image sensor, which is used to control the working mode and calibration process of the image sensor. Calibration refers to checking the functions that the image sensor should have to make it meet the standards. The configuration data can be set by the host computer. The image data input pre-processing will perform preliminary processing on the image data. When the image sensor in this embodiment outputs 16-bit pixel data in four cycles through a 4-bit digital output port, the input of the four cycles needs to be spliced ​​in the FPGA image data input pre-processing to obtain complete pixel data, that is, image data.

[0045] Figure 2The middle part is the image data processing part. The image sensor of this embodiment adopts a row-by-row and column-by-column output method, which is also the output method of most image sensors on the market. In the neural network, the data involved in the calculation each time is often not in a row-by-column order, so the data output by the image sensor cannot be directly sent to RRAM. Therefore, the FPGA of this embodiment is also used to pre-process the image to be identified according to the data format required by the first network layer, obtain the pre-processed data, and transmit the pre-processed data to RRAM, wherein the pre-processing includes format adjustment, that is, the adjustment of the data transmission format is completed. Taking the convolution layer of the convolutional neural network as an example, the data involved in the calculation each time is the data covered by the convolution window k×k, which is a square area on the image, so the image data needs to be formatted in the FPGA before it can be sent to RRAM. This embodiment can use a row cache solution. Taking the most common 3×3 convolution kernel in the current convolutional neural network as an example, assuming that the width of the input image is N, a shift register with a depth of 2N+3 is required. The image data is input from the head of the shift register. Each shift can directly take out a set of data covered by the convolution window from the shift register to obtain the pre-processed data.

[0046] The FPGA of this embodiment is also used to perform pulse encoding on the preprocessed data to obtain a pulse coded signal. At this time, the retrieved data enters the data encoding to generate a pulse coded signal. When the data is pulse encoded, the amplitude and / or width of the pulse signal can be encoded, and the encoding method can adopt the existing method.

[0047] The pooling and activation calculation units of the FPGA are used to implement calculations such as pooling and activation functions that RRAM is not good at. The calculation results of RRAM are sent back to the FPGA. After calculation by the pooling and activation function calculation units, they are resent to the data encoding for pulse encoding, and then sent to RRAM for new calculations. Through the interaction between RRAM and FPGA, the operations of the first network layer and the second network layer in the image recognition model are completed to realize image recognition.

[0048] The control signals required by the RRAM are also given by the FPGA. In this embodiment, the control signals required by the RRAM include a working state control signal STATE, a read / write trigger signal STROBE, a read output enable signal OEN, an address signal, and the like. Figure 2The right side of is the RRAM control part. The FPGA is used to send a second control signal to the RRAM. The second control signal is used to control the RRAM. The second control signal includes a working state control signal, a read / write trigger signal, a read output enable signal and an address signal. The address signal is used to select an array, a word line, a bit line and a source line in the RRAM. For the 1T1R RRAM used in this embodiment, the FPGA needs to send three groups of signals: (1) working mode control, i.e. Figure 2 The RRAM working state control in the byte includes the working state control signal STATE, which is used to control the RRAM to enter four different working states: forming, set, reset and read. Forming refers to the RRAM switching from the initial state to the low impedance state, set refers to the RRAM switching from the high impedance state to the low impedance state, and reset refers to the RRAM switching from the low impedance state to the high impedance state; the read and write trigger signal STROBE, when the working state is in set or reset, a rising edge input will trigger a read operation or a write operation; the read output enable signal OEN, when the working state is in the read state, when OEN is enabled, the read data will be output. (2) Address signal, i.e. Figure 2 The RRAM address is generated in the embodiment. For the RRAM of this embodiment, it includes 30 64×64 1T1R arrays, each of which has multiple word lines WL, multiple bit lines BL and multiple source lines SL. The address signal is used to select one of the arrays, one WL, one BL and one SL. (3) A set of input pulse control signals, namely Figure 2 The RRAM input pulse control in is generated according to the information encoded in the data, that is, the FPGA is used to generate the input pulse control signal based on the pulse coding signal, and control the RRAM to generate the input pulse based on the input pulse control signal, wherein the input pulse control signal includes a first signal for controlling the pulse amplitude of the word line, a second signal for controlling the pulse amplitude of the bit line, a third signal for controlling the pulse width of the word line, and a fourth signal for controlling the pulse width of the bit line. Specifically, the input pulse control signal includes two groups of Vg, Vb, and Vs, which are respectively used to control the pulse amplitude on the selected and unselected WL, BL, and SL, and the corresponding voltage can be directly applied thereon; a group of Vg_pw, Vb_pw, and Vs_pw, which are used to control the pulse width applied to WL, BL, and SL, and pulses of different widths are generated by applying different voltages thereon. In order to achieve precise control of the input pulse, the FPGA outputs a certain digital value according to the input pulse control signal, which is converted into a corresponding voltage through an external DAC, thereby realizing the control of the RRAM input pulse and enabling the RRAM to generate an input pulse. The RRAM calculation result input is used to send the RRAM calculation result back to the FPGA, and the FPGA completes the operation of the second network layer.

[0049] Figure 2 The lower part is the temporary debugging interface part, which is a serial port (i.e., serial debugging interface) designed to communicate with the host computer in order to facilitate debugging and testing of the image recognition system during the development stage. Image data can be output through the serial port and observed on the host computer, or the host computer can input register configuration data to the image sensor through the serial port. In addition, the FPGA receives the output data from the RRAM and returns it to the host computer to help developers observe the reading or calculation results. It should be noted that the temporary debugging interface part is only used in the development stage or troubleshooting, and can be designed according to the actual situation. This part of the function will not be used in the final product.

[0050] The FPGA of this embodiment is also used to perform at least one of the following operations: compression, temporary storage, and transmission timing control on the image to be recognized. When the image sensor captures images at a rate greater than the rate at which the FPGA and RRAM process the images, the FPGA can temporarily store the image to be recognized and perform transmission timing control on the image to be recognized in the order in which the image to be recognized is received, so that the image to be recognized can be processed sequentially in the acquisition order (the same order as the image to be recognized is received).

[0051] like Figure 3 As shown, it is a schematic diagram of the RRAM chip architecture used in this embodiment. It should be noted that the RRAM of this embodiment can use different specifications. Taking the RRAM containing 30 4K 1T1R arrays, each with a 64×64 structure as an example, the RRAM chip implements address decoding logic inside, and can directly select a certain storage unit (located in a certain array, a certain word line, a certain bit line and a certain source line) by inputting the address. The chip implements pulse control logic inside, and the amplitude and width of the input pulse can be controlled by externally applying voltage. The main ports of the chip include:

[0052] (1) Global power supply signal, that is Figure 3 Various power signals in the RRAM chip, including VDD and VSS, are used to provide operating voltage for the internal logic of the RRAM chip.

[0053] (2) State control signal. A set of digital signals, including: state, that is, Figure 2 STATE in is used to control the working state of the chip, including forming, set, reset, and read; strobe, that is, Figure 2 STROBE in, according to the state setting, when the state is set or reset, a rising edge triggers a read or write operation; oen, that is Figure 2 In the read state, OEN in the pin is applied high to enable the read output port and output the read data.

[0054] (3) Selection signal. A group of digital signals, which can be understood as a group of address signals for FPGA, including: ip_sel, selects one of the 30 arrays; wl_sel, selects a WL; bl_sel, selects a BL; sl_sel, selects a SL.

[0055] (4) Read and write pulse configuration signals, that is Figure 3 Input pulse control in. A group of analog signals, including pulse amplitude configuration and pulse width configuration, are used to configure the amplitude and pulse width of the pulses applied to WL, BL and SL. FPGA can assign values ​​to this group of signals through an external DAC, including: Vg1 / Vg2, setting the pulse amplitude on the selected / unselected WL; Vb1 / Vb2, setting the pulse amplitude on the selected / unselected BL; Vs1 / Vs2, setting the pulse amplitude on the selected / unselected SL; Vg_pw, setting the pulse width on WL; Vb_pw, setting the pulse width on BL; Vs_pw, setting the pulse width on SL.

[0056] (5) Output signal. It includes a digital output signal out, which is used to output the data or calculation results (i.e., read data) stored in the RRAM; and a set of pulse detection signals pulse_moniter, which are used to detect the pulse signals on WL, BL, and SL.

[0057] like Figure 4 As shown in the figure, it is a schematic diagram of the principle of deploying weights and implementing multiplication and accumulation on RRAM. First, different weight values ​​must be quantized. In this embodiment, 4-bit quantization is used, so 0 to 15 corresponds to a total of 16 conductance values, represented as g0 to g15. Taking a 3×3 convolution kernel as an example, Figure 4In the array structure shown, the weight values ​​are deployed along the row direction. In order to realize the deployment of positive and negative weights, each weight is deployed on two resistive switching devices in two rows, representing positive conductance +g and negative conductance -g respectively. Therefore, 3×3 total 9 weight values ​​will be deployed on two rows total 18 resistive switching devices, and the remaining resistive switching devices on these two rows are all encoded with the conductance value corresponding to 0. That is, since resistive switching devices cannot deploy negative values, this embodiment deploys one weight through two rows. Specifically, the resistive switching device in the storage unit of the i-th row and j-th column of the RRAM is deployed with a first weight value, and the resistive switching device in the storage unit of the i+1-th row and j-th column is deployed with a second weight value. The difference between the first weight value and the second weight value is equal to the value of the j-th weight of the first network layer to complete the deployment of the weight of the first network layer, wherein the first weight value and the second weight value are both greater than or equal to 0; the input data is a pulse on the BL (i.e., an input pulse). Through pulse coding, different input values ​​correspond to pulses of different amplitudes and widths, and different charge accumulations will be generated after passing through the resistive switching device to complete the multiplication operation. The charges on each resistive switching device in a row are accumulated on the SL to complete the addition operation. Finally, the charges on the +g row and the -g row are subtracted to obtain a set of multiplication and accumulation results, so that the RRAM is used to complete the operation of the first network layer based on the weights and input pulses of the first network layer.

[0058] like Figure 5 As shown in the figure, it is a schematic diagram of the process of running a convolutional neural network when realizing video image recognition. A convolutional neural network includes a convolution layer with a convolution kernel size of 3×3, a pooling layer with a pooling window of 2×2, and a fully connected layer as an example, and it is assumed that the input image has been downsampled by the previous layers and the size is 10×10.

[0059] (1) The weights of the convolutional neural network are Figure 4 The method is deployed on RRAM, assuming that the weights of the convolutional layer are deployed on two rows of resistive switching devices at addresses 0x0000 and 0x0001, and the weights of the fully connected layer are deployed on two rows of resistive switching devices at addresses 0x0100 and 0x0101;

[0060] (2) The image data collected by the image sensor is sent to the FPGA and pre-processed;

[0061] (3) The pre-processed image data is sent to the line buffer;

[0062] (4) Take out a convolution window of data from the line buffer and send it to the data encoder to generate pulse control information (i.e., input pulse control signal);

[0063] (5) The pulse control information is sent to the RRAM. At the same time, the address control unit sends the address corresponding to this input to the RRAM. The pulse control logic inside the RRAM generates an input pulse on the BL corresponding to the address and performs the calculation.

[0064] (6) RRAM returns the calculated results to FPGA;

[0065] (7) FPGA performs activation functions, pooling and other calculations, and caches the frame data;

[0066] (8) Since the next layer is calculated as a fully connected layer, it does not need to go through the row cache, so it can be directly sent to the data encoding to generate pulse control information;

[0067] (9) The pulse control information is sent to the RRAM, and the address corresponding to the weight of the fully connected layer is sent to the RRAM, driving the RRAM to perform calculations and complete the image recognition process.

[0068] The image recognition system of this embodiment has the following three advantages:

[0069] First, the structure is simple and easy to build. The image recognition system only includes image sensor, RRAM and FPGA in hardware.

[0070] Second, it can be customized to a certain extent, with high versatility and flexibility. This advantage is mainly brought by FPGA, which is flexible and programmable. It can be customized for different types and specifications of image sensors and RRAM, as well as different neural networks, and can adapt to the needs of various scenarios, that is, it is highly versatile; it can also be redeveloped according to changes in demand, that is, it is highly flexible.

[0071] Third, the computing efficiency is high and the power consumption is low. The array structure of RRAM itself is naturally suitable for high-parallelism calculations of multiplication and accumulation. The characteristic of RRAM for calculation in the analog domain also makes its power consumption very low. FPGA is a low-frequency, high-parallelism chip, and the logic resource utilization of FPGA in this system is relatively low compared to the total logic resource utilization of most FPGA chips. As a control center, FPGA also has lower power consumption than CPU, GPU and other chips, which can solve the problem that computing units such as CPU, GPU or AI dedicated chips have high power consumption as digital chips, and the cost of GPU and AI dedicated chips themselves is also relatively high.

[0072] Example 2

[0073] A method for image recognition based on RRAM with integrated sensing, storage and computing, controlling the RRAM-based image recognition system with integrated sensing, storage and computing described in Example 1 to work, comprising:

[0074] An image recognition model is constructed using RRAM and FPGA, and the image recognition model is used to recognize an image to be recognized acquired by an image sensor to obtain an image recognition result; wherein the RRAM is used to complete the operation of the first network layer in the image recognition model, and the first network layer is a network layer that completes multiplication and accumulation operations; and the FPGA is used to complete the operation of the second network layer in the image recognition model, and the second network layer is a network layer other than the first network layer in the image recognition model.

[0075] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0076] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A sensor-storage-computation integrated image recognition system based on RRAM, characterized in that: include: Image sensors, RRAM, and FPGAs; The image sensor is used to collect the image to be identified; The RRAM and the FPGA can build an image recognition model, and the image recognition model is used to recognize the image to be recognized to obtain an image recognition result; wherein the RRAM is used to complete the operation of the first network layer in the image recognition model, and the first network layer is a network layer that completes multiplication and accumulation operations; the FPGA is used to complete the operation of the second network layer in the image recognition model, and the second network layer is a network layer other than the first network layer in the image recognition model; The FPGA is also used to pre-process the image to be identified according to the data format required by the first network layer to obtain pre-processed data, and transmit the pre-processed data to the RRAM; wherein the pre-processing includes format adjustment; The FPGA is further used to pulse encode the preprocessed data to obtain a pulse encoding signal, generate an input pulse control signal based on the pulse encoding signal, and control the RRAM to generate an input pulse based on the input pulse control signal; wherein the input pulse control signal includes a first signal for controlling a pulse amplitude of a word line, a second signal for controlling a pulse amplitude of a bit line, a third signal for controlling a pulse width of a word line, and a fourth signal for controlling a pulse width of a bit line; The storage unit in the i-th row and j-th column of the RRAM is deployed with a first weight value, and the storage unit in the i+1-th row and j-th column is deployed with a second weight value, and the difference between the first weight value and the second weight value is equal to the value of the j-th weight of the first network layer, so as to complete the deployment of the weight of the first network layer; wherein the first weight value and the second weight value are both greater than or equal to 0; The RRAM is used to complete the operation of the first network layer based on the weight of the first network layer and the input pulse.

2. The RRAM-based sensor-storage-computation integrated image recognition system according to claim 1, characterized in that: The first network layer includes a convolutional layer and a fully connected layer; the second network layer includes a pooling layer and an activation function layer.

3. The RRAM-based image recognition system of claim 1, characterized in that: The FPGA is also used to send a first control signal and configuration data to the image sensor, the first control signal is used to control the image sensor, the first control signal includes a clock signal, an external frame synchronization signal and a reset signal, and the configuration data is used to configure the image sensor.

4. The RRAM-based sensor-storage-computation integrated image recognition system according to claim 1, characterized in that: The FPGA is also used to send a second control signal to the RRAM, the second control signal is used to control the RRAM, the second control signal includes a working state control signal, a read / write trigger signal, a read output enable signal and an address signal; the address signal is used to select an array, a word line, a bit line and a source line in the RRAM.

5. The RRAM-based sensor-storage-computation integrated image recognition system according to claim 1, characterized in that: The FPGA is also used to perform at least one operation of compressing, temporarily storing, and transmitting timing control on the image to be identified.

6. A method for image recognition based on RRAM with integrated sensing, storage and computing, controlling the operation of an image recognition system based on RRAM with integrated sensing, storage and computing as claimed in any one of claims 1 to 5, characterized in that: include: RRAM and FPGA are used to build an image recognition model, which is used to recognize the image to be recognized collected by the image sensor to obtain an image recognition result; wherein the RRAM is used to complete the operation of the first network layer in the image recognition model, and the first network layer is a network layer that completes multiplication and accumulation operations; The FPGA is used to complete the operation of the second network layer in the image recognition model, and the second network layer is the other network layer in the image recognition model except the first network layer.

Images