Neural network processing device, electronic apparatus, and operation method

Abstract

A neural network processing device, an electronic apparatus, and an operation method The neural network processing device comprises a storage module, an input data processing module, a plurality of matrix compute-in-memory units, an output data processing module, and a control module, wherein the storage module comprises: a first storage sub-module configured to store an input feature map for neural network processing, and a second storage sub-module configured to store an intermediate feature map for neural network processing; and the plurality of matrix compute-in-memory units are configured to store the weight of at least one network layer of a neural network and use the weight to perform matrix-vector multiplication computation of the neural network. The neural network processing device can be applied to a compute-in-memory system for image processing, thereby lowering the computation delay, and reducing the area and power consumption.

Description

Neural network processing device, electronic device and operating method

[0001] This application claims priority to Chinese Patent Application No. 202311499050.4 filed on November 10, 2023, and the contents of the above-mentioned Chinese patent application disclosure are hereby cited in their entirety as part of this application. Technical Field

[0002] Embodiments of the present disclosure relate to a neural network processing device, an electronic device, and an operating method. Background Art

[0003] With the advent of the artificial intelligence era, AI technology has been widely used in the field of image sensing, covering intelligent recognition tasks such as face recognition and object detection, as well as image enhancement tasks such as low-light enhancement and noise reduction. On the one hand, image sensing applications are often deployed on edge devices and require high computational complexity and latency, which poses new challenges to the area, power consumption, and computing power of AI chips. On the other hand, as edge devices continue to optimize for reducing latency, reducing chip area, protecting user privacy, and lowering manufacturing costs, integrating AI chips with image sensor chips is considered an important direction for the development of edge artificial intelligence. However, after integration, AI chips have difficulty accessing off-chip memory, and image processing algorithms require storing large feature maps, which results in a lot of on-chip memory access overhead. This poses a challenge to the architectural design of edge AI chips.

[0004] Currently, AI chips that combine storage and computing with memristors are typically optimized for high computing power. A common example is the fully pipelined architecture, which has high computational parallelism but complex routing and control logic, high power consumption, and low chip utilization, making it unsuitable for edge applications. Another type of architecture optimized for low power consumption is the layer-by-layer computing architecture. Layer-by-layer computing works well in lightweight networks (such as speech recognition), but for image processing tasks, which face a large number of feature maps, this architecture requires a larger storage structure, which results in excessive storage area and poor versatility. At the same time, due to the large computational complexity of image processing tasks and the large number of network layers, the layer-by-layer computing architecture also suffers from excessive latency.

[0005] Summary of the Invention

[0006] At least some embodiments of the present disclosure provide a neural network processing device, including a storage module, an input data processing module, multiple matrix storage and computing units, an output data processing module, and a control module, wherein the storage module includes: a first storage submodule, configured to store input feature maps for neural network processing, and a second storage submodule, configured to store intermediate feature maps for neural network processing; the input data processing module is configured to obtain input data from the storage module and process it into a data structure for input of multiple matrix storage and computing units; the multiple matrix storage and computing units are configured to store weights of at least one network layer of the neural network and use the weights to perform matrix-vector multiplication calculations of the neural network; the output data processing module is configured to process data output by the multiple matrix storage and computing units; and the control module is configured to control the storage module, the input data processing module, the matrix storage and computing units, and the output data processing module according to configuration information.

[0007] For example, in a neural network processing device provided by at least some embodiments of the present disclosure, the input data processing module includes an input register, and the input data processing module is configured to obtain input data from a corresponding storage address and store the input data in the input register after processing under the control of the control module.

[0008] For example, in a neural network processing device provided by at least some embodiments of the present disclosure, each of a plurality of matrix storage and computing units includes a memristor array, a bit line driving circuit, a word line / source line driving circuit, and a digital-to-analog converter, wherein the memristor array is configured to store the weights of at least one network layer of the neural network and use the weights to perform matrix-vector multiplication calculations of the neural network, the bit line driving circuit is configured to provide a bit line signal to the memristor array, the word line / source line driving circuit is configured to provide a word line signal / source line signal to the memristor array, and the bit line driving circuit, the word line / source line driving circuit, and the digital-to-analog converter are further configured to receive a control signal provided by a control module.

[0009] For example, in a neural network processing device provided by at least some embodiments of the present disclosure, the memristor array includes a first memristor array and a second memristor array, and the first memristor array and the second memristor array share the same bit line driving circuit.

[0010] For example, in a neural network processing device provided by at least some embodiments of the present disclosure, an output data processing module includes an output register, and the output data processing module is configured to process the output data into the same data format as the data in the storage module and store it in the output register, and store the value of the output register in the storage module.

[0011] For example, in a neural network processing device provided by at least some embodiments of the present disclosure, the control module includes a control logic generation module, a configuration information storage module and a control logic state machine. The control logic generation module is configured to generate control signals and selection signals for at least one of the storage module, the input data processing module, multiple matrix storage and computing units and the output data processing module according to the configuration information. The configuration information storage module is configured to store configuration information, and the configuration information includes at least one of the degree of parallelism of each layer of the neural network, the number of pipeline stages, the network structure, the address at which the input data processing module obtains input data, and the address at which the output data processing module stores data. The control logic state machine is configured to switch the working state according to the handshake signals with the storage module, the input data processing module, the matrix storage and computing unit and the output data processing module.

[0012] At least some embodiments of the present disclosure also provide an electronic device comprising the neural network processing device provided by any embodiment of the present disclosure.

[0013] At least some embodiments of the present disclosure also provide an operating method for a neural network processing device, including: storing an input feature map in a first storage submodule, writing the weights of at least one network layer of the neural network into at least one object matrix storage unit among multiple matrix storage units, obtaining input data from the storage module through an input data processing module and processing it into a data structure for input to at least one object matrix storage unit, performing matrix-vector multiplication operations of the neural network by at least one object matrix storage unit, and processing data output by the matrix-vector multiplication operation performed by at least one object matrix storage unit through an output data processing module.

[0014] For example, in an operating method for a neural network processing device provided by at least some embodiments of the present disclosure, at least one object matrix storage unit includes a first matrix storage unit and a second matrix storage unit, wherein the weight of the i-th layer network of the neural network is written into the first matrix storage unit, and the weight of the i+1-th layer network of the neural network is written into the second matrix storage unit, i is a positive integer, and in a first time period, the i-th layer network is calculated, the input data processing module obtains the first input data from the first storage submodule, and inputs the first input data into the first matrix storage unit after being processed to perform a first operation to obtain the first output data, and the first output data is processed by the output data processing module and stored in the second storage submodule, and in a second time period, The i+1 layer network is calculated, the input data processing module obtains the second input data from the first storage submodule and the second data storage submodule, and inputs the first input data into the first matrix storage and calculation unit and the second matrix storage and calculation unit after being processed to perform a second operation to obtain second output data, and the second output data is processed by the output data processing module and stored in the second storage submodule. In the third time period, the calculation of the i-th layer network is completed, and the calculation of the i+1-th layer is continued. The input data processing module obtains the third input data from the second storage submodule, and inputs the third input data into the second matrix storage and calculation unit after being processed to perform a third operation to obtain third output data, and the third output data is processed by the output data processing module and stored in the first storage submodule.

[0015] For example, in an operating method for a neural network processing device provided by at least some embodiments of the present disclosure, at least one object matrix storage unit includes a first matrix storage unit and a second matrix storage unit, wherein the weights of the i-th layer network and the i+1-th layer network of the neural network are copied and written into the first matrix storage unit and the second matrix storage unit, and the weights of the i-th layer network and the i+1-th layer network are alternately arranged in the first matrix storage unit and the second matrix storage unit, i is a positive integer, in a first time period, the i-th layer network is calculated, the input data processing module obtains first input data from the first storage submodule, uses the first matrix storage unit and the second matrix storage unit to perform calculations to obtain first output data, the first output data is processed by the output data processing module and then stored in the first storage submodule, in a second time period, the i+1-th layer network is calculated, the input data processing module obtains second input data from the first storage submodule, uses the first matrix storage unit and the second matrix storage unit to perform second calculations to obtain second output data, and the second output data is processed by the output data processing module and then stored in the first storage submodule. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, rather than limiting the present disclosure.

[0017] FIG1A is a schematic diagram of a memristor unit in a memristor array according to some embodiments of the present disclosure;

[0018] FIG1B is a schematic diagram of a memristor unit in a memristor array according to some other embodiments of the present disclosure;

[0019] FIG2 is a schematic diagram of the structure of a memristor storage and computing device;

[0020] FIG3 is a schematic diagram of the structure of a neural network processing device provided by some embodiments of the present disclosure;

[0021] FIG4A is a schematic diagram of the structure of a matrix storage and calculation unit provided in some embodiments of the present disclosure;

[0022] FIG4B is a schematic structural diagram of a matrix storage and calculation unit provided in some other embodiments of the present disclosure;

[0023] FIG5 is a schematic flowchart of an operating method of a neural network processing device provided by some embodiments of the present disclosure;

[0024] FIG6 is a schematic block diagram of an operating method of a neural network processing device provided by some embodiments of the present disclosure;

[0025] FIG7 is a schematic block diagram of an operating method of a neural network processing device provided by other embodiments of the present disclosure; and

[0026] FIG8 is a schematic diagram of multiplexing input data according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0028] Unless otherwise defined, the technical or scientific terms used in this disclosure should have the usual meanings understood by persons of ordinary skill in the field to which this disclosure belongs. The words "first", "second" and similar terms used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0029] The present disclosure is described below using several specific embodiments. To keep the following description of the embodiments of the present disclosure clear and concise, detailed descriptions of known functions and components may be omitted. When any component of an embodiment of the present disclosure appears in more than one drawing, the component is represented by the same or similar reference numeral in each drawing.

[0030] Memristors (e.g., resistive random access memory, phase-change memory, conductive bridge memory, etc.) are non-volatile devices whose conductance state can be adjusted by applying an external stimulus. As two-terminal devices, memristors offer adjustable resistance and are non-volatile, making them widely used in integrated storage and computing. According to Kirchhoff's current law and Ohm's law, an array of memristors can perform multiplication and accumulation calculations in parallel, with both storage and computation occurring within each device in the array. This computing architecture enables integrated storage and computing without requiring large amounts of data movement.

[0031] For example, Figure 1A shows a schematic diagram of a memristor unit in a memristor array according to some embodiments of the present disclosure. As shown in Figure 1A , the memristor unit has a 1T1R structure, including a transistor M1 and a memristor R1.

[0032] For example, when the transistor M1 is an N-type transistor, its gate is connected to the word line terminal WL, for example, when a high level is input to the word line terminal WL, the transistor M1 is turned on; the first electrode of the transistor M1 can be a source electrode and is configured to be connected to the source line terminal SL, for example, the transistor M1 can receive a reset voltage through the source line terminal SL; the second electrode of the transistor M1 can be a drain electrode and is configured to be connected to the second electrode (for example, the negative electrode) of the memristor R1, and the first electrode (for example, the positive electrode) of the memristor R1 is connected to the bit line terminal BL, for example, the memristor R1 can receive a set voltage through the bit line terminal BL. For example, when transistor M1 is a P-type transistor, its gate is connected to the word line terminal WL. For example, when the word line terminal WL is input with a low level, transistor M1 is turned on. The first electrode of transistor M1 can be a drain and configured to be connected to the source line terminal SL. For example, transistor M1 can receive a reset voltage through the source line terminal SL. The second electrode of transistor M1 can be a source and configured to be connected to the second electrode (for example, the negative electrode) of memristor R1. The first electrode (for example, the positive electrode) of memristor R1 is connected to the bit line terminal BL. For example, memristor R1 can receive a set voltage through the bit line terminal BL. It should be noted that the resistive memory structure can also be implemented as other structures, such as a structure in which the second electrode of memristor R1 is connected to the source line terminal SL, and the embodiments of the present disclosure are not limited to this.

[0033] The following description is made by taking the transistor M1 as an N-type transistor as an example.

[0034] The function of the word line terminal WL is to apply a corresponding voltage to the gate of the transistor M1, thereby controlling whether the transistor M1 is turned on or off. When operating the memristor R1, such as performing a set operation or a reset operation, it is necessary to first turn on the transistor M1, that is, it is necessary to apply a turn-on voltage to the gate of the transistor M1 through the word line terminal WL. After the transistor M1 is turned on, for example, a voltage can be applied to the memristor R1 through the source line terminal SL and the bit line terminal BL to change the resistance state of the memristor R1. For example, a set voltage can be applied through the bit line terminal BL to put the memristor R1 in a low-resistance state; for another example, a reset voltage can be applied through the source line terminal SL to put the memristor R1 in a high-resistance state. For example, the resistance value of the high-resistance state is more than 100 times the resistance value of the low-resistance state, for example, more than 1000 times.

[0035] It should be noted that in the embodiments of the present disclosure, for example, by simultaneously applying voltages to the word line terminal WL and the bit line terminal BL, the resistance value of the memristor R1 can be made smaller and smaller, that is, the memristor R1 changes from a high resistance state to a low resistance state. The operation of changing the memristor R1 from a high resistance state to a low resistance state is referred to as a set operation. By simultaneously applying voltages to the word line terminal WL and the source line terminal SL, the resistance value of the memristor R1 can be made larger and larger, that is, the memristor R1 changes from a low resistance state to a high resistance state. The operation of changing the memristor R1 from a low resistance state to a high resistance state is referred to as a reset operation. For example, the memristor R1 has a threshold voltage. When the input voltage amplitude is less than the threshold voltage of the memristor R1, the resistance value (or conductance value) of the memristor R1 does not change. In this case, the resistance value (or conductance value) of the memristor R1 can be used for calculation by inputting a voltage less than the threshold voltage; the resistance value (or conductance value) of the memristor R1 can be changed by inputting a voltage greater than the threshold voltage.

[0036] FIG1B is a schematic diagram of a memristor cell in a memristor array according to another embodiment of the present disclosure. As shown in FIG1B , the memristor cell has a 2T2R structure, including two transistors M1 and M2 and two memristors R1 and R2. The following description uses an example in which both transistors M1 and M2 are N-type transistors.

[0037] The gate of the transistor M1 is connected to the word line terminal WL1. For example, when a high level is input to the word line terminal WL1 of M1, the transistor M1 is turned on. The gate of the transistor M2 is connected to the word line terminal WL2. For example, when a high level is input to the word line terminal WL2 of M2, the transistor M2 is turned on. The first electrode of the transistor M1 can be a source and is configured to be connected to the source line terminal SL. For example, the transistor M1 can receive a reset voltage through the source line terminal SL. The first electrode of the transistor M2 can be a source and is configured to be connected to the source line terminal SL. For example, the transistor M2 can receive a reset voltage through the source line terminal SL. The first electrode of the transistor M1 is connected to the first electrode of the transistor M2, and is connected together to the source line terminal SL. The second electrode of the transistor M1 can be a drain and is configured to be connected to the second electrode (for example, the negative electrode) of the memristor R1, and the first electrode (for example, the positive electrode) of the memristor R1 is connected to the bit line terminal BL1. For example, the memristor R1 can receive a set voltage through the bit line terminal BL1; the second electrode of the transistor M2 can be a drain and is configured to be connected to the second electrode (for example, the negative electrode) of the memristor R2, and the first electrode (for example, the positive electrode) of the memristor R2 is connected to the bit line terminal BL2. For example, the memristor R2 can receive a set voltage through the bit line terminal BL2.

[0038] It should be noted that the transistors M1 and M2 in the 2T2R structure memristor unit may also be P-type transistors, which will not be described in detail here.

[0039] The memristor array provided by the embodiments of the present disclosure may include a plurality of memristor units arranged in an array. For example, according to Kirchhoff's law, the output current of the memristor array can be obtained according to the following formula:

[0040] Wherein, i=1,…,M, j=1,…,n, and n and M are both integers greater than 1.

[0041] In the above formula, v i Represents the voltage excitation of neuron node i in the input layer, i j represents the output current of neuron node j in the next layer, g i,j Represents the conductance matrix of the memristor array.

[0042] For example, a memristor array has a threshold voltage. When the input voltage amplitude is less than the threshold voltage of the memristor array, the conductance of the memristor array does not change. In this case, the conductance of the memristor can be used for calculations by inputting a voltage less than the threshold voltage; the conductance of the memristor can be changed by inputting a voltage greater than the threshold voltage.

[0043] Figure 2 is a schematic diagram of the structure of a memristor storage and computing device. The memristor device includes a memristor (sub) array and its peripheral drive circuit, which is used to implement the input and output module. For example, as shown in Figure 2, the memristor device includes a signal acquisition device, a word line drive circuit, a bit line drive circuit, a source line drive circuit, a memristor array, and a data output circuit.

[0044] For example, the signal acquisition device is configured to convert the digital signal into a plurality of first analog signals through a digital to analog converter (DAC) so as to be input to a plurality of column signal input terminals of the memristor array during convolution processing.

[0045] For example, a memristor array includes M source lines, M word lines, and N bit lines, as well as a plurality of memristor cells arranged in M ​​rows and N columns (M and N are positive integers). For example, as shown in FIG1A , each memristor cell has a 1T1R structure (or a 2T2R structure), and a parameter matrix for Fourier transform can be mapped to the plurality of memristor cells in the memristor array.

[0046] For example, the operation of the memristor array is achieved through a word line driver circuit, a bit line driver circuit, and a source line driver circuit.

[0047] For example, the word line driver circuit includes multiple multiplexers (Mux) for switching word line input voltages; the bit line driver circuit includes multiple multiplexers for switching bit line input voltages; and the source line driver circuit also includes multiple multiplexers (Mux) for switching source line input voltages. For example, the source line driver circuit also includes multiple analog-to-digital converters (ADCs) for converting analog signals into digital signals. In addition, a trans-impedance amplifier (TIA) (not shown) can be further provided between the Mux and ADC in the source line driver circuit to perform current-to-voltage conversion for ADC processing.

[0048] For example, a memristor array includes an operating mode and a computing mode. When the memristor array is in the operating mode, the memristor cells are initialized, and the values ​​of the parameter elements in the parameter matrix can be written into the memristor array. For example, the source line input voltage, bit line input voltage, and word line input voltage of the memristor are switched to corresponding preset voltage ranges through a multiplexer.

[0049] For example, the word line input voltage is switched to a corresponding voltage range by the control signal WL_sw[1:M] of the multiplexer in the word line driving circuit in FIG2 . For example, when performing a set operation on the memristor, the word line input voltage is set to 2V (volts), and when performing a reset operation on the memristor, the word line input voltage is set to 5V. For example, the word line input voltage can be obtained by the voltage signal V_WL[1:M] in FIG2 .

[0050] For example, the source line input voltage is switched to a corresponding voltage range by the control signal SL_sw[1:M] of the multiplexer in the source line driving circuit in FIG2 . For example, when performing a set operation on the memristor, the source line input voltage is set to 0V, and when performing a reset operation on the memristor, the source line input voltage is set to 2V. For example, the source line input voltage can be obtained by the voltage signal V_SL[1:M] in FIG2 .

[0051] For example, the bit line input voltage is switched to a corresponding voltage range by the control signal BL_sw[1:N] of the multiplexer in the bit line driving circuit in FIG2 . For example, when performing a set operation on the memristor, the bit line input voltage is set to 2V, and when performing a reset operation on the memristor, the bit line input voltage is set to 0V. For example, the bit line input voltage can be obtained by the DAC in FIG2 .

[0052] For example, when the memristor array is in computing mode, the memristors in the memristor array are in a conductive state that can be used for computing, and the bit line input voltage input to the column signal input terminal does not change the conductance value of the memristor. For example, multiplication and addition operations can be performed through the memristor array to complete the calculation. For example, the word line input voltage is switched to a corresponding voltage range by the control signal WL_sw[1:M] of the multiplexer in the word line driving circuit in Figure 2, for example, when the start signal is applied, the word line input voltage of the corresponding row is set to 5V, for example, when the start signal is not applied, the word line input voltage of the corresponding row is set to 0V, for example, the GND signal is connected; the source line input voltage is switched to a corresponding voltage range by the control signal SL_sw[1:M] of the multiplexer in the source line driving circuit in Figure 2, for example, the source line input voltage is set to 0V, so that the current signals of the multiple row signal output ends can flow into the data output circuit; the bit line input voltage is switched to a corresponding voltage range by the control signal BL_sw[1:N] of the multiplexer in the bit line driving circuit in Figure 2, for example, the bit line input voltage is set to 0.1V-0.3V, so as to complete the convolution operation by utilizing the characteristic that the memristor array can perform multiplication and addition operations.

[0053] For example, the data output circuit may include multiple TIAs and ADCs, which may convert current signals at multiple row signal output terminals into voltage signals, and then convert them into digital signals for subsequent processing.

[0054] Memristor-based AI chips with integrated computing and storage, such as a fully pipelined architecture optimized for high computing power, typically deploy a network layer per compute unit, storing feature maps in the cache as buffer lines. Each compute unit has its own independent cache. Deploying a single network layer per compute unit results in low compute unit utilization. The number of required compute units is related to the network size and depth, resulting in a large chip area when a large number of units are required. Furthermore, this architecture has complex control logic and wiring, resulting in large routing area and high power consumption. Another architecture optimized for low power consumption is the layer-by-layer computing architecture. In this architecture, different networks can be deployed on the same compute unit. Only one compute unit is activated at a time, and the feature maps output by each layer must be stored. Therefore, the required storage capacity is tightly coupled to the network structure, resulting in excessive storage area and poor versatility. Because the network is computed one layer at a time, latency can be excessive for algorithms with high computational complexity and a large number of network layers, such as image processing.

[0055] The present disclosure provides a neural network processing device, comprising a storage module, an input data processing module, a plurality of matrix storage and calculation units, an output data processing module, and a control module. The storage module comprises: a first storage submodule configured to store input feature maps for neural network processing, and a second storage submodule configured to store intermediate feature maps for neural network processing; the input data processing module is configured to obtain input data from the storage module and process it into a data structure for input to the plurality of matrix storage and calculation units; the plurality of matrix storage and calculation units are configured to store weights of at least one network layer of the neural network and use the weights to perform matrix-vector multiplication calculations of the neural network; the output data processing module is configured to process data output by the plurality of matrix storage and calculation units; and the control module is configured to control the storage module, the input data processing module, the matrix storage and calculation units, and the output data processing module according to configuration information. Compared with the traditional fully pipelined, layer-by-layer architecture, the neural network processing device provided by the above embodiments of the present disclosure uses configuration information to perform parallel or pipeline processing on only key network layers, thereby compressing feature maps, reducing computational latency, improving chip utilization, and saving area and power consumption.

[0056] Figure 3 shows the structure of a neural network processing device provided by some embodiments of the present disclosure. As shown in Figure 3, the neural network processing device includes a storage module, an input data processing module, multiple matrix storage and calculation units, an output data processing module, and a control module.

[0057] For example, the storage module may include two storage submodules, a first storage submodule and a second storage submodule. The embodiments of the present disclosure do not limit the implementation of the storage submodules. For example, the two can be physically separated, can share peripheral drive circuits or have dedicated peripheral drive circuits. For example, the two can be static random access memories (SRAMs). For example, the storage capacities of the two can be different. For example, the first sub-storage module is configured as an SRAM with a larger capacity to store the input feature maps processed by the neural network, for example, to store the complete feature maps. The specific capacity of the first storage submodule can be determined according to the image resolution requirements of the application scenario. For example, the capacity of the first storage submodule can be set to the order of 1MB to 10MB. For example, the second sub-storage module is configured as an SRAM with a smaller capacity, for example, to store intermediate feature maps processed by the neural network, for example, to configure the buffer rows of the feature maps output by the matrix storage and calculation unit in the pipeline processing mode. The bandwidth of the second storage submodule should ensure that the delay in transmitting data is acceptable when the number of pipeline stages reaches the maximum. For example, the capacity of the second storage submodule is set to the order of several hundred KB.

[0058] Under the control of the control module, the input data processing module is responsible for fetching data from the corresponding storage address, performing bit division and rearrangement on the data, and combining it into a data structure suitable for input by multiple matrix storage and computation units. For example, the input data processing module may include an input register, which processes the input data obtained from the corresponding storage address and then stores it in the input register.

[0059] A plurality of matrix storage units are configured to store the weights of at least one network layer of a neural network and use the weights to perform matrix-vector multiplication operations of the neural network. The neural network structure shown in FIG3 includes four matrix storage units. It should be noted that the present disclosure does not limit the number of matrix storage units. The number of matrix storage units can be determined by the network weight scale of the application scenario. For example, for edge applications, the number of matrix storage units is generally 1 to 10. The embodiments of the present disclosure do not limit the connection method between matrix storage units. For example, when the number of matrix storage units is small, such as when the number of matrix storage units is four or less, the matrix storage units are interconnected in a point-to-point manner such as a cross matrix. When the number of matrix storage units is large, they are interconnected in an efficient routing manner such as a data bus or a network on chip (NoC).

[0060] Under the control of the control module, the output data processing module is responsible for processing the data output by the multiple matrix storage and calculation units. For example, it pools and activates the output data according to the network configuration, and according to the data processing rules of the input data processing module, restores the output data of the matrix storage and calculation units to the same data format as the data stored in the storage module, for example, performing operations such as shift, accumulation, and integer multiplication. For example, the output data processing module may include an output register. The output data processing module processes the data output by the matrix storage and calculation units and stores it in the output register, and stores the value in the output register back to the corresponding address of the storage module.

[0061] The control module is configured to control the timing, configuration information, etc. of all modules including the storage module, input data processing module, matrix storage and calculation unit, and output data processing module according to the configuration information.

[0062] For example, the control module includes a control logic generation module, a configuration information storage module, and a control logic state machine. Before the neural network processing device begins operation, configuration information is pre-written into the configuration information storage module. The control logic generation module generates control signals and selection signals for different modules based on the configuration information. The configuration information includes, for example, the parallelism and number of pipeline stages of each layer in the neural network, the data access address of the input data processing module of each layer, the network structure of each layer, and the data storage address of the output data processing module of each layer. The control logic state machine switches its operating state based on handshake signals with other modules.

[0063] The neural network processing device of the disclosed embodiment can use simple control logic to implement a pipeline-configurable hardware acceleration architecture, thereby compressing feature maps and significantly reducing computing delays. It has the advantages of high utilization, low area power consumption, and short delay, and can be applied to scenarios such as edge image processing.

[0064] FIG4A shows the structure of a matrix storage unit provided in some embodiments of the present disclosure, which includes a memristor array, a bit line driver circuit, a word line / source line driver circuit, and a digital-to-analog converter. For example, the memristor array has M rows and N columns (M and N are positive integers), the bit line driver circuit corresponds to the number of rows of the memristor array, and the word line / source line driver circuit corresponds to the number of columns of the memristor array. For example, see, but not limited to, the situation shown in FIG2. The weights of the neural network are stored as the resistance values ​​of the memristor array. The memristor array uses the weights to perform matrix-vector multiplication calculations of the neural network. The bit line driver circuit provides a bit line signal to the memristor array based on the digital signal stored in the input register, for example, using a corresponding analog voltage as the input signal. The bit line driver circuit clamps the word line / source line of the memristor array to a set voltage, and the resulting accumulated current signal is converted into a corresponding digital output by the digital-to-analog converter. Accordingly, the bit line driver circuit, the word line / source line driver circuit, and the digital-to-analog converter can all receive the control signal provided by the control module. It should be noted that the matrix storage and computing unit may also include other peripheral driving circuits, such as data reading circuits, data output circuits, etc. The peripheral driving circuits can be added and tailored accordingly according to the different designs of the memristor array (such as row and column scales). The embodiments of the present disclosure do not limit the specific peripheral circuit implementation methods.

[0065] Figure 4B shows the structure of the matrix storage and computing unit provided by other embodiments of the present disclosure, which is different from the matrix storage and computing unit in Figure 4A. The matrix storage and computing unit includes a first memristor array and a second memristor array. For example, both memristor arrays are P rows and Q columns (P and Q are positive integers), and they share the same bit line driver circuit. The weights of the neural network can be stored in the first memristor array or the second memristor array, or stored in the first memristor array and the second memristor array after being copied. Under the control of the control module, each row of the bit line driver circuit can be connected to the first memristor array or the second memristor array, or to the first memristor array and the second memristor array. This matrix storage and computing structure can better utilize the characteristics of the convolution kernel receptive field, reuse the input data, and thus make the input data more efficient during pipeline configuration, and can reduce the area of ​​the peripheral circuit and improve utilization.

[0066] At least one embodiment of the present disclosure further provides an electronic device, comprising a neural network processing apparatus according to any of the above embodiments. For example, the electronic device in the embodiments of the present disclosure may be a server device or a terminal device. For example, the terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers.

[0067] The following describes in detail an operating method of a neural network processing device provided by some embodiments of the present disclosure with reference to Figure 5. As shown in Figure 5, the operating method for the neural network processing device according to at least one embodiment of the present disclosure includes the following steps S10-S50.

[0068] Step S10: Store the input feature map in the first storage submodule.

[0069] Before the neural network starts calculating, the input feature map is first stored in the first storage submodule.

[0070] Step S20: writing the weights of at least one network layer of the neural network into at least one object matrix storage unit among the plurality of matrix storage units.

[0071] The operating method of the neural network processing device provided by the embodiment of the present disclosure can select the number of matrix storage and computing units to be used according to the requirements of different application scenarios to reduce computing delay and improve chip utilization. For example, at least one object matrix storage and computing unit may include a first matrix storage and computing unit and a second matrix storage and computing unit.

[0072] Step S30: The input data processing module obtains input data from the storage module and processes the data into a data structure for input to at least one object matrix storage and calculation unit.

[0073] In step S40, at least one object matrix storage and calculation unit performs a matrix-vector multiplication operation of the neural network.

[0074] Step S50: Processing the data output by at least one object matrix storage unit after performing the matrix-vector multiplication operation through the output data processing module.

[0075] Compared with the traditional full-pipeline, layer-by-layer architecture operation method, the operation method of the neural network processing device provided by the present invention compresses the feature map, reduces computing delay, improves chip utilization, and saves area and power consumption by performing parallel or pipeline processing only on the key layer network.

[0076] The following first describes in detail the operation process of a pipeline operation method provided by an embodiment of the present disclosure with reference to FIG6 .

[0077] For example, at least one object matrix storage unit includes a first matrix storage unit and a second matrix storage unit. The weight of the i-th layer network of the neural network is written into the first matrix storage unit, and the weight of the i+1-th layer network of the neural network is written into the second matrix storage unit, where i is a positive integer.

[0078] In the first time period (0~t0), the i-th layer network is calculated, the input data processing module obtains the first input data from the first storage submodule, and inputs the first input data into the first matrix storage and calculation unit after processing to perform the first operation to obtain the first output data, and the first output data is processed by the output data processing module and stored in the second storage submodule.

[0079] In the second time period (t0~t1), the i+1th layer network is calculated, the input data processing module obtains the second input data from the first storage submodule and the second data storage submodule, and inputs the first input data into the first matrix storage and calculation unit and the second matrix storage and calculation unit after being processed to perform a second operation to obtain the second output data, and the second output data is processed by the output data processing module and stored in the second storage submodule.

[0080] In the third time period (t1~t2), the calculation of the i-th layer network is completed, and the calculation of the i+1-th layer continues. The input data processing module obtains the third input data from the second storage submodule, and inputs the third input data into the second matrix storage and calculation unit after processing to perform the third operation to obtain the third output data. The third output data is processed by the output data processing module and stored in the first storage submodule.

[0081] This operation is performed on the target layer neural network until the calculation of all target layers is completed.

[0082] It should be noted that at least one object matrix storage and calculation unit may also include, for example, a matrix storage and calculation unit, which includes a first memristor array and a second memristor array. Similarly, the weights of the i-th layer network of the neural network are written into the first memristor array, and the weights of the i+1-th layer network of the neural network are written into the memristor array. The first memristor array is used when calculating the i-th layer network in the first time period (0-t0); the first memristor array and the second memristor array are used when calculating the i+1-th layer network in the second time period (t0-t1); the calculation of the i-th layer network is completed in the third time period (t1-t2), and the second memristor array is used when continuing to calculate the i+1-th layer.

[0083] For example, at least one object matrix storage and computing unit includes a first matrix storage and computing unit and a second matrix storage and computing unit, the first matrix storage and computing unit includes a first memristor array and a second memristor array, and the second matrix storage and computing unit includes a third memristor array and a fourth memristor array. The operating method of the neural network processing device provided by at least one embodiment of the present disclosure includes first starting calculation with the first memristor array, and after the data dependency relationship is satisfied, the second memristor array starts calculation, and then the third memristor array and the fourth memristor array start calculation in sequence respectively.

[0084] For example, at least one object matrix storage and computing unit includes a first matrix storage and computing unit and a second matrix storage and computing unit, the first matrix storage and computing unit includes a first memristor array and a second memristor array, and the operating method of the neural network processing device provided by at least one embodiment of the present disclosure includes first starting calculation by the first memristor array, and after the data dependency relationship is satisfied, the second memristor array starts calculation, and then the second matrix storage and computing unit starts calculation.

[0085] The operation process of a parallel operation method provided by an embodiment of the present disclosure is described in detail below with reference to FIG. 7 .

[0086] For example, an object matrix storage unit includes a first matrix storage unit and a second matrix storage unit. The weights of the i-th layer network and the i+1-th layer network of the neural network are copied and written into the first matrix storage unit and the second matrix storage unit. The weights of the i-th layer network and the i+1-th layer network are alternately arranged in the first matrix storage unit and the second matrix storage unit, and i is a positive integer.

[0087] In the first time period (0-t0), the i-th layer network is calculated. The input data processing module obtains the first input data from the first storage submodule, and uses the first matrix storage unit and the second matrix storage unit to perform calculations to obtain the first output data. The first output data is processed by the output data processing module and then stored in the first storage submodule.

[0088] In the second time period (t0~t1), the i+1th layer network is calculated, the input data processing module obtains the second input data from the first storage submodule, and uses the first matrix storage and calculation unit and the second matrix storage and calculation unit to perform a second operation to obtain the second output data. The second output data is processed by the output data processing module and stored in the first storage submodule.

[0089] It should be noted that at least one object matrix storage and computing unit may also include a matrix storage and computing unit, which includes a first memristor array and a second memristor array. The weights of the i-th layer network and the i+1-th layer network of the neural network are copied and written into the first memristor array and the second memristor array. The weights of the i-th layer network and the i+1-th layer network are alternately arranged in the first memristor array and the second memristor array. The first memristor array and the second memristor array are used when calculating the i-th layer and i+1 layer network.

[0090] It should be understood that when the matrix storage and computing unit includes a first memristor array and a second memristor array, and executes the parallel operation method provided by the embodiment of the present disclosure, as shown in Figure 8, since the weights of the i-th layer network and the i+1-th layer network are interlaced in the first memristor array and the second memristor array, because the receptive field of the convolution kernel is large, there is data overlap between the sliding windows, and there is multiplexing of input data between the first memristor array and the second memristor, which can reduce the data retrieved and the number of occupied rows, making the input data more efficient under the parallel operation method, and can reduce the area of ​​the peripheral circuit and improve utilization.

[0091] Although the present disclosure has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications or improvements may be made based on the embodiments of the present disclosure. Therefore, such modifications or improvements, as long as they do not depart from the spirit of the present disclosure, are within the scope of protection claimed by the present disclosure.

[0092] Regarding this disclosure, the following points need to be explained:

[0093] (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure. Other structures may refer to conventional designs.

[0094] (2) For the sake of clarity, in the drawings used to describe the embodiments of the present disclosure, the thickness of layers or regions is exaggerated or reduced, that is, these drawings are not drawn according to the actual scale.

[0095] (3) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to form new embodiments.

[0096] The above description is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. A neural network processing device, comprising a storage module, an input data processing module, a plurality of matrix storage and calculation units, an output data processing module and a control module, wherein: The storage module comprises: a first storage submodule configured to store input feature maps for neural network processing, and A second storage submodule is configured to store intermediate feature maps for processing by the neural network, The input data processing module is configured to obtain input data from the storage module and process the input data into a data structure for input to the plurality of matrix storage and calculation units. The plurality of matrix storage and calculation units are configured to store weights of at least one network layer of the neural network and perform matrix-vector multiplication calculations of the neural network using the weights, The output data processing module is configured to process the data output by the plurality of matrix storage and calculation units. The control module control is configured to control the storage module, the input data processing module, the matrix storage and calculation unit and the output data processing module according to configuration information.

2. The neural network processing device according to claim 1, wherein: The input data processing module includes an input register. The input data processing module is configured to obtain input data from a corresponding storage address and store the input data in the input register after processing under the control of the control module.

3. The neural network processing device according to claim 1 or 2, wherein: Each of the plurality of matrix storage and computing units comprises a memristor array, a bit line driving circuit, a word line / source line driving circuit and a digital-to-analog converter, wherein: The memristor array is configured to store weights of at least one network layer of the neural network and use the weights to perform matrix-vector multiplication calculations of the neural network, The bit line driving circuit is configured to provide a bit line signal to the memristor array. The word line / source line driving circuit is configured to provide a word line signal / source line signal to the memristor array, The bit line driving circuit, the word line / source line driving circuit and the digital-to-analog converter are further configured to receive a control signal provided by the control module.

4. The neural network processing device according to claim 3, wherein: The memristor array includes a first memristor array and a second memristor array, and the first memristor array and the second memristor array share a same bit line driving circuit.

5. The neural network processing device according to any one of claims 1 to 4, wherein: The output data processing module includes an output register, and is configured to process the output data into the same data format as the data in the storage module and store the data in the output register, and store the value of the output register in the storage module.

6. The neural network processing device according to any one of claims 1 to 5, wherein: The control module includes a control logic generation module, a configuration information storage module and a control logic state machine. The control logic generation module is configured to generate a control signal and a selection signal for at least one of the storage module, the input data processing module, the plurality of matrix storage and calculation units, and the output data processing module according to the configuration information, The configuration information storage module is configured to store the configuration information, wherein the configuration information includes at least one of the degree of parallelism, the number of pipeline stages, the network structure, the address where the input data processing module obtains input data, and the address where the output data processing module stores data. The control logic state machine is configured to switch working states according to handshake signals with the storage module, the input data processing module, the matrix storage and calculation unit, and the output data processing module.

7. An electronic device comprising a neural network processing device as described in any one of claims 1-6.

8. A method for operating the neural network processing device according to any one of claims 1 to 6, comprising: Storing the input feature map in the first storage submodule, Writing the weight of at least one network layer of the neural network into at least one object matrix storage unit among the plurality of matrix storage units, The input data processing module obtains input data from the storage module and processes the input data into a data structure for input to the at least one object matrix storage unit. The at least one object matrix storage unit performs a matrix-vector multiplication operation of the neural network, The output data processing module processes the data output by the at least one object matrix storage unit performing the matrix-vector multiplication operation.

9. The operating method according to claim 8, wherein: The at least one object matrix storage unit includes a first matrix storage unit and a second matrix storage unit, wherein: The weight of the i-th layer of the neural network is written into the first matrix storage unit, and the weight of the i+1-th layer of the neural network is written into the second matrix storage unit, where i is a positive integer. In a first time period, the i-th layer network is calculated, the input data processing module obtains first input data from the first storage submodule, and inputs the first input data into the first matrix storage and calculation unit for a first operation to obtain first output data after being processed, and the first output data is processed by the output data processing module and stored in the second storage submodule, In a second time period, the i+1th layer network is calculated, the input data processing module obtains second input data from the first storage submodule and the second data storage submodule, and inputs the first input data into the first matrix storage unit and the second matrix storage unit after processing to perform a second operation to obtain second output data, and the second output data is processed by the output data processing module and stored in the second storage submodule, In the third time period, the calculation of the i-th layer network is completed, and the calculation of the i+1-th layer continues. The input data processing module obtains the third input data from the second storage submodule, and inputs the third input data into the second matrix storage and calculation unit after being processed to perform a third operation to obtain the third output data. The third output data is processed by the output data processing module and stored in the first storage submodule.

10. The operating method according to claim 8, wherein: The at least one object matrix storage unit includes a first matrix storage unit and a second matrix storage unit, wherein: The weights of the i-th layer network and the i+1-th layer network of the neural network are copied and written into the first matrix storage unit and the second matrix storage unit, and the weights of the i-th layer network and the i+1-th layer network are alternately arranged in the first matrix storage unit and the second matrix storage unit, i is a positive integer, In a first time period, the i-th layer network is calculated, the input data processing module obtains first input data from the first storage submodule, uses the first matrix storage unit and the second matrix storage unit to perform calculations to obtain first output data, and the first output data is processed by the output data processing module and stored in the first storage submodule. In the second time period, the i+1th layer network is calculated, the input data processing module obtains second input data from the first storage submodule, uses the first matrix storage unit and the second matrix storage unit to perform a second operation to obtain second output data, and the second output data is processed by the output data processing module and stored in the first storage submodule.

Images