In-memory computing core, in-memory computing method, in-memory processor and processing method

Abstract

The present invention belongs to the technical field of integrated circuits and processors, and relates to an in-memory computing core, a computing method, an in-memory processor and a processing method. The in-memory processor adopts multiple in-memory computing cores with independent programmable design, a custom instruction architecture and a top-level master-slave communication mode, including multiple independent in-memory computing cores, a top-level controller, a top-level instruction cache, a top-level weight cache and a top-level activation cache and a clock and power module. Each independent in-memory computing core contains an in-core instruction cache, an in-core weight cache, an in-core activation cache, and an in-core calculation result cache, and each in-memory computing core is connected to an adjacent in-memory computing core for bus communication. The in-memory processor uses a single instruction to simultaneously control multiple in-memory computing cores to perform neural network calculations including vector multiplication and addition. Data transfer between in-memory computing cores and between in-memory computing cores and processors is achieved by controlling the in-memory computing cores. Various rich mathematical operations are performed by controlling the in-memory computing cores to achieve high energy efficiency computing.

Description

Technical Field

[0001] The present invention belongs to the technical field of integrated circuits and processors, and in particular, relates to an in-memory computing core, an in-memory computing method, an in-memory processor and a processing method. Background Art

[0002] In recent years, the rise of various large artificial intelligence models has led to an increasing demand for computing power. Traditional high-performance GPUs contain thousands of computing cores. When multiple threads are concurrent in the GPU, data dependencies will occur between threads. Therefore, communication between threads needs to be implemented using shared storage. The shared storage method brings two problems. First, multi-port shared storage will reduce the operating frequency of the memory itself. Second, when multiple threads read and write to the shared memory at the same time, memory access collisions are prone to occur. In order to solve the memory access collision problem, additional computing overhead is also brought. In deep learning tasks, unstructured fine-grained sparse operations often occur, and existing processors treat the operation processing of sparse matrices and dense matrices as the same task. However, there is little valid data in the sparse matrix, which will cause a waste of memory space, greatly reducing the computing efficiency of unstructured fine-grained sparse operations.

[0003] In the traditional von Neumann architecture, the "memory wall" caused by the separation of computing and memory has also become a problem that must be faced in the field of artificial intelligence. For example, traditional processors need to use at least three instructions to complete memory access, computing, and write-back tasks when performing multiplication and addition operations. Data reading needs to go through intermediate registers. The classic matrix multiplication and addition operation accounts for more than 90% of the computing amount in the deep learning process, and a large number of memory access, register access, and register write-back operations are involved during the operation. In-memory technology combines computing and storage into one, and directly uses the data in the memory as operands for calculations. Compared with traditional processors, it can improve energy efficiency. However, the in-memory computing cores provided by existing technologies often lack a complete instruction set, resulting in reduced programmability, so that the advantages of in-memory computing cores are not fully utilized. Redundant operations are required during programming to achieve specific operations, and programming flexibility is greatly limited.

[0004] In addition, the multi-core in-memory processor provided by the prior art has multiple in-memory computing cores, and the multiple in-memory computing cores are generally coprocessors, that is, a main memory computing core is defined or configured, and the main memory computing core directly controls other in-memory computing cores except the main memory computing core through a bus to complete calculations, and the computing efficiency between the in-memory computing cores is low. Summary of the invention

[0005] The purpose of the present invention is to address the technical defects of low computing efficiency and poor control flexibility of existing in-memory computing cores and low inter-core communication efficiency of existing multi-core in-memory processors, and to propose an in-memory computing core, an in-memory computing method, an in-memory processor and a processing method.

[0006] In order to achieve the above object, the present invention adopts the following technical scheme:

[0007] In a first aspect, the present invention provides an in-memory computing core, including a bus interface, an in-core logic control unit, an in-core logic operation unit, a storage-computation multiplier-adder array, a vector register, an in-core scalar register, an in-core instruction cache, an in-core weight cache, an in-core activation cache, and an in-core calculation result cache;

[0008] The bus interface is used to input instructions, weight data and activation data to the in-memory computing core, and output instructions, weight data, activation data and in-core calculation results from the in-memory computing core;

[0009] The in-core logic control unit is used to read instructions from the in-core instruction cache, generate multiple logic control signals based on the instructions, and output each of the logic control signals to the in-core logic operation unit, the in-core scalar register, the in-core vector register, and the storage-calculation multiplier-adder array respectively;

[0010] The in-core logic operation unit reads operation data from the in-core scalar register based on the logic control signal, and generates a feedback signal based on the operation data, wherein the feedback signal is used to control the in-core logic control unit or the storage-calculation multiplier-adder array;

[0011] The vector registers include a vector scalar multiplication result vector register accessed by the in-core logic control unit and an accumulation result vector register embedded in the accumulator of the storage multiplier array;

[0012] The storage-calculation multiplier array reads weight data and activation data from the in-core weight cache and the in-core activation cache respectively, and performs multiplication-addition operations to obtain multiplication-addition operation results; the multiplication-addition operations include dense matrix multiplication-addition operations and sparse matrix multiplication-addition operations; the multiplication-addition operation results of the dense matrix are directly written into the accumulation result vector register; the multiplication operation results of the sparse matrix are first input into the vector scalar multiplication result vector register to complete format adjustment, and then sent to the accumulator to complete the multiplication-addition operation of the sparse matrix to obtain the multiplication-addition operation result;

[0013] The accumulated result in the accumulated result vector register is a result matrix, and a column vector of the result matrix after a round of operation is written into the in-core result cache.

[0014] Compared with the prior art, the vector multiplication and addition operation provided by the prior art needs to run at least the following three instructions: first, write the vector to be calculated in the memory into the register; second, the logic control signal controls the vector in the register to complete the multiplication and addition calculation; third, write the vector multiplication and addition result back to the memory. In the case where the amount of vector multiplication and addition operations increases exponentially, the above-mentioned vector multiplication and addition operation has the problem of low operation efficiency. The in-memory computing core provided by the present invention combines calculation and storage into one, that is, one instruction can be used to complete multi-shot tasks. The in-memory computing core provided by the present invention performs vector multiplication and addition operations by the following method: based on the logic control signal storage multiplier array, a set of data (vector) is read from the in-core weight cache and the in-core activation cache respectively, and then the multiplier and accumulator are used in turn to complete the vector multiplication and addition calculation, that is, the step of moving data from the memory is avoided, and one control instruction can be used to complete the tasks of reading data from the weight cache and the activation cache respectively, and using the multiplier and accumulator in turn to complete the vector multiplication and addition calculation and write the calculation results into the in-core calculation result cache, thereby effectively improving the operation efficiency. In addition, the storage-computation multiplier-adder array provided by the present invention can support both dense matrix multiplication-addition operations and sparse matrix multiplication-addition operations, thereby improving on-chip storage utilization and computing efficiency.

[0015] Furthermore, the in-memory computing core has a vector instruction set that adapts to its structure. The logic control unit can directly read instructions from the in-core instruction cache, generate multiple logic control signals based on the instructions, and further output each logic control signal to the in-core logic operation unit, the in-core scalar register, the in-core vector register, and the storage multiplier array. That is, using the vector instruction set, only one instruction is needed to complete the vector multiplication and addition operation within one clock cycle, thereby improving the flexibility of the in-memory computing core and reducing the complexity of control; the read instruction is an instruction in the vector instruction set.

[0016] In a second aspect, the present invention further provides an in-memory computing method, which is applied to the in-memory computing core provided in the first aspect, and the in-memory computing method comprises the following steps:

[0017] Send instructions to the instruction cache in the core, send weight data to the weight cache in the core, and send activation data to the activation cache in the core through the bus interface;

[0018] The in-memory logic control unit reads instructions from the in-core instruction cache, generates a plurality of logic control signals based on the instructions, and outputs each of the logic control signals to the in-core logic operation unit, the in-core scalar register, the in-core vector register, and the storage-calculation multiplier-adder array respectively;

[0019] The in-memory logic operation unit receives the control signal sent by the in-memory logic control unit, reads operation data from the in-core scalar register, and generates a feedback signal based on the operation data, wherein the feedback signal is used to control the in-core logic control unit or the storage-operation multiplier-adder array;

[0020] The storage-calculation multiplier array receives the control signal sent by the in-memory logic control unit, reads weight data and activation data from the in-core weight cache and the in-core activation cache respectively, and performs multiplication-addition operations to obtain multiplication-addition operation results; the multiplication-addition operations include dense matrix multiplication-addition operations and sparse matrix multiplication-addition operations; the multiplication-addition operation results of the dense matrix are directly written into the accumulation result vector register; the multiplication operation results of the sparse matrix are first input into the vector scalar multiplication result vector register to complete format adjustment, and then sent to the accumulator to complete the multiplication-addition operation of the sparse matrix to obtain the multiplication-addition operation result;

[0021] The accumulated result in the accumulated result vector register is a result matrix, and a column vector of the result matrix after a round of operation is written into the in-core result cache.

[0022] Compared with the prior art, the beneficial effects of the in-memory computing method provided by the present invention are the same as the beneficial effects of the in-memory computing kernel provided by the first aspect and / or any implementation of the first aspect, and will not be elaborated here.

[0023] In a third aspect, the present invention provides an in-memory processor, comprising K independent in-memory computing cores, a top-level controller, a top-level instruction cache, a top-level weight cache, a top-level activation cache, a clock, and a power supply; wherein the in-memory computing core is the in-memory computing core described in the first aspect;

[0024] The top-level controller is connected to K independent in-memory computing cores through an instruction bus interface. The top-level controller reads instructions from the top-level instruction cache and determines whether the instructions are top-level execution instructions or in-memory computing core execution instructions. When the instructions are top-level execution instructions, the control program count continues to run; when the instructions are in-memory computing core execution instructions, the instructions are transmitted to the in-memory computing core to control the running state of the in-memory processor.

[0025] A plurality of in-memory computing cores distributed along a first direction form an in-memory computing core unit, and a plurality of in-memory computing core units are distributed along a second direction, and the first direction and the second direction are perpendicular to each other; the top-level weight cache is communicatively connected to the top-level controller via a weight bus interface; the top-level weight cache is communicatively connected to each of the in-memory computing core units in the same manner as follows: the weight bus interface of the top-level weight cache is communicatively connected to the proximal in-memory computing core and the distal in-memory computing core respectively, and all the in-memory computing cores located in the same in-memory computing core unit are connected via the weight bus interface;

[0026] The top-level activation cache is communicatively connected to the top-level controller via an activation bus interface; the top-level activation cache is communicatively connected to each of the in-memory computing core units in the same manner as follows: the activation bus interface of the top-level activation cache is communicatively connected to the in-memory computing core at the near end and the in-memory computing core at the far end, respectively, and all the in-memory computing cores in the same in-memory computing core unit are connected via the activation bus interface;

[0027] The top-level controller, top-level instruction cache, top-level weight cache, top-level activation cache and all in-memory computing core units are connected through the instruction data interface, weight data interface and activation data interface to form an interconnected topology structure, and adjacent in-memory computing cores perform instruction transmission, weight data transmission and activation data transmission.

[0028] Compared with the prior art, the top-level controller is interconnected with the in-memory computing cores in a topological structure through a bus. One instruction can simultaneously control the operations of multiple in-memory computing cores and control the data transfer between the in-memory computing cores and between the in-memory computing cores and the top-level controller. The data here includes instructions transferred between the top-level instruction cache and the in-core instruction cache, and between the in-core instruction caches, as well as weight data transferred between the top-level weight cache and the in-core weight cache, and between the in-core weight caches, and activation data transferred between the top-level activation cache and the in-core activation cache, and between the in-core activation caches. In this way, the memory access collision problem when multiple threads read and write to the memory at the same time can be effectively avoided.

[0029] In a fourth aspect, the present invention further provides an in-memory processing method, comprising the following steps:

[0030] Send instructions from outside the in-memory processor to the top-level instruction cache, weight data to the top-level weight cache, and activation data to the top-level activation cache via the bus interface;

[0031] The top-level logic control unit reads instructions from the top-level instruction cache, generates a plurality of logic control signals based on the instructions, and outputs each of the logic control signals to the top-level logic operation unit, the top-level scalar register and the instruction distribution module respectively;

[0032] The top-level logic operation unit receives the control signal sent by the top-level logic control unit, reads operation data from the top-level scalar register, and generates a feedback signal based on the operation data, wherein the feedback signal is used to control the top-level logic control unit or the in-memory computing core;

[0033] When the feedback signal controls the in-memory computing core, the instruction distribution module is used to send the in-memory computing core execution instruction to the in-memory logic control unit of the in-memory computing core to control the in-memory weight cache to read weight data from the top-level weight cache and control the in-memory activation cache to read activation data from the top-level activation cache. At least one in-memory computing core performs operations according to the in-memory computing method described in claim 7.

[0034] Compared with the prior art, the beneficial effects of the in-memory processing method provided by the present invention are the same as the beneficial effects of the in-memory processor provided by the first aspect and / or any implementation of the first aspect, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0036] Figure 1 The overall framework diagram of the in-memory computing core provided by the embodiment of the present invention;

[0037] Figure 2 A schematic diagram of re-adjusting the structure of a multiplication operation result represented in a sparse format provided by an embodiment of the present invention;

[0038] Figure 3 The multiplication-addition operation structure framework in the in-memory computing core provided by the embodiment of the present invention;

[0039] Figure 4 A flow chart of in-memory operations provided by an embodiment of the present invention;

[0040] Figure 5 The overall framework diagram of the in-memory processor provided by the embodiment of the present invention;

[0041] Figure 6 The overall framework diagram of the top-level controller provided by the embodiment of the present invention;

[0042] Figure 7 A flow chart of an in-memory processing method provided by an embodiment of the present invention;

[0043] Figure 8 The in-memory processor provided in the embodiment of the present invention implements the calculation process of 48×48 systolic matrix multiplication. DETAILED DESCRIPTION

[0044] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0045] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.

[0046] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined. The meaning of "several" is one or more, unless otherwise clearly and specifically defined.

[0047] In the description of the present invention, it is necessary to understand that the directions or positional relationships indicated by the terms "up", "down", "front", "back", "left", "right", etc. are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention.

[0048] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0049] The purpose of the present invention is to solve the problems of decreased performance of multi-core processor shared storage communication and poor programmability of in-memory processor. The general von Neumann architecture processor contains registers, which are temporary storage areas for storing and operating data. The present application combines the von Neumann architecture and the in-memory architecture. The in-memory processors used in the traditional in-memory architecture have no registers and need to fetch data from the memory for calculation, which involves more data transmission and additional instructions, so the programmability is poor and the processing efficiency is low. The core of the present invention is that the in-memory computing core uses the von Neumann architecture when executing scalar instructions, and uses the in-memory computing architecture alone when executing vector instructions. Based on this, in the process of deep learning, operations with a computing amount of more than 90% can be accelerated, thereby improving the processing efficiency of the in-memory processor. In addition, the overall architecture does not involve shared storage, which can effectively avoid the memory access collision problem when multiple threads read and write to the memory at the same time, thereby improving the multi-core communication efficiency. The in-memory processor constructed based on the in-memory computing core in the embodiment of the present invention adopts a multi-core independent bus communication architecture, that is, the communication architecture adopts a top-level controller to distribute instructions, uses a single instruction to simultaneously control the calculation of multiple in-memory computing cores, and also uses a single instruction to control data transfer between in-memory computing cores and between the in-memory computing core and the top-level controller, specifically: data is transferred between the in-memory caches (including in-memory instruction caches, in-memory weight caches, and in-memory activation caches) of each in-memory computing core and data is directly transferred from the top-level cache (including the top-level instruction cache, the top-level weight cache, and the top-level activation cache) to the in-memory cache.

[0050] In order to facilitate understanding of the specific architecture of in-memory processing provided by the embodiments of the present invention, the following is explained from multiple aspects with specific embodiments. It should be understood that the following examples are only for explanation and not for limitation of the present invention.

[0051] First, see Figure 1 An embodiment of the present invention provides an in-memory computing core, the core idea of ​​which is that when performing vector multiplication and addition operations, the instruction directly uses the content in the memory address as the operand, without the need for additional memory access instructions and write-back instructions.

[0052] The above-mentioned in-memory computing core specifically includes a bus interface, an in-core logic control unit, an in-core logic operation unit, a storage and calculation multiplier array, a vector register, an in-core scalar register, an in-core instruction cache, an in-core weight cache, an in-core activation cache, and an in-core calculation result cache. Among them, the bus interface is used to input instructions, weight data and activation data to the in-memory computing core, and output instructions, weight data, activation data and calculation results in the in-core calculation result cache from the in-memory computing core to the outside;

[0053] The in-core logic control unit is used to read instructions from the in-core instruction cache, generate multiple logic control signals based on the instructions, and output each logic control signal to the in-core logic operation unit, in-core scalar register, in-core vector register and storage multiplier array respectively.

[0054] The logic operation unit in the core reads operation data from the scalar register in the core based on the logic control signal, and generates a feedback signal based on the operation data. The feedback signal is used to control the logic control unit in the core or the storage-calculation multiplier array.

[0055] The vector registers include a vector scalar multiplication result vector register accessed by the logic control unit in the core and an accumulation result vector register embedded in an accumulator of the storage multiplier array.

[0056] The storage multiplier array reads weight data and activation data from the in-core weight cache and in-core activation cache respectively, and performs multiplication and addition operations to obtain multiplication and addition results. Multiplication and addition operations include dense matrix multiplication and addition operations and sparse matrix multiplication and addition operations. The multiplication and addition results of dense matrices are directly written into the accumulation result vector register. The multiplication results of sparse matrices are first input into the vector scalar multiplication result vector register to complete the format adjustment, and then sent to the accumulator to complete the multiplication and addition of sparse matrices to obtain the multiplication and addition results. The accumulated result in the accumulation result vector register is the result matrix, and the column vectors of the result matrix after one round of operation are written into the in-core result cache.

[0057] Compared with the prior art, the vector multiplication and addition operation provided by the prior art needs to run at least the following three instructions: first, write the vector to be calculated in the memory into the register; second, the logic control signal controls the vector in the register to complete the multiplication and addition calculation; third, write the vector multiplication and addition result back to the memory. In the case where the amount of vector multiplication and addition operations increases exponentially, the above-mentioned vector multiplication and addition operation has the problem of low operation efficiency. The in-memory computing core provided by the present invention combines calculation and storage into one, that is, one instruction can be used to complete multi-shot tasks. The in-memory computing core provided by the present invention performs vector multiplication and addition operations by the following method: based on the logic control signal storage multiplier array, a set of data (vector) is read from the in-core weight cache and the in-core activation cache respectively, and then the multiplier and accumulator are used in turn to complete the vector multiplication and addition calculation, that is, the step of moving data from the memory is avoided, and one control instruction can be used to complete the tasks of reading data from the weight cache and the activation cache respectively, and using the multiplier and accumulator in turn to complete the vector multiplication and addition calculation and write the calculation results into the in-core calculation result cache, thereby effectively improving the operation efficiency. In addition, the storage-computation multiplier-adder array provided by the present invention can support both dense matrix multiplication-addition operations and sparse matrix multiplication-addition operations, thereby improving on-chip storage utilization and computing efficiency.

[0058] Furthermore, the in-memory computing core has a vector instruction set adapted to its structure. The logic control unit directly reads instructions from the in-core instruction cache, generates multiple logic control signals based on the instructions, and further outputs each logic control signal to the in-core logic operation unit, the in-core scalar register, the in-core vector register, and the storage multiplier array. That is, using the vector instruction set, only one instruction is needed to complete the vector multiplication and addition operation within one clock cycle, thereby improving the flexibility of the in-memory computing core and reducing the operation complexity. The instruction directly read from the in-core instruction cache is an instruction in the vector instruction set.

[0059] It needs to be further explained that the multiple in-memory computing cores in the in-memory processor provided in the embodiment of the present invention have the same structure. The in-core logic operation unit can operate on two data according to the control signal output by the in-core logic control unit, and the feedback signal controls the in-core logic control unit or the memory multiplier array to output the operation result. The in-core logic operation unit can perform a variety of operations, including bitwise AND, bitwise OR, bitwise XOR, bitwise XOR, left shift, and right shift. The logic control signal fed back by the in-core logic operation unit includes a jump flag, an addition and subtraction flag, and a fixed-point floating-point flag.

[0060] The logic control unit in the core includes instruction decoding function, logic control function, data path function and program counting function. The instruction decoding function is to extract and decode the instruction according to the address pointed to by the program pointer in the instruction cache in the core. The logic control function is to output the control signal for each module according to the instruction decoding result. The data path function is to connect the logic control signal to the port of the corresponding module. The program counting function is to accumulate the address of the program pointer.

[0061] The in-core scalar register obtains two addresses and two read enables of the in-core scalar register according to the decoding result of the in-core logic control unit, which are used to obtain the two data required by the in-core logic operation unit. The in-core scalar register has a write-back address and a write-back enable to ensure that the instruction to modify the register content type can be executed normally. As an example, the bit width of the in-core scalar register is 16 bits, and each read and write operation of the port can only operate one 16-bit data. There are 32 general scalar registers in the in-core scalar register.

[0062] The in-core vector register inputs or outputs vector data according to the logic control result of the in-core logic control unit. The bit width of the in-core vector data is 256 bits. The storage mode of the in-core vector register is composed of 16 groups of 16-bit wide scalar registers, and the address of each group is fixed and cannot be changed. The bit width of the vector register is 256 bits, which can read or write 16-bit data at a certain address, and can also perform read or write operations on the entire 256-bit data. The vector register contains a vector scalar multiplication result vector register that can be accessed by the in-core logic control unit and an accumulation result vector register embedded in the accumulator of the storage multiplication and addition array.

[0063] The intra-core bus interface unit, whose function is to control the direction of activation data flow and weight data flow, includes an intra-core host bus interface unit and an intra-core slave bus interface unit. The output line of the intra-core host bus interface unit is called the intra-core main bus, and the output line of the intra-core slave bus interface unit is called the intra-core slave bus.

[0064] The main bus in the core includes the main activation bus, the main master bus and the data control bus. The main activation bus includes the main activation bus data read and write, the main activation bus address, the main activation bus enable and the main activation bus read and write selection. The main master master bus includes the main master master bus data read and write, the main master master master bus address, the main master master bus enable and the main master master bus read and write selection; the data bus includes data input, data output, data address, data read enable and data write enable.

[0065] The slave bus in the core includes the slave activation bus, the slave weight bus and the data control bus. The slave activation bus includes slave activation bus data read and write, slave activation bus address, slave activation bus enable and slave activation bus read and write selection. The slave weight bus includes slave weight bus data read and write, slave weight bus address, slave weight bus enable and slave weight bus read and write selection. The data control bus includes data input, data output, data address, data read enable and data write enable.

[0066] In general, the in-core master bus of the in-memory computing core is always connected to the in-core slave buses of other cores, including the in-core master activation bus connected to the in-core slave activation bus, and the in-core master weight bus connected to the in-core slave weight bus. The data control buses in the in-core master bus and the in-core slave bus are connected to each other to ensure that the data of each in-memory computing core can be exchanged.

[0067] See also Figure 3There are two special cases. In the first special case, the vertically arranged cores 0, 3, 6, 2, 5, and 8 are connected to the main activation bus of the top-level controller through the intra-core slave activation bus, and are connected to the top-level activation cache of the in-memory processor, and directly exchange data with the top-level activation cache. In the second special case, the horizontally arranged cores 0, 1, 2, 6, 7, and 8 are connected to the main weight bus of the top-level controller through the intra-core slave weight bus, and are connected to the top-level weight cache of the in-memory processor, and directly exchange data with the top-level weight cache.

[0068] The in-core instruction cache can obtain instructions distributed from the top-level instruction cache and write them to the specified address. The in-core instruction cache can be controlled by the in-core logic control unit to obtain instructions at the specified address for execution by the in-memory computing core.

[0069] The in-core weight cache is used to store weight data in a buffered manner. It includes four functions: Function 1: Read and write the specified address according to the data control bus. Function 2: Read and write the specified address according to the master bus. Function 3: Read and write the specified address according to the slave bus. Function 4: Read the specified address according to the in-core logic control unit of the in-memory computing core.

[0070] The in-core activation cache is used to store activation data. It includes four functions: Function 1: Read and write the specified address according to the data control bus. Function 2: Read and write the specified address according to the master bus. Function 3: Read and write the specified address according to the slave bus. Function 4: Read the specified address according to the in-core logic control unit of the in-memory computing core.

[0071] The in-core calculation result cache is used to store the calculation results of the accumulator array. It includes four functions: Function 1, read and write the specified address according to the data control bus. Function 2, read and write the specified address according to the in-core master bus. Function 3, read and write the specified address according to the in-core slave bus. Function 4, read and write the specified address according to the in-core logic control unit of the in-memory calculation core.

[0072] The in-core instruction cache, in-core weight cache, in-core activation cache, and in-core calculation result cache have consecutive storage addresses in the in-memory computing core, which are: the in-core instruction cache capacity is 1K bits, the in-core weight cache capacity is 4K bits, the in-core activation cache capacity is 4K bits, and the in-core calculation result cache capacity is 4K bits, with a total capacity of 13K bits.

[0073] As a possible implementation, the in-memory computing core is an in-memory computing core that can execute scalar instructions. In this case, the in-core logic control unit controls the in-core scalar register memory access and controls the in-core logic operation unit to perform operations.

[0074] As a possible implementation, the in-memory computing core is an in-memory computing core that can execute vector instructions. In this case, the in-core logic control unit controls the memory computing multiplier array to perform operations.

[0075] As a possible implementation method, the storage-computation multiplier array performs dense matrix multiplication-addition calculations in the following manner: based on a logic control signal, a set of data is read from the in-core weight cache and the in-core activation cache respectively; a multiplication operation is performed on the two sets of data, and the multiplication result is input into the accumulator to complete the accumulation operation to obtain the multiplication-addition result; the multiplication-addition result is directly stored in the accumulation result vector register.

[0076] As a possible implementation method, the storage-computation multiplier array performs sparse matrix multiplication-addition calculations in the following manner: based on a logic control signal, a set of data is read from the in-core weight cache and the in-core activation cache respectively; a multiplication operation is performed on the two sets of data to obtain a multiplication result; and the multiplication result is first input into a vector scalar multiplication result vector register; based on the logic control signal, the multiplication result represented in a sparse format is restructured to restore the multiplication result represented in a dense format that can be accumulated normally; the multiplication result in the dense format is sent to the accumulator to obtain a multiplication-addition result; and the multiplication-addition result is stored in the accumulation result vector register.

[0077] See also Figure 2 As a possible implementation method, the multiplication result represented in sparse format is obtained by representing the sparse matrix in a compressed format. At this time, the multiplication result represented in sparse format only includes non-zero elements and the labels of the row and column positions of the non-zero elements; based on this, the multiplication result represented in sparse format is restructured based on the logic control signal to restore it to the multiplication result represented in dense format that can be accumulated normally, specifically including the following steps: transmitting the label to the logic control unit in the core, and transmitting the non-zero element to the vector scalar multiplication result vector register; the logic control unit obtains the row and column position of the non-zero element in the original sparse matrix based on the label; adjusting the non-zero elements in the vector scalar multiplication result vector register through the scalar instruction; and restoring the adjusted non-zero elements to the form of the original sparse matrix.

[0078] It needs to be further explained that the storage-calculation multiplier-adder array can be used to implement multiplication and addition operations of fixed-point numbers and BF16 floating-point numbers.

[0079] As a possible implementation, see Figure 3, the storage-computation multiplier-adder array includes a plurality of storage-computation multiplier-adders, each of which includes a multiplier, a first multiplexer, a second multiplexer and an accumulator. Among them, the data input port of the multiplier is communicatively connected to the in-core weight cache and the in-core activation cache, and the instruction input port of the multiplier is communicatively connected to the in-core logic control unit. One of the data output ports of the multiplier is communicatively connected to the data input port of the first multiplexer, and the other data output port is communicatively connected to the data input port of the vector scalar multiplication result vector register, and the instruction input port of the first multiplexer is communicatively connected to the in-core logic control unit. The data output port of the first multiplexer is communicatively connected to the data input port of the accumulator; the first multiplexer selects the data in the multiplier or the vector scalar multiplication result vector register to be sent to the accumulator. The data output port of the accumulator is connected to one of the data input ports of the second multiplexer, the other data input port of the second multiplexer is connected to the in-core calculation result cache, and the instruction input port of the second multiplexer is connected to the in-core logic control unit; the second multiplexer selects the data in the adder or the in-core calculation result cache to send to the accumulation result vector register. The data output port of the accumulation result connected register is connected to the data input port of the in-core calculation result cache to write a column vector of the result matrix after a round of operation into the in-core result cache.

[0080] As an example, the storage-calculation multiplier-adder array can implement operations other than multiplication and addition, including fixed-point addition, fixed-point subtraction, fixed-point multiplication, fixed-point division, BF16 floating-point addition, BF16 floating-point subtraction, BF16 floating-point multiplication, BF16 floating-point, etc.

[0081] Second, see Figure 4 The present invention also provides an in-memory computing method, which is applied to the in-memory computing core provided in the first aspect. The in-memory computing method comprises the following steps:

[0082] S10. Send instructions to the instruction cache in the core through the bus interface, send weight data to the weight cache in the core, and send activation data to the activation cache in the core.

[0083] S11. The in-memory logic control unit reads instructions from the in-core instruction cache, generates multiple logic control signals based on the instructions, and outputs each logic control signal to the in-core logic operation unit, the in-core scalar register, the in-core vector register, and the storage multiplier array.

[0084] S12. The in-memory logic operation unit receives the control signal sent by the in-memory logic control unit, reads the operation data from the in-core scalar register, and generates a feedback signal based on the operation data. The feedback signal is used to control the in-core logic control unit or the memory-calculation multiplier-adder array.

[0085] S13. The storage-computation multiplier array receives the control signal sent by the in-memory logic control unit, reads the weight data and activation data from the in-core weight cache and the in-core activation cache respectively, and performs multiplication-addition operations to obtain the multiplication-addition results; the multiplication-addition operations include dense matrix multiplication-addition operations and sparse matrix multiplication-addition operations; the multiplication-addition results of the dense matrix are directly written into the accumulation result vector register; the multiplication results of the sparse matrix are first input into the vector scalar multiplication result vector register to complete the format adjustment, and then sent to the accumulator to complete the multiplication-addition operations of the sparse matrix to obtain the multiplication-addition results.

[0086] S14. The accumulated result in the accumulated result vector register is a result matrix. After a round of calculation, the column vectors of the result matrix are written into the result cache in the core.

[0087] Compared with the prior art, the beneficial effects of the in-memory computing method provided by the present invention are the same as the beneficial effects of the in-memory computing kernel provided by the first aspect and / or any implementation of the first aspect, and will not be elaborated here.

[0088] In a third aspect, an embodiment of the present invention further provides an in-memory processor, see Figure 5 , an in-memory processor, comprising K independent in-memory computing cores, a top-level controller, a top-level instruction cache, a top-level weight cache, a top-level activation cache, a clock and a power supply; wherein the in-memory computing core is the in-memory computing core of any one of claims 1 to 6.

[0089] The top-level controller is connected to K independent in-memory computing cores through an instruction bus interface. The top-level controller reads instructions from the top-level instruction cache and determines whether the instruction is a top-level execution instruction or an in-memory computing core execution instruction. When the instruction is a top-level execution instruction, the control program count continues to run; when the instruction is an in-memory computing core execution instruction, the instruction is transmitted to the in-memory computing core to control the running status of the in-memory processor.

[0090] Multiple in-memory computing cores distributed along a first direction form an in-memory computing core unit, and multiple in-memory computing core units are distributed along a second direction, and the first direction and the second direction are perpendicular to each other; the top-level weight cache is communicated with the top-level controller through a weight bus interface; the top-level weight cache is communicated with each in-memory computing core unit in the same manner as follows: the weight bus interface of the top-level weight cache communicates with the proximal in-memory computing core and the distal in-memory computing core respectively, and all in-memory computing cores in the same in-memory computing core unit are connected through the weight bus interface.

[0091] The top-level activation cache is communicated with the top-level controller through the activation bus interface; the top-level activation cache is communicated with each in-memory computing core unit in the same manner as follows: the activation bus interface of the top-level activation cache communicates with the proximal in-memory computing core and the distal in-memory computing core respectively, and all in-memory computing cores in the same in-memory computing core unit are connected through the activation bus interface.

[0092] The top-level controller, top-level instruction cache, top-level weight cache, top-level activation cache and all in-memory computing core units are connected through the instruction data interface, weight data interface and activation data interface to form an interconnected topology structure, and adjacent in-memory computing cores perform instruction transmission, weight data transmission and activation data transmission.

[0093] Compared with the prior art, the top-level controller is interconnected with the in-memory computing cores in a topological structure through a bus. One instruction can simultaneously control the operations of multiple in-memory computing cores and control the data transfer between the in-memory computing cores and between the in-memory computing cores and the top-level controller. The data here includes instructions transferred between the top-level instruction cache and the in-core instruction cache, and between the in-core instruction caches, as well as weight data transferred between the top-level weight cache and the in-core weight cache, and between the in-core weight caches, and activation data transferred between the top-level activation cache and the in-core activation cache, and between the in-core activation caches. In this way, the memory access collision problem when multiple threads read and write to the memory at the same time can be effectively avoided.

[0094] As a possible implementation, see Figure 6 The top-level controller includes a top-level bus interface unit, a top-level logic control unit, a top-level logic operation unit, a top-level scalar register and an instruction distribution module; the top-level logic control unit is the overall control, and is communicated with the top-level logic operation unit, the top-level scalar register, the top-level bus interface and the instruction distribution module.

[0095] Among them, the top-level bus interface unit includes a top-level host bus interface unit and a top-level slave bus interface unit. The top-level host bus interface unit is communicatively connected to the slave bus interface unit of the in-memory computing core, and the top-level slave bus interface unit is communicatively connected to the host bus interface unit of the in-memory computing core.

[0096] The top-level logic control unit reads instructions from the top-level instruction cache and determines whether the instruction is a top-level execution instruction or an in-memory computing core execution instruction. When the instruction is a top-level execution instruction, the control program count continues to run. When the instruction is an in-memory computing core execution instruction, the instruction is transmitted to the in-memory computing core to control the running state of the in-memory processor.

[0097] The top-level logic operation unit reads operation data from the top-level scalar register according to the top-level execution instruction output by the top-level logic control unit, operates on the operation data and outputs the operation result; and obtains a feedback signal, which is returned to the top-level logic control unit.

[0098] The operation result is returned to the top-level scalar register or the second-level data cache; the second-level data cache includes a second-level weight cache and a second-level activation weight cache.

[0099] The instruction distribution module receives instructions output by the top-level logic control unit, distributes the top-level execution instructions to the top-level controller, and distributes the in-core execution instructions to the in-memory computing core.

[0100] It needs to be further explained that the top-level logic operation unit of the top-level controller can operate on two data and output the operation result according to the control signal output by the top-level logic control unit, and the control logic control unit receives the feedback signal; the top-level logic operation unit can implement multiple operations, such as bitwise AND, bitwise OR, bitwise XOR, bitwise XOR, left shift and right shift; the feedback signal received by the top-level logic operation unit includes a jump flag and an addition and subtraction flag; compared with the in-core logic operation unit of the in-memory computing core, it can only perform fixed-point logic feedback.

[0101] The top-level logic control unit of the top-level controller includes instruction decoding function, logic control function, data path function, program counting function and operation arbitration function. The instruction decoding function, logic control function, data path function and program counting function have the same functions as the in-core logic control unit of the in-memory computing core. The operation arbitration function is to obtain instructions from the top-level instruction cache to determine whether this instruction is executed by the top level or the in-memory computing core, control whether the program counting can continue to run, and then control the operation status of the in-memory processor.

[0102] The top-level scalar register of the top-level controller is consistent with the in-memory computing core's in-core scalar register structure, with 32 scalar registers, all of which are 16-bit scalar registers, including 30 general registers, 1 comparison result register, and 1 operation mask register. The comparison result register is used to store comparison jump related flags; the operation mask register is used to control the operation status of the in-memory computing core.

[0103] The top-level bus interface unit of the top-level controller includes a top-level host bus interface unit and a top-level slave bus interface unit. The outgoing line of the top-level host bus interface unit is called the top-level master bus, and the outgoing line of the top-level slave bus interface unit is called the top-level slave bus.

[0104] The top-level master bus includes the master activation bus, the master master bus, and the data control bus. The master activation bus includes bus enable, bus read / write selection, activation cache read enable, and write enable; the master master bus includes bus address, bus enable, bus read / write selection, weight cache address, weight cache read enable, and weight cache write enable. The data control bus includes host enable, host read / write selection, internal address, external address, and weight activation direction selection.

[0105] The top-level slave bus includes the slave activation bus and the slave weight bus. The slave activation bus includes bus read data, bus write data, bus address, bus enable, read / write select, cache input data, cache output data, cache address, cache read enable, and cache write enable. The slave weight bus includes bus enable, bus read / write select, activation cache read enable and write enable, cache input data, cache output data, cache address, cache read enable, and cache write enable.

[0106] The instruction distribution module distributes instructions to the logic operation unit of the top-level controller or each in-memory computing core according to the operation arbitration result.

[0107] The top-level instruction cache obtains instructions from outside the in-memory processor and writes data to a specified address or reads data from a specified address. The top-level instruction cache can be controlled by the top-level logic control unit to write data to a specified address or read data from a specified address for execution inside the in-memory processor.

[0108] The top-level weight cache is consistent with the structure of the instruction cache in the core. The top-level activation cache is also consistent with the structure of the instruction cache in the core. The capacity of the top-level instruction cache is 16K bits, and the capacity of the top-level activation cache and the top-level weight cache are both 64K bits; the clock and power module provides clock, digital and analog power for the in-memory processor.

[0109] Fourthly, see Figure 7 In a fourth aspect, the present invention further provides an in-memory processing method, comprising the following steps:

[0110] S20. Send instructions outside the memory processor to the top-level instruction cache through the bus interface, send weight data to the top-level weight cache, and send activation data to the top-level activation cache.

[0111] S21. The top-level logic control unit reads instructions from the top-level instruction cache, generates multiple logic control signals based on the instructions, and outputs each of the logic control signals to the top-level logic operation unit, the top-level scalar register and the instruction distribution module respectively.

[0112] S22. The top-level logic operation unit receives the control signal sent by the top-level logic control unit, reads operation data from the top-level scalar register, and generates a feedback signal based on the operation data, wherein the feedback signal is used to control the top-level logic control unit or the in-memory computing core.

[0113] S23. When the feedback signal controls the in-memory computing core, the instruction distribution module is used to send the in-memory computing core execution instruction to the in-memory logic control unit of the in-memory computing core to control the in-memory weight cache to read weight data from the top-level weight cache and control the in-memory activation cache to read activation data from the top-level activation cache. At least one in-memory computing core performs operations according to the in-memory computing method described in claim 7.

[0114] Compared with the prior art, the beneficial effects of the in-memory processing method provided by the present invention are the same as the beneficial effects of the in-memory processor provided by the first aspect and / or any implementation of the first aspect, and will not be elaborated here.

[0115] In order to facilitate understanding of the processing process of the in-memory processor provided in an embodiment of the present invention, nine in-memory computing cores jointly implement 48×48 systolic matrix multiplication as an example for explanation. It should be understood that the following example is only for explanation and not for limitation.

[0116] See also Figure 8 In this embodiment, the 48×48 matrix is ​​divided into 3×3 blocks, and a 48×48 matrix is ​​divided into nine 16×16 sub-matrices. The multiplication operation of two 48×48 matrices is converted into the matrix multiplication operation of two groups of 16×16 sub-matrices. Each row and column of the 16×16 sub-matrix can be regarded as a vector, that is, each sub-matrix is ​​stored as 16 vectors by row or column, thereby converting the matrix multiplication operation into the vector multiplication. The vector multiplication operation in this embodiment can be mathematically converted into the matrix multiplication method. The two 48×48 matrices are respectively recorded as the weight matrix W and the activation matrix A. The weight matrix W is divided into nine 16×16 sub-matrices, which are recorded as w00, w01, w02, w10, w11, w12, w20, w21, and w22 from left to right and from top to bottom. The activation matrix A is divided into nine 16×16 sub-matrices, which are recorded as a00, a01, a02, a10, a11, a12, a20, a21, and a22 from left to right and from top to bottom. The calculation result can be obtained by accumulating the corresponding matrix outer product of the corresponding sub-matrix, and this embodiment will explain this process in detail.

[0117] The working process of the in-memory processor in this embodiment includes: instruction acquisition and distribution, data transmission and in-memory calculation, and obtaining calculation results.

[0118] The instruction acquisition and distribution phase of the in-memory processor includes the following steps:

[0119] In step 1, the in-memory processor stores instructions into the top-level instruction cache, activation data into the top-level activation cache, and weight data into the top-level weight cache through external pins.

[0120] Step 2: The top-level controller of the in-memory processor controls the instruction distribution module to send the instruction of the corresponding in-memory computing core to the in-core instruction cache of the corresponding in-memory computing core according to the decoding result of the top-level logic operation unit in the top-level controller. Figure 8 The initialization state at time T=0.

[0121] The data transfer and in-memory calculation phase of the in-memory processor includes the following steps:

[0122] Step 1: The bus interface unit of the top-level controller controls the weight matrix w00 required for the operation of core 0 to be loaded from the top-level weight cache to the in-core weight cache of core 0. The loaded data size is 256 groups of 16-bit vector data, totaling 4K bits. The bus interface unit of the top-level controller controls the activation matrix a00 required for the operation of core 0 to be loaded from the top-level activation cache to the in-core activation cache of core 0. The loaded data size is 256 groups of 16-bit vector data, totaling 4K bits.

[0123] Step 2: Enable core 0 through the top-level controller. Core 0 reads the instructions to be executed from the in-core instruction cache, fetches data from the in-core weight cache and in-core activation cache, and directly uses the storage multiplier array to complete the vector multiplication operation, and sends the calculation result to the in-core calculation result cache. Figure 8 In the state at time T=1, the in-memory computing core 0 completes two groups of 16×16 sub-matrix multiplications and stores the results in the in-core computing result cache.

[0124] Step three, enable core 0 through the top-level controller, and the bus interface unit of core 0 controls the transfer of the weight matrix w00 of the in-core weight cache to the in-core weight cache of core 3, and transfers the activation matrix a00 of the in-core activation cache to the in-core activation cache of core 1.

[0125] Step 4: The bus interface unit of the top-level controller controls the weight matrix 10 required for the core operation of core 0 and the weight matrix w01 required for the core operation of core 1 to be loaded from the top-level weight cache to the in-core weight cache of the corresponding in-memory computing core; the activation matrix a01 required for the core operation of core 0 and the activation matrix a10 required for the core operation of core 3 are loaded from the top-level activation cache to the in-core activation cache of the corresponding in-memory computing core.

[0126] Step 5: Enable core 0, core 1, and core 3 through the top-level controller, read the instructions to be executed from the in-core instruction cache, and retrieve data from the corresponding in-core weight cache and in-core activation cache. Core 1 and core 3 directly use the storage and calculation multiplier array to complete the vector multiplication operation, and send the calculation results to the corresponding in-core calculation result cache; the storage and calculation multiplier array of core 0 completes the vector multiplication and addition operation, accumulates the data in its in-core calculation result cache with the new matrix multiplication calculation result, and sends the new calculation result to the in-core calculation result cache to overwrite the old calculation result. At this point, the in-memory processor is Figure 8 In the state at time T=2, the in-memory computing cores 1 and 3 complete two groups of 16×16 sub-matrix multiplications and store the results in the corresponding in-core calculation result caches, and core 0 completes the vector multiplication and addition calculation to obtain a new calculation result.

[0127] Step 6: Enable core 0, core 1, and core 3 through the top-level controller. The bus interface unit of core 0 controls the weight matrix w10 of the in-core weight cache to be transferred to the in-core weight cache of core 3, and transfers the activation matrix a01 of the in-core activation cache to the in-core activation cache of core 1. The bus interface unit of core 1 controls the weight matrix w01 of the in-core weight cache to be transferred to the in-core weight cache of core 4, and transfers the activation matrix a00 of the in-core activation cache to the in-core activation cache of core 2. The bus interface unit of core 3 controls the weight matrix w00 of the in-core weight cache to be transferred to the in-core weight cache of core 6, and transfers the activation matrix a10 of the in-core activation cache to the in-core activation cache of core 4.

[0128] Step seven, the bus interface unit of the top-level controller controls the weight matrix w20 required for the core operation of core 0, the weight matrix w11 required for the core operation of core 1, and the weight matrix w02 required for the core operation of core 2 to be loaded from the top-level weight cache to the in-core weight cache of the corresponding in-memory computing core; the activation matrix a02 required for the core operation of core 0, the activation matrix a11 required for the core operation of core 3, and the activation matrix a20 required for the core operation of core 6 are loaded from the top-level activation cache to the in-core activation cache of the corresponding in-memory computing core.

[0129] Step eight, enable core 0, core 1, core 2, core 3, core 4, and core 6 through the top-level controller, read the instructions to be executed from the in-core instruction cache, and retrieve data from the corresponding in-core weight cache and in-core activation cache. Core 2, core 4, and core 6 directly use the storage and calculation multiplier array to complete the vector multiplication operation, and send the calculation result to the corresponding in-core calculation result cache; the storage and calculation multiplier array of core 0, core 1, and core 3 completes the vector multiplication and addition operation, accumulates the data in the corresponding in-core calculation result cache with the new matrix multiplication calculation result, and sends the new calculation result to the corresponding in-core calculation result cache to overwrite the old calculation result. At this point, the in-memory processor is Figure 8 In the state at time T=3, the in-memory computing cores 2, 4, and 6 complete two groups of 16×16 sub-matrix multiplications and store the results in the corresponding in-core calculation result caches. Cores 1 and 3 complete the vector multiplication and addition calculations to obtain new calculation results. Core 0 completes all operations.

[0130] Step nine, enable core 0, core 1, core 2, core 3, core 4, and core 6 to work through the top-level controller, and the bus interface unit of core 0 controls the weight matrix w20 of the in-core weight cache to be transferred to the in-core weight cache of core 3, and transfers the activation matrix a02 of the in-core activation cache to the in-core activation cache of core 1. The bus interface unit of core 1 controls the weight matrix w11 of the in-core weight cache to be transferred to the in-core weight cache of core 4, and transfers the activation matrix a01 of the in-core activation cache to the in-core activation cache of core 2. The bus interface unit of core 2 controls the weight matrix w02 of the in-core weight cache to be transferred to the in-core weight cache of core 5. The bus interface unit of core 3 controls the weight matrix w10 of the in-core weight cache to be transferred to the in-core weight cache of core 6, and transfers the activation matrix a11 of the in-core activation cache to the in-core activation cache of core 4. The bus interface unit of core 4 controls the transfer of the weight matrix w01 of the in-core weight cache to the in-core weight cache of core 7, and transfers the activation matrix a10 of the in-core activation cache to the in-core activation cache of core 5. The bus interface unit of core 6 controls the transfer of the activation matrix a20 of the in-core activation cache to the in-core activation cache of core 7.

[0131] Step ten, the bus interface unit of the top-level controller controls the weight matrix w21 required for the core 1 operation and the weight matrix w12 required for the core 2 operation to be loaded from the top-level weight cache to the in-core weight cache of the corresponding in-memory computing core; the activation matrix a12 required for the core 3 operation and the activation matrix a21 required for the core 6 operation are loaded from the top-level activation cache to the in-core activation cache of the corresponding in-memory computing core.

[0132] Step eleven, enable core 1, core 2, core 3, core 4, core 5, core 6, and core 7 through the top-level controller, read the instructions to be executed from the in-core instruction cache, and retrieve data from the corresponding in-core weight cache and in-core activation cache. Core 1, core 2, core 3, core 4, and core 6 directly use the storage and calculation multiplier array to complete the vector multiplication operation, and send the calculation result to the corresponding in-core calculation result cache. The storage and calculation multiplier array of core 5 and core 7 completes the vector multiplication and addition operation, accumulates the data in the corresponding in-core calculation result cache with the new matrix multiplication calculation result, and sends the new calculation result to the corresponding in-core calculation result cache to overwrite the old calculation result. At this point, the in-memory processor is Figure 8 In the state at time T=4, the in-memory computing cores 5 and 7 complete two groups of 16×16 sub-matrix multiplications and store the results in the corresponding in-core calculation result cache. Cores 2, 4, and 6 complete the vector multiplication and addition calculations to obtain new calculation results. Cores 1 and 3 complete all operations.

[0133] Step 12: Enable core 1, core 2, core 3, core 4, core 5, core 6, and core 7 through the top-level controller. The bus interface unit of core 1 controls the weight matrix w21 of the in-core weight cache to be transferred to the in-core weight cache of core 4, and transfers the activation matrix a02 of the in-core activation cache to the in-core activation cache of core 2. The bus interface unit of core 2 controls the weight matrix w12 of the in-core weight cache to be transferred to the in-core weight cache of core 5. The bus interface unit of core 3 controls the weight matrix w20 of the in-core weight cache to be transferred to the in-core weight cache of core 6, and transfers the activation matrix a12 of the in-core activation cache to the in-core activation cache of core 4. The bus interface unit of core 4 controls the weight matrix w11 of the in-core weight cache to be transferred to the in-core weight cache of core 7, and transfers the activation matrix a11 of the in-core activation cache to the in-core activation cache of core 5. The bus interface unit of core 5 controls the weight matrix w02 of the in-core weight cache to be transferred to the in-core weight cache of core 8. The bus interface unit of core 6 controls the transfer of activation matrix a21 of the in-core activation cache to the in-core activation cache of core 7. The bus interface unit of core 7 controls the transfer of activation matrix a20 of the in-core activation cache to the in-core activation cache of core 8.

[0134] Step 13: The bus interface unit of the top-level controller controls the weight matrix w22 required for the operation of core 2 to be loaded from the top-level weight cache to the in-core weight cache of the corresponding in-memory computing core. The activation matrix a22 required for the operation of core 6 is loaded from the top-level activation cache to the in-core activation cache of the corresponding in-memory computing core.

[0135] Step fourteen, enable core 2, core 4, core 5, core 6, core 7, and core 8 through the top-level controller, read the instructions to be executed from the in-core instruction cache, and retrieve data from the corresponding in-core weight cache and in-core activation cache. Core 8 directly uses the storage and calculation multiplier array to complete the vector multiplication operation, and sends the calculation result to the corresponding in-core calculation result cache. The storage and calculation multiplier array of core 2, core 4, core 5, core 6, and core 7 completes the vector multiplication and addition operation, accumulates the data in the corresponding in-core calculation result cache with the new matrix multiplication calculation result, and sends the new calculation result to the corresponding in-core calculation result cache to overwrite the old calculation result. At this point, the in-memory processor is Figure 8 In the state at time T=5, the in-memory computing core 8 completes two groups of 16×16 sub-matrix multiplications and stores the results in the corresponding in-core calculation result cache. Cores 5 and 7 complete the vector multiplication and addition calculations to obtain new calculation results. Cores 2, 4, and 6 complete all operations.

[0136] Step 15: Enable core 2, core 4, core 5, core 6, and core 7 through the top-level controller. The bus interface unit of core 2 controls the weight matrix w22 of the in-core weight cache to be transferred to the in-core weight cache of core 5. The bus interface unit of core 4 controls the weight matrix w21 of the in-core weight cache to be transferred to the in-core weight cache of core 7, and transfers the activation matrix a12 of the in-core activation cache to the in-core activation cache of core 5. The bus interface unit of core 5 controls the weight matrix w12 of the in-core weight cache to be transferred to the in-core weight cache of core 8. The bus interface unit of core 6 controls the activation matrix a22 of the in-core activation cache to be transferred to the in-core activation cache of core 7. The bus interface unit of core 7 controls the activation matrix a21 of the in-core activation cache to be transferred to the in-core activation cache of core 8.

[0137] Step 16: Enable cores 5, 7, and 8 through the top-level controller, read the instructions to be executed from the in-core instruction cache, and retrieve data from the corresponding in-core weight cache and in-core activation cache. The storage multiplier arrays of cores 5, 7, and 8 complete the vector multiplication and addition operation, accumulate the data in the corresponding in-core calculation result cache with the new matrix multiplication result, and send the new calculation result to the corresponding in-core calculation result cache to overwrite the old calculation result. At this point, the in-memory processor is Figure 8 In the state at time T=6, core 8 completes the vector multiplication and addition calculation to obtain a new calculation result, and cores 5 and 7 complete all operations.

[0138] Step 17: The bus interface unit of core 5 controls the weight matrix w22 of the in-core weight cache to be transferred to the in-core weight cache of core 8. The bus interface unit of core 7 controls the activation matrix a22 of the in-core activation cache to be transferred to the in-core activation cache of core 8.

[0139] Step 18: Enable core 8 through the top-level controller, read the instructions to be executed from the in-core instruction cache, fetch data from the in-core weight cache and in-core activation cache, and the storage multiplier array of core 8 completes the vector multiplication and addition operation, accumulates the data in the corresponding in-core calculation result cache with the new matrix multiplication result, and sends the new calculation result to the corresponding in-core calculation result cache to overwrite the old calculation result. Figure 8 In the state at time T=7, all the computing cores in memory have completed the operation.

[0140] The in-memory processor is manufactured using 28nm CMOS technology. By utilizing in-memory computing technology based on outer products, the in-memory processor achieves 4.8ns BF16 activation and BF16 weight access time at 0.95V for BF16 multiply-add output. For BF16 activation and BF16 weight, the average floating-point energy efficiency ranges from 16.55 to 32.18 TFLOPS / W, with a maximum area efficiency of 2.21 TFLOPS / mm 2 For INT8 activations and weights, the average fixed-point energy efficiency ranges from 22.78 to 50.53 TFLOPS / W, with a maximum area efficiency of 2.74 TOPS / mm 2 .

[0141] The specific descriptions or embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above descriptions are only specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An in-memory computing core, characterized in that: Including bus interface, in-core logic control unit, in-core logic operation unit, storage and calculation multiplier array, in-core vector register, in-core scalar register, in-core instruction cache, in-core weight cache, in-core activation cache and in-core calculation result cache; The bus interface is used to input instructions, weight data and activation data to the in-memory computing core, and output instructions, weight data, activation data and in-core calculation results from the in-memory computing core; The in-core logic control unit is used to read instructions from the in-core instruction cache, generate multiple logic control signals based on the instructions, and output each of the logic control signals to the in-core logic operation unit, the in-core scalar register, the in-core vector register, and the storage-calculation multiplier-adder array respectively; The in-core logic operation unit reads operation data from the in-core scalar register based on the logic control signal, and generates a feedback signal based on the operation data, wherein the feedback signal is used to control the in-core logic control unit or the storage-calculation multiplier-adder array; The in-core vector registers include a vector scalar multiplication result vector register accessed by the in-core logic control unit and an accumulation result vector register embedded in the accumulator of the storage multiplier array; The storage-calculation multiplier array reads weight data and activation data from the in-core weight cache and the in-core activation cache respectively, and performs multiplication-addition operations to obtain multiplication-addition operation results; the multiplication-addition operations include dense matrix multiplication-addition operations and sparse matrix multiplication-addition operations; the multiplication-addition operation results of the dense matrix are directly written into the accumulation result vector register; the multiplication operation results of the sparse matrix are first input into the vector scalar multiplication result vector register to complete format adjustment, and then sent to the accumulator to complete the multiplication-addition operation of the sparse matrix to obtain the multiplication-addition operation result; The accumulated result in the accumulated result vector register is a result matrix, and a column vector of the result matrix after a round of calculation is written into the in-core calculation result cache.

2. The in-memory computing core according to claim 1, characterized in that: The in-memory computing core is an in-memory computing core capable of executing scalar instructions. At this time, the in-core logic control unit controls the in-core scalar register to access memory and controls the in-core logic operation unit to perform operations; Alternatively, the in-memory computing core is an in-memory computing core capable of executing vector instructions. In this case, the in-core logic control unit controls the memory-computation multiplier array to perform operations.

3. The in-memory computing core according to claim 1, characterized in that: The storage multiplier array performs dense matrix multiplication and addition calculations in the following manner: Reading a set of data from the in-core weight cache and the in-core activation cache respectively based on the logic control signal; Perform multiplication operation on two sets of data, input the multiplication operation result into the accumulator to complete the accumulation operation to obtain the multiplication and addition operation result; The multiplication and addition operation results are directly stored in the accumulation result vector register.

4. The in-memory computing core according to claim 1, characterized in that: The storage multiplier array performs sparse matrix multiplication and addition calculations in the following manner: Reading a set of data from the in-core weight cache and the in-core activation cache respectively based on the logic control signal; Perform multiplication operation on two sets of data to obtain the multiplication result; and inputting the multiplication result into the vector scalar multiplication result vector register first; Restructuring the multiplication result represented in the sparse format based on the logic control signal to restore the multiplication result represented in the dense format capable of normal accumulation; Sending the multiplication result in dense format to the accumulator to obtain the multiplication-addition result; The multiplication and addition operation result is stored in the accumulation result vector register.

5. The in-memory computing core according to claim 4, characterized in that: The multiplication result represented in the sparse format is obtained by characterizing the sparse matrix in a compressed format. At this time, the multiplication result represented in the sparse format only includes non-zero elements and the labels of the row and column positions of the non-zero elements. Based on this, the multiplication result represented in the sparse format is restructured based on the logic control signal to restore the multiplication result represented in the dense format that can be accumulated normally, which specifically includes the following steps: The label is transmitted to the in-core logic control unit, and the non-zero element is transmitted to the vector scalar multiplication result vector register; The logic control unit obtains the row and column positions of the non-zero elements in the original sparse matrix based on the labels; adjusting the non-zero elements in the vector scalar multiplication result vector register by a scalar instruction; The adjusted non-zero elements are restored to the original form of the sparse matrix.

6. The in-memory computing core according to claim 4, characterized in that: The storage-calculation multiplier-adder array includes a plurality of storage-calculation multiplier-adders, each of which includes a multiplier, a first multiplexer, a second multiplexer and an accumulator; The data input port of the multiplier is communicatively connected to the in-core weight cache and the in-core activation cache, and the instruction input port of the multiplier is communicatively connected to the in-core logic control unit; One of the data output ports of the multiplier is communicatively connected to the data input port of the first multiplexer, the other data output port is communicatively connected to the data input port of the vector scalar multiplication result vector register, and the instruction input port of the first multiplexer is communicatively connected to the in-core logic control unit; The data output port of the first multiplexer is communicatively connected to the data input port of the accumulator; the first multiplexer selects data in the multiplier or the vector scalar multiplication result vector register to be sent to the accumulator; The data output port of the accumulator is communicatively connected to one of the data input ports of the second multiplexer, the other data input port of the second multiplexer is communicatively connected to the in-core calculation result cache, and the instruction input port of the second multiplexer is communicatively connected to the in-core logic control unit; the second multiplexer selects data in the adder or the in-core calculation result cache to be sent to the accumulation result vector register; The data output port of the accumulation result connected register is communicatively connected to the data input port of the in-core calculation result cache to write a column vector of a result matrix after a round of calculation into the in-core calculation result cache.

7. An in-memory computing method, characterized in that: Applied to the in-memory computing core according to any one of claims 1 to 6, the in-memory computing method comprises the following steps: Send instructions to the instruction cache in the core, send weight data to the weight cache in the core, and send activation data to the activation cache in the core through the bus interface; The in-core logic control unit reads instructions from the in-core instruction cache, generates a plurality of logic control signals based on the instructions, and outputs each of the logic control signals to the in-core logic operation unit, the in-core scalar register, the in-core vector register, and the storage-calculation multiplier-adder array respectively; The in-core logic operation unit receives the control signal sent by the in-core logic control unit, reads the operation data from the in-core scalar register, and generates a feedback signal based on the operation data, wherein the feedback signal is used to control the in-core logic control unit or the storage-calculation multiplier-adder array; The storage and calculation multiplier array receives the control signal sent by the in-core logic control unit, reads weight data and activation data from the in-core weight cache and the in-core activation cache respectively, and performs multiplication and addition operations to obtain multiplication and addition operation results; the multiplication and addition operations include dense matrix multiplication and addition operations and sparse matrix multiplication and addition operations; the multiplication and addition operation results of the dense matrix are directly written into the accumulation result vector register; the multiplication operation results of the sparse matrix are first input into the vector scalar multiplication result vector register to complete format adjustment, and then sent to the accumulator to complete the sparse matrix multiplication and addition operation to obtain the multiplication and addition operation results; The accumulated result in the accumulated result vector register is a result matrix, and a column vector of the result matrix after a round of calculation is written into the in-core calculation result cache.

8. An in-memory processor, characterized in that: The device comprises K independent in-memory computing cores, a top-level controller, a top-level instruction cache, a top-level weight cache, a top-level activation cache, a clock, and a power supply; wherein the in-memory computing core is the in-memory computing core according to any one of claims 1 to 6; The top-level controller is connected to K independent in-memory computing cores through an instruction bus interface. The top-level controller reads instructions from the top-level instruction cache and determines whether the instructions are top-level execution instructions or in-memory computing core execution instructions. When the instructions are top-level execution instructions, the control program count continues to run; when the instructions are in-memory computing core execution instructions, the instructions are transmitted to the in-memory computing core to control the running state of the in-memory processor. A plurality of in-memory computing cores distributed along a first direction form an in-memory computing core unit, and a plurality of in-memory computing core units are distributed along a second direction, and the first direction and the second direction are perpendicular to each other; the top-level weight cache is communicatively connected to the top-level controller via a weight bus interface; the top-level weight cache is communicatively connected to each of the in-memory computing core units in the same manner as follows: the weight bus interface of the top-level weight cache is communicatively connected to the proximal in-memory computing core and the distal in-memory computing core respectively, and all the in-memory computing cores located in the same in-memory computing core unit are connected via the weight bus interface; The top-level activation cache is communicatively connected to the top-level controller via an activation bus interface; the top-level activation cache is communicatively connected to each of the in-memory computing core units in the same manner as follows: the activation bus interface of the top-level activation cache is communicatively connected to the in-memory computing core at the near end and the in-memory computing core at the far end, respectively, and all the in-memory computing cores in the same in-memory computing core unit are connected via the activation bus interface; The top-level controller, top-level instruction cache, top-level weight cache, top-level activation cache and all in-memory computing core units are connected through the instruction data interface, weight data interface and activation data interface to form an interconnected topology structure, and adjacent in-memory computing cores perform instruction transmission, weight data transmission and activation data transmission.

9. The in-memory processor according to claim 8, characterized in that: The top-level controller includes a top-level bus interface unit, a top-level logic control unit, a top-level logic operation unit, a top-level scalar register and an instruction distribution module; the top-level logic control unit is a general control, and is in communication connection with the top-level logic operation unit, the top-level scalar register, the top-level bus interface and the instruction distribution module; Wherein, the top-level bus interface unit includes a top-level host bus interface unit and a top-level slave bus interface unit, the top-level host bus interface unit is communicatively connected with the slave bus interface unit possessed by the in-memory computing core, and the top-level slave bus interface unit is communicatively connected with the host bus interface unit possessed by the in-memory computing core; The top-level logic control unit reads instructions from the top-level instruction cache, and determines whether the instructions are top-level execution instructions or in-memory computing core execution instructions. When the instructions are top-level execution instructions, the control program counter continues to run; when the instructions are in-memory computing core execution instructions, the instructions are transmitted to the in-memory computing core to control the running state of the in-memory processor; The top-level logic operation unit reads operation data from the top-level scalar register according to the top-level execution instruction output by the top-level logic control unit, operates on the operation data to obtain a feedback signal, and the feedback signal controls the top-level logic control unit to output the operation result and return it to the top-level scalar register or the secondary data cache; the secondary data cache includes a secondary weight cache and a secondary activation weight cache; The instruction distribution module receives instructions output by the top-level logic control unit, distributes top-level execution instructions to the top-level controller, and distributes in-core execution instructions to in-memory computing cores.

10. An in-memory processing method, characterized in that: The steps include: Send instructions from outside the in-memory processor to the top-level instruction cache, weight data to the top-level weight cache, and activation data to the top-level activation cache via the bus interface; The top-level logic control unit reads instructions from the top-level instruction cache, generates a plurality of logic control signals based on the instructions, and outputs each of the logic control signals to the top-level logic operation unit, the top-level scalar register and the instruction distribution module respectively; The top-level logic operation unit receives the control signal sent by the top-level logic control unit, reads operation data from the top-level scalar register, and generates a feedback signal based on the operation data, wherein the feedback signal is used to control the top-level logic control unit or the in-memory computing core; When the feedback signal controls the in-memory computing core, the instruction distribution module is used to send the in-memory computing core execution instruction to the in-core logic control unit of the in-memory computing core to control the in-core weight cache to read weight data from the top-level weight cache and control the in-core activation cache to read activation data from the top-level activation cache. At least one in-memory computing core performs operations according to the in-memory operation method described in claim 7.

Images