SoC system with in-memory / near-memory computing modules

Abstract

The present invention provides a SoC system with an in-memory / near memory computing module, comprising: a memory general bus, which is configured to transmit memory call instructions; and an in-memory / near memory computing module, which is configured to store data and perform operations on the stored data, wherein the in-memory / near memory computing module has a first interface, and the first interface is connected to the memory general bus, wherein the in-memory / near memory computing module and / or the first interface are constructed so that the in-memory / near memory computing module can be called by the memory call command transmitted on the memory general bus, so as to control and use the in-memory / near memory computing module.

Description

Technical Field

[0001] The present invention relates to the technical field of integrated circuits, and in particular to a SoC system with an in-memory / near-memory computing module. Background Art

[0002] Processing-in-memory (PIM) technology is one of the most promising candidates for AIoT applications due to its attractive properties, such as low computational latency, large throughput, and high energy efficiency. However, how to effectively utilize PIM in system-on-chip (SoC) architectures is rarely discussed.

[0003] The existing in-memory / near-memory computing system needs to use a software and hardware system designed independently for in-memory / near-memory computing to call the in-memory / near-memory computing unit to perform accelerated computing. This method has high design costs and low system versatility. Generally, the software and hardware system can only be designed for a specific algorithm. It will be difficult for the system to implement other algorithms. Summary of the invention

[0004] The object of the present invention is to provide a SoC system with an in-memory / near-memory computing module to solve the problems that the design cost of the existing in-memory / near-memory computing system on a chip is too high and the designed system is not universal.

[0005] In order to solve the above technical problems, the present invention provides a SoC system with an in-memory / near-memory computing module, comprising:

[0006] a memory universal bus configured to transmit memory call instructions; and

[0007] An in-memory / near-memory computing module is configured to store data and perform operations on the stored data, wherein the in-memory / near-memory computing module has a first interface, and the first interface is connected to a memory general bus, wherein the in-memory / near-memory computing module and / or the first interface are constructed so that the in-memory / near-memory computing module can be called by a memory call command transmitted on the memory general bus to control and use the in-memory / near-memory computing module.

[0008] Optionally, the SoC system with the in-memory / near-memory computing module further includes:

[0009] The embedded memory is configured to store data, and has a second interface connected to the memory universal bus.

[0010] Optionally, in the SoC system having an in-memory / near-memory computing module, the in-memory / near-memory computing module includes a heterogeneous memory, and the heterogeneous memory is configured to divide tensors using a tensor mapping method based on heterogeneous memory, and deploy general matrix multiplication operations on the in-memory / near-memory computing module.

[0011] Optionally, in the SoC system having the in-memory / near-memory computing module, the memory universal bus of the SoC system includes an AHB bus and / or an AXI bus.

[0012] Optionally, in the SoC system having an in-memory / near-memory computing module, the working mode of the in-memory / near-memory computing module includes a memory mode and a computing mode;

[0013] In memory mode, data of the in-memory / near-memory computing module is accessed through the R / W port to read and write data from the memory array of the in-memory / near-memory computing module;

[0014] In the calculation mode, the input data V in the vector forms a stream and enters the PIM memory. The input data V directly interacts with the stored data W in the PIM memory, and the calculation result I is output at the latch port.

[0015] Optionally, in the SoC system with in-memory / near-memory computing modules, the PIM memory calculates general matrix multiplication in an analog processing manner, and an analog-to-digital converter is used as a computing circuit interface:

[0016] I = VW, where V is the input data in the vector, W is the storage data in the PIM memory, and I is the calculation result, which is used as the output data;

[0017] The PIM memory array is tiled together with the instruction decoder and data path controller to form a PIM accelerator.

[0018] Optionally, in the SoC system with in-memory / near-memory computing modules, the heterogeneous memory simulates a PIM memory attached to the traditional memory in the SoC system;

[0019] The PIM input buffer and the PIM output buffer are composed of static random access memory as on-chip fast memory, which is used to cache input data and output data of the PIM memory, and the PIM memory is installed to the system bus through an interconnect interface;

[0020] The computation mode input is connected via a physical metal bus so that the PIM memory directly accesses the data of the PIM input buffer;

[0021] The PIM output buffer includes a virtual output buffer including an output data latch, a multiplexer / decoder, and a bus slave module to pass an addressable output data block of the system bus;

[0022] The PIM output buffer includes an additional output buffer that is used to buffer intermediate partial sums of complete GEMM operations.

[0023] Optionally, in the SoC system with the in-memory / near-memory computing module, the addressability of the system bus includes a common address allocation method, and the common addresses are divided according to the shape of the SoC system, the total address width of the SoC system, and the scale of the on-chip memory and the PIM memory, which includes:

[0024] The system address width is 32 bits, and each 64MB forms a block;

[0025] The addresses in Block0 and Block7 are reserved for CPU configuration, instruction memory, and internal peripheral circuit I / O;

[0026] Block1 is used to bridge the on-chip SRAM and external peripheral circuit I / O to the low-speed bus;

[0027] Block3 is used for PIM-related cache and PIM memory, as well as for GEMM acceleration;

[0028] Block 4 to 6 are expandable addresses.

[0029] Optionally, in the SoC system with in-memory / near-memory computing modules, based on the fact that most elements in vectors / matrices are aligned with linearly distributed addresses, the on-chip interconnect network is verified by simulating AHB communication to provide sufficient bandwidth for bidirectional data transmission related to PIM, so that the commonly used AHB can perform the following operations: obtain calculation results from the PIM output buffer and feed the input vector / matrix to the PIM input buffer.

[0030] Optionally, in the SoC system with the in-memory / near-memory computing module, high-dimensional matrix operations of a deep neural network are performed in the PIM memory, including:

[0031] The weight matrices of the two layers of perceptrons and the weight data stored in the memory are partitioned and fed into the PIM memory to improve computational efficiency;

[0032] All matrix multiplications in PIM memory are directly compiled into general memory call commands. PIM calculations are compiled through the gcc tool chain and optimized in software based on the general memory call commands.

[0033] The general memory call commands include: LD.I(x,y) is to export the input matrix from SRAM, LD.W(x,y) is to export the weight matrix from SRAM, SD.R(x,y) is to import the calculation results into SRAM, SD.M is to import the weight matrix into PIM, SD.V is to import the input data into inputbuffer, and LD.O is to export the calculation results from outputbuffer;

[0034] The (x,y) in the general memory call command represents the address deviation of the upper left corner element of the matrix in the traditional memory.

[0035] Optionally, in the SoC system having an in-memory / near-memory computing module,

[0036] Use multiple PIM and HMA structures to enhance system performance. Multiple PIM memories are scheduled in parallel so that the weight data stored in the PIM memory does not change frequently during NN calculations.

[0037] Calculated with each input associated with a certain weight already stored in the PIM matrix to reduce the frequency of reloading weight data in the PIM memory;

[0038] The calculation results loaded into different SRAMs in the same row are superimposed accordingly; when all multiplication results are ready, the addition operation is performed;

[0039] The addition results of different columns are horizontally spliced ​​to complete a GEMM operation.

[0040] The present invention also provides a method for using a SOC system, comprising:

[0041] Designing an in-memory / near-memory computing module based on the memory of the SoC system, so that the in-memory / near-memory computing module has a first interface;

[0042] The embedded memory of the SoC system is provided with a second interface;

[0043] The embedded memory is connected to the system bus through the second interface, wherein the system bus is a memory general bus of the SoC system;

[0044] The in-memory / near-memory computing module is connected to the memory universal bus of the SoC system through the first interface;

[0045] The first interface is made the same as the second interface, so as to call the in-memory / near-memory computing module through a general memory call command to control and use the in-memory / near-memory computing module;

[0046] The general memory call command is also used by the SoC system to call the embedded memory to control and use the embedded memory.

[0047] In the SoC system with in-memory / near-memory computing module provided by the present invention, by constructing the first interface to be able to receive memory call commands like the second interface (such as using interfaces of the same specifications) and constructing the in-memory / near-memory computing module to be able to execute the received memory call commands, the in-memory / near-memory computing module can be called by a general memory call command to perform operations, such as control and use, of the in-memory / near-memory computing module, thereby realizing the memory structure using in-memory computing, that is, designing a usage mode, so that the in-memory computing unit can be universally called in the SoC to perform various in-memory operations in the SoC, thereby greatly expanding the computing power of the SoC and reducing its power consumption and chip area. In terms of structure: the PIM system is mounted on the existing computer memory bus structure, which is convenient for design and use, and there is no need to design a customized interface for the PIM system. In terms of function: the PIM operation is completed by using widely used and unified standard instructions, which simplifies the difficulty of software and hardware collaborative design and solves the problem that the compiler is not easy to find the optimal instruction execution scheduling. Based on the use of in-memory computing and near-memory computing accelerators in a system on chip, the present invention proposes a universal calling method, so that this type of accelerator can be used in a traditional conventional system on chip (SoC) architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a schematic diagram of an application of in-memory computing in a SoC system according to an embodiment of the present invention;

[0049] Figure 2 It is a schematic diagram of an in-memory computing module in a SoC system according to an embodiment of the present invention;

[0050] Figure 3 It is a schematic diagram of the circuit module and mechanism of a PIM macro according to an embodiment of the present invention;

[0051] 4(a)-(b) are a hierarchical block diagram of an HMA and a schematic diagram of heterogeneous memory being managed by a bus subordinate according to an embodiment of the present invention;

[0052] Figure 5 This is a schematic diagram of typical address space allocation in HMA on a 32-bit address width SoC according to an embodiment of the present invention;

[0053] Figure 6 It is a schematic diagram of the bandwidth of the on-chip AHB bus and the throughput of the PIM macro in the computing mode of an embodiment of the present invention;

[0054] Figure 7 Schematic diagram of the optimization process of matrix multiplication according to an embodiment of the present invention;

[0055] Figure 8 It is a visualization diagram of a memory operation compilation sequence of a GEMM operation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention will be further described below in conjunction with specific embodiments with reference to the accompanying drawings.

[0057] It should be noted that the components in the figures may be shown exaggeratedly for the sake of illustration and are not necessarily true to scale. In the figures, identical or functionally identical components are provided with the same reference numerals.

[0058] In the present invention, unless otherwise specified, "arranged on...", "arranged above..." and "arranged above..." do not exclude the existence of an intermediate between the two. In addition, "arranged on or above..." merely indicates the relative positional relationship between two components, and in certain cases, such as after reversing the product direction, it can also be converted into "arranged below or below...", and vice versa.

[0059] In the present invention, each embodiment is only intended to illustrate the aspects of the present invention and should not be construed as limiting.

[0060] In the present invention, unless otherwise specified, the quantifiers "a", "an" and "an" do not exclude the presence of a plurality of elements.

[0061] It should also be noted that in the embodiments of the present invention, for the sake of clarity and simplicity, only a portion of the parts or components may be shown, but those of ordinary skill in the art will understand that under the teachings of the present invention, the required parts or components may be added according to the needs of the specific scenario. In addition, unless otherwise specified, the features in different embodiments of the present invention may be combined with each other. For example, a feature in the second embodiment may be used to replace a corresponding or functionally identical or similar feature in the first embodiment, and the resulting embodiment also falls within the disclosure scope or recorded scope of the present application.

[0062] It should also be noted that within the scope of the present invention, the terms "same", "equal", "equal to" and the like do not mean that the values ​​of the two are absolutely equal, but allow a certain reasonable error, that is, the terms also cover "substantially the same", "substantially equal", "substantially equal to". By analogy, in the present invention, the terms "perpendicular to", "parallel to" and the like indicating directions also cover the meanings of "substantially perpendicular to" and "substantially parallel to".

[0063] In addition, the numbering of the steps of the methods of the present invention does not limit the execution order of the method steps. Unless otherwise specified, the method steps can be executed in different orders.

[0064] The SoC system with in-memory / near-memory computing module proposed by the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become clearer based on the following description. It should be noted that the accompanying drawings are all in a very simplified form and are not in precise proportions, and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0065] The present invention relates to the field of artificial intelligence and the Internet of Things. The combination of artificial intelligence and the Internet of Things (AIoT) has spawned a variety of specific field accelerators to achieve low-power device intelligence. AIoT devices perform data analysis locally near sensors and typically run on batteries or energy harvesting power supplies below 10mW. Therefore, AIoT technology requires a low-cost and small-size system-on-chip (SoC) containing a general-purpose CPU and a programmable machine learning (ML) accelerator to take advantage of programmability and execution efficiency, such as Figure 1 As shown, a PIM design with different computing interface circuits is shown. The embedded data memory is used to store the processed data, the embedded instruction memory is used to store the compiled instructions, and the in-memory computing module can store and calculate, which is a function that some previous accelerators do not have. The present invention optimizes the embedded SoC with memory processing (PIM) technology to process tasks with a large number of general matrix multiplication (GEMM) operations, so that the PIM can perform computing operations directly in the memory array.

[0066] Furthermore, although the PIM design at the circuit level has been well explored, the method of connecting the PIM to the SoC remains unclear. In the state-of-the-art PIM hardware design, it has been considered as a mutated vector processing unit to form an accelerator, i.e., a PIM integrated with a custom instruction set architecture (ISA). However, the inventors found that the architecture efficiency at the SoC level may not be high because the additional custom instructions and shared memory allocation with the main processor cause compilation difficulties. Therefore, peak efficiency is rarely achieved in most cases.

[0067] The object of the present invention is to provide a SoC system with an in-memory / near-memory computing module to solve the problems that the design cost of the existing in-memory / near-memory computing system on a chip is too high and the designed system is not universal.

[0068] To achieve the above purpose, Figure 1As shown, the present invention provides a SoC system with an in-memory / near memory computing module, a memory universal bus, which is configured to transmit memory call instructions; and an in-memory / near memory computing module, which is configured to store data and perform operations on the stored data, wherein the in-memory / near memory computing module has a first interface, and the first interface is connected to the memory universal bus, wherein the in-memory / near memory computing module and / or the first interface are constructed so that the in-memory / near memory computing module can be called by the memory call command transmitted on the memory universal bus, so as to control and use the in-memory / near memory computing module.

[0069] like Figure 2 As shown, the bus interface is generally a popular AMBA bus protocol interface, such as AHB, AXI, AXI-lite, etc. The bus interface and the read-write circuit communicate with each other in the storage mode of read-write data (used as a memory), and the timing control module and the decoder (select control logic) perform timing control on the read-write circuit and the calculation circuit to control the address of the calculation unit array in the memory.

[0070] In one embodiment of the present invention, a SoC system having an in-memory / near-memory computing module further includes an embedded memory configured to store data, wherein the embedded memory has a second interface connected to a memory universal bus.

[0071] The present invention provides a method for using a SoC system, including: designing an in-memory / near-memory computing module based on the memory of the SoC system, so that the in-memory / near-memory computing module has a first interface; making the embedded memory of the SoC system have a second interface; making the embedded memory connected to the system bus through the second interface, wherein the system bus is a memory universal bus of the SoC system; making the in-memory / near-memory computing module connected to the memory universal bus of the SoC system through the first interface; making the first interface the same as the second interface, so as to call the in-memory / near-memory computing module through a universal memory call command, and control and use the in-memory / near-memory computing module; wherein the universal memory call command is also used for the SoC system to call the embedded memory, and control and use the embedded memory. The present invention proposes a series of solutions from hardware architecture to algorithm to maximize the advantages of PIM design.

[0072] In order to solve the above technical problems, the present invention proposes a heterogeneous memory (HMA) that optimizes the existing SoC with PIM through a high-throughput on-chip bus. Then, based on the given HMA structure, the present invention also proposes an HMA tensor mapping method to partition tensors and deploy general matrix multiplication operations on the PIM structure.

[0073] In order to solve the above technical problems, the HMA hardware and HMA tensor mapping methods of the present invention both utilize the programmability of mature embedded CPU solution stacks and maximize the high efficiency of PIM technology. Compared with 44.6% of other accelerator solutions (such as PUMA) in the design field, the entire HMA system can save 416 times of power. The experiments of the present invention also show that compared with the most advanced baseline and unoptimized PIM, the design of the present invention can reduce the operation delay of TinyML applications by 430 times and 11 times respectively.

[0074] In order to solve the above technical problems, the present invention proposes a heterogeneous memory architecture (HMA) to adapt to PIM on an existing embedded SoC architecture. The concept of "heterogeneous memory" is hereby proposed, indicating that this architecture has both PIM memory and traditional memory (memory that can only store and read data) on one SoC. In the proposed HMA, the PIM memory is connected to the on-chip high-throughput system bus like the traditional memory. As HMA significantly simplifies the program interface, the present invention also develops an HMA tensor mapping method as a software-to-hardware optimization for deploying GEMM to the proposed architecture.

[0075] The main contributions of the present invention include:

[0076] The present invention proposes a novel heterogeneous memory concept and architecture to expand the memory space through PIM memory.

[0077] The HMA scheme proposed in the present invention is the first architecture that illustrates how to connect a PIM to an off-the-shelf SoC.

[0078] The present invention designs the HMA tensor mapping method to partition the tensor and deploy the GEMM task to the proposed hardware architecture.

[0079] The present invention not only provides a hardware-independent way to develop PIM hardware, but can also be used as a pre-design specification estimate.

[0080] The present invention creates a software-to-hardware development flow for PIM to use the off-the-shelf gcc compiler, bridging the gap between hardware and software toolchains to develop PIM in SoC.

[0081] The present invention focuses on PIM made of embedded non-volatile memory technology to exploit its zero standby power consumption characteristics. The present invention evaluates the proposed HMA based on resistive random access memory (RRAM).

[0082] The present invention focuses on solving the following technical problems:

[0083] First, application-specific integrated circuit (ASIC) accelerators in SoCs such as Figure 1 shown. Figure 1 An existing scheme for connecting an application-specific integrated circuit (ASIC) accelerator in a SoC is shown. The inventors have found that ASIC accelerators are designed for certain tasks with application-specific optimized hardware implementations. ASIC accelerators or coprocessors often work in parallel with the main processor and share buses and memories with them. A fully customized instruction set architecture (ISA) or an extended ISA of a public standard, such as RISC-V, is used as a program interface from software to hardware. The memory that holds instructions ("instruction memory") is typically loaded outside the accelerator by connecting to the system bus. In such a SoC, the on-chip bus (such as AXI, AHB, etc.) is responsible not only for data communication, but also for transmitting instructions executed by the accelerator. This paradigm of shared memory and mixed transmission of accelerator instructions on the same bus makes compilation and scheduling very complex, and subsequently leads to design work and overhead.

[0084] Secondly, PIM technology transforms memory circuits by adding computing interface circuits. Figure 3 As shown, PIM has two working modes: memory mode and computation mode. In memory mode, data can be accessed through the R / W port to read and write data from the memory array; in computation mode, input data "V" in the vector forms a stream into the PIM memory. The input directly interacts with the data stored in the PIM memory unit, and the output can latch the port "I" at the output. The PIM memory calculates the GEMM in an analog processing manner, with an analog-to-digital converter (ADC) as the computation circuit interface: I = VW, where V is the incoming input, W is the data stored in the PIM memory, and I is the computation result. The PIM memory array is tiled together with the instruction decoder and data path controller to form a PIM accelerator. The most advanced PIM accelerator customizes instructions based on the new programming model of PIM. However, compiling and deploying software programs to PIM hardware is a very complex problem and is still under research and development.

[0085] In addition, for most AIoT applications, ML accelerators, especially deep neural networks (DNNs), focus on the acceleration of GEMM operations, as they account for more than 70% of the overall operations. Optimizing the execution speed of GEMM is a classic problem in computer science. At the algorithmic level, the time complexity of matrix multiplication can be reduced from O(n) to O(n) by the Strassen algorithm. 3 ) is reduced to O(n log 2 7 ), which can be reduced to O(n) by Winograd algorithm 2.376). At the microarchitecture level, optimizations are common in von Neumann computers, including improving memory access locality and using vector instructions to reduce memory access times and improve cache hit rates. In contrast, in the present invention, the present invention uses a hardware / software co-design approach to accelerate GEMM with the help of PIM technology.

[0086] To address the above challenges, the present invention proposes a heterogeneous memory architecture (HMA), which is a minimalist design that can upgrade existing SoC designs using PIM memory to improve performance and efficiency.

[0087] The core idea of ​​the HMA of the present invention is to attach a PIM memory to the SoC like a traditional memory, such as Figure 3 As shown in Figure 4, instead of using PIM memory to build an independent accelerator. Figure 4 illustrates how the entire HMA structure works on the SoC, including how in-memory computing is applied in the system, and the two directions of data transmission. Figure 4 (a) shows the CPU obtaining the calculation results from the in-memory computing module, and Figure 4 (b) shows the CPU sending the input to the in-memory computing module to start computing. The hardware components in the PIM memory are as follows: Figure 3 As shown. The PIM input / output buffer (on-chip fast memory implemented by static random access memory SRAM, used to cache computation input / output data in the PIM) and the PIM memory are both mounted to the system bus through an interconnect interface. The computation mode input is connected through a physical metal bus, so the PIM memory can directly access the data in the PIM input buffer. The output data latch, plus additional multiplexers / decoders and a bus slave module, form a "virtual output buffer", that is, an addressable output data block through the system bus. Additional PIM output buffers are required to cache the intermediate parts and of the complete GEMM operation. Through this hardware structure, the CPU instructs the PIM to program and execute GEMM calculations in memory.

[0088] Figure 5 The typical case of address allocation within the HMA is illustrated, including possible address allocations, where the numbers on the left need to be set according to the application. The entire address space can be divided into several blocks. The present invention takes the AIoT small SoC as an example for SoC design. The total width of the address is set to 32 bits. Each 64MByte forms a block. The addresses in blocks 0 and 7 are reserved for CPU configuration, instruction memory, internal peripherals, etc. Block 1 includes on-chip SRAM and external peripherals bridged to a low-speed bus. PIM-related buffers and PIM memories have block 3 that is used only for GEMM acceleration. The remaining blocks 4 to 6 in this embodiment can be used for future expansion. Note that different form factors of SoCs may result in different partitioning methods based on their unique total address widths and the size of on-chip memories and PIM memories.

[0089] A key prerequisite for HMA implementation is that the on-chip interconnect network can provide sufficient bandwidth for bidirectional data transfer related to PIM, i.e., (a) Case A: getting PIM calculation results from PIM output buffer; (b) Case B: feeding input vector / matrix to PIM input buffer. This is verified by simulating AHB communication. The register transfer level (RTL) code is synthesized and simulated on a 28nm technology node with an on-chip clock of 1GHz and a bus width of 128 bits. The PIM related information is summarized in Table 1.

[0090] Table 1 PIM memory array specifications

[0091] Capacity / Memory Array 256kb Calculate latency @ 4b precision 18.3ns Maximum number of PIM arrays / chip 8

[0092] The results are as follows Figure 6 As shown in Table 1, the Y axis is the data transfer rate. The upper x-axis is the transaction size of the AHB bus in burst increment mode, and the lower x-axis is the number of PIM memory arrays. The present invention simulates this mode because most of the elements in the vector / matrix are aligned with linearly distributed addresses. As shown in Table 1, the size of each PIM memory array is 256kb. Multiple PIM memory arrays work in parallel, making the input / output data transfer rate linear. Figure 6 (a) and (b) show the results for two cases. They correspond to the AHB manager read (case A) and write (case B) operations, respectively. It shows that the more PIM memory arrays are working simultaneously, the higher the data transfer rate of the bus. The transaction size should be increased to 64 bytes to cover the high PIM output throughput of up to 3000MByte / s for 8 PIM memory arrays. In this way, the commonly used AHB can handle PIM input and output without additional congestion or interconnect buffering.

[0093] In addition, based on the HMA hardware infrastructure, the present invention proposes a hardware-software co-design method, namely the HMA tensor mapping method, to improve the computational efficiency and maximize the advantages of the HMA structure.

[0094] Deep neural networks require a large number of high-dimensional matrix operations, which all occur in the PIM memory designed by the present invention. In most cases, the size of the operand matrix is ​​larger than the cell size of the single PIM memory provided. More specifically, in one embodiment of the present invention, the PIM memory performs I=VW in dimensions of I of 1×16, V of 1×64, and W of 64×16. The classic simple two-layer perceptron involves a weight matrix of 784×100, which is much larger than 64×16. Therefore, all relevant high-dimensional large-scale input data and weight data stored in traditional memory need to be partitioned and fed into the PIM memory to improve computational efficiency. Based on the method of the present invention, the present invention directly compiles all matrix multiplications in the PIM memory into a series of ordinary memory operation instructions. Compared with the traditional PIM accelerator attribute PIM in which the GEMM is controlled by custom instructions, this innovation can make PIM calculations easily compiled through the gcc tool chain and optimized in software according to these instructions. The detailed description of the PIM memory instructions is shown in Table 2. The note (x, y) in the instruction represents the address deviation of the upper left corner element of the matrix in the traditional memory.

[0095] Table 2 Description List

[0096] Instructions describe LD.I(x,y) Load matrix A from conventional memory LD.W(x,y) Load matrix B from conventional memory SD.R(x,y) Storing PIM results in traditional memory SD.M Store matrix B into PIM memory SD.V Store matrix A into the PIM input buffer LD.O Load calculation results from the PIM output buffer

[0097] For most designs, multiple PIM and HMA structures can be used to enhance system performance. These PIM memories should be scheduled in parallel to take advantage of the fact that the weight data stored in the PIM memory does not need to be changed frequently in NN calculations. Therefore, how to decompose these operations depends on how many HMA architectures are accessible. If the number of PIM memories is large enough to cover the entire matrix weight, the weight data can simply remain undivided and deployed in each PIM memory. The final result can be obtained by splicing all the corresponding sub-matrix multiplication results. However, in most real-world situations, the PIM and HMA structures are not sufficient to directly cover the matrix weights in NN calculations. Due to such hardware limitations, the weight data in these PIMs cannot remain unchanged. Unlike traditional GEMMs, which run GEMM matrices one by one and frequently exchange weight data, the present invention splits the operation into multiplication and addition groups. Each input related to a certain weight that has been stored in the PIM matrix needs to be calculated together, so that the frequency of reloading weight data in the PIM memory can be reduced. The calculation results loaded into different SRAMs in the same row are correspondingly superimposed. When all multiplication results are ready, the addition operation is performed. In the last step, the addition results of different columns are horizontally spliced ​​to complete a GEMM operation.

[0098] Figure 7The pseudocode of GEMM before and after optimization and the graphical model of the HMA tensor mapping method are shown. For example, in the PIM structure, the matrix multiplication C = A × B is calculated, where the sizes of the operand matrices A and B are 100 × 400 and 400 × 80, respectively. The size of the PIM input is 1 × 64, and the size of the PIM memory GEMM is 64 × 16 for each operation. For simplicity, the present invention assumes that there is only one HMA structure. HMA-PIM (such as Figure 8 This is done in the following steps:

[0099] 1. "LD W (0, 0)" & "SD.M": Take out the data of size 64 × 16 from the (0, 0) position of the B matrix and store it in the PIM memory of the HMA;

[0100] 2. "LD I (0, 0)" & "SD.V": Get 1*16 size data from the (0, 0) position of the A matrix and store it in the input buffer of HMA;

[0101] 3. "LD O" & "SD R (0, 0)": Get the result from the output buffer and store it in the traditional memory;

[0102] 4. “LD I (1.0); LD I (2.0); …; LD I (99, 0)”: repeat this process, take data from one matrix in turn, and calculate in PIM;

[0103] 5. "SD R (1.0); SD.R (2.0); ...; SD R (99.0)": write back the calculation results of PIM,

[0104] 6. “LD W(0,16)”: Update the data in the PIM memory and repeat steps 1-5.

[0105] Frequent memory access will lead to additional overhead. Therefore, the present invention reduces its memory access frequency through PIM. The present invention provides an analysis of memory access in GEMM. For a given matrix multiplication problem, calculate C = A × B, and write it as an element-by-element expression:

[0106]

[0107] The matrix sizes of A, B, and C are m×k, k×n, and m×n, respectively. The total number of memory accesses using CPU cycles is RT classic for:

[0108] RT classic =(2+1+1)·m·n·k=4·m·n·k (2)

[0109] Where m, n, and k are the cumulative sum of the cycles; 2+1+1 is the sum of the frequencies that C, A, and B need to access, which involves data transmission delay. Based on the above formula, the number of memory accesses is 4×100×400×80=1.28×10 7 .

[0110] If PIM is used for calculation, the memory access frequency RT PIM It can be described as:

[0111]

[0112] parRowA, parColA, parRowB, and parColB represent the number of partition matrices A and B in the PIM input buffer and PIM memory, respectively. col is the size of the input buffer, and PIM row and PIM col Is the size of the PIM memory.

[0113] Going back to the example above, the number of memory accesses is reduced to 6*100*400 / 64*80 / 16=2.1*10 4 (The number of accesses brought by PIM memory is reduced by a factor of 100). By using multiple cores and the proposed stitching method, the memory access frequency is further reduced to:

[0114] RT opt =par RouB·parColB·(2+4·parRowA) (4)

[0115] Then, the number of memory accesses in the same example above can be further reduced to 400 / 64*80 / 16*(2+4*100)=1.407*10 4 (This is reduced by a factor of 1.5 due to the proposed splicing method).

[0116] The toolkit compiles matrix multiplication into a PIM-compatible instruction series such as Figure 8 As shown, given arbitrary sizes of matrices A and B, the sizes of the input buffer and PIM memory, and the number of PIM memories working simultaneously.

[0117] This paper proposes a heterogeneous memory to improve the efficiency of PIM on traditional small-scale embedded SoC. On this basis, this paper further proposes a mapping algorithm to better utilize the acceleration of PIM. The power consumption and operation delay are fully explored in several common AIoT applications. This analysis can provide a good guide for top-level hardware-software co-design in the early design stage of PIM-related SoC design.

[0118] In summary, the above embodiments describe in detail different configurations of a SoC system with an in-memory / near-memory computing module. Of course, the present invention includes but is not limited to the configurations listed in the above embodiments. Any content that is transformed based on the configurations provided in the above embodiments belongs to the scope of protection of the present invention. Those skilled in the art can draw inferences based on the contents of the above embodiments.

[0119] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0120] The above description is only a description of the preferred embodiments of the present invention, and is not intended to limit the scope of the present invention. Any changes or modifications made by a person skilled in the art in the field of the present invention based on the above disclosure shall fall within the scope of protection of the claims.

Claims

1. A SoC system with an in-memory / near-memory computing module, comprising: a memory universal bus configured to transmit memory call instructions; as well as An in-memory / near-memory computing module is configured to store data and perform operations on the stored data, wherein the in-memory / near-memory computing module has a first interface, and the first interface is connected to a memory universal bus, wherein the in-memory / near-memory computing module and / or the first interface are constructed so that the in-memory / near-memory computing module can be called by a memory call command transmitted on the memory universal bus to perform operations on the in-memory / near-memory computing module, The in-memory / near-memory computing module includes a PIM input / output buffer, a PIM memory, a virtual output buffer and a PIM output buffer, wherein the PIM input / output buffer and the PIM memory are mounted to a memory universal bus via a first interface, the PIM memory directly accesses data in the PIM input buffer, wherein the virtual output buffer includes an output data latch, a multiplexer / decoder and a bus slave module, the virtual output buffer outputs data blocks addressable via a system bus, wherein the PIM output buffer caches intermediate portions and of complete GEMM operations.

2. The SoC system with in-memory / near-memory computing module according to claim 1, further comprising: The embedded memory is configured to store data, and has a second interface connected to the memory universal bus.

3. The SoC system with in-memory / near-memory computing module according to claim 2, characterized in that: The in-memory / near-memory computing module includes a heterogeneous memory, which is configured to partition tensors using a heterogeneous memory-based tensor mapping method and perform general matrix multiplication operations on the in-memory / near-memory computing module.

4. The SoC system with in-memory / near-memory computing module according to claim 3, characterized in that: The memory universal bus of the SoC system includes an AHB bus and / or an AXI bus.

5. The SoC system with in-memory / near-memory computing module according to claim 4, characterized in that: The working modes of the in-memory / near-memory computing module include memory mode and computing mode; In memory mode, the data of the in-memory / near-memory computing module is accessed through the R / W port. In the calculation mode, the input data V in the vector forms a stream into the PIM memory to read and write data from the memory array of the in-memory / near-memory calculation module; the input data V directly interacts with the storage data W in the PIM memory and outputs the calculation result I at the latch port.

6. The SoC system with in-memory / near-memory computing module according to claim 5, characterized in that: The addressability of the system bus includes common address allocation methods. Common addresses are divided according to the shape of the SoC system, the total address width of the SoC system, and the size of the on-chip memory and PIM memory, including: The system address width is 32 bits, and each 64MB forms a block; The addresses in Block0 and Block7 are reserved for CPU configuration, instruction memory, and internal peripheral circuit I / O; Block1 is used to bridge the on-chip SRAM and external peripheral circuit I / O to the low-speed bus; Block3 is used for PIM-related cache and PIM memory, as well as for GEMM acceleration; Block 4 to 6 are expandable addresses.

7. The SoC system with in-memory / near-memory computing module according to claim 6, characterized in that: High-dimensional matrix operations for deep neural networks in PIM memory include: The weight matrices of the two layers of perceptrons and the weight data stored in the memory are partitioned and fed into the PIM memory to improve computational efficiency; All matrix multiplications in PIM memory are directly compiled into generic memory call commands. PIM calculations are compiled through the gcc toolchain and optimized in software based on generic memory call commands.

8. The SoC system with in-memory / near-memory computing module according to claim 7, characterized in that: The general memory call commands include: LD.I(x,y) is to export the input matrix from SRAM, LD.W(x,y) is to export the weight matrix from SRAM, SD.R(x,y) is to import the calculation results into SRAM, SD.M is to import the weight matrix into PIM, SD.V is to import the input data into the input buffer, and LD.O is to export the calculation results from the output buffer; The (x,y) in the general memory call command represents the address deviation of the upper left corner element of the matrix in the traditional memory.

9. The SoC system with in-memory / near-memory computing module according to claim 8, characterized in that: Use multiple PIM and HMA structures to enhance system performance. Multiple PIM memories are scheduled in parallel so that the weight data stored in the PIM memory does not change frequently during NN calculations. Calculated with each input associated with a certain weight already stored in the PIM matrix to reduce the frequency of reloading weight data in the PIM memory; The calculation results loaded into different SRAMs in the same row are superimposed accordingly; when all multiplication results are ready, the addition operation is performed; The addition results of different columns are horizontally spliced ​​to complete a GEMM operation.

10. A method for using a SoC system, characterized in that: include: Providing an in-memory / near-memory computing module having a first interface; Providing an embedded memory of a SoC system, which has a second interface; Connecting the in-memory / near-memory computing module to the system bus via a first interface; Connecting the embedded memory to the system bus through the second interface, wherein the system bus is a memory general bus of the SoC system; as well as The in-memory / near-memory computing module and / or the first interface are constructed so that a general memory call command can call the in-memory / near-memory computing module to control and use the in-memory / near-memory computing module, The in-memory / near-memory computing module includes a PIM input / output buffer, a PIM memory, a virtual output buffer and a PIM output buffer, wherein the PIM input / output buffer and the PIM memory are installed to the memory universal bus through a first interface, the PIM memory directly accesses data in the PIM input buffer, wherein the virtual output buffer includes an output data latch, a multiplexer / decoder and a bus slave module, and the virtual output buffer outputs data blocks addressable by a system bus, wherein the PIM output buffer caches intermediate portions and of complete GEMM operations.

Images