In-memory processing

Abstract

The application provides a memory computing device and a memory, and belongs to the technical field of the memory. The device comprises a global controller, N operation channels, a gate unit, M memory computing tiles, a read enable signal line and a row address bus. The global controller is used for sequentially controlling the N operation channels to be enabled in response to an operation instruction. When one operation channel is enabled, the gate unit of the operation channel is synchronously activated through the read enable signal line, and a row address is broadcast to each memory bank in the operation channel through the shared row address bus and the row address bus. Weight data in each memory bank is read in parallel, and the weight data is output to a corresponding calculation unit for multiply-accumulate calculation to obtain a partial sum corresponding to the operation channel. The partial sums obtained by the N operation channels are accumulated to obtain an operation result. Through the memory computing device provided by the application, the dependence on an external high-bandwidth memory can be eliminated, and a memory wall bottleneck can be solved.

Description

Technical Field

[0001] This invention belongs to the field of memory technology, specifically relating to a storage computing device and a memory. Background Technology

[0002] As large language models evolve towards a Mixture of Experts (MoE) architecture, the inference phase requires dynamic activation of sparse expert weights, placing extremely high demands on the parallel bandwidth, capacity, and access latency of on-chip storage. Existing solutions face multiple bottlenecks: the capacity of a single SRAM (Static Random-Access Memory) storage unit is limited, making it difficult to support complete expert weights. Summary of the Invention

[0003] In view of the above problems, embodiments of this application provide a storage computing device and a memory to overcome or at least partially solve the above problems.

[0004] In a first aspect, this application provides a memory computing device, comprising: a global controller; N computing channels, each computing channel including a gating unit, M memory tiles, a read enable signal line, and a row address bus; each memory tile including P memory cells and P computing units connected one-to-one with the P memory cells; each memory cell including a memory cell array; and N, M, and P being integers greater than or equal to 2; wherein, the gating terminal of the gating unit is connected to the gating signal output terminal of the global controller; the address terminal of the P memory cells in each memory tile is connected to the same row address bus; the row address buses of the M memory tiles are connected to a shared row address bus; the read enable terminal of the P memory cells in each memory tile is connected to the same read enable signal line; the shared row address bus is connected to the row address output terminal of the global controller; and the read enable signal line is connected to... The output terminal of the gating unit is connected; the global controller is used to respond to the operation instructions, sequentially control the gating units of the N operation channels to select, and for each operation channel selected, it synchronously activates each memory bank in each memory tile of the operation channel through the read enable signal line, and broadcasts the row address to each memory bank in the operation channel through the shared row address bus and the row address bus, so as to read the weight data in each memory bank in parallel and output it to the corresponding calculation unit for multiplication and accumulation calculation to obtain the first part sum corresponding to each memory bank, and accumulate the first part sum corresponding to each memory bank in the same memory tile to obtain the second part sum of each memory tile, for a total of M second part sums, which are used as the operation result of the operation channel; the global controller is also used to accumulate the operation results of the N operation channels to obtain the target operation result.

[0005] In some embodiments, the global controller includes a global control module and a secondary address decoder. The global control module is connected to the input of the secondary address decoder, the address output of the secondary address decoder is connected to the shared row address bus, and the strobe signal output of the secondary address decoder is connected to the gating unit of each computation channel via a strobe signal line. The global control module is used to output a global address to the secondary address decoder in response to an computation instruction. The global address includes a strobe address and a row address. The secondary address decoder is used to output a strobe signal to the gating unit of each computation channel based on the strobe address, so as to control the gating unit of the computation channel corresponding to the strobe address to be in the open state, and synchronously activate each memory bank in each memory tile in the computation channel and broadcast the target weight row address to each memory bank in the computation channel.

[0006] In some embodiments, an output register is also included. The global controller further includes a global adder tree circuit. The input terminals of the global adder tree are connected to the output terminals of each of the N operation channels, and the output terminals of the global adder tree are connected to the output register. The global adder tree is used to accumulate the operation results of the N operation channels to obtain the target operation result, and output the target operation result to the output register.

[0007] In some embodiments, each computation channel further includes a partial sum register. In the same computation channel, the input of the partial sum register is connected to the output of each computation unit in the computation channel, and the output of the partial sum register is connected to the global addition tree. The partial sum register is used to store the first partial sum output by each computation unit in the computation channel, and to accumulate the first partial sums output by P computation units in the same storage tile to obtain a second partial sum, and to output M second partial sums to the global addition tree.

[0008] In some embodiments, the portion and register are a register group with a bit width of M×32 bits, and the calculation unit is a multiply-accumulate unit.

[0009] In some embodiments, the N computing channels are arranged at intervals along a first direction, and the M memory tiles in each computing channel are arranged at intervals along a second direction, wherein the first direction and the second direction are perpendicular to each other.

[0010] In some embodiments, the spacing between each pair of adjacent memory tiles in each computing channel is equal along the second direction; the shared row address bus in the same computing channel extends along the first direction and is connected to each memory cell in the same computing channel through the row address bus.

[0011] In some embodiments, the spacing between any two adjacent computing channels in the N computing channels along the first direction is equal, and a power isolation band and a clock buffer band are provided between any two adjacent computing channels.

[0012] In some embodiments, the P-cells in each of the computing tiles are arranged at intervals along the first direction, the spacing between any two adjacent cells along the second direction is equal, and a wiring gap is reserved between the two adjacent cells.

[0013] A second aspect of this application provides a memory, which includes the memory computing device described in the first aspect of this application.

[0014] The in-memory computing device of this embodiment includes: a global controller; N computing channels, each computing channel including a gating unit, M computing tiles, a read enable signal line and a row address bus. Each computing tile includes P memory cells and P computing units. Each memory cell is directly connected to one computing unit, so that the weight data of all memory cells in a computing channel can be read out in parallel at the same time and sent to their respective directly connected computing units to perform multiplication and accumulation operations synchronously. The weight data does not pass through multiplexers and shared buses on the entire path from reading from the memory cell to entering the computing unit, and there is no need for bus arbitration and multi-level gating, thereby eliminating the combinational logic delay and crosstalk bottleneck introduced by multiple memory cells sharing the row address bus in traditional centralized architecture. Simultaneously, during computation, the global controller sequentially selects each channel through the gating unit and read enable signal line. All memory banks within the selected channel synchronously receive the same row address based on the shared row address bus, ensuring that the expert weights are fully resided and synchronously read out for calculation, thus obtaining the first partial sum of each memory bank. Subsequently, the first partial sums of each memory bank within the same computing tile are accumulated to obtain the second partial sum of a computing tile. Then, the M second partial sums within the same channel are used as the computation result of a computing channel. Finally, the global controller accumulates the second partial sums of all selected channels to obtain the target computation result, thereby completely eliminating the dependence on external high-bandwidth memory and solving the memory wall bottleneck. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the frame of a storage computing device provided in an embodiment of this application; Figure 2 It is aimed at Figure 1 A schematic diagram showing the connection between a single storage tile and the global addition tree; Figure 3 This is a schematic diagram showing the connection of the storage tiles in the N computing channels provided in this application embodiment; Figure 4 This is a schematic diagram showing the connection of the storage cells in the M memory tiles provided in the embodiments of this application; Figure 5 This is a schematic diagram of the computation timing of a single-cycle computing unit provided in an embodiment of this application; Explanation of reference numerals in the attached figures: 1-Strobe signal line; 2-Shared row address bus; 3-Row address bus; 4-Read enable signal line. Detailed Implementation

[0017] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0018] In related technologies, the MC (Memory Compiler) tools provided by mainstream manufacturers are limited by physical design constraints such as memory array decoding drivers and sensitive amplifier layouts. The maximum capacity of a single bank (memory bank) of a standard single-port 6T SRAM IP that can be generated is only 128KB (e.g., a specification of 1024 wide × 1024 bit deep). This capacity is insufficient to completely store the FFN weight matrix of a single MoE expert. Taking a certain artificial intelligence model with a matrix size of 3072 × 7168 as an example, storing FP8 (8-bit / 4-bit Floating-Point) per unit, the total capacity is 24MB. It is necessary to expand the scale horizontally and vertically through multiple banks to build a large-capacity SRAM array to meet the weight storage requirements.

[0019] Secondly, traditional SRAM IPs are designed to adapt to the on-chip bus width specifications of mainstream SoC (System-on-Chip) chip architectures such as NoC (Network-on-Chip) and AMBA (Advanced Microcontroller Bus Architecture). The maximum data bit width of a single port is typically limited to 64 bits, making it impossible to achieve parallel reading of 1024 columns of weighted data in a single cycle. Furthermore, there are no readily available large-width, large-capacity standardized SRAM IPs that can be directly used in MoE in-memory arrays. Developing fully custom SRAM cells presents engineering challenges such as high difficulty in process verification, high tape-out risk, and long development cycles. Moreover, existing large-capacity SRAM arrays mostly adopt a centralized architecture of "multi-bank + shared bus + multiplexer." Weight reading requires multiple levels of multiplexer selection and bus arbitration, introducing additional high combinational logic latency, which severely restricts high-frequency timing convergence above 1GHz. At the same time, the shared bus has crosstalk and load bottlenecks, making it unable to support parallel reading of multiple banks across the entire array, and difficult to match the high bandwidth computing power requirements of MoE in-memory arrays. Mainstream MoE acceleration solutions largely rely on external HBM (High Bandwidth Memory) storage expert weights, resulting in memory access latency of tens of nanoseconds. The bandwidth is far below the computational requirements of the in-memory computing array, creating a severe storage wall – a major core issue plaguing the industry. Furthermore, with the rising cost of HBM, it has surpassed the cost of SRAM, especially in the highly linear, computationally intensive MoE FFN (Feed-Forward Network) stage, significantly limiting the efficiency and cost of token generation.

[0020] In view of this, to solve the above problems, this embodiment provides a memory computing device. Through a global controller, N computing channels, each computing channel including a gating unit, M memory tiles (which can be called titles), read enable signal lines, and a row address bus, a three-level hierarchical expansion of N computing channels, M memory tiles, and P memory chips is achieved. This splices together a large number of standard small-capacity SRAM memory chips into a large-scale memory array with a total capacity sufficient to fully support the MoE expert FFN weight matrix, bypassing the limitation of the single memory chip capacity limit. Each memory chip is directly connected point-to-point to a computing unit. Ultra-wide weight data is read in parallel from each memory chip within the same clock cycle and directly sent to the corresponding computing unit for local multiplication and accumulation operations. The weight data does not pass through multiplexers and shared buses throughout the entire path, eliminating the combinational logic delays and crosstalk bottlenecks introduced by multi-level gating and bus arbitration in the traditional "multi-memory chip + shared bus + multiplexer" architecture, and supporting GHz-level high-frequency timing convergence. Meanwhile, the global controller uses a gating unit to select each computing channel in a time-sharing manner. During the inference phase, only the target computing channel is turned on in a single cycle, while the other computing channels are turned off, minimizing the dynamic power consumption and static leakage power consumption of the idle channels. All expert weights are stored in the on-chip SRAM array and read out for calculation on demand in a time-sharing manner, eliminating the dependence on external HBM and reducing the latency and cost of token generation while ensuring high bandwidth computing power.

[0021] The specific implementation method of the in-memory computing device is described below.

[0022] Figure 1 This is a schematic diagram of the framework of a memory computing device provided in an embodiment of this application, as shown below. Figure 1 As shown, the in-memory computing device includes: a global controller; N computing channels (which can be called lanes), each computing channel including a gating unit, M computing tiles (which can be called titles), a read enable signal line and a row address bus, each computing tile including P memory banks (which can be called banks) and P computing units (which can also be called MACs) connected one-to-one with the P memory banks, where N, M and P are integers greater than or equal to 2.

[0023] The address terminals of the P-chips in each memory tile are connected to the same row address bus, and the read enable terminals of the P-chips in each memory tile are connected to the same read enable signal line. Each memory chip is connected to a computing unit. Therefore, the P-chips in the same memory tile share the same set of row addresses and read enable signals, word lines are activated synchronously, the weights of the P-chips are read synchronously, and each memory chip outputs an independent M / 8-bit row address bus. The parallel outputs of the P-chips are concatenated to a width of M bits, and M FP8 format weights can be output in a single cycle. If P=8, N=8, and M=8, each M / 8-bit row address bus can be directly connected to the corresponding M / 8 input computing unit point-to-point through an equal-length metal wire. There is no shared bus, no multiplexer, and no data transfer logic, eliminating additional combination delays.

[0024] In this embodiment, each memory bank includes a memory cell array, such as an SRAM memory cell array. For example, the memory bank can be a standard 6T (6-Transistor) single-port or dual-port bank, with a single bank (memory bank) specification of 1024 bits wide × 1024 bits deep. It supports parallel readout of the entire row, has no internal multiplexer, and exhibits minimal native readout latency, representing a timing-optimal physical implementation. Each P SRAM bank is horizontally tiled to form a computing tile, and one computing tile can serve as a minimum weight output unit. The row address bus of the M memory tiles is connected to the shared row address bus, the shared row address bus is connected to the row address output of the global controller, the gating terminal of the gate unit is connected to the gating signal output of the global controller, and the read enable signal line is connected to the output of the gate unit.

[0025] In this embodiment, each M storage tiles constitute one computing channel, which can correspond to the 1024-dimensional input segment of the MoE FFN matrix. There are a total of N / 1024 computing channels in the macro, corresponding to three segments of the N-dimensional input.

[0026] The gating unit can be composed of clock gating and power gating, which can individually activate and deactivate each operation channel. During the inference phase, the global controller sequentially selects each operation channel according to the operation instructions. Within a single clock cycle, only the clock gating and power gating of the operation channel are activated, while the clock gating and power gating of the remaining operation channels are deactivated. The internal memory clock of the deactivated operation channel stops toggling, and the power supply is cut off, thus minimizing the dynamic power consumption and static leakage power consumption of idle operation channels. The M second-part sums output by each operation channel are also output independently and are not output together with the M second-part sums of other operation channels.

[0027] The computation unit can be a MAC (Multiply-Accumulate). The data output port of each memory bank is directly connected to the weight input of a multiply-accumulate. The multiply-accumulate also includes a data input for inputting the activation value for multiply-accumulate calculation. Figure 2 It is aimed at Figure 1 A diagram illustrating the connection between a single stored computation tile and the global addition tree is provided, as follows: Figure 2 As shown, when each memory tile contains P memory cells directly connected to P computing units, the memory cells and computing units can be connected without going through a multiplexer or shared bus. During the clock cycle when the computing channel is activated, each memory cell synchronously reads the weight data based on the row address broadcast by the shared row address bus, and sends it directly to the weight input terminal of the corresponding computing unit via the point-to-point direct connection path between the P memory cells and the P computing units. At the same time, the activation data also arrives at the other input terminal of the computing unit. Each computing unit completes the multiplication operation of the weight data and the activation data within a single clock cycle, thus obtaining the first part and output corresponding to that computing unit.

[0028] The global controller responds to computation instructions, sequentially controlling the selection of N computation channels. For each selected channel, the gating unit of the channel is synchronously activated via the read enable signal line. The controller then broadcasts the row address to each memory bank within the selected channel via the shared row address bus. In parallel, it reads the weight data from each memory bank within the selected channel and outputs it to the corresponding computation unit for multiplication and accumulation calculations, obtaining the first partial sum for each memory bank. Then, it accumulates the P first partial sums corresponding to the same memory chip to obtain the second partial sum for each memory chip. The M second partial sums obtained within the same computation channel are used as the computation result for that channel. Finally, other computation channels are sequentially selected to obtain the M second partial sums corresponding to each of the N computation channels. The computation results of the N computation channels are then accumulated to obtain the target computation result.

[0029] In summary, by expanding the N computation channels, the M memory tiles within each computation channel, and the P memory chips per memory tile into a three-level hierarchical structure, with each memory chip directly connected to the computation unit point-to-point, and combined with the time-division multiplexing of the gating unit and the synchronous broadcast of the shared row address bus, the upper limit of single memory chip capacity and the standard SRAM bit width limitation can be overcome. This eliminates the combinational logic delay and crosstalk bottlenecks introduced by multiplexers and bus arbitration in traditional multi-memory shared bus architectures. At the same time, the power consumption of idle computation channels is minimized by the gating strategy of only conducting computation channels in a single cycle. Thus, while fully carrying the MoE expert weight matrix, high-frequency timing convergence and low-power inference are achieved, and the dependence on external high-bandwidth memory is eliminated.

[0030] Figure 3 This is a schematic diagram showing the connection of the storage tiles in the N computing channels provided in the embodiments of this application, as follows: Figure 3 As shown, the global controller includes a global control module and a secondary address decoder. The global control module is connected to the input of the secondary address decoder, the address output of the secondary address decoder is connected to the shared row address bus, and the strobe signal output of the secondary address decoder is connected to the gating unit of each operation channel via the strobe signal line. The global control module is used to output a global address to the secondary address decoder in response to the operation instruction. The global address includes the strobe address and the row address.

[0031] Therefore, in this embodiment, the global control module can generate a global address containing a strobe address and a row address according to the operation instructions. The strobe address is used to specify the operation channel that needs to be activated, and the row address is used to select a row of storage cells in the memory body through the row decoder, so that the weight data stored in that row is read out to the bit line. The full macro can use a 13-bit global address, with the high 2 bits as the strobe address and the low 10 bits as the row address. The highest log(N / 1024) bits of the global address are input to the N / 1024-to-1 decoder, and output N / 1024 lane strobe signals, activating only the address path of the target lane and disabling the clock gating of the address bus of idle lanes; this stage of decoding has a fan-out of only N / 1024 channels, with extremely low load and negligible latency. The low 10 bits of the global address are the row address, which is only broadcast locally within the selected lane, driving the row decoders of all SRAMs in that lane; through distributed buffer tree driving and equal-length wiring, the synchronization of the entire lane address is guaranteed.

[0032] The secondary address decoder is used to output a gating signal to the computation channel corresponding to the gating address based on the gating address, so as to control the gating unit of the computation channel corresponding to the gating address to be in the open state, and simultaneously activate each memory bank in each memory tile in the computation channel and broadcast the target weight row address to each memory bank in the computation channel.

[0033] Therefore, after the secondary address decoder receives the global address output by the global control module, it can decode the gating address to generate a gating signal corresponding to each operation channel. This gating signal is then output to the gating terminal of the corresponding operation channel's gate control unit via the gating signal line, ensuring that only one operation channel is activated at any given time, while the gating units of the remaining operation channels remain off. Simultaneously, the secondary address decoder also outputs the row address from the global address to the shared row address bus via its address output terminal. The row address transmitted on the shared row address bus drives the row decoders of all memory tiles within the operation channel and is synchronously broadcast to each memory bank of each memory tile within the operation channel. This restricts the transmission range of the row address to within the operation channel, reducing the equivalent load capacitance of the address bus and avoiding timing bottlenecks and long-distance interconnect signal deviations caused by the high fan-out global address bus in multi-bank parallel addressing schemes.

[0034] Continue to refer to Figure 1 The in-memory computing device includes an output register, and the global controller also includes a global addition tree. The input of the global addition tree is connected to the output of each operation channel, and the output of the global addition tree is connected to the output register. The global addition tree is used to accumulate the operation results of N operation channels to obtain the target operation result, and output the target operation result to the output register.

[0035] In this embodiment, Figure 4 This is a schematic diagram showing the connection of the storage cells in the M memory tiles provided in the embodiments of this application, combined with... Figure 1 , Figure 2 and Figure 3 It can be seen that each operation channel, after completing the weight readout and multiplication-accumulation operations during its conduction period, outputs the corresponding M second-part sums. The input terminals of the global adder tree are connected to the output terminals of each operation channel, and sequentially receive the M second-part sums generated by the currently activated operation channel in each clock cycle. Since the N operation channels are activated sequentially in a time-division manner, the global adder tree can accumulate and update the sum of the operation results of each operation channel received at the current time and the stored accumulated intermediate value in each clock cycle, until all N operation channels are activated and the corresponding partial sums are accumulated, finally obtaining the target operation result, and finally outputting the target operation result to the output register.

[0036] By using an up-accumulation method, the channel-by-channel serial accumulation replaces the centralized accumulation scheme in the traditional fully parallel architecture, which requires aggregating the partial sums of all channels at the same time. This avoids the high peak power consumption and instantaneous current requirements brought about by fully parallel accumulation. At the same time, the global addition tree only needs to process the partial sum of a single operation channel in each clock cycle, which can reduce the hardware size and wiring complexity.

[0037] Continue to refer to Figure 1and Figure 4 Each operation channel also includes a partial sum register. The input of the partial sum register is connected to the output of multiple computing units, and the output of the partial sum register is connected to the global addition tree. The partial sum register stores the first partial sum output by each computing unit in the operation channel, and accumulates the first partial sums output by P computing units in the same storage tile to obtain the second partial sum. The M second partial sums are then output to the global addition tree.

[0038] In this embodiment, during the clock cycle when the computation channel is activated, the computation units of the M in-memory tiles within that channel synchronously complete multiplication-accumulation operations, generating M second partial sums. These M second partial sums are generated in parallel within the computation channel, but the global addition tree processes data from only one computation channel per clock cycle. The input of the partial sum register is connected to the output of multiple computation units within the same computation channel. At the end of the clock cycle, the first parts output by each computation unit are accumulated to obtain the second partial sum, and then the M partial sums calculated by that computation channel are obtained. The partial sum register can centrally store and output the M second partial sums generated in parallel by each in-memory tile within the computation channel, allowing the global addition tree to be connected only to the partial sum registers of each computation channel, rather than directly to each computation unit of each in-memory tile. This simplifies the number of inputs to the global addition tree and reduces the complexity of the top-level routing.

[0039] In one embodiment, the partial sum register is a register group with a bit width of M×32 bits, and the calculation unit is a multiply-accumulate unit.

[0040] In this embodiment, after each in-memory tile completes multiplication and accumulation operations with P memory cells and P computing units, the in-memory tile generates a partial sum. M in-memory tiles operate in parallel within a single computing channel, thus generating M partial sums simultaneously within one clock cycle. Each partial sum in this embodiment is a 32-bit register group. The partial sum register receives and temporarily stores all M partial sums in parallel with a width of M×32 bits, eliminating the need for time-division multiplexing or serialization. This ensures that the parallel computation results within the computing channel can be completely latched and output to the global addition tree in one clock cycle. This avoids data truncation or multiple transmission delays due to insufficient bit width and ensures that the physical bit width of the partial sum register is linearly related to the number of in-memory tiles in the computing channel, facilitating adjustments to the register bit width as needed for array configurations of different sizes. The computing unit can be a multiply-accumulate unit, which performs multiply-accumulate calculations on the corresponding connected memory bank. Specifically, during the clock cycle when the operation channel is activated, each memory bank synchronously reads out a row of weight data stored therein based on the row address broadcast by the shared row address bus. This weight data is directly sent to the weight input terminal of the corresponding multiply-accumulate unit via the point-to-point direct connection path between the memory bank and the multiply-accumulate unit. At the same time, the input activation data also arrives at the other input terminal of the multiply-accumulate unit. The multiply-accumulate unit performs a multiplication operation on the received weight data and activation data within a single clock cycle, and accumulates the multiplication result with the intermediate accumulation value temporarily stored inside the multiply-accumulate unit to generate the first partial sum corresponding to the multiply-accumulate unit, which is then output to the partial sum summary path of the memory chip. Each multiply-accumulator is directly connected to only one memory bank. After the weight data is read from the memory bank, it directly enters the multiply-accumulator to complete the multiplication and addition operation. The entire path does not pass through the multiplexer and the shared bus. Both multiplication and accumulation are completed locally inside the multiply-accumulator. There is no need to send the multiplication result to a separate addition unit for secondary processing. This compresses the consumption path of the weight data to the shortest possible length, thereby eliminating the combinational logic delay and crosstalk bottlenecks introduced by multi-level gating and bus arbitration in the traditional multi-memory bank shared bus architecture. This can support high-frequency timing convergence.

[0041] Reference Figure 3 N computing channels are arranged at intervals along the first direction, and M storage tiles in each computing channel are arranged at intervals along the second direction. The first and second directions are perpendicular to each other.

[0042] In this embodiment, N computing channels are arranged at intervals along a first direction, and M memory tiles within each computing channel are arranged at intervals along a second direction, with the first and second directions perpendicular to each other. A row address bus can extend along the first direction and connect to the memory tiles in each computing channel via branch lines of equal length, ensuring that the address signal is evenly distributed vertically and that each branch path has a consistent length, guaranteeing that the time difference in receiving the row address by each memory tile within the same computing channel is minimal. The row address bus can also extend along the second direction and be directly connected point-to-point to the input terminals of the computing units of each memory tile, minimizing the horizontal transmission path of the weight data and preventing crosstalk coupling between long-distance parallel lines, thus reducing the risk of wiring congestion.

[0043] Meanwhile, the computing channels are spaced apart along the first direction, reserving space for power isolation bands and clock buffer bands between adjacent computing channels. This effectively isolates power noise and clock jitter from different computing channels, preventing the computational activities of the activated computing channels from interfering with the deactivated ones during time-sharing activation. The memory tiles are spaced apart along the second direction, reserving wiring gaps between tiles to facilitate the passage of the row address bus and shared address bus through the gaps without occupying additional routing resources. The overall array's size ratio in the X and Y directions can be flexibly adapted to the aspect ratio of the photomask exposure window by adjusting the numbers N and M, so that the array's physical shape matches the photomask exposure area. Consequently, the physical layout of all memory cells can fall completely within the same photomask exposure field, without cross-photomask cutting, thus ensuring device characteristic consistency and maximizing lithography yield.

[0044] Reference Figure 3 In each computing channel, the spacing between any two adjacent memory tiles along the second direction is equal; the shared row address bus in the same computing channel extends along the first direction and is connected to each memory cell in the same computing channel through the row address bus.

[0045] In this embodiment, adjacent memory tiles in each computing channel are arranged at equal intervals along the second direction. The shared row address bus in the same computing channel extends along the first direction and is connected to the memory in each memory tile through the row address bus. The equal intervals of the memory tiles along the second direction make the positions of each tile in the same computing channel uniform in the horizontal direction, providing a physical path of equal length for the point-to-point direct connection of the row address bus along the second direction. The trace length between each computing unit and the corresponding memory tends to be consistent, which can ensure the timing synchronization of parallel reading and computing.

[0046] The shared row address bus extends along the first direction, forming branch connections with the row address buses of each memory tile within the computation channel in the vertical direction. The row address signal originates from the shared row address bus, passes through the branch lines of each tile's row address bus, and enters the P-chip memory within the tile. Because the tiles are evenly spaced along the second direction, and the branch points are uniformly distributed, the lengths of the branch lines can be kept highly consistent through equal-length wiring design. The time difference between the arrival times of the row address signal at different memory tiles within the same computation channel is controlled to a minimal range. This ensures that all memory chips within the channel synchronously read weighted data based on the same row address within the same clock cycle, eliminating the risk of data misalignment caused by inconsistent address signal arrival times.

[0047] Reference Figure 3 In the N operation channels, the spacing between any two adjacent operation channels along the first direction is equal, and a power isolation band and a clock buffer band are provided between any two adjacent operation channels.

[0048] In this embodiment, during the inference phase, the global controller activates only one computation channel per clock cycle via a gating unit, while the remaining computation channels are either clock-off or power-off. At this time, the memory and computation units of the computation channels operate synchronously, resulting in drastic instantaneous current changes. This can easily cause noise interference to adjacent power-off channels via power distribution network coupling with the substrate. The power isolation band reserved between adjacent computation channels physically blocks the lateral conduction path of power noise, suppresses the diffusion of dynamic current fluctuations from the activated channels to the power-off channels, and ensures that the memory in the power-off channels retains static data without loss and that the computation units maintain their initial state without flipping.

[0049] At the same time, the clock buffer provides physical isolation and buffered regeneration of the clock tree between adjacent operation channels. Figure 3 The clock buffer band and power isolation band are not shown in the diagram. The clock trees of different operation channels can be independently inserted into the buffer unit to avoid clock edge degradation and duty cycle distortion caused by sharing the same long clock line. When switching operation channels in a time-division multiplexing mode, each operation channel can be turned on or off individually. The clock tree of the off channel stops flipping and does not contribute to dynamic power consumption. The clock signal of the on operation channel is regenerated through the nearest buffer band and maintains a steep edge, ensuring that the clock arrival time deviation of all memory and computing units in the operation channel is controlled within a very small range, meeting the high-frequency timing convergence requirements.

[0050] The equally spaced computing channels further ensure that the power isolation band width and clock buffer band layout of each computing channel are uniform and consistent. The overall array has a regular and orderly size distribution along the first direction, which facilitates alignment with the boundary of the photomask exposure window and avoids exposure distortion or wiring congestion caused by uneven channel spacing in local areas. This helps to ensure the consistency of lithography yield and electrical performance after the array scale is expanded.

[0051] Reference Figure 4 Each storage tile contains P-cells arranged at intervals along a first direction, with equal spacing between adjacent cells along a second direction, and wiring gaps reserved between adjacent cells.

[0052] In this embodiment, the P-cells of memory within each memory tile are arranged at intervals along the first direction, and the spacing between any two adjacent memory cells along the second direction is equal. Wiring gaps are reserved between adjacent memory cells, which can form a regular two-dimensional grid-like array of memory cells inside the memory tile.

[0053] The arrangement of the aforementioned memory cells ensures that the point-to-point direct connection traces between each memory cell and its corresponding computing unit have physical paths of equal length. Since each memory cell is arranged along the first direction with uniform spacing, and its data output ports are distributed in an orderly manner along the second direction, the corresponding computing units can be arranged in the same direction, making the length of the metal connections between them tend to be consistent. This results in a minimal time difference between the arrival of data signals at the input of the computing unit, which is beneficial for ensuring the timing synchronization of parallel operation of each computing unit within the same memory tile.

[0054] Secondly, the spaced arrangement along the first direction allows for complete alignment of the Y-axis boundaries of the memory cells within the memory tiles. Combined with the equidistant spacing of adjacent memory cells along the second direction, the entire memory tile exhibits a regular rectangular outline in both the X and Y directions. This shape facilitates seamless splicing of multiple memory tiles arranged in a line along the second direction within the computing channel. It also ensures a regular and orderly physical boundary of the overall array, which is beneficial for adapting to the aspect ratio of the photomask exposure window. This guarantees that all memory cells are completely located within the same photomask exposure field, without cross-photomask cutting, thereby maintaining device characteristic consistency and maximizing lithography yield.

[0055] For example, Figure 5 This is a schematic diagram of the computation timing of a single-cycle computing unit provided in an embodiment of this application, involving a single round of N×M matrix-vector multiplication, combined with... Figures 1-5 The workflow of a memory-based computing device with N=3, M=8, and P=8 is described in detail. The three computing channels are Lane0, Lane1, and Lane3.

[0056] Period T0: Lane0 is activated, while Lane1 and Lane2 are in the clock-gated off state.

[0057] The global address bus outputs a 13-bit address, with the high 2 bits being 00 to select Lane 0, and the low 10 bits being the target weight row address.

[0058] Lane0 address path is opened, and row address is broadcast to all 64 memory chips in Lane0.

[0059] All storage units in Lane0 are simultaneously triggered to read rows, and 0.38ns later, the 8 sets of storage tiles simultaneously output 8×Mbit weight data.

[0060] The weight data is directly connected to the input section and register of the corresponding computing unit array via a point-to-point bus, and latching is completed on a 1ns clock edge.

[0061] The computational unit array completes the first segment of 1024-dimensional multiplication and accumulation operations, outputting a partial sum of Lane0 and P0 to the global addition tree latch.

[0062] Period T1: Lane1 is activated, while Lane0 and Lane2 are in the clock-gated off state.

[0063] The high 2 bits of the global address are switched to 01 to select Lane 1, while the low 10 bits of the row address remain unchanged.

[0064] The address path of Lane1 is opened, and the row address is broadcast to all 64 memory chips in Lane1.

[0065] All storage units in Lane 1 are simultaneously triggered to read rows, and 0.38ns later, the 8 sets of storage tiles simultaneously output 8×Mbit weight data.

[0066] The computational unit array completes the second 1024-dimensional multiplication and accumulation operation, outputs the partial sum of Lane1 and P1 to the global addition tree, and the global addition tree completes the pre-accumulation of P0+P1 and latches the result.

[0067] Period T2: Lane2 is activated, while Lane0 and Lane1 are in the clock-gated off state.

[0068] The high 2 bits of the global address are switched to 10 to select Lane 2, while the low 10 bits of the row address remain unchanged.

[0069] Lane2 address path is opened, and row address is broadcast to all 64 memory chips in Lane2.

[0070] All storage units in Lane 2 are simultaneously triggered to read rows, and 0.38ns later, the 8 sets of storage tiles simultaneously output 8×Mbit weight data.

[0071] In Lane 2, the computational unit array completes the third segment of the 1024-dimensional multiplication and accumulation operation, outputting part P2 to the global addition tree. The global addition tree completes the final accumulation of P0+P1+P2, outputting the complete M-dimensional 32-bit result to the output register. At this point, the valid flag is pulled high, indicating that a single round of matrix-vector multiplication operation has been completed.

[0072] In summary, the in-memory computing device provided in this embodiment can output Mbit weights in parallel in a single active lane per cycle, with a peak read bandwidth of 1TB / s; the peak bandwidth of N / 1024 lanes fully open reaches NGB / s, far exceeding that of traditional multi-bank SRAM macros of the same capacity. It can meet the requirements of an 8-channel M / 8 input MAC parallel computing array, eliminating the storage bandwidth bottleneck. By optimizing the overall array shape and wiring topology with the photomask window ratio, all memory cells fall within a single photomask window, with no cross-window exposure, low back-end wiring congestion, small timing deviation, balanced power network, and mass production yield and device consistency far superior to irregularly spliced ​​arrays. Abandoning the traditional multi-memory architecture, it adopts a point-to-point direct data path, and the address bus uses two-level hierarchical decoding to reduce load. With equal-length wiring controlling inter-chip spacing to <0.05ns, it fully meets the safety timing margin of a 1GHz main frequency, and the timing convergence difficulty is extremely low. Lane-level independent gating is naturally adapted to the segmented computing characteristics of MoE, enabling N-dimensional matrix multiplication to be completed in a time-sharing manner within a limited clock cycle, and idle lanes can be deeply shut down; at the same time, expert-level storage granularity supports subsequent expansion to multiple experts independently activated, further reducing power consumption in sparse inference scenarios. It solves the problems of insufficient capacity of a single storage bank and the lack of readily available large-width, large-capacity storage.

[0073] In addition, if there is a multi-expert extension, a 384-expert fully parallel MoE array can be realized by copying the SRAM macrocells of this embodiment. Each expert has 3 SRAM macrocells, and independent activation of experts is supported after routing. The in-memory computing device provided in this embodiment can also be seamlessly migrated to process nodes such as 16nm and 7nm. Only the standard SRAM IP of the corresponding node needs to be replaced, and the architecture logic remains unchanged. The output bit width can also be flexibly configured by increasing or decreasing the number of SRAM banks in the Tile, and it can adapt to various quantization formats such as FP4 and INT8 (8-bit Integer).

[0074] This embodiment also provides a memory, which includes the memory computing device described in the above embodiment.

[0075] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0076] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods and apparatus according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0077] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0078] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0079] The present invention has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A storage computing device, characterized in that, include: Global controller; N computing channels, each computing channel includes a gating unit, M memory tiles, a read enable signal line and a row address bus. Each memory tile includes P memory blocks and P computing units connected one-to-one with each of the P memory blocks. Each memory block includes a memory cell array. N, M and P are integers greater than or equal to 2. The gating unit's selection terminal is connected to the gating signal output terminal of the global controller; the address terminal of the P-chip memory in each memory tile is connected to the same row address bus; the row address buses of the M memory tiles are connected to a shared row address bus; the read enable terminal of the P-chip memory in each memory tile is connected to the same read enable signal line; the shared row address bus is connected to the row address output terminal of the global controller; and the read enable signal line is connected to the output terminal of the gating unit. The global controller is used to respond to the operation instructions and sequentially control the gating units of the N operation channels. For each operation channel selected, each memory cell in each memory tile of the operation channel is synchronously activated through the read enable signal line. The row address is broadcast to each memory cell in the operation channel through the shared row address bus and the row address bus to read the weight data in each memory cell in parallel and output it to the corresponding calculation unit for multiplication and accumulation calculation to obtain the first part sum corresponding to each memory cell. The first part sum corresponding to each memory cell in the same memory tile is accumulated to obtain the second part sum of each memory tile. A total of M second part sums are obtained as the operation result of the operation channel. The global controller is also used to accumulate the calculation results of the N calculation channels to obtain the target calculation result.

2. The storage and computing device according to claim 1, characterized in that, The global controller includes a global control module and a secondary address decoder. The global control module is connected to the input terminal of the secondary address decoder, the address output terminal of the secondary address decoder is connected to the shared row address bus, and the strobe signal output terminal of the secondary address decoder is connected to the gating unit of each of the arithmetic channels through a strobe signal line. The global control module is used to output a global address to the secondary address decoder in response to the operation instruction. The global address includes a strobe address and a target weight row address. The secondary address decoder is used to output a gating signal to the gating unit of each computing channel based on the gating address, so as to control the gating unit of the computing channel corresponding to the gating address to be in the open state, and simultaneously activate each memory bank in each computing tile in the computing channel and broadcast the target weight row address to each memory bank in the computing channel.

3. The storage and computing device according to claim 2, characterized in that, It also includes an output register, and the global controller also includes a global addition tree, the input of which is connected to the output of each of the operation channels, and the output of which is connected to the output register; The global addition tree is used to accumulate the operation results of the N operation channels to obtain the target operation result, and output the target operation result to the output register.

4. The storage and computing device according to claim 3, characterized in that, Each operation channel also includes a part and a register. In the same operation channel, the input of the part and the register is connected to the output of each computing unit in the operation channel, and the output of the part and the register is connected to the global addition tree. The aforementioned partial sum register is used to store the first partial sum output by each computing unit in the computing channel, and to accumulate the first partial sum output by P computing units in the same computing tile to obtain the second partial sum, and to output M second partial sums to the global addition tree.

5. The storage and computing device according to claim 4, characterized in that, The aforementioned part and register are a register group with a bit width of M×32 bits, and the calculation unit is a multiply-accumulate unit.

6. The storage and computing device according to claim 1, characterized in that, The N computing channels are arranged at intervals along a first direction, and the M storage tiles in each computing channel are arranged at intervals along a second direction, wherein the first direction and the second direction are perpendicular to each other.

7. The storage and computing device according to claim 6, characterized in that, The spacing between each pair of adjacent storage tiles in each of the aforementioned computing channels is equal along the second direction; The shared row address bus within the same computing channel extends along the first direction and is connected to each of the memory cells within the same computing channel via the row address bus.

8. The storage device according to claim 6, characterized in that, The spacing between any two adjacent computing channels in the N computing channels is equal along the first direction, and a power isolation band and a clock buffer band are provided between any two adjacent computing channels.

9. The storage and computing device according to claim 6, characterized in that, The P-cells in each of the computing tiles are arranged at intervals along the first direction, the spacing between any two adjacent cells along the second direction is equal, and a wiring gap is reserved between any two adjacent cells.

10. A memory, characterized in that, The storage device includes any one of claims 1-9.

Images