Adaptive high-bandwidth memory subsystem using dynamic channel management and operating method thereof

Abstract

An adaptive high-bandwidth memory subsystem using dynamic channel management and a method of operation thereof are disclosed. An adaptive high-bandwidth memory subsystem using dynamic channel management according to one embodiment of the present invention includes: a memory structured such that each of a plurality of data blocks is distributed and stored in a plurality of channels according to a pre-set criterion; and a controller that selectively activates the plurality of channels according to the size and pattern of the corresponding memory in response to a memory request among the plurality of channels.

Description

Technology Field

[0001] The disclosed embodiments relate to an adaptive high-bandwidth memory subsystem using dynamic channel management and a method of operation thereof. Background Technology

[0003] HBM 4 (high-bandwidth memory 4) can transfer up to 2KB of data in a single burst through a 2K-bit wide interface and a burst length of 8 (BL8). HBM 4 consists of 32 channels, each with a 64-bit wide data interface. Each channel operates independently and can directly access the allocated DRAM.

[0004] Meanwhile, artificial intelligence (AI) accelerators, such as graphics processing units (GPGPUs), may frequently require less than 2KB of data per memory request. This discrepancy can lead to wasted memory bandwidth and increased power consumption by accessing unused data.

[0005] While the aforementioned inefficiencies can be mitigated through the channel parallelism of HBM, managing 32 HBM channels independently may lead to relatively high design costs, signal integrity issues, and channel bias. On the other hand, all 32 HBM channels can be managed by a single control and address interface, but this may result in wasted memory bandwidth. Consequently, operators are required to design HBM subsystems that consider both performance and resource efficiency. Prior art literature

[0007] Republic of Korea Published Patent Application No. 10-2019-0097930 (Aug. 21, 2019) The problem to be solved

[0008] The disclosed embodiments aim to provide an adaptive high-bandwidth memory subsystem using dynamic channel management for effectively managing memory requests of an artificial intelligence accelerator and a method of operation thereof. means of solving the problem

[0010] An adaptive high-bandwidth memory subsystem using dynamic channel management according to one embodiment includes: a memory structured such that each of a plurality of data blocks is distributed and stored in a plurality of channels according to a preset standard; and a controller that selectively activates the plurality of channels according to the size and pattern of the corresponding memory in response to a memory request among the plurality of channels.

[0011] The above memory can manage the above multiple channels by dividing them into multiple preset groups.

[0012] The above controller may include a first interface for transmitting and receiving signals between a host device and the controller; and a second interface for transmitting and receiving signals between the controller and the memory.

[0013] The second interface comprises: a control and address interface that is connected to each of the plurality of channels to transmit control and address signals; and a CS interface that is connected to each of the preset channels among the plurality of channels to transmit chip select signals, wherein the number of channels connected to the CS interface may be determined according to the number of maximum dynamic channels.

[0014] The above controller may further include a processor that activates a corresponding group channel among the first group channel and the second group channel that corresponds to the chip selection signal, wherein the plurality of groups includes a first group channel and a second group channel.

[0015] The above memory can perform response processing for a memory request in a corresponding channel when its own chip selection signal, which is pre-matched among the plurality of channels, is activated.

[0016] The processor can transmit first and second memory requests occurring within a preset time difference corresponding to the first group channel and the second group channel to the memory.

[0017] The above memory can process the response to the first memory request and the second memory request transmitted from the processor in parallel.

[0018] A method of operation of an adaptive high-bandwidth memory subsystem according to another embodiment may include, in a method performed by a high-bandwidth memory service system comprising a memory and a controller, a step in which the memory stores each of a plurality of data blocks distributed across a plurality of channels according to a preset standard; and a step in which the controller selectively activates the plurality of channels according to the size and pattern of the corresponding memory in response to a memory request among the plurality of channels.

[0019] The operation method of the above adaptive high-bandwidth memory subsystem can manage the plurality of channels by dividing them into a plurality of preset groups during the distributed storage step.

[0020] The method of operation of the above adaptive high-bandwidth memory subsystem can, in the activation step, activate the group channel corresponding to the chip selection signal among the first group channel and the second group channel, wherein the plurality of groups includes a first group channel and a second group channel.

[0021] The method of operation of the above adaptive high-bandwidth memory subsystem may further include the step of performing a response processing for a memory request in a corresponding channel when the memory is activated by its own chip selection signal that is pre-matched among the plurality of channels.

[0022] The method of operation of the above adaptive high-bandwidth memory subsystem is such that, in the activation step, the controller can transmit first and second memory requests corresponding to the first group channel and the second group channel to the memory.

[0023] The method of operation of the above adaptive high-bandwidth memory subsystem can, in the step of performing the response processing, process the response processing for the first memory request and the second memory request transmitted from the controller in parallel.

[0024] In addition to this, a computer program stored on a computer-readable storage medium for executing a method for implementing the disclosed embodiments may be further provided.

[0025] In addition to this, a computer-readable recording medium may be further provided for recording a computer program for executing a method for implementing the disclosed embodiments. Effects of the invention

[0027] According to the disclosed embodiments, channels within memory can be efficiently managed through adaptive data-channel mapping and dynamic memory scheduling techniques, and as a result, it is expected that relatively high inference performance can be provided compared to a general HBM subsystem. Brief explanation of the drawing

[0029] FIG. 1 is a block diagram illustrating an adaptive high-bandwidth memory subsystem according to one embodiment. FIGS. 2 to 7 are exemplary diagrams for explaining a control method of an adaptive high-bandwidth memory subsystem according to one embodiment. FIG. 8 is a flowchart illustrating a control method for an adaptive high-bandwidth memory subsystem according to one embodiment. Specific details for implementing the invention

[0030] Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, apparatuses, and / or systems described herein. However, this is merely illustrative and the present invention is not limited thereto.

[0031] In describing the embodiments of the present invention, detailed descriptions of known technologies related to the present invention are omitted if it is determined that such detailed descriptions may unnecessarily obscure the essence of the present invention. Furthermore, the terms described below are defined in consideration of their functions within the present invention, and these may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification. Terms used in the detailed description are intended merely to describe the embodiments of the present invention and should not be limiting in any way. Unless explicitly stated otherwise, expressions in the singular form include the meaning of the plural form. In this description, expressions such as "include" or "comprise" are intended to refer to certain characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof, and should not be interpreted to exclude the existence or possibility of one or more other characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof other than those described.

[0032] FIG. 1 is a block diagram illustrating an adaptive high-bandwidth memory subsystem according to one embodiment.

[0033] Hereinafter, a method for controlling an adaptive high-bandwidth memory subsystem according to one embodiment will be described with reference to FIGS. 2 to 7, which are exemplary diagrams for explaining the method.

[0034] The adaptive data-channel mapping technology implemented in the adaptive high-bandwidth memory subsystem (hereinafter referred to as the high-bandwidth memory subsystem) (10) according to one embodiment can optimize memory bandwidth utilization by adaptively grouping or dividing HBM channel operations according to the granularity of memory requests. The memory scheduling technology implemented in the processor (220) within the controller (200) described later can efficiently manage memory requests by allocating service channels according to the granularity of memory requests. This memory approach can minimize design complexity while all channels operate in parallel. A detailed description thereof will be provided later.

[0035] Referring to FIG. 1, the high-bandwidth memory subsystem (10) includes memory (100) and a controller (200).

[0036] FIG. 2 may be a diagram showing an interconnection structure between an AI accelerator and an HBM4 according to one embodiment. Referring to FIG. 2, the memory (100) is an HBM4 (high-bandwidth memory 4) including a DRAM (dynamic random access memory), and the controller (200) may refer to a memory controller within a memory scheduler implemented inside the AI ​​accelerator. That is, the high-bandwidth memory subsystem (10) may be a configuration including an AI accelerator and an HBM4.

[0037] The memory (100) may be configured such that each of the multiple data blocks is distributed and stored according to a pre-set standard in multiple channels.

[0038] That is, when the memory (100) receives a write request for data transmitted from a host device, it can store the data distributed across multiple channels according to a preset standard. The preset standard may refer to a standard for storing data such that data blocks are distributed within memory channels, rather than storing data such that data blocks are mapped sequentially to memory channels. At this time, the method of distributed mapping may be preset by an operator.

[0039] Specifically, the memory (100) may be configured to manage multiple channels by dividing them into multiple preset groups. In this case, each of the multiple groups may be a unit of channel group that is activated by the same chip selection signal.

[0040] The memory (100) can perform response processing for a memory request in a corresponding channel when its own chip selection signal, which is pre-matched among a plurality of channels, is activated.

[0041] The memory (100) can process the response to the first memory request and the second memory request transmitted from the processor (220) described later in parallel.

[0042] The memory (100) can operate based on memory control signals, including memory requests. The memory control signals may include a row access strobe (RAS) signal and a column access strobe (CAS) signal. Additionally, the memory control signals may include signals for controlling the operation of the memory.

[0043] The above RAS signal may be a signal for accessing a row position among the memory address values. Additionally, the RAS signal may be a signal for activating the memory. That is, the RAS signal may be the chip select signal of the present disclosure.

[0044] The above CAS signal may include a CAS RD signal for reading a data word and a CAS WR signal for writing a data word.

[0045] The CAS signal may be a signal for accessing a column position among the memory address values. Additionally, the CAS signal may be a signal for outputting data from an active memory cell. The processor (220) can write or read data to or from a data word of the memory (100) through the CAS signal.

[0046] The controller (200) may include a first interface (210), a processor (220), a buffer section (230), and a second interface (240). The components illustrated in FIG. 1 are not essential for implementing the high-bandwidth memory subsystem (10) according to the present disclosure, so the high-bandwidth memory subsystem (10) described herein may have more or fewer components than those listed above.

[0047] The controller (200) may be configured to selectively activate the plurality of channels according to the size and pattern of the corresponding memory in response to a memory request among the plurality of channels.

[0048] Specifically, it can be assumed that a controller (memory controller) (200) operates an HBM4 having m channels and 2048 bytes of data that can be read at once with n channels. In this case, the controller (200) can access 2048 / n bytes of data at once from each channel. Here, the first CS to be activated can be obtained from the memory address given in FIG. 4. For example, in FIG. 4, it may be the case where n=2. If the memory data required at once is v (<=2048), the total number of activated channels is It can be. Therefore, if CS is j from the memory address, then CS is from j to j+ It can be activated up to -1.

[0049] For example, it can be assumed that a controller (memory controller) (200) operates 8 channels of an HBM4 having 32 channels and 2048 bytes of data that can be read at once. In this case, the controller (200) can access 256 bytes of data at once from each channel. If the memory data required at once is 1024, the total number of active channels can be 4. Accordingly, if the starting CS from the memory address is 1, CS can be activated from 1 to 4.

[0050] Referring to FIG. 2, the controller (200) described above may refer to a memory controller within a memory scheduler implemented inside an AI accelerator. That is, the present disclosure may be intended to describe the operation of an adaptive high-bandwidth memory subsystem using dynamic channel management to effectively manage memory requests of an AI accelerator.

[0051] The first interface (210) may be configured for transmitting and receiving signals between a host device (not shown) and a controller (200). For example, the host device may be configured to access memory (100) for storing data, such as a desktop computer, laptop computer, mobile phone, MP3 player, game console, TV, in-vehicle infotainment system, etc.

[0052] The first interface (210) can interface between the host device and the controller (200) in accordance with the protocol of the host device. For example, the first interface (210) can communicate with the host device through any one of the following protocols: USB (universal serial bus), UFS (universal flash storage), MMC (multimedia card), PATA (parallel advanced technology attachment), SATA (serial advanced technology attachment), SCSI (small computer system interface), SAS (serial attached SCSI), PCI (peripheral component interconnection), and PCI-e (PCI express).

[0053] The processor (220) may be configured such that a plurality of groups include a first group channel and a second group channel, and activates the corresponding group channel among the first group channel and the second group channel that corresponds to a chip selection signal. In this case, the first group channel may refer to a group of channels activated by the same single chip selection signal (e.g., CS0). In this manner, the second group channel may refer to a group of channels activated by the same single chip selection signal (e.g., CS1).

[0054] The processor (220) can transmit first and second memory requests that occur within a preset time difference corresponding to the first group channel and the second group channel to the memory (100).

[0055] Referring to FIG. 7, the processor (220) can schedule memory requests to support dynamic channel management.

[0056] Specifically, the processor (220) may process memory requests in the order in which they are received, while applying potential conflicts and constraints. For each available memory channel, the processor (220) may check whether a memory request in the input queue satisfies two conditions. The conditions may be that the request must not be a read-write or write-read conversion and must not cause a row buffer conflict with the current bank state of the memory (100). If the memory request satisfies the above conditions, the processor (220) may immediately select the memory request and remove the current channel from the list of available channels to prevent duplicate memory operations.

[0057] If there are no memory requests satisfying the conditions, the processor (220) may select the first memory request according to the order of reception of memory requests to ensure that no memory requests remain unserviced for a long time. Then, the algorithm may move to the next available channel. The above-described approach prioritizes collision-free, low-latency operations while maximizing memory throughput by utilizing channel parallelism.

[0058] The buffer section (230) can store request signals, including memory requests received through the first interface (210). In FIG. 1, the buffer section (230) is shown as one, but is not limited thereto. If the buffer section (230) is implemented as a plurality, each buffer section (230) can store different types of request signals.

[0059] The buffer section (230) may store data to be written to the memory (100). At this time, the data may be stored separately from request signals, including memory requests, but is not limited thereto.

[0060] The second interface (240) may be configured for transmitting and receiving signals between the controller (200) and the memory (100).

[0061] The second interface (240) can control the memory (100) according to the control of the processor (220). The second interface (240) can transmit memory control signals generated by the processor (220) to the memory (100).

[0062] Referring to FIG. 2, the second interface (240) may include a CS interface (241) and a control and address interface (243). Additionally, the second interface (240) may include a data interface (245).

[0063] The CS interface (241) may be configured to transmit a chip select signal by being connected to each of the preset channels among the multiple channels.

[0064] The number of channels connected to the CS interface (241) can be determined according to the maximum number of dynamic channels. For example, if the control and address interface (243) is connected to 32 channels of the memory (HBM4) (100), each CS interface (241) can be connected to 16 channels of the memory (HBM4) (100). Additionally, in the case of dynamic 8-channel management, each CS interface (241) can be connected to 4 channels of the memory (HBM4) (100).

[0065] The control and address interface (243) may be configured to be connected to each of the multiple channels to transmit control and address signals.

[0066] Specifically, referring to FIG. 2, a high-bandwidth memory subsystem (10) according to one embodiment may provide dynamic 2-channel management through a single control and address interface (243) having two chip select (CS) interfaces (241). Not limited thereto, the high-bandwidth memory subsystem (10) may be extended to dynamic 8-channel management. As illustrated in FIG. 2, the control and address interfaces (243) may be interconnected to all 32 channels of the memory (HBM4) (100), and each CS interface (241) may be interconnected to 16 channels of the memory (HBM4) (100). Accordingly, control and address signals are transmitted to all channels of the memory (HBM4) (100), but each channel may respond only when the corresponding CS signal is activated. Under this interconnection architecture, data may be mapped to BL8 as shown in FIG. 3. Figure 3 above may be an illustrative diagram for explaining the mapping between data and channels within an HBM4 according to one embodiment.

[0067] As a result, when the CS signal (CS 0) is activated for 16 channels of the first group channel, data blocks 0-127, 256-383, etc. can be accessed. If the CS signal (CS 1) is activated for 16 channels of the second group channel, data blocks 128-255, 384-511, etc. can be accessed.

[0068] FIG. 4 may be a diagram showing data-memory address mapping according to dynamic two-channel management according to one embodiment.

[0069] FIG. 4 may represent the corresponding data-memory address mapping for dynamic 2-channel management. Unlike a typical configuration, the channel address of this embodiment may be selected from a given address bit 7. If a configuration supporting dynamic 4-channel and 8-channel management is in place, the channel address may be selected from a given address bits 6-7 and bits 5-7, respectively.

[0070] When the high-bandwidth memory subsystem (10) of the present embodiment receives a memory request in 2KB units for data blocks 0-255, it can activate and read data from all channels where both CS 0 and CS 1 are activated. That is, the memory (100) can read data under the control of the processor (220). Channels 0-15 of the HBM4 can each provide data blocks 0-127, and channels 16-31 can each provide data blocks 128-143. The size of each data block can be 64 bits. Referring to FIG. 5, the memory (100) can repeat the above-described operation for BL8 to transmit a total of 2KB of data per CAS command to the controller (100) of the AI ​​accelerator.

[0071] In addition, the high-bandwidth memory subsystem (10) of the present embodiment can sufficiently perform 1KB unit memory requests for data blocks 0-127.

[0072] Referring to FIG. 6, the controller (200) uses channels 0-15 of the memory (HBM4) (100) through CS 0 and can sequentially access data blocks 0-15, 16-31, 32-47, 48-63, 64-79, 80-95, 96-111, and 112-127. Accordingly, the data-channel mapping technique proposed in the high-bandwidth memory subsystem (10) of the present embodiment may be suitable for both small-scale and large-scale unit memory requests.

[0073] In addition, the memory (HBM4) (100) can maximize parallelism by using channels 0-15 to transmit 1KB of data while simultaneously processing other 1KB unit memory requests using channels 16-31.

[0074] The data interface (245) may be configured to be connected to each of the multiple channels to transmit data request signals.

[0075] The above-described embodiment may provide a new high-bandwidth memory subsystem that applies a dynamic channel management mechanism to optimize the design cost of HBM channels and efficiently support AI workloads such as Transformers. This mechanism may enable adaptive resource utilization by adjusting the grouping or splitting of high-bandwidth memory channel operations according to the granularity of memory requests. Additionally, for relatively small memory requests, this embodiment may use only some channels to save memory bandwidth and energy. On the other hand, for large granular memory requests, all channels may be used to maximize memory throughput. Furthermore, this embodiment may access data sequentially without wasting memory bandwidth regardless of the number of channels used for each memory request. For example, the controller (200) of the AI ​​accelerator may use all channels before transpose and permute operations to sequentially access 2KB of data. However, after performing the operations, the controller (200) of the AI ​​accelerator may selectively activate channels to maintain efficiency as memory requests become smaller and granular. That is, the high-bandwidth memory subsystem (10) according to the present embodiment can minimize collisions that may occur during data access and maintain throughput by optimizing the access order through data rearrangement.

[0077] FIG. 8 is a flowchart illustrating a control method for an adaptive high-bandwidth memory subsystem according to one embodiment. The method illustrated in FIG. 8 may be performed, for example, by the aforementioned high-bandwidth memory subsystem (10). Although the method is described in the illustrated flowchart in a plurality of steps, at least some of the steps may be performed in a different order, combined with other steps and performed together, omitted, divided into detailed steps, or performed with one or more steps not illustrated added.

[0078] The operation method of the high-bandwidth memory subsystem described below may reflect all the operations of the high-bandwidth memory subsystem (10) described with reference to FIGS. 1 to 7 above, but for the convenience of explanation, redundant detailed descriptions will be omitted.

[0079] In step 1100, the memory (100) can store each of the multiple data blocks distributed across multiple channels according to a preset standard. That is, when the memory (100) receives a write request for data transmitted from a host device, it can store the data distributed across multiple channels according to a preset standard.

[0080] When the memory (100) stores multiple data blocks in a distributed manner, it can manage multiple channels by dividing them into multiple preset groups.

[0081] In step 1200, the controller (200) can selectively activate multiple channels according to the size and pattern of the memory requested among the multiple channels.

[0082] The controller (200) can activate the group channel corresponding to the chip selection signal among the first group channel and the second group channel, wherein a plurality of groups include a first group channel and a second group channel.

[0083] The controller (200) can transmit first and second memory requests that occur in correspondence with the first group channel and the second group channel to the memory (100).

[0084] In step 1300, the memory (100) can perform response processing for a memory request in a corresponding channel when its pre-matched chip selection signal among a plurality of channels is activated.

[0085] The memory (100) can process the response to the first memory request and the second memory request transmitted from the controller (200) in parallel.

[0086] Although representative embodiments of the present invention have been described in detail above, those skilled in the art will understand that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined by the claims set forth below as well as equivalents thereof. Explanation of the symbols

[0087] 10: High-bandwidth memory subsystem 100: Memory 200: Controller 210: First Interface 220: Processor 230: Buffer section 240: Second Interface

Claims

Claim 1 An adaptive high-bandwidth memory subsystem using dynamic channel management, comprising: a memory structure in which each of a plurality of data blocks is distributed and stored according to a preset standard in a plurality of channels; and a controller that selectively activates the plurality of channels according to the size and pattern of the corresponding memory in response to a memory request among the plurality of channels. Claim 2 The adaptive high-bandwidth memory subsystem using dynamic channel management according to claim 1, wherein the memory manages the plurality of channels by dividing them into a plurality of preset groups. Claim 3 An adaptive high-bandwidth memory subsystem using dynamic channel management according to claim 1, wherein the controller comprises: a first interface for transmitting and receiving signals between a host device and the controller; and a second interface for transmitting and receiving signals between the controller and the memory. Claim 4 The adaptive high-bandwidth memory subsystem using dynamic channel management according to claim 3, wherein the second interface comprises: a control and address interface that is connected to each of the plurality of channels and transmits control and address signals; and a CS interface that is connected to each of the preset portion of the plurality of channels and transmits a chip select signal, wherein the number of channels connected to the CS interface is determined according to the number of maximum dynamic channels. Claim 5 An adaptive high-bandwidth memory subsystem using dynamic channel management according to claim 4, wherein the controller further comprises a processor that activates a corresponding group channel among the first group channel and the second group channel corresponding to the chip selection signal, wherein the plurality of groups includes a first group channel and a second group channel. Claim 6 The adaptive high-bandwidth memory subsystem using dynamic channel management according to claim 5, wherein the memory performs response processing for a memory request in a corresponding channel when a pre-matched chip selection signal among the plurality of channels is activated. Claim 7 The adaptive high-bandwidth memory subsystem using dynamic channel management according to claim 6, wherein the processor transmits first and second memory requests occurring within a preset time difference corresponding to the first group channel and the second group channel to the memory. Claim 8 In claim 7, the memory is an adaptive high-bandwidth memory subsystem using dynamic channel management, which processes the response to the first memory request and the second memory request transmitted from the processor in parallel. Claim 9 A method of operation of an adaptive high-bandwidth memory subsystem, wherein the method is performed by a high-bandwidth memory service system including a memory and a controller, the method comprising: the step of the memory distributing and storing each of a plurality of data blocks in a plurality of channels according to a preset standard; and the step of the controller selectively activating the plurality of channels according to the size and pattern of the corresponding memory in response to a memory request among the plurality of channels. Claim 10 A method of operation of an adaptive high-bandwidth memory subsystem according to claim 9, wherein, in the distributed storage step, the plurality of channels are divided into a plurality of preset groups and managed. Claim 11 A method of operation of an adaptive high-bandwidth memory subsystem according to claim 10, wherein, in the activating step, the plurality of groups includes a first group channel and a second group channel, and the corresponding group channel among the first group channel and the second group channel is activated to correspond to a chip selection signal. Claim 12 A method of operation of an adaptive high-bandwidth memory subsystem according to claim 11, further comprising the step of performing a response processing for a memory request in a corresponding channel when the memory is activated by its own chip selection signal that is pre-matched among the plurality of channels. Claim 13 A method of operation of an adaptive high-bandwidth memory subsystem according to claim 12, wherein, in the activating step, the controller transmits first and second memory requests occurring corresponding to the first group channel and the second group channel to the memory. Claim 14 A method of operation of an adaptive high-bandwidth memory subsystem according to claim 13, wherein, in the step of performing the response processing, the memory processes the response processing for the first memory request and the second memory request transmitted from the controller in parallel.

Images