Memory apparatus and method for improved write-read turnaround times

Abstract

A memory includes interface circuitry to receive at least one write command. Multiple banks are organized into multiple bank groups and are operative in compliance with respective bank group timings. Each of the multiple banks include an array of storage cells. Multiple write buffers temporarily store write data associated with the at least one write command. Write buffer selection circuitry, based on the bank group timings, retires the temporarily stored write data from a selected one of the multiple write buffers and directs the retired write data to a given one of the arrays of storage cells.

Description

MEMORY APPARATUS AND METHOD FOR IMPROVED WRITE-READ TURNAROUND TIMESTECHNICAL FIELD

[0001] The disclosure herein relates to memory systems, memory controllers, memory devices, and associated methods.BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Embodiments of the disclosure are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

[0003] FIG. 1 illustrates one embodiment of a memory system that employs a memory controller, and at least one memory device.

[0004] FIG. 2 illustrates high-level steps employed in operating the memory system of FIG. 1.

[0005] FIG. 3 illustrates one specific embodiment of a write buffer circuit for use with the memory system of FIG. 1.

[0006] FIG. 4 illustrates a timing chart that shows a write-read turnaround bubble associated with memory access operations for a memory system without enabled write buffers.

[0007] FIG. 5 illustrates a timing chart that shows a reduced write-read turnaround bubble associated with memory access operations for a memory system having multiple enabled write buffers.

[0008] FIG. 6 illustrates a timing chart that shows a write-read turnaround bubble associated with memory access operations for a memory system having multiple write buffers that retire buffered data in order, without regard to bank or bank group conflicts.

[0009] FIG. 7 illustrates a timing chart that shows a reduced write-read turnaround bubble associated with memory access operations for a memory system having multiple write buffers that retire buffered data in an order that minimizes the impact of bank and bank group conflicts.

[0010] FIG. 8 illustrates a command truth table for a DDR protocol showing availability for an explicit retire command.

[0011] FIG. 9A illustrates one embodiment of a DDR DIMM that includes a registered clock driver (RCD) chip.

[0012] FIG. 9B illustrates a shared CA bus topology for the DDR DIMM of FIG. 9A.

[0013] FIG. 9C illustrates a timing chart for commands generated and passed by theRCD chip of FIG. 9 A corresponding to the shared CA bus topology of FIG. 9B.

[0014] FIG. 10A illustrates a further embodiment of a DDR DIMM that includes a registered clock driver (RCD) chip.

[0015] FIG. 10B illustrates a dedicated per-rank CA bus topology for the DDR DIMM of FIG. 10 A.

[0016] FIG. 10C illustrates a timing chart for commands generated and passed by the RCD chip of FIG. 10A corresponding to the dedicated per-rank CA bus topology of FIG. 10B.

[0017] FIG. 11 illustrates multiple DDR DIMMs with respective RCD chips in communication with a memory controller.

[0018] FIG. 12 illustrates a memory system that employs a DDR DIMM that includes an RCD chip and multiple data buffer chips.

[0019] FIG. 13 illustrates further detail for one embodiment of write buffer circuitry disposed in the RCD chip and data buffer chips of FIG. 12.

[0020] FIG. 14 illustrates a Broadcast Communications (BCOM) bus truth table for the BCOM circuitry of FIG. 13.

[0021] FIG. 15A illustrates a timing chart, from the perspective of a memory controller, of a reduced write-read turnaround bubble accomplished through use of write buffer circuitry in at least one data buffer chip.

[0022] FIG. 15B illustrates a timing chart, from the perspective of a data buffer chip, of a write-read turnaround bubble with disabled write buffer circuitry in at least one data buffer.

[0023] FIG. 15C illustrates a timing chart, from the perspective of a data buffer chip, of a reduced write-read turnaround bubble with enabled write buffer circuitry in at least one data buffer.

[0024] FIG. 16 illustrates one embodiment of a three-dimensional (3D) DRAM architecture with various potential locations for the write buffer circuitry described herein.

[0025] FIG. 17 illustrates one embodiment of a high -bandwidth memory (HBM) architecture with various potential locations for the write buffer circuitry described herein.DETAILED DESCRIPTION

[0026] Memory devices, modules, controllers, systems and associated methods are disclosed. In one embodiment, a memory is disclosed. The memory includes interface circuitry to receive at least one write command. Multiple banks are organized into multiple bank groups and are operative in compliance with respective bank group timings. Each of the multiple banks include an array of storage cells. Multiple write buffers temporarily store write data associated with the at least one write command. Write buffer selection circuitry, based on the bank group timings, retires the temporarily stored write data from a selected one of the multiple write buffers and directs the retired write data to a given one of the arrays of storage cells. For some embodiments, the write buffer selection circuitry is responsive to explicit and / or implicit commands to retire the write data. In some embodiments, the write buffer circuitry is included in a memory device of the memory. Other embodiments may include the write buffer circuitry off-chip through use of one or more buffer chips. Byemploying the write buffer circuitry to temporarily store write data, and to retire the stored write data based on bank group timings, reductions in write-to-read turnaround timings may be realized, resulting in reduced latency parameters.

[0027] Referring now to FIG. 1, a memory system, generally designated 100, is shown that includes a memory controller 102 coupled to one or more memory devices 104 via signaling media 106. For one embodiment, the memory controller 102 is a dynamic random access memory (DRAM) controller, with the memory device 104 realized as a DRAM memory device. In some embodiments, the memory controller 102 and the memory device 104 may be embodied as integrated circuits, or chips. Other embodiments may employ the memory controller 102 as a circuit in a host central processing unit (CPU) (not shown). Specific embodiments for the DRAM memory controller 102 and memory 104 may be compliant with various DRAM standards, including double data rate (DDR) variants, low power (LPDDR) versions, high bandwidth (HBM), graphics (GDDR) types, and multiplexed DDR (MDDR) types. Other embodiments may include multi -chip modules that, for example, employ stacked memory die, or stacked packages. Additional embodiments may stack memory die and logic die together in a common package, or in separate packages stacked upon each other. Yet other embodiments may employ multiple memory devices on a substrate (not shown) in a memory module configuration for high-capacity applications.

[0028] With continued reference to FIG. 1, one embodiment of the memory controller 102 includes write queue circuitry 108 to temporarily store a copy of command information associated with each write command, such as a bank group address, bank address, autoprecharge information, and mask information. For some embodiments, the write queue circuitry 108 of the memory controller 102 also temporarily stores data associated with each write command. The memory controller 102 also includes read queue circuitry 110 that temporarily stores information associated with each read command issued to the memorydevice 104 in a similar manner to that of the write queue circuitry 108. For some embodiments, the memory controller 102 includes prioritization circuitry 112 to reorder one or more operations in the write queue circuitry 108 and / or read queue circuitry 110 and thus prioritize certain write and / or read operations over others based on bank group timings. As explained more fully below, prioritizing certain operations over others from the memory controller side may reduce latency associated with write-to-read timing bubbles.

[0029] Further referring to FIG. 1, for one embodiment, each memory device 104 includes memory interface circuitry 114 for handling communications between the memory controller 102 and memory core circuitry 116 of the memory device 104. For one embodiment, the memory core circuitry 116 is organized in a bank group architecture having multiple bank groups 118 and 120 that are separately addressable. Each bank group includes multiple banks 122, which are addressable via respective bank addresses. Each bank 122 includes an array of storage cells 124, with individual cells being selectable for access through unique row and column addresses.

[0030] With continued reference to FIG. 1, in an effort to reduce a number of clock cycles before read data associated with a read operation may begin transfer along a data bus following an immediately preceding transfer of write data along the data bus, often referred to as a write-to-read turnaround time, the memory system 100 employs write buffer circuitry 126. As described below, and shown in FIG. 1, the write buffer circuitry 126 may be disposed in one of a variety of locations on-chip or off-chip. For one embodiment, the write buffer circuitry 126 includes first storage to store control information corresponding to a given write operation such as the bank group address, bank address, autoprecharge information, and mask information. The write buffer circuitry 126 includes second storage to store the write data associated with each write operation. The write buffer circuitry 126 generally responds to either explicit command information or implicit command information(including predetermined conditions) to retire the temporarily stored control information and write data for transfer to the addressed portion of the memory core circuitry 116 of the memory device 104. Embodiments described below provide further detail regarding the types of commands and conditions that may be employed to control the write buffer circuitry 126. In some embodiments, register circuitry (not shown) may be provided on or off of the memory device 104 to provide register programmability for enabling / disabling specific circuitry associated with expected forms of commands (explicit / implicit) and conditions, and / or other various configurability options described herein.

[0031] Further referring to FIG. 1, certain embodiments may employ multiple write buffers in the write buffer circuitry 126, with one or more of the multiple write buffers corresponding to the multiple bank groups 118 and 120 of the memory device 104. Depending on the application, the write buffer circuitry 126 may be disposed on-chip in the memory device 104 itself, such as in the memory interface circuitry 114 at 128, or in the memory core circuitry 116 associated with a first bank group 118 at 130, or a second bank group, at 132, or in each bank 122 at 134 or 136. Additional detail for on-chip embodiments of the write buffer circuitry are described below with respect to FIGs. 3 through 8. For some embodiments, the write buffer circuitry 126 may be disposed off-chip, such as in a buffer chip or chipset, at 138. Further detail for off-chip write buffer circuitry embodiments are described below with respect to FIGs. 9A through 15C.

[0032] FIG. 2 illustrates one embodiment of a high-level flowchart for a method of operation involving the memory system 100 of FIG. 1 to reduce the idle time of a data interface that manifests itself when a read operation follows a write operation. The idle time is often referred to as a write-read turnaround bubble, and it has an impact on memory system performance. At 202, the memory controller 102 issues current write command information for a current write operation to the memory device 104. For some embodiments, the writecommand information includes write buffer selection information for selecting one from a plurality of write buffers in the write buffer circuitry 126. At 204, the control information and the write data corresponding to the write command information is directed to the write buffer circuitry 126. At 206, in response to either an explicit command, at 208, or to an implicit command, at 210, previously buffered write data and associated control information stored in the selected write buffer is retired from the selected write buffer to the memory core circuitry 116 of the memory device 104 based on timings associated with the multiple bank groups 118 and 120. For one embodiment, the write buffer selection may be implicitly determined through agreed-upon rules between the memory controller 102 and the memory device 104, at 211. In one specific example, such as where each write buffer has its own unique identification number, the write buffer selected might be an empty buffer with a lowest identification number. If no empty buffers exist, then the selected write buffer is the least- recently accessed write buffer. Implicit determination of the write buffer selection reduces the amount of information that needs to be transmitted between the memory controller 102 and the write buffers, reducing the bandwidth consumed for commands.

[0033] With continued reference to FIG. 2, since the prior write data was already buffered proximate the memory core circuitry 116, it arrives at the memory core circuitry 116 much faster than if propagating from the memory controller 102. This reduction in latency allows the memory controller 102 to issue a subsequent read command to the memory device 104 sooner, at 212, resulting in read data being transferred along data paths of the memory interface circuitry 114 within a shorter idle interval (bubble) following transfer of the write data to the memory core circuitry 116.

[0034] FIG. 3 illustrates one specific embodiment of the write buffer circuitry 126 of FIG. 1. For this particular embodiment, the write buffer circuitry 126 is disposed on the memory device 104 as on-chip circuitry, with respective command / address (CA) and data(DQ) paths 302 and 304. As noted above with respect to FIG. 1, the write buffer circuitry 126 may be disposed at various on-chip at locations such as the memory interface 114, the bank group circuitry 118 or the bank circuitry 122.

[0035] Further referring to FIG. 3, the write buffer CA path 302 includes control circuitry 306 that receives incoming CA information, such as chip identification information, bank group address information, bank address information and so forth. Based on the received CA information, the control circuitry 306 generates a read / write select control signal that is fed along a first control path 307 that generally activates / deactivates a first set of multiple write buffers 308 in the write buffer CA path 302 and a second set of multiple write buffers 310 in the write buffer data path 304. While not shown, when the CA information involves a read command, the CA information corresponding to the read command is fed straight to the memory core circuitry 116 without entering the write buffer CA path.

[0036] With continued reference to FIG. 3, when the CA information involves a write operation, the control circuitry 306 generates a write buffer input select signal, fed along control path 312, and a write buffer output select signal, fed along control path 314. In response to the write buffer input select signal, a first input multiplexer 316 in the write buffer CA path 302 steers the received CA information into a selected (based on the value of the write buffer input select signal) one of the write buffers from the first set of write buffers 308. A second input multiplexer 318 in the write buffer data path 304 steers the received write data into a selected (based on the value of the write buffer input select signal) one of the write buffers from the second set of write buffers 310. In response to the write buffer output select signal, a first output multiplexer 320 in the write buffer CA path 302 retires previously buffered CA information from a selected (based on the value of the write buffer output select signal) one of the write buffers of the first set of write buffers 308. A second output multiplexer 322 in the write buffer data path 304 retires previously buffered write data from aselected (based on the value of the write buffer output select signal) one of the write buffers from the second set of write buffers 310.

[0037] Further referring to FIG. 3, the write buffer data path 304 includes a read data path 324 that bypasses the second set of write buffers 310, and includes a serializer 326 to serialize read data from the core for transfer along the data interface DQ. A write data path 328 includes the second set of write buffers 310 and a deserializer 330 to deserialize the write data as it is received from the data interface DQ. Respective first, second and third selectors 332, 334 and 336 respond to the read / write select signal generated by the control circuitry 306 to activate certain read or write circuits, such as the read bypass path 324 or the write data path 328 to accomplish a given data transfer operation.

[0038] Referring now to FIG. 4, a timing chart is presented with detailed timing intervals shown that are representative of a DDR-6400 memory device, without write buffer circuitry, to aid in explaining the context of how a typical write-read bubble arises. The chart is shown from the perspective of a DDR memory device, with clock cycles shown at 402, commands shown at 404, data transfers along a data interface shown at 406, and memory core activity shown at 408. The chart generally shows timings associated with a first read command RD, at 410, followed by a first write command WR, at 412. The number of clock cycles between issuance of the first read command at 410 and the first write command at 412 is controlled by the memory controller 102 and is constrained by a read-to-write timing interval RTW, shown at 414. A similar timing constraint, a write-to-read WTR timing interval shown at 416, is controlled by the memory controller 102 to appropriately place a second read command RD, at 418, that takes into account various delays associated with the preceding write operation.

[0039] Further referring to FIG. 4, in response to the first read command at 410, the memory core 116 performs a variety of read-related sub-operations that ties-up the memorycore 116 for an interval such as at 420. At the expiration of a CAS Latency (CL) timing constraint, shown at 422, and defined as a delay in clock cycles between the internal first read command RD at 410 and the availability of the first bit of output read data, at 423. Once the read data begins transfer along the data interface, at 423, the memory core 116, at 424, is immediately available for the next memory access operation - here, the first write operation. However, write data, at 425, associated with the write command WR at 412, is constrained to be applied to the data interface by a CAS Write Latency (CWL) timing interval, at 426, and defined by memory manufacturer standards as a delay, in clock cycles, between the internal write command WR at 412 and the availability of the first bit of input write data, at 425. After the write data has propagated along the data interface, it is fed to the memory core 116, at 428. Although the memory core 116 was available to receive the write data at a time coinciding with 424, actual write data was not fed to the memory core 116 until time 428, resulting in an idle delay interval 430 within the memory core 116 of several clock cycles.

[0040] With continued reference to FIG. 4, the idle delay interval 430 has a significant impact on the timing for future operations. As noted above, the second read RD command, at 418, is constrained from being issued by the memory controller 102 until the WTR interval has expired, due in-part to the expectation of the idle delay interval 430. In response to the second read RD command at 418, the memory core 102 performs the read-related suboperations that ties-up the memory core 116 for the read interval at 432. Following a second CL timing interval, shown at 434, the second read data begins transfer along the data interface, at 436. The duration of the time interval between transfer of the last bit of write data and the first bit of subsequent read data along the data interface, referred to herein as the write-to-read turnaround bubble, is a direct result of the duration of the memory core idle time interval 430.

[0041] FIG. 5 illustrates how the inclusion of multiple write buffers in the write buffer circuitry of FIG. 1 reduces the memory core idle time interval 430, and correspondingly shortens the write-to-read turnaround bubble. The timing chart of FIG. 5 is similar to that of FIG. 4, albeit with an additional set of timing intervals, at 502, corresponding to three write buffers that each store separate write data corresponding to three write commands, at 504, 506, and 508. For this particular example, the three write buffers are assumed to be empty, with the write operations thus having no impact on the memory core. Following the last write command, at 508, a read command RD, at 510, is received by the memory device following a write-to-read WTR time that is far less than the WTR interval of FIG. 4. The reason for the reduced WTR interval is that the memory controller 102 determines that there will be a small memory core idle time interval since none of the write data corresponding to the three write commands 504, 506 and 508 is fed to the memory core 116, instead sitting in the write buffers. As a result, the subsequent read command RD at 510 may be issued by the memory controller 102 sooner, resulting in read data arriving on the data interface, at 512. The duration of the time interval between transfer of the last bit of write data corresponding to the third write command at 508 and the first bit of subsequent read data along the data interface, the resulting write-to-read turnaround bubble, is significantly shorter than the example of FIG. 4.

[0042] The inclusion of multiple write buffers provides a way to shorten the write-to-read turnaround bubble, especially when the write buffers are originally empty. There is thus an incentive to keep write buffer resources available as-needed to maintain a minimal bubble. Moreover, when bank group architectures are involved, the sequencing of when to retire each of the multiple write buffers may further optimize the reduction of the write-to-read bubbles.

[0043] FIG. 6 illustrates a timing chart similar to FIG. 5, but expanding the detail of the memory core activity by showing activity in three bank groups, at 602, that take advantage ofthree write buffers, at 604, using a retire sequence based on when the write data was stored in the write buffer. With the three write buffers, at 604, already storing prior write data, a sequence of three write commands at 606, 608 and 610 results in three bursts of write data, at 612, 614 and 616 along the data interface that are directed to bank group 0, bank group 1 and bank group 2, respectively. A lone read command for accessing read data from bank group 2 is received by the memory device, at 618. The write buffer storing the previous write data for bank group 0 retires the previous write data to the memory core, at 620, while the write buffers for bank groups 1 and 2 retire their prior write data at 622 and 624. Since the last retire of write data to the memory core happens with the bank group 2 portion of the core, an idle delay interval 626 occurs within the bank group 2 portion of the memory core of several clock cycles. The idle delay interval 626 is taken into account by the memory controller 102, resulting in the read data being applied to the data interface at 628. The resulting write-to- read turnaround bubble is shown at 630.

[0044] FIG. 7 illustrates a way to further reduce the write-to-read turnaround bubble 630 of FIG. 6 by sequencing the retire order of the write buffers based on bank group timings. The timing chart of FIG. 7 is virtually identical to that of FIG. 6, but instead of retiring the write buffers based on the length of time the prior write data was stored in a write buffer, the retire sequence is based on a known bank group timing conflict between a prior write operation and a subsequent read operation to the same bank group. For instance, with the write commands directed to bank groups 0, 1 and 2, and the read command directed to bank group 2, retiring the prior write data directed to bank group 2 from write buffer 2 first, at 702, allows the bank group 2 portion of the memory core to perform the read sub-operations for the read command earlier, at 704, which enables the read data to be placed on the data interface earlier, at 706. This results in a shorter-duration write-to-read timing bubble, at 708, as compared to the bubble 630 shown in FIG. 6.

[0045] The ability to retire the prior write data from a given one of the write buffers may be carried out in a variety of ways. Generally, a desirable overall manner of retiring the write data would be to do so in a way that reduces the latency of ensuing read operations. For one embodiment, an explicit retire command that specifies a unique write buffer to retire write data from may be issued by the memory controller 102 as part of the CA information received by the memory device 104. FIG. 8 shows a command truth table for a current DDR5 command protocol with unused bits for read and write operations at 802, 804, 806, 808, 810 and 812. For one embodiment, use of two to three command bits from the available unused bits of the DDR5 command protocol would support being able to select one from four to eight write buffers to retire write data from.

[0046] In addition to being able to use an explicit command to specify a given write buffer to retire write data to the memory core, an implicit rule set may be employed to optimally retire write data from the multiple write buffers. For example, when data is in the write buffer, an implicit command to retire the data may occur when another write operation is directed to the memory device. Another example may trigger an implicit command to retire data from a given write buffer when a read or write operation is directed to a memory device of a different, or “non-target” rank. The truth table of FIG. 8 specifies that CS_n = Low during the second cycle of a two cycle command controls on-die termination for non-target ranks for write, read and mode register read (MRR) commands. Thus, one may use the CS_n =Low state as a signal to start an implicit retire, with the controller and memory device being compliant to one another that if one or more write buffers are full, then detection of the non- target rank (NTR) signal indicates that a retire should happen. For one embodiment, the write buffers are sequenced from 0 to N, and then wrap around to 0 again.

[0047] While much of the discussion above relates to the memory device side of the memory system, for one embodiment the memory controller 102 generally manages whichdata gets retired and when. Among other things, this involves ensuring no conflicts for internal core resources with read operations and write operations. Further, the correct memory device page in the addressed bank of the memory core should be open when data is being retired from the write buffer. The memory controller 102 also tracks which memory devices 104 have data in their write buffers 126, and the corresponding addresses of the write buffers 126. Lastly, the memory controller 102 is also responsible for taking into account the memory device 104 and overall memory channel timings. Such timing constraints are managed at the row, bank, bank group, rank and channel level.

[0048] For certain embodiments involving buffered memory modules, such as registered dual inline memory modules (RDIMM), multiplexer combined rank DIMMS (MCRDIMM) and multi-ranked buffered DIMMS (MRDIMM), to name but a few, various write buffer retirement operations may be directed to, or offloaded to one or more buffer IC chips, such as a registered clock driver (RCD) chip. The assistance provided by the one or more buffer IC chips may prove useful in saving command bandwidth between the memory controller 102 and the memory devices 104, and improving execution time.

[0049] Referring now to FIG. 9A, one embodiment of a memory system 900 includes a memory controller 902 that is coupled to at least one memory module 904. For one embodiment, the memory module 904 takes the form of a dual inline memory module (DIMM) that includes a first group of memory devices organized as a first rank 906 and mounted on a front side 908 of a module substrate. Mounted on a back side 910 of the substrate is a second group of memory devices organized as a second rank 912. A buffer IC chip 914 also mounts to the front side of the module substrate and includes a primary interface 916 that couples to the memory controller 902 via a command / address (CA) bus 918, and a secondary interface 920 that couples to the first and second ranks of memory devices 906 and 912.

[0050] For some embodiments, each rank of the two ranks 906 and 912 may be further sub-divided to support multiple independent channels, where, for example, the left-side subgroup of memory devices collectively form a first channel “Channel A”, and the right-side sub-group of memory devices collectively form a second channel “Channel B.” The primary interface 916 receives command / address information via the primary CA bus DCA, and chip select information for the two ranks 906 and 912 via respective control buses DCS0 and DCS1. For one embodiment, the secondary interface 920 includes a first channel CA bus 922 that supports Channel A, and a second channel CA bus 924 that supports channel B. As shown in FIG. 9B, the second channel bus 924 of the secondary interface 920 includes a first copy of a CA bus, at 925, for the second channel CHANNEL B, and a second copy of a CA bus, at 927, for the second channel CHANNEL B. The first copy of the CA bus 925 includes a secondary CA bus QCAA that passes a delayed version of the CA information received by the primary CA bus DCA. Respective secondary chip select busses QCS0A and QCS0B pass delayed versions of the signals received from the respective primary control buses DCS0 and DCS1. The second copy of the CA bus 927 is formed similar to the first copy of the CA bus 925. For some embodiments, with the memory devices organized such that the front side memory devices form the first rank 906 and the back-side memory devices form the second rank 912, each of the CA bus copies 925 and 927 are shared between the ranks 906 and 912. The left side of the memory module 904 is formed similar to the right side.

[0051] FIG. 9C illustrates a timing chart that shows one example of how the buffer IC chip 914 can generate, from a single input command received from the memory controller 902, multiple output commands that execute in parallel. At 926, an input write command Wr is received via the primary CA bus DCA. For one embodiment, the input write command may include extended command bits that can specify a retire operation for selected write buffers in a non-target rank. For other embodiments, maintaining a low state for a chip selectcontrol signal presented by the primary interface 916 may indicate that a retire operation is to be carried out for a non-target rank. In FIG. 9C, at 928, such a condition is illustrated for the control bus DCS1. The secondary interface 920 of the buffer chip 914 passes the write command, at 930, with an associated buffer delay tPDM along the secondary CA bus QCA, and the respective chip select signals for the two ranks, along secondary chip select busses QCS0_n and QCSl_n (where “n” denotes the CA bus copy, such as at 925 or 927). The write command (for the target rank), at 930, and the retire command (for the non-target rank), at 932, are thus fed to the memory devices of the selected channel for execution by the devices in parallel.

[0052] FIG. 10A illustrates a memory system 1000 similar to that of FIG. 9 A, but with a memory module 1002 having an RCD chip 1004 and memory devices of a lower row organized into a first rank RANK 0, and the memory devices of the upper row organized into a second rank RANK 1. FIG. 10B illustrates further detail of the right side of the memory module 1002 of FIG. 10A. This top / bottom row rank topology lends itself to allowing for separate CA busses for each rank, with bus QCAA serving the first rank RANK 0, and QCAB serving the second rank RANK 1. In this configuration, the RCD chip 1004 is free to send different operations to each rank without necessarily needing to use non-target commands, such as those described above. FIG. 10C illustrates a timing chart that is similar to FIG. 9C, showing how an input write command, at 1006, can result in the RCD chip 1004 generating multiple output commands, such as a write command, at 1008, and a dedicated retire command, at 1010, to save host-side command bandwidth and provide improved execution timing.

[0053] FIG. 11 illustrates one embodiment of a memory system 1100 that is similar to the system 900 of FIGs. 9A-9C, but including first and second memory modules 1102 and 1104. The first memory module 1102 includes a first RCD chip 1106 that is coupled to a memorycontroller 1108 via a CA bus DCA and chip select busses DCSO and DCS1. The first chip select bus DCSO provides a first rank selection signal for a first rank RANK 0 of memory devices on the front side of the memory module 1102, while the second chip select bus DCS1 provides a second rank selection signal for a second rank RANK 1 of memory devices on the back side of the module 1102. The second memory module 1104 is configured similarly, with the CA bus DCA being shared with the first memory module 1102, and having a second RCD chip 1110 that is coupled to the memory controller 1108 via the CA bus DCA and chip select busses DCS2 (for a third rank RANK 2 of memory devices on the front-side) and DCS3 (for a fourth rank RANK 3 of memory devices on the back-side).

[0054] In operation, the configuration of the memory system 1100 allows for a variety of operations to be performed in parallel in response to a single command received from a host. For example, the memory controller 1108 may issue a write command to the first rank RANK 0 via the first RCD chip 1106. The second RCD chip 1110 may, through the nontarget rank mechanism described above, realize that it is the non-target rank for the right channel, and start a retire operation to the third rank RANK 2 on the right channel. Further, if the second RCD chip 1110 realizes that there are no commands being issued to the left channel, it can simply start a retire on the left channel, to the second rank RANK 1. These actions might be part of "agreed-upon" rules so that the memory controller 1108 knows these operations will be internally initiated, and can manage core timings and bank utilizations, and keep track of which write buffers are full or empty.

[0055] The embodiments described above generally relate to ways of utilizing write buffer circuitry disposed on a memory device, such as a DRAM integrated circuit (IC) chip. Embodiments described below achieve similar benefits, but involve write buffer circuitry that is not disposed on the memory device, but rather located off-chip, such as in one or morebuffer IC chips that couple to one or more memory devices through use of a memory module or other form of chip-to-chip substrate.

[0056] FIG. 12 illustrates one embodiment of a memory system 1200 that includes a memory controller 1202 coupled to a memory module 1204. The memory module 1204 includes not only an RCD chip 1206, but also a plurality of data buffer chips 1208 that cooperate with the RCD chip 1206 to form a buffer chipset 1210. With the RCD chip 1206 generally handling the buffering of control information for a given write operation, the data buffer chips 1208 generally buffer the write data corresponding to the write operation. For one embodiment, the write buffer circuitry described above with respect to FIGs. 1-8 may be included in the buffer chipset 1210, where write data buffers are included in the data buffer chips 1208, and write CA buffers are included in the RCD chip 1206. While FIG. 12 shows the RCD chip 1206 and the data buffer chips 1208 as being separate, for some embodiments, the functionality of the RCD chip 1206 and the data buffer chips 1208 may be combined into one integrated circuit (IC) buffer chip.

[0057] FIG. 13 provides further detail for one embodiment of the write buffer circuitry employed in the buffer chipset 1210. While the circuitry is similar to that described above with respect to the write buffer circuitry 300 of FIG. 3, the interrelationship between the buffer chips entails some separation between a first portion of the buffer circuitry that stores the control information, and a second portion of the buffer circuitry that stores the write data. With that in mind, one embodiment of the RCD chip 1206 employs a primary or input CA interface 1304 that includes a CA receiver 1306 that receives a 7 -bit CA signal via a CA bus DC A. A chip select receiver 1308 receives a 2 -bit chip select signal via a chip select bus DCS. The received CA signal is deserialized by a deserializer 1310, and fed to write CA buffer logic 1312. The write CA buffer logic 1312 includes a write buffer control circuit 1314 that provides control signals for an input selector 1316 and an output, or retire selector 1318.Multiple write CA buffers 1320 are disposed between the selectors 1316 and 1318, and provide temporary storage for command information associated with a given write operation. The write buffer control circuit 1314 is responsive to the chip select information in the form of extended retire commands or non-target retire commands to select and pass write data from, via the output selector 1318, a given one of the multiple write CA buffers 1320 for retiring the associated control information to a specified address of core circuitry for a given memory device of the module. The output selector 1318 feeds the selected control information to a Broadcast Communications BCOM encoder 1322 to generate BCOM command information, and to a secondary or output interface 1324. For one embodiment, the BCOM encoding may be compliant with the BCOM Commands truth table shown in FIG. 14.

[0058] Further referring to FIG. 13, the output interface 1324 includes a clock domain crossing circuit 1326 which synchronizes the BCOM signal, the unencoded write buffer output, the chip select signal, and respective transmit and receive clocks txclk and rxclk. A first portion of the synchronized information, including the BCOM signal BCOM, and a data buffer chip select signal BCS is transmitted to each data buffer 1208 (FIG. 12) by transmit circuitry 1330. A second portion of the synchronized information, including secondary interface CA signals QCA and chip select signal QCS is transferred, by the transmit circuitry 1330, to the appropriate rank / channel of memory devices associated with the particular RCD chip 1206.

[0059] With continued reference to FIG. 13, one embodiment of a data buffer chip 1208 includes a primary data interface 1332 that supports bidirectional data transmissions at a nibble, or 4-bit granularity. The primary data interface 1332 includes data transceiver circuitry 1334 that receives write data for write operations, and transmits read data for read operations. Differential strobe transceiver circuitry 1336 receives source-synchronous writetiming information for the write data, and transmits a source-synchronous read strobe signal to accompany read data transmissions. A BCOM receiver circuit 1338 receives the BCOM signal BCOM and the BCOM chip select signal BCS n from the RCD chip 1206, and feeds the received information to a transaction control circuit 1340.

[0060] Further referring to FIG. 13, the data buffer chip 1208 includes write buffer logic 1342 that is configured similar to the write buffer logic 1312 of the RCD chip 1206, including a write buffer control circuit 1344 that provides control signals for an input selector 1346 and an output, or retire selector 1348. Multiple write data buffers 1350 are disposed between the selectors 1346 and 1348, and provide temporary storage for write data associated with a given write operation. The write buffer control circuit 1344 is responsive to the transaction control circuit 1340 in selecting which of the write data buffers to store new write data, via the input selector 1346, and in selecting which of the write data buffers to retire write data, via the output selector 1348.

[0061] With continued reference to FIG. 13, retired write data that is passed to the output of the output selector 1348 is serialized by a serializer 1350, and transmitted by transmitter 1351 to a secondary data bus MDQ of the data buffer 1208, along with an accompanying differential strobe signal, generated by a differential strobe generation circuit 1352, and transmitted by a strobe transmitter 1354 along secondary strobe paths MDQS t and MDQS c.

[0062] FIG. 15A illustrates a timing chart, from the perspective of the memory controller 1202 in the memory system 1200 (FIG. 12), showing a stream of issued write commands, at 1502, followed by a read command, at 1504. For purposes of illustration, the read command at 1504 is dispatched by the memory controller 1202 a number of clock cycles corresponding to a write-to-read turnaround time following the last write command, at 1506, with theexpectation that write buffer circuitry disposed on the buffered memory module 1204 will reduce the write-to-read turnaround time.

[0063] FIG. 15B illustrates a timing chart, from the perspective of a data buffer 1208 in the memory system 1200 (FIG. 12), with the write buffer circuitry disabled. Without the write buffer circuitry enabled, the write data corresponding to the stream of write commands 1502 cannot begin transfer along the secondary data bus MDQ of the data buffer until after data buffer-related propagation delays tPD2 and tPDMb have expired, at 1508. In contrast, when the write buffer circuitry is enabled, the data buffer propagation delays are effectively hidden, as shown at 1510 in the timing chart of FIG. 15C, allowing the final portion of write data, at 1512, from the stream of write commands 1502, to be pulled in by the interval shown by T, resulting in a reduction in the write-to-read turnaround time by the interval T.

[0064] While the embodiments above describe how write buffer circuitry may be implemented on a standard DDR5 DRAM memory device, or in buffer chips coupled to standard DDR5 DRAM memory devices, the write buffer circuitry may also be employed in other DRAM memory device architectures. FIG. 16 illustrates a high-level exploded view of a three-dimensional (3-D) DRAM device, generally designated 1600. The 3D DRAM includes a primary die 1602 that includes an external memory interface 1604 for communicating with a memory controller (not shown). The primary die 1602 includes a memory core 1606 organized into bank groups 1605 with banks 1607 that are arranged in a manner similar to that of standard DDR memory devices. Multiple secondary die 1608 are stacked on the primary die 1602, with the secondary die 1608 formed similar to the primary die 1602 but without the external memory interface 1604. Through-silicon-vias, such as at 1609, interconnect the secondary die 1608 with the primary die 1602 such that the external memory interface 1604 becomes shared by all the die.

[0065] Like the memory device 104 of FIG. 1, various locations in the 3D DRAM 1600 lend themselves well to employing embodiments of the write buffer circuitry described herein. For example, the control portion of the write buffer circuitry for the primary die 1602 may be positioned in the external memory interface 1604, such as at 1610, or 1612, or the bank group circuitry 1605, at 1614, or in the bank circuitry 1607, at 1616. Likewise, the data portion of the write buffer circuitry may be positioned in the external memory interface 1604, such as at 1618, or in the bank group circuitry 1605, at 1620, or in the bank circuitry, at 1622. For the secondary die 1608, the write buffer circuitry may be positioned in any of the locations noted above for the primary die 1602, except for the external memory interface 1604 (which is only included in the primary die 1602).

[0066] FIG. 17 illustrates a high-level exploded view of a high-bandwidth memory (HBM) device, generally designated 1700. The HBM memory device 1700 includes a base die 1702 that includes memory interface circuitry 1704 for communicating with a memory controller (not shown). Stacked on the base die 1702 are multiple DRAM die 1706 that include a memory core 1708 organized into bank groups with banks in a manner similar to that of standard DDR memory devices. Through-silicon-vias (not shown) interconnect the DRAM die 1706 with the base die 1702. Like the other DRAM embodiments described herein, to employ write buffer circuitry, the control portion of the write buffer circuitry may be positioned in the memory interface circuitry 1704, such as at 1710 for each channel, or at 1712 for each DRAM die 1706, or bank group circuitry 1714, at 1716, or in bank circuitry 1718, at 1720. Likewise, the data portion of the write buffer circuitry may be positioned in the memory interface circuitry 1704, such as at 1722, or 1724, or in the bank group circuitry 1714, at 1726, or in the bank circuitry 1718, at 1728.

[0067] Those skilled in the art will appreciate the relatively straightforward circuitry for buffering write data and associated control information for write operations as describedabove. By employing write buffer circuitry to temporarily store write data, and selectively retiring the stored write data based on bank group timings, reductions in write-to-read turnaround times may be achieved.

[0068] When received within a computer system via one or more computer-readable media, such data and / or instruction-based expressions of the above described circuits may be processed by a processing entity (e.g., one or more processors) within the computer system in conjunction with execution of one or more other computer programs including, without limitation, net-list generation programs, place and route programs and the like, to generate a representation or image of a physical manifestation of such circuits. Such representation or image may thereafter be used in device fabrication, for example, by enabling generation of one or more masks that are used to form various components of the circuits in a device fabrication process.

[0069] In the foregoing description and in the accompanying drawings, specific terminology and drawing symbols have been set forth to provide a thorough understanding of the present disclosure. In some instances, the terminology and symbols may imply specific details that are not required to practice aspects of the disclosure. For example, any of the specific numbers of bits, signal path widths, signaling or operating frequencies, component circuits or devices and the like may be different from those described above in alternative embodiments. Also, the interconnection between circuit elements or circuit blocks shown or described as multi-conductor signal links may alternatively be single-conductor signal links, and single conductor signal links may alternatively be multi-conductor signal links. Signals and signaling paths shown or described as being single-ended may also be differential, and vice-versa. Similarly, signals described or depicted as having active-high or active-low logic levels may have opposite logic levels in alternative embodiments. Component circuitry within integrated circuit devices may be implemented using metal oxide semiconductor(MOS) technology, bipolar technology or any other technology in which logical and analog circuits may be implemented. With respect to terminology, a signal is said to be “asserted” when the signal is driven to a low or high logic state (or charged to a high logic state or discharged to a low logic state) to indicate a particular condition. Conversely, a signal is said to be “deasserted” to indicate that the signal is driven (or charged or discharged) to a state other than the asserted state (including a high or low logic state, or the floating state that may occur when the signal driving circuit is transitioned to a high impedance condition, such as an open drain or open collector condition). A signal driving circuit is said to “output” a signal to a signal receiving circuit when the signal driving circuit asserts (or deasserts, if explicitly stated or indicated by context) the signal on a signal line coupled between the signal driving and signal receiving circuits. A signal line is said to be “activated” when a signal is asserted on the signal line, and “deactivated” when the signal is deasserted. Additionally, the prefix symbol “ / ” attached to signal names indicates that the signal is an active low signal (i.e., the asserted state is a logic low state). A line over a signal name (e.g., ‘ < signal name >’) is also used to indicate an active low signal. The term “coupled” is used herein to express a direct connection as well as a connection through one or more intervening circuits or structures. Integrated circuit device “programming” may include, for example and without limitation, loading a control value into a register or other storage circuit within the device in response to a host instruction and thus controlling an operational aspect of the device, establishing a device configuration or controlling an operational aspect of the device through a one-time programming operation (e.g., blowing fuses within a configuration circuit during device production), and / or connecting one or more selected pins or other contact structures of the device to reference voltage lines (also referred to as strapping) to establish a particular device configuration or operation aspect of the device. The term “exemplary” is used to express an example, not a preference or requirement.

[0070] While aspects of the disclosure have been described with reference to specific embodiments thereof, it will be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the disclosure. For example, features or aspects of any of the embodiments may be applied, at least where practicable, in combination with any other of the embodiments or in place of counterpart features or aspects thereof. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

Claims

CLAIMS1. A memory, comprising: interface circuitry to receive at least one write command; multiple banks organized into multiple bank groups and operative in compliance with respective bank group timings, each of the multiple banks including an array of storage cells; multiple write buffers to temporarily store write data associated with at least one write command; and write buffer selection circuitry, based on the bank group timings, to retire the temporarily stored write data from a selected one of the multiple write buffers and to direct the retired write data to a given one of the arrays of storage cells.

2. The memory of claim 1, wherein: the write buffer selection circuitry includes control circuitry to execute the retire of the temporarily stored write data from the selected one of the multiple write buffers based on an implicit command.

3. The memory of claim 2, wherein: the implicit command includes reception of a subsequent write command.

4. The memory of claim 2, wherein: the implicit command includes reception of a Non-Target Rank indication.

5. The memory of claim 1, wherein: the write buffer selection circuitry includes control circuitry to execute the retire of the temporarily stored write data from the selected one of the multiple write buffers in response to an explicit command.

6. The memory of claim 5, wherein: the explicit command comprises a retire command in the form of encoded control bits.

7. The memory of claim 1, wherein: the bank group timings include timing constraints to avoid bank group conflicts between the multiple bank groups.

8. The memory of claim 1, wherein: the multiple write buffers reside in the interface circuitry or are distributed among the multiple banks or the multiple bank groups.

9. The memory of claim 1, wherein: the multiple write buffers reside in off-chip buffer circuitry.

10. The memory of claim 9, wherein: the off-chip buffer circuitry includes: a command / address (CA) buffer integrated circuit (IC) chip to store control information associated with the at least one write command; and at least one data buffer IC chip to store write data associated with the at least one write command.

11. The memory of claim 1, wherein: each array of storage cells comprises dynamic random access memory (DRAM) storage cells.

12. A dynamic random access memory (DRAM) integrated circuit (IC) memory chip, comprising: multiple banks organized into multiple bank groups and operative incompliance with respective bank group timings, each of the multiple banks including an array of storage cells; interface circuitry to receive a first write command and a second write command, the first write command associated with first write data for writing to a first one of the multiple banks, the second write command associated with second write data; a first write buffer to temporarily store the first write data; a second write buffer to temporarily store the second write data; and write buffer selection circuitry, based on the bank group timings, to retire the temporarily stored first write data and second write data in a prioritized sequence from the first write buffer and the second write buffer, and to direct the retired first write data and second write data to selected ones of the arrays of storage cells according to the prioritized sequence.

13. The DRAM IC chip of claim 12, wherein: the retired first write data and second write data are directed to at least one of the multiple banks in a same one of the multiple bank groups.

14. The DRAM IC chip of claim 12, wherein: the retired first write data and second write data are directed to respective ones of the multiple banks in different ones of the multiple bank groups.

15. The DRAM IC chip of claim 12, wherein: the write buffer selection circuitry includes control circuitry to execute the retire of the temporarily stored first write data and the second write data in the prioritized sequence based on an implicit command.

16. The DRAM IC chip of claim 12, wherein: the write buffer selection circuitry includes control circuitry to execute theretire of the temporarily stored first write data and the second write data in the prioritized sequence in response to an explicit command.

17. The DRAM IC chip of claim 12, wherein: the bank group timings include timing constraints to avoid bank conflicts between the multiple bank groups.

18. The DRAM IC chip of claim 12, wherein: the multiple write buffers reside in the interface circuitry or are distributed among the multiple banks or the multiple bank groups.

19. A buffer circuit, comprising: a first write buffer to temporarily store first write data associated with a first write command and address information specifying one from a plurality of memory banks that are organized into multiple bank groups, the multiple bank groups operative in compliance with respective bank group timings, the first write data originating from a memory controller; a second write buffer to temporarily store second write data associated with a second write command and originating from the memory controller; and write buffer selection circuitry, based on the bank group timings, to retire the temporarily stored first write data and second write data in a prioritized sequence from the first write buffer and the second write buffer, and to direct the retired first write data and second write data to selected ones of the plurality of memory banks according to the prioritized sequence.

20. The buffer circuit of claim 19, wherein: the write buffer selection circuitry includes control circuitry to execute the retire of the temporarily stored first write data and the second write data in theprioritized sequence based on an implicit command.

21. The buffer circuit of claim 19, wherein: the write buffer selection circuitry includes control circuitry to execute the retire of the temporarily stored first write data and the second write data in the prioritized sequence in response to an explicit command.

Images