System for high speed computing using optical exchanger

Abstract

The present disclosure provides high bandwidth processing by sharing a memory die over a plurality of computing dies via an optical switch. The optical switch may be configured to operate both as an optical switch and as an optical demultiplexer. An optical switch configuration for an optical switch allows data to be written from any computing die to one of a plurality of memory chips via an optical connection. The optical demultiplexer configuration allows data to be broadcast from the memory die to the plurality of computing dies.

Description

Background Art

[0001] Computing packages with multiple high bandwidth memories ("HBMs") and multiple compute dies can be used to perform high-speed computing. However, it is becoming increasingly difficult to fit additional memory and compute dies onto the package in a manner that allows data to be efficiently transferred using electrical transport. Additionally, known optical transport systems make inefficient use of memory and compute dies. Summary of the invention

[0002] The present application relates to a system for providing a high bandwidth connection between a memory and a computing unit. Aspects of the present disclosure allow for the disaggregation of high-speed memory and computing units while maintaining high bandwidth at relatively low power and cost. Communication between the memory and the computing unit is achieved via an optical switch that allows optical signals to be transmitted between the memory and the computing unit in an efficient manner. The optical switch can be configured so that any computing unit can write data to any memory unit. In addition, the optical switch can be configured so that the memory unit can broadcast data to multiple computing units simultaneously.

[0003] According to aspects of the present disclosure, a system for computing and memory communication may include: multiple high bandwidth memory units (HBMs), each HBM having a memory optical interface; multiple computing units, each of the computing units having a computing optical interface; and an optical cable optically connected to the memory optical interfaces of the multiple HBMs and the computing optical interfaces of the multiple computing units, wherein the optical I / O includes a demultiplexer configured to transmit a read-related signal received from any one of the multiple HBMs to one or more of the computing units in the multiple computing units, and wherein the optical switch includes a switch configured to transmit a write-related signal received from any one of the multiple computing units to any one of the multiple HBMs.

[0004] According to other aspects of the present disclosure, the memory optical interfaces of the plurality of HBMs may further include a first memory optical interface configured to send a read-related signal and a second memory optical interface configured to receive a write-related signal. The computing optical interfaces of the plurality of computing units may further include a first computing optical interface configured to receive a read-related signal and a second computing optical interface configured to send a write-related signal.

[0005] According to yet other aspects of the present disclosure, the demultiplexer of the optical switch may further include an input optical interface for each of the plurality of HBMs and an output optical interface for each of the computing units. The demultiplexer may further be configured to transmit read-related signals to more than one computing unit simultaneously. The demultiplexer may further be configured to transmit read-related signals while the switch transmits write-related signals.

[0006] According to other aspects of the present disclosure, the plurality of HBMs may include an active HBM and at least one spare HBM, and wherein the switch is configured to transmit write-related signals only to the active HBM.

[0007] According to yet other aspects of the present disclosure, a plurality of HBMs may be located on a first substrate, and a plurality of computing units may be located on a second substrate. In addition, the first substrate may be located on a first device, and the second substrate may be located on a second device.

[0008] According to yet other aspects of the present disclosure, the memory optical interface and the computation optical interface may be electrically connected via one or more interposers.

[0009] According to other aspects of the present disclosure, a system for computing and memory communication may include: a first package having multiple high bandwidth memory cells (HBM); a second package having multiple computing cells; and an optical switch configured to optically connect the HBM and the computing cells, wherein the optical switch includes a demultiplexer configured to provide read-related signals from any one of the multiple HBMs to any group of computing cells in the multiple computing cells, and wherein the optical switch includes a switch configured to provide write-related signals from any one of the multiple computing cells to any one of the multiple HBMs.

[0010] According to other aspects of the present disclosure, each of the plurality of HBMs may further include a first memory optical interface configured to transmit a read-related signal and a second memory optical interface configured to receive a write-related signal.

[0011] According to yet other aspects of the present disclosure, the plurality of computing units may further include a first computing optical interface configured to receive a read-related signal and a second computing optical interface configured to send a write-related signal.

[0012] According to yet other aspects of the present disclosure, the demultiplexer of the optical switch may further include an input optical interface for each of the plurality of HBMs and an output optical interface for each of the computing units. In addition, the demultiplexer may further be configured to transmit the read-related signal to the plurality of computing units simultaneously.

[0013] According to aspects of the present disclosure, the optical switch may also include an optical chiplet optically connected to a plurality of optical fibers, and wherein each of the computing units is optically connected to one or more of the plurality of optical fibers interconnected with an optical interface for the computing unit.

[0014] According to other aspects of the disclosure, the plurality of HBMs may include an active HBM and at least one spare HBM, and wherein the switch is configured to transmit write-related signals to the active HBM. According to other aspects of the disclosure, the plurality of HBMs may be configured as part of a 3D memory stack.

[0015] According to yet other aspects of the present disclosure, the first package may be located on the first device, and the second package may be located on the second device. In addition, the optical switch may be located on the first package or on the second package. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a block diagram of a computing package having an optical switch according to aspects of the present disclosure.

[0017] Figure 2 is a block diagram of two computing packages with optical switches according to aspects of the present disclosure.

[0018] Figure 3 is a diagram of a partial side view of two computing packages having optical switches according to aspects of the present disclosure. DETAILED DESCRIPTION

[0019] The technology involves high-bandwidth processing using optical connections between memory and computing units. Specifically, the present disclosure provides a high-speed communication system between a high-performance computing unit (such as a tensor processing unit (TPU) or other application-specific integrated circuit (ASIC)) and a high-performance memory unit (such as a high-bandwidth memory unit (HBM)). The system disclosed herein can be implemented in conjunction with specific high-performance computing applications, including use of the system in machine learning accelerators, supercomputer systems, and cloud computing applications.

[0020] According to the present disclosure, multiple computing units can be optically interconnected with multiple memory units in a manner that allows efficient high-bandwidth data transfer. These optical connections can be controlled by an optical switch that is configured to allow any of the computing units to optically transfer data from any of the computing units to any of the memory units. In addition, the optical switch can be configured so that data stored at any particular memory unit can be broadcast to multiple computing units simultaneously. By using the disclosed optical switch in conjunction with multiple low-latency, high-bandwidth optical connections, memory in multiple memory units can be efficiently shared across multiple computing units in a manner that allows increased bandwidth while also limiting associated power and operating costs.

[0021] Figure 11 is a block diagram 100 providing an example system 101 according to the present disclosure. The system includes a plurality of compute dies (XPUs) 102 and a plurality of HBM dies 104. Each compute die 102 and each HBM die 104 may be electrically connected to an optical interface 110, which may include an optical output 112 and an optical input 114. The optical output 112 and the optical input 114 are connected to an optical fiber 122, which is configured to transmit an optical signal over a length of the optical fiber 122.

[0022] The system 101 also includes an optical switch 106. The optical switch 106 is configured to send and receive optical signals between the computing die 102 and the HBM die 104. The optical signals transmitted between the computing die 102 and the HBM die 104 can be divided into "write-related" transmissions and "read-related" transmissions. Write-related transmissions include signal transmissions related to the process in which data is written to the HBM die 104. Read-related transmissions include signal transmissions related to the process of reading data from the HBM die 104. The optical switch 106 is configured to include a write switch 108 and a read demultiplexer 106. The write switch 108 is configured to direct write-related transmissions from the computing die 102 to the HBM die 104, while the read demultiplexer 106 is configured to broadcast data from the HBM die 104 to multiple computing dies 102.

[0023] Thus, the optical switch 106 is configured to receive write-related signal transmissions from any of the compute dies 102 and transmit write-related signal transmissions to any of the HBM dies 104. For example, the XPU1 computer die 102 may have a set of data to be written to one of the HBM dies 104. The set of data may be optically transmitted by the optical interface 110 of the XPU1 computer die 102 via the optical output 112 and along the optical fiber 122. The optical signal is received by the optical switch 106 via the write optical input 117 electrically connected to the write switch 108. Upon receiving the optical signal at the write optical input 117, the write switch 108 is configured to selectively switch the transmission of the received signal to one of the HBM dies 104. For example, the received signal may contain an address or other identifier indicating that the data within the received signal is to be stored at a specific HBM die 104 (e.g., HBM2). The write switch 108 may be configured to identify a write optical output 118 corresponding to the identified HBM die 104 and transmit a write-related signal transmission via the identified optical output 118. The optical transmission may then proceed from the identified optical output 118 along the optical fiber 122 to the optical input 114 of the identified HBM die 104. The HBM die 104 may then write the data contained within the received transmission into its memory in accordance with the write-related signal transmission. Similarly, a write-related signal transmission from another computing die 102 (such as XPU2) may be directed by the write switch 108 to any one of the HBM dies 104.

[0024] In addition, the optical switch 106 is configured to allow data to be broadcasted from the HBM die 104 to multiple computing dies 102. For example, the HBM die 104 (such as HBM1) can read data from its memory and transmit the read data via the optical output 112. The data is transmitted as an optical signal via the optical fiber 122 as a read-related transmission to the read optical input 115. As shown in the block diagram 100, each read optical input 115 is electrically connected to the read demultiplexer 107. Upon receiving the optical signal at the read optical input 115, the read demultiplexer 107 is configured to selectively transmit the received optical signal to one or more of the computing dies 102. For example, the read-related transmission received at the read optical input 115 may specify one or more computing dies 102 to receive the read-related transmission. The read demultiplexer can be configured to identify the read optical output 116 corresponding to the specified computing die 102. The read-related transmission is transmitted to each of the identified compute dies 102 via the identified read optical output 116 and the corresponding optical fiber 122. Each of the identified compute dies 102 can then access and process the data from the read-related transmission via the optical input 114.

[0025] Read-related transmissions from the demultiplexer 107 can occur simultaneously along multiple read optical outputs 116. By allowing read-related transmissions from a particular HBM die 104 to be broadcast to multiple computing dies 102 along with accesses to any HBM die 104 by any computing die 102, the system 101 allows the capacity of bandwidth and memory to be increased by an order of magnitude. The implementation of the system 101 also avoids data duplication and data transfer. Therefore, the system 101 reduces the total cost of ownership while improving the overall performance of the computing die 102 and the HBM die 104. In addition, the optical switch 106 can be configured so that the optical fiber 122 is connected to an optical chiplet that is configured to send and receive optical communications over multiple optical fibers.

[0026] The compute die 102 may issue a read request by transmitting the read request to the HBM die 104 via the optical switch 106. The read request from the compute die 102 may be transmitted via the optical output 112 of the compute die 102 and may be transmitted to the appropriate HBM die 104 via the write switch 108. Alternatively, the HBM die 104 may receive the read request from another device, including other HBM dies 114 that are electrically connected to each other.

[0027] The number of computing dies 102 and HBM dies 104 within the system 101 may be based on the desired parameters of the application and hardware of the system. In the block diagram 100, the computing dies 102 are numbered from XPU1 to XPUn. Similarly, the HBM dies 104 are numbered from HBM1 to HBMn. However, this does not mean that the number of computing dies 102 and HBM dies 104 must be the same. In addition, the configuration of the HBM die 104 and the computing die 102 may be different. For example, at least a portion of the HBM die 104 may be included in a 3D stack of HBM dies 104, wherein the signal is transmitted within the stack through silicon vias (TSVs).

[0028] According to aspects of the present disclosure, the system 101 may include one or more spare HBM dies 104. A spare memory die is a memory die that is not currently active, such that it does not perform read or write operations on the system. For example, a memory die 104 may be designated as a spare die and maintained in a sleep state or other low-power state such that the memory die 104 does not consume any power or consumes as much power as the active memory die 104. In addition, the system 101 may be configured such that the memory die 104 is designated as a spare die such that it does not perform specific system operations, such as those associated with read or write requests from the compute die 102.

[0029] Therefore, the optical switch 106 can identify one or more memory dies 104 as spare memory dies, and will not direct read or write transmissions to the memory die designated as the spare memory die. However, the memory die 104 can change its designation so that the spare memory die can become an active die, or the active die can enter a dormant state and be designated as a spare memory die. For example, if the HBM1 memory die 104 is an active die that is determined to be experiencing a failure or requires diagnostic testing, the active HBM1 memory die 104 can be placed in a dormant state and designated as a spare memory die. Once the HBM1 memory die 104 has been designated as a spare die, the optical switch 106 will not transmit read or write requests to the HBM1 memory die 104.

[0030] Additionally, a previously active HBM1 memory die 104 may be replaced by another memory die 104 that is currently designated as a spare die. For example, an HBM2 memory die 104 may currently be designated as a spare die, but it may be restored from a dormant state and designated as an active memory die 104. Once designated as an active memory die, the compute die 102 and the optical switch 106 may be configured to send read and write requests to the HBM2 memory die 104. Thus, the system 101 may be configured to maintain a certain level of operation even in the event that one or more memory die 104 experience a failure or are at least temporarily removed from the set of active memory die 104.

[0031] like Figure 1 As shown, the components of system 101 may be placed on a single silicon substrate 130. For example, compute die 102, HBM die 104, write switch 108, and read demultiplexer 107 may each be placed on silicon substrate 130 along with corresponding chiplets for optical input and optical output.

[0032] However, according to aspects of the present disclosure, the disclosed system may be configured such that the compute die 102 and the HBM die 104 reside on different substrates and may be part of different packages. Figure 2 2 is a block diagram 200 of a system 201 in which a plurality of HBM dies 104 reside on a first substrate 230, and a compute die 102 and an optical switch 106 reside on a second substrate 232. The substrates 230 and 232 may be part of separate packages within the same device, and the separate packages may reside in different devices. For example, the substrate 230 may be part of a package within a first device located in a first server rack, and the substrate 232 may be part of a package within a second device located in a second server rack.

[0033] In addition, the length of the optical fiber 122 may be sufficient to extend between two different devices, including, for example, devices located approximately 10 meters apart from each other on different server racks. Although the block diagram 200 shows individual optical fibers 122 connecting the HBM die 104 to the optical switch 106, these optical fibers 122 may be bundled such that they provide a bundled optical connection between the HBM die 104 and the optical switch 106. Similarly, there may be a bundled optical connection to connect the optical switch 106 to the computing die 102.

[0034] Although the system 201 can be configured such that the HBM die 104 resides in a different package or a different device than the compute die 102, the system 201 can be configured to operate in the manner described above for the system 101. Thus, the optical switch 106 of the system 201 can selectively transmit write-related transmissions from any compute die 102 to any HBM die 104 via the write switch 108, and the optical switch 106 can also selectively transmit read-related transmissions from any HBM die 104 to one or more compute die 102 via the read demultiplexer 107.

[0035] Figure 3 3 is a block diagram 300 of a cross-sectional side view of a system 301, wherein a computing die 102 and an HBM die 104 are located on different packages. Specifically, the computing die 102 is located on a first substrate 332 of a first package, and the HBM die 104 is located on a second substrate 334 of a second package. Although only one computing die 102 and one HBM die 104 are shown in the block diagram 300, each substrate 332 and 334 may be combined with Figure 2 The depicted approach includes multiple compute dies 102 and multiple HBM dies 104. As shown in block diagram 300, HBM dies 104, optical interfaces 110, and optical switches 106 reside on an interposer 310 connected to a substrate 334. Electrical connections 312 are established between the HBM dies 104 and the optical interfaces 110 via the interposer 310. For each HBM die 104, an optical connection may be established between the optical interface 110 of the HBM die 104 and the optical switch 106. Similarly, an optical connection may be established between the optical switch 106 and the optical interface 110 of each compute die 102.

[0036] As shown in block diagram 300, optical fiber 122 can span from optical switch 106 of first packaging substrate 334 to optical interface 110 of second packaging substrate 332. Electrical connection 322 can be established between optical interface 110 and computing die 102 via interposer 320. Although not shown, optical switch 106 can include a write switch and a read demultiplexer, as described herein. Thus, optical switch 106 of system 301 can selectively transmit write-related transmissions from any computing die 102 to any HBM die 104 via a write switch, and optical switch 106 can also selectively transmit read-related transmissions from any HBM die 104 to one or more computing die 102 via a read demultiplexer.

[0037] Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the previous description of the embodiments should be made in an illustrative manner rather than in a manner that limits the subject matter defined by the claims. In addition, the examples described herein and the provision of clauses expressed as "such as", "including", etc. should not be interpreted as limiting the subject matter of the claims to specific examples; on the contrary, these examples are intended to illustrate only one of many possible embodiments. In addition, the same reference numerals in different figures may indicate the same or similar elements.

Claims

1. A system for computing and memory communication, characterized in that, include: A plurality of high bandwidth memory units HBM, wherein each HBM has a memory optical interface; a plurality of computing units, wherein each of the computing units has a computing optical interface; and An optical switch optically connected to the memory optical interfaces of the plurality of HBMs and the computing optical interfaces of the plurality of computing units, wherein the optical switch includes a demultiplexer configured to transmit a read-related signal received from any one of the plurality of HBMs to one or more of the computing units in the plurality of computing units, and wherein the optical switch includes a switch configured to transmit a write-related signal received from any one of the plurality of computing units to any one of the plurality of HBMs.

2. The system according to claim 1, characterized in that The memory optical interfaces of the plurality of HBMs further include a first memory optical interface configured to send read related signals and a second memory optical interface configured to receive write related signals.

3. The system according to claim 1, characterized in that The computational optical interfaces of the plurality of computational units further include a first computational optical interface configured to receive read related signals and a second computational optical interface configured to send write related signals.

4. The system according to claim 1, characterized in that The demultiplexer of the optical switch also includes an input optical interface for each of the plurality of HBMs and an output optical interface for each of the computational units.

5. The system according to claim 1, wherein: The demultiplexer is further configured to simultaneously transmit the read-related signal to more than one computing unit of the plurality of computing units.

6. The system according to claim 1, wherein: The demultiplexer is further configured to transmit the read-related signal, while the switch transmits the write-related signal.

7. The system according to claim 1, characterized in that The plurality of HBMs include an active HBM and at least one spare HBM, and wherein the switch is configured to transmit write-related signals only to the active HBM.

8. The system of claim 1, wherein: The plurality of HBMs are located on a first substrate, and the plurality of computing units are located on a second substrate.

9. The system according to claim 8, characterized in that The first substrate is located on a first device, and the second substrate is located on a second device.

10. The system according to claim 1, wherein: The memory optical interface and the computation optical interface are electrically connected via one or more interposers.

11. A system for computing and memory communication, characterized in that, include: A first package, wherein the first package has a plurality of high bandwidth memory units HBM; a second package, the second package having a plurality of computing units; as well as An optical switch configured to optically connect the HBM and the computing units, wherein the optical switch includes a demultiplexer configured to provide a read-related signal from any one of the plurality of HBMs to any group of computing units in the plurality of computing units, and wherein the optical switch includes a switch configured to provide a write-related signal from any one of the plurality of computing units to any one of the plurality of HBMs.

12. The system according to claim 11, characterized in that Each of the plurality of HBMs also includes a first memory optical interface configured to send read-related signals and a second memory optical interface configured to receive write-related signals.

13. The system according to claim 11, characterized in that Each of the plurality of computational units further includes a first computational optical interface configured to receive read-related signals and a second computational optical interface configured to send write-related signals.

14. The system according to claim 11, characterized in that The demultiplexer of the optical switch also includes an input optical interface for each of the plurality of HBMs and an output optical interface for each of the computational units.

15. The system of claim 11, wherein: The demultiplexer is further configured to simultaneously transmit the read-related signal to more than one computing unit of the plurality of computing units.

16. The system of claim 11, wherein: The optical switch further comprises an optical chiplet optically connected to a plurality of optical fibers, and wherein each of the computational units is optically connected to one or more of the plurality of optical fibers.

17. The system of claim 11, wherein: The plurality of HBMs include an active HBM and at least one spare HBM, and wherein the switch is configured to transmit write-related signals only to the active HBM.

18. The system of claim 11, wherein: The plurality of HBMs are configured as part of a 3D memory stack.

19. The system of claim 18, wherein: The first package is located on a first device, and the second package is located on a second device.

20. The system of claim 11, wherein: The optical switch is located on the first package.

Images