Supplementary compute node for peer-to-peer interface

Abstract

The embodiment of the invention relates to a supplementary computing node of a peer-to-peer interface. New and advanced computing tools and operations require increasing memory and computing power. Disclosed herein are novel devices and methods that provide a scalable, modular, and adaptive design that enables any additional node of a desired configuration to be connected to and used for a host computing node. The design does not need to change the hardware, software or protocol of the host computing node that can view additional nodes as a single source of supplementary computing and memory. The disclosed design includes connections of additional nodes in a peer-to-peer topology that enables multiple additional node interconnect chains to share a single connection with a host node. This avoids limitations imposed by a single node space and configuration of the amount of compute and memory that may be provided to the host node.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 544,941, filed on October 19, 2023, and U.S. Patent Application No. 18 / 680,419, filed on May 31, 2024. The entire teachings of the above applications are incorporated herein by reference. Background Art

[0003] The development of modern computing and server design, including but not limited to the incorporation and use of generative artificial intelligence (AI), has greatly increased the demand for memory and computing power (referred to herein as "compute"). For example, machine learning models often require a large amount of computing power to execute, train, and use large training sets that need to be stored in easily accessible memory. Single chip designs have been unable to keep up with this growing need for more memory and computing power.

[0004] One possible way to meet the growing need for more memory and compute is to utilize more primary ("host") compute nodes or chips, but this can become expensive and inefficient. Due to these limitations, system and chip designs that incorporate off-chip and / or external memory and compute capabilities have begun to develop, such as memory extenders and near memory computing (NMC), which can be connected to and then utilized by the primary host compute nodes. Improvements are needed in how to incorporate additional memory and compute capabilities without requiring changes to the primary host(s), and can be customized to the memory and compute capabilities needs of a particular system. Summary of the invention

[0005] A novel design is provided, including external, off-chip or off-server devices, which can provide sufficient computing and data storage capabilities to meet the growing demand for computing power and / or memory. This design incorporates peer-to-peer connections between non-host nodes, as a non-limiting example in a daisy chain topology, and enables linear computing and memory scalability, very high memory capacity and very high computing power. In addition, this novel design can be modular and customized for the precise memory and computing needs of a specific system or host computing node without requiring changes to the host computing node at the hardware or software level.

[0006] Certain embodiments of the present invention include an apparatus comprising an I / O interface configured to receive at least one operation and a node set configured to execute the received at least one operation. Each node in the node set includes at least one peer interface and a local memory, the peer interface being configured to connect to another node in the node set and send the received at least one operation, and the local memory being configured to store data.

[0007] At least one node in the node set may be a memory extender configured to provide additional memory to perform at least one received operation or a near memory compute (NMC) configured to provide additional computing power to perform at least one received operation.

[0008] The node set may also be configured to perform the received at least one operation in parallel or jointly. In some embodiments, at least one operation originates from at least one external host. In such embodiments, at least one external host may use the node set as a single entity.

[0009] One or both of the I / O interface and the at least one peer interface may be a PCI Express (PCIe) interface, and the PCI Express (PCIe) interface may utilize a Computer Express Link (CXL) protocol. The node assembly may be configured to be modular, and the number and type of nodes comprising the node assembly may be variable.

[0010] The I / O interface may be configured to couple the node assembly to at least one host computing node, and in such embodiments, the at least one computing node may be located on a different chip than the node assembly.

[0011] A peer interface comprising a node of a node assembly may be configured to removably couple the node assembly.

[0012] Additional embodiments of the present invention include a method for performing a computing operation, the method comprising receiving at least one operation through an I / O interface, and performing the received at least one operation through a node set. Each node in the node set includes at least one peer interface and a local memory, the peer interface is configured to connect to another node in the node set and communicate the received at least one operation, and the local memory is configured to store data.

[0013] In such methods, at least one node in the node set may be a memory expander configured to provide additional memory to perform the received at least one operation or a near memory computing (NMC) configured to provide additional computing power to perform the received at least one operation. The node set may be connected in a daisy chain topology via a peer-to-peer interface.

[0014] In selected embodiments, the method further comprises sending at least one operation from the host computing node. In such embodiments, the host computing node may be located on a server external to the node set. Also in such embodiments, the method may further comprise utilizing an I / O interface to removably couple the node set to the host computing node, and the host computing node may use the node set as a single entity.

[0015] The method may also include modulating a number of nodes or a type of nodes comprising the node set based on a demand for the received at least one operation.

[0016] Other embodiments of the present invention include an apparatus for performing computing operations, the apparatus including components for receiving at least one operation and components for performing the received configured at least one operation by a node set, each node in the node set including components for connecting to another node in the node set and transmitting the received at least one operation. In such embodiments, at least one node in the node set can be a memory extender or near memory computing (NMC), the memory extender including components for providing additional memory to perform the received at least one operation, the near memory computing (NMC) including components for providing additional computing power to perform the received at least one operation.

[0017] The apparatus may further include means for removably coupling to at least one operational source.The at least one operational source may use the set of nodes as a single entity.

[0018] The apparatus may also include means for modulating the number or type of nodes comprising the node set based on a demand for the received at least one operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The foregoing will become apparent from the following more particular description of example embodiments, as illustrated in the accompanying drawings, in which like reference numerals refer to the same parts throughout the different views. The drawings are not necessarily drawn to scale, emphasis instead being placed upon illustrating the embodiments.

[0020] Figure 1A A diagram of a set of computing host nodes embodying the prior art.

[0021] Figure 1B A block diagram of a set of host nodes connected to a near-memory computing node embodying the prior art.

[0022] Figure 2A An illustration of a set of host nodes connected to a chain of additional nodes in accordance with an example embodiment of the present invention.

[0023] Figure 2B A block diagram of a set of host nodes connected to additional nodes in accordance with an example embodiment of the present invention.

[0024] Figure 3A A block diagram of a set of host nodes connected to a near-memory computing or memory expander embodying the prior art.

[0025] Figure 3B A block diagram of an alternative system for a set of host nodes connected to near-memory computing, memory extenders, or both, for example embodiments of the present invention.

[0026] Figure 4 A diagram of a chip including a set of additional nodes capable of connecting to a host node, in accordance with an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0027] Following is a description of example embodiments.

[0028] The teachings of all patents, published applications, and references cited herein are incorporated by reference in their entirety.

[0029] Current computer (as used herein, "computer" includes any device or apparatus capable of reading and executing digital instructions) and server architectures are based on a combination of (one or more) graphics processing units (GPUs) and (one or more) central processing units (CPUs) to provide computing power ("compute") to perform tasks or operations. The GPU-CPU combination comprises a compute or compute node. The compute node interfaces with a memory such as a dynamic random access memory (DRAM) or other data storage device or component, and can read data into the memory and / or write data out of the memory during the execution of a task or operation. The compute provided by the compute node uses the connected memory provided by the DRAM, data storage device or other data storage component to perform received operations, tasks and instructions. These operations, tasks and instructions may be derived from and / or executed in response to received input, software programs, or caused by any other function of the execution computing device.

[0030] Traditionally, memory is provided by components directly incorporated into the same device, chip or equipment, which also includes computing nodes. Memory integrated in this way is called local memory, such as local DRAM. Due to space and power limitations, this single design can only include the amount of memory and computing provided by the components fitted on a single device or chip. In addition, the amount of memory and computing on a single device or chip can be set during manufacturing, or limited by a fixed number of component connections or port configurations, thereby limiting flexibility and adaptability. Due to its limitations, a single chip design is difficult to meet the growing needs of complex computer operations for memory and computing. Some developments in computer architecture and design, such as non-uniform memory access (NUMA) memory designs, have attempted to increase the capacity of local memory and computing, but they still cannot meet the growing memory and computing needs, especially those driven by the adaptation of generative AI and machine learning.

[0031] Due to the limitations of single or single chip design, designs utilizing multiple connected chips, devices, components or nodes have been developed. The computing and memory of the system can be increased by adding more computing nodes, but this will also increase the power consumption, cost and complexity of the system. However, in an embodiment of the present invention, it is not necessary to add a complete computing node with its own GPU, CPU and memory, but to add a node configured to provide only specific functions or resources required by the system. For example, if more memory is needed, a memory expander node can be used to provide additional data storage space without any corresponding computing power, thereby avoiding the addition of unnecessary computing components. Alternatively, if more computing is needed, a near memory computing (NMC) node can be used to provide additional computing power, such as by including an additional CPU, which can process data stored on the local memory of the NMC or on the connected node, but may not include other components such as GPU, because it does not require complete computing node functionality. These additional dedicated nodes (referred to herein as "additional nodes") can be connected to computing nodes (referred to herein as "host nodes" or "host computing nodes"), and their resources can be utilized by computing nodes. The connection can use a computer expresslink (CXL) or an equivalent interface with sufficient bandwidth built on PCI Express (PCIe) to enable high-speed operation and function. However, in existing system designs, the ability to connect additional nodes such as memory expanders and NMCs to the host computing node is limited by the ability of the interface or port that they are directly connected to (one or more) host computing nodes. For example, a host computing node with only three external input / output ports or interfaces (such as PCIe ports) will only be able to connect and utilize three additional nodes, or if some of these ports or interfaces are used to connect to other host computing nodes or devices, then it may be less. In addition, the host computing node must manage the communication of the additional nodes connected to it separately, increasing the complexity of task management and data fidelity operations. Some prior art designs also incorporate additional nodes on the same chip, board or device including the host computing node, which results in the ability to add additional nodes (and the resources they provide) being limited by the space and structure provided by the shared chip, board or device.

[0032] Figure 1A The present invention is an illustration of a computing host node set 100 embodying the prior art. Figure 1AThree host node sets 100 are shown, each including a compute 101, memory bandwidth 102, and memory load and store 103. The host node set 100 receives, originates from, and / or performs computer-operable tasks and instructions. The compute 101, such as a CPU-GPU combination or other similar computing components that can perform or execute computer-operable tasks and instructions, provides computing power. Each compute 101 utilizes memory bandwidth 102 to store, read, and / or write data to and from memory load and store 103. The memory load and store 103 can be local memory on the same chip / device of the compute 101, such as but not limited to DRAM. If the host node 100 is connected, then the compute 101 can utilize the memory bandwidth 102 and memory load and store 103 associated with other computes 101 or located on other host nodes. Each compute node in the compute node set 100 is a complete computing device in itself, and while they can serve as a supplemental resource to the memory provided by memory bandwidth 102 and memory load and store 103 and the compute 101 provided by compute components such as CPUs and GPUs, it would become extremely expensive and complex to continue to connect complete compute nodes to the set 100 to meet the ever-growing memory and compute needs.

[0033] Figure 1B A block diagram of a set of host nodes 100a, 100b connected to a near memory compute node 110 embodying the prior art. To address the compute and memory limitations of the compute nodes 100a, 100b, some prior art designs connect them to additional nodes, such as near memory compute (NMC) nodes 110. The additional nodes 110 need not have all of the components of a full compute node, and therefore provide a less resource intensive way to add additional compute or memory to the system when compared to adding additional full compute nodes 100a, 100b. In such a configuration, the compute nodes 100a, 100b are referred to as hosts, host nodes, or host compute nodes. The compute nodes 100a, 100b retain their compute components 101a, 101b and memory components 102a, 102b, 103a, 103b, which are located in Figure 1A Shown in and about Figure 1ADescribed. Computing nodes 100a, 100b can be connected using interface 104, which allows them to share or operate in parallel using other node components. Interface 104 can be a PCIe port, which utilizes CXL or other similar interfaces, devices or connections that can enable host-host communication. Host computing nodes 100a, 100b can be connected to additional nodes using interfaces 105a, 105b, such as near memory computing (NMC) nodes 110 that can provide additional memory, additional computing or both. Interfaces 105a, 105b can be PCIe ports, which utilize CXL or other similar interfaces that can host-node communication. However, in such prior art designs, even if additional nodes 110 are separated from host computing nodes 100a, 100b, it is often integrated into a single server, device or board, and cannot be easily removed, supplemented or modified. In addition, the amount of supplementary memory or computing that can be added is subject to the constraints of the space and configuration limitations of additional nodes 110.

[0034] The additional node 110 can provide additional computing using its own computing component 111 (such as a CPU, GPU or other similar components), and can use its own memory bandwidth 112 and memory loading and storage 113 and / or other similar memory components to provide additional memory. The host computing node 100a, 100b can access and use the additional computing and / or memory through the interface 105a, 105b, and enable parallel operation. The additional node 110 can also include hardware or software that can provide error protection to ensure data fidelity with the host computing node 100a, 100b. If the additional node is an NMC, it can store data locally on its memory components 112, 113, and use its computing component 111 to perform local operations based on the communication it receives from the host computing node 100a, 100b. The data stored on the memory components 112, 113 of the additional node 110 can be copied from the memory components 102a, 102b, 103a, 103b of the host computing node 100a, 100b. If the additional nodes are memory expanders, then the host compute nodes 100a, 100b and their compute components 101a, 101b can directly access and utilize data stored on their memory components 112, 113 without any local compute operations occurring on the node 110. However, because in prior art embodiments, the additional nodes 110 are directly connected to the host compute nodes 100a, 100b, the number of additional nodes 110 that can be added to the system, as well as any compute or memory provided by the node components 111, 112, 113, is limited by the number of available interfaces 115a, 115b.

[0035] Figure 2AA diagram of a host node set 200a, 200b, 200c connected to an additional node chain 210a, 210b, 210c, 210d of an example embodiment of the present invention. In order to address the limitations of the prior art, a design is disclosed herein, which includes multiple additional nodes 210a-210d connected in a peer-to-peer daisy chain topology, and enables linear computing and memory scalability, while only requiring a single connection with the host node 200a, 200b, 200c. Other connection topologies other than or in place of the daisy chain topology can be used to connect additional nodes 210a-210d in a peer-to-peer configuration. Additional nodes 210a-210d can be NMCs (for increasing computing power) or memory expanders (for increasing data storage and memory), and are connected to each other peer to peer to form a chain. The chain of additional nodes 210a-210d can then be connected to host computing nodes 200a-200c using a shared input-output interface through an additional node at the end of the chain 210a. Because the shared input-output interface can be connected to all of the additional nodes 210a-210d (directly or indirectly through a host-to-host connection, such as interface 104), the host computing nodes 200a-200c can view and process the chain of additional nodes 210a-210d as a single additional node and in a manner similar to Figure 1B The prior art designs shown take advantage of the computation and memory they provide. This enables the disclosed novel design to be incorporated with existing host computing nodes without significant hardware, software or communication protocol changes. However, compared to the prior art designs, the chain of nodes 210a-210d is scalable through unlimited connections of more additional nodes and can provide much more computing power or memory than a single additional node 110 can provide.

[0036] Figure 2AShown are three host nodes 200a, 200b, 200c sets, which include their own corresponding computing components 201 and storage components (not shown). Each host node 200a, 200b, 200c includes an interface 205, which can be used as direct memory access (DMA), and is convenient for the transmission and communication of data, requests and operations between host nodes 200a, 200b, 200c. Host nodes 200a, 200b, 200c can be connected to each other using interface 205 to share calculations and memory between them. Additional nodes 210a-210d are connected to each other in a daisy chain topology by peer interfaces or ports. Specifically, additional node 210a is connected to additional node 210b, additional node 210b is connected to additional node 210c, and additional node 210c is connected to additional node 210d. Any number of additional nodes can be included in the chain, and due to the sequential mode of connection, additional nodes 210a-210d only need two peer interfaces, regardless of the number of nodes connected. The peer interfaces of the additional nodes 210a-210d enable communication and transmission of requests, and operations between the additional nodes 210a-210d, similar to how the interface 205 allows communication and operations of requests between the host nodes 200a, 200b, 200c. The requests and operations may originate from the host nodes 200a, 200b, 200c, the additional nodes 210a-210d, or external devices, and utilize computations provided by the computing components 201, 211a-211d and / or memory provided by the memory components 211a-211d, 212a-212d of the additional nodes 210a-210d or the memory component host nodes 200a, 200b, 200c. Data may be replicated and transferred between the memory components 211a-211d, 212a-212d of the additional nodes 210a-210d and the memory component host nodes 200a, 200b, 200c as needed, and the host node 201 and the additional nodes 210a-210d may perform data fidelity operations to ensure that the data is not corrupted or incorrectly edited. Requests and operations may be performed in parallel or jointly by any combination of hosts 200a, 200b, 200c and additional nodes 210a-210d, and utilize any combination of computing and memory components located on the hosts 200a, 200b, 200c and / or additional nodes 210a-210d.

[0037] If the additional node 210a-210d is a near memory computing (NMC) node, then its own compute components 211a-211d, memory bandwidth 212a-212d, and memory stores and loads 213a-213d may be included. If the additional node 210a-210d is a memory expander node, to reduce complexity and cost, the local compute components 211a-211d may be excluded, and only memory components 211a-211d, 212a-212d may be provided. The additional node 210a-210d may use the compute components 211a-211d to perform operations using data stored on its corresponding local memory components 212a-212d, memory components 212a-212d located on other additional nodes 210a-210d, memory components located on the host node 200a, 200b, 200c, or any other memory components or data storage locations incorporated or connected to the system. The memory components 212a-212d of the additional nodes 210a-210d can be used to read data, write data, or otherwise be used for computing tasks originating from and / or performed by local computing components 211a-211d, computing components 211a-211d located on other additional nodes 210a-210d, computing components 201 of host nodes 200a, 200b, 200c, or other sources of computing instructions incorporated into or connected to the system.

[0038] The chain of additional nodes 210a-210d is connected to at least one of the host nodes 200a, 200b, 200c using interface 205. The chain of additional nodes 210a-210d may be connected to any number of the host nodes 200a, 200b, 200c directly through any of its component nodes, or indirectly through connections between the host nodes 200a, 200b, 200c. Figure 2A A set of chains of multiple additional nodes 210a-210d is shown, which provides a visual representation of how each host node 200a, 200b, 200c can view and interact with the chain of additional nodes 210a-210d independently of or in coordination with other host nodes 200a, 200b, 200c through interface 205. In addition, the host nodes 200a, 200b, 200c do not need to distinguish between the additional nodes 210a-210d and their computation 211a-211d and / or memory components 212a-212d, 213a-213d, but can treat them as a single source of additional computation and / or memory accessible through interface 205.

[0039] Figure 2BA block diagram of a set of host nodes 200a, 200b connected to a chain of additional nodes 210a-210e for an example embodiment of the present invention. The host nodes 200a, 200b include computing components 201a, 201, such as but not limited to a CPU and / or a GPU, and memory components 202a, 202b, 202a, 202b, such as but not limited to DRAM and components necessary to access and utilize data stored on DRAM or other data storage devices. The host nodes 200a, 200b are connected using interface 205a, which can be a PCIe port utilizing CXL or an equivalent interface capable of direct memory access (DMA) and / or host-to-host connection. The host nodes are connected to the chain of additional nodes 210a-210e using interfaces 205b, 205c. Interfaces 205b, 205c can be PCIe ports that utilize CXL or an equivalent interface capable of direct memory access (DMA) and / or node-to-node connection.

[0040] With reference to additional node 210a as an example, additional node 210a-210e may include computing component 211a and memory component, such as memory bandwidth 212a and memory storage and loading 212b, which can make host node 200a, 200b supply supplementary computing and / or memory available. Memory component, computing component and multiple additional nodes 210a-210e can be selected to provide memory and computing corresponding to the specific number of needs of host 200a, 200b. Additional nodes 210a-210e are connected with peer interfaces 215a-215d in a daisy chain topology. Additional nodes 210a-210e may include multiple peer interfaces 215a-215d to allow additional nodes 210a-210e to be connected with any desired topology or structure. Interface 215a-215d can be a PCIe port, which utilizes CXL or an equivalent interface capable of direct memory access (DMA) and / or peer node connection. Interfaces 215a-215d enable communication between additional nodes 210a-210e and indirect communication from host nodes 200a, 200b to specific additional nodes 210a-210e through other nodes. As a non-limiting example, host node 200a can send a communication to additional node 211a through interface 205b to access additional node 210b and utilize the computation and / or memory provided by its components, and additional node 211a transmits the communication to additional node 211b through peer interface 215a. If desired, the CXL protocol can be used to minimize the latency bottleneck caused by transmission through interfaces 205a-c, 215a-215d.

[0041] Although Figure 2BTwo host nodes 200a, 200b and five additional nodes 210a-210e are shown, but it should be clear to those skilled in the art that any number of host nodes or additional nodes can be connected in a similar configuration. It should also be understood by those skilled in the art that additional interfaces can be added to additional nodes 210a-210e so that more additional node chains can be connected. Additional nodes 210a-210e can be located on separate chips, boards, devices or components different from host nodes 201a, 201b, and can be removably coupled to host nodes 201a, 201b using interfaces 205b, 205c. This modular capability enables the variable needs based on host nodes 200a, 200b or other host devices or components to add, modify or remove additional calculations and / or memory provided by additional nodes 210a-210e.

[0042] The chain of additional nodes 210a-210e can be designed and constructed to fit the specific needs of the host node 200a, 200b or other host devices or components to which it can be connected. The chain of additional nodes 210a-210e can also be constructed to provide the needs of specific computer-implemented applications and tools utilized by the host nodes 200a, 200b that require a lot of computing and memory, such as generative AI and machine learning. Each node 210a-210e can be a near memory computer (NMC) that provides additional computing, or a memory expander that provides additional memory. The specific internal components of a node 210a-210e, for example, if it contains a computing component 211a, a memory component 212a, 213a, other components that supplement the functionality of the host node 200a, 200b, or any combination of the foregoing components, do not interfere with the ability to connect additional nodes 210a-210e using peer interfaces 215a-215d and to connect a chain of additional nodes 210a-210e to the host node through shared input-output interfaces 205b, 205c. Thus, a chain of additional nodes 210a-210e may be constructed or modified, including a set of nodes that provide a precise amount of additional computing, storage, or other functionality to the host node 200a, 200b. In addition, changing the composition of the additional node chain 210a-210e does not require any changes to the host nodes 200a, 200b or their local components, because they can be configured to view, communicate and interact with the additional node chain 210a-210e as a single system, and utilize the components of the chain of additional nodes 210a-210e without distinguishing which additional node 210a-210e they are located on. If the host nodes 200a, 200b change their needs for computing and memory, the disclosed design can also allow disconnection and connection of different additional node 210a-210e chains to the host nodes 200a, 200b. In summary, the modularity, flexibility and scalability of the disclosed design and topology of the additional nodes 210a-210e provide significant improvements to the prior art designs.

[0043] Figure 3A 1 is a block diagram of a set of host nodes 300a, 300b connected to a near memory computing or memory expander 310 embodying the prior art. The host nodes 300a, 300b and the additional nodes 310 include Figure 1B The host nodes 300a, 300b are connected using interfaces 104, 305a, and are connected to the additional node 310 using input and output interfaces 105a, 305b, 105b, 305c. Figure 3AAs shown, without the novel peer-to-peer capabilities disclosed herein, the system is limited to a single predetermined additional node 310 due to the lack of additional input-output interfaces 305a, 305b of the host nodes 300a, 300b. Therefore, even if the host nodes 300a, 300b require both additional computation and memory, the additional node 310 can only be used as a near memory computing (NMC) or memory expander, and thus cannot meet the needs of the host nodes 300a, 300b. In addition, if the needs of the hosts 300a, 300b change, there is no simple way to add additional computation or memory to the system or reallocate the functionality of the additional node 310.

[0044] Figure 3B 1 is a block diagram of an alternative system 320a, 320b, 320c of a set of host nodes 300a, 300b connected to near memory computing, memory expanders, or both 310a-310e for an example embodiment of the present invention. The host nodes 300a, 300b and the additional nodes 310a-310e include Figure 2B 1 and 12. The internal components shown and described with respect to the additional nodes 310a-310e may have any combination of compute components 111, memory components 112, 113, or additional components to enable them to be used as near memory computing, memory expanders, or both. The host nodes 300a, 300b are connected using interfaces 205a, 305a and are connected to the additional node 310 using input and output interfaces 205b, 305b, 205c, 305c. The additional nodes 310a-310e are connected using peer interfaces 215a-215d, 315a-315d. The interfaces 305a, 305b, 305c, 315a-315d may be PCIe Express ports that enable CXL connections. Alternatively, interfaces 305a, 305b, 305c, 315a-315d may be any type of port, connection, architecture, or device that enables inter-device or inter-chip or node-to-node communication and utilizes any compatible communication standard or protocol. Figure 3ACompared to the prior art system shown in , the systems 320a, 320b, 320c can adapt and customize the needs of the hosts 300a, 300b. If the hosts 300a, 300b need more computing, then the system 320a in which the additional nodes 310a-310e are all NMCs can be utilized. Alternatively, if the hosts 300a, 300b need more memory, then the system 320c in which the additional nodes 310a-310e are all memory expanders can be utilized. If both memory and computing are needed, then the system 320b in which the additional nodes 310a-310e include a mixture of NMCs and memory expanders can be used. The chain of additional nodes 310a-310e of the system 320a, 320b, 320c can be configured to be removably coupled to the host 300a, 300b using the input-output interface 305a, 305b, so that when the needs of the host 300a, 300b change, they can connect to the system 320a-320c and / or disconnect from the system 320a-320c with different combinations of NMCs and memory expanders. Similarly, peer interfaces 315a-315d can also be removably coupled to the additional nodes 310a-310e to allow the removal, addition or exchange of individual additional nodes 310a-310e. This ability to "mix and match" NMCs and memory expanders in the chain of additional nodes 310a-310e ensures that the host 300a, 300b always has the optimal amount of additional computing and / or memory available to them, while avoiding the generation of inefficient excess computing and / or memory or bottlenecks.

[0045] The systems 320a, 320b, 320c provide a further improvement over prior art systems that limit the number of additional nodes 310a-310e that can be connected to a host 300a, 300b based on the available number of direct connections provided by the input-output interfaces 305b, 305c. The systems 320a, 320b, 320c are configured to allow additional nodes 310a-310e to connect to each other using peer interfaces, which enables a theoretically unlimited number of NMCs or memory expanders to be connected to the host nodes 300a, 300b (through a chain of peer interfaces 315a-315d of the nodes 310a-310e and the ultimate input-output interfaces 305b, 305c). This enables the systems 320a, 320b, 320c to provide scalable amounts of memory and computation to the hosts 300a, 300b. The scalability and mix and match capabilities of the node chain 310a-310e do not require any changes to the hosts 300a, 300b, as they continue to view and interact with the node chain 310a-310e as a single NMC and / or memory expander. The chain of nodes 310a-310e can be located on a separate device or chip from the hosts 300a, 300b, and changing between systems 320a, 320b, 320c (or any other system with any desired combination of peer-connected NMCs or memory expanders) can be as simple as unplugging and switching connections to the input-output interfaces 305b, 305c.

[0046] If desired, the systems 320a, 320b, 320c can use a heat tracking method to classify data stored in the memory provided by the node chain 310a, such as "hot", "warm" and "cold", including allocating different memory areas on the chain of nodes 310a-310e based on the proximity to the host 300a, 300b. Although increasing the number of additional nodes in the chain including nodes 310a-310e can result in additional latency, due to the increase in communication length through the peer interface 315a-315d, in many modern computing applications, minimizing latency is not as important as the ability to scale computing power and memory size. Specifically, the ability to enable parallel operation on the host 300a, 300b and the additional nodes 310a-310e provides tremendous benefits to many modern computing technologies and applications. For example, but not limited to machine learning, generative AI, electronic design automation (EDA) tools, and predictive analysis. Physical EDA tools are incorporating AI into fine-tuning settings in placement and routing tools to achieve better power, performance, and area, and to help determine the root cause of vulnerabilities. These EDA tools are bottlenecked by their large memory requirements, and the systems 320a, 320b, 320c can be used to overcome these bottlenecks by making large memory expander chains available to the hosts 300a, 300b running the EDA tools, or by providing AI-based assistance. Given the benefits of being able to overcome memory (or compute) bottlenecks, any additional latency is intolerable.

[0047] Figure 4A diagram of a chip 400 including an additional node set 411a-h that can be connected to a host node for an exemplary embodiment of the present invention. The chip 400 includes an additional node set 411a-h connected by peer interfaces 415a-g. The peer interfaces 415a-g can be PCIe ports utilizing CXL. The peer interfaces 415a-g can also be any other component capable of establishing a peer connection (permanent and removable / reconfigurable) between the additional nodes 411a-h, including but not limited to components capable of direct memory access. Referring to the additional node 410a as an example, the additional nodes 410a-h can include a computing component 411 such as a CPU, a GPU, or both and a memory component 412, 413 such as a DRAM and / or a component connected and utilizing a DRAM or other memory storage device. The chip 400 can include an area 401, which includes components that can be utilized by the chip 400 and the additional nodes 411a-h, such as but not limited to fans, power supplies, and / or baseboard management controllers. The chip 400 includes external interfaces 405a, 405b, which are configured to connect to a host computing device, for example, through input-output interfaces 205b, 205c, 305b, 305c, so that the host computing device can access and use the computing and / or memory provided by the additional nodes 410a-h. The external interface 405b can also be configured to connect to other chips 400 to expand the chain of additional nodes 410a-h or other computing components, chips or devices containing more additional nodes or host nodes. The chip 400 can be located on a removable board or component separate from the host computing device or a server containing the host computing device.

[0048] Additional nodes 410a-h may be any combination of memory expanders or NMCs required to provide the required amount of compute and / or memory. An embodiment of chip 400 utilizing sixteen additional nodes 410 may provide access to up to 32TB of additional memory or up to 256 ARM V2 processors provided by any desired combination of NMCs or memory expanders. In contrast, prior art designs incorporated into solutions on servers or on chips may only provide up to 2TB or 16 ARM V2 processors with dedicated, non-configurable or reselectable memory expanders or NMCs.

[0049] While example embodiments have been particularly shown and described above, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the scope of the embodiments as encompassed by the appended claims.

Claims

1. A device comprising: An I / O interface configured to receive at least one operation; as well as A node set is configured to perform the at least one operation received, each node in the node set comprising: at least one peer interface configured to connect to another node in the set of nodes and communicate the received at least one operation; as well as A local storage is configured to store data. 2 . The apparatus of claim 1 , wherein at least one node in the set of nodes is a memory expander configured to provide additional memory for performing the received at least one operation. 3 . The apparatus of claim 1 , wherein at least one node in the set of nodes is a near memory computing (NMC) configured to provide additional computing power for performing the received at least one operation. 4 . The apparatus of claim 1 , wherein the set of nodes is further configured to perform the received at least one operation in parallel.

5. The apparatus of claim 1, wherein the set of nodes are connected in a daisy chain topology via the peer interface. The apparatus of claim 1 , wherein the at least one operation originates from at least one external host.

7. The apparatus of claim 6, wherein the at least one external host uses the set of nodes as a single entity.

8. The apparatus of claim 1, wherein at least one of the I / O interfaces and the at least one peer interface are PCI Express (PCIe) interfaces.

9. The apparatus of claim 8, wherein the PCI Express (PCIe) interface utilizes a Compute Express (CXL) protocol.

10. The apparatus of claim 1, wherein the node assembly is modular, and the number of nodes and the types of nodes comprising the node assembly are variable.

11. The apparatus of claim 1, wherein the I / O interface is configured to removably couple the node set to at least one host computing node.

12. The apparatus of claim 11, wherein the at least one computing node is located on a different chip than the set of nodes.

13. The apparatus of claim 1, wherein the one peer interface is further configured to removably couple the set of nodes.

14. The apparatus of claim 1, wherein the set of nodes jointly performs the received at least one operation.

15. A method of performing a computing operation, the method comprising: receiving at least one operation from the I / O interface; as well as The at least one operation received is performed by a set of nodes, each node in the set of nodes comprising: at least one peer interface configured to connect to another node in the set of nodes and communicate the received at least one operation; as well as A local storage is configured to store data.

16. The method of claim 15, wherein at least one node in the set of nodes is a memory expander configured to provide additional memory for performing the received at least one operation. 17 . The method of claim 15 , wherein at least one node in the set of nodes is a near memory computing (NMC) configured to provide additional computing power for performing the received at least one operation.

18. The method of claim 1, wherein the set of nodes are connected in a daisy chain topology via the peer interface.

19. The method of claim 1, further comprising sending the at least one operation from a host computing node.

20. The method of claim 19, wherein the host computing node is located on a server external to the set of nodes.

21. The method of claim 19, further comprising removably coupling the node assembly to the host computing node using the I / O interface.

22. The method of claim 19, wherein the host computing node uses the set of nodes as a single entity.

23. The method of claim 15, wherein at least one of the I / O interfaces and the at least one peer interface are PCI Express (PCIe) interfaces.

24. The method of claim 15, further comprising: Based on the received demand for the at least one operation, the number of nodes or the type of nodes comprising the set of nodes is modulated.

25. An apparatus for performing a computing operation, the apparatus comprising: means for receiving at least one operation; as well as Means for performing, by a set of nodes, the at least one operation received, each node in the set of nodes comprising means for connecting to another node in the set of nodes and transmitting the at least one operation received.

26. The apparatus of claim 25, wherein at least one of the nodes in the set of nodes is a memory expander, the memory expander comprising means for providing additional memory to perform the received at least one operation.

27. The apparatus of claim 25, wherein at least one node in the set of nodes is a near memory computing (NMC), the near memory computing (NMC) comprising means for providing additional computing power for performing the received at least one operation.

28. The apparatus of claim 25, further comprising means for removably coupling to a source of the at least one operation.

29. The apparatus of claim 28, wherein the source of the at least one operation uses the set of nodes as a single entity.

30. The apparatus of claim 25, further comprising: Means for modulating the number of nodes or node types comprising the node set based on demand for the received at least one operation.

Images