Memory computing system and related method

Abstract

Various examples relate to systems and methods for loop execution in a reconfigurable computing structure. A first flow controller may start a first thread at a first synchronization flow to execute a first portion of a first iteration of the loop. A second flow controller may receive a first asynchronous message instructing the second flow controller to start a first thread at a second synchronization flow to execute a second portion of the first iteration. The second flow controller may determine that the first iteration of the loop is the last iteration of the loop to be executed, and start the first thread at the second synchronization flow if a last iteration flag is set.

Description

[0001] Priority application

[0002] This application claims priority to U.S. Provisional Application Serial No. 63 / 132,779 filed on December 31, 2020 (which is incorporated herein by reference in its entirety) and U.S. Application Serial No. 17 / 405,371 filed on August 18, 2021 (which is incorporated herein by reference in its entirety). Technical Field

[0003] The present application relates generally to computing architectures, and more particularly to loop execution in reconfigurable computing structures. Background Art

[0004] Various computer architectures (e.g., von Neumann architecture) typically use shared memory for data, buses for accessing the shared memory, arithmetic units, and program control units. However, moving data between the processor and memory can require a significant amount of time and effort, which in turn can limit the performance and capacity of the computer system. Given these limitations, new computing architectures and devices are needed to improve computing performance beyond the practice of transistor scaling (i.e., Moore's Law). Summary of the invention

[0005] According to one aspect of the present application, a method is provided. The method includes: starting a first thread at a first synchronous stream by a first stream controller of a first synchronous stream, the first thread at the first synchronous stream executing a first part of a first iteration of a loop, the loop having multiple iterations, the first synchronous stream being configured by a reconfigurable computing structure including a set of multiple stream controllers and a set of multiple computing elements; receiving a first asynchronous message by a second stream controller of a second synchronous stream, the first asynchronous message instructing the second stream controller to start the first thread at the second synchronous stream to execute a second part of the first iteration, the second part of the first iteration being executed after the first part of the first iteration; determining by the second stream controller that the first iteration of the loop is the last iteration of the loop to be executed; and in response to determining that the first iteration of the loop is the last iteration of the loop to be executed, starting the first thread at the second synchronous stream by the second stream controller, the starting including providing a first synchronization message to a first computing element of the second synchronous stream, the first synchronization message including a last iteration flag indicating that the first iteration is the last iteration of the loop to be executed.

[0006] According to another aspect of the present application, a reconfigurable computing structure is provided. The reconfigurable computing structure includes: a first synchronization stream, the first synchronization stream includes a first flow controller and at least one computing element, the first flow controller is programmed to start a first thread at the first synchronization stream to execute a first part of a first iteration of a loop, the loop having multiple iterations; a second synchronization stream, the second synchronization stream includes a second flow controller and at least one computing element, the second flow controller is programmed to perform operations including: receiving a first asynchronous message, the first asynchronous message instructing the second flow controller to start a first thread at the second synchronization stream to execute a second part of the first iteration, the second part of the first iteration being executed after the first part of the first iteration; and determining that the first iteration of the loop is the last iteration of the loop to be executed; and in response to determining that the first iteration of the loop is the last iteration of the loop to be executed, starting the first thread at the second synchronization stream, the starting including providing a first synchronization message to a first computing unit of the second synchronization stream, the first synchronization message including a last iteration flag indicating that the first iteration is the last iteration of the loop to be executed.

[0007] According to yet another aspect of the present application, a machine-readable medium is provided. The machine-readable medium includes instructions stored thereon, which when executed at a reconfigurable computing structure cause the reconfigurable computing structure to perform operations including: starting, by a first flow controller of a first synchronization flow, a first thread at the first synchronization flow, the first thread at the first synchronization flow executing a first portion of a first iteration of a loop, the loop having a plurality of iterations, the first synchronization flow being configured by a reconfigurable computing structure including a set of a plurality of flow controllers and a set of a plurality of computing elements; receiving, by a second flow controller of a second synchronization flow, a first asynchronous message instructing the second flow controller to start the first thread at the second synchronization flow to execute a second portion of the first iteration, the second portion of the first iteration being executed after the first portion of the first iteration; determining, by the second flow controller, that the first iteration of the loop is a last iteration of the loop to be executed; and in response to determining that the first iteration of the loop is the last iteration of the loop to be executed, starting, by the second flow controller, the first thread at the second synchronization flow, the starting including providing a first synchronization message to a first computing element of the second synchronization flow, the first synchronization message including a last iteration flag indicating that the first iteration is the last iteration of the loop to be executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] To easily identify the discussion of any particular element or act, the most significant digit(s) in a reference number refers to the figure number in which the element is first introduced.

[0009] Figure 1 A first example of a first memory computing device in the context of a memory computing system according to an embodiment is generally shown.

[0010] Figure 2 An example of a memory subsystem of a memory computing device according to an embodiment is generally shown.

[0011] Figure 3 An example of a programmable atomic cell for a memory controller according to an embodiment is generally shown.

[0012] Figure 4 An example of a hybrid thread processor (HTP) accelerator of a memory computing device according to an embodiment is shown.

[0013] Figure 5 An example of a representation of a hybrid thread structure (HTF) of a memory computing device according to an embodiment is shown.

[0014] Fig. 6A An example of a chiplet system according to an embodiment is generally shown.

[0015] Figure 6B Overall, the diagram shows the Fig. 6A A block diagram of the various components in an example chiplet system.

[0016] Figure 7 An example of a chiplet-based implementation for a memory computing device according to an embodiment is generally shown.

[0017] Figure 8 An exemplary tiling of chiplets of a memory computing device according to an embodiment is shown.

[0018] Fig. 9 is a diagram illustrating an example of a workflow for performing operations on a reconfigurable computing fabric using more than one synchronization stream.

[0019] Fig.10 is a diagram illustrating one example of a workflow for performing operations at a reconfigurable computing fabric, where synchronization flows interact with a memory interface.

[0020] Fig.11 is a diagram illustrating an example of a workflow for executing a loop using a reconfigurable computing structure.

[0021] Fig.12is a flow chart illustrating one example of a process flow that may be implemented using a reconfigurable computing structure to execute a loop using a last iteration flag.

[0022] Fig.13 is a diagram illustrating an example of a workflow for executing a loop with a reconfigurable computing structure using a last iteration flag.

[0023] Fig.14 A block diagram of an exemplary machine is shown with, in, or by which any one or more of the techniques (eg, methodologies) discussed herein may be implemented. DETAILED DESCRIPTION

[0024] The latest advances in materials, devices, and integration techniques can be used to provide memory-centric computing topologies. Such topologies can achieve improvements in computing efficiency and workload throughput, for example, for applications limited by size, weight, or power requirements. The topology can be used to promote low-latency computing near or inside a memory or other data storage element. These methods can be particularly suitable for various computationally intensive operations with sparse lookups, such as in transform calculations (e.g., fast Fourier transform calculations (FFT)), or in applications such as neural networks or artificial intelligence (AI), financial analysis, or simulation or modeling such as for computational fluid dynamics (CFD), engineer enhanced acoustic simulator (EASE), simulation program with integrated circuit emphasis (SPICE), etc.

[0025] The systems, devices and methods discussed herein may include or use a memory computing system with a processor or processing power, which is provided in, near or integrated with a memory or data storage component. Such systems are generally referred to as neighboring memory computing (CNM) systems herein. A CNM system may be a node-based system in which individual nodes in the system are coupled using a system-level structure. Each node may include or use a dedicated or general-purpose processor and a user-accessible accelerator, with a customized computing structure for intensive operations, particularly in environments where a higher cache miss rate is expected.

[0026] In an example, each node in the CNM system can have one or more host processors. In each node, a dedicated hybrid thread processor can occupy discrete endpoints of the network on chip. The hybrid thread processor can access some or all of the memories in a specific node of the system, or the hybrid thread processor can access the network memory across multiple nodes via a system-level structure. The custom computing structure or hybrid thread structure on each node can have its own processor or accelerator, and can operate at a higher bandwidth than the hybrid thread processor. Different nodes in the adjacent memory computing system can be configured differently, for example, with different computing capabilities, different types of memories, different interfaces or other differences. However, these nodes can be coupled together to share data and computing resources in a defined address space.

[0027] In an example, a neighboring memory computing system or a node in the system can be configured by a user for custom operations. The user can provide instructions using a high-level programming language (such as C / C++), which can be compiled and directly mapped to the data flow architecture of the system, or the data flow architecture of one or more nodes in the CNM system. In other words, the nodes in the system can include hardware blocks (e.g., memory controllers, atomic units, other customer accelerators, etc.) that can be configured to directly implement or support user instructions, thereby enhancing system performance and reducing latency.

[0028] In an example, the adjacent memory computing system can be particularly suitable for implementing a hierarchy of instructions and nested loops (e.g., two, three or more loops, deep loops or multi-dimensional loops). A standard compiler can be used to accept high-level language instructions, and then directly compile into the data flow architecture of one or more nodes. For example, a node in the system can include a mixed thread structure accelerator. The mixed thread structure accelerator can be executed in the user space of the CNM system, and can start its own thread or sub-thread, which can operate in parallel. Each thread can be mapped to different loop iterations, thereby supporting multi-dimensional loops. In addition to other capabilities, by starting the ability of this nested loop, the CNM system can achieve significant time savings and latency improvements for computationally intensive operations.

[0029] In some examples, the near-memory computing system is programmed to arrange components of a reconfigurable computing structure (such as the various HTFs described herein) into one or more synchronized flows. The reconfigurable computing structure includes one or more hardware flow controllers and one or more hardware computing elements, which can be arranged to form one or more synchronized flows, as described herein.

[0030] The computing element includes a computing element memory and a processor or other suitable logic circuitry that forms a computing pipeline for processing received data. In some examples, the computing element includes multiple parallel processing channels, such as single instruction multiple data (SIMD) processing channels. As described herein, the computing element also includes circuitry for sending and receiving synchronous and asynchronous messages to the flow controller, other computing elements, and other system components. Reference is made to Figure 5 Blocks 504 , 510 , 512 of EMBODIMENTS depict exemplary computing elements.

[0031] As described herein, the flow controller includes a processor or other logic circuitry for managing synchronous flows. As described herein, the flow controller includes circuitry for sending synchronous messages and asynchronous messages to computing elements, other flow controllers, and other system components. In some examples, the flow controller is implemented using a tile basis of one or more of the tiles 504, 510, 512 described herein.

[0032] A synchronization flow is a hardware arrangement in a reconfigurable computing structure that includes a hardware flow controller and an ordered synchronization data path between a set of one or more hardware computing elements. A synchronization flow can execute one or more work threads. To execute a thread, the hardware components of the synchronization flow pass synchronization messages and perform a set of predetermined operations in the order of the synchronization flow.

[0033] A flow controller of a synchronization stream starts a thread at a synchronization stream by providing a first synchronization message to a first computing element of the synchronization stream. The first synchronization message includes data (e.g., data processed by the computing element) and may also include control information that provides various flags and other configuration and / or instruction data to the computing element. The first computing element is programmed to perform one or more operations, for example, based on the data provided by the first synchronization message. The first computing element generates a second synchronization message, which may also include data and control information. For example, the second synchronization message may describe the results of one or more operations performed by the first computing element.

[0034] The first computing element provides the second synchronization message to the next computing element according to the order of the synchronization flow. The next computing element of the synchronization flow can be another computing element of the reconfigurable computing structure, although in some instances, a single computing element can perform continuous operations of the synchronization flow, which means that in some arrangements, the computing elements of the synchronization flow can direct the synchronization message to themselves. The next computing element is programmed to perform one or more operations, which can include preparing a third synchronization message and sending the third synchronization message to the subsequent computing element.

[0035] A thread is completed when all of the computational elements of the synchronization flow have completed their programmed operations in the predetermined order of the synchronization flow. When a thread has completed, a pipeline of synchronization messages will propagate between the various computational elements starting from the flow controller in the predetermined order of the synchronization flow. Because the arrangement is synchronous, completion of a thread can occur in a fixed amount of time (e.g., a predictable number of clock cycles from when the flow controller initiates the synchronization flow).

[0036] Arranging the HTF to include synchronization streams may facilitate parallel processing. For example, a flow controller of a synchronization stream does not need to wait for one thread to complete before launching additional threads. Consider an exemplary synchronization stream including a flow controller and multiple computing elements. The flow controller starts a first thread by providing a synchronization message to a first computing element of the synchronization stream. The first computing element performs its processing and directs a second synchronization message to the next computing element, and so on. After the first computing element completes its processing and directs the synchronization message to the next computing element, the flow controller can start additional threads at the synchronization stream, for example, by providing additional synchronization messages to the first computing element.

[0037] Additional parallelization of synchronization flows at a reconfigurable computing structure can be obtained by utilizing computing elements that operate at a predetermined cadence or branch count (such as the various tiles described herein). For example, a computing element can use a predetermined number of clock cycles to perform various operations, such as receiving a synchronization message, performing a processing operation, sending a synchronization message, etc. A computing element can be configured to receive a new synchronization message and begin operations on a thread while operations from a previous thread are still propagating through the computing element. The new thread can be a different thread of the same synchronization flow as the previous thread, or can be a thread of a different synchronization flow.

[0038] Synchronous streams can use the asynchronous structure of the reconfigurable computing structure to use asynchronous messages to communicate with other synchronous streams and / or other components of the reconfigurable computing structure. For example, a stream controller can receive an asynchronous message from a scheduling interface and / or from another stream controller, and the asynchronous message indicates that the stream controller is to start a thread at the synchronous stream. The scheduling interface can interface between the reconfigurable computing structure and other system components. In addition, in some instances, a synchronous stream can send an asynchronous message to the scheduling interface to indicate the completion of a thread.

[0039] Asynchronous messages can also be used by various synchronization streams to access memory. For example, a reconfigurable computing structure may include one or more memory interfaces. A memory interface is a hardware component or its component used by a synchronization stream for accessing an external memory that does not belong to a synchronization stream. A thread executed at a synchronization stream may include sending a read and / or write request to a memory interface. Because reading and writing are asynchronous, a thread that initiates a read or write request to a memory interface may not receive the result of the request. Instead, the result of the read or write request may be provided to different threads executed at different synchronization streams.

[0040] Consider an exemplary reconfigurable computing structure, which is arranged with a first synchronous flow for initiating a read request and a second synchronous flow for receiving the result of the read request. The first thread at the first synchronous flow sends an asynchronous read request message to the memory interface. The first thread can also send an asynchronous continuation type message to the flow controller of the second synchronous flow, where the continuation message indicates the read request. The memory interface obtains the requested data from the memory and directs the read data to the appropriate computing element of the second synchronous flow. The computing element then directs the asynchronous message to the second flow controller, indicating that the data has been received. (In some instances, the memory interface directly provides the read data to the second flow controller). After receiving an indication that the read data has been received, the second flow controller starts the thread at the second synchronous flow to further process the result of the read request.

[0041] A near-memory computing system or a node or component of a near-memory computing system may include or use various memory devices, controllers, and interconnects, among others. In an instance, the system may include various interconnected nodes, and the nodes or groups of nodes may be implemented using chiplets. Chiplets are an emerging technology for integrating various processing functions. Typically, a chiplet system consists of discrete chips (e.g., integrated circuits (ICs) on different substrates or dies) integrated on an interposer and packaged together. This arrangement is different from a single chip (e.g., IC) containing different device blocks (e.g., intellectual property (IP) blocks) on one substrate (e.g., a single die), such as a system on a chip (SoC), or a discrete packaged device integrated on a board. In general, chiplets offer more production benefits than single-die chips, including higher yields or reduced development costs. The following discussion Fig. 6A and Figure 6B An example of a chiplet system as may include a near-memory computing system is generally shown.

[0042] In some examples, a reconfigurable computing structure of an HFT as described herein is used to execute one or more loops, such as a set of nested loops. As described herein, to execute the loop, the reconfigurable computing structure utilizes a flow controller and computing elements arranged into one or more synchronous streams. For example, a flow controller of a synchronous stream can start a thread at the synchronous stream for each iteration of the loop. Consider a simple exemplary loop given by the following code segment [1]: [1]

[0044] For i=1,10{

[0045] x[i]=x[i-1]*2;

[0046] }

[0047] The flow controller can start an exemplary loop by starting the first thread at the synchronization stream for the i=1 loop iteration. In this example, the initial value of x[i-1] is passed to the first computing element by the flow controller along with the payload data of the initial synchronization message. One or more computing elements of the synchronization stream determine the value of x[1] and return the value of x[1] to the flow controller as a synchronization message or an asynchronous message. The flow controller then starts the second thread at the synchronization stream for the i=2 loop iteration and passes the return value of x[1] as x[i-1] in the synchronization message. This process continues until all iterations of the loop are completed and the value of x

[10] is returned.

[0048] The example loop above uses a single synchronization stream for each iteration of the loop. However, in some instances, multiple synchronization streams may be used for each loop iteration. Consider the example loop given by the following code segment [2]:

[0049]

[0050] In this example, each loop iteration involves multiplying i by the value y[i] read from the memory and then writing the result to the memory. Therefore, each loop iteration includes an asynchronous memory read and an asynchronous memory write. As described herein, a memory read involves sending an asynchronous message to a memory interface and then waiting for the memory interface to reply with another asynchronous message including the requested data. Because the memory read is asynchronous, each loop iteration may use threads executing at two different synchronization streams. For the i=1 loop iteration, the thread at the first synchronization stream sends an asynchronous message to the memory interface, which includes a read request for the value of y[1]. The thread at the first synchronization stream may also send an asynchronous message to the second flow controller of the second synchronization stream, which indicates that the second flow controller expects the result of the read request (directly from the memory interface or from the computing element of the second synchronization stream that has received the read data). The memory interface initiates the reading of the value of y[1] and provides the value of y[1] to the second synchronization stream via an asynchronous message. When an asynchronous message indicating that the read data has been received is received, the second flow controller starts the thread at the second synchronization stream. (The return value of y[1] may be provided to the computation unit, for example, via synchronous communication of the threads and / or directly from the memory interface before the threads are started.) The second thread determines the value of x[1] and sends a synchronization message to the memory interface that includes a write request for x[1].

[0051] In loops that use more than one synchronization stream per loop iteration, it can be difficult, and sometimes impossible, to predict the order in which the loop iterations will complete. Consider again the exemplary loop above. A first flow controller starts a first thread at a first synchronization stream for the i=1 iteration of the loop. After waiting a predetermined number of clock cycles (e.g., 4 cycles), the first flow controller starts a second thread at the first synchronization stream for the i=2 iteration of the loop. After again waiting a predetermined number of clock cycles, the first flow controller starts a third thread at the first synchronization stream for the i=3 iteration of the loop, and so on.

[0052] However, as explained above, each iteration of the loop includes an asynchronous operation (in this instance, a read request). Even if the read requests are provided to the memory interface by various threads in the order of the loop iterations (e.g., first i=1, then i=2, and so on), the system may not return the requested data to the second synchronous stream in the same order. For example, the second flow controller may receive an asynchronous message indicating that the i=10 iteration value (y

[10] ) has been received before receiving an asynchronous message indicating that the i=9 iteration value (y[9]) has been received. If this occurs, the second flow controller may start a thread at the second synchronous stream using the i=10 iteration value y

[10] . After the second synchronous stream subsequently receives the i=9 iteration value y[9], the second flow controller starts another thread at the second synchronous stream using y[9].

[0053] Sometimes, asynchronous or out-of-order completion of loop iterations is not a problem. However, in some instances, one or more operations in one loop iteration depend on operations performed in one or more previous loop iterations. Therefore, if the loop iterations are executed out of order, incorrect results may be returned. Consider the example loop given by the following code segment [3]:

[0054]

[0055] In this example, each loop iteration loads a data value (data[i]) from memory and adds the data value to a local accumulation variable (localAccum). In the last loop iteration (i=10), the local accumulation variable is loaded into memory and then cleared.

[0056] When implemented using a reconfigurable computing structure (such as the HTF described herein), this example can use two synchronization streams for each loop iteration. For each loop iteration, the first thread at the first synchronization stream sends an asynchronous read request to the memory interface to load data [i], and also sends an asynchronous message to the second flow controller of the second domain, the asynchronous message indicating that the second synchronization stream expects the data [i] value from the memory interface. After the second flow controller receives the asynchronous message indicating that the data [i] value is received, it starts the thread at the second synchronization stream. The thread at the second synchronization stream updates the local accumulation variable, and if i=10, sends an asynchronous message to the memory interface to save and clear the local accumulation variable.

[0057] However, as can be seen from this example, if the second synchronous flow thread of the i=10 loop iteration is not the last thread to start at the second synchronous flow, the loop will not execute correctly. For example, if the second synchronous flow thread of the i=10 loop iteration is started at the second synchronous flow before the thread of the i=9 loop iteration, the value of the local accumulation variable loaded into the memory will be incorrect because it will not take into account the data [9] value considered in the i=9 loop iteration that has not yet completed.

[0058] There are various approaches to this problem, but many introduce their own additional problems. Loops (such as the example above) can be implemented in a serialized manner. In a serialized loop execution, the second loop iteration does not begin until the first loop iteration is completed. The third loop iteration does not begin until the second loop iteration is completed, and so on. However, this approach can significantly slow down the execution of the loop and may reduce or eliminate the benefits of an adjacent memory computing system.

[0059] Another exemplary approach uses additional synchronization streams to perform looping. Consider the arrangement given by the following code snippet [4]:

[0060]

[0061] In this example, the preservation and removal of local accumulation variables occur outside the loop. For example, each thread at the second synchronous flow can send an asynchronous message to the third flow controller of the third synchronous flow. The asynchronous message indicates to the third flow controller that the corresponding loop iteration has been completed. When the third flow controller receives ten asynchronous messages, indicating that all ten iterations of the loop have been completed, it starts the thread at the third synchronous flow to preserve and remove the local accumulation variables. This arrangement allows the loop to be executed correctly, however, it also requires the additional resources of a reconfigurable computing structure. For example, the resources of the flow controller and computing element used to implement the third synchronous flow may not be available for other processing.

[0062] Various examples solve these and other problems by utilizing a last iteration flag managed by a flow controller of the last synchronization flow of a loop iteration. For example, various flow controllers of a reconfigurable computing structure can be configured with a programmable field that controls whether the last iteration flag of a thread is set locally in the flow controller or provided by a previously executed thread. The flow controller of the synchronization flow that executes the last part of the loop iteration is configured to set the last iteration flag locally. The flow controller is also provided with an iteration count that indicates the total number of loop iterations it should execute.

[0063] The flow controller counts the number of loop iterations it processes, for example, by counting the number of threads launched. When launching the final loop iteration, the flow controller sets the last iteration flag in a synchronization message sent to the first computational element of the synchronization flow to launch the final thread. The thread is executed using operations that are conditional on the state of the last iteration flag. In this way, the conditional operation is executed in the thread of the last loop iteration to be executed, even if that iteration is not the last to be launched.

[0064] To further illustrate, consider the following example code segment [5], which provides a variation of the loop given by the example code segments [3] and [4] above:

[0065]

[0066] In this example, as with the arrangement of the exemplary code segment [3] above, two synchronization streams are used for each loop iteration. The thread at the first synchronization stream sends an asynchronous read request to the memory interface to load data [i], and also sends an asynchronous message to the second flow controller of the second synchronization stream. After the second flow controller receives an indication that the data [i] value has been received, it starts the thread at the second synchronization stream to update the local accumulation variable. However, in this example, the second flow controller is configured to track the total number of threads it starts at the second synchronization stream. When the number of threads started at the second synchronization stream meets the iteration count, the second flow controller sets the last iteration flag (lastlter) in the synchronization message that starts the last thread. The computing element of the last synchronization stream is configured to send one or more asynchronous messages to the memory interface to save and clear the local accumulation variable conditioned on the setting of the last iteration flag.

[0067] In this arrangement, execution of the loop can be robust even if one or more asynchronous operations of the loop iteration cause the iteration to complete out of order. Consider an exemplary execution of the code segment [5] above, in which the second synchronous stream receives the value of data [i] in order, except that data

[10] of the i=10 loop iteration is received before data [9] of the i=9 loop iteration. In this example, the second flow controller will start the thread at the second synchronous stream of the i=10 loop when data

[10] is received. Because, in this example, the i=10 iteration is not the last iteration to be completed, when the second synchronous stream thread of the i=10 iteration is started, the second flow controller will not set the last iteration flag. Therefore, the second synchronous stream thread of the i=10 iteration will not save and clear the local accumulation variable. Thereafter, when the flow controller receives data [9] of the i=9 iteration, it starts the thread at the second synchronous stream of the i=9 iteration and sets the last iteration flag. Therefore, the thread of the i=9 iteration (which is the last to be executed in this example) saves and clears the local accumulation variable. In this manner, as described herein, use of a last iteration flag at the last flow controller of a loop iteration may allow for robust and correct loop execution without slowing down processing by serializing loop iterations and without the additional resource requirements of using additional synchronization flows.

[0068] Figure 1 A first example of a neighboring memory computing system or CNM system 102 is generally shown. The example of the CNM system 102 includes a plurality of different memory computing nodes, such as each of which may include various neighboring memory computing devices. Each node in the system may operate in its own operating system (OS) domain (e.g., Linux, etc.). In an example, the nodes may coexist in a common OS domain of the CNM system 102.

[0069] Figure 1 The example includes an instance of a first memory computing node 104 of the CNM system 102. The CNM system 102 may have multiple nodes, such as including different instances of the first memory computing node 104, which are coupled using an expansion structure 106. In an example, the architecture of the CNM system 102 may support expansion using up to n different memory computing nodes (e.g., n=4096) using the expansion structure 106. As discussed further below, each node in the CNM system 102 may be an assembly of multiple devices.

[0070] The CNM system 102 may include a global controller for various nodes in the system, or a particular memory computing node in the system may optionally serve as a host or controller for one or more other memory computing nodes in the same system. Thus, various nodes in the CNM system 102 may be configured similarly or differently.

[0071] In an example, each node in the CNM system 102 may include a host system that uses a specified operating system. The operating system may be common or different between the various nodes in the CNM system 102. Figure 1 In an example, the first memory computing node 104 includes a host system 108, a first switch 110, and a first memory computing device 112. The host system 108 may include a processor, such as an X86, ARM, RISC-V, or other type of processor. The first switch 110 may be configured to facilitate communication between the first memory computing node 104 or the devices of the CNM system 102, such as using a dedicated or other communication protocol, which is generally referred to as a chip-to-chip protocol interface (CT) herein. That is, CTCPI may include a dedicated interface unique to the CNM system 102, or may include or use other interfaces, such as a computing express link (CXL) interface, a peripheral component interconnect express (PCIe) interface, or a chiplet protocol interface (CPI). The first switch 110 may include a switch configured to use CTCPI. For example, the first switch 110 may include a CXL switch, a PCIe switch, a CPI switch, or other types of switches. In an example, the first switch 110 may be configured to couple endpoints of different configurations. For example, the first switch 110 may be configured to convert packet formats, such as converting between PCIe and CPI formats.

[0072] The CNM system 102 is described herein in various exemplary configurations, such as a system including nodes, and each node may include various chips (e.g., processors, switches, memory devices, etc.). In an example, the first memory computing node 104 in the CNM system 102 may include various chips implemented using chiplets. In the chiplet-based configuration of the CNM system 102 discussed below, inter-chiplet communications and additional communications within the system may use a CPI network. The CPI network described herein is an example of CTCPI, that is, a chiplet-specific implementation of CTCPI. Therefore, the structure, operation, and function of the CPI described below can be equally applied to structures, operations, and functions that may be implemented in other ways using a non-chiplet-based CTCPI. Unless otherwise explicitly stated, any discussion of CPI herein also applies to CTCPI.

[0073] The CPI interface includes a packet-based network that supports virtual channels to enable flexible and high-speed interactions between chiplets, such as may include a first memory compute node 104 or a portion of a CNM system 102. CPI enables bridging from a network within a chiplet to a wider network of chiplets. For example, the Advanced Extensible Interface (AXI) is a specification for intra-chip communications. However, the AXI specification covers a large number of physical design options, such as the number of physical channels, signal timing, power, etc. Within a single chip, these options are typically selected to meet design goals, such as power consumption, speed, etc. However, in order to achieve the flexibility of a chiplet-based memory compute system, an adapter (such as using CPI) can interface between various AXI design options that can be implemented in various chiplets. By implementing a mapping of physical channels to virtual channels and encapsulating time-based signaling with a packetized protocol, CPI can be used to bridge intra-chiplet networks, such as within a specific memory compute node, across a wider chiplet network, such as across the first memory compute node 104 or across the CNM system 102.

[0074] The CNM system 102 is scalable to include multi-node configurations. That is, multiple different instances of the first memory computing node 104 or other differently configured memory computing nodes can be coupled using the expansion structure 106 to provide an extended system. Each memory computing node can run its own operating system and can be configured to jointly coordinate system-wide resource usage.

[0075] exist Figure 1 In an example, the first switch 110 of the first memory computing node 104 is coupled to the expansion structure 106. The expansion structure 106 can provide a switch (e.g., a CTCPI switch, a PCIe switch, a CPI switch, or other switches) that can facilitate communication between different memory computing nodes. In an example, the expansion structure 106 can help various nodes communicate in a partitioned global address space (PGAS).

[0076] In an example, a first switch 110 from a first memory computing node 104 is coupled to one or more different memory computing devices, such as including a first memory computing device 112. The first memory computing device 112 may include a chiplet-based architecture, which is referred to herein as a neighboring memory computing (CNM) chiplet. A packaged version of the first memory computing device 112 may include, for example, one or more CNM chiplets. These chiplets may be communicatively coupled using CTCPI for high bandwidth and low latency.

[0077] exist Figure 1In an example of, the first memory computing device 112 may include a network on chip (NOC) or a first NOC 118. Typically, a NOC is an interconnect network within a device that connects a specific set of endpoints. Figure 1 In the embodiment, the first NOC 118 can provide communication and connection between various memories, computing resources and ports of the first memory computing device 112.

[0078] In an example, the first NOC 118 can include a folded Clos topology, such as within each instance of a memory computing device, or as a mesh of multiple memory computing devices in a coupled node. The Clos topology (such as multiple smaller radix crossbars can be used to provide the functionality associated with a higher radix crossbar topology) provides various benefits. For example, the Clos topology can exhibit consistent latency and equally divided bandwidth across the NOC.

[0079] The first NOC 118 can include various switch types, including hub switches, edge switches, and endpoint switches. Each switch can be constructed as a crossbar that provides substantially uniform latency and bandwidth between input nodes and output nodes. In an example, the endpoint switch and the edge switch can include two independent crossbars, one for traffic heading to the hub switch, and the other for traffic away from the hub switch. The hub switch can be constructed as a single crossbar that switches all inputs to all outputs.

[0080] In an example, the hub switches may each have multiple ports (eg, four or six ports each), such as depending on whether a particular hub switch participates in inter-chip communications. The number of hub switches participating in inter-chip communications may be set by the inter-chip bandwidth requirements.

[0081] The first NOC 118 can support various payloads between computing elements and memory (e.g., payloads from 8 to 64 bytes; similarly, other payload sizes can be used). In an example, the first NOC 118 can be optimized for relatively small payloads (e.g., 8-16 bytes) to efficiently handle access to sparse data structures.

[0082] In an example, the first NOC 118 can be coupled to an external host via a first physical layer interface 114, a PCIe slave module 116 or endpoint, and a PCIe master module 126 or root port. That is, the first physical layer interface 114 can include an interface that allows an external host processor to be coupled to the first memory computing device 112. The external host processor can optionally be coupled to one or more different memory computing devices, such as using a PCIe switch or other local protocol switch. Communication with an external host processor through a PCIe-based switch can limit device-to-device communication to the range supported by the switch. In contrast, communication through a memory computing device local protocol switch (such as using CTCPI) can allow more complete communication between different memory computing devices, including support for partitioned global address spaces, such as creating threads of work and sending events.

[0083] In an example, the CTCPI protocol may be used by the first NOC 118 in the first memory computing device 112, and the first switch 110 may include a CTCPI switch. The CTCPI switch may allow CTCPI packets to be transmitted from a source memory computing device (such as the first memory computing device 112) to a different destination memory computing device (e.g., on the same or other nodes) without being converted to another packet format.

[0084] In an example, the first memory computing device 112 may include an internal host processor 122. The internal host processor 122 may be configured to communicate with the first NOC 118 or other components or modules of the first memory computing device 112, for example, using an internal PCIe master module 126, which may help eliminate a physical layer that would consume time and energy. In an example, the internal host processor 122 may be based on a RISC-V ISA processor and may use a first physical layer interface 114 to communicate outside the first memory computing device 112, such as communicating with other storage, network, or other peripheral devices of the first memory computing device 112. The internal host processor 122 may control the first memory computing device 112 and may act as a proxy for operating system-related functions. The internal host processor 122 may include a relatively small number of processing cores (e.g., 2-4 cores) and a host memory device 124 (e.g., including a DRAM module).

[0085] In an example, the internal host processor 122 may include a PCI root port. When the internal host processor 122 is in use, one of its root ports may be connected to the PCIe slave module 116. Another of the root ports of the internal host processor 122 may be connected to the first physical layer interface 114, such as to provide communication with an external PCI peripheral device. When the internal host processor 122 is disabled, the PCIe slave module 116 may be coupled to the first physical layer interface 114 to allow the external host processor to communicate with the first NOC 118. In an example of a system with multiple memory computing devices, the first memory computing device 112 may be configured to act as a system host or controller. In this example, the internal host processor 122 may be used, and other instances of the internal host processor in the corresponding other memory computing devices may be disabled.

[0086] The internal host processor 122 may be configured when the first memory computing device 112 is powered up, such as to allow host initialization. In an example, the internal host processor 122 and its associated data path (e.g., including the first physical layer interface 114, the PCIe slave module 116, etc.) may be configured from input pins to the first memory computing device 112. One or more pins may be used to enable or disable the internal host processor 122 and configure the PCI (or other) data path accordingly.

[0087] In an example, the first NOC 118 can be coupled to the extended structure 106 via an extended structure interface module 136 and a second physical layer interface 138. The extended structure interface module 136 or SIF can facilitate communication between the first memory computing device 112 and a device space such as a partitioned global address space (PGAS). The PGAS can be configured so that a specific memory computing device such as the first memory computing device 112 can access memory or other resources on different memory computing devices (e.g., on the same or different nodes) using a load / store paradigm. Various scalable structural technologies can be used, including CTCPI, CPI, Gen-Z, PCI, or Ethernet bridged on CXL. The extended structure 106 can be configured to support various packet formats. In an example, the extended structure 106 supports out-of-order packet communication, or supports sequential packets, such as using a path identifier to extend bandwidth across multiple equivalent paths. The extended structure 106 can generally support remote operations, such as remote memory reads, writes, and other built-in atoms, remote memory atoms, remote memory computing device sends events, and remote memory computing device calls and returns operations.

[0088] In an example, the first NOC 118 can be coupled to one or more different memory modules, such as including a first memory device 128. The first memory device 128 can include various types of memory devices, such as LPDDR5 or GDDR6, etc. Figure 1 In an example, the first NOC 118 can coordinate communication with the first memory device 128 via a memory controller 130 that can be dedicated to a particular memory module. In an example, the memory controller 130 can include a memory module cache and an atomic operation module. The atomic operation module can be configured to provide relatively high throughput atomic operators, such as including integer and floating point operators. The atomic operation module can be configured to apply its operators to data within a memory module cache (e.g., including an SRAM memory side cache), thereby allowing back-to-back atomic operations using the same memory location with minimal throughput degradation.

[0089] The memory module cache can provide storage for frequently accessed memory locations, such as without having to re-access the first memory device 128. In an example, the memory module cache can be configured to cache data only for a specific instance of the memory controller 130. In an example, the memory controller 130 includes a DRAM controller configured to interface with the first memory device 128, such as including a DRAM device. The memory controller 130 can provide access scheduling and bit error management, among other functions.

[0090] In an example, the first NOC 118 may be coupled to a hybrid thread processor (HTP 140), a hybrid thread fabric (HTF 142), and a host interface and scheduling module (HIF 120). The HIF 120 may be configured to facilitate access to host-based command request queues and response queues. In an example, the HIF 120 may schedule new execution threads on processors or computing elements of the HTP 140 or HTF 142. In an example, the HIF 120 may be configured to maintain workload balance across the HTP 140 module and the HTF 142 module.

[0091] A hybrid thread processor or HTP 140 may include an accelerator, such as may be based on the RISC-V instruction set. The HTP 140 may include a highly threaded, event-driven processor, where threads may execute in a single instruction cycle, such as to maintain high instruction throughput. The HTP 140 includes relatively few custom instructions to support low-overhead threading capabilities, event send / receive, and shared memory atomic operators.

[0092] The hybrid thread structure or HTF 142 may include an accelerator, such as a non-von Neumann, coarse-grained, reconfigurable processor. The HTF 142 may be optimized for high-level language operations and data types (e.g., integer or floating point). In an example, the HTF 142 may support data flow computing. The HTF 142 may be configured to use substantially all of the memory bandwidth available on the first memory computing device 112, such as when executing a memory-bound computing kernel.

[0093] The HTP and HTF accelerators of the CNM system 102 can be programmed using various high-level structured programming languages. For example, the HTP and HTF accelerators can be programmed using C / C++, such as using the LLVM compiler framework. The HTP accelerator can utilize an open source compiler environment, such as adding various custom instruction sets, which are configured to improve memory access efficiency, provide message passing mechanisms, and manage events, etc. In an example, the HTF accelerator can be designed to be able to program HTF 142 using a high-level programming language, and the compiler can generate a simulator configuration file or binary file running on HTF 142 hardware. HTF 142 can provide a mid-level language for accurately and concisely expressing algorithms while hiding the configuration details of the HTF accelerator itself. In an example, the HTF accelerator tool chain can use the LLVM front-end compiler and the LLVM intermediate representation (IR) to interface with the HTF accelerator back end.

[0094] Figure 2 An example of a memory subsystem 200 of a memory computing device according to an embodiment is generally shown. The example of the memory subsystem 200 includes a controller 202, a programmable atomic unit 208, and a second NOC 206. The controller 202 may include or use the programmable atomic unit 208 to perform operations using information in the memory device 204. In the example, the memory subsystem 200 includes Figure 1 An instance of a portion of a first memory computing device 112 , such as including a portion of a first NOC 118 or a memory controller 130 .

[0095] exist Figure 2 In an example, the second NOC 206 is coupled to the controller 202, and the controller 202 may include a memory control module 210, a local cache module 212, and a built-in atomic module 214. In an example, the built-in atomic module 214 may be configured to process relatively simple single-cycle integer atomics. The built-in atomic module 214 may perform atomics at the same throughput as, for example, a normal memory read or write operation. In an example, an atomic memory operation may include a combination of storing data to a memory, performing an atomic memory operation, and then responding with the load data from the memory.

[0096] A local cache module 212 (e.g., which may include an SRAM cache) may be provided to help reduce latency for repeatedly accessed memory locations. In an example, the local cache module 212 may provide a read buffer for sub-memory line accesses. The local cache module 212 may be particularly beneficial for computing elements that have relatively small or no data caches.

[0097] The memory control module 210 (e.g., which may include a DRAM controller) may provide low-level request buffering and scheduling, such as to provide efficient access to the memory device 204 (e.g., which may include a DRAM device). In an example, the memory device 204 may include or use a GDDR6 DRAM device, such as having a density of 16 Gb and a peak bandwidth of 64 Gb / sec. Similarly, other devices may be used.

[0098] In an example, the programmable atomic unit 208 may include single-cycle or multi-cycle operators, such as being configured to perform integer addition or more complex multi-instruction operations, such as Bloom filter insertion. In an example, the programmable atomic unit 208 may be configured to perform load and store operations to memory. The programmable atomic unit 208 may be configured to utilize a RISC-V ISA with a set of dedicated instructions to facilitate interaction with the controller 202 to atomically perform user-defined operations.

[0099] A programmable atomic request such as received from a host on a node or off a node can be routed to the programmable atomic unit 208 via the second NOC 206 and the controller 202. In an example, a custom atomic operation (e.g., performed by the programmable atomic unit 208) can be the same as a built-in atomic operation (e.g., performed by the built-in atomic module 214), except that the programmable atomic operation can be defined or programmed by a user rather than a system architect. In an example, a programmable atomic request packet can be sent to the controller 202 via the second NOC 206, and the controller 202 can identify the request as a custom atom. The controller 202 can then forward the identified request to the programmable atomic unit 208.

[0100] Figure 3 An example of a programmable atomic unit 302 for use with a memory controller according to an embodiment is generally shown. In an example, the programmable atomic unit 302 may include or may correspond to a programmable atomic unit 302 from Figure 2 An example of a programmable atomic unit 208. That is, Figure 3 Components in an example of a programmable atomic unit 302 (PAU) are shown, as described above with respect to Figure 2 (e.g., in programmable atomic unit 208) or with respect to Figure 1(e.g., in an atomic operation module of the memory controller 130). Figure 3 As shown, the programmable atomic unit 302 includes a PAU processor or PAU core 306, a PAU thread controller 304, an instruction SRAM 308, a data cache 310, and a memory interface 312 that interfaces with a memory controller 314. In the example, the memory controller 314 includes Figure 2 An instance of a controller 202 of an instance of.

[0101] In an example, the PAU core 306 is a pipeline processor so that multiple stages of different instructions are executed together per clock cycle. The PAU core 306 may include a barrel multithreaded processor with a thread control 304 circuit system to switch between different register files (e.g., a set of registers containing the current processing state) at each clock cycle. This achieves efficient context switching between currently executing threads. In an example, the PAU core 306 supports eight threads, thereby generating eight register files. In an example, some or all of the register files are not integrated into the PAU core 306, but reside in a local data cache 310 or an instruction SRAM 308. This reduces the circuit complexity in the PAU core 306 by eliminating traditional triggers for registers in such memory.

[0102] The local PAU memory may include an instruction SRAM 308, such as instructions for various atoms. These instructions include instruction sets that support atomic operators loaded by various applications. When an atomic operator is requested, such as by an application chiplet, a set of instructions corresponding to the atomic operator is executed by the PAU core 306. In an example, the instruction SRAM 308 can be partitioned to establish an instruction set. In this example, a specific programmable atomic operator requested by a requesting process can identify the programmable atomic operator by a partition number. When the programmable atomic operator is registered to (e.g., loaded to) the programmable atomic unit 302, the partition number can be established. Other metadata of the programmable instruction can be stored in a memory in a memory local to the programmable atomic unit 302 (e.g., in a partition table).

[0103] In an example, the atomic operator manipulates the data cache 310, which is typically synchronized (e.g., flushed) when the thread of the atomic operator completes. Thus, latency can be reduced for most memory operations during execution of the programmable atomic operator thread, except for the initial load from external memory (such as from the memory controller 314).

[0104] When an executing thread attempts to issue a memory request, a pipeline processor such as the PAU core 306 may experience problems if a potentially hazardous condition would prevent such a request. Here, the memory request is to retrieve data from the memory controller 314, whether it is from a cache on the memory controller 314 or an off-die memory. To address this issue, the PAU core 306 is configured to deny memory requests to threads. In general, the PAU core 306 or thread control 304 may include circuitry to enable one or more thread rescheduling points in the pipeline. Here, the denial occurs at a point in the pipeline outside (e.g., after) these thread rescheduling points. In an example, the hazard occurs outside of a rescheduling point. Here, the previous instruction in the thread created a hazard after the memory request instruction passed through the last thread rescheduling point before the pipeline stage where the memory request could be made.

[0105] In an example, to deny a memory request, the PAU core 306 is configured to determine (e.g., detect) that a hazard exists on the memory indicated in the memory request. Here, a hazard represents any condition under which allowing (e.g., executing) the memory request will cause an inconsistent state for the thread. In an example, the hazard is an ongoing memory request. Here, regardless of whether the data cache 310 includes data for the requested memory address, the presence of an ongoing memory request makes it uncertain what the data in the data cache 310 at the address should be. Therefore, the thread must wait for the ongoing memory request to complete in order to operate on the current data. When the memory request completes, the hazard is cleared.

[0106] In an example, the hazard is a dirty cache line in the data cache 310 for the requested memory address. While a dirty cache line generally indicates that the data in the cache is current and the memory controller version of the data is not current, problems may occur on thread instructions that do not operate from the cache. An example of such an instruction uses a built-in atomic operator or other separate hardware block of the memory controller 314. In the context of the memory controller, the built-in atomic operator may be separate from the programmable atomic unit 302 and does not have access to the cache within the PAU or the data cache 310. If the cache line is dirty, the built-in atomic operator will not operate on the latest data until the cache is refreshed to synchronize the cache with other or off-die memory. The same situation may also occur with other hardware blocks of the memory controller (such as encryption blocks, encoders, etc.).

[0107] Figure 4An example of a hybrid thread processor (HTP) accelerator or HTP accelerator 400 is shown. According to an embodiment, the HTP accelerator 400 may include a portion of a memory computing device. In an example, the HTP accelerator 400 may include or include a processor from Figure 1 140. The HTP accelerator 400 includes, for example, an HTP core 402, an instruction cache 404, a data cache 406, a conversion block 408, a memory interface 410, and a thread controller 412. The HTP accelerator 400 may further include a scheduling interface 414 and a NOC interface 416, such as for communicating with a NOC (e.g., from Figure 1 The first NOC in the example of 118, from Figure 2 A second NOC 206 or other NOC) interface in an instance of.

[0108] In an example, the HTP accelerator 400 includes a module based on the RISC-V instruction set, and may include a relatively small amount of other or additional custom instructions to support low overhead, threaded hybrid thread (HT) language. The HTP accelerator 400 may include a highly threaded processor core, i.e., an HTP core 402, in which or using the processor core, threads may be executed in a single instruction cycle, such as to maintain high instruction throughput. In an example, a thread may be suspended while waiting for other pending events to complete. This may allow computing resources to be effectively used for related work, rather than polling. In an example, multithreaded barrier synchronization may use efficient HTP to HTP and HTP to host / from host message delivery, such as allowing thousands of threads to be initialized or awakened in, for example, tens of clock cycles.

[0109] In an example, the scheduling interface 414 may include a functional block of the HTP accelerator 400 for handling hardware-based thread management. That is, the scheduling interface 414 may manage the scheduling of the work of the HTP core 402 or other accelerators. However, non-HTP accelerators are generally unable to schedule work. In an example, the work scheduled from the host may use a scheduling queue residing in, for example, the host main memory (e.g., a DRAM-based memory). On the other hand, the work scheduled from the HTP accelerator 400 may use a scheduling queue residing in the SRAM, such as within the scheduling of the target HTP accelerator 400 in a particular node.

[0110] In an example, the HTP core 402 may include one or more cores that execute instructions on behalf of threads. That is, the HTP core 402 may include an instruction processing block. The HTP core 402 may further include a thread controller 412 or may be coupled to the thread controller. The thread controller 412 may provide thread control and status for each active thread in the HTP core 402. The data cache 406 may include a cache for a host processor (e.g., for local and remote memory computing devices, including for the HTP core 402), and the instruction cache 404 may include a cache for use by the HTP core 402. In an example, the data cache 406 may be configured for read and write operations, and the instruction cache 404 may be configured for read-only operations.

[0111] In an example, data cache 406 is a small cache provided for each hardware thread. Data cache 406 can temporarily store data for use by the owning thread. Data cache 406 can be managed by hardware or software in HTP accelerator 400. For example, the hardware can be configured to automatically allocate or evict lines as needed as load and store operations are performed by HTP core 402. Software (such as using RISC-V instructions) can determine which memory accesses should be cached and when lines should be invalidated or written back to other memory locations.

[0112] The data cache on the HTP accelerator 400 has various benefits, including making larger access more effective for the memory controller, thereby allowing execution threads to avoid stagnation. However, there are situations that lead to inefficiency when using cache. Examples include accesses where data is only accessed once, and cause the impact of cache lines. In order to help solve this problem, the HTP accelerator 400 can use a set of customized load instructions to force the load instruction to check cache hits, and issue a memory request to the requested operand when the cache misses, and the obtained data is not placed in the data cache 406. Therefore, the HTP accelerator 400 includes various different types of load instructions, including non-cached and cached line loads. If there is dirty data in the cache, the non-cached load instruction uses the cached data. The non-cached load instruction can ignore the clean data in the cache, and the accessed data will not be written to the data cache. For cache line load instructions, a complete data cache line (e.g., containing 64 bytes) can be loaded from the memory into the data cache 406, and the addressed memory can be loaded into the specified register. These loads may use cached data if the clean data or dirty data is in the data cache 406. If the referenced memory location is not in the data cache 406, the entire cache line may be accessed from memory. The use of cache line load instructions may reduce cache misses when sequential memory locations are referenced (such as memory copy operations), but may also waste memory and bandwidth at the NOC interface 416 if the referenced memory data is not used.

[0113] In an example, the HTP accelerator 400 includes a non-cacheable custom store instruction. The non-cacheable store instruction can help avoid impacting the data cache 406 with write data that is not sequentially written to memory.

[0114] In an example, the HTP accelerator 400 further includes a conversion block 408. The conversion block 408 may include a virtual to physical conversion block for a local memory of a memory computing device. For example, a host processor such as in the HTP core 402 may execute a load or store instruction, and the instruction may generate a virtual address. The virtual address may be converted to a physical address of the host processor, such as using a conversion table from the conversion block 408. The memory interface 410 may include, for example, an interface between the HTP core 402 and the NOC interface 416.

[0115] Figure 5 An example of a representation of a hybrid thread structure (HTF) or HTF 500 of a memory computing device according to an embodiment is shown. In the example, HTF 500 may include or contain Figure 1500 is a coarse-grained, reconfigurable computation structure that can be optimized for high-level language operand types and operators (e.g., using C / C++ or other high-level languages). In an example, HTF 500 can include a configurable n-bit wide (e.g., 512-bit wide) data path that interconnects hardened SIMD arithmetic units.

[0116] In an example, HTF 500 includes HTF cluster 502, which includes multiple HTF tiles (including exemplary tile 504 or tile N). Each HTF tile can implement one or more computing elements using local tile or computing element memory and arithmetic functions. For example, each tile can include a computing pipeline that supports integer and floating point operations. In an example, data paths, computing elements, and other infrastructure can be implemented as hardened IP to provide maximum performance while minimizing power consumption and reconfiguration time.

[0117] exist Figure 5 In the example of , the tiles comprising the HTF cluster 502 are arranged linearly, and each tile in the cluster can be coupled to one or more other tiles in the HTF cluster 502. Figure 5 In the example of , the exemplary tile 504 or tile N is coupled to four other tiles, including a tile base (e.g., tile N-2) coupled to tile 510 via a port labeled SF Input N-2, coupled to an adjacent tile 512 (e.g., tile N-1) via a port labeled SF Input N-1, and coupled to tile N+1 via a port labeled SF Input N+1 and coupled to tile N+2 via a port labeled SF Input N+2. The tile base is a hardware portion of a tile (e.g., tiles 504, 510, 512) that is configured to launch threads and / or otherwise act as a flow controller. The exemplary tile 504 can be coupled to the same or other tiles via corresponding output ports (e.g., output ports labeled SF Output N-1, SF Output N-2, SF Output N+1, and SF Output N+2). In this example, the ordered list of the names of the various tiles is a conceptual indication of the locations of the tiles. In other examples, the tiles comprising HTF cluster 502 may be arranged in a grid or other configuration, where each tile is similarly coupled to one or a few of its nearest neighbors in the grid. Tiles disposed at the edge of a cluster may optionally have fewer connections to adjacent tiles. For example, tile N-2 or Figure 5 The tile basis of the example tile 510 may be coupled only to the adjacent tile 512 (tile N-1) and the example tile 504 (tile N). Similarly, fewer or additional inter-tile connections may be used.

[0118] The HTF cluster 502 may further include memory interface modules, including a first memory interface module 506. The memory interface module may couple the HTF cluster 502 to a NOC (e.g., the first NOC 118). In an example, the memory interface module may allow tiles within the cluster to make requests to other locations in the memory computing system, such as in the same or different nodes in the system. That is, the representation of the HTF 500 may comprise a portion of a larger structure that may be distributed across multiple nodes, such as having one or more HTF tiles or HTF clusters on each node. Requests may be made between tiles or nodes in the context of a larger structure.

[0119] exist Figure 5 In the example of FIG. 5 , a synchronization fabric (SF) is used to couple tiles in the HTF cluster 502. As described above, the synchronization fabric can provide communication between a particular tile and its neighboring tiles in the HTF cluster 502. Each HTF cluster 502 can further include an asynchronous fabric (AF) that can provide communication between, for example, tiles in the cluster, memory interfaces in the cluster, and a scheduling interface 508 in the cluster.

[0120] In an example, the synchronization structure can exchange messages including data and control information. The control information can include instruction RAM address information or thread identifiers, among others. The control information can be used to establish a data path, and the data message field can be selected as the source of the path. Typically, the control fields can be provided or received earlier so that they can be used to configure the data path. For example, to help minimize any delays in the tile through the synchronous flow pipeline, the control information can arrive at the tile several clock cycles before the data field. Various registers can be provided to help coordinate the data flow timing in the pipeline.

[0121] In an example, each tile in the HTF cluster 502 may include one or more tile memories. Each tile memory may have the same width as the data path (e.g., 512 bits) and may have a specified depth, such as in the range of 512 to 1024 elements. The tile memories may be used to store data that supports data path operations. For example, the stored data may include constants loaded as part of the cluster configuration of the kernel, or may include variables calculated as part of the data flow. In an example, the tile memory may be written from an asynchronous structure as a data transfer from another synchronous flow, or may include the result of a load operation such as initiated by another synchronous flow. The tile memory may be read via synchronous data path instruction execution in a synchronous flow.

[0122] In an example, each tile in the HTF cluster 502 can have a dedicated instruction RAM (INST RAM). In an example of an HTF cluster 502 with sixteen tiles and a corresponding instruction RAM instance with sixty-four entries, the cluster can allow algorithms to be mapped with up to 1024 multiply shifts and / or ALU operations. The various tiles can optionally be pipelined together, such as using synchronous structures to allow data flow calculations to be performed with minimal memory accesses, thereby minimizing latency and reducing power consumption. In an example, asynchronous structures can allow memory references to be performed in parallel with calculations, thereby providing a more efficient streaming kernel. In an example, the various tiles can include built-in support for loop-based constructs and can support nested loop kernels.

[0123] The synchronization structure can allow multiple tiles (e.g., multiple computing elements thereof) to be pipelined, such as without the need for data queuing. For example, the computing elements participating in the synchronization flow can act as a single pipeline data path. The flow controller for the synchronization flow can be or include a tile (e.g., in Figure 5 In the example of the block N-2, the computing elements on the block and / or the block base or controller on the block. The flow controller of the synchronous flow can start the threads of work through the pipelined blocks. The flow controller can be responsible for starting the threads at a predefined rhythm (referred to as the branch count in this article). For example, if the branch count is 3, the tile base can start a thread every three clock cycles.

[0124] In an example, a synchronization stream includes a set of connected computing elements in the HTF cluster 502. The execution of a thread can be started at a flow controller, and can be advanced from the flow controller to other computing elements (e.g., other tiles in the same synchronization stream) via a synchronization structure. The flow controller can provide instructions to be executed for the first computing element. By default, the first computing element can provide the same instructions for other connected computing elements to execute. However, in some examples, the flow controller or subsequent computing elements can implement conditional operations that conditionally specify or use alternative instructions. Alternative instructions can be selected by causing the data path of the computing element to generate a Boolean conditional value, which can then be used to select between the instruction set of the current computing element and the alternative instruction.

[0125] The asynchronous structure can be used to perform operations that occur asynchronously with respect to the synchronous flow. Each tile in the HTF cluster 502 can include an interface to the asynchronous structure. The inbound interface can include, for example, a FIFO buffer or queue (e.g., an AF entry queue) to provide storage for messages that cannot be processed immediately. Similarly, the outbound interface of the asynchronous structure can include a FIFO buffer or queue (e.g., an AF exit queue) to provide storage for messages that cannot be sent out immediately.

[0126] In an example, messages in an asynchronous structure can be classified as data messages or control messages. Data messages can include SIMD-width data values ​​written to tile memory 0 (MEM_0) or memory 1 (MEM_1). Control messages can be configured to control thread creation, release resources, or issue external memory references.

[0127] The tiles in the HTF cluster 502 can perform various computational operations for the HTF. The computational operations can be performed by configuring the data paths within the tiles and / or their computational elements. In an example, the tiles include two functional blocks that perform the computational operations of the tiles: a multiplication and shift operation block (MS OP) and an arithmetic, logic, and bit operation block (ALB OP). These two blocks can be configured to perform pipelined operations, such as multiplication and addition, or shift and addition, etc.

[0128] In an example, each instance of a memory computing device in a system may have a complete set of supported instructions for its operator blocks (e.g., MS OPs and ALB OPs). In this case, binary compatibility may be achieved across all devices in the system. However, in some examples, it may be helpful to maintain a base set of functions and optional instruction set classes (e.g., to meet various design tradeoffs, such as die size). This approach may be similar to how the RISC-V instruction set has a base set and multiple optional instruction subsets.

[0129] In an example, the exemplary tile 504 may include a branch RAM. The branch RAM may be used to specify which input (e.g., from four SF tile inputs and tile base input) is the main input for each clock cycle. The branch RAM read address input may be derived from a counter that counts from zero to a branch count minus one. In an example, different branch counts may be used on different tiles, such as within the same HTF cluster 502, to allow the use of multiple slices or unique tile instances by an internal loop to determine the performance of a specific application or instruction set. In an example, the branch RAM may specify when a synchronous input is written to the tile memory, such as when multiple inputs of a specific tile instruction are used and one input arrives before the other inputs. The earliest input that arrives may be written to the tile memory and may be read later when all inputs are available. In this example, the tile memory may be accessed as a FIFO memory, and the FIFO read and write pointers may be stored in a register-based memory area or structure in the tile memory.

[0130] Fig. 6A and Figure 6BAn example of a chiplet system that can be used to implement one or more aspects of the CNM system 102 is generally shown. As similarly mentioned above, a node in the CNM system 102 or a device within a node in the CNM system 102 may include a chiplet-based architecture or a neighboring memory computing (CNM) chiplet. The packaged memory computing device may include, for example, one, two, or four CNM chiplets. The chiplets can be interconnected using a high-bandwidth, low-latency interconnect (such as using a CPI interface). Typically, a chiplet system consists of discrete modules (each a "chiplet") integrated on an interposer, and in many instances, interconnected as needed through one or more established networks to provide a system with the desired functionality. The interposer and the included chiplets can be packaged together to facilitate interconnection with other components of the larger system. Each chiplet may include one or more separate integrated circuits (ICs) or "chips," which are potentially combined with discrete circuit components and may be coupled to a corresponding substrate for attachment to the interposer. Most or all of the chiplets in the system can be individually configured for communication over an established network.

[0131] The configuration of chiplets as separate modules of a system is different from such systems implemented on a single chip containing different device blocks (e.g., intellectual property (IP) blocks) on one substrate (e.g., a single die), such as a system on a chip (SoC), or multiple discrete packaged devices integrated on a printed circuit board (PCB). In general, chiplets provide better performance (e.g., lower power consumption, reduced latency, etc.) than discrete packaged devices, and chiplets provide greater production benefits than a single bare die chip. These production benefits can include higher yields or reduced development costs and time.

[0132] A chiplet system may include, for example, one or more application (or processor) chiplets and one or more support chiplets. Here, the distinction between application chiplets and support chiplets is merely a reference to possible design scenarios for a chiplet system. Thus, for example, a synthetic vision chiplet system may include (by way of example only) an application chiplet to produce a synthetic vision output, and support chiplets such as a memory controller chiplet, a sensor interface chiplet, or a communication chiplet. In a typical use case, a synthetic vision designer may design an application chiplet and obtain support chiplets from other parties. Thus, design expenditure (e.g., in terms of time or complexity) is reduced by avoiding the design and production of functionality embodied in support chiplets.

[0133] Chiplets also support tight integration of IP blocks that might otherwise be difficult, such as IP blocks manufactured using different process technologies or using different feature sizes (or utilizing different contact technologies or spacings). Thus, multiple ICs or IC components with different physical, electrical, or communication characteristics can be assembled in a modular manner to provide components with various desired functions. Chiplet systems can also facilitate adaptation to the needs of different larger systems that will incorporate the chiplet system. In an example, an IC or other component can be optimized for power, speed, or heat generation for a specific function—as might happen with a sensor—and can be more easily integrated with other devices than trying to integrate on a single die. In addition, by reducing the overall size of the die, the yield of chiplets is often higher than that of more complex single-die devices.

[0134] Fig. 6A and Figure 6B An example of a chiplet system according to an embodiment is generally shown. Fig. 6A is a representation of a chiplet system 602 mounted on a peripheral board 604, which may be connected to a wider computer system via, for example, Peripheral Component Interconnect Express (PCIe). The chiplet system 602 includes a package substrate 606, an interposer 608, and four chiplets, an application chiplet 610, a host interface chiplet 612, a memory controller chiplet 614, and a memory device chiplet 616. Other systems may include many additional chiplets to provide additional functionality, as will be apparent from the discussion below. The packaging of the chiplet system 602 is shown as having a lid or cover 618, although other packaging techniques and structures for chiplet systems may be used. Figure 6B is a block diagram of the components in the chiplet system labeled for clarity.

[0135] The application chiplet 610 is shown to include a chiplet system NOC 620 to support a chiplet network 622 for inter-chiplet communication. In an exemplary embodiment, the chiplet system NOC 620 can be included on the application chiplet 610. In an example, the chiplet system NOC 620 from Figure 1 The first NOC 118 of the example can be defined in response to the selected supporting chiplets (e.g., host interface chiplet 612, memory controller chiplet 614, and memory device chiplet 616), thereby enabling the designer to select the appropriate number or chiplet network connections or switches for the chiplet system NOC 620. In an example, the chiplet system NOC 620 can be located on a separate chiplet, or within the interposer 608. In an example as discussed herein, the chiplet system NOC 620 implements a chiplet protocol interface (CPI) network.

[0136] In an example, the chiplet system 602 may include or comprise a portion of the first memory computing node 104 or the first memory computing device 112. That is, the various blocks or components of the first memory computing device 112 may include chiplets that may be mounted on the peripheral board 604, the package substrate 606, and the interposer 608. The interface components of the first memory computing device 112 may generally include a host interface chiplet 612, the memory and memory control related components of the first memory computing device 112 may generally include a memory controller chiplet 614, the various accelerator and processor components of the first memory computing device 112 may generally include an application chiplet 610 or instances thereof, and so on.

[0137] The CPI interface (such as can be used for communication between chiplets in the system) is a packet-based network that supports virtual channels to enable flexible and high-speed interaction between chiplets. CPI implements bridging from the intra-chiplet network to the chiplet network 622. For example, the Advanced Extensible Interface (AXI) is a widely used specification for designing intra-chip communications. However, the AXI specification covers a very large number of physical design options, such as the number of physical channels, signal timing, power, etc. Within a single chip, these options are usually selected to meet design goals, such as power consumption, speed, etc. However, in order to achieve the flexibility of the chiplet system, an adapter such as CPI is used to interface between various AXI design options that can be implemented in various chiplets. By implementing a mapping of physical channels to virtual channels and encapsulating time-based signaling with a packetized protocol, CPI bridges the intra-chiplet network across the chiplet network 622.

[0138] CPI can use a variety of different physical layers to transmit packets. The physical layer may include a simple conductive connection, or may include a driver to increase the voltage, or otherwise facilitate the transmission of signals over a longer distance. An example of such a physical layer may include an advanced interface bus (AIB), which in various examples may be implemented in the interposer 608. The AIB uses source synchronous data transmission with a forwarding clock to transmit and receive data. Packets are transmitted across the AIB relative to the transmitted clock at a single data rate (SDR) or a double data rate (DDR). Various channel widths are supported by the AIB. The channel can be configured to have a symmetrical number of transmit (TX) and receive (RX) inputs / outputs (I / O), or an asymmetrical number of transmitters and receivers (e.g., all transmitters or all receivers). The channel can act as an AIB master or slave depending on which chiplet provides the master clock. The AIB I / O unit supports three clock modes: asynchronous (i.e., non-clocked), SDR, and DDR. In various examples, the non-clocked mode is used for clocks and some control signals. The SDR mode can use a dedicated SDR-only I / O cell, or a dual purpose SDR / DDR I / O cell.

[0139] In an example, a CPI packet protocol (e.g., point-to-point or routable) can use symmetrical receive and transmit I / O units within an AIB channel. The CPI streaming protocol allows for more flexible use of AIB I / O units. In an example, an AIB channel in streaming mode can configure I / O units as all TX, all RX, or half TX and half RX. The CPI packet protocol can use the AIB channel in SDR or DDR operating mode. In an example, the AIB channel is configured in increments of 80 I / O units (i.e., 40TX and 40RX) for SDR mode and 40 I / O units for DDR mode. The CPI streaming protocol can use the AIB channel in SDR or DDR operating mode. Here, in an example, the AIB channel is in increments of 40 I / O units for both SDR and DDR modes. In an example, a unique interface identifier is assigned to each AIB channel. The identifier is used during CPI reset and initialization to determine paired AIB channels across adjacent chiplets. In an example, the interface identifier is a 20-bit value that includes a seven-bit chiplet identifier, a seven-bit column identifier, and a six-bit link identifier. The AIB physical layer uses the AIB out-of-band shift register to transmit the interface identifier. The 20-bit interface identifier is transmitted in both directions across the AIB interface using bits 32 to 51 of the shift register.

[0140] AIB defines a stacked group of AIB channels as an AIB channel column. An AIB channel column has a certain number of AIB channels, plus auxiliary channels. Auxiliary channels contain signals for AIB initialization. All AIB channels in a column (except auxiliary channels) have the same configuration (for example, all TX, all RX, or half TX and half RX, and have the same number of data I / O signals). In an example, the AIB channels are numbered in a continuous increasing order starting with the AIB channel adjacent to the AUX channel. The AIB channel adjacent to AUX is defined as AIB channel zero.

[0141] Typically, the CPI interface on a separate chiplet may include serialization-deserialization (SERDES) hardware. SERDES interconnects are well suited for scenarios requiring high-speed signaling and low signal counts. However, SERDES may result in additional power consumption and longer latency for multiplexing and demultiplexing, error detection or correction (e.g., using block-level cyclic redundancy checks (CRCs)), link-level retries, or forward error correction. However, when low latency or energy consumption is the primary concern for ultra-short distance chiplet-to-chiplet interconnects, a parallel interface may be utilized whose clock rate allows data transfer with minimal latency. CPI includes elements for minimizing both latency and energy consumption for these ultra-short distance chiplet interconnects.

[0142] For flow control, CPI employs a credit-based technique. A receiver such as application chiplet 610 provides credits representing available buffers to a sender such as memory controller chiplet 614. In an example, a CPI receiver includes a buffer for each virtual channel for a given transmit time unit. Thus, if a CPI receiver supports five messages and a single virtual channel in time, the receiver has five buffers arranged in five rows (e.g., one row per unit time). If four virtual channels are supported, the receiver has twenty buffers arranged in five rows. Each buffer holds the payload of one CPI packet.

[0143] When the sender transmits to the receiver, the sender decrements the available credits based on the transmission. Once all the credits of the receiver have been consumed, the sender stops sending packets to the receiver. This ensures that the receiver always has an available buffer to store the transmission.

[0144] When the receiver processes the received packet and frees up the buffer, the receiver transfers the available buffer space back to the sender. The sender can then use this credit back to allow the transmission of additional information.

[0145] Fig. 6A Examples include a chiplet mesh network 624 that uses direct chiplet-to-chiplet technology without the need for a chiplet system NOC 620. The chiplet mesh network 624 can be implemented with CPI or another chiplet-to-chiplet protocol. The chiplet mesh network 624 typically implements a pipeline of chiplets, where one chiplet acts as an interface to the pipeline, while other chiplets in the pipeline only interface with themselves.

[0146] In addition, dedicated device interfaces, such as one or more industry standard memory interfaces (such as, for example, synchronous memory interfaces, such as DDR5, DDR6), can be used to connect the device to the chiplet. The connection of the chiplet system or an individual chiplet to an external device (such as a larger system) can be through a desired interface (e.g., a PCIe interface). In an example, such an external interface can be implemented through a host interface chiplet 612, which in the depicted example provides a PCIe interface external to the chiplet system. Such a dedicated chiplet interface 626 is typically used when industry conventions or standards have converged on such an interface. The illustrated example of a double data rate (DDR) interface connecting a memory controller chiplet 614 to a dynamic random access memory (DRAM) memory device chiplet 616 is such an industry convention.

[0147] Among the various possible supporting chiplets, the memory controller chiplet 614 may be present in the chiplet system because of the almost ubiquitous use of memory for computer processing and the advanced technology of memory devices. Therefore, using a memory device chiplet 616 and a memory controller chiplet 614 produced by others enables chiplet system designers to have access to robust products produced by sophisticated producers. Typically, the memory controller chiplet 614 provides a memory device-specific interface to read, write, or erase data. Typically, the memory controller chiplet 614 may provide additional features such as error detection, error correction, maintenance operations, or atomic operator execution. For some types of memory, maintenance operations are often specific to the memory device chiplet 616, such as garbage collection in NAND flash memory or storage class memory, temperature adjustment in NAND flash memory (e.g., cross temperature management). In an instance, maintenance operations may include logical to physical (L2P) mapping or management to provide a level of indirection between the physical and logical representations of data. In other types of memory, such as DRAM, some memory operations, such as refresh, may be controlled at certain times by a host processor or memory controller and at other times by the DRAM memory device or logic associated with one or more DRAM devices, such as an interface chip (in an example, a buffer).

[0148] Atomic operators are data manipulations that may be performed, for example, by the memory controller chiplet 614. In other chiplet systems, atomic operators may be performed by other chiplets. For example, an atomic operator of "increment" may be specified by the application chiplet 610 in a command that includes a memory address and possibly an increment value. Upon receiving the command, the memory controller chiplet 614 retrieves a number from the specified memory address, increments the number by the amount specified in the command, and stores the result. Upon successful completion, the memory controller chiplet 614 provides an indication of the success of the command to the application chiplet 610. Atomic operators avoid transmitting data across the chiplet mesh network 624, resulting in lower latency execution of such commands.

[0149] Atomic operators can be classified as built-in atomics or programmable (e.g., custom) atomics. Built-in atomics are a limited set of operations that are implemented unchanged in hardware. Programmable atomics are small programs that can be executed on the programmable atomic unit (PAU) (e.g., custom atomic unit (CAU)) of the memory controller chiplet 614.

[0150] The memory device chiplet 616 may be or may include any combination of volatile memory devices or non-volatile memory. Examples of volatile memory devices include, but are not limited to, random access memory (RAM) such as DRAM, synchronous DRAM (SDRAM), Graphic Double Data Rate Type 6 SDRAM (GDDR6 SDRAM), etc. Examples of non-volatile memory devices include, but are not limited to, NAND-type flash memory, storage class memory (e.g., phase change memory or memristor-based technology), ferroelectric RAM (FeRAM), etc. The illustrated example includes a memory device chiplet 616 as a chiplet, however, the device may reside elsewhere, such as in a different package on the peripheral board 604. For many applications, multiple memory device chiplets may be provided. In an example, these memory device chiplets may each implement one or more storage technologies and may include an integrated computing host. In an example, a memory chiplet may include multiple stacked memory dies of different technologies, such as one or more static random access memory (SRAM) devices stacked or otherwise communicating with one or more dynamic random access memory (DRAM) devices. In an example, the memory controller chiplet 614 can be used to coordinate operations between multiple memory chiplets in the chiplet system 602, for example, using one or more memory chiplets in one or more levels of cache storage and using one or more additional memory chiplets as main memory. The chiplet system 602 may include multiple memory controller chiplet 614 instances, such as can be used to provide memory control functions for separate hosts, processors, sensors, networks, etc. The chiplet architecture in the system as shown provides advantages in allowing different memory storage technologies and different memory interfaces to be accommodated through updated chiplet configurations, such as without requiring redesign of the rest of the system architecture.

[0151] Figure 7 Generally, an example of a chiplet-based implementation for a memory computing device according to an embodiment is shown. The example includes an implementation with four neighboring memory computing or CNM chiplets, and each CNM chiplet may include or contain from Figure 1 The first memory computing device 112 or the first memory computing node 104 of the instantiation of the present invention may be a portion of the first memory computing device 112 or the first memory computing node 104. The respective portions themselves may include or contain respective chiplets. The chiplet-based implementation may include or use CPI-based intra-system communication, as described above in the context of Fig. 6A and Figure 6B The example chiplet system 602 is similarly discussed above.

[0152] Figure 7An example of includes a first CNM package 700 including a plurality of chiplets. The first CNM package 700 includes a first chiplet 702, a second chiplet 704, a third chiplet 706, and a fourth chiplet 708 coupled to a CNM NOC hub 710. Each of the first to fourth chiplets may include an instance of the same or substantially the same components or modules. For example, the chiplets may each include an instance of a HTP accelerator, an HTF accelerator, and a memory controller for accessing internal or external memory.

[0153] exist Figure 7 In the example of FIG. 7 , the first chiplet 702 includes a first NOC hub edge 714 coupled to a CNM NOC hub 710. Other chiplets in the first CNM package 700 similarly include NOC hub edges or endpoints. Switches in the NOC hub edge facilitate communication within a chiplet or within a chiplet system via the CNM NOC hub 710.

[0154] The first chiplet 702 may further include one or more memory controllers 716. The memory controllers 716 may correspond to respective different NOC endpoint switches that interface with the first NOC hub edge 714. In an example, the memory controller 716 includes the memory controller chiplet 614, or includes the memory controller 130, or includes the memory subsystem 200, or other memory computing implementations. The memory controller 716 may be coupled to respective different memory devices, such as including the first external memory module 712a or the second external memory module 712b. The external memory modules may include, for example, GDDR6 memory, which may be selectively accessed by respective different chiplets in the system.

[0155] The first chiplet 702 may further include a first HTP chiplet 718 and a second HTP chiplet 720, which are coupled to the first NOC hub edge 714, such as via respective different NOC endpoint switches. The HTP chiplet may correspond to an HTP accelerator, such as from Figure 1 HTP 140 for example, or from Figure 4 The HTP chiplet can communicate with the HTF chiplet 722. The HTF chiplet 722 can correspond to the HTF accelerator (such as from Figure 1 HTF 142 of the example, or from Figure 5 Example HTF 500).

[0156] The CNM NOC hub 710 can be coupled to other chiplets or NOC hub instances in other CNM packages through various interfaces and switches. For example, the CNM NOC hub 710 can be coupled to a CPI interface through multiple different NOC endpoints on the first CNM package 700. Each of the multiple different NOC endpoints can be coupled to, for example, a different node outside the first CNM package 700. In an example, the CNM NOC hub 710 can be coupled to other peripheral devices, nodes, or devices using CTCPI or other non-CPI protocols. For example, the first CNM package 700 can include a PCIe extended structure interface (PCIE / SFI) or a CXL interface (CXL) configured to interface the first CNM package 700 with other devices. In an example, the devices to which the first CNM package 700 is coupled using various CPI, PCIe, CXL, or other structures can constitute a common global address space.

[0157] exist Figure 7 In the example of FIG. 7 , the first CNM package 700 includes a host interface 724 (HIF) and a host processor (R5). The host interface 724 may correspond to, for example, a Figure 1 The host processor or R5 may correspond to the HIF 120 from Figure 1 The host interface 724 may include a PCI interface for coupling the first CNM package 700 to other external devices or systems. In an example, operations may be initiated on the first CNM package 700 or a cluster of tiles within the first CNM package 700 through the host interface 724. For example, the host interface 724 may be configured to command individual HTF tile clusters such as in various chiplets in the first CNM package 700 to enter and exit power / clock gating modes.

[0158] Figure 8 An exemplary tiling of a memory computing device according to an embodiment is shown. Figure 8 , tiled chiplet instance 800 includes four instances of different memory-near compute clusters of a chiplet, where the clusters are coupled together. Each instance of a memory-near compute chiplet can itself include one or more constituent chiplets (e.g., a host processor chiplet, a memory device chiplet, an interface chiplet, and so on).

[0159] The tiled chiplet example 800 includes Figure 7The first CNM package 700 of an instance of the embodiment of the invention may be coupled to one or more of its neighboring compute-on-memory (CNM) clusters. For example, the tiled chiplet instance 800 may include a first CNM cluster 802, which includes a first chiplet 810 (e.g., corresponding to the first chiplet 702), a second chiplet 812 (e.g., corresponding to the second chiplet 704), a third chiplet 814 (e.g., corresponding to the third chiplet 706), and a fourth chiplet 816 (e.g., corresponding to the fourth chiplet 708). The chiplets in the first CNM cluster 802 may be coupled to a common NOC hub, which in turn may be coupled to a NOC hub in one or more neighboring clusters (e.g., in the second CNM cluster 804 or the fourth CNM cluster 808).

[0160] exist Figure 8 In the example of FIG. 8 , the tiled chiplet instance 800 includes a first CNM cluster 802, a second CNM cluster 804, a third CNM cluster 806, and a fourth CNM cluster 808. Various different CNM chiplets can be configured in a common address space so that the chiplets can allocate and share resources across different tiles. In the example, the chiplets in the cluster can communicate with each other. For example, the first CNM cluster 802 can be communicatively coupled to the second CNM cluster 804 via an inter-chiplet CPI interface 818, and the first CNM cluster 802 can be communicatively coupled to the fourth CNM cluster 808 via another or the same CPI interface. The second CNM cluster 804 can be communicatively coupled to the third CNM cluster 806 via the same or other CPI interface, and so on.

[0161] In an example, one of the neighboring memory compute chiplets in the tiled chiplet instance 800 may include a host interface (e.g., corresponding to a host interface from the tiled chiplet instance 800) responsible for workload balancing across the tiled chiplet instance 800. Figure 7 The host interface can facilitate access to host-based command request queues and response queues, such as from outside the tiled chiplet instance 800. In the tiled chiplet instance 800, the host interface can schedule new execution threads using a hybrid threaded processor and a hybrid threaded fabric in one or more neighboring memory compute chiplets.

[0162] Fig. 9 is a diagram illustrating one example of a workflow 900 for performing operations on a reconfigurable computing structure such as HTF 142 or HTF 500 using more than one synchronization stream. Fig. 9The workflow 900 utilizes a scheduling interface 902 and two synchronization streams 901, 903. As described herein, the scheduling interface 902 is configured to receive instructions and / or data from other parts of the adjacent memory computing system. In some instances, the scheduling interface 902 is arranged in a manner similar to the scheduling interface 508 described herein.

[0163] The synchronization flows 901, 903 include respective flow controllers 904, 906 and ordered synchronization data paths 908, 910, as described herein. The synchronization data paths 908, 910 may be, for example, between computing elements of the respective synchronization flows 901, 903, as described herein. The flow controllers 904, 906 and the synchronization data paths 908, 910 are arranged from a reconfigurable computing structure (such as the HTF 142 or HTF 500 described herein).

[0164] The reconfigurable computing structure can be arranged to implement various computing elements of the synchronization flow 901, 903, including flow controllers 904, 906 and synchronization data paths 908, 910. For example, the flow controllers 904, 906 can be implemented using the corresponding tile basis of the tiles 510, 512, etc. The synchronization data paths 908, 910 can be implemented between the tiles 510, 512, etc., where the tiles 510, 512, etc. implement the computing elements described herein. As described herein and Figure 5 The tiles, tile bases, etc. shown may be components of a hybrid thread structure (HTF) such as HTF 142, which is a memory computing device such as Figure 1 The other synchronization streams described herein may be similarly arranged using the tiles and tile basis of the HTF as described herein.

[0165] exist Fig. 9 In the workflow 900, the domain interface 902 starts the processing task by providing an asynchronous message 912 to the flow controller 904 via the asynchronous structure of the reconfigurable computing structure. The asynchronous message 912 can be a continuous type of message. The domain interface 902 can start the processing task based on its own programming and / or in response to an instruction from another component of the reconfigurable computing structure of the adjacent memory computing system. In some instances, the domain interface 902 also configures the reconfigurable computing structure to the correct kernel configuration for executing the processing task, and the processing task includes, for example, programming various flow controllers 904, 906 and computing elements to implement synchronous flows 901, 903.

[0166] The asynchronous message 912 may instruct the flow controller 904 to start a synchronous flow thread. For example, when the branch RAM selects the flow controller 904 for the synchronous data path 908 between the components of the synchronous flow 901, the flow controller may start the thread by providing a first synchronization message to the first computing element of the synchronous data path 908. As described herein, the synchronization message may include data and control information. The first computing element of the synchronous data path 908 starts the execution of the thread by performing a processing task and generating a next synchronization message that is sent to the second computing element of the synchronous data path 908. The second computing element performs the processing task and generates a third synchronization message, which is sent to the third computing element of the synchronous data path 908, and so on, until the thread is executed on all computing elements of the synchronous data path 908.

[0167] The computing elements of the synchronization data path 908 are programmed to perform various operations when executing threads, including, for example, operations for configuring the synchronization flow 903. For example, during the execution of a thread at the synchronization flow 901, one or more computing elements of the synchronization data path 908 send an asynchronous message 914 to the flow controller 906 of the synchronization flow 903. The asynchronous message 914 instructs the flow controller 906 to start the thread at the synchronization flow 903 conditional on the occurrence of at least one condition. In this example, the one or more conditions include receiving another asynchronous message from the synchronization flow 901, which indicates that the thread is completed or that it is acceptable to start the thread at the synchronization flow 903 in other ways. Later in the execution of the thread at the synchronization flow 901, the computing elements of the synchronization data path 908 send an asynchronous message 916 to the flow controller 906 of the synchronization flow 903. The asynchronous message 916 indicates that the thread at the synchronization flow 901 has completed and / or reached a point where the corresponding thread at the synchronization flow 903 can start. In some examples, asynchronous message 916 indicates the results of processing during execution of a thread at synchronous flow 903 .

[0168] Upon receiving the asynchronous message 916 (and when other conditions (if any) of the asynchronous message 914 are satisfied), the flow controller 906 starts the thread at the synchronous flow 903, for example, by sending a first synchronous message to the first computing element of the synchronous data path 910. In some examples, for example, when the thread at the synchronous flow 903 is completed, the computing element of the synchronous data path 910 will send an asynchronous message 918 to the scheduling interface 902 to indicate that the requested processing task has been completed.

[0169] Fig.101 is a diagram showing an example of a workflow 1000 for performing operations on a reconfigurable computing structure such as HTF 142 or HTF 500, wherein synchronization flows 1001, 1003 interact with a memory interface 1016. In workflow 1000, domain interface 1002 initiates a processing task by providing an asynchronous message 1010 to flow controller 1004 via an asynchronous structure of the reconfigurable computing structure. Asynchronous message 1010 instructs flow controller 1004 to start a thread at synchronization flow 1001 when one or more conditions occur. (In some examples, message 1010 may instruct flow controller 1004 to start execution of a thread immediately without waiting for any condition to occur).

[0170] As described herein, flow controller 1004 starts a thread by providing a synchronization message to a first computing element of synchronization data path 1022. During execution of the thread at synchronization flow 1001, the computing element of synchronization data path 1022 provides an asynchronous message 1012 to flow controller 1006 of synchronization flow 1003. Asynchronous message 1012 may instruct flow controller 1006 to start the thread at synchronization flow 1003 upon receiving an indication that data from memory interface 1016 has been received.

[0171] As part of the execution of the thread at synchronization flow 1001, another computing element of synchronization data path 1022 provides an asynchronous message 1014 to memory interface 1016. Asynchronous message 1014 contains a read request to be directed to a memory of a memory-adjacent computing system (such as, for example, a memory computing device including a reconfigurable computing structure). In some examples, asynchronous message 1014 can also identify the computing element of synchronization flow 1003 that will receive the result of the read request.

[0172] The memory interface 1016 performs a read request on the appropriate system memory to obtain the load data. When the load data is received, the memory interface 1016 guides the asynchronous message 1018 including the load data to the computing element (e.g., one of the computing elements of the synchronous data path 1024) of the synchronous flow 1003 indicated by the asynchronous message 1014. When the load data is received, the computing element writes the load data to the computing element memory and sends an asynchronous message 1017 to the flow controller 1006 of the synchronous flow 1003. Message 1017 indicates that the load data has been received. In some instances, similar to message 1018, the memory interface 1016 provides an asynchronous message including the load data to multiple computing elements of the synchronous flow 1003. Similar to message 1017, each computing element that has received the load data can provide a corresponding asynchronous message to the flow controller 1006.

[0173] Upon receiving the asynchronous message 1017 (and after satisfying any other conditions of the continuation type message 1012), the flow controller 1006 starts the thread at the synchronization flow 1003, for example, by providing a synchronization message to the first computing element of the synchronization data path 1024. If more than one computing element of the synchronization flow 1003 is to receive load data, the flow controller 1006 can start the thread when it has received all expected asynchronous messages from the computing elements that have received the load data. In some examples, when the thread at the synchronization flow 1003 is completed, one or more computing elements (e.g., the last computing element thereof) of the synchronization data path 1024 sends an asynchronous message 1020 indicating completion to the scheduling interface 1002.

[0174] Fig.11 1 is a diagram showing one example of a workflow 1100 that uses a reconfigurable computing structure such as HTF 142 or HTF 500 to execute a loop 1122. The workflow 1100 is illustrated using three synchronization streams 1101, 1103, 1105. In this example, each iteration of the loop 1122 is executed at a single synchronization stream 1103 using a single thread.

[0175] Flow controller 1102 starts a first thread at synchronization flow 1101, for example, by providing a synchronization message to a first computing element of synchronization data path 1108. Execution of the thread at synchronization flow 1101 includes sending an asynchronous message 1114 to flow controller 1106 of synchronization flow 1105. Asynchronous message 1114 may indicate that flow controller 1106 will execute the thread at synchronization flow 1105 after receiving an indication that loop 1122 has been completed. Execution of the thread at synchronization flow 1101 also includes sending an asynchronous message 1116 to flow controller 1104 of synchronization flow 1103. Asynchronous message 1116 instructs flow controller 1104 to execute the loop. For example, asynchronous message 1116 may include an iteration count indicating the number of iterations in the loop.

[0176] The flow controller 1104 starts a thread for the first iteration of the loop. The computing elements of the synchronous data path 1110 of the synchronization domain 1103 execute the first iteration of the loop. When the thread is completed, one or more computing elements send an asynchronous message 1120 to the flow controller 1104 indicating that the first iteration of the loop has been completed.

[0177] In some instances, after launching the first thread for the first iteration of the loop, the flow controller 1104 continues to launch additional threads for executing additional iterations of the loop. For example, if the flow controller 1104 has a branch count of three, it can launch a new thread at the synchronization stream 1103 for a new iteration of the loop every three clock cycles. This can occur regardless of whether the thread of the previous loop iteration has completed propagation through the computational elements of the synchronization data path 1110. In some instances, various threads implementing different loop iterations can be executed simultaneously at the computational elements of the synchronization data path 1110 of the synchronization stream 1103 (e.g., offset branch counts).

[0178] When the threads implementing all iterations of loop 1122 have completed, flow controller 1104 will have received asynchronous messages from all threads implementing the iterations. When this has occurred, flow controller 1104 sends an asynchronous message 1118 to flow controller 1106 indicating that loop 1122 has completed. Flow controller 1106 may respond by launching threads at computing elements 1112 to implement the next portion of the programming of the reconfigurable computing structure.

[0179] Fig.11 Various exemplary elements that can be combined to implement loops in different contexts are shown. First, although Fig.11 The example of shows a loop 1122 started by a thread at the synchronization flow 1101, but in some examples, the loop can be started by another component such as a scheduling interface. In addition, although in Fig.11 In the example of , the completion of loop 1122 triggers another thread at another synchronization flow 1105, but in some examples, the completion of the loop can indicate the completion of the processing task. Therefore, the synchronization message 1118 indicating the completion of the loop can be provided to the scheduling interface in addition to or instead of being provided to the scheduling interface such as Fig.11 Another flow controller 1106 is shown.

[0180] also, Fig.11 Elements that can be used to implement nested loops are shown. For example, a thread at synchronization flow 1101 that initiates loop 1122 may be executing an iteration of an outer loop that calls loop 1122 as an inner loop. Furthermore, in some instances, for example, when a thread at synchronization flow 1101 causes synchronization flow 1103 to execute loop 1122, each thread at synchronization flow 1103 that implements an iteration of loop 1122 may cause another synchronization flow (not shown) to implement another loop.

[0181] exist Fig.11In the example of, each iteration of loop 1122 is executed using a single synchronization stream 1103. However, in some examples, the execution of loop iterations can utilize multiple synchronization streams. For example, executing a thread for an iteration of loop 1122 can include sending an asynchronous message to a second flow controller and / or a memory interface. The second flow controller starts a thread at another synchronization stream (not shown) to execute another part of the loop iteration. In this example, when the loop iteration is completed, the synchronization stream that completes the loop iteration sends an asynchronous message to the flow controller that started the loop (e.g., flow controller 1104).

[0182] Fig.12 1 is a flow chart showing an example of a process flow 1200 that can be implemented using a reconfigurable computing structure such as HTF 142 or HTF 500 to execute a loop using a last iteration flag. The process flow 1200 includes four columns 1201, 1203, 1205, 1207. Each column 1201, 1203, 1205, 1207 shows actions performed by different components of the adjacent memory computing system (including, for example, other components of the reconfigurable computing structure). Column 1201 includes actions performed by a loop start component. Column 1203 includes actions performed by a first loop flow controller (e.g., a flow controller that executes a synchronization flow of threads that start loop iteration execution). Column 1205 indicates one or more asynchronous operation components that perform asynchronous operations that may occur during the execution of the loop iteration. For example, column 1205 can indicate operations performed by a memory interface or any other asynchronous operation. Column 1207 indicates operations performed by a last loop flow controller. The last loop flow controller is a flow controller at the synchronization flow that executes the last thread of the loop iteration.

[0183] At operation 1202, a loop initiation component sends an asynchronous message 1209 to a first loop flow controller via an asynchronous structure of a reconfigurable computing structure. The asynchronous message 1209 instructs the first loop flow controller to start execution of the loop. The asynchronous message 1209 may include an iteration count indicating the number of iterations of the loop to be completed. The initiation component that sends the asynchronous message 1209 may be, for example, a scheduling interface, a memory interface, a component of another synchronization flow, etc.

[0184] At operation 1204, the first loop flow controller receives an asynchronous message 1209. At operation 1206, the first loop flow controller starts a thread at the first synchronization flow for executing the first iteration of the loop. At operation 1208, the first loop flow controller determines whether there are any remaining or remaining iterations for which threads have not yet been started. If so, the first loop flow controller returns to operation 1206 and starts the next thread at the first synchronization flow for the next iteration of the loop. For example, the next thread can be started after the branch count number of clock cycles have passed since the thread of the previous iteration of the loop was started.

[0185] If the first loop flow controller determines at operation 1208 that threads at the first synchronization flow have been started for all iterations of the loop, then in some instances, it determines at operation 1210 whether it has received an indication that all iterations of the loop have been completed. For example, the component that completed the loop iteration can send an asynchronous message to the first loop flow controller, the asynchronous message indicating that the particular loop iteration has been completed. If the first loop flow controller has not yet received an indication that all iterations of the loop are completed, it can continue to wait and recheck at operation 1210. When the first loop flow controller has determined that all iterations of the loop are completed, it can send an asynchronous message 1211 to the loop start component at operation 1212, the asynchronous message indicating that the loop has been completed. At operation 1214, the loop start component can receive message 1211. After receiving the load completion message, the loop start component can continue its processing.

[0186] At operation 1206, various threads started by the first loop flow controller for different loop iterations may prompt at least one asynchronous operation to be performed by one or more asynchronous operation components (e.g., column 1205). For example, a synchronous thread may start a memory load and / or write performed by a memory interface, or any other operation that is not completed synchronously. Based on the initial thread started at operation 1206, any suitable number of asynchronous operations and / or synchronous threads may be executed.

[0187] The completion of the asynchronous operation performed by one or more asynchronous operation components at column 1205 can be indicated by an asynchronous message 1215 provided to the last cycle flow controller. The asynchronous message 1215 can be sent to the last cycle flow controller, for example, through a memory interface, a computing element of the last cycle synchronization flow, etc. The asynchronous message 1215 can indicate that the last cycle flow controller starts the thread at the last cycle synchronization flow to complete the corresponding loop iteration. In some instances, the asynchronous message 1215 includes data that will be used by the last cycle synchronization flow to execute the last cycle thread, such as load data, or otherwise indicates that the load data has been received by the last cycle synchronization flow. For each iteration of the loop, the last cycle flow controller can receive an instance of the asynchronous message 1215. As explained herein, the message 1215 may not be received in the order in which the loop iteration is started. Because the threads started by the last cycle flow controller are synchronous, they may be completed in the order in which they are started. For example, the loop iteration corresponding to the last thread started by the last cycle flow controller will also be the last loop iteration to be completed.

[0188] The last loop flow controller may also be provided with an iteration count 1213 indicating the number of iterations in the loop. The iteration count 1213 may be provided to the last loop flow controller in any suitable manner. In some instances, the iteration count 1213 is included in a synchronization message provided by the first loop flow controller to start threads for various iterations. The iteration count may be passed, for example, between asynchronous operation components via asynchronous messages, and ultimately provided to the last loop flow controller via asynchronous messages 1215. In another instance, the iteration count 1213 is provided to the last loop flow controller when the configuration computing structure is programmed. In another instance, for example, when the scheduling interface instructs the first flow controller to start executing the first thread of the processing task, the iteration count 1213 is provided to the last loop flow controller by the scheduling interface.

[0189] At operation 1216, the last loop flow controller receives the asynchronous message 1215. At operation 1218, the last loop flow controller updates the iteration counter. The iteration counter is used to track the total number of loop iterations that have been processed by the last loop flow controller. For example, the last loop flow controller can start the counter at zero and increment the counter whenever an asynchronous message 1215 for a loop iteration is received. When the counter is equal to the iteration count 1213, the current asynchronous message 1215 corresponds to the last loop iteration to be completed. In another example, the last loop flow controller can start the counter at the iteration count 1213 and decrement the counter whenever an asynchronous message 1215 for a loop iteration is received. When the counter is equal to zero, the current asynchronous message 1215 corresponds to the last loop iteration to be completed.

[0190] At operation 1220, the last loop flow controller determines whether the current iteration (e.g., the iteration corresponding to the asynchronous message 1215) is the last iteration to be completed. For reasons described herein, the last loop iteration to be completed may not be the last loop iteration to be started. If the asynchronous message 1215 does not correspond to the last loop iteration to be completed, then at operation 1222, the last loop flow controller starts a thread at the last loop synchronization flow to continue execution of the loop iteration without setting the last iteration flag, and waits for the next asynchronous message 1215. If the asynchronous message 1215 does correspond to the last loop iteration to be completed, then at operation 1224, the last loop flow controller starts a thread to continue execution of the loop iteration with the last iteration flag set.

[0191] The computing element of the last cycle synchronization flow is programmed to execute a thread including at least one conditional operation, which is conditioned on the value of the last iteration flag. If the last iteration flag is set, such operation will be performed in the thread, and if the last iteration flag is not set, such operation will not be performed. Therefore, if the last iteration flag is not set, the thread started at operation 1222 may not perform the conditional operation. If the last iteration flag is set, the thread started at operation 1224 can perform the conditional operation.

[0192] Fig.13 is a diagram showing an example of a workflow 1300 for executing loop 1340 using a reconfigurable computing structure such as HTF 142 or HTF 500 using a last iteration flag. For example, workflow 1300 provides an example of how loop 1340 may be executed. Fig.12 1300 shows four exemplary synchronization flows 1301, 1303, 1305, 1307, which include respective flow controllers 1302, 1304, 1306, 1308 and computing elements constituting respective synchronization data paths 1310, 1312, 1314, 1316. Workflow 1300 also shows a scheduling interface 1318 and memory interfaces 1320A, 1320B.

[0193] exist Fig.13 In the example of , at least two synchronization streams 1303 and 1305 are used to perform loop 1340. Synchronization stream 1303 corresponds to Fig.12 The first loop synchronization flow of , and the execution thread to start the execution of the loop iteration. Synchronization flow 1305 execution thread to end the execution of the loop iteration, and corresponds to Fig.12 The final loop of the synchronization stream.

[0194] Loop 1340 can be started in any suitable manner. In some instances, loop 1340 is started by a thread at synchronous stream 1301. The thread at synchronous stream 1301 can implement iterations of a loop, which is an external loop relative to loop 1340. In other instances, the thread at synchronous stream 1301 implements other non-loop work. The thread at synchronous stream 1301 starts loop 1340 by providing an asynchronous message 1322A to the flow controller 1304, and the asynchronous message indicates that the flow controller 1304 starts starting threads for various iterations of loop 1340. Asynchronous message 1322A may include an iteration count indicating the number of loop iterations to be executed. In some instances, the thread at synchronous stream 1301 also provides another asynchronous message indicating the iteration count to the flow controller 1306, as described herein.

[0195] In another example, loop 1340 is started by scheduling interface 1318. For example, scheduling interface 1318 can provide asynchronous message 1322B to flow controller 1304, which indicates that flow controller 1304 starts starting threads for iterations of loop 1340. Asynchronous message 1322B can include an iteration count indicating the number of loop iterations to be performed. In some examples, scheduling interface 1318 also provides another asynchronous message indicating the iteration count to flow controller 1306, as described herein.

[0196] Iterations of loop 1340 may switch between synchronization stream 1303 (first loop synchronization stream) and synchronization stream 1305 (last loop synchronization stream) in any suitable manner. In some instances, a thread at synchronization stream 1303 directly prompts flow controller 1306 to start a corresponding thread for ending loop iterations. For example, a thread at synchronization stream 1303 may provide an asynchronous message 1324 to flow controller 1306. Asynchronous message 1324 may instruct flow controller 1306 to start a thread when a condition occurs, wherein the condition may be receiving another asynchronous message 1330A from a thread at synchronization stream 1303, receiving a synchronization message 1330B from memory interface 1320B, and / or receiving an asynchronous message 1330C from an execution element of synchronization stream 1305, indicating that load data from a read request has been received (e.g., in asynchronous message 1331 from memory interface 1320A). In some examples, a thread at the synchronous stream 1303 may send an asynchronous message 1333 to the memory interface 1320A, causing the memory interface 1320A to perform the memory operation reported to the synchronous stream 1305 via the asynchronous message 1331 or 1330B.

[0197] For example, as described herein, flow controller 1306 receives an iteration count indicating the total number of iterations of loop 1340. When flow controller 1306 launches threads for various loop iterations, it tests whether the thread to be launched corresponds to the last iteration of loop 1340 to be completed. If the thread corresponds to the last iteration to be completed, flow controller 1306 launches the thread with the last iteration flag set. The threads of synchronization flow 1305 include at least one conditional operation that is executed (or not executed) based on the value of the last iteration flag.

[0198] Fig.13 Also shown are examples of one or more conditional operations performed by the synchronization stream 1305 in the thread with the last iteration flag set. For example, the conditional operation may be or may include sending an asynchronous message 1332A to a flow controller 1308 of another synchronization stream 1307 to execute one or more additional threads to perform additional operations. In other examples, the conditional operation may be or may include sending an asynchronous message 1332B to a memory interface (e.g., memory interface 1320B) to, for example, write to a memory location, clear a memory location, read from a memory location, or perform any other suitable operation.

[0199] Fig.14A block diagram of an exemplary machine 1400 is shown, with which, in or by which any one or more of the techniques (e.g., methods) discussed herein may be implemented. As described herein, an example may include or may be operated by logic or multiple components or mechanisms in the machine 1400. A circuit system (e.g., a processing circuit system) is a collection of circuits implemented in a tangible entity of the machine 1400, which includes hardware (e.g., simple circuits, gates, logic, etc.). The membership of the circuit system may be flexible over time. The circuit system includes members that can perform specific operations individually or in combination when operating. In an example, the hardware of the circuit system may be designed in an unchangeable manner to perform specific operations (e.g., hard-wired). In an example, the hardware of the circuit system may include variably connected physical components (e.g., execution units, transistors, simple circuits, etc.), including physically modified machine-readable media (e.g., magnetic, electrical, movable placement of unchanging aggregated particles, etc.), to encode instructions for specific operations. When connecting physical components, the fundamental electrical properties of the hardware components are changed, such as from an insulator to a conductor, or vice versa. Instructions enable embedded hardware (e.g., an execution unit or a loading mechanism) to create components of a circuit system in hardware via variable connections to perform portions of a specific operation when in operation. Thus, in an example, a machine-readable media element is part of a circuit system or is communicatively coupled to other components of the circuit system when the device is in operation. In an example, any of the physical components can be used in more than one component in more than one circuit system. For example, under operation, an execution unit can be used in a first circuit in a first circuit system at one point in time and reused by a second circuit in the first circuit system, or reused by a third circuit in a second circuit system at a different time. Additional examples of these components are described herein with respect to machine 1400.

[0200] In alternative embodiments, machine 1400 can operate as a standalone device, or can be connected (e.g., networked) to other machines. In a networked deployment, machine 1400 can operate with the capabilities of a server machine, a client machine, or both in a server-client network environment. In an example, machine 1400 can act as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment. Machine 1400 can be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a network appliance, a network router, a switch or a bridge, or any machine capable of executing (sequentially or otherwise) specifying an instruction for an action to be taken by the machine. In addition, although only a single machine is shown, the term "machine" should also be deemed to include any one or more of any machine collections that execute a set (or multiple sets) of instructions to perform the methods discussed herein, such as cloud computing, software as a service (SaaS), and other computer cluster configurations.

[0201] The machine 1400 (e.g., a computer system) may include a hardware processor 1402 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 1404, a static memory 1406 (e.g., memory or storage for firmware, microcode, basic input output (BIOS), unified extensible firmware interface (UEFI), etc.), and a mass storage device 1408 (e.g., a hard drive, a tape drive, a flash storage device, or other block device), some or all of which may communicate with each other via an interconnect 1430 (e.g., a bus). The machine 1400 may further include a display device 1410, an alphanumeric input device 1412 (e.g., a keyboard), and a user interface (UI) navigation device 1414 (e.g., a mouse). In an example, the display device 1410, the input device 1412, and the UI navigation device 1414 may be a touch screen display. The machine 1400 may additionally include a mass storage device 1408 (e.g., a drive unit), a signal generating device 1418 (e.g., a speaker), a network interface device 1420, and one or more sensors 1416, such as a global positioning system (GPS) sensor, a compass, an accelerometer, or other sensors. The machine 1400 may include an output controller 1428, such as a serial (e.g., universal serial bus (USB), parallel or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate with or control one or more peripheral devices (e.g., printers, card readers, etc.).

[0202] A register of the hardware processor 1402, main memory 1404, static memory 1406, or mass storage 1408 may be or may include a machine-readable medium 1422 on which is stored one or more sets of data structures or instructions 1424 (e.g., software) that embody or are used by any one or more of the techniques or functions described herein. During execution of the instructions 1424 by the machine 1400, the instructions may also reside, in whole or in part, in any register of the hardware processor 1402, main memory 1404, static memory 1406, or mass storage 1408. In an example, one or any combination of the hardware processor 1402, main memory 1404, static memory 1406, or mass storage 1408 may constitute the machine-readable medium 1422. Although the machine-readable medium 1422 is illustrated as a single medium, the term “machine-readable medium” may include a single medium or multiple media (eg, a centralized or distributed database, or associated caches and servers) configured to store one or more instructions 1424 .

[0203] The term "machine-readable medium" may include any medium capable of storing, encoding, or carrying instructions for execution by the machine 1400 and causing the machine 1400 to perform any one or more of the techniques of the present disclosure, or any medium capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non-limiting machine-readable medium examples may include solid-state memory, optical media, magnetic media, and signals (e.g., radio frequency signals, other photon-based signals, sound signals, etc.). In an example, a non-transitory machine-readable medium includes a machine-readable medium having a set of multiple particles having a constant (e.g., stationary) mass and therefore being a composition of matter. Therefore, a non-transitory machine-readable medium is a machine-readable medium that does not include a transient propagating signal. Specific examples of non-transitory machine-readable media may include: non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0204] In an example, information stored or otherwise provided on machine-readable medium 1422 may represent instructions 1424, such as instructions 1424 themselves or a format from which instructions 1424 may be derived. Such formats from which instructions 1424 may be derived may include source code, encoded instructions (e.g., in compressed or encrypted form), packaged instructions (e.g., split into multiple packages), etc. Information representing instructions 1424 in machine-readable medium 1422 may be processed by processing circuitry into instructions to implement any of the operations discussed herein. For example, deriving instructions 1424 from information (e.g., processed by processing circuitry) may include: compiling (e.g., from source code, object code, etc.), interpreting, loading, organizing (e.g., dynamically or statically linking), encoding, decoding, encrypting, decrypting, packing, unpacking, or otherwise manipulating information into instructions 1424.

[0205] In an example, the derivation of instructions 1424 may include the assembly, compilation, or interpretation of information (e.g., by processing circuitry) to create instructions 1424 from some intermediate or pre-processed format provided by machine-readable media 1422. When provided in multiple parts, the information may be combined, unpacked, and modified to create instructions 1424. For example, the information may be in multiple compressed source code packages (or object code, or binary executable code, etc.) on one or more remote servers. The source code packages may be encrypted when transmitted over the network, and may be decrypted, decompressed, assembled (e.g., linked) as necessary, and compiled or interpreted at the local machine (e.g., compiled or interpreted into a library, a stand-alone executable file, etc.) and executed by the local machine.

[0206] The instructions 1424 may further be sent or received over a communication network 1426 using a transmission medium via the network interface device 1420 using any of a variety of transmission protocols (e.g., frame relay, Internet Protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Exemplary communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), a mobile telephone network (e.g., a cellular network), a plain old telephone (POTS) network, and a wireless data network (e.g., the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards (known as ), the IEEE 802.16 family of standards (known as )), IEEE 802.15.4 family of standards, peer-to-peer (P2P) networks, etc. In an example, the network interface device 1420 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas to connect to the network 1426. In an example, the network interface device 1420 may include a set of multiple antennas to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technology. The term "transmission media" should be deemed to include any intangible media that can store, encode, or carry instructions for execution by the machine 1400, and includes digital or analog communication signals or other intangible media to facilitate the communication of such software. Transmission media is machine-readable media.

[0207] In order to better illustrate the methods and apparatus described herein, a set of non-limiting exemplary embodiments are set forth below as numbered examples.

[0208] The above detailed description includes reference to the accompanying drawings, which form a part of the detailed description. The accompanying drawings show by means of illustration specific embodiments in which the present invention can be put into practice. These embodiments are also referred to as "examples" in this article. Such examples may include elements other than those shown or described. However, the inventors also contemplate that only examples of those elements shown or described are provided. In addition, the inventors also contemplate using any combination or arrangement of those elements shown or described relative to a specific example (or one or more aspects thereof) or relative to other examples (or one or more aspects thereof) shown or described herein (or one or more aspects thereof).

[0209] In this document, as is common in patent documents, the terms "a / an" are used to include one or more than one, regardless of any other examples or uses of "at least one" or "one or more". In this document, unless otherwise indicated, the term "or" is used to refer to a non-exclusive or, such that "A or B" may include "A but not B", "B but not A", and "A and B". In the appended claims, the terms "including" and "in which" are used as the plain English equivalents of the corresponding terms "comprising" and "wherein". In addition, in the appended claims, the terms "including" and "comprising" are open-ended, that is, in addition to the elements listed after such terms in the claims, systems, devices, articles or processes that include these elements are still considered to fall within the scope of the claim. In addition, in the appended claims, the terms "first", "second", and "third", etc. are used only as labels and are not intended to impose numerical requirements on their objects.

[0210] The above description is intended to be illustrative, not restrictive. For example, the examples described above (or one or more aspects thereof) can be used in combination with each other. Other embodiments can be used by those of ordinary skill in the art after consulting the above description. The abstract is submitted with the following understanding: it will not be used to interpret or limit the scope or meaning of the claims. Similarly, in the above specific embodiments, various characteristics can be grouped together to simplify the present disclosure. This situation should not be interpreted as expecting that the disclosed features that are not claimed for protection are necessary for any claim. In fact, the subject matter of the present invention may be less than all the features of the specific disclosed embodiments. Therefore, the attached claims are incorporated into the specific embodiments accordingly, wherein each claim exists independently as a separate embodiment, and it is expected that such embodiments can be combined with each other in various combinations or arrangements. The scope of the present invention should be determined by reference to the complete scope of the attached claims and the equivalents granted by such claims.

Claims

1. A method for a memory computing system, the method comprising: launching, by a first flow controller of a first synchronization flow, a first thread at the first synchronization flow, the first thread at the first synchronization flow executing a first portion of a first iteration of a loop, the loop having a plurality of iterations, the first synchronization flow being configured by the memory computing system including a set of a plurality of flow controllers and a set of a plurality of computing elements; After launching the first thread, launching at least one thread to execute at least a portion of a second iteration of the loop; receiving, by a second flow controller of a second synchronous flow, a first asynchronous message instructing the second flow controller to start a first thread at the second synchronous flow to execute a second portion of the first iteration of the loop, the second portion of the first iteration being executed after the first portion of the first iteration; determining, by the second flow controller, that the first iteration of the loop is a last iteration of the loop to be performed; In response to determining that the first iteration of the loop is the last iteration of the loop to be executed, launching, by the second flow controller, the first thread at the second synchronization flow, the launching comprising providing a first synchronization message to a first computing element of the second synchronization flow, the first synchronization message comprising a last iteration flag indicating that the first iteration is the last iteration of the loop to be executed; as well as Executing the first thread at the second synchronization flow, the execution comprising: Determining that the first synchronization message includes the last iteration flag; as well as In response to determining that the first synchronization message includes the last iteration flag, performing a first conditional operation based at least in part on an output produced by the first iteration of the loop and at least in part on an output produced by the second iteration of the loop.

2. The method of claim 1, further comprising executing a second thread at the second synchronization flow, the executing comprising: providing a second synchronization message to the first computing element of the second synchronization flow; determining that the second synchronization message does not include the last iteration flag; and In response to determining that the second synchronization message does not include the last iteration flag, omitting the first conditional operation.

3. The method according to claim 1, further comprising: receiving, by the second flow controller, an iteration count indicating a number of iterations of the loop; and The number of threads previously launched by the second flow controller is compared to the iteration count. The method of claim 3 , wherein the first asynchronous message includes the iteration count.

5. The method according to claim 3, further comprising: Sending, by the scheduling interface of the memory computing system, a second asynchronous message to the first flow controller, the second asynchronous message instructing the first flow controller to start execution of the loop; and A third asynchronous message is sent by the scheduling interface to the second flow controller, where the third asynchronous message includes the iteration count.

6. The method of claim 3, further comprising programming the memory computing system to include the first synchronization stream and the second synchronization stream, the programming comprising providing the iteration count to the second stream controller.

7. The method of claim 3, further comprising sending the first asynchronous message to the second flow controller by at least a portion of the first synchronous flow.

8. The method according to claim 1, further comprising: After starting the first thread at the first synchronization flow, starting a second thread at the first synchronization flow by the first flow controller, the second thread at the first synchronization flow executing a first part of a second iteration of the loop; receiving, by the second flow controller, a second asynchronous message prior to receiving the first asynchronous message, the second asynchronous message instructing the second flow controller to start a second thread at the second synchronous flow to execute a second portion of a second iteration of the loop; determining, by the second flow controller, that the second iteration of the loop is not a last iteration of the loop to be performed; as well as A second thread at the second synchronization flow is started by the second flow controller, the starting comprising providing a second synchronization message that does not include the last iteration flag to the first computing element of the second synchronization flow.

9. A memory computing system comprising: a first synchronization stream, the first synchronization stream comprising a first stream controller and at least one computing element, the first stream controller being programmed to launch a first thread at the first synchronization stream to execute a first portion of a first iteration of a loop, the loop having a plurality of iterations, and after launching the first thread at the first synchronization stream, launch at least one thread to execute at least a portion of a second iteration of the loop; as well as A second synchronization flow, the second synchronization flow comprising a second flow controller and at least one computing element, the second flow controller being programmed to perform operations comprising: receiving a first asynchronous message, the first asynchronous message instructing the second flow controller to start a first thread at the second synchronous flow to execute a second portion of the first iteration, the second portion of the first iteration being executed after the first portion of the first iteration; as well as determining that the first iteration of the loop is a last iteration of the loop to be performed; In response to determining that the first iteration of the loop is the last iteration of the loop to be executed, starting the first thread at the second synchronization stream, the starting comprising providing a first synchronization message to a first computation unit of the second synchronization stream, the first synchronization message comprising a last iteration flag indicating that the first iteration is the last iteration of the loop to be executed; as well as Executing the first thread at the second synchronization flow, the execution comprising: determining that the first synchronization message includes the last iteration flag; and In response to determining that the first synchronization message includes the last iteration flag, performing a first conditional operation, The first conditional operation is based at least in part on an output produced by the first iteration of the loop and at least in part on an output produced by the second iteration of the loop.

10. The memory computing system of claim 9, the second synchronization stream being programmed to execute the first thread at the second synchronization stream at least in part by performing operations further comprising: providing a second synchronization message to the first computing element of the second synchronization flow; Determining that the second synchronization message does not include the last iteration flag; as well as In response to determining that the second synchronization message does not include the last iteration flag, omitting the first conditional operation.

11. The memory computing system of claim 9, wherein the second flow controller is programmed to perform operations further comprising: receiving an iteration count indicating a number of iterations of the loop; and The number of threads previously launched by the second flow controller is compared to the iteration count.

12. The memory computing system of claim 11, wherein the first asynchronous message includes the iteration count.

13. The memory computing system of claim 11, further comprising a dispatch interface programmed to perform operations comprising: sending a second asynchronous message to the first flow controller, the second asynchronous message instructing the first flow controller to start execution of the loop; and A third asynchronous message is sent to the second flow controller, the third asynchronous message including the iteration count.

14. The memory computing system of claim 11, programmed to include the iteration count at the second flow controller.

15. The memory computing system of claim 11, the first synchronization flow being programmed to execute the first thread at the first synchronization flow at least in part by performing operations including sending the first asynchronous message to the second flow controller by at least a portion of the first synchronization flow.

16. The memory computing system of claim 9, wherein the first flow controller is further programmed to perform operations comprising: After starting the first thread at the first synchronization flow, starting a second thread at the first synchronization flow, the second thread at the first synchronization flow executing a first portion of a second iteration of the loop; and The second stream controller is further programmed to perform operations comprising: before receiving the first asynchronous message, receiving a second asynchronous message, the second asynchronous message instructing the second flow controller to start a second thread at the second synchronous flow to execute a second portion of a second iteration of the loop; determining that the second iteration of the loop is not a last iteration of the loop to be performed; as well as A second thread at the second synchronization flow is started, the starting comprising providing a second synchronization message that does not include the last iteration flag to the first computing element of the second synchronization flow.

17. A non-transitory machine-readable medium comprising instructions stored thereon, which when executed at a reconfigurable computing structure causes the reconfigurable computing structure to perform operations comprising: launching, by a first flow controller of a first synchronization flow, a first thread at the first synchronization flow, the first thread at the first synchronization flow executing a first portion of a first iteration of a loop, the loop having a plurality of iterations, the first synchronization flow being configured by a reconfigurable computing structure including a set of a plurality of flow controllers and a set of a plurality of computing elements; After launching the first thread, launching at least one thread to execute at least a portion of a second iteration of the loop; receiving, by a second flow controller of a second synchronous flow, a first asynchronous message instructing the second flow controller to start a first thread at the second synchronous flow to execute a second portion of the first iteration, the second portion of the first iteration being executed after the first portion of the first iteration; determining, by the second flow controller, that the first iteration of the loop is a last iteration of the loop to be performed; In response to determining that the first iteration of the loop is a last iteration of the loop to be executed, launching, by the second stream controller, the first thread at the second synchronization stream, the launching comprising providing a first synchronization message to a first computing element of the second synchronization stream, the first synchronization message comprising a last iteration flag indicating that the first iteration is the last iteration of the loop to be executed; as well as Executing the first thread at the second synchronization flow, the execution comprising: Determining that the first synchronization message includes the last iteration flag; as well as In response to determining that the first synchronization message includes the last iteration flag, performing a first conditional operation based at least in part on an output produced by the first iteration of the loop and at least in part on an output produced by the second iteration of the loop.

Images