Dynamic decomposition and thread allocation

Abstract

This application relates to dynamic decomposition and thread allocation. Apparatuses and techniques for thread scheduling control and memory partitioning in a processor are described herein. One apparatus includes a hardware interface configured to receive a first request to execute a first thread, the first request including an indication of a workload, and processing circuitry configured to determine the workload to generate a metric based at least in part on the indication, compare the metric to a threshold to determine that the metric exceeds the threshold, split the workload into a set of sub-workloads consisting of a predefined number of identical portions from the workload based at least in part on the comparison, create a second request to execute a second thread, the second request including a first member of the set of sub-workloads, and process a second member of the set of sub-workloads in the first thread.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 132,754, filed on December 31, 2020, which is incorporated herein by reference in its entirety. Technical Field

[0003] This application involves dynamic decomposition and thread allocation. Background Technology

[0004] For example, various computer architectures based on the von Neumann architecture typically use shared memory for data, a bus for accessing shared memory, an arithmetic unit, and a program control unit. However, moving data between the processor and memory requires significant time and energy, which constrains the performance and capacity of the computer system. Given these limitations, new computing architectures and devices are expected to improve computing performance beyond the convention of transistor expansion (i.e., Moore's Law). Summary of the Invention

[0005] One embodiment of this disclosure provides an apparatus comprising: a hardware interface configured to receive a first request to execute a first thread, the first request including an indication of workload; and a processing circuitry configured to: determine the workload to generate a metric at least partially based on the indication; compare the metric with a threshold to determine that the metric exceeds the threshold; divide the workload into a set of sub-workloads consisting of a predefined number of identical portions from the workload, at least partially based on the comparison; create a second request to execute a second thread, the second request including a first component of the set of sub-workloads; and a second component to process the set of sub-workloads in the first thread.

[0006] Another embodiment of this disclosure provides a method comprising: receiving a first request to execute a first thread, the first request including an indication of workload; determining the workload to generate a metric at least in part based on the indication; comparing the metric with a threshold to determine that the metric exceeds the threshold; dividing the workload into a set of sub-workloads consisting of a predefined number of identical portions from the workload, at least in part based on the comparison; creating a second request to execute a second thread, the second request including a first component of the set of sub-workloads; and processing a second component of the set of sub-workloads in the first thread.

[0007] Another embodiment of this disclosure provides an apparatus comprising: a hardware interface configured to receive a first request to execute a first thread, the first request including an indication of workload; and a processing circuitry configured to: recursively divide the workload into a plurality of sub-workloads and cause a sub-thread to process each of the plurality of sub-workloads until the sub-workloads are less than a threshold size; and process the sub-workloads less than the threshold size in the sub-threads. Attached Figure Description

[0008] This disclosure will be more fully understood from the specific embodiments given below and from the accompanying drawings of the various embodiments of this disclosure. However, the drawings should not be construed as limiting this disclosure to the specific embodiments, but are provided for explanation and understanding only.

[0009] To facilitate easy identification of any particular element or action, the highest or first few most significant digits in the reference element symbol refer to the figure number in which the element is first introduced.

[0010] Figure 1 This section generally describes a first example of a first memory computing device within the context of a memory computing system according to an embodiment.

[0011] Figure 2 Examples of memory subsystems of memory computing devices according to embodiments are generally described.

[0012] Figure 3 This section generally illustrates examples of programmable atomic units for a memory controller according to embodiments.

[0013] Figure 4 An example of a hybrid thread processor (HTP) accelerator for a memory computing device according to an embodiment is described.

[0014] Figure 5 An example of a representation of a hybrid thread structure (HTF) of a memory computing device according to an embodiment is described.

[0015] Figure 6A This section provides a general description of an example of a chiplet system according to an embodiment.

[0016] Figure 6B The general description shows the information from Figure 6A A block diagram of the components in an example chiplet system.

[0017] Figure 7 This section generally illustrates examples of chiplet-based implementations of the memory computing device according to embodiments.

[0018] Figure 8 The illustration shows an example of a memory computing device chiplet according to an embodiment.

[0019] Figure 9 This is a table illustrating the fiber creation instructions according to the embodiments.

[0020] Figure 10 This is a list of pseudocode illustrating the divide-and-conquer process according to the embodiments.

[0021] Figure 11 This is another list of codes illustrating the divide-and-conquer process according to the embodiments.

[0022] Figure 12 This is a diagram illustrating the thread call tree according to an embodiment.

[0023] Figure 13 This is a flowchart of an example of a method for thread execution control according to an embodiment.

[0024] Figure 14 A block diagram illustrating any one or more of the techniques (e.g., methods) discussed in this article, or an instance machine in which they are implemented, implemented, or carried out. Detailed Implementation

[0025] Recent advances in materials, devices, and integration technologies can be leveraged to provide memory-centric computing topologies. Such topologies can improve computational efficiency and workload handling, for example, for applications constrained by size, weight, or power requirements. Topologies can facilitate low-latency computations near or within memory or other data storage elements. The approach is particularly well-suited for various computationally intensive operations with sparse lookups, such as in transform computations (e.g., Fast Fourier Transform (FFT) calculations) or in applications such as neural networks or artificial intelligence (AI), financial analysis, or simulation or modeling (e.g., for computational fluid dynamics (CFD), Engineer's Enhanced Acoustic Simulator (EASE), SPICE (Simultaneous Integrated Circuit Priority Project), and others).

[0026] The systems, apparatuses, and methods discussed herein may include or use memory computing systems with processors or processing capabilities provided in, near, or integrated with memory or data storage components. Such systems are generally referred to herein as near-memory computing (CNM) systems. CNM systems may be node-based systems in which individual nodes in the system are coupled using a system extension architecture. Each node may include or use a dedicated or general-purpose processor and a user-accessible accelerator with a custom computing architecture that facilitates intensive operations, particularly in environments where high cache miss rates are expected.

[0027] In this example, each node in a CNM system can have one or more host processors. Within each node, a dedicated hybrid-threaded processor can occupy discrete endpoints of the on-chip network. The hybrid-threaded processor can access some or all of the memory in a specific node of the system, or it can access memory across a network of multiple nodes via a system extension architecture. Custom compute architectures or hybrid-threaded architectures can have their own processors or accelerators at each node and can operate with higher bandwidth than hybrid-threaded processors. Different nodes in a near-memory computing system can be configured differently, for example, with different compute capabilities, different types of memory, different interfaces, or other differences. However, nodes can be co-coupled to share data and compute resources within a defined address space.

[0028] In this example, near-memory computing systems or nodes within a system can be user-configurable for custom operations. Users can provide instructions using high-level programming languages ​​such as C / C++, which can be compiled and directly mapped to the dataflow architecture of one or more nodes in the system or CNM system. That is, nodes in the system can contain hardware blocks (e.g., memory controllers, atomic units, other client accelerators, etc.) that can be configured to directly implement or support user instructions to enhance system performance and reduce latency.

[0029] In practice, near-memory computing systems are particularly well-suited for implementing hierarchical structures of instructions and nested loops (e.g., two, three, or more deep loops or multidimensional loops). Standard compilers can be used to accept high-level language instructions and compile them directly into one or more dataflow architectures within a node. For example, nodes in the system can contain hybrid threading accelerators. Hybrid threading accelerators can execute in the user space of the CNM system and can launch their own threads or sub-threads that can operate in parallel. Each thread can be mapped to a different loop iteration to support multidimensional loops. Using the ability to launch such nested loops, along with other capabilities, CNM systems can achieve significant time savings and latency improvements for computationally intensive operations.

[0030] Near-memory computing systems, or nodes or components of near-memory computing systems, may include or utilize various memory devices, controllers, interconnects, and other components. In some examples, the system may include various interconnect nodes, and nodes or groups of nodes may be implemented using chiplets. Chiplets are an emerging technology for integrating various processing functionalities. Typically, a chiplet system consists of discrete chips (e.g., integrated circuits (ICs) on different substrates or dies) integrated on an interposer and packaged together. This arrangement differs from a single chip (e.g., an IC) containing disparate device blocks (e.g., intellectual property (IP) blocks) on a single substrate (e.g., a single die), such as a system-on-a-chip (SoC) or a discrete packaged device integrated on a board. Generally, chiplets offer manufacturing benefits compared to single-die chips, including higher yields or reduced development costs. The following discussion... Figure 6A and Figure 6B Examples of chiplet systems that can be broadly described may include those used in near-memory computing systems.

[0031] In some computational tasks, hundreds or even thousands of threads can be used for parallel processing. At this scale, such as with thousands of threads, the amount of work that can be parallelized is a key factor in achieving performance. Another consideration is how quickly threads can be started.

[0032] To process large lists in parallel, one strategy is to have a main thread dedicated to dividing the list into chunks and creating child threads to process those chunks. This can be done with very little scheduling overhead, and the loop can iterate quickly. However, this strategy has two main drawbacks. First, if there are many chunks to start, a single source of work means that the time required to start each unit of work becomes significant. Second, this linear scheduling of tasks also puts pressure on the task interface because of the single point of joining and releasing child threads.

[0033] A more efficient mechanism is needed to process large tasks in parallel. The system and method described in this paper use a divide-and-conquer strategy to partition the work. Instead of a single main thread creating all worker threads, each thread participates in scheduling. If the workload assigned to a thread is larger than a certain threshold-sized chunk, the thread first attempts to split the work in half. The thread creates a new thread to handle one half of the work and continues processing the remaining half. If thread creation fails (e.g., all system threads are already running), the thread will process a threshold-sized chunk of work before attempting to split it again. This mechanism provides a logarithmic scheduling tree for threads.

[0034] Efficiency can be further improved by leveraging the busy-failure thread creation application programming interface (API) provided by the Hybrid Thread Processor (HTP) runtime. The logarithmic thread creation tree also serves as a log-depth return path. This must be done if a single result from each subtask is required. However, if no result is needed (e.g., all results are already in memory), we can utilize a special "no-return" variant of thread creation. A thread with no return is automatically terminated by the host interface upon completion of execution. This means it can be freely reused, and the calling code does not need to handle adding and releasing threads. This greatly increases the flexibility of work partitioning, as threads can easily continue to loop as new work is generated.

[0035] Using this strategy, work continues to be processed and dynamic thread creation does not stall. An additional advantage is that the scheduling mechanism results in a logarithmic depth of balanced tree scheduling with an exponential number of threads running at several levels. Further processing advantages can be obtained by using "busy-failed" threads that provide fast dispatch and return to all resources in the system, and "no-return" thread variants that enable fast termination and reuse of system threads.

[0036] Near-memory computing systems, or nodes or components of near-memory computing systems, may include or utilize various memory devices, controllers, interconnect structures, and other elements. In some instances, the system may include computing architectures with various interconnect nodes, and nodes or groups of nodes may be implemented using chiplets. Chiplets are an emerging technology for integrating various processing functionalities. Typically, a chiplet system consists of discrete chips (e.g., integrated circuits (ICs) on different substrates or dies) integrated on an interposer and packaged together. This arrangement differs from a single chip (e.g., an IC) containing disparate device blocks (e.g., intellectual property (IP) blocks) on a single substrate (e.g., a single die), such as a system-on-a-chip (SoC) or a discrete packaged device integrated on a board. Generally, chiplets offer better performance (e.g., lower power consumption, reduced latency) than discrete packaged devices, and chiplets offer greater manufacturing benefits than single-die chips. These manufacturing benefits may include higher yields or reduced development costs and time.

[0037] Figure 1 This section describes a first instance of a near-memory computing system, or CNM system 102. The instance of CNM system 102 comprises multiple different memory computing nodes, each potentially containing various near-memory computing devices. Each node in the system operates within its own operating system (OS) domain (e.g., Linux, etc.). In this instance, the nodes may coexist within a common OS domain of CNM system 102.

[0038] Figure 1 Examples include an instance of the first memory compute node 104 of the CNM system 102. The CNM system 102 may have multiple nodes coupled using an extension structure 106, such as different examples including the first memory compute node 104. In the example, the architecture of the CNM system 102 may support scaling to up to n different memory compute nodes (e.g., n = 4096) using the extension structure 106. As discussed further below, each node in the CNM system 102 may be a combination of multiple devices.

[0039] CNM system 102 may include a global controller for various nodes in the system, or a specific memory compute node in the system may optionally be used as a host or controller for one or more other memory compute nodes in the same system. Therefore, the individual nodes in CNM system 102 may be configured similarly or differently.

[0040] In this example, each node in CNM system 102 may include a host system using a specified operating system. The operating system may be common or different across the nodes in CNM system 102. Figure 1In this example, the first memory computing node 104 includes a host system 108, a first switch 110, and a first memory computing device 112. The host system 108 may include a processor, such as an x86, ARM, RISC-V, or other type of processor. The first switch 110 may be configured to facilitate communication between or within devices of the first memory computing node 104 or the CNM system 102, for example, using a dedicated or other communication protocol commonly referred to herein as the Chip-to-Chip Protocol Interface (CTCPI). That is, CTCPI may include a dedicated interface unique to the CNM system 102, or may include or use other interfaces such as a Compute High-Speed ​​Link (CXL) interface, a Peripheral Component Interconnect High-Speed ​​(PCIe) interface, or a Chiplet Protocol Interface (CPI). The first switch 110 may include a switch configured to use CTCPI. For example, the first switch 110 may include a CXL switch, a PCIe switch, a CPI switch, or other types of switches. In this example, the first switch 110 may be configured to couple endpoints with different configurations. For example, the first switch 110 can be configured to convert packet formats, for example, between PCIe and CPI formats and others.

[0041] The CNM system 102 is described herein with various example configurations, such as a system including nodes, each of which may include various chips (e.g., processors, switches, memory devices, etc.). In one example, the first memory computing node 104 in the CNM system 102 may include various chips implemented using chiplets. In the chiplet-based configuration of the CNM system 102 discussed below, inter-chiplet communication and additional intra-system communication may use a CPI network. The CPI network described herein is an example of CTCPI, i.e., a chiplet-specific implementation of CTCPI. Therefore, the following structures, operations, and functionalities of CPI are equally applicable to structures, operations, and functionalities that could otherwise be implemented using non-chiplet-based CTCPI implementations. Unless otherwise expressly indicated, any discussion of CPI herein also applies to CTCPI.

[0042] The CPI interface includes a packet-based network that supports virtual channels for flexible and high-speed interaction between chiplets, and may include portions of the first memory compute node 104 or CNM system 102. CPI enables bridging of intra-chiplet networks to a wider chiplet network. For example, the Advanced Scalable Interface (AXI) is a specification for intra-chip communication. However, the AXI specification covers various physical design options, such as the number of physical channels, signal timing, power, etc. Within a single chip, these options are typically selected to meet design goals, such as power consumption and speed. However, to achieve the flexibility of chiplet-based memory compute systems, adapters using CPI can interface between various AXI design options that can be implemented in individual chiplets. By implementing physical-to-virtual channel mapping and encapsulating time-based signaling with packet protocols, CPI can be used to bridge intra-chiplet networks, such as within a specific memory compute node, across a wider chiplet network, such as across the first memory compute node 104 or across CNM system 102.

[0043] CNM system 102 is scalable to include multi-node configurations. That is, multiple different examples of the first memory compute node 104 or other memory compute nodes with different configurations can be coupled using extension structure 106 to provide an extended system. Each of the memory compute nodes can run its own operating system and can be configured to coordinate the use of system resources collectively.

[0044] exist Figure 1 In this example, the first switch 110 of the first memory compute node 104 is coupled to the extension structure 106. The extension structure 106 provides a switch (e.g., a CTCPI switch, PCIe switch, CPI switch, or other switch) that facilitates communication between and among different memory compute nodes. In this example, the extension structure 106 may facilitate communication between nodes in the Partitioned Global Address Space (PGAS).

[0045] In this example, a first switch 110 from a first memory computing node 104 is coupled to one or more different memory computing devices, such as a first memory computing device 112. The first memory computing device 112 may include a chiplet-based architecture referred to herein as a near-memory computing (CNM) chiplet. A packaged version of the first memory computing device 112 may include, for example, one or more CNM chiplets. The chiplets may use CTCPI communication coupling to achieve high bandwidth and low latency.

[0046] exist Figure 1 In an example, the first memory computing device 112 may include an on-chip network (NOC) or a first NOC 118. Typically, an NOC is an interconnect network within a device that connects a specific set of endpoints. Figure 1In this context, the first NOC 118 can provide communication and connectivity between various memories, computing resources and ports of the first memory computing device 112.

[0047] In an example, the first NOC 118 may include a folded Clos topology, such as within each instance of a memory computing device or as a grid of multiple memory computing devices in a coupling node. Clos topology (e.g., using multiple smaller-radix cross switches to provide the functionality associated with a higher-radix cross switch topology) offers various benefits. For example, Clos topology can exhibit consistent latency and bi-directional bandwidth across NOCs.

[0048] The first NOC 118 can include various different switch types, including hub switches, edge switches, and endpoint switches. Each of these switches can be configured as a crossbar switch that provides substantially uniform latency and bandwidth between input and output nodes. In examples, endpoint switches and edge switches can contain two separate crossbar switches: one for traffic heading to the hub switch and the other for traffic leaving the hub switch. Hub switches can be configured as a single crossbar switch that switches all inputs to all outputs.

[0049] In this example, hub switches may each have multiple ports (e.g., four or six ports each), depending on whether a particular hub switch participates in inter-chip communication. The number of hub switches participating in inter-chip communication can be set by inter-chip bandwidth requirements.

[0050] The first NOC 118 can support a variety of payloads between computing elements and memory (e.g., payloads from 8 to 64 bytes; other payload sizes can be used similarly). In some instances, the first NOC 118 can be optimized for relatively small payloads (e.g., 8 to 16 bytes) to efficiently handle access to sparse data structures.

[0051] In this example, the first NOC 118 may be coupled to an external host via a first physical layer interface 114, a PCIe slave module 116 or endpoint, and a PCIe master module 126 or root port. That is, the first physical layer interface 114 may include an interface that allows an external host processor to couple to the first memory computing device 112. The external host processor may optionally couple to one or more different memory computing devices, for example, using a PCIe switch or other native protocol switches. Communicating with an external host processor via a PCIe-based switch limits device-to-device communication to communication supported by the switch. In contrast, communication via a memory computing device native protocol switch, for example using CTCPI, allows for more comprehensive communication between or within different memory computing devices, including support for partitioning the global address space, such as for creating worker threads and sending events.

[0052] In this example, the CTCPI protocol may be used by a first NOC 118 in a first memory computing device 112, and a first switch 110 may include a CTCPI switch. The CTCPI switch may allow CTCPI packets to be transmitted from a source memory computing device, such as the first memory computing device 112, to different destination memory computing devices (e.g., on the same or other nodes), without requiring conversion to another packet format.

[0053] In one example, the first memory computing device 112 may include an internal host processor 122. The internal host processor 122 may be configured to communicate with the first NOC 118 or other components or modules of the first memory computing device 112, for example, using an internal PCIe master module 126, which may help eliminate time-consuming and energy-intensive physical layers. In one example, the internal host processor 122 may be based on a RISC-VISA processor and may communicate externally to the first memory computing device 112, for example, with other storage devices, networks, or other peripherals connected to the first memory computing device 112, using a first physical layer interface 114. The internal host processor 122 may control the first memory computing device 112 and may act as a proxy for operating system-related functionalities. The internal host processor 122 may include a relatively small number of processing cores (e.g., 2 to 4 cores) and a host memory device 124 (e.g., including DRAM modules).

[0054] In this example, the internal host processor 122 may include a PCI root port. When the internal host processor 122 is in use, one of its root ports may be connected to a PCIe slave module 116. Another of the root ports of the internal host processor 122 may be connected to a first physical layer interface 114 to provide, for example, communication with an external PCI peripheral device. When the internal host processor 122 is deactivated, the PCIe slave module 116 may be coupled to the first physical layer interface 114 to allow an external host processor to communicate with a first NOC 118. In an example of a system with multiple memory computing devices, the first memory computing device 112 may be configured to function as a system host or controller. In this example, the internal host processor 122 may be in use, and other examples of internal host processors in corresponding other memory computing devices may be deactivated.

[0055] The internal host processor 122 can be configured, for example, to allow host initialization when the first memory computing device 112 is powered on. In an example, the internal host processor 122 and its associated data paths (e.g., including the first physical layer interface 114, PCIe slave module 116, etc.) can be configured to the first memory computing device 112 from input pins. One or more of these pins can be used to enable or disable the internal host processor 122 and configure the PCI (or other) data paths accordingly.

[0056] In this example, the first NOC 118 may be coupled to the extension structure 106 via the extension structure interface module 136 and the second physical layer interface 138. The extension structure interface module 136 or SIF facilitates communication between the first memory computing device 112 and a device space such as a Partitioned Global Address Space (PGAS). The PGA may be configured such that a particular memory computing device, such as the first memory computing device 112, may access memory or other resources on different memory computing devices (e.g., on the same or different nodes) using a load / store paradigm. Various scalable architecture technologies may be used, including CTCPI, CPI, Gen-Z, PCI, or Ethernet bridged via CXL. The extension structure 106 may be configured to support various packet formats. In this example, the extension structure 106 supports out-of-order packet communication or supports ordered packets, for example, by using path identifiers to extend bandwidth across multiple equivalent paths. The extension structure 106 may typically support remote operations such as remote memory reads, writes, and other built-in atoms, remote memory atoms, remote memory computing device send events, and remote memory computing device call and return operations.

[0057] In this example, the first NOC 118 may be coupled to one or more different memory modules, such as including a first memory device 128. The first memory device 128 may include various memory devices, such as LPDDR5 or GDDR6. Figure 1 In one example, the first NOC 118 may coordinate communication with the first memory device 128 via a memory controller 130, which may be dedicated to a specific memory module. In this example, the memory controller 130 may include a memory module cache and an atomic operation module. The atomic operation module may be configured to provide relatively high-volume atomic operators, such as integer and floating-point operators. The atomic operation module may be configured to apply its operators to data within a memory module cache (e.g., including an SRAM-side cache), thereby allowing back-to-back atomic operations using the same memory location with minimal volume degradation.

[0058] The memory module cache can provide storage for frequently accessed memory locations, such as eliminating the need to re-access the first memory device 128. In one example, the memory module cache can be configured to cache data used only by a specific instance of the memory controller 130. In another example, the memory controller 130 includes a DRAM controller configured to interface with, for example, the first memory device 128 containing a DRAM device. The memory controller 130 can provide access scheduling, bit error management, and other functions.

[0059] In this example, the first NOC 118 can be coupled to the hybrid thread processor (HTP 140), the hybrid thread architecture (HTF 142), and the host interface and dispatch module (HIF 120). HIF 120 can be configured to facilitate access to host-based command request and response queues. In this example, HIF 120 can dispatch new threads to execute on the processors or computing elements of the HTP 140 or HTF 142. In this example, HIF 120 can be configured to maintain workload balancing across the HTP 140 and HTF 142 modules.

[0060] The hybrid-threaded processor, or HTP 140, may include accelerators, such as those based on the RISC-V instruction set. The HTP 140 may include a highly threaded, event-driven processor, where threads can execute in a single instruction rotation to, for example, maintain high instruction throughput. The HTP 140 includes relatively few custom instructions supporting low-overhead threading capabilities, event send / receive, and shared-memory atomic operators.

[0061] The hybrid thread architecture or HTF 142 may include accelerators, such as non-von Neumann, coarse-grained, reconfigurable processors. The HTF 142 may be optimized for high-level language operations and data types (e.g., integers or floating-point). In some instances, the HTF 142 may support dataflow computation. The HTF 142 may be configured to utilize virtually all memory bandwidth available on the first memory computing device 112, for example, when executing a memory-limited computational kernel.

[0062] The HTP and HTF accelerators of the CNM System 102 can be programmed using various high-level structured programming languages. For example, the HTP and HTF accelerators can be programmed in C / C++, such as using the LLVM compiler framework. The HTP accelerator can leverage open-source compiler environments, such as those with various additional custom instruction sets configured to improve memory access efficiency, provide message passing mechanisms, and manage events, among other things. In one example, the HTF accelerator can be designed to program the HTF 142 using a high-level programming language, and the compiler can generate simulator configuration files or binaries that run on the HTF 142 hardware. The HTF 142 provides a mid-level language to express algorithms precisely and concisely, while hiding the configuration details of the HTF accelerator itself. In another example, the HTF accelerator toolchain can use an LLVM front-end compiler and an LLVM intermediate representation (IR) to interface with the HTF accelerator back-end.

[0063] Figure 2An example of a memory subsystem 200 of a memory computing device according to an embodiment is generally described. The example of memory subsystem 200 includes a controller 202, a programmable atom unit 208, and a second NOC 206. The controller 202 may include or use the programmable atom unit 208 to perform operations using information in the memory device 204. In this example, the memory subsystem 200 includes components from… Figure 1 A portion of the first memory computing device 112, such as a portion including the first NOC 118 or memory controller 130.

[0064] exist Figure 2 In this example, the second NOC 206 is coupled to the controller 202, and the controller 202 may include a memory control module 210, a local cache module 212, and a built-in atom module 214. In this example, the built-in atom module 214 may be configured to handle relatively simple, single-loop integer atoms. The built-in atom module 214 can perform atoms with the same processing volume as, for example, normal memory read or write operations. In this example, an atomic memory operation may include a combination of storing data into memory, performing an atomic memory operation, and then responding to loaded data from memory.

[0065] A local cache module 212 (e.g., which may include an SRAM cache) may be provided to help reduce latency at repeatedly accessed memory locations. In an example, the local cache module 212 may provide a read buffer for sub-memory line access. The local cache module 212 may be particularly beneficial for computing elements with relatively small or no data cache.

[0066] Memory control module 210 (e.g., may include a DRAM controller) can provide low-level request buffering and scheduling to provide efficient access to memory device 204, which may contain DRAM devices. In an example, memory device 204 may include or use GDDR6 DRAM devices, for example, with a density of 16Gb and a peak bandwidth of 64Gb / s. Other devices may be used similarly.

[0067] In this example, programmable atomic unit 208 may include single-loop or multi-loop operators, such as those configured to perform integer addition or more complex multi-instruction operations like Bloom filter insertion. In this example, programmable atomic unit 208 may be configured to perform load and store-to-memory operations. Programmable atomic unit 208 may be configured to utilize a RISC-V ISA with a dedicated set of instructions to facilitate interaction with controller 202 to atomically execute user-defined operations.

[0068] Programmable atomic requests (e.g., received from a host on or outside the node) can be routed to programmable atomic units 208 via the second NOC 206 and controller 202. In this example, custom atomic operations (e.g., implemented by programmable atomic unit 208) can be identical to built-in atomic operations (e.g., implemented by built-in atomic module 214), except that the programmable atomic operations can be defined or programmed by a user rather than a system architect. In this example, programmable atomic request packets can be sent to controller 202 via the second NOC 206, and controller 202 can recognize the requests as custom atoms. Controller 202 can then forward the recognized requests to programmable atomic units 208.

[0069] Figure 3 This section generally illustrates an example of a programmable atom unit 302 for use with a memory controller according to an embodiment. In this example, the programmable atom unit 302 may include or correspond to data from... Figure 2 The programmable atomic unit 208 of the instance. That is, Figure 3 This describes the components in an example of a programmable atomic unit 302 (PAU), such as those mentioned above. Figure 2 (For example, in programmable atom unit 208) or Figure 1 (For example, in the atomic operation module of memory controller 130) the components indicated. Figure 3 The description states that the programmable atomic unit 302 includes a PAU processor or PAU core 306, a PAU thread control 304, an instruction SRAM 308, a data cache 310, and a memory interface 312 that interfaces with the memory controller 314. In this example, the memory controller 314 includes components from... Figure 2 An instance of controller 202.

[0070] In this example, the PAU core 306 is a pipelined processor, enabling multiple levels of different instructions to execute together per clock cycle. The PAU core 306 may contain a barrel-shaped multithreaded processor with a thread control circuitry 304 that switches between different register files (e.g., several sets of registers containing the current processing state) after each clock cycle. This implementation enables efficient context switching between currently executing threads. In this example, the PAU core 306 supports eight threads, resulting in eight register files. In this example, some or all register files are not integrated into the PAU core 306 but reside in the local data cache 310 or instruction SRAM 308. This reduces the circuit complexity of the PAU core 306 by eliminating conventional flip-flops used for registers in such memories.

[0071] The local PAU memory may contain instruction SRAM 308, such as instructions for individual atoms. The instructions include instruction sets supporting various application-based atomic operators. When an atomic operator is requested, for example, by an application chiplet, a set of instructions corresponding to the atomic operator is executed by the PAU core 306. In this example, the instruction SRAM 308 is partitioned to establish the instruction set. In this example, a specific programmable atomic operator requested by the request process can be identified by a partition number. The partition number can be established when the programmable atomic operator is registered on (e.g., loaded onto) programmable atomic unit 302. Other metadata for the programmable instructions may be stored in memory (e.g., in a partition table) in the local memory of programmable atomic unit 302.

[0072] In this example, atomic operators manipulate instructions in SRAM 308, which are typically synchronized (e.g., flushed) when the thread of the atomic operator completes. Therefore, latency for most memory operations during the execution of the programmable atomic operator thread is reduced, except for initial loading from external memory (e.g., from memory controller 314).

[0073] If a potentially dangerous situation would prevent a memory request, a pipelined processor, such as PAU core 306, would encounter a problem when an execution thread attempts to make such a request. Here, the memory request would retrieve data from memory controller 314, whether from a cache on memory controller 314 or off-die memory. To address this, PAU core 306 is configured to reject the thread's memory request. Typically, PAU core 306 or thread control 304 may contain circuitry that enables one or more thread rescheduling points in the pipeline. Here, rejection occurs at points in the pipeline beyond these thread rescheduling points (e.g., after these rescheduling points). In this instance, the danger occurs outside of a rescheduling point. Here, a previous instruction in the thread creates the danger after the last thread rescheduling point before the memory request instruction passes through the pipeline stage where the memory request can be made.

[0074] In this example, to reject a memory request, the PAU core 306 is configured to determine (e.g., detect) that a danger exists on the memory indicated in the memory request. Here, danger refers to any condition that would cause allowing (e.g., executing) the memory request to result in an inconsistent state for the thread. In this example, the danger is an in-flight memory request. Here, regardless of whether the instruction SRAM 308 contains data for the requested memory address, the existence of an in-flight memory request does not determine what data should be in the instruction SRAM 308 at that address. Therefore, the thread must wait for the in-flight memory request to complete before operating on the current data. When the memory request completes, the danger is cleared.

[0075] In this example, the danger lies in a dirty cache line in the instruction SRAM 308 used for the requested memory address. While a dirty cache line typically indicates that the data in the cache is current and the memory controller version of this data is not, the problem can arise with thread instructions that do not operate from the cache. Examples of such instructions use the built-in atomic operators of the memory controller 314 or other separate hardware blocks. In the context of the memory controller, the built-in atomic operators may be separate from the programmable atomic unit 302 and do not access the cache or instruction SRAM 308 within the PAU. Therefore, memory requests for the built-in atomic operators are processed from the cache or off-die memory. If the cache line is dirty, the built-in atomic operators will not operate on most of the current data until the cache is flushed to synchronize the cache or other or off-die memory. The same situation can occur with other hardware blocks of the memory controller, such as encryption blocks, encoders, etc.

[0076] The PAU core 306 can be configured to place threads into the memory response path. Although a thread's memory request has not actually been issued, the thread can be treated as if a memory request had been made. This implements smooth thread rescheduling without affecting other threads and eliminates the need to add thread stopping circuitry to the various pipeline stages of the PAU core 306. This approach introduces two problems. First, because the memory request has not actually been issued, the thread can skip some stages of the memory request and response path. Second, because no memory response is imminent, data will not be updated by the thread (e.g., no register write-back will occur).

[0077] To address the first issue, the PAU core 306 can be configured to add threads to a queue of threads waiting to be inserted into the memory response path. Here, the PAU core 306 maintains the queue and pushes threads onto it. In this way, the processor core's memory response path can pop the next thread from the queue to be placed into the standard memory response path.

[0078] To address the second issue, the PAU core 306 is configured to provide an indicator that a thread will not perform a register write-back. This indicator (e.g., a flag) can be some bits added to thread metadata in a queue or elsewhere to indicate that the write-back mechanism will not be applied to the thread in this example. Therefore, when a thread appears at a register file write block in the memory response path, no register file write occurs.

[0079] In this example, to provide an indicator that a thread will not perform register write-back, the PAU core 306 is configured to create a memory response that includes the indicator. Here, the thread is inserted into the memory response path via a memory response created by the PAU core 306, rather than from the memory controller 314. Therefore, the thread handles standard memory responses for the memory response path. However, including the indicator means that no action is taken on register or cache write blocks. In this example, the memory response is queued to await insertion into the memory response path after an earlier stage (e.g., decoding from the memory controller, error correction, etc.).

[0080] Thread control 304 can be configured to reschedule a thread with other threads in the memory response path. In one instance, the thread is popped from the thread queue waiting to be inserted into the memory response path. In another instance, an indicator is processed by the memory response path to move the thread from the thread queue waiting for a response from memory to the memory response rescheduling queue in the memory response path. Here, the indicator or other metadata is used by the memory response path to determine the correct insertion time for a thread among other pending threads.

[0081] Figure 4 This describes an example of a hybrid-threaded processor (HTP) accelerator or HTP accelerator 400. According to an embodiment, HTP accelerator 400 may include a portion of a memory computing device. In an example, HTP accelerator 400 may include or include components from... Figure 1 The HTP 140 is an example. The HTP accelerator 400 includes, for example, an HTP core 402, an instruction cache 404, a data cache 406, a translation block 408, a memory interface 410, and a thread controller 412. The HTP accelerator 400 may further include a dispatch interface 414 and a NOC interface 416, for example, for interfacing with an NOC, such as from... Figure 1 The first NOC 118 instance, from Figure 2 The second NOC 206 or other NOCs for the example.

[0082] In this example, the HTP accelerator 400 includes modules based on the RISC-V instruction set and may include a relatively small number of other or additional custom instructions supporting a low-overhead, thread-capable Hybrid Threading (HT) language. The HTP accelerator 400 may include a highly threaded processor core (HTP core 402), in which threads can be executed in a single instruction rotation to, for example, maintain high instruction throughput. In this example, threads can be paused while waiting for other pending events to complete. This allows computational resources to be used efficiently for related work rather than polling. In this example, multi-threaded barrier synchronization can utilize efficient HTP-to-HTP and HTP-to-host / slave message sending, allowing, for example, thousands of threads to initialize or wake up in, for example, dozens of clock cycles.

[0083] In this example, dispatch interface 414 may include functional blocks of HTP accelerator 400 for handling hardware-based thread management. That is, dispatch interface 414 can manage the dispatch of work to HTP core 402 or other accelerators. However, non-HTP accelerators typically cannot dispatch work. In this example, work dispatched from the host may use dispatch queues residing in, for example, host main memory (e.g., DRAM-based memory). On the other hand, work dispatched from HTP accelerator 400 may use dispatch queues residing in SRAM (e.g., within the dispatch of the target HTP accelerator 400 within a specific node).

[0084] In an example, HTP core 402 may include one or more cores representing threads executing instructions. That is, HTP core 402 may contain an instruction processing block. HTP core 402 may further include or be coupled to thread controller 412. Thread controller 412 provides thread control and state for each active thread within HTP core 402. Data cache 406 may include a cache for the host processor (e.g., for local and remote memory computing devices, including caches for HTP core 402), and instruction cache 404 may include a cache used by HTP core 402. In an example, data cache 406 may be configured for read and write operations, and instruction cache 404 may be configured for read-only operations.

[0085] In this example, data cache 406 is a small cache provided per hardware thread. Data cache 406 can temporarily store data used by the owning thread. Data cache 406 can be managed by hardware or software in the HTP accelerator 400. For example, the hardware can be configured to automatically allocate or evict lines as needed, since load and store operations are performed by the HTP core 402. Software (e.g., using RISC-V instructions) can determine which memory accesses should be cached and when lines should be invalidated or written back to other memory locations.

[0086] The data cache on the HTP accelerator 400 offers various benefits, including making larger accesses more efficient for the memory controller, thus allowing execution threads to avoid stopping. However, in some cases, using the cache leads to inefficiency. Examples include accesses where data is accessed only once, causing cache line thrashing. To help address this issue, the HTP accelerator 400 uses a custom set of load instructions to force load instructions to check for cache hits and, if a cache miss occurs, to issue a memory request for the requested operand without placing the acquired data into the data cache 406. Therefore, the HTP accelerator 400 includes various types of load instructions, including non-cache and cache line loads. If dirty data exists in the cache, the non-cache load instruction uses the cached data. The non-cache load instruction ignores clean data in the cache and does not write the accessed data to the data cache. For cache line load instructions, a complete data cache line (e.g., including 64 bytes) can be loaded from memory into the data cache 406, and addressed memory can be loaded into a specified register. If clean or dirty data is in data cache 406, then these loads can use cached data. If the referenced memory location is not in data cache 406, then the entire cache line can be accessed from memory. Using cache line load instructions can reduce cache misses when referencing sequential memory locations (e.g., memory copy operations), but it also wastes memory and bandwidth at NOC interface 416 if the referenced memory data is not used.

[0087] In this example, the HTP accelerator 400 includes custom, non-cached store instructions. These non-cached store instructions help prevent data cache 406 from being thrashed by out-of-order writes to memory.

[0088] In this example, the HTP accelerator 400 further includes a translation block 408. Translation block 408 may contain a virtual-to-physical translation block for the local memory of the memory computing device. For example, a host processor (e.g., in the HTP core 402) may execute load or store instructions, and these instructions may generate virtual addresses. The virtual addresses may be translated into physical addresses of the host processor, for example, using a translation table from translation block 408. Memory interface 410 may, for example, include an interface between the HTP core 402 and the NOC interface 416.

[0089] Figure 5 Examples of a hybrid threading architecture (HTF) or HTF 500 of a memory computing device according to embodiments are described. In these examples, HTF 500 may include or include components from... Figure 1The HTF 142 is an example of this. The HTF 500 is a coarse-grained, reconfigurable computational architecture that can be optimized for high-level language operand types and operators (e.g., using C / C++ or other high-level languages). In an instance, the HTF 500 may contain configurable, n-bit-wide (e.g., 512-bit-wide) data paths that interconnect hardened SIMD arithmetic units.

[0090] In this instance, the HTF 500 includes an HTF cluster 502, which contains multiple HTF blocks, including instance block 504 or block N. Each HTF block may contain one or more computing elements with local memory and arithmetic functions. For example, each block may contain a computing pipeline that supports integer and floating-point operations. In this instance, data paths, computing elements, and other infrastructure may be implemented as hardened IP to provide maximum performance while minimizing power consumption and reconfiguration time.

[0091] exist Figure 5 In this example, the pieces constituting HTF cluster 502 are arranged linearly, and each piece in the cluster can be coupled to one or more other pieces in HTF cluster 502. Figure 5 In this example, instance block 504 or block N is coupled to four other blocks, including a port labeled SF IN N-2 coupled to a base block 510 (e.g., block N-2), a port labeled SF IN N-1 coupled to an adjacent block 512 (e.g., block N-1), a port labeled SF IN N+1 coupled to block N+1, and a port labeled SF IN N+2 coupled to block N+2. Instance block 504 may be coupled to the same or other blocks via corresponding output ports (e.g., output ports labeled SF OUT N-1, SF OUT N-2, SF OUT N+1, and SF OUT N+2). In this example, the ordered list of block names is a conceptual indication of the block's location. In other examples, the blocks constituting HTF cluster 502 may be arranged in a mesh or other configuration, where each block is similarly coupled to one or more of its nearest neighbors in the mesh. Blocks provided at the edge of the cluster may optionally have fewer connections to adjacent blocks. For example, piece N-2 or Figure 5 In an instance, the base piece 510 may be coupled only to the adjacent piece 512 (piece N-1) and the instance piece 504 (piece N). Similarly, fewer or additional inter-piece connections may be used.

[0092] HTF cluster 502 may further include a memory interface module, which includes a first memory interface module 506. The memory interface module couples HTF cluster 502 to a NOC, such as a first NOC 118. In an example, the memory interface module may allow fragments within the cluster to make requests to other locations in the memory computing system (e.g., on the same or different nodes in the system). That is, the representation of HTF 500 may include a portion of a larger structure that can be distributed across multiple nodes, such as having one or more HTF fragments or HTF clusters at each node. Requests can be made between fragments or nodes within the context of the larger structure.

[0093] exist Figure 5 In this example, the slices in HTF cluster 502 are coupled using a synchronous structure (SF). The synchronous structure provides communication between a specific slice in HTF cluster 502 and its neighboring slices, as described above. Each HTF cluster 502 may further include an asynchronous structure (AF), which provides, for example, communication between slices in the cluster, memory interfaces in the cluster, and dispatch interfaces 508 in the cluster.

[0094] In this example, the synchronization structure can exchange messages containing data and control information. Control information may specifically include instruction RAM address information or thread identifiers. Control information can be used to establish data paths, and data message fields can be selected as the source of the path. Typically, control fields can be provided or received earlier, making them available for configuring the data path. For example, to help minimize any delays through the synchronization domain pipeline in a slice, control information may arrive in the slice several clock cycles before the data fields. Various registers may be provided to help coordinate the timing of data flow in the pipeline.

[0095] In this example, each piece in the HTF cluster 502 may contain multiple memories. Each memory may have the same width as the data path (e.g., 512 bits) and may have a specified depth, for example, in the range of 512 to 1024 elements. The piece memory may be used to store data that supports data path operations. The stored data may contain, for example, constants partially loaded as part of the kernel's cluster configuration or variables partially computed as part of the data stream. In this example, the piece memory may be written from an asynchronous structure as a data transfer from another synchronous domain, or may contain, for example, the result of a load operation initiated by another synchronous domain. The piece memory may be read via synchronous data path instructions executed in the synchronous domain.

[0096] In the example, each block in the HTF cluster 502 may have a dedicated instruction RAM (INST RAM). In an example of an HTF cluster 502 with 16 blocks and an instruction RAM with 64 entries, the cluster can allow algorithms to be mapped to up to 1024 multiplication-shift and / or ALU operations. The individual blocks may optionally be piped together, for example, using a synchronous structure to allow data stream computation with minimal memory access, thus minimizing latency and reducing power consumption. In the example, an asynchronous structure can allow memory references and computations to run in parallel, thereby providing a more efficient streaming kernel. In the example, the individual blocks may include built-in support for loop-based constructions and can support nested loop kernels.

[0097] Synchronization structures allow multiple tiles to be transported via pipelines, eliminating the need for data queuing. Tiles participating in a synchronization domain can, for example, be used as a single pipelined data path. The first or base tile of the synchronization domain (e.g., Figure 5 In the example, slice N-2 can start worker threads via pipelined slices. The base slice is responsible for starting work at a predefined rhythm, referred to in this document as the spoke count. For example, if the spoke count is 3, then the base slice can cycle through and start working every 2 clock cycles.

[0098] In this example, a synchronization domain comprises a set of connected segments within an HTF cluster 502. Thread execution can begin at a base segment of the domain and can progress from the base segment to other segments within the same domain via the synchronization structure. The base segment provides instructions for execution on the first segment. The first segment may by default provide the same instructions for execution on other connected segments. However, in some instances, the base segment or subsequent segments may conditionally specify or use alternative instructions. Alternative instructions are selected by causing a Boolean condition value to be generated for the segment's data path, and then a Boolean value can be used to select between the instruction set of the current segment and the alternative instructions.

[0099] Asynchronous structures can be used to perform operations that occur asynchronously relative to synchronous domains. Each piece in an HTF cluster 502 may contain an interface to an asynchronous structure. Inbound interfaces may contain, for example, FIFO buffers or queues (e.g., AF IN QUEUE) that provide storage for messages that cannot be processed immediately. Similarly, outbound interfaces of asynchronous structures may contain FIFO buffers or queues (e.g., AF OUT QUEUE) that provide storage for messages that cannot be sent immediately.

[0100] In this example, messages in an asynchronous architecture can be categorized as data messages or control messages. Data messages may contain SIMD-width data values ​​written to fragment memory 0 (MEM_0) or memory 1 (MEM_1). Control messages can be configured to control threads to create and release resources or issue external memory references.

[0101] In an HTF cluster 502, segments can perform various computational operations on the HTF. These operations can be executed by configuring the data paths within the segment. In this example, a segment contains two functional blocks that perform computational operations on the segment: the Multiplication and Shift Operations Block (MS OP) and the Arithmetic, Logical, and Bitwise Operations Block (ALB OP). Both blocks can be configured to perform pipelined operations, such as multiplication and addition, or shift and addition, and others.

[0102] In one instance, each memory computing device in the system may have a full set of supported instructions for its operator blocks (e.g., MSOP and ALB OP). In this case, binary compatibility can be achieved across all devices in the system. However, in some instances, it can help maintain a basic set of functionalities and optional instruction set classes, such as satisfying various design trade-offs, like die size. The approach can be similar to how the RISC-V instruction set has a basic set and multiple optional instruction subsets.

[0103] In this example, instance block 504 may contain spoke RAM. The spoke RAM can be used to specify which input (e.g., from the four SF block inputs and the base block input) is the primary input for each clock cycle. The spoke RAM read address input may originate from a counter that counts from zero to a spoke count decremented by 1. In this example, different spoke counts may be used on different blocks (e.g., within the same HTF cluster 502) to allow several slices or a single block example to be used by the inner loop to determine the execution of a particular application or instruction set. In this example, the spoke RAM may specify when synchronization inputs are written to the block memory, for example, when multiple inputs using a particular block instruction arrive before the others. The first arriving input may be written to the block memory and can later be read when all inputs are available. In this example, the block memory may be accessed as a FIFO memory, and the FIFO read and write pointers may be stored in register-based memory regions or structures within the block memory.

[0104] Figure 6A and Figure 6BThis section describes examples of chiplet systems that can be used to implement one or more aspects of CNM system 102. As similarly mentioned above, nodes in CNM system 102 or devices within nodes in CNM system 102 may contain chiplet-based architectures or near-memory computing (CNM) chiplets. Packaged memory computing devices may contain, for example, one, two, or four CNM chiplets. Chipslets may be interconnected using high-bandwidth, low-latency interconnects (e.g., using CPI interfaces). Typically, a chiplet system consists of discrete modules (each a “chiplet”) integrated on an interposer and, in many instances, interconnected as needed via one or more established networks to provide the desired functionality to the system. The interposer and the contained chiplets may be packaged together to facilitate interconnection with other components of a larger system. Each chiplet may contain one or more individual integrated circuits (ICs) or “chips” (possibly combined with discrete circuit components) and may be coupled to a corresponding substrate to facilitate attachment to the interposer. Most or all of the chiplets in the system may be individually configured to communicate via established networks.

[0105] Chiplets are configured as individual modules of a system that differ from a single chip containing dissimilar device blocks (e.g., intellectual property (IP) blocks) implemented on a substrate (e.g., a single die), such as a system-on-a-chip (SoC) or multiple discrete packaged devices integrated on a printed circuit board (PCB). Generally, chiplets offer better performance (e.g., lower power consumption, reduced latency) than discrete packaged devices, and provide greater manufacturing benefits than a single die chip. These manufacturing benefits may include higher yields or reduced development costs and time.

[0106] A chiplet system may include, for example, one or more application (or processor) chiplets and one or more support chiplets. Here, the distinction between application and support chiplets is merely a possible design case for a reference chiplet system. Thus, for example, a synthetic vision chiplet system may include (by example only) application chiplets that generate synthetic vision output and support chiplets, such as memory controller chiplets, sensor interface chiplets, or communication chiplets. In typical use cases, synthetic vision designers may design the application chiplets and supply the support chiplets from other sources. Therefore, design costs (e.g., in terms of time or complexity) are reduced by avoiding the design and manufacturing of functionality embodied in the support chiplets.

[0107] Chiplets also enable the tight integration of IP blocks that might otherwise be difficult, such as those manufactured using different processing technologies or with different feature sizes (or employing different contact technologies or spacing). Therefore, multiple ICs or IC assemblies with different physical, electrical, or communication characteristics can be modularly assembled to provide a variety of desired functionalities to the assembly. Chiplet systems also facilitate adaptation to the needs of different larger systems into which the chiplet system will be incorporated. In examples, ICs or other assemblies can be optimized for power, speed, or heat generation (as can occur with sensors) for specific functions, making integration with other devices much easier than attempting to do so on a single die. Furthermore, by reducing the overall die size, chiplet yields tend to be higher than those of more complex single-die devices.

[0108] Figure 6A and Figure 6B This section provides a general description of an example of a chiplet system according to an embodiment. Figure 6A This is an illustration of a chiplet system 602 mounted on a peripheral board 604, which can be connected to a larger computer system via, for example, peripheral component interconnect (PCIe). The chiplet system 602 includes a package substrate 606, an interposer 608, and four chips: an application chiplet 610, a host interface chiplet 612, a memory controller chiplet 614, and a memory device chiplet 616. Other systems may include numerous additional chipsets to provide additional functionality as will be understood from the following discussion. The package of the chiplet system 602 is illustrated using a cap / cover 618, but other packaging technologies and structures used for chiplet systems may be used. Figure 6B This is a block diagram that labels the components in a chiplet system for clarity.

[0109] Application chip 610 is described as including chiplet system NOC 620 to support chiplet network 622 for inter-chiplet communication. In an example embodiment, chiplet system NOC 620 may be included on application chip 610. In an example, from Figure 1 The first NOC 118 of this example can be defined in response to the selected supporting chiplets (e.g., host interface chiplet 612, memory controller chiplet 614, and memory device chiplet 616), thus enabling the designer to select the appropriate number of chiplet system NOCs 620 or chiplet network connections or switches. In this example, the chiplet system NOC 620 can be located on a single chiplet or within an intermediary layer 608. In the example discussed herein, the chiplet system NOC 620 implements a chiplet protocol interface (CPI) network.

[0110] In an example, the chiplet system 602 may include or comprise a portion of the first memory computing node 104 or the memory first memory computing device 112. That is, various blocks or components of the first memory computing device 112 may include chiplets that can be mounted on the peripheral board 604, the package substrate 606, and the interposer 608. The interface components of the first memory computing device 112 may substantially include a host interface chiplet 612; the memory and memory control-related components of the first memory computing device 112 may substantially include a memory controller chiplet 614; the various accelerator and processor components of the first memory computing device 112 may substantially include an application chiplet 610 or an example thereof, and so on.

[0111] CPI interfaces (e.g., for communication between or within chiplets in a system) are packet-based networks that enable flexible and high-speed interaction between chiplets via virtual channels. CPI implementations bridge the chiplet network 622 from the chiplet network itself. For example, the Advanced Extensible Interface (AXI) is a widely used specification for designing intra-chip communication. However, the AXI specification encompasses various physical design options, such as the number of physical channels, signal timing, and power. Within a single chip, these options are typically selected to meet design goals, such as power consumption and speed. However, to achieve flexibility in chiplet systems, CPI adapters are used to intersect between various AXI design options that can be implemented in individual chiplets. By enabling physical-to-virtual channel mapping and encapsulating time-based signaling with packet protocols, CPI bridges the intra-chiplet network 622.

[0112] CPI can use various physical layers to transmit packets. A physical layer may contain simple conductive connections, or it may contain drivers to increase voltage or otherwise facilitate signal transmission over longer distances. An instance of this physical layer may contain an Advanced Interface Bus (AIB), which in various instances may be implemented in the intermediate layer 608. The AIB uses source-synchronous data transfer to transmit and receive data within a forwarding clock. Packets are transmitted across the AIB at Single Data Rate (SDR) or Double Data Rate (DDR) with respect to the transmitted clock. The width of each channel is supported by the AIB. Channels can be configured to have a symmetrical number of transmit (TX) and receive (RX) inputs / outputs (I / O), or an asymmetrical number of transmitters and receivers (e.g., all transmitters or all receivers). A channel may be used as a master or slave AIB depending on which chip provides the master clock. The AIB I / O unit supports three clock modes: asynchronous (i.e., non-clocked), SDR, and DDR. In various instances, the non-clocked mode is used for clock and some control signals. SDR mode can use a dedicated SDR-only I / O unit or use dual SDR / DDR I / O units.

[0113] In this example, CPI packet protocols (e.g., point-to-point or routable) can use symmetrical receive and transmit I / O units within an AIB channel. CPI streaming protocols allow for more flexible use of AIB I / O units. In this example, AIB channels for streaming mode can be configured with I / O units as all TX, all RX, or half TX and half RX. CPI packet protocols can use AIB channels in SDR or DDR operating modes. In this example, AIB channels are configured incrementally with 80 I / O units (i.e., 40 TX and 40 RX) for SDR mode and 40 I / O units for DDR mode. CPI streaming protocols can use AIB channels in SDR or DDR operating modes. Here, in this example, AIB channels increment with 40 I / O units for both SDR and DDR modes. In this example, each AIB channel is assigned a unique interface identifier. This identifier is used during CPI reset and initialization to identify paired AIB channels across adjacent chiplets. In this example, the interface identifier is a 20-bit value comprising a 7-bit chiplet identifier, a 7-bit column identifier, and a 6-bit link identifier. The AIB physical layer uses an AIB out-of-band shift register to transmit the interface identifier. The 20-bit interface identifier is transmitted in both directions across the AIB interface using bits 32 to 51 of the shift register.

[0114] AIB defines a stacked set of AIB channels as an AIB channel column. An AIB channel column has a certain number of AIB channels plus auxiliary channels. The auxiliary channels contain signals used for AIB initialization. All AIB channels within a column (except for the auxiliary channels) have the same configuration (e.g., all TX, all RX, or half TX and half RX, and the same number of data I / O signals). In this example, AIB channels are numbered sequentially in ascending order, starting with the AIB channel adjacent to the AUX channel. The AIB channel adjacent to the AUX channel is defined as AIB channel 0.

[0115] Typically, the CPI interface on an individual chiplet may include serialization-deserialization (SERDES) hardware. SERDES interconnects are well-suited for cases where high-speed signaling with low signal counts is desired. However, SERDES can result in additional power consumption and longer latency for multiplexing and demultiplexing, error detection or correction (e.g., using block-level cyclic redundancy check (CRC)), link-level retries, or forwarding error correction. However, when low latency or power consumption is the primary concern for ultra-short-range chiplet-to-chiplet interconnects, parallel interfaces with clock rates that allow data transmission with minimal latency can be utilized. CPIs contain elements that minimize both latency and power consumption in these ultra-short-range chiplet interconnects.

[0116] For flow control, CPI employs a credit-based technique. For example, the receiver using chip 610 provides credits representing available buffers to the sender, such as memory controller chip 614. In this example, for a given time unit of transmission, the CPI receiver contains buffers for each virtual channel. Therefore, if the CPI receiver supports 5 time messages and a single virtual channel, the receiver has 5 buffers arranged in 5 rows (e.g., 1 row per unit time). If 4 virtual channels are supported, the receiver has 20 buffers arranged in 5 rows. Each buffer holds the payload of one CPI packet.

[0117] When a sender transmits to a receiver, the sender decrements its available credits based on the transmission. Once all of the receiver's credits are exhausted, the sender stops sending packets to the receiver. This ensures that the receiver always has an available buffer to store transmissions.

[0118] As the receiver processes the received packets and releases the buffer, it returns the available buffer space to the sender. This credit return can then be used by the sender to allow the transmission of additional information.

[0119] Figure 6A An example is the chiplet mesh network 624 that uses direct chiplet-to-chiplet technology without requiring a chiplet system NOC 620. The chiplet mesh network 624 can be implemented in a CPI or another chiplet-to-chiplet protocol. The chiplet mesh network 624 typically enables a chiplet pipeline, where one chiplet acts as an interface to the pipeline while other chips in the pipeline only interface to themselves.

[0120] Additionally, dedicated device interfaces, such as one or more industry-standard memory interfaces (e.g., synchronous memory interfaces, such as DDR5, DDR6), can be used to connect devices to the chiplet. Connections from the chiplet system or individual chiplets to external devices (e.g., larger systems) can be made via a desired interface (e.g., a PCIe interface). In this example, this external interface is implemented via a host interface chiplet 612, which provides a PCIe interface external to the chiplet system. This dedicated chiplet interface 626 is typically used when industry conventions or standards have converged on this interface. The illustrated example of connecting the memory controller chiplet 614 to the Double Data Rate (DDR) interface of the dynamic random access memory (DRAM) memory device chiplet 616 is precisely this industry convention.

[0121] Regarding various possible chiplets, the memory controller chiplet 614 is likely to exist in chiplet systems due to its almost ubiquitous use in storage devices for computer processing and advanced memory devices. Therefore, using the memory device chiplet 616 and the memory controller chiplet 614, both manufactured by others, allows chiplet system designers to use robust products from advanced manufacturers. Typically, the memory controller chiplet 614 provides a memory device-specific interface for reading, writing, or erasing data. Typically, the memory controller chiplet 614 can provide additional features such as error detection, error correction, maintenance operations, or atomic operator execution. For some types of memory, maintenance operations tend to be specific to the memory device chiplet 616, such as the collection of discarded items in NAND flash or storage-class memory, or temperature regulation (e.g., cross-temperature management) in NAND flash memory. In instances, maintenance operations may involve logic-to-physical (L2P) mapping or management to provide an indirect level between the physical and logical representations of data. In other types of memory, such as DRAM, some memory operations, such as refresh, may be controlled by the host processor of the memory controller at some times, and by the DRAM memory device or by logic associated with one or more DRAM devices, such as an interface chip (in this example, a buffer).

[0122] Atomic operators are data manipulations that can be executed, for example, by the memory controller chip 614. In other chiplet systems, atomic operators can be executed by other chipsets. For example, the "increment" atomic operator can be specified by the application chip 610 in a command that includes a memory address and a possible increment value. After receiving the command, the memory controller chip 614 retrieves a number from the specified memory address, increments it by the amount specified in the command, and stores the result. Upon successful completion, the memory controller chip 614 provides a success indication to the application chip 610. Atomic operators avoid data transfer across the chiplet mesh network 624, resulting in lower latency execution of such commands.

[0123] Atomic operators can be classified as built-in atoms or programmable (e.g., custom) atoms. Built-in atoms are a finite set of operations that are implemented immutably in the hardware. Programmable atoms are small programs that can be executed on programmable atom units (PAUs) (e.g., custom atom units (CAUs)) of the memory controller chip 614.

[0124] The memory device chip 616 can be a volatile memory device or a non-volatile memory, or any combination thereof. Examples of volatile memory devices include (but are not limited to) random access memory (RAM), such as DRAM, synchronous DRAM (SDRAM), graphics double data rate type 6 SDRAM (GDDR6 SDRAM), etc. Examples of non-volatile memory devices include (but are not limited to) NAND flash memory, memory-type memory (e.g., phase-change memory or memristor-based technology), ferroelectric RAM (FeRAM), etc. The illustrated example includes memory device chip 616 as a chip; however, the device may reside elsewhere, such as in a different package on peripheral board 604. Multiple memory device chips may be provided for many applications. In the examples, these memory device chips may each implement one or more memory technologies and may include an integrated computing host. In examples, a memory chiplet may comprise multiple stacked memory dies of different technologies, such as one or more static random access memory (SRAM) devices stacked with or communicating with one or more dynamic random access memory (DRAM) devices. In examples, a memory controller chiplet 614 may be used to coordinate the operation between multiple memory chipsets in a chiplet system 602, for example, using one or more memory chipsets in one or more cache memory levels and using one or more additional memory chipsets as main memory. The chiplet system 602 may include multiple examples of memory controller chipset 614, such as those used to provide memory control functionality for individual hosts, processors, sensors, networks, etc. Chiplet architectures (e.g., in the illustrated system) offer the advantage of allowing adaptation to different memory storage technologies and providing different memory interfaces by upgrading chiplet configurations, for example, without requiring redesign of the remainder of the system architecture.

[0125] Figure 7 This section generally describes examples of chiplet-based implementations of a memory computing device according to embodiments. Examples include implementations having four near-memory computing or CNM chiplets, each of which may include or incorporate components from… Figure 1 This refers to a portion of the first memory computing device 112 or the first memory computing node 104. Each portion may itself contain or include a corresponding chiplet. Chiplet-based implementations may include or utilize CPI-based intra-system communication, as described above. Figure 6A and Figure 6B The example chip system 602 is discussed similarly.

[0126] Figure 7An example includes a first CNM package 700, which comprises a plurality of chiplets. The first CNM package 700 includes a first chiplet 702, a second chiplet 704, a third chiplet 706, and a fourth chiplet 708 coupled to a CNM NOC hub 710. Each of the first to fourth chiplets may include examples of the same or substantially the same components or modules. For example, each chiplet may each include a corresponding example of an HTP accelerator, an HTF accelerator, and a memory controller for accessing internal or external memory.

[0127] exist Figure 7 In one example, the first chiplet 702 includes a first NOC hub edge 714 coupled to the CNM NOC hub 710. Other chipslets in the first CNM package 700 similarly include NOC hub edges or endpoints. Switches in the NOC hub edges facilitate communication within the chiplet or within the chiplet system via the CNM NOC hub 710.

[0128] The first chiplet 702 may further include one or more memory controllers 716. The memory controllers 716 may correspond to different NOC endpoint switches that interface with the first NOC hub edge 714. In examples, the memory controllers 716 may include a memory controller chiplet 614, or a memory controller 130, or a memory subsystem 200 or other memory computing implementation. The memory controllers 716 may be coupled to different memory devices, such as a first external memory module 712A or a second external memory module 712B. The external memory modules may include, for example, GDDR6 memory that can be selectively accessed by the corresponding different chiplets in the system.

[0129] The first chiplet 702 may further include, for example, a first HTP chiplet 718 and a second HTP chiplet 720 coupled to the edge 714 of the first NOC hub via a corresponding different NOC endpoint switch. The HTP chipsets may correspond to an HTP accelerator, for example, from... Figure 1 The instance of HTP 140 or from Figure 4 An example of an HTP accelerator 400. The HTP chiplet can communicate with the HTF chiplet 722. The HTF chiplet 722 can correspond to an HTF accelerator, for example, from... Figure 1 The instance of HTF 142 or from Figure 5 The HTF 500 instance.

[0130] The CNM NOC hub 710 can be coupled to other chiplets or other CNM packaged NOC hub examples via various interfaces and switches. For example, the CNM NOC hub 710 can be coupled to a CPI interface via multiple different NOC endpoints on the first CNM package 700. Each of the multiple different NOC endpoints can, for example, be coupled to a different node outside the first CNM package 700. In an example, the CNM NOC hub 710 can be coupled to other peripheral devices, nodes, or apparatuses using CTCPI or other non-CPI protocols. For example, the first CNM package 700 may include a PCIe expansion architecture interface (PCIe / SFI) or a CXL interface (CXL) configured to interface the first CNM package 700 with other devices. In an example, the devices coupled to the first CNM package 700 using various CPI, PCIe, CXL, or other architectures can form a common global address space.

[0131] exist Figure 7 In this example, the first CNM package 700 includes a host interface 724 (HIF) and a host processor (R5). The host interface 724 may correspond to, for example, a source... Figure 1 The instance is HIF 120. The host processor or R5 can correspond to the one from... Figure 1 The instance includes an internal host processor 122. Host interface 724 may include a PCI interface for coupling the first CNM package 700 to other external devices or systems. In this instance, operation can be initiated by host interface 724 on the first CNM package 700 or a cluster of chips within the first CNM package 700. For example, host interface 724 may be configured to command, for example, individual HTF chip clusters among various chiplets in the first CNM package 700 to enter or exit power / clock gate modes.

[0132] Figure 8 This describes an example tile of a memory computing device according to an embodiment. Figure 8 In the example 800, the tiled chiplet instance contains four examples of different near-memory computing clusters of chiplets, where the clusters are coupled together. Each example of a near-memory computing chiplet may itself contain one or more constituent chips (e.g., host processor chiplets, memory device chiplets, interface chiplets, etc.).

[0133] The 800 tiled chiplet instances include those from Figure 7An example of a first CNM package 700 is one or more of its near-memory computing (CNM) clusters. For example, a tiled chiplet instance 800 may include a first CNM cluster 802, which includes a first chiplet 810 (e.g., corresponding to the first chiplet 702), a second chiplet 812 (e.g., corresponding to the second chiplet 704), a third chiplet 814 (e.g., corresponding to the third chiplet 706), and a fourth chiplet 816 (e.g., corresponding to the fourth chiplet 708). The chipsets in the first CNM cluster 802 may be coupled to a common NOC hub, which may in turn be coupled to NOC hubs in one or more adjacent clusters (e.g., in the second CNM cluster 804 or the fourth CNM cluster 808).

[0134] exist Figure 8 In this example, the tiled chiplet instance 800 includes a first CNM cluster 802, a second CNM cluster 804, a third CNM cluster 806, and a fourth CNM cluster 808. These different CNM chipsets can be configured in a common address space, allowing chipsets to allocate and share resources across different chipsets. In this example, chipsets within a cluster can communicate with each other. For instance, the first CNM cluster 802 can be coupled to the second CNM cluster 804 via a chiplet-to-chiplet CPI interface 818, and the first CNM cluster 802 can be coupled to the fourth CNM cluster 808 via another or the same CPI interface. The second CNM cluster 804 can be coupled to the third CNM cluster 806 via the same or another CPI interface, and so on.

[0135] In this example, one of the near-memory computing chips in the tiled chiplet instance 800 may include a host interface responsible for workload balancing across the tiled chiplet instance 800 (e.g., corresponding to a host interface from a local storage device). Figure 7 The host interface (724) of the instance. The host interface can facilitate, for example, access to host-based command request queues and response queues from outside the tiled chiplet instance 800. The host interface can use hybrid thread processors and hybrid thread architectures in one or more near-memory computing chips in the tiled chiplet instance 800 to dispatch new execution threads.

[0136] Threads begin on the accelerator when CPI or CTCPI call packets are received and processed by the HTP accelerator. CPI or CTCPI call packets can originate from the Host Interface (HIF) or the HTP accelerator. Calls originating from the HIF are issued by the host processor and defined as the main thread. Calls originating from HTP are defined as fiber threads. The main difference between main threads and fiber threads must be related to the limitations of creating new fibers.

[0137] It should be noted that the stack memory used for all threads is pre-allocated when an application running on the host processor makes a system call to indicate that it needs to access HTP resources. Thereafter, the pre-allocated stack becomes available when a thread starts in the context of HTP. The stack size of the main thread may differ from the stack size of the fibers. Typically, the main thread has a larger stack size than the fibers.

[0138] HTP threads can execute Thread Creation (ETC) instructions to launch threads on accelerator resources. Accelerator resources include HTP, HTF, and generic accelerators. Generic accelerators allow interaction with accelerators that have not yet been defined.

[0139] The ETC instruction specifies the information to be returned when the created thread completes. Space must be reserved for this return information before the thread can be created. When the thread finally completes, it writes its return information to this reserved space and waits for the parent thread to execute the thread join instruction. The thread join instruction retrieves the return information from the reserved space and transfers it to the thread's X register state for later use by the parent thread. Once the thread has been joined, the reserved space is released.

[0140] The ETC instruction indicates that no return information will be returned when the created thread completes. This instruction allows the HTP accelerator to create threads without reserving space for return information. In this case, a counter is used to track the number of outstanding threads that return no information.

[0141] The parent thread executes join instructions (EFJ, EFJA) to determine if the child thread has completed and obtains a return result. All accelerator resources (HTP, HTF, and general) use this common method to return the thread completion status to the starting parent thread. The status returned from the child thread can include: the caller ID and 0, 1, or 2 64-bit return parameters. Threads can be created with no return specified (nr).

[0142] The Fiber Join (EFJ) instruction joins a thread created without a no-return indicator and ignores threads created with a no-return indicator. The EFJ instruction processes a single completed child thread and writes the returned call ID and parameters to the thread's X register for the parent thread to access.

[0143] The Fiber Creation (EFC) instruction initiates a thread on the Hybrid Thread Accelerator (HTA). Up to four HTAs can exist within a CNM device that includes HTP and HTF.

[0144] Fiber creation command in Figure 9 The explanation is as follows. Bits 0 to 6 are the opcodes specifying the EFC function, and bits 27 to 31 are the function codes that distinguish the EFC type (e.g., HTA0, HTA1, HTA0.BF, etc.). Bits 7 to 11, 15 to 19, and 20 to 24 are reserved bits for future use.

[0145] The EFC instruction specifies several parameters used for the fiber creation operation. The return argument count (rc) field can take values ​​(NR, 0, 1, 2). If the calling HTP has space to store the return information, then the HTP can create the thread only on the accelerator resource. The return information includes both the caller ID and up to two 64-bit return values. The fiber creation instruction specifies the storage space required for the return information. The return information suffix options are NR, R0, R1, and R2. The NR suffix indicates that no return information will be stored (i.e., the join instruction will not provide any information). The R0 suffix indicates that the caller ID will be available for the join instruction. The R1 and R2 suffixes indicate that one or two 64-bit return arguments, in addition to the caller ID, will be stored and available for the join instruction.

[0146] Return information is allocated during fiber creation instruction execution. If insufficient space is available for return information, the fiber creation instruction is paused until space becomes available. Fiber creation instructions with the NR suffix require no return information storage, and the number of protrusions is limited by the size of a counter that tracks the number of protrusions. The R0 suffix requires storage for the caller ID but no return value, while R1 and R2 require both caller ID and return value space. The more information a set of created fibers requires, the smaller the maximum number of protruding fibers.

[0147] A busy-fail version of the EFC instruction is provided to allow the accelerator resource dispatcher to fail the EFC instruction when supported.

[0148] The HTP dispatcher supports busy failures and will return a failure status if the target CNM device is unavailable in the HTP context. The status of the fiber creation instruction is written to register A0 of the X register.

[0149] The HTF dispatcher does not support busy failures and will pause EFC instructions until the dispatcher can find the HTF to start the created thread.

[0150] Other accelerator resource assigners can choose whether to support busy failure.

[0151] Example assembly:

[0152] a.EFC.HTF

[0153] b.EFC.HTP.C4.NR

[0154] c.EFC.HTP.C2.R1

[0155] d.EFC.HTP.BF.C2.R1

[0156] Once the EFC instruction completes, the thread continues with instructions immediately following the EFC instruction.

[0157] It should be noted that the main thread (i.e., the thread created by the host) can execute any of the defined EFC instructions. HTP fiber threads (i.e., threads created by HTP) can execute busy-failure type EFC instructions or EFC instructions targeting non-HTP accelerator resources. If a fiber attempts to execute a non-busy-failure HTP fiber creation, a trap will occur.

[0158] Threads can be created on accelerators such as HTP 140 or HTF 142. In this document, "thread" can refer to a system or host thread, while "fiber" can refer to a thread created at HTP 140 or HTF 142. In other words, work dispatched by the host to an HTP accelerator is called a main thread, and work dispatched by HTP to an HTP accelerator is called a fiber thread. In many cases, the terms "fiber thread" and "thread" are used interchangeably.

[0159] Multiple thread types are supported by the HTP and HTF architectures. Threads can be designed as either return or "non-return" types. If a thread is a return type, the fiber creation instructions specify the storage space for return information. If a thread is a "non-return" type, then storage for return information is necessary, and the only limitation is the maximum number of available threads in the system.

[0160] The busy failure instruction feature is provided to allow the accelerator resource dispatcher to fail thread creation instructions. The HTP dispatcher supports busy failure and will return a failure status when the HTP context is unavailable on the target memory computing system. Using this failure status, a thread or fiber attempting to initiate a subfiber to undertake some workload can determine whether to continue processing the workload or divide the workload and assign a portion of the divided workload to the initiated fiber. This is described in more detail below.

[0161] Figure 10 This is a list of pseudocode illustrating the divide-and-conquer process 1000 according to an embodiment. It describes the Main() function, which treats the array WORK[] as a parameter. The size of the WORK[] array is stored in the local variable SIZE_WORK. If the size of the WORK[] array is greater than a threshold, then the WORK[] array is divided into two roughly equal parts. A thread is allocated, and the second half of the WORK[] array is passed to the thread as a parameter. The thread type is set to NO_RETURN, which allows the thread to terminate without returning a value after completing its allocation work. The thread is then started using the Start() function.

[0162] If the thread starts successfully, the child thread uses the second half of the WORK[] array as its local WORK[] array to recursively execute the main program.

[0163] If a thread cannot start, the current thread can handle some work locally. For example, it can process up to a threshold number of WORK[] arrays and then allocate and start another thread.

[0164] When the size of the WORK[] array is less than the threshold, the current thread can process the WORK[] array without triggering any other threads (fibers).

[0165] It should be understood that the Process() function, which operates on the WORK[] array, is for illustrative purposes only. Furthermore, it should be understood that the input to the Main() function (i.e., the WORK[] array) is for illustrative purposes only. The input can be a range of addresses to be processed, a linked list, or other enumerated data structures. Processing the input to the Main() function can be performed using bitwise operations (e.g., bit shifting, logical bitwise generation, or other bitwise operations). Additionally, although pseudocode in a high-level language is provided here, it should be understood that the pseudocode can be used to program hardware components such as application-specific integrated circuits (ASICs) or other programmable units.

[0166] Furthermore, it should be understood that Figure 9 The threshold comparisons in the text are for illustrative purposes. Thresholds may be based on, for example, operands that can be executed in a single clock cycle or some other metric that allows HTP to perform work within a given time.

[0167] Figure 11 This is another code list illustrating the divide-and-conquer process 1100 according to an embodiment. In the code list, the `fiber_init_parent()` function takes `lower_bound`, `upper_bound`, and a pointer to the `parent_segment` array and loops within a range assigning -1 to each element of the array. The function `init_parent_divide_and_conquer()` takes two bounds: a pointer to the `parent_segment` and a granularity (i.e., a threshold size for the work to be done on individual processes). In this code, `emd_create_fiber_htp_c4_nr_bf()` is explicitly called to create threads, where the second argument is a recursive reference (as a function pointer) to this same function `init_parent_divide_and_conquer()`. Boundary calculations are also added to establish child threads.

[0168] Figure 12This is a diagram illustrating the thread call tree 1200 according to an embodiment. Thread 0 receives the result of 8 work units, denoted as numbers 0 to 7. The threshold is equal to 1 work unit. Therefore, since there are 8 work units (which exceed the threshold), thread 0 divides the work and causes thread 1 to handle half of the work. In the next loop, thread 0 checks again and finds that it has assigned 4 work units ([0,1,2,3]) (which exceed the threshold) and then causes another thread (thread 2). Meanwhile, thread 1 also uses a similar evaluation and causes thread 3 to complete the half of the work assigned to thread 1 by thread 0. This work division continues until a valid thread has a single work unit (at or below the threshold amount).

[0169] The resulting thread (fiber) can be configured as a non-returning thread. In this case, the thread can terminate directly when it has completed processing its unit of work.

[0170] Figure 13 This is a flowchart illustrating an example of a method 1300 for thread execution control according to an embodiment. The operation of method 1300 is performed by computer hardware, for example, regarding... Figures 1 to 8 (e.g., host system 108, host processor 122, PAU thread control 304, HTP core 402, thread controller 412 or dispatch interface 414) or Figure 14 Computer hardware described as (e.g., processing circuitry systems).

[0171] At 1302, a request to execute the first thread is received, the request containing the workload. In various embodiments, the request can be received by interpreting code at the compiler that causes the new thread, at the thread manager, or at the thread dispatcher.

[0172] In 1304, method 1300 measures workload to generate a metric in a first thread. In an embodiment, the first thread is the main thread.

[0173] At 1306, the metric is compared with the threshold to determine if the metric exceeds the threshold.

[0174] At 1308, in response to a metric exceeding a threshold, the workload is divided into a set of sub-workloads consisting of a predefined number of identical portions from the workload. For example, the workload may be roughly divided in half, allowing multiple threads to process each half. Therefore, in this embodiment, the predefined number is 2.

[0175] At 1310, a second request is created to execute the second thread, wherein the second request contains the first component of the set of subworkloads. In an embodiment, the second thread is a fiber thread.

[0176] In one embodiment, the request includes a set busy-failure field. In another embodiment, if a second request to execute a second thread fails, and in response to the failure of the second request to execute the second thread and the busy-failure field being set, method 1300 continues by continuing to process a second component of the set of subworkloads and creating a third request to execute the second thread, the third request including a first component of the set of subworkloads. In one embodiment, the busy-failure field is a bit in a chip-to-chip protocol interface (CTCPI) packet.

[0177] In one embodiment, the second request includes a configured no-return field. In another embodiment, the no-return field is a bit in a chip-to-chip protocol interface (CTCPI) packet. In yet another embodiment, the no-return field is used to signal to the second thread not to return a value to the stack location. In yet another embodiment, the no-return field allows the first thread to avoid waiting for the second thread to return.

[0178] At 1312, the first thread continues to process the second component of the set of sub-workloads.

[0179] In an embodiment, the second component that continues processing the set of subwork tasks includes the following operations: measuring the second component to generate a second metric; comparing the metric with a threshold to determine that the second metric exceeds the threshold; in response to the second metric exceeding the threshold, dividing the second component into another set of subwork tasks consisting of a predefined number of identical parts from the second component; creating a third request to execute a third thread, the third request including the first component of the other set of subwork tasks; and continuing to process the second component of the other set of subwork tasks.

[0180] In one embodiment, in response to a failure to create a second request to execute a second thread, the first thread processes a second component of the set of sub-workloads until a threshold is reached. In another embodiment, method 1300 includes repeating the operation of creating a second request to execute a second thread after processing the second component of the set of sub-workloads until a threshold is reached.

[0181] Figure 14This document illustrates a block diagram of an instance machine 1400 in which any or more of the techniques (e.g., methods) discussed herein can be implemented, implemented therein, or implemented by. As described herein, an instance may contain logic or components or mechanisms in or operable by said logic or components or mechanisms. A circuit system (e.g., a processing circuit system) is a collection of circuits implemented in a tangible entity containing hardware (e.g., simple circuits, gates, logic, etc.) of machine 1400. The relationships between the members of a circuit system can change flexibly over time. A circuit system contains members that can perform specified operations individually or in combination during operation. In an instance, the hardware of the circuit system may be designed invariably to perform a specific operation (e.g., hardwired). In an instance, the hardware of the circuit system may contain variablely connected physical components (e.g., execution units, transistors, simple circuits, etc.) containing machine-readable media that are physically modified (e.g., magnetic, electrical, movable placement of particles of invariant mass, etc.) to encode instructions for a specific operation. When connecting physical components, the basic electrical properties of the hardware configuration change, for example, from insulator to conductor, or vice versa. Instructions enable embedded hardware (such as an execution unit or load mechanism) to generate members of a circuit system within the hardware via variable connections to perform specific operations in operation. Thus, in an example, a machine-readable media element is part of the circuit system or another component communicatively coupled to the circuit system during device operation. In an example, any of the physical components can be used in more than one member of more than one circuit system. For example, in operation, an execution unit may be used at one point in time in a first circuit of a first circuit system and reused at different times by a second circuit of the first circuit system or by a third circuit of the second circuit system. Additional examples of these components of machine 1400 are provided.

[0182] In alternative embodiments, machine 1400 may operate as a standalone device or be connected (e.g., networked) to other machines. In a networked deployment, machine 1400 may operate as a server machine, a client machine, or both in a server-client network environment. In an example, machine 1400 may act as a peer-to-peer (P2P) (or other distributed) network environment. Machine 1400 may be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, network device, network router, switch, or bridge, or any machine capable of (sequentially or otherwise) executing instructions specifying actions to be taken by the machine. Furthermore, while only a single machine is described, the term "machine" should also be considered as any collection of machines that individually or jointly execute one (or more) sets of instructions to perform any or more of the methods discussed herein (e.g., cloud computing, Software as a Service (SaaS), other computer cluster configurations).

[0183] Machine 1400 (e.g., a computer system) may include a hardware processor 1402 (e.g., a central processing unit (CPU), graphics processing unit (GPU), hardware processor core, or any combination thereof), main memory 1404, static memory 1406 (e.g., memory or storage device for firmware, microcode, basic input-output (BIOS), unified extensible firmware interface (UEFI), etc.), and mass storage device 1408 (e.g., a hard drive, tape drive, flash memory, or other block device), some or all of which may communicate with each other via interconnect 1430 (e.g., a bus). Machine 1400 may further include a display device 1410, an alphanumeric input device 1412 (e.g., a keyboard), and a user interface (UI) navigation device 1414 (e.g., a mouse). In an example, the display device 1410, the input device 1412, and the UI navigation device 1414 may be a touchscreen display. Machine 1400 may additionally include a mass storage device 1408 (e.g., a drive unit), a signal generation device 1418 (e.g., a speaker), a network interface device 1420, and one or more sensors 1416, such as a Global Positioning System (GPS) sensor, a compass, an accelerometer, or other sensors. Machine 1400 may include an output controller 1428, for example, for serial (e.g., Universal Serial Bus (USB)), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connections for communicating or controlling one or more peripheral devices (e.g., printers, card readers, etc.).

[0184] The registers of the hardware processor 1402, main memory 1404, static memory 1406, or mass storage device 1408 may be or contain machine-readable medium 1422 storing one or more sets of data structures or instructions 1424 (e.g., software) embodying any one or more of the techniques or functions described herein or used by any one or more of the techniques or functions described herein. Instructions 1424 may also reside entirely or at least partially in any of the registers of the hardware processor 1402, main memory 1404, static memory 1406, or mass storage device 1408 during execution by the machine 1400. In an example, one or any combination of the hardware processor 1402, main memory 1404, static memory 1406, or mass storage device 1408 may constitute machine-readable medium 1422. Although machine-readable media 1422 is described as a single medium, the term "machine-readable media" may include a single medium or multiple media (e.g., a centralized or distributed database or associated cache and server) configured to store one or more instructions 1424.

[0185] The term "machine-readable medium" may include any medium capable of storing, encoding, or transmitting instructions executable by machine 1400 and causing machine 1400 to perform any or more of the technologies disclosed herein, or any medium capable of storing, encoding, or transmitting data structures used by or associated with such instructions. Examples of non-limiting machine-readable media may include solid-state memory, optical media, magnetic media, and signals (e.g., radio frequency signals, other photon-based signals, sound signals, etc.). In examples, non-transitory machine-readable media includes machine-readable media having a plurality of particles with invariant (e.g., rest) mass and therefore being composed of matter. Therefore, non-transitory machine-readable media is machine-readable media that does not contain transiently propagating signals. Specific examples of non-transitory machine-readable media may include: non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0186] In this example, information stored or otherwise provided on machine-readable medium 1422 may represent instructions 1424, such as instructions 1424 itself or a format from which instructions 1424 may be derived. This format from which instructions 1424 may be derived may contain source code, encoded instructions (e.g., in compressed or encrypted form), packaged instructions (e.g., divided into multiple packages), or the like. The information representing instructions 1424 in machine-readable medium 1422 may be processed by a processing circuitry system into instructions that perform any of the operations discussed herein. For example, deriving instructions 1424 from information (e.g., processed by a processing circuitry system) may include: compiling (e.g., from source code, object code, etc.), interpreting, loading, organizing (e.g., dynamic or static linking), encoding, decoding, encrypting, decrypting, packaging, depackaging, or otherwise manipulating the information into instructions 1424.

[0187] In an example, the derivation of instruction 1424 may include assembly, compilation, or interpretation information (e.g., by processing a circuit system) to create instruction 1424 from some intermediate or preprocessed format provided by machine-readable media 1422. Information provided in multiple parts may be combined, unpacked, and modified to create instruction 1424. For example, the information may be in multiple compressed source code packages (or object code or binary executables, etc.) on one or more remote servers. The source code packages may be encrypted while transiting over a network and decoded, decompressed, assembled (e.g., linked) (if necessary), and compiled or interpreted (e.g., into libraries, standalone executables, etc.) at the local machine and executed by the local machine.

[0188] Instruction 1424 can be further transmitted or received via a communication network 1426 using a transmission medium through a network interface device 1420 that utilizes any of several transmission protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Example communication networks may include local area networks (LANs), wide area networks (WANs), packet data networks (e.g., the Internet), mobile phone networks (e.g., cellular networks), simple old-style telephone (POTS) networks, and wireless data networks (e.g., referred to as…). The Institute of Electrical and Electronics Engineers (IEEE) 802.11 series of standards, known as The network interface device 1420 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas for connection to the network 1426. In an example, the network interface device 1420 may include multiple antennas for wireless communication using at least one of Single-Input Multiple-Output (SIMO), Multiple-Input Multiple-Output (MIMO), or Multiple-Input Single-Output (MISO) technologies. The term "transmission medium" should be considered as any intangible medium capable of storing, encoding, or transmitting instructions executable by the machine 1400, and includes digital or analog communication signals or other intangible media that facilitate such software communication. The transmission medium is a machine-readable medium.

[0189] To better illustrate the methods and devices described herein, a set of non-limiting example embodiments are presented below as examples of digital recognition.

[0190] Example 1 is a device comprising: a hardware interface configured to receive a first request to execute a first thread, the first request including an indication of workload; and a processing circuitry configured to: determine the workload to generate a metric at least partially based on the indication; compare the metric with a threshold to determine that the metric exceeds the threshold; divide the workload into a set of sub-workloads consisting of a predefined number of identical portions from the workload, at least partially based on the comparison; create a second request to execute a second thread, the second request including a first component of the set of sub-workloads; and a second component to process the set of sub-workloads in the first thread.

[0191] In Example 2, the subject matter according to Example 1 includes, wherein the predefined number is 2.

[0192] In Example 3, the object described in Examples 1 to 2 includes, wherein the first thread is the main thread.

[0193] In Example 4, the subject matter according to Examples 1 to 3 includes the fiber thread of the second thread.

[0194] In Example 5, the subject matter described in Examples 1 to 4 includes, wherein the first request includes a configured busy failure field.

[0195] In Example 6, the object according to Example 5 includes, wherein the second request to execute the second thread fails, and wherein, in response to the failure of the second request to execute the second thread and the busy failure field is set, the processing circuitry is configured to: continue processing the second component of the set of subworkloads; and create a third request to execute the second thread, the third request including the first component of the set of subworkloads.

[0196] In Example 7, the subject matter described in Examples 5 and 6 includes wherein the busy failure field is a bit in a chip-to-chip protocol interface (CTCPI) packet.

[0197] In Example 8, the subject matter described in Examples 1 to 7 includes, wherein the second request includes a configured no-return field.

[0198] In Example 9, the subject matter according to Example 8 includes, wherein the no-return field is a bit in a chip-to-chip protocol interface (CTCPI) packet.

[0199] In Example 10, the object described in Examples 8 to 9 includes, wherein the no-return field is used to signal the second thread not to return a value to the stack position.

[0200] In Example 11, the object described in Examples 8 to 10 includes a no-return field that allows the first thread to avoid waiting for the second thread to return.

[0201] In Example 12, the object according to Examples 1 to 11 includes, in order to process the second component of the set of subworkloads, the processing circuitry is configured to: determine the second component to generate a second metric; compare the second metric with the threshold to determine that the second metric exceeds the threshold; divide the second component into another set of subworkloads consisting of the predefined number of identical parts from the second component, at least in part based on the comparison; create a third request to execute a third thread, the third request including the first component of the other set of subworkloads; and process the second component of the other set of subworkloads.

[0202] In Example 13, the subject matter according to Examples 1 to 12 includes an operation that creates the second request to execute the second thread that fails, and in response, the first thread will process the second component of the set of sub-workloads in the first thread until the threshold.

[0203] In Example 14, the subject matter according to Example 13 includes, wherein the processing circuitry system will repeatedly create the operation of executing the second request of the second thread after processing the second component of the set of sub-workloads up to the threshold.

[0204] Example 15 is a method comprising: receiving a first request to execute a first thread, the first request including an indication of workload; determining the workload to generate a metric based at least in part on the indication; comparing the metric with a threshold to determine that the metric exceeds the threshold; dividing the workload into a set of sub-workloads consisting of a predefined number of identical portions from the workload, based at least in part on the comparison; creating a second request to execute a second thread, the second request including a first component of the set of sub-workloads; and processing the set of sub-workloads in the first thread.

[0205] In Example 16, the subject matter according to Example 15 includes, wherein the predefined number is 2.

[0206] In Example 17, the subject matter described in Examples 15 and 16 includes, wherein the first thread is the main thread.

[0207] In Example 18, the subject matter according to Examples 15 to 17 includes the fiber thread of the second thread.

[0208] In Example 19, the subject matter as described in Examples 15 through 18 includes, wherein the first request includes a configured busy failure field.

[0209] In Example 20, the object according to Example 19 includes, wherein the second request to execute the second thread fails, and wherein in response to the failure of the second request to execute the second thread and the busy failure field is set, the method includes: continuing to process the second component of the set of subworkloads; and creating a third request to execute the second thread, the third request including the first component of the set of subworkloads.

[0210] In Example 21, the subject matter described in Examples 19 and 20 includes wherein the busy failure field is a bit in a chip-to-chip protocol interface (CTCPI) packet.

[0211] In Example 22, the subject matter as described in Examples 15 to 21 includes, wherein the second request includes a configured no-return field.

[0212] In Example 23, the subject matter according to Example 22 includes, wherein the no-return field is a bit in a chip-to-chip protocol interface (CTCPI) packet.

[0213] In Example 24, the object described in Examples 22 to 23 includes, wherein the no-return field is used to signal the second thread not to return a value to the stack position.

[0214] In Example 25, the object described in Examples 22 to 24 includes a no-return field that allows the first thread to avoid waiting for the second thread to return.

[0215] In Example 26, the object described according to Examples 15 to 25 includes a second component that processes the set of subworkloads comprising: determining the second component to generate a second metric; comparing the second metric with the threshold to determine that the second metric exceeds the threshold; dividing the second component into another set of subworkloads composed of the predefined number of identical portions from the second component, at least in part based on the comparison; creating a third request to execute a third thread, the third request including a first component of the other set of subworkloads; and a second component that processes the other set of subworkloads.

[0216] In Example 27, the subject matter according to Examples 15 to 26 includes, in response to the failure of the second request to create and execute the second thread, the first thread will process the second component of the set of sub-workloads in the first thread until the threshold.

[0217] In Example 28, the object described in Example 27 includes repeating the second request to create and execute the second thread after processing the second component of the set of sub-workloads up to the threshold.

[0218] Example 29 is a device comprising: a hardware interface configured to receive a first request to execute a first thread, the first request including an indication of workload; and a processing circuitry configured to: recursively divide the workload into a plurality of sub-workloads and cause a sub-thread to process each of the plurality of sub-workloads until the sub-workloads are less than a threshold size; and process the sub-workloads less than the threshold size in the sub-threads.

[0219] In Example 30, the object described in Example 29 includes a workload that is divided into two sub-workloads.

[0220] In Example 31, the subject matter according to Examples 29 and 30 includes, wherein the first thread is the main thread.

[0221] In Example 32, the subject matter according to Examples 29 to 31 includes the fiber thread of the sub-thread.

[0222] In Example 33, the subject matter as described in Examples 29 to 32 includes, wherein the first request includes a configured busy failure field.

[0223] In Example 34, the object described in Examples 29 to 33 includes a sub-thread that processes the sub-workload and does not return a value to the calling thread.

[0224] In Example 35, the object described in Examples 29 to 34 includes an instruction for executing the sub-thread to process the sub-workload, which includes a set no-return field.

[0225] In Example 36, the subject matter according to Example 35 includes, wherein the no-return field is a bit in a chip-to-chip protocol interface (CTCPI) packet.

[0226] In Example 37, the object described in Examples 35 and 36 includes, wherein the no-return field is used to signal the child thread not to return a value to the stack position.

[0227] In Example 38, the object described in Examples 35 to 37 includes a no-return field that allows the first thread to avoid waiting for the child thread to return.

[0228] Example 39 is at least one machine-readable medium containing instructions that, when executed by a processing circuitry system, cause the processing circuitry system to perform any of the operations described in Examples 1 to 38.

[0229] Example 40 is a device that includes components for implementing any of Examples 1 to 38.

[0230] Example 41 is a system that implements any one of Examples 1 to 38.

[0231] Example 42 is a method for implementing any of Examples 1 through 38.

[0232] The detailed description above includes reference to the accompanying drawings, which form a part of the detailed description. The drawings illustrate, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are also referred to herein as “examples.” Such examples may include elements other than those shown or described. However, the inventors also contemplate examples in which only those elements shown or described are provided. Furthermore, the inventors contemplate examples (or examples) using any combination or arrangement of those elements shown or described with respect to a particular example (or one or more aspects thereof) or with respect to other examples (or one or more aspects thereof) shown or described herein.

[0233] In this document, as is common in patent documents, the term "a / an" is used to include one or more, independent of any other example or use of "at least one" or "one or more". In this document, the term "or" is used to refer to a non-exclusive "or", such that "A or B" may include "A but not B", "B but not A", and "A and B", unless otherwise indicated. In the appended claims, the terms "comprising" and "in which" are used as their common English equivalents to the corresponding terms "including" and "wherein". Furthermore, in the appended claims, the terms "comprising" and "including" are open-ended, meaning that a system, apparatus, article, or process that includes elements other than those listed after the term is still considered to fall within the scope of the claim. Additionally, in the appended claims, the terms "first", "second", and "third", etc., are used only as designations and are not intended to impose numerical requirements on their objects.

[0234] The foregoing description is intended to be illustrative and not restrictive. For example, the above examples (or one or more aspects thereof) may be used in combination with each other. Other embodiments may be used by those skilled in the art, for instance, after reviewing the foregoing description. It should be understood that it is not intended to interpret or limit the scope or meaning of the claims. Furthermore, in the detailed description above, various features may be grouped together to simplify this disclosure. This should not be interpreted as an intention that unclaimed features are necessary for any claim. Rather, the subject matter of the invention may have not all features of a particular disclosed embodiment. Therefore, the appended claims are hereby incorporated into the detailed description, wherein each claim is itself an individual embodiment, and such embodiments are contemplated to be combined or arranged in various ways. The scope of the invention should be determined with reference to the appended claims and the full scope of the equivalents granted by these claims.

Claims

1. An apparatus comprising: A hardware interface configured to receive a first request to execute a first thread, the first request including an indication of workload and a configured busy failure field; and The processing circuit system is configured to: The workload is determined to generate a metric based at least in part on the indication; The metric is compared with a threshold to determine if the metric exceeds the threshold; The workload is divided into a set of sub-workloads consisting of a predefined number of identical parts from the workload, at least in part based on the comparison. Create a second request to execute the second thread, the second request containing the first component of the set of subworkloads; and In the first thread, the second component of the set of sub-workloads is processed, wherein executing the second request of the second thread fails, and wherein in response to the failure of executing the second request of the second thread and the busy failure field is set, the processing circuitry is configured to: Continue processing the second component of the set of sub-workloads; and A third request is created to execute the second thread, the third request containing the first component of the set of subworkloads.

2. The device according to claim 1, wherein the predefined number is 2.

3. The device according to claim 1, wherein the first thread is the main thread.

4. The device according to claim 1, wherein the second thread is a fiber thread.

5. The device according to claim 1, wherein the busy failure field is a bit in a chip-to-chip protocol interface (CTCPI) packet.

6. The device of claim 1, wherein the second request includes a configured no-return field.

7. The device of claim 6, wherein the no-return field is a bit in a chip-to-chip protocol interface (CTCPI) packet.

8. The device of claim 6, wherein the no-return field is used to signal the second thread not to return a value to the stack position.

9. The device of claim 6, wherein the no-return field allows the first thread to avoid waiting for the second thread to return.

10. The apparatus of claim 1, wherein, in order to process the second component of the set of sub-workloads, the processing circuitry is configured to: The second component is determined to generate the second metric; Compare the second metric with the threshold to determine if the second metric exceeds the threshold; The second component is divided into another set of sub-workloads, consisting of the predefined number of identical parts from the second component, based at least in part on the comparison. Create a third request to execute a third thread, the third request containing the first component of the other set of sub-workloads; and The second component that processes the other set of sub-workloads.

11. The device of claim 1, wherein the operation of creating the second request to execute the second thread fails, and in response, the first thread will process the second component of the set of sub-workloads in the first thread until the threshold.

12. The device of claim 11, wherein the processing circuitry system will repeatedly create the operation of executing the second request of the second thread after processing the second component of the set of sub-workloads up to the threshold.

13. A method comprising: Receive a first request to execute the first thread, the first request containing an indication of workload and a set busy failure field; The workload is determined to generate a metric based at least in part on the indication; The metric is compared with a threshold to determine if the metric exceeds the threshold; The workload is divided into a set of sub-workloads consisting of a predefined number of identical parts from the workload, at least in part based on the comparison. Create a second request to execute the second thread, the second request containing the first component of the set of subworkloads; and The method includes processing a second component of the set of sub-workloads in a first thread, wherein executing the second request of the second thread fails, and wherein the busy failure field is set in response to the failure of executing the second request of the second thread. Continue processing the second component of the set of sub-workloads; and A third request is created to execute the second thread, the third request containing the first component of the set of subworkloads.

14. The method of claim 13, wherein the second component processing the set of sub-workloads comprises: The second component is determined to generate the second metric; Compare the second metric with the threshold to determine if the second metric exceeds the threshold; The second component is divided into another set of sub-workloads, consisting of the predefined number of identical parts from the second component, based at least in part on the comparison. Create a third request to execute a third thread, the third request containing the first component of the other set of sub-workloads; and The second component that processes the other set of sub-workloads.

Images