Techniques for providing a scalable architecture for performing in-memory computational operations - Patents.com

A scalable memory architecture with tensor logic units and computational logic units addresses inefficiencies in AI operations by performing computations directly in memory, enhancing speed and energy efficiency for complex tensor operations.

JP7581605B2Active Publication Date: 2024-11-13INTEL CORP
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
JP2020013821
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-03-29
Filing Date
2020-01-30
Publication Date
2024-11-13
Estimated Expiration
2040-01-30

AI Technical Summary

Technical Problem

Existing computing systems face inefficiencies in energy usage and speed when performing tensor operations for artificial intelligence applications due to the need to transfer data from memory to computational devices, which consumes time and energy, especially as data complexity increases.

Method used

A scalable architecture is implemented in computing devices that performs computational operations directly in memory using a processor coupled with a memory medium featuring a three-dimensional cross-point architecture and media access circuitry, including tensor logic units and computational logic units to perform operations efficiently.

Benefits of technology

This approach enhances computational efficiency by reducing the need for data transfer, allowing for faster and more energy-efficient performance of complex AI operations, such as matrix multiplications, by distributing operations across multiple computational logic units.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007581605000001
    Figure 0007581605000001
  • Figure 0007581605000002
    Figure 0007581605000002
  • Figure 0007581605000003
    Figure 0007581605000003
Patent Text Reader

Abstract

To provide a scalable architecture for efficiently performing compute operations in memory.SOLUTION: In a scalable architecture for more efficiently performing compute operations by performing compute operations in a memory 104 rather than calculating data by transmitting the data to a processor 102 via a bus, the memory has memory media and media access circuitry coupled to the memory media. The media access circuitry is to access data from the memory media to perform a requested operation, perform, with each of multiple compute logic units included in the media access circuitry, the requested operation concurrently on the accessed data, and write, to the memory media, resultant data produced from execution of the requested operation.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Background technology]

[0001] Artificial intelligence applications, such as applications that use neural networks to train neural networks and / or perform inference (e.g., to identify objects in images, perform voice recognition, etc.), typically use a relatively large amount of computing power to perform tensor operations (e.g., matrix calculations such as matrix multiplication) on matrix data. In some computing devices, the computational operations to support artificial intelligence applications may be offloaded from a general-purpose processor to an accelerator device, such as a graphics processing unit (GPU). However, while a GPU may be able to perform tensor operations faster than a processor, the efficiency (e.g., energy use and speed) with which a computing device can perform its operations is still impaired by the fact that the data to be computed (e.g., matrix data) resides in memory and is transmitted from memory over a bus to the device that performs the computational operations (e.g., the GPU), consuming time and energy. As the complexity and amount of data to be computed increases (e.g., with increasingly complex artificial intelligence applications), the inefficiency of existing systems in terms of energy use and speed may correspondingly increase. [Brief description of the drawings]

[0002] The concepts described herein are illustrated by way of example, and not by way of limitation, in the accompanying drawings. For simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. Where considered appropriate, reference labels have been repeated among the figures to indicate corresponding or analogous elements.

[0003] [Figure 1] 1 is a simplified diagram of at least one embodiment of a computing device having a scalable architecture for efficiently performing computational operations in memory.

[0004] [Diagram 2]2 is a simplified diagram of at least one embodiment of a memory medium included in the computing device of FIG. 1;

[0005] [Diagram 3] 2 is a simplified diagram of at least one embodiment of a partition of a memory medium and components of memory medium access circuitry included in the computing device of FIG. 1;

[0006] [Figure 4] 2 is a simplified diagram of at least one embodiment of a pair of dual in-line memory modules that may be included in the computing device of FIG. 1.

[0007] [Diagram 5] 2 is a simplified diagram of at least one embodiment of a tensor operation that may be performed in the memory of the computing device of FIG. 1.

[0008] [Figure 6] 2 is a simplified diagram of at least one embodiment of a method for performing efficient computational operations in a memory that may be performed by the computing device of FIG. 1; [Figure 7] 2 is a simplified diagram of at least one embodiment of a method for performing efficient computational operations in a memory that may be performed by the computing device of FIG. 1; [Figure 8] 2 is a simplified diagram of at least one embodiment of a method for performing efficient computational operations in a memory that may be performed by the computing device of FIG. 1;

[0009] [Figure 9] 2 is a simplified diagram of two dies of memory of the computing device of FIG. 1 integrated into a package. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] While the concepts of the present disclosure are susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and are herein described in detail. It should be understood, however, that there is no intention to limit the concepts of the present disclosure to the particular forms disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives not inconsistent with the scope of the present disclosure and the appended claims.

[0011] References herein to "one embodiment," "an embodiment," "an exemplary embodiment," and the like indicate that the described embodiment may include a particular feature, structure, or characteristic, but that not all embodiments may or may not include that particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in connection with one embodiment, it is believed to be within the knowledge of one of ordinary skill in the art to achieve such feature, structure, or characteristic in connection with other embodiments, whether or not explicitly described. Furthermore, it should be appreciated that an item included in a list in the form "at least one of A, B, and C" may mean (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). Similarly, an item included in a list in the form "at least one of A, B, or C" may mean (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C).

[0012] The disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof, in some cases. The disclosed embodiments may be implemented as instructions held or stored by a transitory or non-transitory machine-readable (e.g., computer-readable) storage medium, which may be read and executed by one or more processors. A machine-readable storage medium may be embodied as any storage device, mechanism, or other physical structure (e.g., a volatile or non-volatile memory, a media disk, or other media device) for storing or transmitting information in a form readable by a machine.

[0013] In the figures, some structural or method features may be shown in a particular arrangement and / or order. However, it should be appreciated that such a particular arrangement and / or order may not be required. Rather, in some embodiments, such features may be arranged in a different manner and / or order than that shown in the example figures. Also, the inclusion of a structural or method feature in a particular figure does not imply that such feature is required in all embodiments, and in some embodiments may not be included or may be combined with other features.

[0014] 1, a computing device 100 for efficiently executing artificial intelligence operations (e.g., tensor operations) in memory includes a processor 102, a memory 104, an input / output (I / O) subsystem 112, a data storage device 114, and communications circuitry 122. Of course, in other embodiments, the computing device 100 may include other or additional components such as those typically found in a computer (e.g., a display, peripherals, etc.). Also, in some embodiments, one or more of these illustrative components may be incorporated in or form part of another component. As described herein, computing device 100, and in particular memory 104 of computing device 100, has a scalable architecture in which one or more circuit boards (e.g., dual in-line memory modules (DIMMs) 160) adapted to be removably connected (e.g., inserted) into computing device 100 each include a set of packages (e.g., each being a set of one or more semiconductor dies 190, 192) 150, 152 having a computational logic unit (e.g., computational logic unit 136) configured to perform computational operations such as tensor operations (e.g., matrix calculations) on data present in memory 104. By performing computational operations closer to the data (e.g., in memory 104) rather than sending the data over a bus to another component (e.g., processor 102) to operate on the data, computing device 100 may perform operations more efficiently. Furthermore, by adding additional DIMMs or similar circuit boards having computational logic units 136 to memory 104, the ability of computing device 100 to perform more complex computational operations (e.g., more complex artificial intelligence operations) on the data in memory 104 can be correspondingly expanded.

[0015] As described in more detail herein, in an exemplary embodiment, memory 104 includes medium access circuitry 108 configured to access memory medium 110 to perform computational operations. In an exemplary embodiment, memory medium 110 has a three-dimensional cross-point architecture that allows access of one bit per tile and has data access characteristics that differ from other memory architectures (e.g., dynamic random access memory (DRAM)) that incur time delays between reads or writes to the same or other partitions. In an exemplary embodiment, 128 bits may be read from one partition at a time.

[0016] Processor 102 may be embodied as any device or circuitry (e.g., a multi-core processor, microcontroller, or other processor or processing / control circuitry) capable of performing the operations described herein, such as executing an application (e.g., an artificial intelligence related application that may learn and infer using neural networks or other machine learning structures). In some embodiments, processor 102 may be embodied as, include, or be coupled to an FPGA, an application specific integrated circuit (ASIC), reconfigurable hardware or hardware circuitry, or other specialized hardware to facilitate performance of the functions described herein.

[0017] The memory 104, which may include a non-volatile memory (e.g., a far memory in a two-level memory scheme), includes a memory medium 110 and a media access circuit 108 (e.g., a device or circuit such as an integrated circuit constructed of complementary metal-oxide semiconductor (CMOS) or other materials) that is located underneath (e.g., at a lower position) and coupled to the memory medium 110. The media access circuit 108 is also connected to a memory controller 106, which may be embodied as any device or circuit (e.g., a processor, a co-processor, a dedicated circuit, etc.) configured to selectively read from and / or write to the memory medium 110, as well as to perform computational operations (e.g., tensor operations) on data (e.g., matrix data) present on the memory medium 110 (e.g., in response to requests from the processor 102, which may be running an artificial intelligence related application that relies on tensor operations to train a neural network and / or perform inference). Briefly referring to FIG. 2, in an exemplary embodiment, the memory medium 110 includes a tile architecture, also referred to herein as a cross-point architecture (e.g., an architecture in which memory cells are at the intersections of word lines and bit lines and are individually addressable, and bit storage is based on changes in bulk resistance), in which each tile (e.g., memory cell) is addressable by x and y parameters (e.g., row and column). The memory medium 110 includes multiple partitions, each of which includes a tile architecture. The partitions may be stacked as layers 202, 204, 206 to form a three-dimensional cross-point architecture (e.g., Intel 3D XPoint™ memory). Unlike typical memory devices, in which only fixed-size multi-bit data structures (e.g., bytes, words, etc.) are addressable, the media access circuitry 108 is configured to read individual bits or other units of data from the memory medium 110 upon request of the memory controller 106, which may generate the request in response to receiving a corresponding request from the processor 102.In an exemplary embodiment, 128 bits can be accessed (eg, read or written) simultaneously in a given partition.

[0018] 1, in an exemplary embodiment, media access circuitry 108 includes a tensor logic unit 130, which may be embodied as any device or circuitry (e.g., a CMOS circuit) configured to offload the performance of computational operations, such as tensor operations, from other portions of media access circuitry 108. In an exemplary embodiment, tensor logic unit 130 includes a plurality of memory scratch pads 132, each of which may be embodied as any device or circuitry (e.g., a static random access memory (SRAM), a register file, etc.) usable to provide relatively fast (e.g., low latency) access to data, such as matrix data, read from memory medium 110. In an exemplary embodiment, scratch pad 132 provides faster read and write access times than memory medium 110, which has a relatively slower access time and larger capacity. Tensor logic unit 130 may further include an error correction code (ECC) logic unit 134, which may be embodied as any device or circuit (e.g., a reconfigurable circuit, an application specific integrated circuit (ASIC), etc.) configured to determine whether data read from memory medium 110 contains errors and to correct any errors using an error correction algorithm such as a Reed-Solomon code or a Bose-Chaudhuri-Ockhenghem (BCH) code. Also, in an exemplary embodiment, tensor logic unit 130 includes a plurality of computation logic units 136, each of which may be embodied as any device or circuit (e.g., a reconfigurable circuit, an ASIC, etc.) configured to perform computational operations (e.g., tensor operations) on data (e.g., matrix data) in a corresponding set of scratch pads 132.

[0019] 3, in an exemplary embodiment, components of memory 104 are divided into clusters 310, 320, 330. Cluster 310 includes a plurality of partitions 311 of memory medium 110, a set of scratch pads 312, 314, 316, each similar to scratch pad 132 of FIG. 1, and a corresponding compute logic unit 318, similar to compute logic unit 136 of FIG. 1. Similarly, cluster 320 includes another set of partitions 321 of memory medium 110, a corresponding set of scratch pads 322, 324, 326, and a corresponding compute logic unit 328. Cluster 330 also includes a set of partitions 331 of memory medium 110, a corresponding set of scratch pads 332, 334, 336, and a compute logic unit 338. In some embodiments, a compute logic unit 136 (e.g., compute logic unit 318) may broadcast (e.g., provide) a set of data from a partition (e.g., one of partitions 311) associated with that compute logic unit 318 to one or more other compute logic units 136 (e.g., to scratch pads 132 associated with those other compute logic units 136). By broadcasting data read from a corresponding set of partitions of memory medium 110 to other scratch pads, media access circuitry 108 reduces the number of times a given section (e.g., set of partitions) of memory medium 110 must be accessed to obtain matrix data (e.g., read matrix data may be broadcast to multiple scratch pads after being read once from memory medium 110, rather than reading the same matrix data from memory medium 110 multiple times). Furthermore, by using multiple compute logic units 318, 328, 338, each associated with a corresponding scratch pad 312, 314, 316, 322, 324, 326, 332, 334, 336, the media access circuit 108 can perform portions of the tensor operations (e.g., matrix multiplication and accumulation) simultaneously (e.g., in parallel).For simplicity, three clusters 310, 320, 330 are shown in FIG. 3, but it should be understood that the actual number of clusters and corresponding partitions, scratch pads, and computational logic units may vary depending on the particular embodiment. With brief reference to FIG. 9, the memory medium 110 may be included on a die 190, and the media access circuitry 108 may be included on another die 192. Each die 190, 192 may be embodied as a pattern on a semiconductor wafer that defines a circuit for performing a set of functions. As shown in FIG. 9, both dies 192, 192 may be integrated into a package 150. The package 150 may include an enclosure (e.g., a metal enclosure, a plastic enclosure, a glass enclosure, a ceramic enclosure, etc.) that houses the dies 190, 192. In other embodiments, the memory medium 110 and the media access circuitry 108 are located on the same die.

[0020] Briefly referring to FIG. 4, a set 400 of dual in-line memory modules (DIMMs) 420, 430 is shown. The DIMM 420 includes a number of packages 402, 404, 406, 408, 410, 412, 414, and 416, each similar to the package 150 of FIG. 1 and FIG. 9. Moreover, in an exemplary embodiment, each package includes 64 partitions, with one compute logic unit 136 assigned to each set of four partitions. Thus, in a DIMM with eight packages (e.g., DIMM 420), the DIMM has 128 compute logic units 136. By adding another DIMM (e.g., DIMM 430) to the memory 104, an additional 128 compute logic units 136 are provided, doubling the computational capacity within the memory. Thus, by successively adding DIMMs to the memory 104, the computational capacity within the memory can be expanded, significantly increasing the ability of the computing device 100 to efficiently perform artificial intelligence operations (e.g., tensor operations). In some embodiments, a DIMM (e.g., DIMM 420) may include one die having media access circuitry 108 and multiple other dies of memory medium 110 such that the entire DIMM 420 has a similar set of components (e.g., dies) as a package (e.g., package 150). In other embodiments, one or more of the packages (e.g., packages 402, 404, 406, 408, 410, 412, 414, 416) may be located on another type of circuit board other than a DIMM. For example, the packages may be included on a relatively small circuit board that can be inserted into a universal serial bus (USB) interface of computing device 100.

[0021] Briefly referring to FIG. 5, an example of a matrix multiplication (e.g., matrix multiplication and subtraction) operation 500 that may be performed by memory 104 is shown. As illustrated, matrix data in an input matrix A is multiplied with matrix data in another matrix B (e.g., weight data for a layer of a convolutional neural network), and the resulting data is written to an output matrix C. Each matrix shown in FIG. 5 is temporarily stored as matrix data in a scratch pad 132 of the media access circuit 108. In some embodiments, the output matrix C may be used as an input matrix for a subsequent tensor operation (e.g., as an output matrix for a subsequent layer of a convolutional neural network).

[0022] Referring back to FIG. 1 , memory 104 may include non-volatile and volatile memories. Non-volatile memory may be embodied as any type of data storage capable of storing data in a persistent manner (even if power is interrupted to the non-volatile memory). For example, non-volatile memory may be embodied as one or more non-volatile memory devices. Non-volatile memory devices may include one or more storage devices configured with a cross-point architecture that allows for bit-level addressability (e.g., the ability to read and / or write individual bits of data rather than bytes or other larger units of data), illustratively embodied as three-dimensional (3D) cross-point memories. In some embodiments, non-volatile memory may further include other types of memory, including any combination of storage devices using chalcogenide phase change materials (e.g., chalcogenide glasses), ferroelectric transistor random access memory (FeTRAM), nanowire-based non-volatile memory, phase change memory (PCM), memory incorporating memristor technology, magnetoresistive random access memory (MRAM), or spin transfer torque (STT)-MRAM. Volatile memory may be embodied as any type of data storage capable of storing data while power is supplied to the volatile memory. For example, volatile memory may be embodied as one or more volatile storage devices, and in other embodiments, may be embodied as other types of non-persistent data storage, and will be referred to hereinafter periodically as volatile memory. Volatile memory may have an architecture that allows for bit-level addressability, similar to the architectures described above.

[0023] The processor 102 and memory 104 are communicatively coupled to other components of the computing device 100 via an I / O subsystem 112, which may be embodied as circuits and / or components for facilitating input / output operations between the processor 102 and / or main memory 104 and other components of the computing device 100. For example, the I / O subsystem 112 may be embodied as or include a memory controller hub, an input / output control hub, an integrated sensor hub, a firmware device, communications links (e.g., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.), and / or other components and subsystems for facilitating input / output operations. In some embodiments, the I / O subsystem 112 may form part of a system-on-chip (SoC) and may be incorporated within a single chip along with one or more of the processor 102, main memory 104, and other components of the computing device 100.

[0024] The data storage device 114 may be embodied as any type of device configured for short-term or long-term storage of data, such as, for example, memory devices and circuits, memory cards, hard disk drives, solid state drives, or other data storage devices. In an exemplary embodiment, the data storage device 114 includes a memory controller 116 similar to the memory controller 106, a storage medium 120 similar to the memory medium 110, and a media access circuit 118 similar to the media access circuit 108, including a tensor logic unit 140 similar to the tensor logic unit 130, a scratch pad 142 similar to the scratch pad 132, an ECC logic unit 144 similar to the ECC logic unit 134, and a compute logic unit 146 similar to the compute logic unit 136. Additionally, similar to the memory 104, the storage medium 120 and the media access circuit 118 may reside in multiple packages 170, 172 (e.g., each having a set of one or more dies 194, 196) and multiple DIMMs 180. Thus, in an exemplary embodiment, data storage device 114 (e.g., medium access circuitry 118) can efficiently perform computational operations (e.g., tensor operations) on data (e.g., matrix data) stored on storage medium 120, and the computational capacity can be easily expanded by adding additional DIMMs 180. Data storage device 114 may include a system partition that stores data and firmware code for data storage device 114, and one or more operating system partitions that store data files and executable files for the operating system.

[0025] Communications circuitry 122 may be embodied as any communications circuitry, device, or collection thereof capable of enabling communications between computing device 100 and another device over a network. Communications circuitry 122 may be configured to use any one or more communications technologies (e.g., wired or wireless communications) and associated protocols (e.g., Ethernet, Bluetooth, Wi-fi, WiMAX, etc.) to achieve such communications.

[0026] Exemplary communication circuitry 122 includes a network interface controller (NIC) 122, which may also be referred to as a host fabric interface (HFI). NIC 124 may be embodied as one or more add-in boards, daughter cards, network interface cards, controller chips, chipsets, or other devices that may be used by computing device 100 to connect with another computing device. In some embodiments, NIC 124 may be embodied as part of a system-on-chip (SoC) that includes one or more processors, or may be included in a multi-chip package that also includes one or more processors. In some embodiments, NIC 124 may include a local processor (not shown) and / or local memory (not shown), both of which are local to NIC 124. In such embodiments, the local processor of NIC 124 may be capable of performing one or more of the functions of processor 102. Additionally or alternatively, in such embodiments, the local memory of NIC 124 may be integrated into one or more components of computing device 100 at the board level, socket level, chip level, and / or other levels.

[0027] Referring now to FIG. 6, the computing device 100 may perform a method 600 for performing efficient in-memory (e.g., in memory 104) computational operations during operation. The method 600 is described with reference to the memory 104. However, it should be understood that the method 600 may additionally or alternatively be performed using the memory of the data storage device 114. The method 600 begins at block 602, where the computing device 100 (e.g., memory 104) determines whether to enable efficient execution of computational operations (e.g., tensor operations) in the memory 104. The computing device 100 may enable efficient execution of computational operations in the memory 104 in response to a determination that the media access circuitry 108 includes a tensor logic unit 130 in response to a determination that a configuration setting (e.g., in a configuration file) indicates to enable efficient execution of computational operations in memory, and / or based on other factors.

[0028] Regardless, in response to a determination to enable efficient in-memory execution of computational operations, method 600 proceeds to block 604, where computing device 100 may obtain a request to perform one or more operations on data in memory 104. For example, as shown in block 606, memory 104 (e.g., media access circuitry 108) may receive a request from a processor (e.g., processor 102) that may be running an artificial intelligence related application (e.g., an application that may use a neural network or other machine learning structure to learn and infer). As shown in block 608, memory 104 (e.g., media access circuitry 108) may receive a request that includes a descriptor (e.g., a parameter or other data) that indicates a location (e.g., an address) of data to be operated on in memory 104. As shown in block 610, memory 104 may receive a request that includes a descriptor that indicates a dimension of a matrix to be operated on in memory 104.

[0029] Method 600 then proceeds to block 612, where computing device 100 accesses data associated with the requested operation (e.g., data to be operated on) from a memory medium (e.g., memory medium 110) included in memory 104 using media access circuitry (e.g., media access circuitry 108) included in memory 104. As shown in block 614, in an exemplary embodiment, computing device 100 accesses data (e.g., from memory medium 110) using complementary metal oxide semiconductor (CMOS) (e.g., media access circuitry 108 may be formed from CMOS). Also, as shown in block 616, in an exemplary embodiment, memory 104 (e.g., media access circuitry 108) reads data from a memory medium (e.g., memory medium 110) having a cross-point architecture (e.g., an architecture in which memory cells are individually addressable at the intersections of word lines and bit lines and bit storage is based on changes in bulk resistance). Additionally, as indicated at block 618, the media access circuit 108 may read the matrix data from a memory medium (e.g., memory medium 110) having a three-dimensional cross-point architecture (e.g., an architecture in which sets of tiles are stacked in layers, as described with reference to FIG. 2). In other embodiments, the computing device 100 may read data from a memory medium having a different architecture.

[0030] As shown in block 620, the computing device 100 (e.g., media access circuitry 108) may read data from multiple partitions of the memory medium 110 (e.g., simultaneously). In doing so, as shown in block 622, the computing device 100 may read a predefined number of bits (e.g., 128 bits) from each partition. As shown in block 624, the computing device 100 (e.g., media access circuitry 108) may read data into a scratch pad associated with a corresponding partition. For example, the media access circuitry 108 may read data from one of the partitions 311 into scratch pad 312, data from one of the partitions 321 into scratch pad 322, etc. In reading the data into the scratch pad, the computing device 100 (e.g., media access circuitry 108) may read the data into a register file (e.g., an array of registers) or a static random access memory (SRAM), respectively. The method 600 then proceeds to block 630 of FIG. 7, where the computing device 100 performs at least a portion of the requested operation on the data accessed from the memory medium 110 using the media access circuitry 108.

[0031] 7, in performing at least a portion of the requested operation, as shown in block 632, the computing device 100 (e.g., the media access circuitry 108) performs the operation using the compute logic units 136 in the media access circuitry 108. As shown in block 634, in an exemplary embodiment, the computing device 100 performs the operation (e.g., a sub-operation of the requested operation, such as operating on one or more portions of a matrix) simultaneously across multiple compute logic units 136. As shown in block 636, the computing device 100 may perform the operation (e.g., a sub-operation of the requested operation) using a compute logic unit associated with the partition from which the data was read. In an exemplary embodiment, as shown in block 638, the computing device 100 performs the operation on the data loaded into a scratch pad associated with the partition from which the data was loaded (e.g., scratch pads 312, 314, 316 associated with partition 311, scratch pads 322, 324, 326 associated with partition 321, etc.). As shown in block 640, computing device 100 may perform operations using multiple computational logic units (e.g., computational logic units 318, 328, 338) located on a die or package (e.g., package 402). Additionally, as shown in block 642, computing device 100 may perform operations using multiple computational logic units distributed across multiple dies or packages (e.g., packages 402, 404, 406, 408, 410, 412, 414, 416) of a DIMM (e.g., DIMM 420). Additionally, as shown in block 644, computing device 100 may perform operations using multiple computational logic units distributed across multiple DIMMs (e.g., DIMMs 420, 430).

[0032] As shown in block 646, in performing the operations, the computing device 100 (e.g., the media access circuitry 108) may perform one or more tensor operations on the data accessed from the memory medium 110. In doing so, as shown in block 648, the computing device 100 (e.g., the media access circuitry 108) may perform one or more matrix multiplication operations (e.g., multiplying an input matrix A by a weight matrix B to obtain an output matrix C). As shown in block 650, in performing the one or more matrix multiplication operations, the computing device 100 (e.g., the media access circuitry 108) may perform one or more matrix multiply-accumulate operations. As shown in block 652, in other embodiments, the computing device 100 (e.g., the media access circuitry 108) may perform other types of operations (e.g., other than tensor operations) on the data. For example, as shown in block 654, the computing device 100 (e.g., the media access circuitry 108) may perform one or more bitwise operations (e.g., logical operations on individual bits such as AND, OR, XOR, NOR) on the data accessed from the memory medium 110.

[0033] Method 600 then proceeds to block 656, where computing device 100 determines whether to access additional data to continue performing the requested operation. For example, if a tensor operation is requested and only a portion of the matrix data associated with the tensor operation (e.g., less than all of the rows and / or columns) has been operated on, computing device 100 may determine whether additional data (e.g., additional matrix data) should be accessed to complete the requested operation. If computing device 100 determines to access additional data, method 600 loops back to block 612 of FIG. 6 to access the additional data. Otherwise, method 600 proceeds to block 658 of FIG. 8, where computing device 100 (e.g., media access circuitry 108) may write result data generated from performing the requested operation to memory (e.g., memory medium 110).

[0034] 8, in writing the result data, as shown in block 660, the media access circuit 108 may first write the result data to one or more scratch pads each associated with a set of partitions of the memory medium 110 (e.g., adding the result of multiplication of the input matrix A from the scratch pad 312 with the weight matrix B from the scratch pad 314 to the scratch pad 316, adding the result of multiplication of the input matrix A from the scratch pad 322 with the weight matrix B from the scratch pad 324 to the scratch pad 326, etc.). As shown in block 662, the media access circuit 108 may write the result data to scratch pads (e.g., scratch pads 316, 326, 336) each holding an output matrix (e.g., matrix C) to be used as an input matrix for a layer (e.g., a subsequent layer) of the convolutional neural network. As shown in block 664, the media access circuit 108 may write the result data from the scratch pads 316, 326, 336 to the memory medium 110 (e.g., the corresponding partitions). Memory 104 may then provide data to another component of computing device 100 indicating completion of the requested operation, at block 666. In doing so, memory 104 may provide result data (e.g., from block 658) to a processor (e.g., processor 102) executing an application (e.g., the application that generated the request for the operation to be executed in memory 104), as shown in block 668. Memory 104 may provide result data indicative of an artificial intelligence operation to another component of computing device 100 (e.g., processor 102), as shown in block 670. For example, memory 104 may provide data indicative of an inference (e.g., identification of an object in an image, etc.), as shown in block 672. Method 600 then loops back to block 602 of FIG. 6, where computing device 100 determines whether to continue to enable efficient in-memory computing operations.

[0035] [Example] Illustrative examples of the techniques disclosed herein are provided below. Certain embodiments of these techniques may include any one or more, and any combination, of the examples described below.

[0036] Example 1 includes a memory having a media access circuit coupled to a memory medium, the media access circuit accessing data from the memory medium to perform a requested operation, simultaneously performing the requested operation on the accessed data using each of a plurality of computational logic units included in the media access circuit, and writing result data generated from the performance of the requested operation to the memory medium.

[0037] Example 2 includes the subject matter of example 1, wherein performing the requested operation with each of the plurality of computational logic units includes performing the requested operation with a plurality of computational logic units associated with different partitions of the memory medium.

[0038] Example 3 includes the subject matter of any of examples 1 and 2, wherein performing the requested operation with each of the plurality of computational logic units includes performing the requested operation with each of the plurality of computational logic units disposed within a die or package.

[0039] Example 4 includes the subject matter of any of examples 1 to 3, wherein performing the requested operation with each of the plurality of computational logic units includes performing the requested operation with each of a plurality of computational logic units disposed within a plurality of dies or packages of the dual in-line memory module.

[0040] Example 5 includes the subject matter of any of examples 1 to 4, wherein performing the requested operation with each of the plurality of compute logic units includes performing the requested operation with each of the plurality of compute logic units distributed across the plurality of dual in-line memory modules.

[0041] Example 6 includes the subject matter of any of examples 1-5, wherein accessing data from the memory medium includes reading data from multiple partitions of the memory medium.

[0042] Example 7 includes the subject matter of any of examples 1-6, wherein accessing the data from the memory medium includes reading the data into a scratch pad associated with each of the plurality of partitions.

[0043] Example 8 includes the subject matter of any of examples 1-7, wherein reading the data into the scratch pad includes reading the data into a register file or a static random access memory of the media access circuitry.

[0044] Example 9 includes the subject matter of any of examples 1-8, wherein accessing data from the memory medium includes accessing data from a memory medium having a three-dimensional cross-point architecture.

[0045] Example 10 includes the subject matter of any of examples 1-9, wherein performing the requested operations includes performing tensor operations.

[0046] Example 11 includes the subject matter of any of examples 1-10, wherein performing the requested operation includes performing a matrix multiplication operation.

[0047] Example 12 includes the subject matter of any of examples 1-11, wherein performing the requested operation includes performing a bitwise operation on the accessed data.

[0048] Example 13 includes the subject matter of any of examples 1-12, wherein the media access circuitry further provides data to the computing device component indicating completion of the requested operation.

[0049] Example 14 includes a method including: a media access circuit included in a memory accessing data from a memory medium coupled to the media access circuit to perform a requested operation; each of a plurality of computational logic units in the media access circuit simultaneously performing the requested operation on the accessed data; and the media access circuit writing result data generated from the performance of the requested operation to the memory medium.

[0050] Example 15 includes the subject matter of example 14, wherein each of the plurality of compute logic units performing the requested operation includes a plurality of compute logic units associated with different partitions of the memory medium performing the requested operation.

[0051] Example 16 includes the subject matter of any of examples 14 and 15, wherein each of the multiple computational logic units performing the requested operation includes each of the multiple computational logic units arranged within a die or package performing the requested operation.

[0052] Example 17 includes the subject matter of any of examples 14 to 16, wherein each of the multiple computational logic units performing the requested operation includes each of multiple computational logic units disposed within multiple dies or packages of the dual in-line memory module performing the requested operation.

[0053] Example 18 includes the subject matter of any of examples 14 to 17, wherein each of the plurality of compute logic units performing the requested operation includes each of a plurality of compute logic units distributed across a plurality of dual in-line memory modules performing the requested operation.

[0054] Example 19 includes the subject matter of any of examples 14-18, wherein accessing data from the memory medium includes reading data from multiple partitions of the memory medium.

[0055] Example 20 includes one or more machine-readable storage media having a plurality of instructions stored thereon that, in response to being executed, cause media access circuitry included in the memory to access data from a memory medium to perform a requested operation, cause each of a plurality of computational logic units included in the media access circuitry to simultaneously perform the requested operation on the accessed data, and write result data generated from the execution of the requested operation to the memory medium. [Other possible items] (Item 1) 1. A memory comprising a medium access circuit coupled to a memory medium, The medium access circuit includes: accessing data from said memory medium to perform the requested operation; simultaneously performing the requested operations on the accessed data using each of a plurality of computational logic units included in the media access circuitry; and writing result data generated from performance of said requested operations to said memory medium; Memory. (Item 2) 2. The memory of claim 1, wherein performing the requested operation using each of a plurality of computational logic units includes performing the requested operation using a plurality of computational logic units associated with different partitions of the memory medium. (Item 3) 2. The memory of claim 1, wherein performing the requested operation with each of a plurality of computational logic units includes performing the requested operation with each of a plurality of computational logic units disposed within a die or package. (Item 4) 2. The memory of claim 1, wherein performing the requested operation with each of a plurality of computational logic units includes performing the requested operation with each of a plurality of computational logic units disposed within a plurality of dies or packages of a dual in-line memory module. (Item 5) 2. The memory of claim 1, wherein performing the requested operation with each of a plurality of computational logic units includes performing the requested operation with each of a plurality of computational logic units distributed across a plurality of dual in-line memory modules. (Item 6) 2. The memory of claim 1, wherein accessing data from the memory medium includes reading the data from multiple partitions of the memory medium. (Item 7) 7. The memory of claim 6, wherein accessing data from the memory medium includes reading the data into a scratch pad associated with each of the plurality of partitions. (Item 8) 8. The memory of claim 7, wherein reading the data into a scratch pad includes reading the data into a register file or a static random access memory of the media access circuitry. (Item 9) 2. The memory of claim 1, wherein accessing data from the memory medium includes accessing data from a memory medium having a three-dimensional cross-point architecture. (Item 10) 2. The memory of claim 1, wherein performing the requested operation includes performing a tensor operation. (Item 11) 2. The memory of claim 1, wherein performing the requested operation includes performing a matrix multiplication operation. (Item 12) 2. The memory of claim 1, wherein performing the requested operation includes performing a bitwise operation on the accessed data. (Item 13) 2. The memory of claim 1, wherein the media access circuitry further provides data to the computing device component indicating completion of the requested operation. (Item 14) accessing data from a memory medium coupled to said media access circuitry included in the memory to perform the requested operation; each of a plurality of compute logic units in said media access circuitry simultaneously performing said requested operation on said accessed data; said medium access circuitry writing result data generated from said performance of said requested operation to said memory medium; The method includes: (Item 15) 15. The method of claim 14, wherein the step of each of a plurality of computational logic units performing the requested operation comprises a step of a plurality of computational logic units associated with different partitions of the memory medium performing the requested operation. (Item 16) Item 15. The method of item 14, wherein the step of each of a plurality of computational logic units performing the requested operation comprises the step of each of a plurality of computational logic units arranged in a die or package performing the requested operation. (Item 17) Item 15. The method of item 14, wherein the step of each of a plurality of computational logic units performing the requested operation includes the step of each of a plurality of computational logic units disposed within a plurality of dies or packages of a dual in-line memory module performing the requested operation. (Item 18) 15. The method of claim 14, wherein the step of each of a plurality of computational logic units performing the requested operation comprises the step of each of a plurality of computational logic units distributed across a plurality of dual in-line memory modules performing the requested operation. (Item 19) 15. The method of claim 14, wherein accessing data from the memory medium comprises reading the data from multiple partitions of the memory medium. (Item 20) One or more machine-readable storage media containing a plurality of instructions stored thereon, the instructions being executed to cause medium access circuitry included in the memory to: accessing data from said memory medium to perform the requested operation; causing each of a plurality of compute logic units included in said media access circuitry to simultaneously perform said requested operations on said accessed data; and causing result data produced from performance of said requested operations to be written to said memory medium; A machine-readable storage medium.

Claims

1. 1. A memory comprising a medium access circuit coupled to a memory medium, the medium access circuitry includes a plurality of computational logic units associated with different partitions of the memory medium; accessing, by one of the plurality of computational logic units, data from a corresponding partition of the memory medium to perform the requested operation; broadcasting the data accessed by the one of the plurality of compute logic units to other compute logic units of the plurality of compute logic units; performing different portions of the requested operation in parallel on the accessed data using each of the plurality of compute logic units included in the media access circuitry; and writing to the memory medium respective result data generated from execution of the different portions of the requested operation performed by each of the plurality of computational logic units. Memory.

2. the medium access circuitry is connected to the plurality of computational logic units, and includes a plurality of scratch pads, each of which corresponds to one of the plurality of partitions; the one of the plurality of compute logic units broadcasting the accessed data to the other compute logic units by broadcasting the accessed data to a scratch pad of a plurality of scratch pads connected to the other compute logic units of the plurality of compute logic units; said plurality of compute logic units writing respective result data generated from execution of said different portions of said requested operations to their corresponding scratch pads; The memory of claim 1 .

3. 3. The memory of claim 1 or 2, wherein performing the requested operation with each of a plurality of compute logic units comprises performing the requested operation with each of a plurality of compute logic units arranged in a die or package.

4. 4. The memory of claim 1, wherein performing the requested operation using each of a plurality of computational logic units includes performing the requested operation using each of a plurality of computational logic units disposed within a plurality of dies or packages of a dual in-line memory module.

5. 5. The memory of claim 1, wherein performing the requested operation with each of a plurality of compute logic units comprises performing the requested operation with each of a plurality of compute logic units distributed across a plurality of dual in-line memory modules.

6. The memory of claim 1 , wherein accessing data from the memory medium comprises reading the data from multiple partitions of the memory medium.

7. 7. The memory of claim 6, wherein accessing data from the memory medium comprises reading the data into a scratch pad associated with each of the plurality of partitions.

8. 8. The memory of claim 7, wherein reading the data into a scratch pad comprises reading the data into a register file or a static random access memory of the media access circuitry.

9. The memory of claim 1 , wherein accessing data from the memory medium comprises accessing data from a memory medium having a three-dimensional cross-point architecture.

10. The memory of claim 1 , wherein performing the requested operation comprises performing a tensor operation.

11. The memory of claim 1 , wherein performing the requested operation comprises performing a matrix multiplication operation.

12. The accessed data includes matrix data in an input matrix and matrix data in a weight matrix; 12. The memory of claim 10 or 11, wherein the result data comprises matrix data in an output matrix obtained by multiplying the input matrix by the weighting matrix.

13. The memory of claim 1 , wherein performing the requested operation comprises performing a bitwise operation on the accessed data.

14. 14. A memory as claimed in any preceding claim, wherein the medium access circuitry is further adapted to provide data indicating completion of the requested operation.

15. A memory, a media access circuit included in the memory having means for accessing data from a corresponding partition of the memory medium coupled to the media access circuit in order to perform a requested operation by one of a plurality of computational logic units associated with different partitions of the memory medium; means for broadcasting the data accessed by the one of the plurality of computing logic units to other computing logic units of the plurality of computing logic units; means for each of said plurality of compute logic units in said medium access circuitry to perform in parallel different portions of said requested operation on said accessed data; means for said medium access circuitry to write to said memory medium respective result data generated from execution of said different portions of said requested operation performed by each of said plurality of computational logic units; A memory comprising:

16. The means for each of a plurality of computational logic units to perform the requested operation comprises: means for a plurality of computational logic units associated with different partitions of said memory medium to perform said requested operations; 20. The memory of claim 15, comprising:

17. The means for each of a plurality of computational logic units to perform the requested operation comprises: Means for performing the requested operations on each of a plurality of computational logic units disposed within the die or package.

17. The memory of claim 15 or 16, comprising:

18. The means for each of a plurality of computational logic units to perform the requested operation comprises: means for each of a plurality of computational logic units disposed within a plurality of dies or packages of the dual in-line memory module to perform the requested operation; 18. The memory of claim 15, further comprising:

19. The means for each of a plurality of computational logic units to perform the requested operation comprises: means for each of a plurality of computational logic units distributed across a plurality of dual in-line memory modules to perform the requested operation; 19. The memory of claim 15, further comprising:

20. The means for accessing data from the memory medium comprises: means for reading the data from a plurality of partitions of the memory medium; 20. The memory of claim 15, further comprising:

21. The means for accessing data from the memory medium comprises: means for reading said data into a scratch pad associated with each of said plurality of partitions; 21. The memory of claim 20, comprising:

22. said means for reading said data into a scratch pad comprising: means for reading said data into a register file or a static random access memory of said media access circuitry; 22. The memory of claim 21, comprising:

23. The means for accessing data from the memory medium comprises: MEANS FOR ACCESSING DATA FROM A MEMORY MEDIUM HAVING A THREE-DIMENSIONAL CROSSPOINT ARCHITECTURE - Patent application 23. The memory of claim 15, further comprising:

24. The means for performing the requested operation comprises: A means for performing tensor operations 24. The memory of claim 15, further comprising:

25. A medium access circuit included in the memory medium includes: accessing, by one of a plurality of computational logic units associated with different partitions of the memory medium, data from a corresponding partition of the memory medium to perform a requested operation, the computational logic units being included in the media access circuitry; broadcasting, by the one of the plurality of compute logic units, the data accessed to other compute logic units of the plurality of compute logic units; performing different portions of the requested operation in parallel on the accessed data using each of the plurality of compute logic units included in the media access circuitry; writing to the memory medium respective result data generated from execution of the different portions of the requested operation performed by each of the plurality of computational logic units; A program for executing the above.

26. A machine-readable storage medium storing the program according to claim 25.

Citation Information

Patent Citations

  • Inter-system communication control system

    JP2001043198A

  • Method and apparatus for performing atomic semaphore operation

    JP2011048834A

  • Programmable processing unit providing concurrent datapath operation of multiple instructions

    US20070192571A1

  • Computer BIOS protection and authentication

    US20120208619A1

  • 3DIC memory chips including computational logic-in-memory for performing accelerated data processing

    US20150199266A1