Stacked Die for Machine Learning Accelerators

Abstract

A device is disclosed that includes a machine learning die including a memory and one or more machine learning accelerators, and a processing core die stacked with the machine learning die, the processing core die configured to execute shader programs to control operations in the machine learning die, where the memory is configurable as either or both of a cache and a directly accessible memory.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Application No. 63 / 031,954, filed May 29, 2020, entitled "STACKED DIES FOR MACHINE LEARNING ACCELERATOR," and U.S. Patent Application No. 17 / 129,739, filed December 21, 2020, entitled "STACKED DIES FOR MACHINE LEARNING ACCELERATOR," which are incorporated by reference as if fully set forth herein. [Background technology]

[0002] Improvements in processor speed have generally not been matched by improvements in memory bandwidth speed, and a wide variety of techniques have been used to compensate for this fact.

[0003] A more detailed understanding may be had from the following description, given by way of example in conjunction with the accompanying drawings, in which: [Brief explanation of the drawings]

[0004] [Figure 1] FIG. 1 is a block diagram of an example device in which one or more features of the present disclosure may be implemented. [Figure 2A] 2 illustrates details of the acceleration processing device of FIG. 1, according to an example. [Figure 2B] FIG. 1 illustrates details of a device and accelerator processing device core die, according to an example. [Figure 3] FIG. 2C is a block diagram illustrating additional details of the graphics processing pipeline shown in FIG. 2B, according to an example. [Figure 4] FIG. 1 illustrates a diagram of a memory and machine learning accelerator die, according to an example. [Figure 5] FIG. 1 illustrates reconfiguration of memory between memory configured as a cache and memory configured as directly accessible, according to one example. [Figure 6] FIG. 1 illustrates an example layout of a memory and machine learning accelerator die. [Figure 7] FIG. 1 is a flow diagram of a method for performing machine learning operations using a stacked machine learning and accelerator processing device, according to an example. DETAILED DESCRIPTION OF THE INVENTION

[0005] A device is disclosed that includes a machine learning die including a memory and one or more machine learning accelerators, and a processing core die stacked with the machine learning die, the processing core die configured to execute shader programs to control operations in the machine learning die, where the memory is configurable as either or both of a cache and a directly accessible memory.

[0006] 1 is a block diagram of an example device 100 capable of implementing one or more features of the present disclosure. Device 100 may be, for example, but not limited to, a computer, a gaming device, a handheld device, a set-top box, a television, a mobile phone, a tablet computer, or any other computing device. Device 100 includes a processor 102, a memory 104, a storage device 106, one or more input devices 108, and one or more output devices 110. Device 100 also includes one or more input drivers 112 and one or more output drivers 114. Any of the input drivers 112 may be embodied as hardware, a combination of hardware and software, or software, and are responsible for controlling (e.g., controlling the operation of, receiving input from, and providing data to) input driver 112. Similarly, any of the output drivers 114 may be embodied as hardware, a combination of hardware and software, or software and serve to control (e.g., control the operation of, receive input from, and provide data to) the output device 110. It should be understood that device 100 may include additional components not shown in FIG.

[0007] In various alternatives, processor 102 may include a central processing unit (CPU), a graphics processing unit (GPU), a CPU and a GPU located on the same die, or one or more processor cores, each of which may be a CPU or a GPU. In various alternatives, memory 104 may be located on the same die as processor 102 or may be located separately from processor 102. Memory 104 may include volatile or non-volatile memory (e.g., random access memory (RAM), dynamic RAM, cache).

[0008] Storage devices 106 include fixed or removable storage devices (e.g., but not limited to, hard disk drives, solid state drives, optical disks, flash drives). Input devices 108 include, but are not limited to, keyboards, keypads, touchscreens, touchpads, detectors, microphones, accelerometers, gyroscopes, biometric scanners, or network connections (e.g., wireless local area network cards for transmitting and / or receiving wireless IEEE 802 signals). Output devices 110 include, but are not limited to, displays, speakers, printers, haptic feedback devices, one or more optics, antennas, or network connections (e.g., wireless local area network cards for transmitting and / or receiving wireless IEEE 802 signals).

[0009] The input driver 112 and the output driver 114 include one or more hardware, software, and / or firmware components configured to interface with and drive the input device 108 and the output device 110, respectively. The input driver 112 communicates with the processor 102 and the input device 108, enabling the processor 102 to receive input from the input device 108. The output driver 114 communicates with the processor 102 and the output device 110, enabling the processor 102 to send output to the output device 110. The output driver 114 includes an accelerated processing device (APD) 116. In some implementations, the APD 116 is coupled to a display device 118, which in some examples is a physical display device or a simulated device that shows output using a remote display protocol. The APD 116 is configured to perform machine learning-related tasks. In some implementations, the APD 116 is configured to accept one or both of general-purpose computational commands and graphics rendering commands from the processor 102, process the computational and graphics rendering commands, and in some implementations, provide pixel output to the display device 118 for display.

[0010] In some implementations, the APD 116 includes one or more parallel processing units configured to perform calculations according to the single-instruction-multiple-data (SIMD) paradigm. Although various functions are described herein as being performed by or in conjunction with the APD 116, in various alternatives, functions described as being performed by the APD 116 are not driven by a host processor (e.g., the processor 102) but are additionally or alternatively performed by other computing devices having similar capabilities that are configured to provide graphics output to the display device 118. For example, it is contemplated that any processing system that performs processing tasks according to the SIMD paradigm may be configured to perform the functions described herein. Alternatively, it is contemplated that computing systems that do not perform processing tasks according to the SIMD paradigm perform the functions described herein.

[0011] FIG. 2A illustrates details of the APD 116, according to an example. The APD 116 has a stacked die configuration including the APD core die 115 and the memory and machine learning accelerator die 260. These dies are physically stacked, one on top of the other. The dies are operatively coupled via interconnects that enable the transfer of data and commands. The memory and machine learning accelerator die 260 includes memory, such as static random access memory, and a machine learning accelerator, such as a matrix multiplication arithmetic logic unit (ALU) configured to perform matrix multiplication operations that may be useful for machine learning operations. The memory of the memory and machine learning accelerator die 260 is used as a cache by the APD core die 115 in some implementations or modes of operation.

[0012] 2B illustrates details of device 100 and APD core die 115, according to an example. Processor 102 (FIG. 1) executes operating system 120, drivers 122, and applications 126, and may alternatively or additionally execute other software. Operating system 120 controls various aspects of device 100, such as managing hardware resources, handling service requests, scheduling and controlling process execution, and performing other operations. APD driver 122 controls the operation of APD core die 115 and sends tasks, such as graphics rendering tasks or other work, to APD core die 115 for processing. Driver 122 also includes a just-in-time compiler that compiles programs for execution by processing components of APD core die 115 (such as SIMD unit 138, discussed in more detail below).

[0013] The APD core die 115 executes commands and programs for selected functions, such as graphics and non-graphics operations, that may be suitable for parallel processing. The APD core die 115 may be used to perform graphics pipeline operations, such as pixel operations, geometry calculations, and rendering of images to a display device 118, based on commands received from the processor 102. The APD core die 115 also performs computational processing operations not directly related to graphics operations, such as operations related to video, physics simulation, computational fluid dynamics, or other tasks, based on commands received from the processor 102. In some examples, these computational processing operations are performed by executing computational shaders on the SIMD units 138.

[0014] The APD core die 115 includes a compute unit 132 that includes one or more SIMD units 138 configured to perform operations in a parallel manner according to the SIMD paradigm at the request of the processor 102 (or another unit). The SIMD paradigm allows multiple processing elements to share a single program control flow unit and program counter, thus executing the same program but with different data. In one example, each SIMD unit 138 includes 16 lanes, each of which executes the same instruction simultaneously with other lanes in the SIMD unit 138 but can execute the instruction on different data. Lanes can be predictively switched off if not all lanes need to execute a given instruction. Prediction can also be used to execute programs with branching control flow. More specifically, for programs with conditional branches or other instructions where control flow is based on calculations performed by individual lanes, prediction of lanes corresponding to currently unexecuted control flow paths and serial execution of different control flow paths enables arbitrary control flow.

[0015] The basic unit of execution within the compute unit 132 is the work item. Each work item represents a single instantiation of a program executing in parallel on a particular lane. Work items can execute simultaneously (or partially concurrently and partially sequentially) as a “wavefront” on a single SIMD unit 138. One or more wavefronts are included in a “workgroup,” which contains a collection of work items designated to execute the same program. A workgroup can be executed by executing each of the wavefronts that make up the workgroup. In alternative examples, a wavefront executes on a single SIMD unit 138 or on different SIMD units 138. A wavefront can be thought of as the largest collection of work items that can execute simultaneously (or pseudo-simultaneously) on a single SIMD unit 138. “Pseudo-simultaneous” execution occurs when there are more wavefronts than the number of lanes in the SIMD unit 138. In such a situation, the wavefront executes over multiple cycles, with different collections of work items executing in different cycles. APD scheduler 136 is configured to perform operations related to scheduling various workgroups and wavefronts on compute units 132 and SIMD units 138 .

[0016] The parallel processing provided by the compute units 132 is well suited to graphics-related operations such as pixel value calculations, vertex transformations, and other graphics operations. Thus, in some cases, the graphics processing pipeline 134, which accepts graphics processing commands from the processor 102, provides computational tasks to the compute units 132 for execution in parallel.

[0017] Computation unit 132 is also used to perform computational tasks that are not related to graphics or that are not performed as part of the "normal" operation of graphics processing pipeline 134 (e.g., custom operations performed to supplement the operations performed on graphics processing pipeline 134). Applications 126 or other software executing on processor 102 send programs defining such computational tasks to APD 116 for execution.

[0018] Shader programs, such as compute shader programs, executing on the APD core die 115 can request that the memory and machine learning accelerator die 260 perform machine learning related operations. Examples of such operations include operations to move data into or out of the memory of the memory and ML accelerator die 260, or to perform ALU operations, such as matrix multiplication, using data in the memory of the memory and ML accelerator die 260 as operands.

[0019] It should be noted that while the APD core die 115 of FIG. 2B includes a graphics processing pipeline 134 and performs graphics operations, it is understood that the teachings of the present disclosure may be used in conjunction with an APD core die 115 that does not include a graphics processing pipeline 134, but that includes a compute unit 132 and is therefore capable of executing shader programs.

[0020] 3 is a block diagram illustrating additional details of the graphics processing pipeline 134 shown in FIG. 2B, according to one example. The graphics processing pipeline 134 includes stages, each of which performs a specific function of the graphics processing pipeline 134. Each stage is implemented partially or fully as a shader program executing within the programmable compute unit 132, or partially or fully as fixed-function, non-programmable hardware external to the compute unit 132.

[0021] The input assembler stage 302 reads user-filled buffers (e.g., buffers filled with requests from software executed by the processor 102, such as applications 126) and assembles the data into primitives for use by the rest of the pipeline. The input assembler stage 302 can generate different types of primitives based on the primitive data contained in the user-filled buffers. The input assembler stage 302 formats the assembled primitives for use by the rest of the pipeline.

[0022] The vertex shader stage 304 processes the vertices of the primitives assembled by the input assembler stage 302. The vertex shader stage 304 performs various per-vertex operations such as transformation, skinning, morphing, and per-vertex lighting. Transformation operations include various operations for transforming the coordinates of vertices. These operations include one or more of modeling transformations that modify vertex coordinates, display transformations, projection transformations, perspective divisions, viewport transformations, and other operations that modify non-coordinate attributes.

[0023] The vertex shader stage 304 is implemented partially or completely as a vertex shader program that runs on one or more compute units 132. The vertex shader program is provided by the processor 102 and is based on a program pre-written by a computer programmer. The driver 122 compiles such a computer program to generate a vertex shader program having a form suitable for execution within the compute units 132.

[0024] The hull shader stage 306, tessellator stage 308, and domain shader stage 310 work together to implement tessellation, which converts simple primitives into more complex primitives by subdividing the primitives. The hull shader stage 306 generates patches for tessellation based on the input primitives. The tessellator stage 308 generates a sample set for the patch. The domain shader stage 310 calculates vertex positions for vertices corresponding to the samples in the patch. The hull shader stage 306 and domain shader stage 310, like the vertex shader stage 304, can be implemented as shader programs compiled by the driver 122 and executed on the compute unit 132.

[0025] The geometry shader stage 312 performs vertex operations on a primitive basis. A variety of different types of operations can be performed by the geometry shader stage 312, including operations such as point sprite expansion, dynamic particle system operations, fur-fin generation, shadow volume generation, single pass render-to-cubemap, per-primitive material swapping, and per-primitive material setup. In some cases, a geometry shader program compiled by the driver 122 and executed on the compute unit 132 performs the operations of the geometry shader stage 312.

[0026] The rasterizer stage 314 accepts and rasterizes simple primitives (triangles) generated upstream from the rasterizer stage 314. Rasterization consists of determining which screen pixels (or subpixel samples) are covered by a particular primitive. Rasterization is performed by fixed-function hardware.

[0027] The pixel shader stage 316 calculates the output values ​​of the screen pixels based on the primitives generated upstream and the results of rasterization. The pixel shader stage 316 can apply textures from texture memory. The operations of the pixel shader stage 316 are performed by pixel shader programs compiled by the driver 122 and executed on the compute unit 132.

[0028] The output merge stage 318 accepts the outputs from the pixel shader stage 316 and merges them with a target surface, performing operations such as z-testing and alpha blending to determine the final color of the screen pixel. The target surface is the final target for a frame of rendering operations in the graphics processing pipeline 134. The target surface can be anywhere in memory (such as in the memory of the APD 116 or in memory 104).

[0029] 4 is a diagram of a memory and machine learning accelerator die 260, according to an example. The memory and machine learning accelerator die 260 includes a memory 402, a machine learning accelerator 404, a memory interconnect 406, an inter-die interconnect 408, and a controller 410.

[0030] In some examples, memory 402 is a cache memory (such as a level 3 cache memory) that functions as a cache for APD core die 115 (such as compute unit 132). However, in certain configurations, at least a portion of memory 402 is used directly for operations performed by machine learning accelerator 404. Machine learning accelerator 404 includes a hardware accelerator for performing operations associated with machine learning. Examples of such operations include matrix multiplication operations.

[0031] In some operational modes, the APD core die 115 executes shader programs associated with machine learning. The shader programs include instructions for configuring the memory 402 for direct use by the machine learning accelerator 404. The shader programs also include instructions for storing data in such memory. The shader programs also include instructions for causing the machine learning accelerator 404 to perform machine learning operations on the data stored in the memory. When the APD core die 115 encounters such an instruction, the APD core die 115 instructs the memory 402 to load the data and instructs the machine learning accelerator 404 to perform the requested operation.

[0032] The memory interconnect 406 couples different portions of the memory 402 together to the machine learning accelerator 404 so that machine learning operations can be performed by the machine learning accelerator 404 on data stored throughout the memory 402. The inter-die interconnect 408 communicates data and commands between the memory and the machine learning accelerator die 260, the APD core die 115, and the processor 102. In some examples, the inter-die interconnect 408 is directly coupled to the compute units 132 of the APD core die 115. The controller 410 controls operations in the memory and the machine learning accelerator die 260, such as data transfer and machine learning operations in the machine learning accelerator 404.

[0033] 5 illustrates reconfiguration of memory 402 between memory configured as cache 502 and memory configured as directly accessible 504, according to one example. At the request of a shader program or other entity, memory controller 410 changes the amount of memory 402 used as cache 502 and the amount of memory 504 configured as directly accessible. The amount of memory 402 used as cache 502 and the amount of memory 504 configured as directly accessible are reconfigurable. In some configurations, cache memory 502 is used for graphics operations (e.g., on graphics processing pipeline 134), and directly accessible memory 504 is used for machine learning operations.

[0034] 6 illustrates an example layout 600 of a memory and machine learning accelerator die 260. The large rectangles shown represent portions of memory 602. Several machine learning arithmetic logic units (ALUs) 604 are shown. A controller (Cnt) 606 is also shown adjacent to the machine learning ALUs 604. Several inter-die interconnects 608 coupled to the controllers are shown. Memory interconnects 610 couple the controllers 606 together.

[0035] The memory portion 602 is a portion of the memory of the machine learning accelerator die 260. These items represent the portion of the total memory of the die 260 that is local to, and therefore directly controlled by, a particular controller 606 and ML ALU 604. The inter-die interconnects 608 provide external connections from the controller 606. In various implementations, these inter-die interconnects 608 couple to either or both of the APD core die 115 and the processor 102. The controller 606 communicates with the memory, the ML ALU 604, and either or both of the APD core die 115 and the processor 102 via the inter-die interconnects 608.

[0036] The memory interconnect 610 is coupled to the multiple controllers 606 and provides a communication link between the controllers 606. This link allows the controllers 606 to read from or write to memory portions 602 other than the memory portion local to the controller 606. In one example, the ML ALU 604 can perform operations on data from different memory portions 602 by utilizing the memory interconnect 610.

[0037] Although various numbers of elements are shown in FIG. 6, it should be understood that a wide variety of implementations having different numbers of illustrated elements are contemplated.

[0038] Although this disclosure describes the memory and machine learning accelerator die being coupled to the APD die, in alternative implementations, the APD die is instead a microprocessor die.

[0039] 7 is a flow diagram of a method 700 for performing operations on an APD 116, according to an example. Although described with respect to the systems of FIGS. 1-6, one skilled in the art will understand that any system configured to perform the steps of method 700 in any technically feasible order is within the scope of the present disclosure.

[0040] The method 700 begins at step 702, where the APD core die 115 executes a shader. The shader includes instructions for performing machine learning operations using one or more ML ALUs 604.

[0041] In step 704, according to the shader instructions, the APD core die 115 instructs the set of machine learning ALUs 604 to perform the set of machine learning tasks via one or more die interconnects 608.

[0042] In step 706, the machine learning ALUs 604 perform a set of machine learning tasks. In various examples, the controller 606 receives a communication from the APD core die 115 that includes information indicating which action to perform, and the controller 606 instructs one or more ML ALUs 604 to perform the action. In some examples, the controller 606 communicates with another controller 606 via the memory interconnect 610 to coordinate the action to be performed.

[0043] Each of the illustrated functional units represents hardware circuitry configured to perform the operations described herein, software configured to perform the operations described herein, or a combination of software and hardware configured to perform the steps described herein. A non-exclusive list of such units includes storage 106, processor 102, output driver 114, APD 116, memory 104, input driver 112, input device 108, output device 110, display device 118, operating system 120, driver 122, application 126, APD scheduler 136, graphics processing pipeline 134, compute unit 132, SIMD unit 138, and any of the stages of graphics processing pipeline 134.

[0044] It should be understood that many variations are possible based on the disclosure herein, and although features and elements are described above in particular combinations, each feature or element can be used alone without other features and elements, or in various combinations with or without other features and elements.

[0045] The provided methods can be implemented in a general-purpose computer, processor, or processor core. Suitable processors include, by way of example, a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), and / or a state machine. Such processors can be fabricated by configuring a manufacturing process using the results of processed hardware description language (HDL) instructions and other intermediate data, such as a netlist (such instructions can be stored on a computer-readable medium). The result of such processing can be a mask work, which is then used in a semiconductor manufacturing process to produce a processor implementing features of the present disclosure.

[0046] The methods or flow diagrams provided herein may be implemented in a computer program, software, or firmware embodied in a non-transitory computer-readable storage medium for execution by a general-purpose computer or processor. Examples of non-transitory computer-readable storage media include read-only memory (ROM), random-access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media (e.g., internal hard disks and removable disks), magneto-optical media, and optical media (e.g., CD-ROM disks and digital versatile disks (DVDs)).

Claims

1. A device, a machine learning die including a memory and one or more machine learning accelerators; an accelerated processing device (APD) core die stacked with the machine learning die, the APD core die configured to execute a shader program to control machine learning-related operations in the machine learning die; The memory is configurable as either or both of a cache and a directly accessible memory, the cache being configured to perform cache operations for the APD core die, and the directly accessible memory being configured to perform memory operations for the machine learning die. device.

2. the machine learning accelerator is configured to perform matrix multiplication using the data in the memory. The device of claim 1.

3. the machine learning die and the APD core die are coupled via one or more die-to-die interconnects; The device of claim 1.

4. the APD core die is configured to redirect a portion of the memory from use for the one or more machine learning accelerators to use as a cache for the APD core die; The device of claim 1.

5. the APD core die is configured to redirect a portion of the memory from use as a cache for the APD core die to use for the one or more machine learning accelerators; The device of claim 1.

6. the APD core die is configured to execute shader instructions to store data in the memory; The device of claim 1.

7. the machine learning die further includes one or more controllers that execute instructions from the APD core die to control operation of the memory and the machine learning accelerator. The device of claim 1.

8. the machine learning die further includes a memory interconnect coupling the one or more controllers. The device of claim 7.

9. the memory interconnect is configured to provide data from one portion of the memory to a controller local to a different portion of the memory. The device of claim 8.

10. 1. A method comprising: Executing a shader program on an accelerated processing device (APD) core die to control machine learning related operations; directing, via one or more inter-die interconnects, a set of machine learning (ML) arithmetic logic units (ALUs) of a machine learning (ML) accelerator die to perform a set of machine learning tasks according to instructions of the shader program, the ML accelerator die including memory, the ML accelerator die being stacked with the APD core die; executing a set of machine learning tasks on an ML ALU; and configuring at least a portion of the memory as a cache during a first period and as a directly accessible memory for the ML ALU during a second period, the cache being configured to perform cache operations for the APD core die and the directly accessible memory being configured to perform memory operations for the ML accelerator die. method.

11. The method of claim 10, further comprising: performing a matrix multiplication in said ML ALU using data in said memory. The method of claim 10.

12. the ML accelerator die and the APD core die are coupled via one or more die-to-die interconnects; The method of claim 10.

13. the APD core die changing a portion of the memory from being used for one or more of the ML ALUs to being used as a cache for the APD core die; The method of claim 10.

14. the APD core die changing a portion of the memory from being used as a cache for the APD core die to being used for one or more of the ML ALUs; The method of claim 10.

15. executing, at the APD core die, shader instructions to store data in the memory; The method of claim 10.

16. the ML accelerator die further includes one or more controllers that control operation of the memory and the ML ALU; The method of claim 10.

17. the ML accelerator die further includes a memory interconnect coupling the one or more controllers; 17. The method of claim 16.

18. The method of claim 17, further comprising providing data from one portion of the memory to a controller local to a different portion of the memory.

18. The method of claim 17.

19. A system comprising: a processor; an accelerated processing device (APD), The APD is a machine learning die including a memory and one or more machine learning accelerators; an APD core die stacked with the machine learning die, the APD core die configured to execute shader programs for controlling machine learning-related operations in the machine learning die, one or more of the shader programs being specified by the processor; The memory is configurable as either or both of a cache and a directly accessible memory, the cache being configured to perform cache operations for the APD core die, and the directly accessible memory being configured to perform memory operations for the machine learning die. system.

20. the machine learning accelerator is configured to perform matrix multiplication using the data in the memory.

20. The system of claim 19.

Images