Stacked dies for machine learning accelerator

Abstract

An apparatus is disclosed. The apparatus includes a machine learning die including a memory and one or more machine learning accelerators; and a processing core die stacked with the machine learning die, the processing core die configured to execute a shader program to control operations on the machine learning die, wherein the memory is configurable as either or both of a cache and directly accessible memory.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefits of U.S. Provisional Application No. 63 / 031,954, filed May 29, 2020, entitled “STACKED DIES FOR MACHINE LEARNING ACCELERATOR,” and U.S. Non-Provisional Application No. 17 / 129,739, filed December 21, 2020, entitled “STACKED DIES FOR MACHINE LEARNING ACCELERATOR,” which are incorporated herein by reference as fully set forth herein. Background Technology

[0003] Increased processor speeds often cannot keep pace with increases in memory bandwidth. Various techniques have been used to compensate for this fact. Attached Figure Description

[0004] A more detailed understanding can be obtained from the following description, provided with examples and accompanying figures, wherein:

[0005] Figure 1 It is a block diagram of an example device that can implement one or more features of this disclosure;

[0006] Figure 2A An example is shown. Figure 1 Details of the accelerated processing equipment;

[0007] Figure 2B Details of the core die of an accelerator processing device, based on an example, are shown.

[0008] Figure 3 This is based on an example. Figure 2B A block diagram showing additional details of the graphics processing pipeline;

[0009] Figure 4 This is a schematic diagram based on an example memory and machine learning accelerator die;

[0010] Figure 5 The diagram illustrates a memory reconfiguration between memory configured as a cache and memory configured for direct access, according to one example.

[0011] Figure 6 An exemplary layout of a memory and machine learning accelerator die is shown; and

[0012] Figure 7 This is a flowchart illustrating an example of a method for performing machine learning operations using stacked machine learning and accelerator processing devices. DETAILED DESCRIPTION

[0013] An apparatus is disclosed. The apparatus includes a machine learning die including a memory and one or more machine learning accelerators; and a processing core die stacked with the machine learning die, the processing core die configured to execute a shader program to control operations on the machine learning die, wherein the memory is configurable as either or both of a cache and directly accessible memory.

[0014] Figure 1 is a block diagram of an example device 100 in which one or more features of the present disclosure can be implemented. The device 100 can be, for example, one of a computer, a gaming device, a handheld device, a set-top box, a television, a mobile phone, a tablet computer, or other computing device, but is not limited to these. The device 100 includes a processor 102, a memory 104, a storage 106, one or more input devices 108, and one or more output devices 110. The device 100 also includes one or more input drivers 112 and one or more output drivers 114. Any of the input drivers 112 is embodied as hardware, a combination of hardware and software, or software, and serves the purpose of controlling the input devices 112 (e.g., controlling operation, receiving input from, and providing data to the input drivers 112). Similarly, any of the output drivers 114 is embodied as hardware, a combination of hardware and software, or software, and serves the purpose of controlling the output devices 114 (e.g., controlling operation, receiving input from, and providing data to the output drivers 114). It is understood that the device 100 can include additional components not shown in FIG. 1. Figure 1

[0015] In various alternatives, the processor 102 includes a central processing unit (CPU), a graphics processing unit (GPU), a CPU and GPU located on the same die, or one or more processor cores, where each processor core can be a CPU or GPU. In various alternatives, the memory 104 is located on the same die as the processor 102, or is located separately from the processor 102. The memory 104 includes volatile or non-volatile memory, such as random access memory (RAM), dynamic RAM, or a cache.

[0016] ​Storage device 106 includes fixed or removable storage devices, such as, but not limited to, hard disk drives, solid-state drives, optical disk drives, or flash drives. Input device 108 includes, but is not limited to, keyboards, keypads, touchscreens, touchpads, detectors, microphones, accelerometers, gyroscopes, biometric scanners, or network connections (e.g., wireless LAN cards for transmitting and / or receiving wireless IEEE 802 signals). Output device 110 includes, but is not limited to, displays, speakers, printers, haptic feedback devices, one or more lights, antennas, or network connections (e.g., wireless LAN cards for transmitting and / or receiving wireless IEEE 802 signals).

[0017] Input driver 112 and output driver 114 include one or more hardware, software, and / or firmware components configured to interface with and drive input device 108 and output device 110, respectively. Input driver 112 communicates with processor 102 and input device 108 and allows processor 102 to receive input from input device 108. Output driver 114 communicates with processor 102 and output device 110 and allows processor 102 to send output to output device 110. Output driver 114 includes an accelerated processing device (“APD”) 116. In some implementations, APD 116 is coupled to display device 118, which in some examples is a physical display device or analog device that uses a remote display protocol to display output. APD 116 is configured to perform machine learning-related tasks. In some implementations, APD 116 is configured to accept one or both of general computing commands and graphics rendering commands from processor 102, process those computing commands and graphics rendering commands, and in some implementations, provide pixel output to display device 118 for display.

[0018] In some specific implementations, APD 116 includes one or more parallel processing units configured to perform computations according to the Single Instruction Multiple Data (“SIMD”) paradigm. Although various functions are described herein as being performed by or in conjunction with APD 116, in various alternatives, the functions described as being performed by APD 116 may additionally or alternatively be performed by other computing devices with similar capabilities, which are not driven by a host processor (e.g., processor 102) and are configured to provide graphics output to display device 118. For example, it is conceivable that any processing system performing processing tasks according to the SIMD paradigm could be configured to perform the functions described herein. Alternatively, it is conceivable that a computing system not performing processing tasks according to the SIMD paradigm could perform the functions described herein.

[0019] Figure 2ADetails of an example APD 116 are shown. The APD 116 has a stacked die configuration, including an APD core die 115 and memory and machine learning accelerator dies 260. These dies are physically stacked, with one die situated on top of another. The dies are operatively coupled via interconnects that allow the transfer of data and commands. The memory and machine learning accelerator die 260 includes memory, such as static random access memory, and machine learning accelerators, such as a matrix multiplication arithmetic logic unit (“ALU”), which is configured to perform matrix multiplication operations that can be used for machine learning operations. In some specific implementations or operating modes, the memory of the memory and machine learning accelerator die 260 is used as a cache by the APD core die 115.

[0020] Figure 2B Details of a sample device 100 and APD core die 115 are shown. Processor 102 ( Figure 1 The device 100 executes operating system 120, driver 122, and application program 126, and alternatively or additionally, may execute other software. Operating system 120 controls various aspects of device 100, such as managing hardware resources, handling service requests, scheduling and controlling process execution, and performing other operations. Driver 122 controls the operation of APD core die 115, sending tasks such as graphics rendering tasks or other jobs to APD core die 115 for processing. Driver 122 also includes a just-in-time (JIT) compiler that compiles programs for execution by processing units of APD core die 115, such as SIMD unit 138 detailed below.

[0021] The APD core die 115 executes commands and procedures for selected functions, such as graphics and non-graphics operations suitable for parallel processing. The APD core die 115 can be used to perform graphics pipeline operations, such as pixel manipulation, geometric calculations, and rendering images to the display device 118 based on commands received from the processor 102. The APD core die 115 also performs computational processing operations not directly related to graphics operations based on commands received from the processor 102, such as operations related to video, physics simulations, computational fluid dynamics, or other tasks. In some examples, these computational processing operations are performed by executing computational shaders on the SIMD unit 138.

[0022] The APD core die 115 includes a computation unit 132 comprising one or more SIMD units 138 configured to perform operations in parallel upon request from processor 102 (or another unit) according to a SIMD paradigm. A SIMD paradigm is one in which multiple processing elements share a single program control flow unit and program counter and thereby execute the same program, but are able to execute the program with different data. In one example, each SIMD unit 138 includes sixteen channels, where each channel executes the same instruction simultaneously with other channels in the SIMD unit 138, but can execute the instruction with different data. If not all channels need to execute a given instruction, channels can be shut down by prediction. Prediction can also be used to execute programs with divergent control flow. More specifically, for programs with conditional branches or other instructions where the control flow is based on computations executed by a single channel, prediction corresponds to channels of control flow paths that are not currently being executed, and the serial execution of different control flow paths can achieve arbitrary control flow.

[0023] The basic execution unit in computing unit 132 is a work item. Each work item represents a single instance of a program to be executed in parallel on a specific channel. Work items can be executed simultaneously (or partially simultaneously and partially sequentially) as “wavefronts” on a single SIMD processing unit 138. One or more wavefronts are included in a “workgroup”, which comprises a set of work items designated to execute the same program. A workgroup can be executed by executing each of the wavefronts that constitute the workgroup. Alternatively, wavefronts may be executed on a single SIMD unit 138 or on different SIMD units 138. A wavefront can be considered as the largest set of work items that can be executed simultaneously (or pseudo-simultaneously) on a single SIMD unit 138. Pseudo-simultaneous execution occurs when the number of wavefronts is greater than the number of channels in SIMD unit 138. In this case, wavefronts are executed in multiple cycles, and different sets of work items are executed in different cycles. APD scheduler 136 is configured to perform operations involving scheduling various workgroups and wavefronts on computing unit 132 and SIMD unit 138.

[0024] The parallelism provided by the computing unit 132 is suitable for graphics-related operations, such as pixel value calculation, vertex transformation, and other graphics operations. Therefore, in some instances, the graphics pipeline 134, which receives graphics processing commands from the processor 102, provides computational tasks to the computing unit 132 for parallel execution.

[0025] The computing unit 132 is also used to perform computational tasks that do not involve graphics or are not part of the “normal” operation of the graphics pipeline 134 (e.g., custom operations performed to supplement the processing performed for the operation of the graphics pipeline 134). An application program 126 or other software executing on the processor 102 sends programs defining such computational tasks to the APD 116 for execution.

[0026] Shader programs (such as compute shader programs) executing on the APD core die 115 can request the memory and machine learning accelerator die 260 to perform machine learning-related operations. Examples of such operations include moving data into or out of the memory and the memory of the ML accelerator die 260, or performing ALU operations (such as matrix multiplication) using data in the memory and the memory of the ML accelerator die 260 as operands.

[0027] It should be noted that, although Figure 2B The APD core die 115 includes a graphics processing pipeline 134 and performs graphics operations, but it should be understood that the teachings of this disclosure can be used in conjunction with an APD core die 115 that does not include a graphics processing pipeline 134 but includes a computing unit 132 and is therefore capable of executing shader programs.

[0028] Figure 3 This is based on an example. Figure 2B A block diagram showing additional details of the graphics processing pipeline 134. The graphics processing pipeline 134 includes stages, each performing a specific function of the graphics processing pipeline 134. Each stage is implemented partly or entirely as a shader program that executes in the programmable computing unit 132, or partly or entirely as fixed-function, non-programmable hardware located outside the computing unit 132.

[0029] Input assembler stage 302 reads graph data from a user-filled cache (e.g., a cache filled at the request of software executed by processor 102, such as application 126) and assembles that data into primitives for use by the rest of the pipeline. Based on the graph data included in the user-filled cache, input assembler stage 302 can generate different types of primitives. Input assembler stage 302 formats the assembled primitives for use by the rest of the pipeline.

[0030] Vertex shader stage 304 processes the vertices of primitives assembled by input assembler stage 302. Vertex shader stage 304 performs various per-vertex operations, such as transform, skinning, warping, and per-vertex lighting. Transformation operations include various operations that transform vertex coordinates. These operations include one or more of the following: modeling transformations that modify vertex coordinates, viewpoint transformations, projection transformations, perspective division, and viewport transformations, as well as other operations that modify non-coordinate attributes.

[0031] The vertex shader stage 304 is partially or fully implemented as a vertex shader program that will execute on one or more computing units 132. The vertex shader program is provided by processor 102 and is based on a program pre-written by a computer programmer. Driver 122 compiles such a computer program to generate a vertex shader program having a format suitable for execution within computing unit 132.

[0032] The shell shader stage 306, the tessellation stage 308, and the domain shader stage 310 work together to perform surface tessellation, which transforms simple primitives into more complex ones by subdividing them. Based on the input primitives, the shell shader stage 306 generates a patch program for surface tessellation. The tessellation stage 308 generates a set of samples for the patch program. The domain shader stage 310 calculates the vertex positions of the vertices corresponding to the samples in the patch program. The shell shader stage 306 and the domain shader stage 310 can be implemented as shader programs that will execute on the computation unit 132, compiled by the driver 122 as in the case of the vertex shader stage 304.

[0033] The geometry shader stage 312 performs vertex operations on a primitive-by-primitive basis. Various types of operations can be performed by the geometry shader stage 312, including operations such as: point sprite expansion, dynamic particle system operations, fur-fin generation, shadow volume generation, single-pass rendering to a cubemap, primitive-by-primitive material exchange, and primitive-by-primitive material setting. In some cases, the operations of the geometry shader stage 312 are performed by a geometry shader program compiled by the driver 122 and executed on the compute unit 132.

[0034] Rasterizer stage 314 receives simple primitives (triangles) generated upstream of rasterizer stage 314 and rasterizes them. Rasterization involves determining which screen pixels (or sub-pixel samples) are covered by a specific primitive. Rasterization is performed by fixed-function hardware.

[0035] Based on the primitives generated upstream and the results of rasterization, pixel shader stage 316 calculates the output value of the screen pixels. Pixel shader stage 316 may apply textures from texture memory. The operation of pixel shader stage 316 is executed by a pixel shader program, which is compiled by driver 122 and executed on computation unit 132.

[0036] Output merger stage 318 accepts output from pixel shader stage 316 and merges these outputs into a target surface, thereby performing operations such as z-testing and alpha blending to determine the final color of the screen pixels. The target surface is the final target of the frames of rendering operations within the graphics processing pipeline 134. The target surface can be located anywhere in memory (such as within the memory of APD 116, or in memory 104).

[0037] Figure 4 This is a schematic diagram of a memory and machine learning accelerator die 260 according to an example. The memory and machine learning accelerator die 260 includes a memory 402, a machine learning accelerator 404, a memory interconnect 406, an inter-die interconnect 408, and a controller 410.

[0038] In some examples, memory 402 is a cache memory (such as a Level 3 cache memory) used as a cache for the APD core die 115 (such as for the compute unit 132). However, in some configurations, at least a portion of memory 402 is directly used for operations performed on the machine learning accelerator 404. The machine learning accelerator 404 includes hardware accelerators for performing operations associated with machine learning. Examples of such operations include matrix multiplication.

[0039] In some operating modes, the APD core die 115 executes shader programs associated with machine learning. The shader programs include instructions for configuring memory 402 for direct use in operations performed on the machine learning accelerator 404. The shader programs also include instructions for storing data in such memory. Furthermore, the shader programs include instructions for causing the machine learning accelerator 404 to perform machine learning operations on the data stored in memory. When the APD core die 115 encounters such instructions, it instructs memory 402 to load data and instructs the machine learning accelerator 404 to perform the requested operation.

[0040] Memory interconnect 406 couples different portions of memory 402 together and to machine learning accelerator 404, enabling machine learning operations to be performed by machine learning accelerator 404 on data stored throughout memory 402. Inter-die interconnect 408 transfers data and commands between memory and machine learning accelerator die 260 and APD core die 115 and processor 102. In some examples, inter-die interconnect 408 is directly coupled to computing unit 132 of APD core die 115. Controller 410 controls operations on memory and machine learning accelerator die 260, such as data transfers and machine learning operations on machine learning accelerator 404.

[0041] Figure 5 The diagram illustrates a reconfiguration of memory 402, according to an example, between memory 502 configured as a cache and memory 504 configured for direct access. Upon request from a shader program or other entity, memory controller 410 changes the amount of memory 402 used as cache 502 and the amount of memory 504 configured for direct access. The amount of memory 402 used as cache 502 and the amount of memory 504 configured for direct access are reconfigurable. In some configurations, cache memory 502 is used for graphics operations (e.g., on graphics processing pipeline 134), and directly accessible memory 504 is used for machine learning operations.

[0042] Figure 6 An exemplary layout 600 of the memory and machine learning accelerator die 260 is shown. The large rectangle shown represents the memory portion 602. Multiple machine learning arithmetic logic units (“ALUs”) 604 are shown. A controller (“Cnt”) 606 is also shown adjacent to the machine learning ALU 604. Multiple inter-die interconnects 608 coupled to the controller are shown. Memory interconnects 610 couple the controller 606 together.

[0043] Memory section 602 is a portion of the memory and machine learning accelerator die 260. These items represent portions of the total memory of die 260 that are local to and directly controlled by a particular controller 606 and ML ALU 604. Inter-die interconnects 608 provide external connectivity from controller 606. In various implementations, these inter-die interconnects 608 are coupled to either or both of the APD core die 115 or the processor 102. Controller 606 communicates via inter-die interconnects 608 between the memory, ML ALU 604, and either or both of the APD core die 115 and the processor 102.

[0044] The memory interconnect 610 is coupled to multiple controllers 606 and provides a communication link between those controllers 606. This link allows the controllers 606 to read from or write to memory portions 602 other than their local memory portions. In one example, the ML ALU 604 is able to perform operations on data from different memory portions 602 by utilizing the memory interconnect 610.

[0045] although Figure 6 Various numbers of elements are shown, but it should be understood that various specific implementations with different numbers of the shown elements are envisioned.

[0046] Although this disclosure describes memory and machine learning accelerator dies coupled to an APD die, in an alternative embodiment, the APD die is a microprocessor die.

[0047] Figure 7 This is a flowchart of a method 700 for performing operations using an APD 116, based on an example. (Although referenced...) Figures 1 to 6 The system described herein is intended to perform the steps of method 700 in any technically feasible order, but those skilled in the art will understand that any system configured to perform the steps of method 700 in any technically feasible order is within the scope of this disclosure.

[0048] Method 700 begins at step 702, where the APD core die 115 executes a shader. This shader includes instructions for performing machine learning operations using one or more ML ALUs 604.

[0049] At step 704, according to the instructions of the shader, the APD core die 115 instructs a set of machine learning ALUs 604 to perform a set of machine learning tasks. This instruction occurs via one or more inter-die interconnects 608.

[0050] At step 706, the machine learning ALU 604 performs the set of machine learning tasks. In various examples, controller 606 receives communication from APD core die 115, including information indicating which operations to perform, and controller 606 instructs one or more ML ALUs 604 to perform those operations. In some examples, controller 606 communicates with another controller 606 via memory interconnect 610 to coordinate the execution of operations.

[0051] Each of the functional units shown in the accompanying drawings represents hardware circuitry configured to perform the operations described herein, software configured to perform the operations described herein, or a combination of software and hardware configured to perform the steps described herein. A non-exclusive list of such units includes storage device 106, processor 102, output driver 114, APD 116, memory 104, input driver 112, input device 106, output device 110, display device 118, operating system 120, driver 122, application program 126, APD scheduler 136, graphics processing pipeline 134, computing unit 132, SIMD unit 138, and any stage of the graphics processing pipeline 134.

[0052] It should be understood that many variations are possible based on the disclosure herein. Although the features and elements described above are described in specific combinations, each feature or element may be used alone without other features and elements, or in various combinations with or without other features or elements.

[0053] The provided methods can be implemented in a general-purpose computer, processor, or processor core. Suitable processors include, for example, general-purpose processors, special-purpose processors, conventional processors, digital signal processors (DSPs), multiple microprocessors, one or more microprocessors associated with a DSP core, controllers, microcontrollers, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), any other type of integrated circuit (IC), and / or state machines. Such processors can be manufactured by configuring a manufacturing process using the results of processed hardware description language (HDL) instructions and other intermediate data, including netlists (such instructions capable of being stored on a computer-readable medium). The result of this processing can be a mask, which is then used in a semiconductor manufacturing process to manufacture processors implementing the features of this disclosure.

[0054] The methods or flowcharts provided herein can be implemented in a computer program, software, or firmware incorporated in a non-transitory computer-readable storage medium for execution by a general-purpose computer or processor. Examples of non-transitory computer-readable storage media include read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media (e.g., internal hard disks and removable disks), magneto-optical media and optical media (e.g., CD-ROM disks), and digital versatile disks (DVDs).

Claims

1. An apparatus comprising: A machine learning die, the machine learning die including memory and one or more machine learning accelerators; as well as A processing core die, stacked with the machine learning die, is configured to execute shader programs to control operations on the machine learning die. Each machine learning accelerator is coupled to a local memory section, coupled to each other machine learning accelerator via a memory interconnect, and coupled to the processing core die via an inter-die interconnect. The memory can be configured to include both cache and directly accessible memory. The memory includes a portion configured to switch between being used as a cache and being used as direct access memory, wherein the switching includes changing the amount of memory used as the cache based on a specified amount; and The cache is configured to perform cache operations for the processing core die, and the directly accessed memory is configured to perform memory operations for the machine learning die.

2. The device of claim 1, wherein the machine learning accelerator is configured to perform matrix multiplication using data in the memory.

3. The device of claim 1, wherein the machine learning die and the processing core die are coupled via one or more inter-die interconnects.

4. The device of claim 1, wherein the processing core die is configured to modify a portion of the memory from being used for the one or more machine learning accelerators to be used as a cache for the processing core die.

5. The device of claim 1, wherein the processing core die is configured to modify a portion of the memory from being used as a cache for the processing core die to be used for the one or more machine learning accelerators.

6. The apparatus of claim 1, wherein the processing core die is configured to execute shader instructions to store data in the memory.

7. The device of claim 1, wherein the machine learning die further includes one or more controllers for controlling the operation of the memory and the machine learning accelerator.

8. The device of claim 1, wherein the machine learning die further includes a memory interconnect that couples the one or more controllers together.

9. The device of claim 8, wherein the memory interconnect is configured to provide data from a portion of the memory to a controller local to a different portion of the memory.

10. A method comprising: Execute shader programs on the core die of the accelerated processing equipment; According to the instructions of the shader program, a set of machine learning arithmetic logic units of a machine learning accelerator die are instructed to perform a set of machine learning tasks via one or more inter-die interconnects, wherein the machine learning accelerator die includes memory, wherein the machine learning accelerator die is stacked with the acceleration processing device core die, wherein the machine learning accelerator die includes one or more machine learning accelerators, and wherein each machine learning accelerator is coupled to a local memory portion, coupled to each other machine learning accelerator via a memory interconnect, and coupled to the acceleration processing device core die via an inter-die interconnect. The set of machine learning tasks is executed using a machine learning arithmetic logic unit; as well as The memory is configured as a cache and a directly accessible memory, wherein the cache is configured to perform cache operations for the core die of the accelerated processing device, and the directly accessible memory is configured to perform memory operations for the machine learning accelerator die, wherein the memory includes a portion configured to switch between being used as a cache and being used as directly accessible memory, wherein the switching includes changing the amount of memory used as the cache based on a specified amount.

11. The method of claim 10, wherein the machine learning accelerator is configured to perform matrix multiplication using data in the memory.

12. The method of claim 10, wherein the machine learning accelerator die and the acceleration processing device core die are coupled via one or more inter-die interconnects.

13. The method of claim 10, wherein the accelerated processing device core die is configured to modify a portion of the memory from being used for the one or more machine learning accelerators to be used as a cache for the accelerated processing device core die.

14. The method of claim 10, wherein the accelerated processing device core die is configured to modify a portion of the memory from being used as a cache for the accelerated processing device core die to be used for the one or more machine learning accelerators.

15. The method of claim 10, wherein the accelerated processing device core die is configured to execute shader instructions to store data in the memory.

16. The method of claim 10, wherein the machine learning accelerator die further includes one or more controllers for controlling the operation of the memory and the machine learning accelerator.

17. The method of claim 10, wherein the machine learning accelerator die further comprises a memory interconnect that couples the one or more controllers together.

18. The method of claim 17, wherein the memory interconnect is configured to provide data from a portion of the memory to a controller local to a different portion of the memory.

19. An apparatus comprising: processor; as well as Accelerated processing equipment, the accelerated processing equipment comprising: Machine learning die, the machine learning die including memory and one or more machine learning accelerators; and A processing core die, stacked with the machine learning die, is configured to execute shader programs to control operations on the machine learning die, wherein one or more shader programs are specified by the processor. Each machine learning accelerator is coupled to a local memory section, coupled to each other machine learning accelerator via a memory interconnect, and coupled to the processing core die via an inter-die interconnect. The memory can be configured to include both cache and directly accessible memory. The memory includes a portion configured to switch between being used as a cache and as direct access memory, wherein the switching includes changing the amount of memory used as the cache based on a specified amount; and The cache is configured to perform cache operations for the processing core die, and the directly accessed memory is configured to perform memory operations for the machine learning die.

20. The apparatus of claim 19, wherein the machine learning accelerator is configured to perform matrix multiplication using data in the memory.

Images