Hierarchical programming model for artificial intelligence hardware

Abstract

Embodiments of the present disclosure include systems and methods for providing hierarchical programming models for artificial intelligence hardware. The system includes a set of lower-level control threads. The system also includes a higher level control thread configured to receive a command from a device, generate a set of commands based on the command, and provide the set of commands to a subset of the set of lower level control threads. A lower-level control thread in the subset of the set of lower-level control threads is configured to instruct a subset of the plurality of processing threads to execute the set of operations based on a particular command in the set of commands.

Description

Background Art

[0001] The present disclosure relates to artificial intelligence (AI). More particularly, the present disclosure relates to artificial intelligence hardware programming techniques for training and reasoning.

[0002] A neural network is a type of machine learning model used in a variety of different applications, such as image classification, computer vision, natural language processing, speech recognition, handwriting recognition, etc. A neural network can be trained for a specific purpose by running a dataset through the neural network, comparing the results from the neural network to known results, and updating the network based on the differences. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Various embodiments of the present disclosure are shown by way of example and not limitation in the figures of the accompanying drawings.

[0004] Figure 1 An architecture of a single-level control hierarchy is shown according to some embodiments.

[0005] Figure 2 An architecture of a two-level control hierarchy is shown according to some embodiments.

[0006] Figure 3 An architecture of a three-level control hierarchy is shown according to some embodiments.

[0007] Figure 4 An artificial intelligence hardware programming process is shown according to some embodiments.

[0008] Figure 5 A simplified block diagram of an example computer system is depicted in accordance with some embodiments.

[0009] Figure 6 A neural network processing system is shown according to some embodiments. DETAILED DESCRIPTION

[0010] In the following description, for the purpose of explanation, a large number of examples and specific details are set forth to provide a thorough understanding of the present disclosure. These examples and details should not be interpreted as unduly limiting the elements of the claim or the entire protected subject matter as a whole. Based on the language of the different claims, it is clear to those skilled in the art that the claimed subject matter may include some or all of the features in these examples, either individually or in combination, and may also include modifications and equivalents to the features and techniques described herein.

[0011] This article describes a technology that provides a hierarchical programming model for artificial intelligence hardware. In some embodiments, the architecture of the programming model includes a hierarchical structure of control threads. The hierarchical structure of control threads may include any number of different hierarchies. At each level of the hierarchical structure, each control thread can asynchronously send commands and parameters to the control threads of the lower hierarchical levels. The lowest level of the hierarchical structure has one or more workgroups, each of which includes a set of threads and a shared memory. The workgroup can perform data calculations and move data. Users can program the architecture to perform a variety of different tasks. For example, users can program the architecture to train neural networks and use the trained neural networks for reasoning. Compared with multi-threaded programming with a flat control hierarchy, the use of a hierarchical control structure provides better scalability.

[0012] In some embodiments, a thread is a static hardware execution context. Each thread may include private state. Examples of private state include command queues, program counters, private memory, register files, etc. In some cases, a thread may receive commands in a queue and execute them in the order in which the thread receives the commands. In some embodiments, the execution of each thread is asynchronous with respect to other threads (unless the program specifies explicit synchronization). Threads may be heterogeneous in the sense that the definition and control mechanisms of thread private state may be different between threads.

[0013] Many different types of threads can be used in the architecture of the programming model. Examples of such threads include control threads, computation threads, and memory threads. In some instances, computation threads and memory threads are collectively referred to as processing threads. In some embodiments, the control thread is configured to send commands to the computation thread, the memory thread, or the lower-level control thread. The computation thread can be configured to perform a set of data computation operations (e.g., matrix multiplication operations, element-by-element operations, etc.). The memory thread can be configured to perform memory operations (e.g., read data from memory, write data to memory, etc.) and / or perform data conversion operations (e.g., matrix transposition operations, collection and distribution operations, data type and layout changes, etc.).

[0014] Figure 1According to some embodiments, the architecture 100 of a single-level control hierarchical structure is shown. In some embodiments, the architecture 100 is implemented on a device (e.g., an artificial intelligence accelerator). As shown, the architecture 100 includes a control thread 105, a synchronization primitive 110, a vector calculation thread 115, a tensor calculation thread 125, a memory thread 135, a shared partition memory (SPM) 140, SPM145, SPM 150, a memory interconnect 155, a main memory 160, and an input / output (I / O) interface 165. In some embodiments, the architecture 100 is implemented in hardware (e.g., a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.), while in other embodiments, the architecture 100 is implemented in software. In some other embodiments, the architecture 100 is implemented by a mixture of hardware and software. For example, in some such embodiments, the control thread 105 and the vector thread 115 can run on a programmable processor (e.g., a traditional central processing unit (CPU) and / or a vector processor), while the tensor calculation thread and the memory thread can run on a custom hardware processor.

[0015] The control thread 105 is responsible for controlling the vector computation thread 115, the tensor computation thread 125, and the memory thread 135. For example, when the control thread 105 receives a specific command (e.g., from a processor in a device implementing the architecture 100, from another device, etc.), the control thread 105 determines a command set for implementing the specific command and then generates the command set. Next, the control thread 105 sends the command set to one or more of the vector computation thread 115, the tensor computation thread 125, and the memory thread 135 via their respective queues. For example, the control thread 105 may send vector-related commands in the command set to the vector computation thread 115, tensor-related commands in the command set to the tensor computation thread 125, and memory-related commands in the command set to the memory thread 135. In some embodiments, the structure of the commands determined by the control thread 105 for each of the vector computation thread 115, the tensor computation thread 125, and the memory thread 135 may be different and suitable for their respective purposes.

[0016] The synchronization primitive 110 provides a mechanism for providing synchronization between the vector computing thread 115, the tensor computing thread 125, the memory thread 135, and the main memory 160. For example, the synchronization primitive 110 can be used for producer-consumer relationships between threads, specifying ordered access of the vector computing thread 115, the tensor computing thread 125, and / or the memory thread 135 to SPM 140, SPM 145, SPM 150, and / or the main memory 160, communicating with remote devices (e.g., artificial intelligence accelerators, hosts, etc.) over a network, multi-workgroup barriers, debug breakpoints, etc. Examples of mechanisms for providing synchronization include semaphores, mutexes, monitors, and / or barriers. Methods for implementing synchronization may include point-to-point messages between threads, shared semaphore register groups, and atomic operations. In some embodiments, the vector computing thread 115, the tensor computing thread 125, and the memory thread 135 execute asynchronously. Thus, in some such embodiments, when a user desires to synchronize vector computation threads 115, tensor computation threads 125, and / or memory threads 135 when creating an executable program for architecture 100, the user explicitly specifies the use of synchronization primitives 110 in the program.

[0017] The vector computing thread 115 is configured to perform vector operations. That is, the vector computing thread 115 can be configured to perform element-by-element operations, that is, operations applied to each element in the vector. As shown in the figure, the vector computing thread 115 includes a queue (Q), through which the vector computing thread 115 receives commands from the control thread 105. Therefore, when the vector computing thread 115 receives a command from the control thread 105 in its queue, the vector computing thread 115 executes the command. In some embodiments, the vector computing thread 115 can be programmed to implement specific commands in different ways. The vector computing thread 115 also includes a private memory 120. In some embodiments, the private memory 120 can only be accessed by the vector computing thread 115, and the tensor computing thread 125 and the memory thread 135 cannot access it. That is, only the vector computing thread 115 can initiate the movement of data into and out of the private memory 120. In some cases, when the vector computing thread 115 executes a command, it may involve the vector computing thread 115 accessing the private memory 120 and / or the SPM 140.

[0018] The tensor computation thread 125 performs tensor operations. Examples of tensor operations include matrix multiplication operations. Figure 1As shown, the tensor computing thread 125 includes a queue (Q) through which the tensor computing thread 125 receives commands from the control thread 105. Therefore, when the tensor computing thread 125 receives a command from the control thread 105 in its queue, the tensor computing thread 125 executes the command. In some embodiments, the tensor computing thread 125 can be programmed to implement specific commands in different ways. In addition, the tensor computing thread 125 includes a private memory 130. In some embodiments, the private memory 130 can only be accessed by the tensor computing thread 125, and the vector computing thread 115 and the memory thread 135 cannot access it. In other words, only the tensor computing thread 125 can initiate data to and from the private memory 130. In some cases, when the tensor computing thread 125 executes a command, it may involve the tensor computing thread 125 accessing the private memory 130, SPM 140, SPM145 and / or SPM 150.

[0019] The memory thread 135 is configured to perform memory operations, perform data conversion operations, and facilitate data communication with other devices. Examples of memory operations include reading data from SPM 140, SPM 145, SPM 150, main memory 160, and / or input / output (I / O) interface 165, and writing data to SPM 140, SPM 145, SPM 150, main memory 160, and / or I / O interface 165. Examples of data conversion operations include matrix transposition operations, collection and distribution operations, data type and layout changes, etc. The memory thread 135 can facilitate data communication with other devices by receiving data from the I / O interface 165 and sending data to the I / O interface 165. As shown, the memory thread 135 includes a queue (Q), and the memory thread 135 receives commands from the control thread 105 through the queue. Therefore, when the memory thread 135 receives a command in its queue from the control thread 105, the memory thread 135 executes the command. In some embodiments, memory thread 135 may be programmed to implement specific commands in different ways. In some cases, when memory thread 135 executes a command, it may involve memory thread 135 accessing SPM 140, SPM 145, SPM 150, main memory 160, and / or I / O interface 165 via memory interconnect 155.

[0020] Each of SPM 140, SPM 145, and SPM 150 is a shared partition memory. In some embodiments, a shared partition memory is a memory that can be accessed by one or more threads and may have affinity with certain threads. An example of a shared partition memory is a temporary storage memory for transferring results between worker threads without entering the main memory. Another example of a shared partition memory is a tensor calculation thread output memory that can be directly accessed by a vector calculation thread. Vector calculation threads 115, tensor calculation threads 125, and memory threads 135 can access each of SPM 140, SPM 145, and SPM 150. Memory interconnect 155 provides access to SPM 140, SPM 145, SPM 150, and main memory 160. Main memory 160 is a memory used by architecture 100 to store data. I / O interface 165 is an interface for accessing external I / O devices (e.g., a network interface, a component bus interface, etc.).

[0021] Figure 2 2 shows an architecture 200 of a two-level control hierarchy according to some embodiments. In some embodiments, the architecture 200 is implemented on a device (e.g., an artificial intelligence accelerator). Figure 2 As shown, architecture 200 includes control thread 202, partition synchronization primitives 204, workgroup 206, workgroup 230, memory thread 254, memory interconnect 256, main memory 258, and I / O interface 260. In some embodiments, architecture 200 is implemented in hardware (e.g., FPGAs, ASICs, etc.), while in other embodiments, architecture 200 is implemented in software. In still other embodiments, architecture 200 is implemented by a mixture of hardware and software.

[0022] The control thread 202 is similar to the control thread 105 in that they are both root control threads at the top of the control hierarchy. Here, the control thread 202 controls the workgroups 206 and 230, and the memory thread 254 via the control threads 208 and 232, respectively. For example, when the control thread 202 receives a specific command (e.g., from a processor included in a device implementing the architecture 200, or from another device, etc.), the control thread 202 determines a command set for implementing the specific command and then generates the command set. The control thread 202 then sends the command set to one or more of the control thread 208, the control thread 232, and the memory thread 254 via their respective queues.

[0023] The partition synchronization primitive 204 provides a mechanism for providing synchronization between the workgroup 206, the workgroup 232, the memory thread 254, and the main memory 258. In some embodiments, the architecture 200 includes a set of global synchronization primitives (not shown), and the partition synchronization primitive 204 is a subset of the set of global synchronization primitives. For example, the partition synchronization primitive 204 can be used for producer-consumer relationships between threads, specifying ordered access to the main memory 258 and / or I / O interface 260 by the workgroup 206 and the workgroup 230, communicating with remote devices (e.g., artificial intelligence accelerators, hosts, etc.) over a network, multi-workgroup barriers, debug breakpoints, etc. Examples of mechanisms for providing synchronization include semaphores, mutexes, monitors, and / or barriers. Methods for implementing synchronization may include point-to-point messages between threads, shared semaphore register groups, and atomic operations. In some embodiments, the workgroup 206, the workgroup 230, and the memory thread 254 execute asynchronously. Thus, in some such embodiments, when a user wants to synchronize workgroups 206, workgroups 230, and / or memory threads 254 when creating a program for execution on architecture 200, the user needs to explicitly specify the use of partition synchronization primitives 204 in the program.

[0024] As mentioned above, a workgroup consists of a set of threads and shared memory. Figure 2 As shown, workgroup 206 includes control thread 208 , partition synchronization primitive 210 , vector computation thread 212 , tensor computation thread 216 , memory thread 220 , SPM 222 , SPM 224 , SPM 226 , and memory interconnect 228 .

[0025] The control thread 208 is configured to control the vector computation thread 212, the tensor computation thread 216, and the memory thread 220. For example, when the control thread 208 receives a specific command from the control thread 202, the control thread 208 determines a command set for implementing the specific command and generates the command set. Then, the control thread 208 sends the command set to one or more of the vector computation thread 212, the tensor computation thread 216, and the memory thread 220 via their respective queues. For example, the control thread 208 may send vector-related commands in the command set to the vector computation thread 212, tensor-related commands in the command set to the tensor computation thread 216, and memory-related commands in the command set to the memory thread 220. In some embodiments, the structure of the commands determined by the control thread 208 for each of the vector computation thread 212, the tensor computation thread 216, and the memory thread 220 may be different and suitable for their respective purposes.

[0026] The partition synchronization primitive 210 is used to provide a mechanism for providing synchronization between the vector computing thread 212, the tensor computing thread 216, and the memory thread 220. As mentioned above, in some embodiments, the architecture 200 includes a set of global synchronization primitives. In some such embodiments, the partition synchronization primitive 210 is a subset of the set of global synchronization primitives. For example, the partition synchronization primitive 210 is used for producer-consumer relationships between threads, specifies the ordered access of the vector computing thread 212, the tensor computing thread 216, and / or the memory thread 220 to SPM 222, SPM 224, and / or SPM 226, communicates with remote devices (e.g., artificial intelligence accelerators, hosts, etc.) over the network, multi-workgroup barriers, debug breakpoints, etc. Examples of mechanisms for providing synchronization include semaphores, mutexes, monitors, and / or barriers. Methods for implementing synchronization may include point-to-point messages between threads, shared semaphore register groups, and atomic operations. In some embodiments, the vector computing thread 212, the tensor computing thread 216, and the memory thread 220 execute asynchronously. Thus, in some such embodiments, when a user desires to synchronize vector compute threads 212, tensor compute threads 216, and / or memory threads 220 when creating a program for execution for architecture 200, the user explicitly specifies the use of partition synchronization primitives 210 in the program.

[0027] 246, SPM 248, and SPM 250. SPM 222, SPM 224, and SPM 226 operate the same as SPM 140, 145, and 150, respectively. Memory interconnect 228 provides access to SPM 222, SPM 224, SPM 226, and memory interconnect 256.

[0028] like Figure 2As shown, the work group 230 includes a control thread 232, a partition synchronization primitive 234, a vector computing thread 236, a tensor computing thread 240, a memory thread 244, an SPM 246, an SPM 248, an SPM 250, and a memory interconnect 252. The control thread 232 is configured to control the vector computing thread 236, the tensor computing thread 240, and the memory thread 244. For example, when the control thread 232 receives a specific command from the control thread 202, the control thread 232 determines a command set for executing the specific command and generates the command set. Then, the control thread 232 sends the command set to one or more of the vector computing thread 236, the tensor computing thread 240, and the memory thread 244 via their respective queues. For example, the control thread 232 may send vector-related commands in the command set to the vector computing thread 236, send tensor-related commands in the command set to the tensor computing thread 240, and send memory-related commands in the command set to the memory thread 244. In some embodiments, the structure of the commands determined by the control thread 232 for each of the vector compute thread 236, the tensor compute thread 240, and the memory thread 244 may be different and suitable for their respective purposes.

[0029] The partition synchronization primitive 234 provides a mechanism for providing synchronization between the vector computing thread 236, the tensor computing thread 240, and the memory thread 244. As mentioned above, in some embodiments, the architecture 200 includes a set of global synchronization primitives. In some such embodiments, the partition synchronization primitive 234 is a subset of the set of global synchronization primitives. For example, the partition synchronization primitive 234 can be used for producer-consumer relationships between threads, specifying ordered access of the vector computing thread 236, the tensor computing thread 240, and / or the memory thread 244 to SPM 222, SPM 224, and / or SPM 226, communicating with remote devices (e.g., artificial intelligence accelerators, hosts, etc.) over a network, multi-workgroup barriers, debug breakpoints, etc. Examples of mechanisms for providing synchronization include semaphores, mutexes, monitors, and / or barriers. Methods for implementing synchronization may include point-to-point messages between threads, shared semaphore register groups, and atomic operations. In some embodiments, the vector computing thread 236, the tensor computing thread 240, and the memory thread 244 execute asynchronously. Thus, in some such embodiments, when a user desires to synchronize vector compute threads 236, tensor compute threads 240, and / or memory threads 244 when creating a program for execution for architecture 200, the user explicitly specifies the use of partition synchronization primitives 234 in the program.

[0030] 224, and SPM 226). SPM 246, SPM 248, and SPM 250 operate the same as SPM 140, 145, and 150, respectively. Memory interconnect 252 provides access to SPM 246, SPM 248, SPM 250, and memory interconnect 256.

[0031] The memory thread 254 is responsible for performing memory operations, data conversion operations, and facilitating data communications with other devices. Examples of memory operations include reading data from the workgroup 206 (e.g., SPM 222, SPM 224, and SPM 226), the workgroup 230 (e.g., SPM 246, SPM 248, and SPM 250), the main memory 258, and / or the I / O interface 260, and writing data to the workgroup 206 (e.g., SPM 222, SPM 224, and SPM 226), the workgroup 230 (e.g., SPM 246, SPM 248, and SPM 250), the main memory 258, and / or the I / O interface 260. Examples of data conversion operations include matrix transposition operations, gather and distribute operations, data type and layout changes, etc. The memory thread 254 can facilitate data communications with other devices by receiving data from and sending data to the I / O interface 260. Figure 2 As shown, the memory thread 254 includes a queue (Q) through which the memory thread 254 receives commands from the control thread 202. Therefore, when the memory thread 254 receives a command in its queue from the control thread 202, the memory thread 254 executes the command. In some embodiments, the memory thread 254 can be programmed to implement specific commands in different ways. In some cases, when the memory thread 254 executes a command, it may involve the memory thread 254 accessing SPM 222, SPM 224, SPM 226, SPM 246, SPM 248, SPM 250, main memory 258 and / or I / O interface 260 via the memory interconnect 256.

[0032] Memory interconnect 256 is configured to provide access to memory interconnect 228, memory interconnect 252, main memory 258, and I / O interface 260. Main memory 258 is a memory used by architecture 200 to store data. I / O interface 260 is an interface for accessing external I / O devices (e.g., a network interface, a component bus interface, etc.).

[0033] Figure 3 300 of a three-level control hierarchy according to some embodiments is shown. In some embodiments, the architecture 300 is implemented on a device (e.g., an artificial intelligence accelerator). Figure 3 As shown, architecture 300 includes control thread 302, partition synchronization primitive 304, cluster 306, cluster 344, memory thread 382, ​​memory interconnect 384, main memory 386 and 388, and I / O interface 390. In some embodiments, architecture 300 is implemented in hardware (e.g., FPGAs, ASICs, etc.), while in other embodiments, architecture 300 is implemented in software. In still other embodiments, architecture 300 is implemented by a mixture of hardware and software.

[0034] Control thread 302 is similar to control threads 105 and 202, which are root control threads at the top of the control hierarchy. In this example, control thread 302 controls clusters 306 and 342, and memory thread 382, ​​via control threads 308 and 344, respectively. For example, when control thread 302 receives a specific command (e.g., from a processor in a device implementing architecture 300, or from another device, etc.), control thread 302 determines a command set for implementing the specific command and generates the command set. Then, control thread 302 sends the command set to one or more of control thread 308, control thread 344, and memory thread 382 through their respective queues.

[0035] Partition synchronization primitive 304 provides a mechanism for providing synchronization between cluster 306, cluster 344, memory thread 382, ​​and main memory 386 and 388. In some embodiments, architecture 300 includes a set of global synchronization primitives (not shown), and partition synchronization primitive 304 is a subset of the set of global synchronization primitives. For example, partition synchronization primitive 304 is used for producer-consumer relationships between threads, specifies ordered access of cluster 306 and cluster 344 to main memory 386 and 388 and / or I / O interface 390, communicates with remote devices (e.g., artificial intelligence accelerators, hosts, etc.) over a network, multi-workgroup barriers, debug breakpoints, etc. Examples of mechanisms for providing synchronization include semaphores, mutexes, monitors, and / or barriers. Methods for implementing synchronization may include point-to-point messages between threads, shared semaphore register groups, and atomic operations. In some embodiments, cluster 306, cluster 344, and memory thread 382 execute asynchronously. Thus, in some such embodiments, when a user desires to synchronize cluster 306, cluster 344, and / or memory threads 382 when creating a program for execution for architecture 300, the user explicitly specifies the use of partition synchronization primitive 304 in the program.

[0036] As mentioned above, a workgroup includes a set of threads and shared memory. In this example, a cluster is a workgroup with a multi-level control hierarchy (two levels in this example), while a tile is a workgroup with a single-level control hierarchy. Figure 3 As shown, cluster 306 includes control thread 308 , partition synchronization primitive 310 , tile 312 , tile 314 , memory thread 338 , memory interconnect 340 , and cluster memory 342 .

[0037] The control thread 308 is responsible for handling the control of the tiles 312, 314, and the memory thread 338. For example, when the control thread 308 receives a specific command from the control thread 302, the control thread 308 determines a command set for executing the specific command and generates the command set. Then, the control thread 308 sends the command set to one or more tiles or memory threads among the tiles 312, 314, and the memory thread 338.

[0038] Partition synchronization primitives 310 are configured to provide a mechanism for providing synchronization between tiles 312, tiles 314, memory threads 338, and cluster memory 342. As mentioned above, in some embodiments, architecture 300 includes a set of global synchronization primitives. In some such embodiments, partition synchronization primitives 310 are a subset of the set of global synchronization primitives. For example, partition synchronization primitives 310 can be used for producer-consumer relationships between threads, specifying ordered access to cluster memory 342 by tiles 312, tiles 314, and memory threads 338, and / or network communications between cluster memory 342 and remote devices (e.g., artificial intelligence accelerators, hosts, etc.), multi-workgroup barriers, debug breakpoints, etc. Examples of mechanisms for providing synchronization include semaphores, mutexes, monitors, and / or barriers. Methods for implementing synchronization may include point-to-point messages between threads, shared semaphore register groups, and atomic operations. In some embodiments, tiles 312, tiles 314, and memory threads 338 execute asynchronously. Thus, in some such embodiments, when a user wishes to synchronize tiles 312, tiles 314, and / or memory threads 338 when creating a program for execution on architecture 300, the user explicitly specifies the use of partition synchronization primitives 310 in the program.

[0039] As explained above, a tile is a work group with a single-level control hierarchy. The following will only describe the operation mode of tile 312. Although tile 314 will not be described in detail, tile 314 is configured to operate in the same or substantially similar manner as tile 312. As shown in the figure, tile 312 includes a control thread 316, a partition synchronization primitive 318, a vector calculation thread 320, a tensor calculation thread 324, a memory thread 328, an SPM 330, an SPM 332, an SPM 334, and a memory interconnect 336. The control thread 316 is responsible for controlling the vector calculation thread 320, the tensor calculation thread 324, and the memory thread 328. For example, when the control thread 316 receives a specific command from the control thread 308, the control thread 316 determines the command set for implementing the specific command, and then generates the command set. Then, the control thread 316 sends the command set to one or more threads in the vector calculation thread 320, the tensor calculation thread 324, and the memory thread 328 via their respective queues. For example, the control thread 316 may send vector-related commands in the command set to the vector computation thread 320, send tensor-related commands in the command set to the tensor computation thread 324, and send memory-related commands in the command set to the memory thread 328. In some embodiments, the structure of the commands determined by the control thread 316 for each of the vector computation thread 320, the tensor computation thread 324, and the memory thread 328 may be different and suitable for their respective purposes.

[0040] The partition synchronization primitive 318 provides a mechanism for providing synchronization between the vector computing thread 320, the tensor computing thread 324, and the memory thread 328. As mentioned above, in some embodiments, the architecture 300 includes a set of global synchronization primitives. In some such embodiments, the partition synchronization primitive 318 is a subset of the set of global synchronization primitives. For example, the partition synchronization primitive 318 can be used for producer-consumer relationships between threads, specifying the ordered access of the vector computing thread 320, the tensor computing thread 324, and / or the memory thread 328 to SPM 330, SPM 332, and / or SPM 334, network communication with remote devices (e.g., artificial intelligence accelerators, hosts, etc.), multi-workgroup barriers, debug breakpoints, etc. Examples of mechanisms for providing synchronization include semaphores, mutexes, monitors, and / or barriers. Methods for implementing synchronization may include point-to-point messages between threads, shared semaphore register groups, and atomic operations. In some embodiments, the vector computing thread 320, the tensor computing thread 324, and the memory thread 328 are executed asynchronously. Therefore, in some such embodiments, when a user wants to create a program for execution for the architecture 300 and synchronize the vector compute threads 320, the tensor compute threads 324 and / or the memory threads 328, the user needs to explicitly specify the use of the partition synchronization primitive 318 in the program.

[0041] 314. Vector compute thread 320 including private memory 322 operates identically to vector compute thread 115 and private memory 120. Additionally, tensor compute thread 324 including private memory 326 operates identically to tensor compute thread 125 and private memory 130. Memory thread 328 operates identically to memory thread 135, except that memory thread 328 can also access the memory of tile 314 as well as data in cluster memory 342. SPM 330, SPM 332, and SPM 334 operate identically to SPM 140, 145, and 150, respectively. Memory interconnect 336 provides access to SPM 330, SPM 332, SPM 334, and memory interconnect 340.

[0042] The memory thread 338 is configured to perform memory operations, data conversion operations, and facilitate data communications with other devices. Examples of memory operations include reading data from tiles 312 (e.g., SPM 330, SPM 332, and SPM 334), tiles 314, cluster memory 342, cluster 344, main memory 386 and 388, and / or input / output (I / O) interface 390, and writing data to tiles 312 (e.g., SPM 330, SPM 332, and SPM 334), tiles 314, cluster memory 342, cluster 344, main memory 386 and 388, and / or I / O interface 390. Examples of data conversion operations include matrix transposition operations, gather and distribute operations, data type and layout changes, etc. The memory thread 338 can facilitate data communications with other devices by receiving data from and sending data to the I / O interface 390. Figure 3 As shown, the memory thread 338 includes a queue (Q) through which the memory thread 338 receives commands from the control thread 308. Therefore, when the memory thread 338 receives a command in its queue from the control thread 308, the memory thread 338 executes the command. In some embodiments, the memory thread 338 can be programmed to implement specific commands in different ways. In some cases, when the memory thread 338 executes a command, it may involve the memory thread 338 accessing SPM 330, SPM 332, SPM 334, SPM 368, SPM 370, SPM 372, cluster memory 342, cluster memory 380, main memory 386 and 388, and / or I / O interface 390 via memory interconnect 340. Memory interconnect 340 is configured to provide access to tile 312, tile 314, cluster memory 342, and memory interconnect 384. Cluster memory 342 is a memory used by cluster 306 to store data.

[0043] As mentioned above, a cluster is a workgroup with multiple levels of control hierarchy (two levels in this example), while a tile is a workgroup with a single level of control hierarchy. Figure 3 As shown, cluster 344 includes control thread 346 , partition synchronization primitives 348 , tile 350 , tile 352 , memory thread 376 , memory interconnect 378 , and cluster memory 380 .

[0044] The control thread 346 is responsible for handling control work of the tile 350, the tile 352, and the memory thread 376. For example, when the control thread 346 receives a specific command from the control thread 302, the control thread 346 determines a command set for executing the specific command and generates the command set. Then, the control thread 346 sends the command set to one or more tiles or memory threads among the tile 350, the tile 352, and the memory thread 376.

[0045] Partition synchronization primitives 348 are configured to provide a mechanism for providing synchronization between tiles 350, tiles 352, memory threads 376, and cluster memory 380. As mentioned above, in some embodiments, architecture 300 includes a set of global synchronization primitives. In some such embodiments, partition synchronization primitives 348 are a subset of the set of global synchronization primitives. For example, partition synchronization primitives 348 can be used for producer-consumer relationships between threads, specifying ordered access to cluster memory 380 by tiles 350, tiles 352, and memory threads 376, and / or network communications between cluster memory 380 and remote devices (e.g., artificial intelligence accelerators, hosts, etc.), multi-workgroup barriers, debug breakpoints, etc. Examples of mechanisms for providing synchronization include semaphores, mutexes, monitors, and / or barriers. Methods for implementing synchronization may include point-to-point messages between threads, shared semaphore register groups, and atomic operations. In some embodiments, tiles 350, tiles 352, and memory threads 376 execute asynchronously. Thus, in some such embodiments, when a user wants to create a program to be executed for architecture 300 and synchronize tiles 350, tiles 352, and / or memory threads 376, the user needs to explicitly specify the use of partition synchronization primitives 348 in the program.

[0046] As described above, a tile is a workgroup with a single level of control hierarchy. Only the operation of tile 350 will be described below. Although tile 352 will not be described in detail, tile 352 is configured to operate the same or substantially similar to tile 350. Figure 3 As shown, tile 350 includes a control thread 354, a partition synchronization primitive 356, a vector computing thread 358, a tensor computing thread 362, a memory thread 366, an SPM 368, an SPM 370, an SPM 372, and a memory interconnect 374. The control thread 354 is configured to control the vector computing thread 358, the tensor computing thread 362, and the memory thread 366. For example, when the control thread 354 receives a specific command from the control thread 346, the control thread 354 determines a command set for executing the specific command and then generates the command set. Then, the control thread 354 sends the command set to one or more threads of the vector computing thread 358, the tensor computing thread 362, and the memory thread 366 through their respective queues. For example, the control thread 354 can send vector-related commands in the command set to the vector computing thread 358, send tensor-related commands in the command set to the tensor computing thread 362, and send memory-related commands in the command set to the memory thread 366. In some embodiments, the structure of the commands determined by the control thread 354 for each of the vector computation thread 358, the tensor computation thread 362, and the memory thread 366 may be different and suitable for their respective purposes.

[0047] The partition synchronization primitive 356 provides a mechanism for providing synchronization between the vector computing thread 358, the tensor computing thread 362, and the memory thread 366. As mentioned above, in some embodiments, the architecture 300 includes a set of global synchronization primitives. In some such embodiments, the partition synchronization primitive 356 is a subset of the set of global synchronization primitives. For example, the partition synchronization primitive 356 can be used for producer-consumer relationships between threads, specifying the ordered access of the vector computing thread 358, the tensor computing thread 362, and / or the memory thread 366 to SPM 368, SPM 370, and / or SPM 372, network communication with remote devices (e.g., artificial intelligence accelerators, hosts, etc.), multi-workgroup barriers, debug breakpoints, etc. Examples of mechanisms for providing synchronization include semaphores, mutexes, monitors, and / or barriers. Methods for implementing synchronization may include point-to-point messages between threads, shared semaphore register groups, and atomic operations. In some embodiments, the vector computing thread 358, the tensor computing thread 362, and the memory thread 366 execute asynchronously. Therefore, in some such embodiments, when a user wants to create a program for execution for the architecture 300 and synchronize the vector compute threads 358, the tensor compute threads 362 and / or the memory threads 366, the user needs to explicitly specify the use of the partition synchronization primitive 356 in the program.

[0048] 360 operates identically to vector compute thread 115 and private memory 120. Likewise, tensor compute thread 362 operates identically to tensor compute thread 125 and private memory 130, including private memory 364. Memory thread 366 operates identically to memory thread 135, except that memory thread 366 can also access the memory of tile 352 and data in cluster memory 380. SPM 368, SPM 370, and SPM 372 operate identically to SPM 140, 145, and 150, respectively. Memory interconnect 374 provides access to SPM 368, SPM 370, SPM 372, and memory interconnect 378.

[0049] Memory thread 376 is used to perform memory operations, data conversion operations, and facilitate data communication with other devices. Examples of memory operations include reading data from tile 350 (e.g., SPM 368, SPM 370, and SPM 372), tile 352, cluster memory 380, cluster 306, main memory 386 and 388, and / or input / output (I / O) interface 390, and writing data to tile 350 (e.g., SPM 368, SPM 370, and SPM 372), tile 352, cluster memory 380, cluster 306, main memory 386 and 388, and / or I / O interface 390. Examples of data conversion operations include matrix transposition operations, gather and distribute operations, data type and layout changes, etc. Memory thread 376 can facilitate data communication with other devices by receiving data from and sending data to I / O interface 390. Figure 3 As shown, the memory thread 376 includes a queue (Q) through which the memory thread 376 receives commands from the control thread 346. Therefore, when the memory thread 376 receives a command from the control thread 346 in its queue, the memory thread 376 executes the command. In some embodiments, the memory thread 376 can be programmed to implement specific commands in different ways. In some cases, when the memory thread 376 executes a command, it may involve the memory thread 376 accessing SPM 330, SPM 332, SPM 334, SPM 368, SPM 370, SPM 372, cluster memory 342, cluster memory 380, main memory 386 and 388, and / or I / O interface 390 via the memory interconnect 374. The memory interconnect 378 is configured to provide access to the tile 350, tile 352, cluster memory 380, and memory interconnect 384. The cluster memory 380 is a memory used by the cluster 344 to store data.

[0050] The memory thread 382 is responsible for performing memory operations, data conversion operations, and facilitating data communications with other devices. Examples of memory operations include reading data from clusters 306 (e.g., tiles 312 and 314), clusters 344 (e.g., tiles 350 and 352), main memories 386 and 388, and / or I / O interfaces 390, and writing data to clusters 306 (e.g., tiles 312 and 314), clusters 344 (e.g., tiles 350 and 352), main memories 386 and 388, and / or I / O interfaces 390. Examples of data conversion operations include matrix transposition operations, gather and distribute operations, data type and layout changes, and the like. The memory thread 382 can facilitate data communications with other devices by receiving data from and sending data to the I / O interface 260. As Figure 3As shown, memory thread 382 includes a queue (Q) through which memory thread 382 receives commands from control thread 302. Thus, when memory thread 382 receives a command in its queue from control thread 302, memory thread 382 executes the command. In some embodiments, memory thread 382 may be programmed to implement specific commands in different ways. In some cases, when memory thread 382 executes a command, it may involve memory thread 382 accessing SPM 330, SPM 334, SPM 336, SPM 368, SPM 370, SPM 372, main memory 386 and 388, and / or I / O interface 390 via memory interconnect 384, memory interconnect 340, and / or memory interconnect 378.

[0051] Memory interconnect 384 is responsible for providing access to memory interconnect 340, memory interconnect 378, main memories 386 and 388, and I / O interface 390. Main memories 386 and 388 are memories used by architecture 300 to store data. I / O interface 390 is an interface for accessing external I / O devices (e.g., a network interface, a component bus interface, etc.).

[0052] Figure 1-Figure 3 An example architecture of a control hierarchy with different numbers of layers is shown. At the lowest level of each control hierarchy is a set of workgroups, each of which includes vector computing threads, tensor computing threads, and memory threads. It should be understood by those of ordinary skill in the art that additional and / or different types of threads may be included in a workgroup. For example, a workgroup may include a network thread configured to perform network operations (e.g., processing data packets, forwarding data packets, etc.), an encryption thread configured to perform encryption operations, a decryption thread configured to perform decryption operations, a compression thread configured to perform data compression operations, a decompression thread configured to perform data decompression operations, and the like. In addition, different workgroups may contain different sets of threads. For example, a first workgroup in an architecture may include vector computing threads, tensor computing threads, and memory threads, while a second workgroup in the architecture may include network threads, compression threads, and decompression threads.

[0053] Figure 4 4 shows an artificial intelligence hardware programming process 400 according to some embodiments. In some embodiments, a user who creates a program using the artificial intelligence hardware hierarchical programming model can perform process 400. Process 400 first programs a higher-level control thread at step 410 to generate a command set based on a command received from a device. Figure 2 For example, a user may program control thread 202 to determine a command set upon receiving a command from a device implementing architecture 200 and then generate the command set.

[0054] Next, process 400 programs the higher-level control thread at step 420 to provide the command set to a subset of the lower-level control threads. Figure 2 For example, the user can program the control thread 202 to send the generated command set to the control thread 208, the control thread 232 and / or the memory thread 254. In turn, the control thread 208 and / or the control thread 232 can generate a command set based on the received command, and then send the command set generated by them to one or more threads of the vector calculation thread 212, the tensor calculation thread 216, the memory thread 220, the vector calculation thread 236, the tensor calculation thread 240 and / or the memory thread 244.

[0055] Finally, process 400 programs, at step 430, a lower-level control thread in the subset of lower-level control threads to instruct a subset of the plurality of processing threads to perform the set of operations based on a specific command in the command set. Figure 2 For example, a user may program the control thread 208 and / or 232 to generate a command set based on the command received from the control thread 202, as mentioned above. The command set generated by the control thread 208 and / or the control thread 232 instructs one or more of the vector computing thread 212, the tensor computing thread 216, the memory thread 220, the vector computing thread 236, the tensor computing thread 240, and / or the memory thread 244 to execute the command set.

[0056] The above-described techniques can be implemented in a variety of computer systems configured to process neural networks. Figure 5 5 shows a simplified block diagram of an example computer system 500 that can be used to implement the techniques described in the above disclosure. Figure 5 As shown, computer system 500 includes one or more processors 502, which communicate with multiple peripheral devices via bus subsystem 504. These peripheral devices may include storage subsystem 506 (e.g., including memory subsystem 508 and file storage subsystem 510) and network interface subsystem 516. Some computer systems may also include user interface input device 512 and / or user interface output device 514.

[0057] The bus subsystem 504 may provide a mechanism that enables the various components and subsystems of the computer system 500 to communicate with each other as intended. Although the bus subsystem 504 is shown in the diagram as a single bus, alternative embodiments of the bus subsystem may use multiple busses.

[0058] The network interface subsystem 516 may serve as an interface for data communication between the computer system 500 and other computer systems or networks. Embodiments of the network interface subsystem 516 may include, for example, Ethernet, Wi-Fi and / or cellular adapters, modems (telephone, satellite, cable TV, integrated services digital network, etc.), digital subscriber line (DSL) units, and the like.

[0059] Storage subsystem 506 includes memory subsystem 508 and file / disk storage subsystem 510. Subsystems 508 and 510, as well as other memories described herein, are examples of non-transitory computer-readable storage media that can store executable program code and / or data that provide the functionality of embodiments of the present disclosure.

[0060] The memory subsystem 508 includes a plurality of memories, including a main random access memory (RAM) 518 for storing instructions and data during program execution and a read-only memory (ROM) 520 for storing fixed instructions. The file storage subsystem 510 can provide persistent (e.g., non-volatile) storage for program and data files, and can include a magnetic or solid-state hard drive, an optical drive and associated removable media (e.g., CD-ROM, DVD, Blu-ray Disc, etc.), a removable flash-based drive or memory card, and / or other types of storage media known in the art.

[0061] It should be understood that computer system 500 is merely an example and that many other configurations are possible having more or fewer components than this system.

[0062] Figure 6 A neural network processing system according to some embodiments is shown. In various embodiments, the neural network in the present disclosure can be implemented and trained in a hardware environment including one or more neural network processors. The neural network processor may refer to various graphics processing units (GPUs) (e.g., GPUs produced by NVIDIA for processing neural networks), field programmable gate arrays (FPGAs) (e.g., FPGAs produced by Xilinx for processing neural networks), or various application-specific integrated circuits (ASICs) or neural network processors including hardware architectures optimized for neural network computing. In this example environment, one or more servers 602 (which may include the above-mentioned Figure 5 The architecture shown in FIG. 6A and FIG. 6B may be connected to a plurality of controllers 610 (1)-610 (M) via a communication network 601 (e.g., a switch, a router, etc.). The controllers 610 (1)-610 (M) may also include Figure 5The architecture shown. Each controller 610(1)-610(M) may be connected to one or more neural network processors, such as processors 611(1)-611(N) and 612(1)-612(N). The neural network processors 611(1)-611(N) and 612(1)-612(N) may include various configurations of functional processing modules and memories optimized for neural network processing (e.g., training or inference). The neural network processors are optimized for neural network computations. In some embodiments, Figure 1-Figure 3 Any of the architectures shown in can be used to implement the neural network processors 611 (1)-611 (N) and 612 (1)-612 (N). The server 602 can configure the controller 610 with a neural network model and input data for the model, which can be loaded and executed in parallel by the neural network processors 611 (1)-611 (N) and 612 (1)-612 (N). The model may include layers and associated weights as mentioned above. The neural network processor can load the model and apply the input to generate output results. The neural network processor can also implement the training algorithm described herein.

[0063] More Example Embodiments

[0064] In various embodiments, the present disclosure includes systems, methods, and apparatus for providing a hierarchical programming model for artificial intelligence hardware. The techniques described herein may be embodied in a non-transitory machine-readable medium storing a program executable by a computer system, the program comprising a set of instructions for performing the techniques described herein. In some embodiments, the system includes a set of processing units and a non-transitory machine-readable medium storing instructions that, when executed by at least one of the processing units, cause the at least one processing unit to perform the above-described techniques. In some embodiments, the non-transitory machine-readable medium may be a memory, for example, which may be connected to one or more controllers or one or more artificial intelligence processors.

[0065] The following techniques may be embodied alone or in various combinations, and may also be embodied together with other techniques described herein.

[0066] For example, in an embodiment, the present disclosure includes a system comprising a set of lower-level control threads and a higher-level control thread, the higher-level control thread being configured to receive a command from a device, generate a command set based on the command, and provide the command set to a subset of the lower-level control threads, wherein the set of lower-level control threads in the subset of lower-level control threads is configured to instruct a subset of multiple processing threads to execute an operation set based on specific commands in the command set, wherein the higher-level control thread and the lower-level control thread form a hierarchical control structure that can provide better scalability.

[0067] In one embodiment, the higher-level control thread provides the command set to the lower-level control thread set by sending each command in the command set to the lower-level control thread in the subset of the lower-level control thread set. The lower-level control thread in the subset of the lower-level control thread set is also configured to receive specific commands in the command set from the higher-level control thread.

[0068] In one embodiment, the command set is a first command set. The system also includes a set of intermediate control threads, wherein the higher-level control thread provides the first command set to the set of lower-level control threads by sending each command in the first command set to an intermediate control thread set in a subset of the set of intermediate control threads, wherein each intermediate control thread in the subset of the set of intermediate control threads is configured to receive the commands in the first command set, generate a second command set, and send each command in the second command set to a lower-level control thread in the set of lower-level control threads.

[0069] In one embodiment, the set of operations includes a set of data transformation operations.

[0070] In one embodiment, the set of operations includes a set of memory access operations.

[0071] In one embodiment, a processing thread in the plurality of processing threads is a memory access thread.

[0072] In one embodiment, the set of operations includes a set of computational operations.

[0073] In one embodiment, a processing thread in the plurality of processing threads is a compute thread.

[0074] In one embodiment, the compute thread is a vector compute thread configured to perform element-wise operations.

[0075] In one embodiment, the compute thread is a tensor compute thread configured to perform matrix multiplication operations.

[0076] In one embodiment, the present disclosure includes a first device comprising a lower-level control thread set; and a higher-level control thread configured to receive a command from a second device, generate a command set based on the command, and provide the command set to a subset of the lower-level control thread set, wherein the lower-level control threads in the subset of the lower-level control thread set are configured to instruct a subset of multiple processing threads to execute an operation set based on specific commands in the command set, wherein the higher-level control thread and the lower-level control thread constitute a hierarchical control structure that can provide better scalability.

[0077] In one embodiment, the present disclosure includes a method, comprising programming a higher-level control thread to generate a command set based on a command received from a device; programming the higher-level control thread to provide the command set to a subset of a lower-level control thread set; and programming a lower-level control thread in the subset of the lower-level control thread set to instruct a subset of multiple processing threads to execute an operation set based on a specific command in the command set, wherein the higher-level control thread and the lower-level control thread constitute a hierarchical control structure that can provide better scalability.

[0078] The above description shows various embodiments of the present disclosure and examples of how aspects of specific embodiments can be implemented. The above examples should not be regarded as the only embodiments, and they are intended to illustrate the flexibility and advantages of specific embodiments defined by the following claims. Based on the above disclosure and the following claims, other arrangements, embodiments, implementations and equivalent forms may be adopted without departing from the scope of the present disclosure defined by the claims.

Claims

1. A system comprising: A set of lower-level control threads; as well as a higher-level control thread configured to: receive commands from a device, generate a command set based on the commands, and provide the command set to a subset of the set of lower-level control threads; wherein a lower-level control thread in the subset of the set of lower-level control threads is configured to instruct a subset of the plurality of processing threads to perform a set of operations based on a particular command in the set of commands; The higher-level control thread and the lower-level control thread set form a hierarchical control structure, and the hierarchical control structure provides better scalability.

2. The system of claim 1 , wherein the higher-level control thread provides the command set to the lower-level control thread set by sending each command in the command set to a lower-level control thread in the subset of the lower-level control thread set, wherein the lower-level control threads in the subset of the lower-level control thread set are also configured to receive the specific command in the command set from the higher-level control thread.

3. The system of claim 1 , wherein the command set is a first command set, wherein the system further comprises a set of intermediate control threads, wherein the higher-level control thread provides the first command set to the lower-level control thread set by sending each command in the first command set to an intermediate control thread in a subset of the intermediate control thread set, wherein each intermediate control thread in the subset of the intermediate control thread set is configured to: receive commands in the first command set, generate a second command set, and send each command in the second command set to a lower-level control thread in the set of lower-level control threads. The system of claim 1 , wherein the set of operations comprises a set of data transformation operations. The system of claim 1 , wherein the set of operations comprises a set of memory access operations. The system of claim 5 , wherein a processing thread of the plurality of processing threads is a memory access thread.

7. The system of claim 1, wherein the set of operations comprises a set of computational operations.

8. The system of claim 7, wherein a processing thread of the plurality of processing threads is a computing thread.

9. The system of claim 8, wherein the compute thread is a vector compute thread configured to perform element-wise operations.

10. The system of claim 8, wherein the computation thread is a tensor computation thread configured to perform a matrix multiplication operation.

11. A first device, comprising: A set of lower-level control threads; as well as a higher-level control thread configured to: receive a command from a second device, generate a command set based on the command, and provide the command set to a subset of the set of lower-level control threads; wherein a lower-level control thread in the subset of the set of lower-level control threads is configured to instruct a subset of the plurality of processing threads to perform a set of operations based on a particular command in the set of commands; The higher-level control thread and the lower-level control thread set form a hierarchical control structure, and the hierarchical control structure provides better scalability.

12. A first device according to claim 11, wherein the higher-level control thread provides the command set to the lower-level control thread set by sending each command in the command set to the lower-level control threads in the subset of the lower-level control thread set, and wherein the lower-level control threads in the subset of the lower-level control thread set are also configured to receive the specific command in the command set from the higher-level control thread.

13. A first device according to claim 11, wherein the command set is a first command set, wherein the system further includes a set of intermediate control threads, wherein the higher-level control thread provides the first command set to the lower-level control thread set by sending each command in the first command set to an intermediate control thread in a subset of the intermediate control thread set, wherein each intermediate control thread in the subset of the intermediate control thread set is configured to: receive commands in the first command set, generate a second command set, and send each command in the second command set to a lower-level control thread in the lower-level control thread set.

14. A method comprising: programming a higher level control thread to generate a command set based on a command received from the device; programming the higher level thread of control to provide the set of commands to a subset of the set of lower level threads of control; as well as programming a lower-level control thread in the subset of the set of lower-level control threads to instruct a subset of the plurality of processing threads to perform a set of operations based on a particular command in the set of commands, The higher-level control thread and the lower-level control thread set form a hierarchical control structure, which provides better scalability.

15. The method of claim 14, wherein the higher-level control thread provides the command set to the lower-level control thread set by sending each command in the command set to a lower-level control thread in the subset of the lower-level control thread set, wherein the lower-level control threads in the subset of the lower-level control thread set are also configured to receive the specific command in the command set from the higher-level control thread.

Images