Tensor-based hardware accelerators including scalar processing units

Abstract

This document describes a computing system that uses a hardware accelerator to accelerate deep neural network (DNN) operations or other processing operations. The hardware accelerator further includes a tensor processing engine that works in conjunction with a scalar processing unit (SPU). The tensor processing engine handles various tensor-based operations required by DNNs, such as multiplying vectors by matrices, combining vectors with other vectors, transforming individual vectors, etc. The SPU performs scalar-based operations, such as forming the reciprocal of a scalar, generating the square root of a scalar, etc. According to one illustrative implementation, the computing system uses the same vector-based programming interface to interact with both the tensor processing engine and the SPU.

Description

Background Art

[0001] Many deep neural networks (DNNs) operate in environments that require low-latency responses. These types of environments include search engines, real-time video analysis engines, interactive speech recognition engines, sensor analysis engines, etc. For example, a user interacting with a search engine will expect a near-real-time response to his or her search request. Summary of the Invention

[0002] This article describes a computing system that uses a hardware accelerator to accelerate deep neural network (DNN) operations or other processing operations. The hardware accelerator further includes a tensor processing engine that works in conjunction with a scalar processing unit (SPU). The tensor processing engine handles the various tensor-based operations required for DNNs (or other analytical tasks). These tensor-based operations make up the majority of the work performed by the hardware accelerator and cover operations such as multiplying vectors by matrices, combining vectors with other vectors, and transforming individual vectors. The SPU performs scalar-based operations such as forming the reciprocal of a scalar, generating the square root of a scalar, and so on.

[0003] According to an illustrative implementation, the computing system uses the same vector-based programming interface to interact with both the tensor processing engine and the SPU. To operate in this manner, the SPU includes a vector-to-scalar interface that converts input vectors received from the tensor processing engine into one or more scalars. The SPU stores (multiple) input scalars in a global scalar register file. The SPU also includes one or more scalar operation hardware components (HCs). Each scalar operation HC reads at least one input scalar from the scalar register file, performs some scalar operations on (multiple) input scalars to generate at least one output scalar, and then writes (multiple) output scalars back to the scalar register file. The SPU also includes a scalar-to-vector interface that converts one or more scalars stored in the scalar register file into output vectors.

[0004] According to one implementation, the vector-to-scalar interface can generate a scalar by generating the sum, maximum, or minimum of the element values ​​in the input vector. The scalar-to-vector interface can replicate the scalar retrieved from the scalar register file to generate an output vector. Alternatively or additionally, the scalar register file includes a write port for receiving an input vector in vector form and a read port for providing an output vector in vector form.

[0005] According to one technical advantage, a computing system seamlessly integrates scalar operations into a processing pipeline that is otherwise dedicated to tensor-based operations. This feature improves the efficiency of the computing system. It also facilitates the task of writing computer programs that include both tensor-based and scalar operations.

[0006] This summary is provided to introduce a series of concepts in a simplified form; these concepts are further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 An overview of an illustrative computing system including a hardware accelerator is shown. The hardware accelerator, in turn, includes a tensor processing engine and a scalar processing unit.

[0008] Figure 2 Shown Figure 1 A more detailed view of the computing system.

[0009] Figure 3 Shown by Figure 1 and Figure 2 An implementation of a tensor processing engine provided by a computing system.

[0010] Figure 4 Shown by Figure 1 and Figure 2 An implementation of a scalar processing unit provided by a computing system.

[0011] Figure 5 Shown Figure 4 An alternative implementation of the parts of the scalar processing unit shown in .

[0012] Figure 6 Shown Figure 1 and Figure 2 An example of how a computing system can be used to perform a softmax (normalized exponential function) operation.

[0013] Figure 7 It shows Figure 1 and Figure 2 A flowchart of an illustrative mode of operation of a computing system.

[0014] Figure 8 Illustrative computing hardware is shown that can be used to implement any aspect of the features shown in the preceding figures.

[0015] The same reference numerals are used throughout the disclosure and drawings to refer to the same components and features. Figure 1 , 200 series reference numerals refer to features originally found in Figure 2 , 300 series reference numerals refer to features originally found in Figure 3 , and so on. DETAILED DESCRIPTION

[0016] This disclosure is organized as follows. Section A describes a computing system having a hardware accelerator that accelerates operations associated with a deep neural network (DNN) or other type of analytical logic. Section B sets forth an illustrative method for explaining the operation of the computing system of Section A. And, Section C describes illustrative computing hardware that can be used to implement any aspect of the features described in Sections A and B.

[0017] As a preliminary matter, some of the figures in the accompanying drawings illustrate structural features of computing systems. The separation of various components in these figures into distinct units may reflect the use of corresponding distinct physical and tangible parts in actual implementations. Alternatively or additionally, any single component illustrated in the figures may be implemented by multiple actual physical parts. Alternatively or additionally, the depiction of any two or more separate components in the figures may reflect different functions performed by a single actual physical part.

[0018] Other drawings have described these concepts in the form of flow charts. In this form, certain operations are described as constituting different blocks that are performed in a particular order. Such implementation is illustrative and not restrictive. Certain blocks described herein can be grouped together and performed in a single operation, certain blocks can be decomposed into multiple component blocks, and certain blocks can be performed in an order different from the order illustrated herein (including the mode of executing blocks in parallel).

[0019] Any storage resource or any combination of storage resources described herein can be considered a computer-readable medium. In many cases, a computer-readable medium represents some form of physical and tangible entity. The term computer-readable medium also encompasses propagated signals, such as those transmitted or received via physical conduits and / or air or other wireless media. However, the specific term "computer-readable storage medium" explicitly excludes propagated signals themselves, while encompassing all other forms of computer-readable media.

[0020] The following explanation may identify one or more features as "optional." Such a statement should not be interpreted as an exhaustive description of features that may be considered optional; that is, other features may be considered optional even though not explicitly stated in the text. Furthermore, any description of a single entity is not intended to exclude the use of multiple such entities; likewise, a description of multiple entities is not intended to exclude the use of a single entity. Furthermore, while the description may explain certain features as alternative ways of performing the identified function or implementing the identified mechanism, these features may also be combined in any combination. Finally, the term "exemplary" or "illustrative" refers to one implementation among potentially many implementations.

[0021] A. Descriptive Computing Systems

[0022] Figure 1 An illustrative computing system 102 is shown that includes a hardware accelerator 104. Hardware accelerator 104, in turn, includes two types of processing engines 106: a tensor processing engine 108 and a scalar processing unit (SPU) 110. Tensor processing engines 108 perform operations in the form of tensors. SPUs 110 perform operations in the form of scalars. Typically, most of the work performed by the processing engines is assigned to tensor processing engines 108. In these contexts, SPUs 110 can be viewed as additional processors that handle residual non-tensor-related work within a tensor-based data flow.

[0023] As used herein, a tensor includes an array of two or more elements. A tensor includes at least a matrix (having at least two dimensions) and a vector. A scalar refers to a single element with a value. As used herein, the terms "scalar" and "scalar value" are considered synonyms.

[0024] In one implementation, the hardware accelerator 104 is implemented as a dedicated hardware component. For example, the hardware accelerator 104 may include a structure of a field programmable gate array (FPGA) unit. Alternatively or additionally, the hardware accelerator 104 may be implemented using one or more application-specific integrated circuits (ASICs). Alternatively or additionally, the hardware accelerator 104 may be implemented using a general-purpose graphics processing unit (GPGPU), etc. The above examples are provided in the spirit of illustration and not limitation; the hardware accelerator 104 may be implemented by other types of dedicated hardware units.

[0025] In some implementations, the hardware accelerator 104 represents a "soft" processor that can be reprogrammed after its manufacture to perform different corresponding operations. The hardware accelerator 104 can achieve this result using a programmable gate array device such as an FPGA. In other implementations, the hardware accelerator 104 is hardwired to perform a fixed set of operations. The hardware accelerator 104 can achieve this result using, for example, a hardwired gate array, an ASIC, etc.

[0026] In some implementations, the hardware accelerator 104 represents a neural processing unit (NPU) that performs operations associated with (multiple) any type of deep neural network (DNN) or (multiple) other types of neural networks (i.e., other than DNNs). Exemplary types of DNNs include convolutional neural networks (CNNs), recurrent neural networks (RNNs), etc. RNNs may include long short-term memory (LSTM) units, gated recurrent units (GRUs), etc. More specifically, the computing system 102 uses the hardware accelerator 104 to speed up the application phase operations of an already trained DNN (where the term "application phase" is intended to distinguish it from offline training of the DNN). In some other implementations, the hardware accelerator 104 implements some other computationally intensive analytical tasks, not necessarily associated with DNNs. The computing system 102 may also use the hardware accelerator 104 to train DNNs or other types of neural networks.

[0027] The hardware accelerator 104 also includes an optional vector memory manager (VMM) 112. Among other tasks, the VMM 112 manages the exchange of data between the processing engine 106 and various other components. For example, the VMM 112 acts as a vector interface between the processing engine 106 and memory (e.g., DRAM) 114. The VMM 112 also acts as an interface between the processing engine 106 and one or more network interfaces 116.

[0028] The network interface(s) 116 interact with one or more consumer systems 118 via one or more channels 120. To this end, the network interface(s) 116 maintain input queues for receiving input data and commands from the consumer systems 118. The network interface(s) 116 forward the input data to the processing engine 106 for processing. The network interface(s) 116 also maintain output queues for receiving output data generated by the processing engine 106. The network interface(s) forward the output data to the consumer systems 118 via the channel(s) 120. The channel(s) 120 may include any of the following: a local area network (such as a local area network provided by a data center), a wide area network (such as the Internet), any type of bus (such as a PCIe bus), and the like.

[0029] Consuming system 118 can provide any combination of services that interact with processing engine 106 for any purpose(s). One type of consuming system corresponds to a search engine that uses processing engine 106 to perform semantic analysis of input queries using a CNN. Another type of consuming system corresponds to an interactive speech recognition engine that uses processing engine 106 to analyze a user's input speech in real time, and so on. Any consuming system can also forward arguments or other commands that govern the type of processing operations performed by processing engine 106.

[0030] Control logic 122 supplies instructions that govern the operation of hardware accelerator 104. It does so based on a computer program 124 written by a human programmer. In one embodiment, the instructions describe the operations associated with each layer or recurrent unit of a neural network or other type of analysis engine. In one implementation, control logic 122 represents a computing device including a central processing unit (CPU). Alternatively or additionally, control logic 122 represents a portion of one or more other components of computing system 102. For example, control logic 122 may represent a component within processing engine 106 and / or VMM 112, among others.

[0031] The instruction decoder 126 maps the instructions of the computer program 124 to more primitive control commands that control the operation of different corresponding parts of the hardware accelerator. For example, it is assumed that the tensor processing engine 108 includes multiple tile engines (not shown) that work in parallel to calculate the product of a vector and a matrix. It is further assumed that the computer program includes instructions that describe the calculation from a high-level perspective. Here, the instruction decoder 126 maps the high-level instructions into multiple primitive control commands, each of which controls a different tile engine. The instruction decoder 126 performs the mapping between high-level instructions and low-level commands based on specified rules and based on one or more setting parameters. For example, one or more setting parameters can instruct the tensor processing engine 108 to use a specified degree of parallelism when processing an input request.

[0032] like Figure 1 As shown in high-level form in FIG, the tensor processing engine 108 includes one or more tensor register files 128 for storing tensor information (e.g., vectors, matrices, etc.). The tensor processing engine 108 also includes one or more tensor operation hardware components (HCs) 130 for performing operations in a tensor-based manner. The SPU 110 includes one or more scalar register files 132 for storing scalar information. The SPU 110 also includes one or more scalar operation hardware components (HCs) 134 for performing operations in a scalar-based manner.

[0033] As will be described in more detail below, the tensor processing engine 108 and the SPU 110 use the same type of vector-based programming interface. This means that both the tensor processing engine 108 and the SPU 110 accept input data in vector form and supply output data in vector form. However, internally, the tensor processing engine 108 performs operations on input data fed to it in vector form, while the SPU 110 performs operations on input data fed to it in scalar form. To operate in this manner, the SPU 110 includes a vector-to-scalar interface that converts input data from vector form to scalar form. The SPU 110 then performs one or more scalar-based operations on the input data in scalar form. The SPU 110 then uses the scalar-to-vector interface to convert output data from scalar form to vector form.

[0034] With the above design, the SPU 110 can be programmably processed in the same manner as the tensor processing engine 108. For example, in one implementation, developers can use the same type of write instructions to write input vectors to the tensor processing engine 108 or the SPU 110. Similarly, developers can use the same type of read instructions to read vectors from the tensor processing engine 108 or the SPU 110. The following figures and accompanying explanations will clarify how the hardware accelerator 104 achieves the above results.

[0035] Figure 2 and Figure 3 Shown Figure 1 A more detailed description of the tensor processing engine 108 of the computing system 102 is provided below. These details are presented in the spirit of illustration rather than limitation. Other implementations may use a tensor processing engine with any other architecture than described below. Furthermore, other implementations may use a tensor processing engine that provides any other type of tensor processing flow than described below.

[0036] Figure 2 The tensor processing engine 108 shown in FIG. 1 includes multiple channels (channel 1, channel 2, . . . , channel C). Figure 1 Illustrative details of the first channel (channel 1) are shown. The other channels include the same components. Different implementations of the tensor processing engine 108 can assign work to different channels based on different environment-specific considerations. For example, assume that the computer program 124 includes at least one loop that involves repeating the same series of operations. The instruction decoder 126 can assign work associated with different iterations of the loop to different corresponding channels. This allows the tensor processing engine 108 to execute work associated with different iterations in parallel.

[0037] Note further that the tensor processing engine 108 can exploit other dimensions of parallelism when performing its tasks. For example, consider the case where the computer program 124 specifies that an input vector is to be multiplied by a matrix. The tensor processing engine 108 can decompose this task into subtasks of multiplying the input vector by each row of the matrix, and then perform these subtasks in parallel.

[0038] As shown in the figure, in this non-limiting implementation, the first (representative) channel provides a processing pipeline consisting of two or more tensor processing units (e.g., tensor processing unit 202, tensor processing unit 204, and tensor processing unit 206). In some other implementations, each channel may include fewer or additional tensor processing units. Each tensor processing unit includes one or more tensor register files (RFs) and one or more tensor operation hardware components (HCs). More specifically, the first tensor processing unit 202 includes one or more tensor RFs 208 and one or more tensor operation HCs 210, the second tensor processing unit 204 includes one or more tensor RFs 212 and one or more tensor operation HCs 214, and the third tensor processing unit 206 includes one or more tensor RFs 216 and one or more tensor operation HCs 218. As explained above, the tensor RFs store data in the form of tensors, for example, as matrices or vectors. The tensor operation HCs perform functions in the form of tensors. For example, a tensor operation HC can multiply a vector by a matrix. Another tensor operation HC may add the first vector to the second vector, and so on. Figure 3 Additional information (described below) is provided regarding a possible distribution of operations among the tensor processing units (202, 204, 206).

[0039] The first channel forms a pipeline in the sense that the output data of the first tensor processing unit 202 serves as input data fed into the second tensor processing unit 204. The output data from the second tensor processing unit 204 serves as input data fed into the third tensor processing unit 206. Each individual tensor processing unit also organizes its tensor operations HC in a pipeline. For example, consider the case where the second tensor processing unit 204 includes three tensor operations HC. The first tensor operation HC provides output data that serves as input data fed into the second tensor operation HC, and the second tensor operation HC provides output data that serves as input data fed into the third tensor operation HC. However, the computer program 124 can call only a subset of the tensor operations HC in any given tensor processing unit.

[0040] In one implementation, a developer writes the computer program 124 in a manner that takes advantage of the order in which the functions are executed in the pipeline described above. Figure 1). For example, assume that the tensor processing engine 108 includes a tensor operation HC that performs a vector multiplication, followed by a tensor operation HC that performs a vector addition or subtraction. Given this order, the developer can write a program in which the instruction identifying the multiplication operation precedes the instruction identifying the addition or subtraction operation. More generally, the developer writes a program to specify chains of operations to be performed by the processing engine 106. The developer creates these chains based on the order in which the tensor operations HC are arranged in the tensor processing engine 108. Further note that in many cases, the computer program 124 instructs the tensor processing engine 108 to repeat one or more chains of operations for a specified number of iterations. The tensor processing engine 108 can perform the operations by performing a pass through each channel. Figure 2 This mode of operation is achieved through multiple passes of the pipeline shown in .

[0041] In some cases, the tensor processing engine 108 may include at least one tensor operation HC that, as a default rule, receives input data from a previous ("upstream") tensor operation HC. This behavior eliminates or reduces the need for the computer program 124 to explicitly identify the source of the input data. It also eliminates the need to allocate dedicated registers to store the input data. In the above example, the tensor operation HC that performs an addition or subtraction operation receives a first operand from a previous tensor operation HC that performs a multiplication operation without the developer explicitly identifying the source of the input data in the computer program 124; it may receive a second operand from a vector-based RF specifically associated with the tensor operation HC that performs the addition or subtraction.

[0042] Figure 2 The tensor processing engine 108 is shown interacting with the SPU 110 via a vector memory manager (VMM) 112. More specifically, in this configuration, the VMM 112 treats the SPU 110 in the same manner as any memory, such as memory 114. That is, the VMM 112 can send write requests to the SPU 110, which operates to send input vectors to the SPU 110 for processing. The VMM 112 can also send read requests to the SPU 110, which are used to pull output vectors from the SPU 110. However, other implementations may integrate the SPU 110 into the computing system 102 using other corresponding connection strategies. For example, in another implementation, any tensor processing unit may include an instantiation of the SPU 110, optionally along with one or more tensor operation HCs. In another case, the VMM 112 itself incorporates the SPU 110.

[0043] In one implementation, Figure 2The tensor processing engine 108 processes matrices with native matrix sizes, such as N rows and N columns. Likewise, the tensor processing engine 108 processes vectors with a specified native vector size, such as 1×N elements. The tensor processing engine 108 can handle vectors with dimensions greater than N by treating two or more N-degree vectors as a single unit to be processed over two or more consecutive instruction cycles. The tensor processing engine 108 can handle vectors with dimensions less than N by padding the vector with dummy values ​​until it includes N dimensions. Likewise, the tensor processing engine 108 can process matrices with dimensions greater than or less than N×N using the above strategies. The SPU 110 processes vectors of non-native sizes in a manner that will be explained below.

[0044] Figure 3 A more specific (but still illustrative and non-limiting) example of a processing pipeline associated with a first channel is shown. Similarly, other channels may include the same series of tensor processing units, although Figure 3 No specific display.

[0045] In this example, the first tensor processing unit corresponds to a matrix-vector multiplier (MVM) processing unit 302. The MVM processing unit 302 performs the primary task of multiplying a vector by a matrix. The second tensor processing unit corresponds to a multifunction processing unit 304 that performs two or more different types of vector-based operations. The third tensor processing unit corresponds to another multifunction processing unit 306 that performs two or more different types of vector-based operations. Figure 3 The pipeline shown in FIG may include additional MVM processing units and / or multi-function processing units. As shown, the MVM processing unit 302 forwards its calculation results as input data to the first multi-function processing unit 304, and the first multi-function processing unit 304 forwards its calculation results as input data to the second multi-function processing unit 306.

[0046] More specifically, the MVM processing unit 302 includes a first tensor register file (Mvm_Inital_Vrf 308) that stores input vectors fed into one or more tile engines 310. A second tensor register file (Mvm_Matrix_Rf 312) includes input matrices fed into the tile engine(s) 310. The tile engine(s) 310 multiply the vectors retrieved from Mvm_Initial_Vrf 308 by the matrices retrieved from Mvm_Matrix_Rf 312. Each tile engine may include an architecture (not shown) that includes multiple dot product engines operating in multiple lanes. In one configuration, each dot product engine multiplies the input vector by a row of the input matrix. It performs this operation in parallel on an element-by-element basis and then sums the results across the dimensions of the input vector. Additionally, the MVM processing unit 302 may include one or more accumulators (not shown) that sum the output results of the multiple dot product engines.

[0047] The first multi-function processing unit 304 receives the initial vector from the initial register file (MFU_Intial_Vrf 314). It also receives the tensor output data provided by the MVM processing unit 302. The tensor arbiter 316 controls what input data is fed to the first multi-function processing unit 304. Figure 3For illustrative purposes only, the first multi-function processing unit 304 includes three tensor operations (HCs): an add-subtract (Add-Sub) HC 318 that adds two vectors together or subtracts a first vector from a second vector; a multiplier HC 320 that multiplies two vectors together; and an activation HC 322 that performs a neural network activation function (such as a hyperbolic tangent function, a sigmoid function, a ReLU function, etc.) on an input vector. A first register file (Add_Sub_Vrf 324) supplies a vector operand to the add-subtract HC 318. Another vector operand to the add-subtract HC 318 originates from a previous source, such as the MVM processing unit 302, the MFU_Initial_Vrf 314, etc. A second register file (Multiply_Vrf 326) supplies a vector operand to the multiplier HC 320. The other vector operand of the multiplier HC 318 originates from some previous source, such as the add / sub HC 318 (if used), the MFU_Initial_Vrf 314, etc. The input to the activate HC 322 originates from some previous source, such as the multiplier HC 320, the add / sub HC 318, the MFU_Initial_Vrf 314, etc. The crossbar switch (xBar) 328 operates as a conduit to pass information in a top-down manner, for example, by passing data from the MVM processing unit 302 or the MFU_Initial_Vrf 314 to the add / sub HC 318, by passing data from the add / sub HC 318 to the multiplier HC 320, and so on.

[0048] The second multi-function processing unit 306 receives input data from the first multi-function processing unit 304. It includes the same components as the first multi-function processing unit 304, including: Add_Sub HC 330, Multiplier HC 332, Activate HC 334, Add_Sub_Vrf 336, Multiply_Vrf 338, and xBar 340. Note that the computer program 124 is not required to utilize all of the tensor HCs in each multi-function unit. For example, the computer program 124 may not use the Add_Sub HC 318 in the first multi-function processing unit 304; therefore, the hardware component may be considered deactivated. On the other hand, the computer program 124 may not use the Multiplier HC 332 in the second multi-function processing unit 306; therefore, the hardware component may be considered deactivated. Further note that Figure 3 The types of tensor-based operations shown in are presented for purposes of illustration and not limitation; other implementations may use other types of tensor-based functions, and / or may omit Figure 3 One or more of the tensor-based functions shown in .

[0049] More generally, as noted above, with Figure 1 and Figure 2 Compared to those shown in FIG, SPU 110 can operate in association with other types of tensor processing engine architectures and tensor processing flows. In one variation of the above architecture, tensor processing engine 108 includes a single tensor processing unit with any combination of one or more tensor register files and tensor operation HCs. In another variation, tensor processing engine 108 does not necessarily arrange its tensor operations HCs in a series (i.e., in a pipeline). For example, tensor processing engine 108 can provide a bus or other type of routing logic that directly connects one or more tensor RFs to each tensor operation HC.

[0050] Figure 4 An implementation of the scalar processing element (SPE) 110 described above is shown. The main components of the SPE 110 include a vector-to-scalar interface 402, a scalar register file (SRF) 404, a scalar-to-vector interface 406, and a set of scalar operation hardware components (HC) 408. The vector-to-scalar interface 402 converts input vectors into scalar form. The SRF 404 stores the scalar(s) provided by the vector-to-scalar interface 402 at corresponding write addresses. The scalar-to-vector interface 406 converts one or more scalars read from the SRF 404 into vector form. And the scalar operation HC 408 performs corresponding scalar operations on the scalars read from the SRF 404; after each such scalar operation, the scalar operation HC writes the scalar result value to the SRF 404. The following explanation describes the operation of each of these components in turn.

[0051] Generally speaking, Figure 4 The various components of the SPU 110 are instructed to perform their operations based on the commands provided by the instruction decoder 126. In addition, the SPU 110 buffers data and input commands in various first-in, first-out (FIFO) buffers. Further, it is noted that each component of the SPU 110 can be visualized as comprising a controller combined with logic that executes instructions from the controller. For example, the scalar operations HC 408 can be conceptualized as a scalar controller that interacts with a logic component that executes instructions from the scalar controller.

[0052] Scalar to Vector Interface

[0053] Starting from the vector-to-scalar interface 402, this component uses different selectable types of hardware components (HCs) to convert an input vector into at least one scalar. For example, the sum reduction HC 410 uses tree reduction to sum the values associated with the elements in the input vector. The max reduction HC 412 uses tree reduction to determine the maximum value among the element values included in the input vector. The min HC (not shown) uses tree reduction to determine the minimum value among the element values included in the input vector. The Argmax HC (not shown) determines the index associated with the element having the maximum value in the input vector, for example, by using tree reduction to identify the maximum value and then recording the index associated with that maximum value. The Argmin HC (not shown) determines the index associated with the element having the minimum value in the input vector, and so on. These types of vector-to-scalar HCs are presented herein in the spirit of illustration rather than limitation; other implementations may use other HCs that convert a vector into at least one scalar. Further note that in most scenarios, the vector-to-scalar interface 402 uses a single hardware component to process the input vector. However, it can also apply two or more hardware components to produce two or more scalars, for example, by using the sum reduction HC 410 to determine the sum of the element values in the input vector and using the max reduction HC 412 to determine the maximum value element in the input vector.

[0054] In some input scenarios, the vector-to-scalar interface 402 can perform its work over multiple instruction cycles. In one such scenario, the vector-to-scalar interface 402 processes an input vector having a width w, where the width w is less than the native vector size N of the vectors processed by the tensor processing engine 108. In another case, the vector-to-scalar interface 402 receives a set of input vectors, each having a size of N, and control information indicating that the set of native vectors is to be treated as a single unit. In these cases, the vector-to-scalar interface 402 performs its scalar extraction operation in two or more stages.

[0055] To address the first-mentioned input scenario, the vector-to-scalar interface 402 can include an optional input selector component 414 that operates to select a portion of the input vector. For example, assume the input vector includes N dimensions, but the SRF 404 is designed to accept a set of w vector elements, where w < N. In this case, the input selector component 508 can select the first w elements of the input vector in a first pass and the remaining elements of the input vector in one or more subsequent passes.

[0056] Each hardware component of the vector-to-scalar interface 402 can also adapt its behavior to handle piecemeal input vector data. More specifically, consider the following example: the input vector has a size of N (e.g., 300 elements), and the width w of the sum-reduce HC 410 is 100 elements—that is, one-third of N. The sum-reduce HC 410 handles this scenario by successively processing three component input vectors, each of which has a size of one-third of N (i.e., 100 elements). That is, the sum-reduce HC 410 can process the first component input vector to produce a first sum. It can then store the first sum internally in the sum-reduce HC 410. For example, by adding the sum of the element values ​​in the second component input vector to the first sum, the sum-reduce HC 410 can process the second component input vector together with the first sum to produce a second sum. The sum-reduce HC then stores the second sum internally. Similarly, the Sum Reduce HC 410 can process the third component input vector together with the second sum to produce a third or final sum, for example by adding the sum of the element values ​​in the third component input vector to the second sum. The Sum Reduce HC 410 then outputs the third or final sum.

[0057] In an alternative implementation, sum-reduce HC 410 may store each intermediate sum it produces in scalar register file (SRF) 404, that is, rather than storing the intermediate sums internally in sum-reduce HC 410 itself. After processing each new component input vector, sum-reduce HC 410 may perform a read-write-modify operation by: (1) pulling the most recent intermediate sum from SRF 404; (2) adding the value associated with the current component input vector to the intermediate sum; and (3) writing the updated sum back to the same address location of SRF 404.

[0058] In some implementations, computing system 102 can initialize storage locations in SRF 404 prior to a multi-pass reduce operation. For example, at the start of a multi-pass sum-reduce operation, control logic 122 can store a value of 0 in the location where sum-reduce HC 410 will store its sum. Alternatively, sum-reduce HC 410 can be configured to suppress read operations for the initial read-modify-write operation. Both examples prevent the values ​​initially stored in SRF 404 from corrupting the sum at the start of the sum-reduce operation (if any).

[0059] Different input scenarios can make the above-mentioned operation mode further complicated. For example, in some cases, vector to scalar interface 402 can receive the input vector filled with dummy values. These dummy values ​​may destroy the determination made by vector to scalar interface 402. The possibility of destruction depends on which hardware component vector to scalar interface 402 uses to produce scalar and the type of dummy value added to the input vector. For example, consider the situation that vector to scalar interface 402 uses maximum reduction HC 412 to determine the maximum value in the input vector. Suppose that all the valid elements in the valid elements of the input vector have negative values, and each dummy value has value 0. Maximum reduction HC 412 will mistakenly determine that the maximum value in the input vector is zero.

[0060] To address this issue, the Max Reduce HC 412 can receive an input vector and an input parameter specifying the logical length of the input vector ("logical length"). For example, assume the native vector length is 300, but only the first 200 elements of the input vector correspond to valid values. The remaining 100 elements correspond to dummy values. In addition to receiving the input vector, the Max Reduce HC 412 also receives a logical length value identifying the logical length of the input vector (corresponding to 200 elements in this case). The Max Reduce HC 412 uses the logical length value to determine the maximum value among the first 200 elements of the input vector, ignoring the dummy values ​​that fill the last 100 elements of the input vector.

[0061] The argmax HC (not shown) differs from the sum-reduce HC 410 and the maximum-reduce HC 412 in that it must propagate multiple values ​​over multiple corresponding instruction cycles (where the argmax HC processes multiple component input vectors). These values ​​include the maximum value encountered so far and the index value of the element associated with the maximum value. The argmax HC stores these two pieces of information internally or in different corresponding storage locations of the SRF 404.

[0062] Different implementations may update the index values ​​across multiple iterations in different corresponding manners. Consider the specific case where the input vector has a size of 300 and the width w of the argmax HC is 100 elements. In a merely illustrative example, the argmax HC stores a base index value that identifies the segment of the input vector currently being processed. That is, when the argmax HC processes the first one hundred elements of the input vector (elements 1 to 100), the base index value is 0; when the argmax HC processes the next one hundred elements of the input vector (elements 101 to 200), the base index value is 100; and when the argmax HC processes the last one hundred elements of the input vector (elements 201 to 300), the base index value is 200. The argmax HC can update the index by: (1) determining whether the segment of the input vector under consideration has a value greater than any previously encountered 300-element input vector; (2) if so, storing the updated maximum value; (3) determining an offset index value that indicates the location where the maximum value occurs in the currently considered vector segment; (4) adding the offset index value to the current base index value (where the base index value identifies the currently considered vector segment) to provide an updated index value; and (5) storing the updated index value.

[0063] Note that the operation of the vector-to-scalar interface 402 has been described in the context of specific types of hardware components, such as the sum-reduce HC 410, the max-reduce HC 412, and the argmax HC (not shown). However, the same principles apply to any other hardware components used by the vector-to-scalar interface 402.

[0064] Scalar register file

[0065] The scalar register file (SRF) 404 includes a set of write ports 416 with corresponding write addresses, and a set of read ports 418 with corresponding read addresses. The vector-to-scalar interface 402 writes scalars to the write ports 416. The scalar-to-vector interface 406 retrieves one or more stored scalars from the SRF 404 from the read ports 418.

[0066] In one implementation, SRF 404 represents a global resource shared by processing pipelines in channels of computing system 102. That is, each tensor processing unit interacts with the same SRF 404. However, SPU 102 may include multiple instantiations of vector-to-scalar interface 402 and multiple instantiations of scalar-to-vector interface 406. For example, SPU 110 may assign a separate instance of vector-to-scalar interface 402 and a separate instance of scalar-to-vector interface 406 to each channel, and / or to each tensor processing unit within each channel, and / or so on. This means that vector-to-scalar interfaces associated with different components of tensor processing engine 108 operate in parallel. Each instance of vector-to-scalar interface 402 stores its results in a designated location in global SRF 404. Similarly, scalar-to-vector interfaces associated with different components of tensor processing engine 108 operate in parallel. Each instance of scalar-to-vector interface 406 pulls scalar values ​​from a specified location in global SRF 404.

[0067] Scalar operation hardware components

[0068] The collection of scalar operation hardware components (HCs) 408 performs corresponding scalar-based operations. More specifically, each scalar operation HC retrieves one or more scalars from the SRF 404 via a read port 418, performs some scalar-based function on the retrieved scalar(s) to generate one or more output scalars, and then writes the output scalar(s) back to the SRF 404 via a write port 416. This constitutes a read-modify-write operation. For example, a scalar reciprocal HC 420 reads a scalar x at a specified memory location in the SRF 404, generates the reciprocal of the scalar (1 / x), and then writes the output result scalar to a specified memory location in the SRF 404. A scalar square root HC 422 reads the scalar x into a specified memory location in the SRF 404, generates the square root of the scalar x, and then writes the output result scalar to a specified memory location in the SRF 404. And then write the output result scalar to the specified memory address.

[0069] The above-described scalar operations are presented for purposes of illustration and not limitation; other implementations may include any subset of scalar operations HC. For example, another implementation may provide a scalar operation HC that performs a multiplication operation, a scalar operation HC that performs an addition operation, and so on. Note that these types of scalar operations HC receive two input operands, while the scalar reciprocal operation HC 420 and the scalar square root HC 422 each receive one input operand. In general, a scalar operation HC may receive any number of input operands and provide any number of output results.

[0070] Yet another class of scalar operations HC performs shift operations by reading a set of scalar values ​​from first corresponding locations in SRF 404 and storing them in second corresponding storage locations in SRF 404. Another class of scalar operations HC performs operations in a conditional manner, e.g., based on one or more input parameter values ​​supplied to SPU 110. Still another class of scalar operations HC performs masking operations by selectively reading values ​​within a larger set of values, and so on.

[0071] Note that the specific computer program 124 may call any scalar operation HC in the scalar operation HC or may not call the scalar operation HC. In the latter case, the computing system 102 uses the hardware components of the vector-to-scalar interface 402 to perform a certain reduction operation on the input vector to produce at least one scalar; then, the computing system 102 uses a certain hardware component of the scalar-to-vector interface 406 to produce an output vector based on the scalar(s) provided by the vector-to-scalar interface 402 without any further transformation provided by the scalar operation HC.

[0072] Scalar to Vector Interface

[0073] The scalar-to-vector interface 406 includes one or more hardware components (HCs) that convert one or more scalars read from the SRF 404 into an output vector (or multiple component output vectors). For example, the broadcast HC 424 reads a single scalar from the SRF 404 and then replicates the scalar across all elements of the N-dimensional output vector.

[0074] Vector Assembly HC 426 reads multiple scalars stored at different corresponding memory locations from SRF 404. It then assembles these scalars into one or more output vectors. In one implementation, SPU 110 generates these multiple scalar values ​​in multiple corresponding scalar operations, executed over multiple instruction cycles. For example, SPU 110 may determine the maximum value for each column of a matrix in multiple consecutive operations and store each maximum value at a specified location in SRF 404. Vector Assembly HC 426 may read these maximum values ​​from SRF 404 and assemble them into an output vector; the output vector represents the maximum value across different columns of the matrix.

[0075] The vector assembly HC 426 can also insert scalars at appropriate locations in the composite output vector based on index information stored in the SRF 404. In other words, the scalar-to-vector interface 406 includes logic that complements the logic used by the vector-to-scalar interface 402 to process multiple component input vectors. For example, the scalar-to-vector interface 406 can determine the placement of element values ​​in an output vector assembled from multiple segments based on a base index value (which identifies the segment under consideration) and an offset index value (which identifies the position of the element within the segment under consideration).

[0076] Assume that the scalar-to-vector interface 406 has a width w that is less than the native vector length N. In this case, the output selector component 428 can concatenate multiple component output vectors of size w to generate a composite output vector of size N.

[0077] SPU programming interface

[0078] As explained above, in one non-limiting implementation, the computing system 102 can provide an input vector to the SPU 110 by executing a write instruction included in the computer program 124. The first operand of the write instruction identifies a specific type of hardware component in the vector-to-scalar interface 402 that will be used to convert the input vector into a scalar. For example, the identifier "Spu_Max_Reduce" instructs the vector-to-scalar interface 402 to process the input vector using the maximum reduce HC 412—for example, by determining the maximum element value in the input vector. The second operand in the write instruction identifies the memory location in the SRF 404 where the maximum value is to be stored, such as "spu_accum".

[0079] The computing system 102 can retrieve the output vector from the SPU 110 by executing a read instruction included in the computer program 124. The first operand of the read instruction identifies a specific type of hardware component in the scalar-to-vector interface 406 that will be used to convert one or more scalars into an output vector. For example, the identifier "Spu_Broadcast" instructs the scalar-to-vector interface 406 to use the broadcast HC 424 to retrieve a scalar from the SRF 404 and copy it across the dimensions of the output vector. The second operand of the read instruction specifies the location in the SRF 404 where the scalar is to be stored.

[0080] Optional vector-based interface to the SPU

[0081] and Figure 4 Compared to the example, Figure 54. An alternative implementation of various portions of SPU 110 is shown. In this case, vector-to-scalar interface 402 includes mechanisms for simultaneously reading entire input vectors into corresponding write addresses 502 of SRF 404. Each individual write address can be said to store a scalar corresponding to one of the element values ​​of the input vector. SPU 110 can then perform scalar operations on any scalar associated with the individual write address. Scalar-to-vector interface 406 also includes mechanisms for reading entire input vectors from corresponding read addresses 504 of SRF 404.

[0082] More specifically, the vector-to-scalar interface 402 may include a set of multiplexers 506 that control whether the SRF 404 receives a group of scalars or individual scalars. For example, when the computing system 102 instructs the SRF 404 to simultaneously receive and store an entire input vector, the SRF 404 may receive a group of scalars. The scalars correspond to the element values ​​of the input vector. For example, when an individual scalar operation (HC) instructs the SRF 404 to store a calculated new scalar, such as by storing the square root of the input value x, the SRF 404 may receive a single scalar.

[0083] The scalar to vector interface 402 may include a multiplexer 508 that controls whether the SRF 404 supplies a single output scalar or a group of output scalars simultaneously. In the former case, the SRF 404 may read out a single scalar to the broadcast HC 424 ( Figure 4 ), or as input to a scalar operation HC. The broadcast HC 424 can then replicate the single scalar to form an output vector. In the latter case, the SRF 404 reads out a group of scalars simultaneously.

[0084] Figure 5 Alternative implementations of are suitable for specific kinds of scalar operations. For example, computing system 102 may use Figure 5 The functionality of HC receives a complete input vector, which the computing system 102 stores at a corresponding write location in SRF 404. The scalar operation HC can then operate as a selector to read a specific scalar within the input vector, for example, by specifying the index value of the desired element of the input vector. In one case, the SPU 110 can determine the index value in a previous argmax reduction operation.

[0085] The computing system 102 described above has various technical advantages. For example, the computing system 102 can seamlessly integrate scalar operations into a processing data flow that is otherwise directed toward tensor-based operations. The computing system 102 achieves this result by using the same vector-based interface to write data, regardless of whether the data is destined for a tensor processing unit or an SPU 110. It also uses the same vector-based interface to read data, regardless of whether the data originates from a tensor processing unit or an SPU 110. These features also facilitate the task of writing computer programs. Developers can use the same type of programming instructions to control the tensor processing engine 108 and the SPU 110.

[0086] The efficiency of computing system 102 can be measured by comparing it to two alternatives. In the first alternative, the computing system can express scalars in vector form, for example, by adding dummy values ​​to the scalars to produce the vectors. The computing system then performs operations based on the vectors. However, this alternative solution is inefficient because it involves adding wasteful padding to the scalars to produce the vectors. This alternative solution may also rely on slow and cumbersome techniques for extracting scalars from vectors, for example, involving iteratively shifting and / or masking the vectors. Figure 1 and Figure 2 The present computing system 102 shown in addresses these shortcomings by providing a technique for operating on scalars directly in scalar form within a data flow that would otherwise be dedicated to tensor-based computations.

[0087] In a second alternative, the computing system can use a general-purpose CPU to handle scalar operations. This alternative solution requires the tensor processing engine to transfer data to shared memory. The CPU retrieves the data from the shared memory and operates on it. This alternative solution is not very efficient because it requires the inclusion of a CPU to perform what is typically (proportionally to the large number of processing operations occurring in tensor space) only a small number of scalar operations. In other words, this alternative wastes most of the resources of the general-purpose CPU. This alternative solution also uses slow and cumbersome techniques to transfer information between the tensor processing engine and the general-purpose CPU. In contrast, Figure 1 and Figure 2 The present computing system 102 shown in FIG relies on a dedicated SPU 110 with a scalar operation HC. The SPU 110 includes logic specifically configured to operate with the tensor processing engine 108 and therefore does not include the same degree of resource waste as a general-purpose CPU. The tensor processing engine 108 can also interact with the SPU 110 with relatively low latency because the SPU 110 operates as a native element of vector-based data flow.

[0088] Example 1: Softmax

[0089] Figure 6 Shown Figure 1 and Figure 2 An example of how the computing system 102 can be used to perform a softmax (normalized exponential function) operation. Mathematically, the softmax operation operates on the input vector x as follows:

[0090]

[0091] In other words, the softmax operation determines the maximum value among the elements of the vector x (i.e., max(x)), and then subtracts this maximum value from each element value of the input vector x to produce a set of per-element difference values ​​(x i -max(x)). It then raises each difference value of e to a power to produce a set of exponential differences. It then normalizes each exponential difference by the sum of the exponential differences (corresponding to the denominator of the above equation).

[0092] Assume that computer program 124 uses the following code to express software operations:

[0093] 1.v_wr(Initial_Vrf,tmp)

[0094]

[0095]

[0096] 4.v_wr(Add_Sub_Vrf,vec_max)

[0097] 5.v_rd(Initial_Vrf,tmp)

[0098] 6.vv_sub(Add_Sub_Vrf,vec_max)

[0099] 7.vv_exp()

[0100] 8.v_wr(Initial_Vrf,tmp2)

[0101]

[0102]

[0103]

[0104] 12.v_wr(Multiply_Vrf,denom)

[0105] 13.v_rd(Initial_Vrf,tmp2)

[0106] 14.vv_mul(denom)

[0107] 15.v_wr(<dest mem id> ,result)

[0108] Assumptions Figure 6 The top portion of corresponds to components provided by one or more tensor processing units of the tensor processing engine 108, such as one or more multi-function units. Figure 6 The bottom portion of corresponds to the components provided by SPU 110. Instructions executed by SPU 110 are presented in bold in the above computer program; the remaining operations are performed by tensor processing engine 108.

[0109] In operation (1), the computing system 102 executes a write command that causes the tensor processing engine 108 to write the initial vector representing the input vector x to the vector register file named Initial_Vrf 602 at the memory address associated with "tmp". In operation (2), the computing system 102 performs a write to the max reduce HC 604 (associated with the memory ID "Spu_Max_Reduce"). This causes the max reduce HC 604 to calculate the maximum value of x and store the scalar at the location associated with the memory index "spu_accum" in the scalar register file (SRF) 606. In operation (3), the computing system 102 performs a read operation on the broadcast HC 608, which causes the broadcast HC 608 to retrieve the scalar stored at "spu_accum" and copy the scalar across the dimensions of the output vector. In operation (4), computing system 102 executes a write command that stores the copied maximum value at a memory location associated with “vec_max” in vector register file 610 called “Add-Sub_Vrf”.

[0110] In operation (5), the computing system 102 executes a read command to read the value x from the input register file Initial_Vrf 602. In operation (6), the computing system 102 uses subtraction HC 612 to subtract the vector vec_max in Add_Sub_Vrf 610 from the vector x in the vector register file Initial_Vrf 602. In this case, the subtraction command implicitly identifies the vector x in the vector register file Initial_Vrf 602 as its first operand. In operation (7), the computing system 102 uses exponent HC 614 to calculate the value of (x) i -max(x)) to the power. In operation (8), computing system 102 stores the result of operation (7) in vector register file 602 at the location identified by "tmp2".

[0111] In operation (9), computing system 102 uses sum-reduce HC 616 to reduce the value of each element in x by The denominator of the above equation is calculated by summing the values ​​of . The computing system 102 instructs the SPU 110 to store the scalar sum at the memory index identified by "spu_accum2" in the SRF 606. In operation (10), the computing system 102 instructs the scalar reciprocal HC 618 to generate the reciprocal of the sum generated in operation (9). It performs this task by pulling the sum from the SRF 606 at the location identified by "spu_accum2", calculating the reciprocal, and storing the reciprocal at the memory location identified by "spu_recip" in the SRF 606. In operation (11), the computing system 102 instructs the broadcast HC 608 to generate the output vector by copying the reciprocal value stored at "spu_recip".

[0112] In operation (12), the computing system 102 instructs the vector register file "Multiply_Vrf" 620 to store the reciprocal value at the location identified by "denom". More specifically, the computing system 102 performs its operation by making at least two passes through the operation pipeline associated with at least one multi-function unit. Assume that the computing system 102 accesses the Multiply_Vrf vector register file 620 in the second pass through the pipeline. In operation (13), the computing system 102 reads the vector at "tmp2" previously stored in the Initial_Vrf vector register file 602. In operation (14), the computing system 102 instructs Multiply HC 622 to multiply the vector stored at "tmp2" by the vector in the Multiply_Vrf vector register file 620 (identified by "denom"). In operation (15), the computing system 102 writes the output result generated in operation (14) to the specified destination location.

[0113] Example 2: Layer Normalization

[0114] In a second example, computing system 102 performs operations to normalize the values ​​in a layer of a neural network. It performs these operations by first calculating the mean (μ) and bias (σ) of the values ​​in vector x according to the following equations:

[0115]

[0116] The computing system 102 uses the mean and deviation metrics to Normalize x by some function of , where b is a bias factor. Note that H refers to the number of elements in x.

[0117] The following explanation will explain Figure 4SPU 110 Combination Figure 2 One non-limiting way that the tensor processing engine 108 may perform a layer normalization operation is as follows. The computing system 102 first generates the sum using the sum-reduce HC 410. It then uses broadcast HC 424 to generate an output vector representing the sum replicated across its elements. Computing system 102 then uses multiplier HC provided by the tensor processing unit to multiply the inverse of H by the sum of the element values ​​in x to produce the mean μ replicated across the vector. (It is assumed that the inverse of H is precomputed and stored as a known parameter value, rather than being calculated in real time.)

[0118] The computing system 102 can then use the subtraction HC of the tensor processing unit to subtract the mean μ (expressed in vector form) from each value of x to produce a plurality of difference values ​​(x i -μ). The computing system 102 then uses the multiplier HC of the tensor processing unit to calculate (x i -μ) squared, for example, by expressing (x i The computing system 102 then calls the Sum Reduce HC 410 to multiply the two vectors of (x i -μ) 2 The computing system 102 then uses the multiplier HC provided by the tensor processing unit to multiply the inverse of H (which is a pre-computed value) by the expression The computing system 102 then calls the addition HC in the tensor processing unit to add the value of ε to It is again expressed in the form of a vector.

[0119] Computing system 102 then writes the result of the above calculation to SPU 110 via Max Reduce HC 412. This effectively converts the result of the above calculation from vector form to scalar form and stores it in SRF 404. System 102 then calls Scalar Square Root HC 422 to calculate the square root of the scalar thus generated. This produces the deviation value σ. Computing system 102 then uses Scalar Inverse HC 420 to calculate the inverse of σ. It then uses Broadcast HC 424 to provide an output vector with a replicated value of 1 / σ.

[0120] Finally, the computing system 102 uses the multiplier HC of the tensor processing unit to multiply 1 / σ by (x i -μ) to generate a vector product, and the adder HC of the tensor processing unit is used to add the bias factor b to the vector product.

[0121] The above allocation of operations to the tensor processing engine 108 and the SPU 110 is set forth in an illustrative, non-limiting manner. As will be appreciated, in some cases, developers have some discretion in determining whether it is more efficient for the tensor processing unit to perform an operation in a tensor-based form or for the SPU 110 to perform the operation in a scalar form. For example, both the tensor processing engine 108 and the SPU 110 may include multipliers HC. The developer may decide whether it is more efficient to perform a multiplication operation in a tensor-based form or in a scalar form. The developer may make this determination based on the process and / or other considerations associated with the particular DNN being implemented.

[0122] B. Illustrative Process

[0123] Figure 7 A process 702 is shown that illustrates the operation of computing system 102 in Section A in flowchart form. Since the basic principles of computing system 102 operation have already been described in Section A, certain operations will be discussed in general terms in this section. As noted in the introduction to the detailed description, the flowcharts are presented as a series of operations to be performed in a specific order. However, the order of these operations is representative and may be altered in any manner.

[0124] In block 704, during execution of a set of instructions that perform operations associated with a neural network or other analytical task, computing system 102 uses tensor processing engine 108 to convert at least one instance of input tensor information into at least one instance of output tensor information. In block 706, during execution of the set of instructions, computing system 102 uses scalar processing unit 110 to: (706.1) provide one or more scalars based on an input vector received from tensor processing engine 108; (706.2) store the one or more scalars in scalar register file 404; (706.3) retrieve at least one input scalar from scalar register file 404; (706.4) perform a scalar processing function to generate at least one output scalar based on at least one input scalar; (706.5) store the at least one output scalar in scalar register file 404; and (706.6) provide an output vector based on the one or more scalars stored in scalar register file 404. The method uses the same vector-based programming interface to interact with the tensor processing engine 108 and the scalar processing unit 110 .

[0125] C. Representative computing hardware

[0126] Figure 81. Computing hardware 802 is shown, which can be used to implement any aspect of the mechanisms described in the above figures. Computing hardware 802 may include one or more hardware processors 804 that execute instructions stored in memory. For example, hardware processor(s) 804 may include one or more central processing units (CPUs). According to one implementation, computing system 102 may use hardware processor(s) 804 to implement at least control logic 122 and instruction decoder 126.

[0127] More generally, the hardware processor(s) 804 interact with any information 806 stored in a computer-readable storage medium 808. Information 806 may include instructions (e.g., computer program 124), data, settings, and the like. Any instance of the computer-readable storage medium 808 may use any technology to store and retrieve information. In one embodiment, the computer-readable storage medium 808 represents a hardware memory unit (such as random access memory (RAM)) for storing transient information during execution of a program by the computing hardware 802. Alternatively or additionally, the computer-readable storage medium 808 represents a hardware storage unit (such as a hard disk) for retaining / archiving information on a more permanent basis. Furthermore, any instance of the computer-readable storage medium 808 may represent a fixed or removable unit of the computing hardware 802.

[0128] The computing hardware 802 may rely on an additional hardware logic unit 810 to implement at least some aspects of the hardware accelerator 104. In one embodiment, the hardware logic unit(s) 810 may include a fixed configuration of hardware logic gates, e.g., they are created and set at the time of manufacture and cannot be changed thereafter. Alternatively or additionally, the additional hardware logic unit(s) 810 may include a collection of programmable hardware logic gates that can be configured to perform different application-specific tasks. The latter class of devices includes, but is not limited to, programmable array logic devices (PALs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), and the like.

[0129] Figure 8Hardware logic circuitry 812 is generally indicated to include any combination of hardware processor(s) 804, computer-readable storage media 808, and / or other hardware logic unit(s) 810. That is, computing hardware 802 may employ any combination of hardware processor(s) 804 that execute machine-readable instructions provided in computer-readable storage media 808 and / or one or more other hardware logic units 810 that perform operations using fixed and / or programmable sets of hardware logic gates. More generally, hardware logic circuitry 812 corresponds to one or more hardware logic units of any type that perform operations based on logic stored in and / or otherwise embodied in the hardware logic unit(s).

[0130] The computing hardware 802 may also include one or more network interfaces 814 for exchanging data with other devices via one or more communication conduits 816. The communication conduit(s) 816 may be implemented in any manner, such as through a local computer network provided by a data center, a wide-area computer network (e.g., the Internet), a point-to-point connection, or any combination thereof. The communication conduit(s) 816 may include any combination of hardwired links, wireless links, routers, gateway functionality, name servers, and the like, governed by any protocol or combination of protocols. Finally, one or more communication buses 818 communicatively couple the aforementioned elements together.

[0131] Figure 8 The computing hardware 802 is shown as being composed of a discrete collection of separate units. In some cases, the collection of units may correspond to discrete hardware units provided in a computing device chassis having any form factor. In other cases, the computing hardware 802 may include hardware logic units that integrate Figure 1 For example, computing hardware 802 may include a system on a chip (SoC or SOC), combined with Figure 8 The functions of two or more units shown in the integrated circuit correspond.

[0132] In one environment, a data center provides, for example, Figure 8 8. One or more instances of computing hardware 802 are shown in . Network interface(s) 816 receive requests from external entities and forward responses to those requests to the external entities.

[0133] The following summary provides a non-exhaustive set of illustrative aspects of the technology described herein.

[0134] According to a first aspect, a computing system for performing operations using a hardware accelerator is described. The hardware accelerator includes a tensor processing engine and a scalar processing unit. The tensor processing engine includes: at least one tensor register file for storing tensors; and two or more tensor operation hardware components for performing corresponding tensor processing functions based on the tensors stored in the at least one tensor register file. The scalar processing unit includes: a vector-to-scalar interface for providing one or more scalars based on an input vector received from the tensor processing engine; a scalar register file for storing each scalar provided by the vector-to-scalar interface; one or more scalar operation hardware components for performing corresponding scalar processing functions, each scalar operation hardware component retrieving at least one input scalar from the scalar register file, performing the scalar processing function to generate at least one output scalar based on the one input scalar, and storing the at least one output scalar in a scalar register file; and a scalar-to-vector interface for providing output vectors based on the one or more scalars stored in the scalar register file. The hardware accelerator also includes control logic for providing instructions for governing the operation of the tensor processing engine and the scalar processing unit. The hardware accelerator uses the same vector-based programming interface to interact with the tensor processing engine and the scalar processing unit.

[0135] According to a second aspect, the instructions express operations associated with a deep neural network.

[0136] According to a third aspect, a tensor processing engine includes multiple lanes that perform tensor-based operations in parallel.

[0137] According to a fourth aspect, a tensor processing engine includes a processing pipeline composed of two or more tensor operation hardware components operating in series.

[0138] According to a fifth aspect, a vector to scalar interface comprises at least one hardware component that generates a sum of element values ​​present in an input vector.

[0139] According to a sixth aspect, a vector to scalar interface comprises at least one hardware component that identifies a maximum or minimum scalar by selecting among element values ​​in an input vector.

[0140] According to a seventh aspect, a vector to scalar interface comprises functionality for simultaneously receiving an input vector and for storing scalars associated with element values ​​in the input vector.

[0141] According to an eighth aspect, a scalar to vector interface includes a broadcast hardware component for replicating a scalar stored in a scalar register file across elements of a vector at output.

[0142] According to a ninth aspect, a scalar-to-vector interface comprises a vector packing component for packing an output vector based on a plurality of scalars stored in a scalar register file.

[0143] According to a tenth aspect, a hardware accelerator includes multiple channels for executing operations in parallel, wherein a scalar register file corresponds to a global resource used by all channels in the channel, a vector-to-scalar interface is duplicated for each channel of the hardware accelerator, and a scalar-to-vector interface is duplicated for each channel of the hardware accelerator.

[0144] According to an eleventh aspect, the hardware accelerator further comprises a vector memory manager, which provides an interface between the hardware accelerator and the memory, wherein the scalar processing unit interacts with the tensor processing engine via the vector memory manager.

[0145] According to a twelfth aspect, the instruction uses a write command to provide an input vector to a scalar processing unit.

[0146] According to a thirteenth aspect, the instructions retrieve the output vector from the scalar processing unit using a read command.

[0147] According to a fourteenth aspect, a method for performing an operation using a hardware accelerator is described. The method includes: using a tensor processing engine to convert at least one instance of input tensor information into at least one instance of output tensor information during execution of a set of instructions, the set of instructions performing operations associated with a neural network or other analytical tasks. The method also includes: using a scalar processing unit during execution of the set of instructions to: provide one or more scalars based on an input vector received from the tensor processing engine; store the one or more scalars in a scalar register file; retrieve at least one input scalar from the scalar register file; perform a scalar processing function to generate at least one output scalar based on at least one input scalar; store the at least one output scalar in a scalar register file; and provide an output vector based on the one or more scalars stored in the scalar register file. The method uses the same vector-based programming interface to interact with the tensor processing engine and the scalar processing unit.

[0148] According to a fifteenth aspect related to the fourteenth aspect, the scalar processing unit provides one or more scalars based on the received input vector by reducing the input vector to a single scalar.

[0149] According to a sixteenth aspect related to the fourteenth aspect, the scalar processing unit provides the output vector by replicating a scalar stored in the scalar register file across elements of the output vector.

[0150] According to a seventeenth aspect related to the fourteenth aspect, the instructions provide the input vector to the scalar processing unit using a write command, and the instructions retrieve the output vector from the scalar processing unit using a read command.

[0151] According to the eighteenth aspect, a hardware accelerator is described, which includes a tensor processing engine and a scalar processing unit. The tensor processing engine includes: at least one tensor register file for storing tensors; and two or more tensor operation hardware components for performing corresponding tensor processing functions based on the tensors stored in the at least one tensor register file. The scalar processing unit includes: a vector-to-scalar interface for providing one or more scalars based on an input vector received from the tensor processing engine; a scalar register file for storing each scalar provided by the vector-to-scalar interface; one or more scalar operation hardware components for performing corresponding scalar processing functions, each scalar operation hardware component being configured to retrieve at least one input scalar from the scalar register file, perform a scalar processing function to generate at least one output scalar based on the at least one input scalar, and store the at least one output scalar in a scalar register file; and a scalar-to-vector interface for providing an output vector based on the one or more scalars stored in the scalar register file. The hardware accelerator uses the same vector-based programming interface to interact with the tensor processing engine and the scalar processing unit.

[0152] According to a nineteenth aspect related to the eighteenth aspect, the hardware accelerator further includes a vector memory manager, which provides an interface between the hardware accelerator and the memory, wherein the scalar processing unit interacts with the tensor processing engine via the vector memory manager.

[0153] According to a twentieth aspect related to the eighteenth aspect, the hardware accelerator includes a plurality of channels for executing operations in parallel, a scalar register file and global resources used by all channels in the channels, a vector-to-scalar interface is replicated for each channel of the hardware accelerator, and a scalar-to-vector interface is replicated for each channel of the hardware accelerator.

[0154] The twenty-first aspect corresponds to any combination (eg, any logically consistent enumeration or subset) of the above-mentioned first to twentieth aspects.

[0155] The twenty-second aspect corresponds to any method counterpart, device counterpart, system counterpart, apparatus plus function counterpart, computer-readable storage medium counterpart, data structure counterpart, product counterpart, graphical user interface presentation counterpart, etc. associated with the first to twenty-first aspects.

[0156] Finally, the specification may have presented various concepts in the context of an illustrative challenge or problem. This interpretation is not intended to imply that others have understood and / or addressed the challenge or problem in the manner specified herein. Furthermore, this interpretation is not intended to imply that the subject matter recited in the claims is limited to solving only the identified challenges or problems; that is, the claimed subject matter may be applied in the context of challenges or problems other than those described herein.

[0157] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A computing system for performing an operation, comprising: Hardware accelerators that implement vector-based data flow and include tensor processing engines and scalar processing units, The tensor processing engine includes: at least one tensor register file for storing tensors; and two or more tensor operation hardware components for performing respective tensor processing functions based on tensors stored in the at least one tensor register file; The scalar processing unit is configured to operate as a native element in the vector-based data flow and comprises: a vector-to-scalar interface for providing one or more scalars based on an input vector received from the tensor processing engine; a scalar register file for storing each scalar provided by the vector-to-scalar interface; a set of multiplexers for controlling whether the scalar register file receives a group of scalars or a single scalar; One or more scalar operation hardware components for performing corresponding scalar processing functions, Each scalar operation hardware component retrieves at least one input scalar from the scalar register file, performs a scalar processing function to generate at least one output scalar based on the at least one input scalar, and stores the at least one output scalar in the scalar register file; and a scalar to vector interface for providing an output vector based on one or more scalars stored in the scalar register file; and control logic for providing a program for controlling the operation of the tensor processing engine and the scalar processing unit, the computing system invoking the scalar processing unit to perform at least one scalar operation specified by the program within the vector-based data flow; The hardware accelerator uses the same kind of vector-based programming interface instructions to interact with the tensor processing engine and the scalar processing unit, the vector-based programming interface instructions including vector write instructions for providing input vectors to both the tensor processing engine and the scalar processing unit, and vector read instructions for retrieving output vectors from both the tensor processing engine and the scalar processing unit. The hardware accelerator and the control logic are implemented using hardware logic circuitry, which in turn includes one or more hardware processors in combination with memory, and / or one or more other hardware logic units that perform operations using fixed and / or programmable sets of hardware logic gates.

2. The computing system of claim 1 , wherein the program expresses operations associated with a deep neural network. 3 . The computing system of claim 1 , wherein the tensor processing engine comprises a plurality of lanes that perform vector-based operations in parallel. 4 . The computing system of claim 1 , wherein the tensor processing engine comprises a processing pipeline consisting of two or more tensor operation hardware components operating in series. 5 . The computing system of claim 1 , wherein the vector-to-scalar interface comprises at least one hardware component that generates a sum of element values ​​present in the input vector.

6. The computing system of claim 1, wherein the vector-to-scalar interface comprises at least one hardware component that identifies a maximum or minimum scalar by selecting from among element values ​​in the input vector.

7. The computing system of claim 1, wherein the vector-to-scalar interface includes functionality for simultaneously receiving the input vector and for storing scalars associated with element values ​​in the input vector. 8 . The computing system of claim 1 , wherein the scalar-to-vector interface comprises a broadcast hardware component for replicating scalars stored in the scalar register file across elements of the output vector. 9 . The computing system of claim 1 , wherein the scalar-to-vector interface comprises a vector assembly component for assembling the output vector based on a plurality of scalars stored in the scalar register file.

10. The computing system according to claim 1, wherein the hardware accelerator includes a plurality of channels for performing operations in parallel, and The scalar register file corresponds to a global resource used by all channels in the channel, while the vector-to-scalar interface is replicated for each channel of the hardware accelerator, and the scalar-to-vector interface is replicated for each channel of the hardware accelerator.

11. The computing system of claim 1 , wherein the hardware accelerator further comprises a vector memory manager that provides an interface between the hardware accelerator and a memory, wherein the scalar processing unit interacts with the tensor processing engine via the vector memory manager.

12. The computing system according to claim 1, wherein the vector to scalar interface has a plurality of selectable vector to scalar interface components, an operand in a vector write instruction selects a vector to scalar interface component for providing the one or more scalars based on the input vector, and The scalar-to-vector interface has a plurality of selectable scalar-to-vector interface components, and an operand in a vector read instruction selects a scalar-to-vector interface component for providing the output vector.

13. A method for performing an operation using a hardware accelerator, comprising: using a tensor processing engine to convert at least one instance of input tensor information to at least one instance of output tensor information during execution of a set of instructions in a program that performs operations associated with a neural network or other analytical task and implements vector-based data flow; as well as In executing the set of instructions, a scalar processing unit is used to: providing one or more scalars based on an input vector received from the tensor processing engine; storing the one or more scalars in a scalar register file; fetching at least one input scalar from the scalar register file; executing a scalar processing function to generate at least one output scalar based on the at least one input scalar, the scalar processing function being executed to perform a scalar operation specified by the program within the vector-based data stream; storing the at least one output scalar in the scalar register file; wherein the scalar processing unit further comprises a set of multiplexers for controlling whether the scalar register file receives a group of scalars or a single scalar; as well as providing an output vector based on one or more scalars stored in the scalar register file, The scalar processing unit is configured to operate as a native element in the vector-based data flow, The method uses the same kind of vector-based programming interface instructions to interact with the tensor processing engine and the scalar processing unit, The vector-based programming interface instructions include vector write instructions and vector read instructions, wherein the vector write instructions are used to provide input vectors to both the tensor processing engine and the scalar processing unit, and the vector read instructions are used to retrieve output vectors from both the tensor processing engine and the scalar processing unit. The hardware accelerator is implemented using hardware logic circuitry, which in turn includes one or more hardware processors in conjunction with memory, and / or one or more other hardware logic units that perform operations using fixed and / or programmable sets of hardware logic gates. 14 . The method of claim 13 , wherein the scalar processing unit provides the one or more scalars based on the input vector by reducing the received input vector to a single scalar.

15. The method of claim 13, wherein the scalar processing unit provides an output vector by replicating a scalar stored in the scalar register file across elements of the output vector.

16. The method according to claim 13, wherein the scalar processing unit has a vector to scalar interface, the vector to scalar interface having a plurality of selectable vector to scalar interface components, an operand in a vector write instruction selects a vector to scalar interface component for providing the one or more scalars based on the input vector, and The scalar processing unit has a scalar to vector interface having a plurality of selectable scalar to vector interface components, and an operand in a vector read instruction selects a scalar to vector interface component for providing the output vector.

17. A hardware accelerator for implementing vector-based data flow, the hardware accelerator comprising: Tensor processing engine, including: at least one tensor register file for storing tensors; and two or more tensor operation hardware components for performing respective tensor processing functions based on tensors stored in the at least one tensor register file; and a scalar processing unit configured to operate as a native element in the vector-based data flow and comprising: a vector-to-scalar interface for providing one or more scalars based on an input vector received from the tensor processing engine in response to a vector write instruction; a scalar register file for storing each scalar provided by the vector-to-scalar interface; a set of multiplexers for controlling whether the scalar register file receives a group of scalars or a single scalar; One or more scalar operation hardware components for performing corresponding scalar processing functions, each scalar operation hardware component configured to retrieve at least one input scalar from the scalar register file, perform a scalar processing function to generate at least one output scalar based on the at least one input scalar, and store the at least one output scalar in the scalar register file, the scalar processing function being executed to perform a scalar operation specified in an instruction executed by the hardware accelerator within the vector-based data stream; and a scalar to vector interface for providing an output vector based on one or more scalars stored in the scalar register file in response to a vector read instruction; The hardware accelerator uses the same kind of vector-based programming interface instructions to interact with the tensor processing engine and the scalar processing unit, the vector-based programming interface instructions including vector write instructions for providing input vectors to both the tensor processing engine and the scalar processing unit, and vector read instructions for retrieving output vectors from both the tensor processing engine and the scalar processing unit. The hardware accelerator is implemented using hardware logic circuitry, which in turn includes one or more hardware processors in conjunction with memory, and / or one or more other hardware logic units that perform operations using fixed and / or programmable sets of hardware logic gates.

18. The hardware accelerator of claim 17, further comprising a vector memory manager providing an interface between the hardware accelerator and a memory, wherein the scalar processing unit interacts with the tensor processing engine via the vector memory manager.

19. The hardware accelerator according to claim 17, wherein the hardware accelerator includes a plurality of channels for performing operations in parallel, and The scalar register file corresponds to a global resource used by all channels in the channel, while the vector-to-scalar interface is replicated for each channel of the hardware accelerator, and the scalar-to-vector interface is replicated for each channel of the hardware accelerator.

20. The hardware accelerator according to claim 17, wherein the vector to scalar interface has a plurality of selectable vector to scalar interface components, an operand in the vector write instruction selects a vector to scalar interface component for providing the one or more scalars based on the input vector, and The scalar-to-vector interface has a plurality of selectable scalar-to-vector interface components, and an operand in the vector read instruction selects a scalar-to-vector interface component for providing the output vector.

Images