Computing engine with sparse matrix instructions

Abstract

A computer system that can execute sparse matrix instructions is disclosed. The computer system includes a processor circuit that can retrieve an input matrix and a weight matrix in response to receiving an instruction. The input matrix can include a plurality of input vectors and the weight matrix can include a plurality of weight vectors, where a number of zero elements in the weight vectors exceeds a threshold. The processor circuit can generate a packed weight vector that includes an orthogonal subset of the plurality of weight vectors. A compute engine can execute a plurality of computations in parallel using a subset of the plurality of input vectors that correspond to a subset of the plurality of weight vectors included in the packed weight vector.

Description

Technical Field

[0001] The described implementation scheme relates generally to a computing engine for auxiliary processor circuitry, and more specifically to a computing engine capable of performing computations in parallel. Background Technology

[0002] Many applications performed by computer systems involve conducting large amounts of computation on relatively small numbers. For example, certain Long Short-Term (“LSTM”) learning algorithms are used in a variety of contexts, such as language detection, card readers, natural language processing, handwriting processing, and machine learning. LSTM processing involves many multiplication and accumulation operations.

[0003] General-purpose processor circuitry (e.g., the central processing unit (“CPU”)) tends to exhibit low performance on workloads of the types described above. Even in processor circuitry that includes vector instructions, CPU performance remains limited and power consumption is high. Low-performance, high-power workloads are problematic for any computer system, but they are particularly problematic for battery-powered computer systems such as smartphones, tablets, mobile computers, etc. Attached Figure Description

[0004] Figure 1 It is a block diagram that depicts the implementation scheme of a computer system.

[0005] Figure 2 It is a block diagram depicting the implementation scheme of the computing engine.

[0006] Figure 3A It is a block diagram depicting an implementation scheme for a computing circuit.

[0007] Figure 3B It is a block diagram depicting an implementation scheme for computing elements.

[0008] Figure 4 This is a diagram depicting the generation of the packing weight vector.

[0009] Figure 5 It is a block diagram depicting the implementation schemes of different computing engines.

[0010] Figure 6 It is a block diagram of a computing engine that includes a group of processing components.

[0011] Figure 7 It is a flowchart of an implementation scheme for a method of operating a computer system including a computing engine.

[0012] Figure 8 It is a flowchart of implementation schemes for different methods of operating a computer system including a computing engine.

[0013] Figure 9It is a block diagram of an implementation scheme for a device that includes a computing engine.

[0014] Figure 10 It can be a block diagram of various implementation schemes of a computer system that may include processor circuitry.

[0015] Figure 11 Examples of non-transitory computer-readable storage media for storing circuit design information are provided. Detailed Implementation

[0016] Some applications running on computer systems (such as language detection and processing) may require significant computation. In some cases, this computation may involve vector operations, such as vector multiplication (also known as the "vector cross product"). On many computer systems, performance may be low when performing vector operations.

[0017] Low performance of a computer system due to certain computations can result in latency that is perceptible to the user. For example, in speech recognition applications, the time required for the necessary computations to identify speech, remove noise, etc., will be noticeable to the user because of the delays that occur while the computer is processing. In some cases, the computations associated with forming the response that needs to generate speech can also cause latency that is observable to the user.

[0018] In some cases, some of the vectors being operated on during vector operations can be sparse vectors. As used in this paper, a sparse vector is a vector that includes a threshold number of elements with a value of zero (called "zero elements"). The threshold number can be based on the size of the vector, the desired performance of the system, or any other suitable metric.

[0019] During vector multiplication, multiplying a non-sparse vector (referred to as a "dense vector") by a sparse vector can include multiplication operations that only generate zero elements corresponding to the zero elements in the sparse vector. The matrix multiplication circuitry used to generate all zero elements is essentially unused because the result of multiplying by zero elements is known. Therefore, the matrix multiplication circuitry is underutilized during sparse computation, thus limiting performance and increasing power consumption.

[0020] The embodiments illustrated in the accompanying figures and described below provide techniques for performing computations using dense and sparse vectors. By packing different sparse data vectors corresponding to different dense input vectors into a single packed vector, multiple computations using different dense input vectors can be performed in parallel, thereby improving system performance.

[0021] A block diagram illustrating an implementation of a computer system is provided. As illustrated, computer system 100 includes processor circuitry 101, lower-level cache circuitry 102, and computing engine 103.

[0022] Processor circuitry 101 includes a data cache circuitry 108, an instruction cache circuitry 104, and a pipeline 105, which includes pipeline stages 106A to 106N. In various embodiments, pipeline stages 106A to 106N may be coupled in series.

[0023] In various implementations, processor circuitry 101 is coupled to lower-level cache circuitry 102 and compute engine 103. In some implementations, compute engine 103 may also be coupled to lower-level cache circuitry 102 and / or to data cache circuitry 108.

[0024] In some embodiments, processor circuitry 101 may fetch instructions from main memory (not shown) and store the fetched instructions in instruction cache circuitry 104. In various embodiments, processor circuitry 101 may be configured to retrieve the fetched instructions from instruction cache circuitry 104 and process the retrieved instructions via pipeline 105. In various embodiments, instruction cache circuitry 104 may be implemented as static random access memory (“SRAM”) circuitry using any suitable cache line size and configuration (e.g., set-associative, direct-mapped, etc.).

[0025] When an instruction is retrieved from instruction cache circuitry 104, it is processed through pipeline stages 106A to 106N. As depicted, pipeline 105 is generalized and can include any level of complexity and performance enhancement features in various embodiments. For example, processor circuitry 101 may be superscalar, and one or more pipeline stages 106A to 106N may be configured to process multiple instructions concurrently. The length of pipeline 105 can vary for different types of instructions. For example, arithmetic logic unit (“ALU”) instructions may have scheduling, address generation, translation / cache access, data forwarding, and miss processing stages. In various embodiments, pipeline stages 106A to 106N may include register renaming, prefetching, etc.

[0026] In various embodiments, processor circuitry 101 can be configured to fetch various instructions intended for use in computing engine 103, such as instruction 109, and send the fetched instructions to computing engine 103 for execution. The overhead of the processor circuitry 101's "front end" (i.e., fetching, decoding, etc.) can be amortized over the computation performed by computing engine 103. In one embodiment, processor circuitry 101 can be configured to propagate instructions through pipeline stages 106A to 106N to the point where the instruction becomes non-speculative. Figure 1As depicted, pipeline stage 106M is a non-speculative stage of pipeline 105. From pipeline stage 106M, instruction 112 is sent to computation engine 103. In various embodiments, processor circuitry 101 may retrieve instruction 112 before computation engine 103 completes the associated computation. In other embodiments, where instruction 112 is queued after other instructions within computation engine 103, processor circuitry 101 may deselect instruction 112 before computation engine 103 begins the associated computation.

[0027] Data cache circuitry 108 is configured to store frequently accessed data for processor circuitry 101 and computing engine 103. In various embodiments, data cache circuitry 108 may be configured to store input vector 113 and packing weight vector 114. In some cases, data cache circuitry 108 may also be configured to store results from computing engine 103 before storing them back into main memory. In various embodiments, data cache circuitry 108 may be implemented as SRAM memory circuitry using any suitable cache line size and configuration (e.g., set-associative, direct-mapped, etc.).

[0028] Lower-level cache circuitry 102 is configured to store frequently fetched instructions. In various embodiments, if previously fetched instructions are stored in lower-level cache circuitry 102, processor circuitry 101 may be configured to retrieve previously fetched instructions from lower-level cache circuitry 102 instead of requesting them from main memory. In some embodiments, compute engine 103 may also be configured to retrieve previously fetched instructions from lower-level cache circuitry 102. In various embodiments, lower-level cache circuitry 102 may be implemented as SRAM memory using any suitable cache line size and configuration (e.g., set-associative, direct-mapped, etc.).

[0029] In various implementations, computing engine 103 is configured to perform one or more computational operations. In some implementations, computing engine 103 may be configured to employ an instruction set that may be a subset of the instruction set of processor circuitry 101. In such cases, processor circuitry 101 can recognize instructions implemented by computing engine 103 and can pass such instructions to computing engine 103.

[0030] In some implementations, processor circuitry 101 may be configured to retrieve instruction 109. In some cases, processor circuitry 101 may be configured to retrieve instruction 109 from instruction cache circuitry 104 or from main memory. In some implementations, instruction 109 may be a “cross-switched outer product and accumulate instruction”. A prototype call to such a cross-switched outer product instruction is depicted in code snippet 1, where ZA is the destination tile, S is the data type, Zm points to a register containing the packed weight vector, Zn points to one or more registers containing the input vector (e.g., input vector 113), and Zk points to a register containing the selection vector (also referred to as a “lookup table”). Pseudocode for the cross-switched outer product and accumulate instruction is depicted in Appendix A.

[0031] Code snippet 1 - Prototype call for cross switch outer product and accumulate instructions XBAROPA ZA.S, Zn.S, Zm.S, Zk.S In various embodiments, processor circuitry 101 is configured to retrieve input matrix 110 and weight matrix 111 in response to execution instruction 109. In various embodiments, input matrix 110 includes input vector 113, and weight matrix 111 includes a plurality of weight vectors. In some embodiments, weight matrix 111 may be a sparse matrix, i.e., the number of zero elements included in weight matrix 111 may be greater than a threshold. In various embodiments, input matrix 110 and weight matrix 111 may be stored in main memory. Alternatively, corresponding copies of input matrix 110 and weight matrix 111 may be stored in data cache circuitry 108.

[0032] Processor circuitry 101 may be additionally configured to generate a packed weight vector 114 in response to execution instruction 109. In various embodiments, the packed weight vector 114 comprises an orthogonal subset of the weight vectors included in weight matrix 111. As used herein, an orthogonal subset of a vector set is a subset in which the dot product of any two vectors is zero.

[0033] The computation engine 103 is configured to perform multiple computations in parallel using a subset of the input vector 113 corresponding to a subset of vectors included in the packed weight vector 114. In some cases, the computation engine 103 may be configured to perform multiple computations based on the execution of instruction 109. In various embodiments, the computation engine 103 may be configured to retrieve the input vector 113 and the packed weight vector 114 from the data cache circuit 108. Alternatively or additionally, the computation engine 103 may be configured to retrieve the input vector 113 and / or the packed weight vector 114 from main memory or a lower-level cache circuit 102.

[0034] In other embodiments, processor circuitry 101 may be configured to generate instruction 112 based on instruction 109, and computation engine 103 may be configured to perform multiple computations in response to receiving instruction 112 from pipeline 105 of processor circuitry 101. It should be noted that in some embodiments, instruction 112 may be the same as instruction 109, while in other embodiments, each pipeline stage of pipeline stages 106A to 106N may modify instruction 109 to generate instruction 112.

[0035] Go to Figure 2 A block diagram of a computing engine 103 is depicted. As illustrated, the computing engine 103 includes computing circuitry 201, instruction buffer 202, memory circuitry 203, interface circuitry 204, lookup table 205, and cache memory circuitry 206.

[0036] Computational circuitry 201 is configured to perform multiple computations in parallel using a subset of input vectors 113 corresponding to a subset of a plurality of weight vectors included in the packed weight vectors 114. In various embodiments, the multiple computations may include vector multiplication. In some embodiments, computational circuitry 201 may, in response to receiving instruction 112 via interface circuitry 204, retrieve a subset of input vectors 113 and the packed weight vectors 114 from memory circuitry 203 before performing the multiple computations. As described below, computational circuitry 201 may include multiple computational elements configured to perform corresponding portions of the multiple computations.

[0037] Lookup table 205 is configured to store information that maps weights included in the packed weight vector 114 to specific input vectors in the input vector 113. In various embodiments, to perform a computation, computation circuitry 201 is configured to retrieve information from lookup table 205 and use that information to select a specific input vector from the input vector 113 to be computed. In some embodiments, lookup table 205 may be implemented using any suitable combination of SRAM memory circuitry, register file circuitry, or flip-flops and / or latching circuitry.

[0038] Interface circuitry 204 is configured to relay information between computing engine 103 and processor circuitry 101. In various embodiments, interface circuitry 204 may relay instruction 112 from pipeline 105 to instruction buffer 202. Additionally, interface circuitry 204 may relay input vector 113 and packing weight vector 114 from data cache circuitry 108 to memory circuitry 203.

[0039] Instruction buffer 202 is configured to receive instructions via interface circuitry 204. In some embodiments, instruction buffer 202 may be configured to communicate with processor circuitry 101 via an interface of interface circuitry 204 to indicate acceptance of instructions, requests for instructions, etc. In various embodiments, instruction buffer 202 may be configured to queue instructions while other instructions are being executed. In other embodiments, instruction buffer 202 may be configured to perform out-of-order processing of instructions. In some embodiments, instruction buffer 202 may be implemented using a first-in-first-out (FIFO) buffer circuit or any other suitable type of buffer circuit.

[0040] Memory circuit 203 includes Zn memory circuit 207, Zm memory circuit 208, and ZA memory circuit 209, each of which can be configured to store data received from processor circuit 101 and accumulated data generated by computing circuit 201 in response to performing computational operations. For example, Zn memory circuit 207 may store input vector 113, and Zm memory circuit 208 may store packing weight vector 114, as well as intermediate and final results from computations performing operations such as multiplication and accumulation. Note that in some embodiments, Zn memory circuit 207 may be referred to as y memory circuit, Zm memory circuit 208 may be referred to as x memory circuit, and ZA memory circuit 209 may be referred to as z memory circuit. In various embodiments, memory circuit 203 may be implemented using SRAM circuitry or any other suitable memory device circuitry.

[0041] As described below, computing circuitry 201 may include a plurality of computing elements arranged in an array of rows and columns. In various embodiments, a given computing element among the plurality of computing elements may receive selected elements from Zn memory circuitry 207 and Zm memory circuitry 208, and multiply such received elements. In some embodiments, a given computing element may receive the current value of the destination location in ZA memory circuitry 209, and may add the current value to the result of the multiplication to generate a result to be stored in ZA memory circuitry 209 (thus accumulating the multiplication result with the previous result).

[0042] In matrix mode, each vector element from Zn memory circuit 207 is multiplied by each vector element from Zm memory circuit 208 to generate matrix elements for the output matrix. Specifically, input vectors can be loaded into Zn memory circuit 207 and Zm memory circuit 208, and computation instructions can be executed by computation engine 103. In response to computation instructions (and particularly computation instructions for matrix mode), computation engine 103 can be configured to perform outer product operations and write the resulting outer product matrix into ZA memory circuit 209. If the vectors loaded into Zn memory circuit 207 have a first number of vector elements and the vectors loaded into Zm memory circuit 208 have a second number of vector elements, the resulting matrix (element i, j) is the product of the corresponding vector elements X(i) and Y(j). In one embodiment, the first number of vector elements is equal to the second number of matrix elements, thereby producing a squared output matrix. Other embodiments may implement non-square matrices, or different outer product operations may produce square or non-square results based on the input vector elements.

[0043] In one implementation, the computation engine 103 can be configured to perform output product operations and accumulate the resulting matrix with previous results in the ZA memory circuit 209 (where accumulation can be performed by addition or subtraction). That is, the external product instruction can be a fused multiply-add (FMA) operation, defined as multiplying the elements of a vector (“x vector”) stored in the Zn memory circuit 207 by the elements of a vector (“y vector”) stored in the Zm memory circuit 208, and adding the product to the corresponding element of a matrix (“z matrix”) stored in the ZA memory circuit 209; or it can be a fused multiply-subtract (FMS) operation, defined as multiplying the elements of the x vector by the elements of the y vector, and subtracting the product from the corresponding element of the z matrix. Alternatively, the FMS operation can include subtracting the corresponding element of the z matrix from the product. In one implementation, the FMA and FMS operations can be performed on floating-point vector elements. Integer vector elements can also support multiplication and accumulation (MAC) computation instructions.

[0044] Furthermore, the computation instructions (FMA, FMS, and MAC) can be code used in vector mode. In vector mode, vector multiplication can be performed (e.g., each element of one vector can be multiplied by the corresponding element of another vector). The result can be accumulated at the target entry of ZA memory circuit 209 along with the current value stored in ZA memory circuit 209. That is, in vector mode, a single entry (or row) stored in ZA memory circuit 209 can be updated, rather than multiple entries (or rows) representing a matrix in matrix mode.

[0045] Additionally, the computation engine 103 can be configured to read operands from any offset within the Zn memory circuit 207 and / or the Zm memory circuit 208. Operands can be selected using the register address of the entry in the memory circuit from which the operand is to be read and the offset to that entry. An initial operand element (vector element) can be selected from the offset, and additional vector elements can be read from adjacent positions within the entry until the end of the entry is reached. The computation engine 103 can be configured to complete the vector operation by reading additional vector elements from the beginning of the next entry (register address plus one). Therefore, the data to be operated on may be "unaligned" within the entry, and the correct data for a given operation can be read without moving data back and forth between the Zn memory circuit 207 and the Zm memory circuit 208. This operation can be useful when the operation to be performed uses partially overlapping data.

[0046] In one implementation, vector elements can be 8-bit or 16-bit integers of 16, 32, or 64-bit floating-point numbers. Therefore, a 64-bit field in Zn memory circuit 207 or Zm memory circuit 208 may include four 16-bit integers or eight 8-bit integers. Similarly, a 64-bit field in Zn memory circuit 207 or Zm memory circuit 208 may include four 16-bit floating-point numbers, two 32-bit floating-point numbers, or one 64-bit floating-point number.

[0047] As previously mentioned, the computation circuit 201 can be implemented as an array of computational elements, used not only to perform multiplication and addition operations that generate elements for a result matrix or result vector, but also to perform multiplication and addition operations for matrix / vector operations in parallel. For example, if the Zn memory circuit 207 and Zm memory circuit 208 include 512-bit entries and implement 8-bit vector elements, then 64 vector element input matrices are stored in each entry stored in the Zn memory circuit 207 and Zm memory circuit 208, and can be processed in parallel in response to a computation instruction. Similarly, if each entry of the memory circuit supports 1024-bit entries, 128 vector elements can be processed in parallel. If 128-bit entries are supported, 16 vector elements can be processed in parallel. If 256-bit entries are supported, 32 vector elements can be processed in parallel. Alternatively, the computation circuit 201 may include fewer MACs than the number of MACs used to perform matrix / vector element multiplications in parallel on the input operands. In this implementation, the computing engine 103 can use an array of matrix calculations performed by multiple passes through the computing circuit 201 for different portions of the input data from the Zn memory circuit 207 and the Zm memory circuit 208.

[0048] As mentioned above, the computation engine 103 can support matrix / vector elements of multiple sizes in the accumulated result. For example, it can support 16-bit and 32-bit result elements for 16-bit input elements. For 32-bit input elements, it can support 32-bit or 64-bit elements. The maximum number of result elements in the ZA memory circuit 209 can be set by the size of the ZA memory circuit 209 and the size of the accumulated elements used for a given operation. A smaller size can consume less memory in the ZA memory circuit 209. For matrix operations, the ZA memory circuit 209 can be arranged to write smaller matrix elements into certain entries of the ZA memory circuit 209, leaving other entries unused (or unmodified). For example, if the matrix element is half the size of the largest element, every other entry in the ZA memory circuit 209 may be unused. If the matrix element is one-quarter the size of the largest element, every fourth row may be used, and so on. In one implementation, the ZA memory circuit 209 can be considered to have multiple banks, wherein entries in the ZA memory circuit 209 are distributed across the banks (e.g., for a two-bank implementation, even-addressed entries may be in bank 0, and odd-addressed entries may be in bank 1). If four banks exist, each fourth entry may be in a different bank (e.g., entries 0, 4, 8, etc. may be in bank 0, entries 1, 5, 9, etc. may be in bank 1, and so on). The vector result may consume one row of the ZA memory circuit 209.

[0049] In some embodiments, the instructions executed by the computing engine 103 may also include memory instructions (e.g., load / store instructions). Load instructions can transfer vectors / matrices from system memory (not shown) to Zn memory circuit 207, Zm memory circuit 208, or ZA memory circuit 209. Store instructions can write matrices / vectors from ZA memory circuit 209 to system memory. Some embodiments may also include store instructions for writing vectors / matrices from Zn memory circuit 207 and Zm memory circuit 208 to system memory. System memory may be memory accessed at the bottom of a cache hierarchy including instruction cache circuit 104, data cache circuit 108, and lower-level cache circuit 102. System memory may be implemented using static random access memory (SRAM) circuitry, dynamic random access memory (DRAM) circuitry, or any other suitable type of memory circuitry. Memory controller circuitry may also be included to interact with system memory. In one embodiment, the computing engine 103 may be cached with processor circuitry 101 and may have access to data cache circuitry 108 to read / write data. Alternatively, the computing engine 103 may have access to the lower-level cache circuitry 102, and the lower-level cache circuitry 102 may ensure cache coherence with the data cache circuitry 108. In other embodiments, the computing engine 103 may have access to the memory system, and coherence points in the memory system may ensure consistent access.

[0050] Cache memory circuitry 206 can be configured to store data recently accessed by computing engine 103. In various embodiments, the inclusion and size of cache memory circuitry 206 can be based on the effective latency experienced by computing engine 103 and the expected performance level of computing engine 103. In various embodiments, cache memory circuitry 206 can be implemented using any suitable cache line size and configuration (e.g., set-associative, direct-mapped, etc.).

[0051] Go to Figure 3A A block diagram depicting an embodiment of computing circuit 201 is shown. As illustrated, computing circuit 201 includes computing elements (denoted as "CE 301A to 301I"). Although in Figure 3A The embodiment depicts only nine computing elements, but in other embodiments, any suitable number of computing elements can be used. In some embodiments, the number of computing elements can be based on the size of the vector to be operated on by the computing circuit 201. For example, in the case of vector multiplication of two 8-bit vectors, the computing circuit 201 would include 64 computing elements.

[0052] In various embodiments, CEs 301A to 301I can be coupled together in an array and can be configured to perform various operations, such as multiplication and accumulation. In other embodiments, CEs 301A to 301I can be configured to perform other mathematical operations, such as addition. CEs 301A to 301I can be configured to receive Zn vector 302 and Zm vector 303, which can be stored in Zn memory circuit 207 and Zm memory circuit 208, respectively. In some embodiments, CEs 301A to 301I can also be configured to perform accumulation operations in ZA memory circuit 209 and / or cache memory circuit 206.

[0053] The elements of Zn vector 302 are labeled as Zn0 to Zn M Furthermore, the elements of the Zm vector 303 are labeled as Zm0 to Zm0. M The elements of the resulting matrix are labeled ZA. 00 toZA MM And in some embodiments, it can be stored in the ZA memory circuit 209. As illustrated, the first number of the matrix element is the number of the element from the Zn vector 302, and the second number of the matrix element is the number of the corresponding element from the Zm vector 303. Therefore, each row of the calculation circuit 201 corresponds to a specific element of the Zm vector 303, and each column of the calculation circuit 201 corresponds to a specific element of the Zn vector 302. Each calculation element of CE 301A to 301I can perform operations on the corresponding elements of the Zn vector 302 and the Zm vector 303 to generate the matrix element ZA. 00 toZA MM One of the corresponding ones in the equation. For example, CE 301A can be configured to generate a product of Zn0 and Zm0, which is added to the product against ZA. 00 The existing value.

[0054] In various implementations, the computation engine 103 can be configured to support a variety of instructions, such as fused multiply-accumulate (FMA), fused multiply-subtract (FMS), multiply-accumulate (MAC), or any other suitable instructions. FMA and FMS can perform operations on floating-point elements (e.g., 16-bit, 32-bit, or 64-bit elements). FMA can calculate ZA = ZA + Zn. Zm, where FMS can calculate ZA = ZA - Zn Zm. MAC can perform operations on integer operands (e.g., 8-bit and 16-bit integer operands) and can calculate ZA = ZA + Zn. Zm. In some implementations, the MAC may allow an optional right shift of the result before adding the multiplication result to ZA.

[0055] Go to Figure 3B A block diagram depicting an embodiment of a computing element is shown. As illustrated, computing element 306 includes a multiplier circuit 305 and a storage device circuit 304. In various embodiments, computing element 306 may correspond to, for example... Figure 3A Any of CE 301A to 301D as depicted in the implementation scheme.

[0056] Multiplier circuit 305 is configured to multiply two numbers. In various embodiments, one of the numbers may be part of an input vector in input vector 113, while the other number may be part of a packed weight vector 114. In some cases, multiplier circuit 305 may be configured to support both integer and floating-point multiplication operations. In some embodiments, the two numbers may be in binary format, two's complement format, or any other suitable format. In various embodiments, multiplier circuit 305 may be implemented using a Wallace tree multiplier circuit, a Booth multiplier circuit, or any other suitable type of multiplier circuit.

[0057] The storage device circuit 304 is configured to store at least one of the multiplicand or multiplier before the multiplication operation is performed by the multiplier circuit 305. In some embodiments, the storage device circuit 304 may also be configured to store the product of the multiplication operation performed by the multiplier circuit 305 to allow for accumulation operations. In various embodiments, the storage device circuit 304 may be implemented using SRAM circuitry, one or more register file circuitry including multiple flip-flops or latch circuitry, or any other suitable circuitry configured to store multiple data bits.

[0058] Although only two circuits are depicted in computing element 306, it is possible and contemplated to include other circuit blocks in other embodiments. For example, in some embodiments, computing element 306 may include an addition circuit, a subtraction circuit, a division circuit, etc.

[0059] Go to Figure 4 The diagram illustrates the generation of a packing weight vector. In various embodiments, the generation of the packing weight vector may be in response to the execution of instruction 109 by processor circuitry 101, or it may be in response to the execution of one or more instructions fetched prior to instruction 109 by processor circuitry 101.

[0060] In various implementation schemes, it can correspond to, for example Figure 1The weight matrix 401 of the depicted weight matrix 111 includes weight vectors 402A to 402F. As illustrated, weight vector 402A includes two non-zero elements (zero elements are depicted as squares without a fill pattern). Similarly, weight vectors 402B and 402E also include two non-zero elements, while weight vector 402C includes only one non-zero element. Weight vectors 402D and 402F include only zero elements. In various embodiments, the number of zero elements in weight matrix 401 is greater than a threshold, thus making weight matrix 401 a sparse matrix.

[0061] It should be noted that although the vectors included in weight matrix 401 are depicted as having 6 elements, in other embodiments, the vectors included in weight matrix 401 may include any suitable number of elements. It should also be noted that although weight matrix 401 includes 6 vectors, in other embodiments, weight matrix 401 may include any suitable number of vectors.

[0062] The corresponding locations of the non-zero elements of weight vectors 402A, 402C, and 402E do not overlap; that is, the vector dot product between any two of weight vectors 402A, 402C, and 402E is zero, thus making the vectors orthogonal. Since weight vectors 402A, 402C, and 402E are orthogonal, they can be combined (or "packed") into a packed weight vector 403. It should be noted that in various embodiments, the packed weight vector 403 may correspond to, for example,... Figure 1 The depicted packing weight vector is 114.

[0063] Where the weight vectors are not orthogonal, portions or subsets of different weight vectors can be combined to form a packaged weight vector, such as packaged weight vector 403. In some implementations, different subsets of a given weight vector can be used together with subsets of other weight vectors to form a corresponding packaged weight vector. By allowing the use of different subsets of weight vectors to be used in the formation of the packaged weight vector, the sparse computation techniques described herein can be employed where the weight matrix does not include orthogonal vectors or includes a small number of orthogonal vectors.

[0064] Input matrix 404 includes input vectors 405A to 405F. In various embodiments, input matrix 404 and input vectors 405A to 405F may correspond to input matrix 110 and weight matrix 111, respectively. Figure 4As shown in the common padding pattern, a cross product operation will be performed between the input vectors in input vectors 405A to 405F and the corresponding weight vectors in weight vectors 402A to 402F. For example, the cross product operation will be performed using input vector 405A and weight vector 402A. Similarly, the cross product operation will be performed using input vector 405B and weight vector 402B.

[0065] Using the packed weight vector 403, computational circuitry 406 can dedicate different sets of computational elements 407A to 407F to different vector cross product operations in multiple vector cross product operations. As illustrated, computational elements 407A and 407C are used to compute the vector cross product of input vector 405A and weight vector 402A, while computational elements 407B and 407F are used to compute the vector cross product of input vector 405E and weight vector 402E. Similarly, computational element 407D is used to compute the vector cross product of input vector 405C and weight vector 402C. Note that computational element 407E is not used. By mapping different computational elements to different input vectors based on the packed weight vector 403, the vector cross product operations described above can be performed in parallel, thus saving computation cycles and power. Without the packed weight vector 403, the vector cross products described above would have to be performed sequentially, with a larger number of computational elements 407A to 407F remaining unused.

[0066] Go to Figure 5 Block diagrams depicting different implementations of computing engines are shown. As illustrated, computing engine 500 includes computing element 501 and register circuits 502 to 507. It should be noted that in various implementations, computing engine 500 may correspond to, for example... Figure 1 The computing engine 103 is described.

[0067] Although computing element 501 is depicted as comprising 16 computing circuits, in other embodiments, computing element 501 may include any suitable number of computing circuits. In various embodiments, any computing element in computing element 501 may correspond to, for example... Figure 3B The computing element 306 is depicted.

[0068] Register circuit 502 can be configured to store input vector 508, and register circuit 503 can be configured to store input vector 509. In various embodiments, input vectors 508 and 509 can be included in input matrix 110. Note that in some embodiments, computation engine 500 can be configured to reorder or encode the corresponding elements included in input vectors 508 and 509 to combine sparse and non-sparse portions of the vectors.

[0069] Register 504 can be configured to store selection vector 510. In various embodiments, computation engine 500 can be configured to use selection vector 510 to determine which portions of input vectors 508 and 509 are used for computation. For example, selection vector 510 can indicate that portion 512 of input vector 508 and portion 513 of input vector 509 will be used for a specific computation. In some embodiments, the elements of selection vector 510 can be reordered to aid in selecting the non-sparse portions of input vectors 508 and 509.

[0070] In various implementations, registers 502 to 503 may be included in the Zn memory circuit 207. In some cases, different registers in registers 502 to 503 may correspond to different entries, address ranges, or memory banks within the Zn memory circuit 207.

[0071] Register 507 may be configured to store weight vector 511. In various embodiments, weight vector 511 may be included in weight matrix 111. In some embodiments, register 506 may be configured to store predicate information, lookup table information, or any other suitable information that the computation engine 500 may use when performing computations. In various embodiments, registers 504, 506, and 507 may be included in Zm memory circuitry 208. In some cases, registers 504, 506, and 507 may correspond to different entries, address locations, or memory banks within Zm memory circuitry 208.

[0072] The computing element 501 can correspond to, for example Figure 3A The CE 301A to 301I are depicted. As described below, computing element 501 can be coupled to multiple communication buses for receiving data from registers 502 to 507 and for transferring or receiving data from ZA memory circuit 209.

[0073] As described above, elements from different portions of portions 512 and 513 can be transferred to corresponding computational elements in computational element 501. Additionally, different elements of weight vector 511 can be transferred to corresponding computational elements in computational element 501. Once the elements have been transferred to different computational elements in computational element 501, operations can be performed, where the different computational elements in computational element 501 execute the corresponding portions of the operations in parallel.

[0074] It should be noted that in some cases, if one or more computing elements in computing element 501 are to receive sparse data, then one or more computing elements in computing element 501 may not receive elements from input vector 508 or 509 or weight vector 511. In such cases, one or more computing elements in computing element 501 can be deactivated when an operation is to be performed. By deactivating unused computing elements, the power consumption of computing engine 500 can be reduced.

[0075] Go to Figure 6 A block diagram depicts a computing engine comprising groups of processing elements. As illustrated, computing engine 600 includes computing elements (also referred to as “processing elements”) denoted as CE 601A to 601D, CE 602A to 602D, CE 603A to 603D, and CE 604A to 604D, which are organized into different groups (also referred to as “processing element groups” or “pegs”). For example, CE 601A to 601D are included in group 605, CE 602A to 602D are included in group 606, and so on. Although the rows of CEs are organized into corresponding groups, different arrangements (e.g., two rows of CEs included in a single group) are possible and contemplated in other embodiments.

[0076] Each of groups 605 to 608 is connected to a common bus 611. In various embodiments, the common bus 611 can be used to allow each CE included in groups 605 to 608 to transfer data to or receive data from the ZA memory circuit 209. In various embodiments, the common bus 611 can be 512 bits wide or any other suitable bit width.

[0077] Different CEs are connected to the Zn memory circuit 207 via a set of buses. As described above, this set of buses can have any suitable bit width. In various embodiments, there are fewer buses than rows in the computing circuit, such as computing circuit 201. Thus, a given bus can be mapped to more than one set of CEs. For example, bus 609 is mapped to groups 605 and 606. In a similar manner, bus 610 is mapped to groups 607 and 608. In other words, a set of CEs can share the bus used to receive data from the Zn memory circuit 207. Although two sets of CEs coupled to a shared bus are shown, in other embodiments, a different number of groups (e.g., four sets of CEs) can be coupled to a single shared bus. The number of CE groups coupled to the shared bus can depend on the size of computing circuit 201, the number of elements in Zn vector 302 and Zm vector 303, or any other suitable parameter of computing engine 103.

[0078] Since a given bus connecting the ZA memory circuit 209 and the computing circuit 201 can be coupled to multiple sets of CEs, data can be transferred from the ZA memory circuit 209 to different CE sets within these CE sets using multiple passes. In the first pass, a first set of elements from a vector stored in the ZA memory circuit 209 is transferred to a first set of CEs included in a CE set coupled to the shared bus. In the second pass, a second set of elements from the vector is transferred to a second set of CEs included in the CE set, and so on.

[0079] In summary, various implementations of a computer system have been disclosed. Broadly speaking, a computer system may include a computation engine and multiple register circuits. The multiple register circuits may be configured to store corresponding vectors from a plurality of vectors. The computation engine may include multiple computation circuits arranged in an array of rows and columns. The computation engine may be configured to select a first portion of a first input vector from a first register circuit in the plurality of register circuits using a selection vector stored in a specific register circuit. The computation engine may also be configured to select a second portion of a second input vector from a second register circuit in the plurality of register circuits using the selection vector. Additionally, the computation engine may be configured to perform multiple computations in parallel using multiple computation circuits, using the first portion of the first input vector, the second portion of the second input vector, and a weight vector. In some cases, the multiple computations include a vector outer product computation, and the weight vector is stored in different register circuits in the plurality of register circuits.

[0080] Go to Figure 7 A flowchart depicting an implementation of a method for operating a computing engine included in processor circuitry is shown. The method, which can be applied to various computer systems (e.g., computer system 100), begins in box 701.

[0081] The method includes retrieving an input matrix and a weight matrix by processor circuitry included in a computer system and in response to receiving a first instruction (box 702). In various embodiments, the input matrix includes multiple input vectors, and the weight matrix includes multiple weight vectors. In some embodiments, the number of zero elements included in the weight matrix exceeds a threshold.

[0082] The method also includes generating a packed weight vector comprising an orthogonal subset of multiple weight vectors by processor circuitry (box 703). In some embodiments, generating the packed weight vector involves the processor circuitry storing information in a lookup table. In such cases, the information would include mapping a specific weight in a subset of multiple weight vectors to a corresponding input vector in a subset of multiple input vectors.

[0083] The method also includes performing multiple computations in parallel by a computation engine included in a computer system using a subset of multiple input vectors corresponding to a subset of multiple weight vectors included in a packaged weight vector (box 704). In some embodiments, the multiple computations include corresponding multiple vector multiplication operations.

[0084] In some embodiments, the method may include generating a second instruction using a first instruction by processor circuitry, and transmitting the second instruction to a computing engine by the processor circuitry. In such cases, the method may additionally include performing multiple computations by the computing engine in response to receiving the second instruction. In various embodiments, performing multiple computations may include retrieving information from a lookup table by the computing engine. The method ends at block 705.

[0085] Go to Figure 8 The diagram depicts flowcharts illustrating implementations of different methods for operating a computing engine included in processor circuitry. Methods applicable to various computer systems (e.g., computer system 100) begin in box 801.

[0086] The method includes selecting a first portion of a first input vector from different register circuits among a plurality of register circuits using a selection vector stored in a specific register circuit among a plurality of register circuits (block 802). In various embodiments, the computing engine includes a plurality of computing circuits arranged in an array of rows and columns. In some embodiments, computing circuits in a specific row of the plurality of computing circuits may be coupled to a first communication bus. Similarly, computing circuits in different rows of the plurality of computing circuits may be coupled to a second communication bus. While each row of an array of computing elements is described as coupled to a corresponding communication bus, in other embodiments, any suitable group of a subset of computing elements in the array of computing elements may be coupled to a corresponding communication bus.

[0087] The method also includes a second portion of the second input vector selected by the computing engine from different register circuits among a plurality of register circuits using a selection vector (box 803).

[0088] In some implementations, the method may additionally include the computing engine using a first communication bus to transfer specific elements of a first portion of the first input vector to corresponding computing circuitry included in a specific row of computing circuitry. Furthermore, the method may include the computing engine using a second communication bus to transfer different elements of the first portion of the first input vector to corresponding computing circuitry included in different rows of computing circuitry. It should be noted that elements of the second portion of the second input vector may be transferred to corresponding computing elements in a similar manner.

[0089] In some cases, the first portion of the first input vector and the second portion of the second input vector can be "dense" subvectors. In other words, the first number of zeros included in the first portion of the first input vector can be less than a threshold, and the second number of zeros included in the second portion of the second input vector can be less than a threshold. In other embodiments, the weight vector can be included in a weight matrix, where the number of zero elements in the weight matrix exceeds different thresholds.

[0090] The method also includes multiple computations performed in parallel by multiple computational elements using a first portion of a first input vector, a second portion of a second input vector, and a weight vector (block 804). In various embodiments, the multiple computations are included in a vector outer product computation, and the weight vector is stored in a fourth register circuit among multiple register circuits.

[0091] In some cases, performing multiple computations involves using specific elements of a first portion of the first input vector and corresponding elements of the weight vector to perform corresponding operations among multiple computations. In various implementations, a given computation among the multiple computations may include multiplication and accumulation operations. The method ends in block 805.

[0092] Now for reference Figure 9 This diagram illustrates a block diagram of an example embodiment of a device including processor circuitry performed out of order. In some embodiments, the components of device 900 may be included within a system-on-a-chip. In some embodiments, device 900 may be included in a mobile device that may be battery-powered. Therefore, the power consumption of device 900 may be an important design consideration. In the illustrated embodiment, device 900 includes a structure 910, a computing complex 920, an input / output (I / O) bridge 950, a cache / memory controller 945, a graphics unit 975, and a display unit 965. In some embodiments, as a supplement to or alternative to the illustrated components, device 900 may include other components (not shown), such as video processor encoders and decoders, image processing or recognition elements, computer vision elements, etc.

[0093] Structure 910 may include various interconnects, buses, MUXs, controllers, etc., and may be configured to facilitate communication between various components of device 900. In some embodiments, portions of structure 910 may be configured to implement various different communication protocols. In other embodiments, structure 910 may implement a single communication protocol, and components coupled to structure 910 may internally switch from a single communication protocol to other communication protocols.

[0094] In the illustrated embodiments, computing complex 920 includes a bus interface unit (BIU) 925, a cache 930, and cores 935 and 940. In various embodiments, computing complex 920 may include various numbers of processors, processor cores, computing engines (e.g., computing engine 103), and cache memory circuitry. For example, computing complex 920 may include one, two, or four processor cores or any other suitable number of processor cores. In one embodiment, cache 930 is a set-associative L2 cache. In some embodiments, cores 935 and 940 may include internal instruction and data caches. In some embodiments, a coherence unit (not shown) in architecture 910, cache 930, or elsewhere in device 900 may be configured to maintain coherence between various caches of device 900. BIU 925 may be configured to manage communication between computing complex 920 and other elements of device 900. Processor cores (such as cores 935 and 940) may be configured to execute instructions of a specific instruction set architecture (ISA) that may include operating system instructions and user application instructions. These instructions can be stored in a computer-readable medium, such as a memory coupled to a cache memory controller 945, which is discussed below.

[0095] As used herein, the term "coupled to" can indicate one or more connections between elements, and coupling may include intermediate elements. For example, in Figure 9 In this context, the graphics unit 975 can be described as being "coupled" to memory via structure 910 and cache / memory controller 945. In contrast, in... Figure 9 In the exemplary implementation, the graphics unit 975 is "directly coupled" to the structure 910 because there are no intermediate elements.

[0096] The cache / memory controller 945 can be configured to manage data transfer between the structure 910 and one or more caches and memories. For example, the cache / memory controller 945 may be coupled to an L3 cache, which in turn may be coupled to system memory. In other embodiments, the cache / memory controller 945 may be directly coupled to memory. In some embodiments, the cache / memory controller 945 may include one or more internal caches. The memory coupled to the cache / memory controller 945 can be any type of volatile memory, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), dual data rate (DDR, DDR2, DDR3, etc.), SDRAM (including mobile versions of SDRAM, such as mDDR3, etc., and / or low-power versions of SDRAM, such as LPDDR4, etc.), RAMBUS DRAM (RDRAM), static RAM (SRAM), etc. One or more memory devices may be coupled onto a circuit board to form a memory module, such as a single in-line memory module (SIMM), a dual in-line memory module (DIMM), etc. Alternatively, these devices can be mounted with integrated circuits in a chip-stacked configuration, a package-stacked configuration, or a multi-chip module configuration. The memory coupled to the cache / memory controller 945 can be any type of non-volatile memory, such as NAND flash memory, NOR flash memory, nanometer RAM (NRAM), magnetoresistive RAM (MRAM), phase-change RAM (PRAM), raceway memory, memristor memory, etc. As noted above, the memory can store program instructions executable by the computing complex 920 to enable the computing device to perform the functionality described herein.

[0097] The graphics unit 975 may include one or more processors, such as one or more graphics processing units (GPUs). For example, the graphics unit 975 may receive graphics-oriented instructions, such as OpenGL. ® Metal ® Or DIRECT3D ®Instructions. The graphics unit 975 can execute dedicated GPU instructions or perform other operations based on received graphics-oriented instructions. The graphics unit 975 can typically be configured to process large blocks of data in parallel and can construct an image in a frame buffer for output to a display, which may be included in a device or may be a separate device. The graphics unit 975 may include a transformation engine, a lighting engine, a triangle engine, and a rendering engine in one or more graphics processing pipelines. The graphics unit 975 can output pixel information for displaying the image. In various embodiments, the graphics unit 975 may include programmable shader circuitry, which may include highly parallel execution cores configured to execute graphics programs, which may include pixel tasks, vertex tasks, and computation tasks (which may be graphics-dependent or not).

[0098] Display unit 965 can be configured to read data from a frame buffer and provide a stream of pixel values ​​for display. In some embodiments, display unit 965 can be configured as a display pipeline. Additionally, display unit 965 can be configured to blend multiple frames to produce an output frame. Furthermore, display unit 965 may include one or more interfaces (e.g., MIPI) for coupling to a user display (e.g., a touchscreen or external display). ® Or embedded display port (eDP)).

[0099] I / O bridge 950 may include various components configured to implement, for example, Universal Serial Bus (USB) communication, security, audio, and low-power always-on functionality. I / O bridge 950 may also include interfaces such as, for example, pulse-width modulation (PWM), general-purpose input / output (GPIO), serial peripheral interface (SPI), internal integrated circuit (I2C), and / or radio frequency interfaces. Various types of peripheral devices and equipment can be coupled to device 900 via I / O bridge 950.

[0100] In some embodiments, device 900 includes network interface circuitry (not explicitly shown) that can be connected to structure 910 or I / O bridge 950. This network interface circuitry can be configured to communicate via various networks, which can be wired networks, wireless networks, or both. For example, the network interface circuitry can be configured to communicate via a wired local area network (LAN), a wireless LAN (e.g., via Wi-Fi), or Wi-Fi. ™ The network interface circuitry can communicate via one or more cellular networks using one or more radio access technologies, or wide area networks (e.g., the Internet or a virtual private network). In some embodiments, the network interface circuitry is configured to communicate via one or more cellular networks using one or more radio access technologies. In some embodiments, the network interface circuitry is configured to use device-to-device communication (e.g., Bluetooth). ® or Wi-Fi™ Communication can be achieved via Direct, etc. In various implementations, the network interface circuitry can provide device 900 with connectivity to various types of other devices and networks.

[0101] Now go to Figure 10 This illustrates various types of systems that may include any of the circuits, devices, or systems discussed above. The system or device 1000, which may utilize one or more of the techniques described herein in combination with or otherwise, can be used in a wide range of fields. For example, the system or device 1000 can be used as part of the hardware of a system such as a desktop computer 1010, a laptop computer 1020, a tablet computer 1030, a cellular or mobile phone 1040, or a television 1050 (or a set-top box coupled to a television).

[0102] Similarly, the disclosed components can be used in wearable devices 1060, such as smartwatches or health monitoring devices. In many embodiments, a smartwatch can perform a variety of different functions—for example, access to email, cellular services, calendars, health monitoring, etc. Wearable devices can also be designed to perform only health monitoring functions, such as monitoring a user's vital signs, performing epidemiological functions such as contact tracing, providing communication to emergency medical services, etc. Other types of devices are also envisioned, including devices worn around the neck, implantable devices, and glasses or helmets designed to provide computer-generated reality experiences, such as those based on augmented reality and / or virtual reality.

[0103] System or device 1000 can also be used in a variety of other contexts. For example, system or device 1000 can be used in the context of a server computer system (such as a dedicated server) or on shared hardware implementing cloud-based services 1070. Furthermore, system or device 1000 can be implemented in a wide range of dedicated everyday devices, including common household appliances 1080 such as refrigerators, thermostats, security cameras, etc. The interconnection of such devices is often referred to as the "Internet of Things" (IoT). Components can also be implemented in various modes of transportation. For example, system or device 1000 can be used in control systems, guidance systems, entertainment systems, etc., of various types of vehicles 1090.

[0104] Figure 10 The applications illustrated herein are merely exemplary and are not intended to limit the potential future applications of the disclosed systems or devices. Other example applications include, but are not limited to, portable gaming devices, music players, data storage devices, and unmanned aerial vehicles.

[0105] Various example circuits have been described in detail above in this disclosure. It is intended that this disclosure cover not only embodiments including such circuits, but also computer-readable storage media that include design information specifying such circuits. Therefore, this disclosure is intended to support claims that specify storage media for circuits in formats that cover not only means including the disclosed circuits, but also analog models for programming computing systems to generate hardware circuits, computing systems configured to generate hardware (e.g., integrated circuits) including the disclosed circuits. Claims for such storage media are intended to cover entities that, for example, generate circuit designs but do not themselves perform complete operations (such as design simulation, design synthesis, circuit fabrication, etc.).

[0106] Figure 11 This is a block diagram illustrating an example of a non-transitory computer-readable storage medium containing circuit design information 1115 and code sequence 1170 according to some embodiments. In the illustrated embodiment, computing system 1140 is configured to process design information 1115. This may include executing instructions included in design information 1115, interpreting instructions included in design information 1115, compiling, transforming, or otherwise updating design information 1115, etc. Thus, in some embodiments, design information 1115 (e.g., by programming computing system 1140) controls computing system 1140 to perform the various operations discussed below.

[0107] In the illustrated example, computing system 1140 processes design information 1115 to generate both a computer simulation model 1160 of integrated circuit 1130 and low-level design information 1150. In other embodiments, computing system 1140 may generate only one of these outputs, may generate other outputs based on design information 1115, or both. Regarding computer simulation model 1160, computing system 1140 may execute instructions in a hardware description language, including register-transfer-level (RTL) code, behavioral code, structural code, or some combination thereof. The simulation model may perform the functionality specified by design information 1115, facilitate verification of the functional correctness of the hardware design, generate power consumption estimates, generate timing estimates, etc.

[0108] In the illustrated example, computing system 1140 also processes design information 1115 to generate low-level design information 1150 (e.g., gate-level design information, netlists, etc.). As shown, this may include synthesis operations such as constructing multi-level networks, optimizing the network using technology-independent techniques, technology-dependent techniques, or both, and outputting a gate network (with potential constraints based on a library of available gate pair techniques, size adjustments, delays, power, etc.). Based on the low-level design information 1150 (and potentially other inputs), semiconductor manufacturing system 1120 is configured to manufacture integrated circuit 1130 (which may correspond to the functionality of computer simulation model 1160). Note that computing system 1140 may generate different simulation models based on design information at various descriptive levels, including low-level design information 1150, design information 1115, etc. Data representing low-level design information 1150 and computer simulation model 1160 may be stored on non-transitory computer-readable storage medium 1110, or on one or more other media.

[0109] In some implementations, low-level design information 1150 controls (e.g., programs) the semiconductor manufacturing system 1120 to manufacture integrated circuit 1130. Therefore, when processed by the manufacturing system, the design information can program the manufacturing system to manufacture circuits including the various circuits disclosed herein.

[0110] The nontransitory computer-readable storage medium 1110 may include any of a variety of suitable types of memory devices or storage devices. The nontransitory computer-readable storage medium 1110 may be an installation medium, such as a CD-ROM, floppy disk, or magnetic tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory; magnetic media, such as hard disk drives or optical storage devices; registers, or other similar types of memory elements. The nontransitory computer-readable storage medium 1110 may also include other types of nontransitory memory or combinations thereof. Therefore, the nontransitory computer-readable storage medium 1110 may include two or more memory media; such media may reside in different locations—for example, in different computer systems connected via a network.

[0111] Design information 1115 can be specified using any of a variety of suitable computer languages, including hardware description languages ​​such as, but not limited to, VHDL, Verilog, SystemC, SystemVerilog, RHDL, M, MyHDL, etc. The formats of various design information can be recognized by one or more applications executed by computing system 1140, semiconductor manufacturing system 1120, or both. In some embodiments, design information 1115 may also include one or more cell libraries specifying the composition, layout, or both of integrated circuit 1130. In some embodiments, design information 1115 is specified wholly or partially in the form of a netlist specifying cell library elements and their connectivity. Individually, the design information discussed herein may or may not include sufficient information for manufacturing the corresponding integrated circuit. For example, design information may specify circuit elements to be manufactured but not their physical layout. In this case, design information may need to be combined with layout information to actually manufacture the specified circuit.

[0112] Code sequence 1170 may include one or more instructions utilizing computation engine 103. In some cases, code sequence 1170 may include one or more instances of vector outer product and accumulation instructions as described above.

[0113] In various implementations, integrated circuit 1130 may include one or more custom macrocells, such as memory, analog or mixed-signal circuitry, etc. In such cases, design information 1115 may include information associated with the included macrocells. Such information may include, but is not limited to, schematic capture databases, mask design data, behavioral models, and device or transistor-level netlists. Mask design data may be formatted according to a Graphical Data System (GDSII) or any other suitable format.

[0114] The semiconductor manufacturing system 1120 may include any of the various suitable elements configured to manufacture integrated circuits. This may include, for example, elements for depositing semiconductor material (e.g., on a wafer that may include a mask), removing material, changing the shape of the deposited material, modifying the material (e.g., by doping the material or by using ultraviolet treatment to modify the dielectric constant), etc. The semiconductor manufacturing system 1120 may also be configured to perform various tests on the manufactured circuits to ensure proper operation.

[0115] In various implementations, integrated circuit 1130 and computer simulation model 1160 are configured to operate according to a circuit design specified by design information 1115, which may include performing any of the functionalities described herein. For example, integrated circuit 1130 may include Figures 1 to 4Any of the various components shown. Additionally, integrated circuit 1130 can be configured to perform the various functions described herein in conjunction with other components. Furthermore, the functionality described herein can be performed by multiple interconnected integrated circuits.

[0116] As used herein, a phrase in the form of “design information specifying a circuit configured to…” does not imply that the circuit in question must be manufactured to satisfy the element. Rather, the phrase indicates that the design information describes a circuit that, when manufactured, will be configured to perform the indicated actions or will include the specified components. Similarly, stating that “instructions of a hardware description programming language” are “executable” for programming a computing system to generate a computer simulation model does not mean that the instructions must be executed to satisfy the element, but rather specifies the characteristics of those instructions. In this context, additional features associated with the model (or the circuit represented by the model) can similarly relate to the characteristics of these instructions. Therefore, an entity selling a computer-readable medium having instructions that satisfy the stated characteristics may provide an infringing product even if another entity actually executes those instructions on the medium.

[0117] It is important to note that a given design, at least within the context of digital logic, can be implemented using multiple different gate arrangements, circuit techniques, etc. As an example, different designs may select or connect gates based on design trade-offs (e.g., focusing on power consumption, performance, circuit area, etc.). Furthermore, different manufacturers may have proprietary libraries, gate designs, physical gate implementations, etc. Different entities may also use different tools to process design information at various layers (e.g., from behavioral specifications to the physical layout of gates).

[0118] However, once a digital logic design is specified, those skilled in the art do not need to perform extensive experiments or studies to determine these implementations. Instead, they understand the process for reliably and predictably producing one or more circuit implementations that provide the functionality described by design information 1115. Different circuit implementations may affect the performance, area, power consumption, etc., of a given design (potentially trade-offs between different design goals), but the logic function remains unchanged between different circuit implementations of the same circuit design.

[0119] In some implementations, instructions included in design information 1115 provide RTL information (or other higher-level design information) and can be executed by a computing system to synthesize a gate-level netlist representing the hardware circuitry based on the RTL information as input. Similarly, these instructions can provide behavioral information and can be executed by a computing system to synthesize a netlist or other lower-level design information included in low-level design information 1150. Low-level design information 1150 can program semiconductor manufacturing system 1120 to manufacture integrated circuit 1130.

[0120]

[0121] This disclosure includes references to “implementation” or groups of “implementation” (e.g., “some implementations” or “various implementations”). An implementation is a different specific implementation or instance of the disclosed concepts. References to “implementation,” “an implementation,” “a particular implementation,” etc., do not necessarily refer to the same implementation. A large number of possible implementations are contemplated, including those specifically disclosed, as well as modifications or alternatives that fall within the substance or scope of this disclosure.

[0122] This disclosure may discuss potential advantages that may arise from the disclosed embodiments. Not all specific implementations of all these embodiments will necessarily exhibit any or all of the potential advantages. Whether a particular embodiment achieves an advantage depends on many factors, some of which are outside the scope of this disclosure. In fact, there are many reasons why an embodiment falling within the scope of the claims may not exhibit some or all of any of the disclosed advantages. For example, a particular embodiment may include other circuitry outside the scope of this disclosure, in conjunction with one embodiment of the disclosed embodiments, which negates or diminishes one or more of the disclosed advantages. Furthermore, suboptimal design execution of a particular embodiment (e.g., the implementing technique or tool) may also negate or diminish the disclosed advantages. Even assuming an implementation of the technique, the realization of advantages may still depend on other factors, such as the environmental circumstances in which the implementation is deployed. For example, the inputs provided to a particular embodiment may prevent one or more problems addressed in this disclosure from occurring in a particular context, and as a result, the benefits of its solution may not be realized. In view of the existence of possible factors outside this disclosure, it is hereby expressed that any potential advantages described herein should not be construed as a limitation of the claims that must be satisfied in order to prove infringement. Rather, the identification of such potential advantages is intended to illustrate the types of improvements available to the designer who benefits from this disclosure. Describing such advantages permanently (e.g., stating that a particular advantage "may occur") is not intended to convey a question about whether such advantages can actually be realized, but rather to recognize that the realization of such advantages often depends on the technological reality of additional factors.

[0123] Unless otherwise stated, the embodiments are non-limiting. That is, the disclosed embodiments are not intended to limit the scope of the claims drafted based on this disclosure, even where only a single example is described with respect to a particular feature. The disclosed embodiments are intended to be illustrative and not restrictive, without any statement to the contrary in this disclosure. Therefore, this application is intended to allow for claims covering the disclosed embodiments, as well as such alternatives, modifications, and equivalents, which will be apparent to those skilled in the art to the advantage of this disclosure.

[0124] For example, features in this application can be combined in any suitable manner. Therefore, new claims may be made for any such combination of features during the filing of this application (or an application claiming priority thereto). Specifically, referring to the appended claims, features of dependent claims may be combined with features of other dependent claims, including claims dependent on other independent claims, where appropriate. Similarly, features from the respective independent claims may be combined where appropriate.

[0125] Accordingly, while the appended dependent claims are drafted such that each claim depends on a single other claim, additional dependency relationships are also contemplated. Any combination of features of the dependent claims consistent with the present disclosure is contemplated, and such combinations may be claimed in this patent application or another patent application. In short, combinations are not limited to those specifically recited in the appended claims.

[0126] Where appropriate, it is also contemplated that claims drafted in one format or statutory type (e.g., apparatus) are intended to support corresponding claims in another format or statutory type (e.g., method).

[0127] Since the present disclosure is a legal document, various terms and phrases may be subject to regulatory and judicial interpretation. Notice is hereby given that the following paragraphs, along with definitions provided throughout this disclosure, will be used to determine how to interpret claims drafted based on the present disclosure.

[0128] Unless the context clearly dictates otherwise, reference to an item in the singular (i.e., a noun or noun phrase preceded by "a", "an" or "the") is intended to mean "one or more". Therefore, reference to "an element" in a claim, absent contrary context, does not exclude additional instances of the element. A "plurality" of items refers to a set of two or more of the items.

[0129] The word "may" is used herein in the permissive sense (i.e., having the potential to, capable of) rather than the mandatory sense (i.e., must).

[0130] The terms "comprise" and "include" and their forms are open-ended and mean "including but not limited to".

[0131] When the term "or" is used in the present disclosure with reference to a list of options, unless the context indicates otherwise, it is generally understood to be used in the inclusive sense. Accordingly, the expression "x or y" is equivalent to "x or y, or both", and thus covers 1) x but not y, 2) y but not x, and 3) both x and y. On the other hand, phrases such as "either x or y, but not both" make clear that "or" is used in the exclusive sense.

[0132] The expressions “w, x, y, or z, or any combination thereof” or “...at least one of w, x, y, and z” are intended to cover all possibilities involving a single element up to the total number of elements in the set. For example, given the set [w, x, y, z], these phrases cover any single element in the set (e.g., w but not x, y, or z), any two elements (e.g., w and x, but not y or z), any three elements (e.g., w, x, and y, but not z), and all four elements. The phrase “...at least one of w, x, y, and z” therefore refers to at least one element in the set [w, x, y, z], thus covering all possible combinations of that list of elements. This phrase should not be interpreted as requiring the existence of at least one instance of w, at least one instance of x, at least one instance of y, and at least one instance of z.

[0133] In this disclosure, various “labels” may precede nouns or noun phrases. Unless the context otherwise provides, different labels used for features (e.g., “first circuit,” “second circuit,” “specific circuit,” “given circuit,” etc.) refer to different instances of the feature. Furthermore, unless otherwise stated, the labels “first,” “second,” and “third” do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) when applied to features.

[0134] The phrase "based on" is used to describe one or more factors that influence the determination. This term does not exclude the possibility that additional factors might influence the determination. That is, the determination may be based solely on the specified factors or on the specified factors along with other unspecified factors. Consider the phrase "A is determined based on B." This phrase specifies that B is a factor used to determine A or that B influences the determination of A. This phrase does not exclude the possibility that the determination of A may also be based on some other factor, such as C. This phrase is also intended to cover implementations where A is determined solely based on B. As used herein, the phrase "based on" is synonymous with the phrase "at least partially based on."

[0135] The phrases “responding to” and “responding” describe one or more factors that trigger an effect. This phrase does not exclude the possibility that additional factors may influence or otherwise trigger the effect, whether these factors are used in conjunction with or independently of the specified factor. That is, the effect may respond solely to these factors, or it may respond to the specified factor as well as other unspecified factors. Consider the phrase “responding to B and executing A.” This phrase specifies that B is a factor that triggers the execution of A or triggers a specific result of A. This phrase does not exclude that the execution of A may also respond to certain other factors, such as C. This phrase also does not exclude that the execution of A may be jointly executed in response to B and C. This phrase is also intended to cover implementation schemes where A is executed solely in response to B. As used herein, the phrase “responding” is synonymous with the phrase “at least partially responding to.” Similarly, the phrase “responding to” is synonymous with the phrase “at least partially responding to.”

[0136] Within this disclosure, different entities (which may be referred to differently as “units,” “circuits,” other components, etc.) may be described or protected by the claims as being “configured” to perform one or more tasks or operations. This expression—[entity] configured to [perform one or more tasks]—is used herein to refer to a structure (i.e., a tangible thing). More specifically, this expression is used to indicate that the structure is arranged to perform one or more tasks during operation. A structure may be considered “configured” to perform a task even if the structure is not currently being operated. Thus, an entity described or stated as being “configured” to perform a task refers to a tangible thing, such as a device, circuit, or system having processor units and memory storing program instructions executable to perform the task. This phrase is not used herein to refer to intangible things.

[0137] In some cases, various units / circuits / components may be described herein as a collection of entities that perform tasks or operations. It should be understood that these entities are "configured" to perform those tasks / operations, even if not specifically stated otherwise.

[0138] The term "configured as" is not intended to mean "able to be configured as." For example, an unprogrammed FPGA is not considered "configured as" to perform a specific function. However, the unprogrammed FPGA can be "configurable as" to perform that function. After proper programming, the FPGA can then be considered "configured as" to perform a specific function.

[0139] For the purposes of this U.S. patent application based on this disclosure, the statement in the claims that a structure is “configured” to perform one or more tasks is expressly intended not to invoke 35 USC § 112(f) for that claim element. If an applicant wishes to invoke part 112(f) during the filing of a U.S. patent application based on this disclosure, it will use the “component for [performing a function]” structure to state the elements of the claims.

[0140] Different “circuits” may be described in this disclosure. These circuits, or “circuits,” constitute hardware that includes various types of circuit elements, such as combinational logic, clock storage devices (e.g., flip-flops, registers, latches, etc.), finite state machines, memories (e.g., random access memory, embedded dynamic random access memory), programmable logic arrays, etc. Circuits may be custom-designed or taken from standard libraries. In various specific implementations, circuits may include digital components, analog components, or a combination of both, depending on the circumstances. Certain types of circuits may be commonly referred to as “cells” (e.g., decoding units, arithmetic logic units (ALUs), functional units, memory management units (MMUs), etc.). Such cells also refer to circuits.

[0141] Therefore, the circuits / units / components and other elements illustrated in the accompanying drawings and described herein include hardware elements, such as those described in the preceding paragraphs. In many cases, the internal arrangement of hardware elements in a particular circuit can be specified by describing the function of that circuit. For example, a particular “decoding unit” can be described as having the function of executing “the opcode of a processing instruction and routing that instruction to one or more of a plurality of functional units,” meaning that the decoding unit is “configured” to perform that function. To those skilled in the art of computers, this functional specification is sufficient to suggest a set of possible structures for the circuit.

[0142] In various implementations, as discussed in the preceding paragraphs, circuits, cells, and other elements can be defined by the functions or operations they are configured to perform. The arrangement of such circuits / cells / components relative to each other and the way they interact form a microarchitecture definition of hardware, which is ultimately manufactured in an integrated circuit or programmed into an FPGA to form a physical implementation of the microarchitecture definition. Therefore, a microarchitecture definition is considered by those skilled in the art to be a structure from which many physical implementations can be derived, all of which fall within the broader structure described by the microarchitecture definition. That is, those skilled in the art, with the microarchitecture definition provided according to this disclosure, can implement this structure without excessive experimentation and using the application of a person of ordinary skill in the art, by encoding the description of the circuits / cells / components in a hardware description language (HDL) such as Verilog or VHDL. The HDL description is often expressed in a way that can be revealed as functional. However, for those skilled in the art, this HDL description is a way of transforming the structure of a circuit, cell, or component into the details of the next level of implementation. Such HDL descriptions can take the following forms: behavioral code (which is typically non-synthesizable), register-passing language (RTL) code (which is typically synthesizable compared to behavioral code), or structural code (e.g., a netlist specifying logic gates and their connectivity). HDL descriptions can be sequentially synthesized against a library of cells designed for a given integrated circuit manufacturing technology and can be modified for timing, power, and other reasons to obtain a final design database that is sent to the factory to generate masks and ultimately produce integrated circuits. Some hardware circuitry or portions thereof can also be custom-designed in a schematic editor and captured into the integrated circuit design along with the synthesized circuitry. The integrated circuit can include transistors and other circuit elements (e.g., passive components such as capacitors, resistors, inductors, etc.), as well as interconnections between transistors and circuit elements. Some implementations may implement multiple integrated circuits coupled together to implement the hardware circuitry, and / or discrete components may be used in some implementations. Alternatively, the HDL design can be synthesized into a programmable logic array such as a field-programmable gate array (FPGA) and can be implemented in the FPGA. This decoupling between the design of a set of circuits and their subsequent low-level implementations often leads to scenarios where circuit or logic designers never specify a particular set of structures for the low-level implementations that goes beyond a description of what the circuits are configured to do, because that process is performed at different stages of the circuit implementation process.

[0143] The fact that a circuit can be implemented to the same specifications using many different low-level combinations of circuit elements results in a large number of equivalent circuit structures. As noted, these low-level circuit implementations can vary depending on the manufacturing technology, the foundry chosen to manufacture the integrated circuit, the cell library provided for a particular project, and so on. In many cases, the choice of different design tools or methods to produce these different implementations can be arbitrary.

[0144] Furthermore, for a given implementation, a single concrete implementation of the circuit's specific functional specifications typically involves a large number of devices (e.g., millions of transistors). Therefore, the shearing volume of this information makes it impractical to provide a complete description of the low-level structure used to implement a single implementation, let alone a large number of equivalent possible implementations. To this end, this disclosure describes the structure of a circuit using functional abbreviations commonly used in industry.

[0145] Appendix A Pseudocode for cross-switch outer product and accumulation instructions

[0146]

Claims

1. An apparatus comprising: Multiple register circuits, the multiple register circuits being configured to store corresponding vectors among multiple vectors; and A computing engine comprising a plurality of computing elements arranged in an array of rows and columns, wherein the computing engine is configured to: A first portion of a first input vector is selected from a first register circuit among the plurality of register circuits using a selection vector stored in a specific register circuit among the plurality of register circuits; The selection vector is used to select a second portion of the second input vector from the second register circuit among the plurality of register circuits; as well as Using the plurality of computing elements, a plurality of computations are performed in parallel using a first portion of the first input vector, a second portion of the second input vector, and a weight vector, wherein the plurality of computations include a vector outer product computation, and wherein the weight vector is stored in different register circuits in the plurality of register circuits.

2. The apparatus of claim 1, further comprising a first number of zero elements in the first portion of the first input vector being less than a threshold, and further comprising a second number of zero elements in the second portion of the second input vector being less than the threshold.

3. The apparatus of claim 1, wherein the computing circuitry of a specific row of the plurality of computing elements is coupled to a first communication bus, and wherein the computing circuitry of different rows of the plurality of computing elements is coupled to the first communication bus.

4. The apparatus of claim 3, wherein the computing engine is further configured to: During the first pass, the first communication bus is used to transfer specific elements of the first portion of the first input vector to the corresponding computational circuitry included in the computational circuitry of the specific row; and During the second pass, the first communication bus is used to transmit different elements of the first portion of the first input vector to the corresponding computation circuits included in the computation circuits of the different rows.

5. The apparatus of claim 1, wherein, in order to perform the plurality of calculations, the calculation engine is further configured to perform a corresponding operation of the plurality of calculations using specific elements of the first portion of the first input vector and corresponding elements of the weight vector.

6. The apparatus of claim 1, wherein a given calculation in the plurality of calculations includes multiplication and accumulation operations.

7. A method comprising: A computing engine included in a computer system uses a selection vector from a specific register circuit among a plurality of register circuits to select a first portion of a first input vector from different register circuits among the plurality of register circuits, wherein the computing engine includes a plurality of computing circuits arranged in an array of rows and columns. The computing engine uses the selection vector to select a second portion of the second input vector from another register circuit among the plurality of register circuits; as well as The plurality of computing elements perform a plurality of computations in parallel using a first portion of the first input vector, a second portion of the second input vector, and a weight vector, wherein the plurality of computations include a vector outer product computation, and wherein the weight vector is stored in a given register circuit among the plurality of register circuits.

8. The method of claim 7, further comprising that a first number of zero elements in the first portion of the first input vector is less than a threshold, and further comprising that a second number of zero elements in the second portion of the second input vector is less than the threshold.

9. The method of claim 7, wherein a specific row of the plurality of computing circuits is coupled to a first communication bus, and wherein different rows of the plurality of computing circuits are coupled to the first communication bus.

10. The method according to claim 9, further comprising: The computing engine uses the first communication bus during the first pass to transmit specific elements of the first portion of the first input vector to the corresponding computing circuit included in the computing circuit of the specific row. as well as During the second pass, the computing engine uses the first communication bus to transmit different elements of the first portion of the first input vector to the corresponding computing circuits included in the computing circuits of the different rows.

11. The method of claim 7, wherein performing the plurality of calculations includes performing a corresponding operation of the plurality of operations using specific elements of the first portion of the first input vector and corresponding elements of the weight vector.

12. The method of claim 7, wherein a given calculation in the plurality of calculations includes multiplication and accumulation operations.

13. The method of claim 7, wherein the weight vector is included in a weight matrix, and includes a number of zero elements in the weight matrix exceeding a threshold.

14. A system comprising: Processor circuitry, the processor circuitry being configured to: In response to executing a first instruction to retrieve an input matrix and a weight matrix, wherein the input matrix includes a plurality of input vectors, wherein the weight matrix includes a plurality of weight vectors, and wherein the number of zero elements in the weight matrix exceeds a threshold; as well as In response to executing the first instruction, a packed weight vector comprising an orthogonal subset of the plurality of weight vectors is generated; and A computation engine configured to perform multiple computations in parallel using a subset of a plurality of input vectors corresponding to a subset of the plurality of weight vectors included in the packaged weight vectors, wherein the computation engine includes a plurality of computational elements.

15. The system of claim 14, wherein the plurality of calculations includes a plurality of corresponding vector multiplication operations.

16. The system of claim 14, wherein, in order to generate the packaged weight vector, the processor circuitry is further configured to store information in a lookup table, wherein the information maps specific weights included in the subset of the plurality of weight vectors to corresponding input vectors included in the subset of the plurality of input vectors.

17. The system of claim 16, wherein, in order to perform the plurality of calculations, the calculation engine is further configured to retrieve the information from the lookup table.

18. The system of claim 16, wherein a given computing element of the plurality of computing elements is configured to store a specific weight value included in the packaged weight vector based on the information stored in the lookup table.

19. The system of claim 14, wherein the processor circuitry is further configured to: Use the first instruction to generate the second instruction; and The second instruction is transmitted to the computing engine.

20. The system of claim 19, wherein the computing engine is further configured to perform the plurality of computing in response to receiving the second instruction.

Images