Lossy sparsely loaded simd instruction family

Abstract

Systems, apparatus, and methods for implementing a lossy sparse loading single instruction multiple data (SIMD) instruction family are disclosed. A lossy sparse loading unit (LSLU) loads multiple values ​​from one or more input vector operands and determines how many non-zero values ​​are included in the one or more input vector operands for a given instruction. If the one or more input vector operands have less than a threshold number of non-zero values, the LSLU causes the instruction for processing the one or more input vector operands to be skipped. In this case, the processing of the one or more input vector operands of the instruction is considered redundant. If the one or more input vector operands have more than or equal to the threshold number of non-zero values, the LSLU causes the instruction for processing the one or more input vector operands to be executed.

Description

Background Technology

[0001] Description of related technologies

[0002] A new emerging technology area is machine learning, in which neural networks are one type of machine learning model. Neural networks have demonstrated excellent performance in tasks such as handwritten digit classification and face detection. Furthermore, they have shown potential for good performance in other, more challenging visual classification tasks. Other applications of neural networks include speech recognition, language modeling, sentiment analysis, and text prediction.

[0003] As is well known, deep neural networks (DNNs) exhibit sparsity or zero values ​​in their various data structures. For example, activations in ResNet-50 and AlexNet exhibit average sparsity of 58% and 55%, respectively, while weights in DeepCompression AlexNet exhibit 65% sparsity during the inference phase. Zero values ​​in DNN data structures lead to unnecessary and inefficient multiply-accumulate (MAD) operations (potentially part of multiply-accumulate (MAC) operations). This results in inefficient implementations of DNNs on typical hardware platforms. Attached Figure Description

[0004] The advantages of the methods and mechanisms described herein can be better understood by referring to the following description in conjunction with the accompanying drawings, in which:

[0005] Figure 1 It is a block diagram of one implementation method of a computing system.

[0006] Figure 2 This is a block diagram of another implementation of a computing system.

[0007] Figure 3 It is a block diagram of one implementation method of the computing unit logic.

[0008] Figure 4 This is a generalized flowchart illustrating an implementation of a method for executing vector lossy combination sparse loading instructions.

[0009] Figure 5 This is a generalized flowchart illustrating an implementation of a method for executing a vector-lossy single sparse load instruction.

[0010] Figure 6 This is a generalized flowchart illustrating one implementation of a method for processing an array of cumulative non-zero counts.

[0011] Figure 7 This is a generalized flowchart illustrating an implementation of a method for performing lossy sparse loading and skipping instructions for vectors.

[0012] Figure 8 This is a generalized flowchart illustrating one implementation of a method for executing lossy sparse loading instructions.

[0013] Figure 9 This is a generalized flowchart illustrating one implementation of a method for implementing a neural network.

[0014] Figure 10 An example of pseudocode for implementing inner and outer product matrix multiplication operations is shown, based on one implementation method.

[0015] Figure 11 An example of pseudocode for implementing a vector lossy combination sparse loading instruction is shown, based on one implementation method.

[0016] Figure 12 This is a block diagram of one implementation of the logic for implementing the lossy combination sparse loading instruction for vectors.

[0017] Figure 13 An example of pseudocode for implementing a single sparse loading instruction with a lossy vector is shown, based on one implementation method.

[0018] Figure 14 An example of pseudocode for implementing lossy sparse loading and skip instructions for vectors, based on one implementation method, is shown. Detailed Implementation

[0019] In the following description, numerous specific details are set forth to provide a comprehensive understanding of the methods and mechanisms presented herein. However, those skilled in the art will recognize that various implementations can be practiced without these specific details. In some cases, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail to avoid obscuring the methods described herein. It should be understood that, for the sake of simplicity and clarity, the elements shown in the figures are not necessarily drawn to scale. For example, the dimensions of some elements may be enlarged relative to other elements.

[0020] This paper discloses various systems, devices, and methods for implementing a lossy sparse-loaded Single Instruction Multiple Data (SIMD) instruction family. Deep neural network (DNN) data structures typically include a certain amount of sparsity (i.e., zero values). In various implementations, the systems utilize the SIMD lossy sparse-loaded instruction family and associated microarchitectural extensions. This instruction family optimizes processor performance and power consumption for the dynamic sparsity encountered during DNN training and inference. In one implementation, redundant multiply-accumulate (MAC) or multiply-add (MAD) operations are identified and eliminated by subjecting the input vector operands to a user-defined mask. The user-defined mask may optionally impose additional sparsity on the input data structure to improve performance and power gains by leveraging the resilience of a particular DNN. In various implementations, the SIMD scheduler skips instructions in blocks based on wavefronts (i.e., twists) or selectively skips instructions via a skip queue.

[0021] A system includes at least a processor having a plurality of computing units coupled to one or more memories. In one implementation, each computing unit includes a lossy sparse loading unit and a plurality of processing elements. In one implementation, the processing elements are SIMD units. In one implementation, the lossy sparse loading unit determines how many non-zero values ​​exist in one or more input vector operands of a pending instruction. In one implementation, in response to determining that the number of non-zero values ​​in one or more input vector operands is greater than or equal to a threshold, the lossy sparse loading unit causes the plurality of processing elements to process one or more input vector operands. Otherwise, if the number of non-zero values ​​in one or more input vector operands is less than the threshold, the lossy sparse loading unit causes the plurality of processing elements to skip processing one or more input vector operands. In one implementation, the threshold is programmable.

[0022] In one implementation, the system receives program code for a neural network for execution by the system's processing element, wherein the program code includes one or more first instructions and one or more second instructions. The system executes the first instructions on the processing element based on the number of non-zero values ​​in the input vector operands of the first and second instructions, and skips the execution of the second instructions. The system implements the neural network by executing the first instructions and skipping the second instructions. In one implementation, the system implements the neural network to generate a classification of a first dataset. It should be noted that in the above representation, the first and second instructions are not pre-identified. Instead, the first instructions are identified at runtime based on the sparsity of their operands having a sparsity less than a threshold amount, while the second instructions are identified at runtime based on the sparsity of their operands having a sparsity greater than or equal to a threshold amount. In various implementations, the sparsity threshold amount can be configured by the user based on the error tolerance of the neural network implementation.

[0023] Now for reference Figure 1 The diagram illustrates a block diagram of one implementation of a computing system 100. In one implementation, the computing system 100 includes at least a neural network 105, one or more processors 110, an input / output (I / O) interface 120, a bus 125, and a memory subsystem 130. In other implementations, the computing system 100 may include other components and / or the computing system 100 may be arranged differently. In various implementations, the neural network 105 includes logic for implementing any of various machine learning algorithms or machine learning models. In one implementation, the neural network 105 implements one or more layers of a convolutional neural network. For example, in this implementation, the neural network 105 implements one or more convolutional layers and / or one or more fully connected layers. In another implementation, the neural network 105 implements one or more layers of a recurrent neural network. In various implementations, the neural network 105 is trained using a backpropagation algorithm via stochastic gradient descent (SGD). In one implementation, the logic of the neural network 105 includes multiple multiply-accumulate (MAC) units, peripheral devices, and internal memory storage devices. Depending on the implementation, any of the various software deep learning frameworks (e.g., Caffe, TensorFlow, Torch) can be used to train neural networks on a specific processing unit (e.g., a graphics processing unit (GPU)) 105.

[0024] The neural network 105 is used in a variety of different applications, depending on the implementation. For example, in one implementation, the neural network 105 analyzes video frames to generate one or more labeled probabilities for those video frames. Possible use cases include, for example, eye tracking, object recognition, point cloud estimation, ray tracing, light field modeling, depth tracking, etc. For the eye tracking use case, the probabilities generated by the neural network 105 are based on learned patterns, dwell times, transition angles, blinks, etc. In other implementations, the neural network 105 is trained and customized for other types of use cases.

[0025] Generally, neural network implementations exhibit two broad types of sparsity. The first sparsity is static sparsity of weights caused by pruning the network. The second sparsity is dynamic sparsity of activations and errors during training. Activation sparsity is caused by the presence of activation functions such as Rectified Linear Units (ReLUs), which return negative inputs to zero. Conversely, error sparsity stems from the presence of both activation functions and max-pooling layers, because zeros can be backpropagated for negative inputs of ReLU activation functions and non-maximum inputs of max-pooling layers. To leverage the static and dynamic sparsity present in the data structures processed by the neural network 105, this disclosure introduces an instruction family and associated microarchitectural extensions. These instruction family and associated microarchitectural extensions enable the neural network 105 to improve performance and reduce power consumption.

[0026] One or more processors 110 represent any number and type of processing units (e.g., central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC)). In one implementation, a portion of the processing associated with neural network 105 is executed by one or more processors 110. Additionally, neural network 105 is implemented using any of these types of processing units and / or other types of processing elements. Memory subsystem 130 represents any number and type of memory devices. For example, the types of memory in memory subsystem 130 may include high-bandwidth memory (HBM), non-volatile memory (NVM), dynamic random access memory (DRAM), static random access memory (SRAM), NAND flash memory, NOR flash memory, ferroelectric random access memory (FeRAM), etc. Neural network 105 and one or more processors 110 can access memory subsystem 130. I / O interface 120 represents any number and type of I / O interface (e.g., Peripheral Component Interconnect (PCI) bus, PCI Expansion (PCI-X), PCIe (PCI Fast) bus, Gigabit Ethernet (GBE) bus, Universal Serial Bus (USB)). Various types of peripheral devices can be connected to I / O interface 120. Such peripheral devices include (but are not limited to) monitors, keyboards, mice, printers, scanners, joysticks or other types of game controllers, media recording devices, external storage devices, network interface cards, etc.

[0027] In various implementations, computing system 100 is any of a computer, laptop, mobile device, game console, server, streaming device, wearable device, or various other types of computing system or device. It should be noted that the number of components in computing system 100 varies depending on the implementation. For example, in other implementations, there are components related to… Figure 1 The number shown is compared to more or fewer of each component. It should also be noted that in other implementations, the computing system 100 includes... Figure 1 Other components not shown. Additionally, in other implementations, the computing system 100 is coupled with... Figure 1 The different ways of structuring are shown.

[0028] Turn now Figure 2 A block diagram of another implementation of the computing system 200 is shown. In one implementation, the system 200 includes a GPU 205, system memory 225, and local memory 230. In one implementation, ( Figure 1The neural network 105 executes on the GPU 205. System 200 also includes other components not shown to avoid obscuring the figures. GPU 205 includes at least an instruction processor 235, control logic 240, dispatch units 250, compute units 255A to 255N, a memory controller 220, a global data sharing mechanism 270, a Level 1 (L1) cache 265, and a Level 2 (L2) cache 260. In other implementations, GPU 205 includes other components, one or more of the components shown are omitted, or multiple instances of the components are present (although in...). Figure 2 (Only one instance is shown in the text) and / or organized in other suitable ways.

[0029] In various implementations, computing system 200 executes any of various types of software applications. As part of executing a given software application, the host CPU (not shown) of computing system 200 starts a kernel to be executed on GPU 205. Command processor 235 receives the kernel from the host CPU and uses dispatch unit 250 to dispatch the kernel to computing units 255A to 255N. Control logic 240 monitors various resources of GPU 205 and assists dispatch unit 250 in determining how to dispatch the kernel to computing units 255A to 255N. Threads within the kernel executing on computing units 255A to 255N read data and write data to global data share 270, L1 cache 265, and L2 cache 260 within GPU 205. Although Figure 2 Not shown, but in one implementation, computing units 255A to 255N also include one or more caches and / or local memories within each computing unit 255A to 255N.

[0030] Now for reference Figure 3 The diagram illustrates a block diagram of one implementation of the computing unit logic 300. In one implementation, the computing unit logic 300 includes ( Figure 2The logic 300 is located in each of the computing units 255A to 255N. In one implementation, the logic 300 includes a lossy sparse load unit (LSLU) 305, a memory 310, a vector general-purpose register (VGPR) 315, a single instruction multiple data (SIMD) unit 320, and a scheduler 325. It should be noted that the LSLU 305 may also be referred to herein as a “load unit.” It should also be noted that the SIMD unit 320 may also be referred to herein as a “processing element.” In one implementation, the logic 300 is included within a graphics processing unit (GPU). In another implementation, the logic 300 is included within a field-programmable gate array (FPGA). In yet another implementation, the logic 300 is included within an application-specific integrated circuit (ASIC). In still other implementations, the logic 300 is included within other types of processing units, computing devices, and / or computing systems. It should be noted that the logic 300 may also be referred to herein as “control logic.”

[0031] In one implementation, LSLU 305 loads the operand from memory 310 into VGPR 315 and then determines how many non-zero values ​​are present in the input vector operand. In another implementation, LSLU 305 determines how many non-zero values ​​are present in the input vector operand before loading the input vector operand from memory 310 into VGPR 315. It should be noted that in one implementation, the term "non-zero value" is defined as a value that is not equal to zero. In another implementation, the term "non-zero value" is defined as a value greater than a threshold or whose absolute value is greater than a threshold. For example, in one implementation, the threshold is a programmable small positive value (e.g., 0.1). In some implementations, if the number of non-zero values ​​in a given input vector operand is less than the threshold number, LSLU 305 will not load the given vector operand from memory 310 into VGPR 315. This threshold number is represented in LSLU 305 as a mask / threshold 306. The comparison logic used to compare the number of non-zero values ​​with the threshold number is shown as comparator 307 in LSLU 305. In other implementations, even if the number of non-zero values ​​in the given input vector operand is less than the threshold number, LSLU 305 will still load the given input vector operand from memory 310 into VGPR 315 or buffer 308. However, if the number of non-zero values ​​in the given input vector operand is less than the threshold number, LSLU 305 will discard, invalidate, and / or overwrite the given input vector operand in VGPR 315.

[0032] Logic 300 also shows a scheduler 325 that issues instructions for execution on the SIMD unit 320. In one implementation, the SIMD unit 320 performs matrix multiplication on the input vector operands of the instructions issued for execution by the scheduler 325. The matrix multiplication can be an inner product or an outer product matrix multiplication, depending on the type of instruction to be executed. In other implementations, the SIMD unit 320 performs other types of operations on the input vector operands of the instructions issued for execution by the scheduler 325. In one implementation, if the LSLU 305 determines that one or more input vector operands of a given instruction have fewer than a threshold number of non-zero values, the scheduler 325 will not schedule the given instruction on the SIMD unit 320. Instead, the scheduler 325 will continue to the next instruction. By skipping instructions with fewer than a threshold number of non-zero values, the efficiency of the neural network implementation on the SIMD unit 320 is improved.

[0033] Turn now Figure 4 This illustrates one implementation of a method 400 for executing vector lossy combination sparse loading instructions. For discussion purposes, the steps in this implementation are shown in sequence. Figures 5 to 9 Those steps. However, it should be noted that in various implementations of the described method, one or more of the described elements may be performed simultaneously, in a different order than shown, or completely omitted. Other additional elements may also be performed as needed. Any of the various systems or devices described herein is configured to implement method 400.

[0034] The Lossy Sparse Loading Unit (LSLU) loads input vector operands A and B, along with the current index and the maximum index of the dataset (box 405). Next, the LSLU determines the number of non-zero values ​​in each of the input vector operands A and B (box 410). Then, if the number of non-zero values ​​in each input vector operand is less than a threshold, and the current index in the dataset is less than the maximum index (condition box 415, "Yes" branch), the LSLU increments the pointers by a step to the address of the input vector operands, and the LSLU increments the current dataset index (box 420). After box 420, method 400 returns to box 410. If the number of non-zero values ​​in either input vector operand is greater than or equal to the threshold, or if the current index in the dataset is equal to the maximum index (condition box 415, "No" branch), the LSLU returns the values ​​of input vector operand A and input vector operand B to the vector register file (box 425). After box 425, method 400 ends. In one implementation, the target of the vector lossy combination sparse load instruction is the simplest inner and outer product implementation with a block size of 1, which iteratively loads one operand for each of A and B before performing a multiply-accumulate (MAC) operation on the operands.

[0035] Now for reference Figure 5 This illustrates one implementation of a method 500 for executing a lossy single sparse load instruction for a vector. The LSLU receives a load instruction for an input vector operand at a specified address, an operand identifier (ID), and a value of N, where the N value specifies the total number of input vector operands (box 505). Next, the LSLU sets the variable “i” to 0 (box 510), and then checks if the variable “i” is less than the total number of threads before the wave (condition box 515). If the variable “i” is less than the total number of threads (condition box 515, “Yes” branch), the LSLU loads the input vector operands from memory into a vector register file, and the LSLU counts the number of non-zero values ​​in the input vector operands and stores that count in the “accNZCount” array (box 520). Next, the LSLU increments the variable “i” (box 525), and then method 500 returns to condition box 515. If the variable “i” equals the number of threads (condition box 515, “No” branch), the LSLU returns the value of the vector “v” (box 530). After box 530, method 500 ends. The following is in conjunction with... Figure 6 An implementation for handling the “accNZCount” array is described in the related discussion.

[0036] Now go to Figure 6This illustrates one implementation of method 600 for processing an array of accumulated non-zero counts (i.e., accNZCount). In one implementation, method 600 is executed after executing the vector lossy single sparse load instruction described in method 500. The LSLU receives the accNZCount array (box 605). Next, the variable “i” is initialized to zero (box 610). Then, the LSLU determines whether the variable “i” is less than the total number of threads before the wave (condition box 615).

[0037] If the variable "i" is less than the total number of threads before the wave (condition box 615, "Yes" branch), then LSLU determines whether the number of non-zero values ​​in each of the two input operands is less than a threshold (i.e., NZThres) (condition box 620). If the non-zero count of each of the two input operands is less than the threshold (condition box 620, "Yes" branch), then the thread redundancy indicator is set to 1 for the current index "i" (box 625). Otherwise, if the non-zero count of either of the two input operands is greater than or equal to the threshold (condition box 620, "No" branch), then the thread redundancy indicator is set to 0 for the current index "i" (box 630). After boxes 625 and 630, the current index "i" is incremented (box 635), and then method 600 returns to condition box 615.

[0038] If the variable "i" equals the total number of threads (condition box 615, "No" branch), then the LSLU determines whether the entire wavefront is redundant by performing a bitwise AND operation on multiple thread redundancy indicators (box 640). The LSLU then returns a redundant wavefront value indicating whether the entire wavefront is redundant (box 645). After box 645, method 600 ends.

[0039] Now for reference Figure 7 This illustrates one implementation of a method 700 for performing lossy sparse loading and skipping instructions for vectors. The LSLU detects lossy sparse loading and skipping instructions in the program code and retrieves different encoded fields of the instructions (box 705). In one implementation, the encoded fields include address, N (the total number of input vector operands), r_offset1, r_offset2, and r_base, which are used to compute redundant indices for a given zero value. In other implementations, lossy sparse loading and skipping instructions for vectors include other numbers and / or types of encoded fields.

[0040] Next, LSLU sets the variable "i" to zero (box 710). Then, LSLU determines if the variable "i" is less than the value of "N" (condition box 715). If the variable "i" is less than the value of "N" (condition box 715, "Yes" branch), LSLU loads the next set of values ​​from the dataset from memory into the vector register file, and then generates a count of the number of non-zero values ​​in the loaded set of values ​​(box 720). This count of non-zero values ​​is... Figure 7 The value is represented by "NZCount[i]".

[0041] If the number of non-zero values ​​is less than the threshold (i.e., NZThres) (condition box 725, "Yes" branch), the LSLU generates a redundant multiply-accumulate (MAC) index (i.e., ridx) and writes the redundant MAC index to the skip queue (box 730). After box 730, the LSLU increments the variable "i" (box 735), and then method 700 returns to condition box 715. If the number of non-zero values ​​is greater than or equal to the threshold (condition box 725, "No" branch), the LSLU increments the variable "i" (box 735), and then method 700 returns to condition box 715. If the variable "i" equals the value of "N" (condition box 715, "No" branch), the LSLU returns a set of values ​​"v" (box 740). After box 740, method 700 ends. It should be noted that the scheduler queries the skip queue before issuing an instruction to be executed to the SIMD unit, and if the index of the instruction is stored in the skip queue, the scheduler proceeds to the next instruction.

[0042] Turn now Figure 8 This illustrates one implementation of method 800 for executing lossy sparse loading instructions. A lossy sparse loading unit receives lossy sparse loading instructions for execution (box 805). The lossy sparse loading unit determines how many non-zero values ​​are included in one or more input vector operands of the received lossy sparse loading instructions (box 810). If the number of non-zero values ​​is less than a threshold (condition box 815, "Yes" branch), the lossy sparse loading unit prevents the issuance of instructions to execute for one or more input vector operands (box 820). In other words, in box 820, the lossy sparse loading unit causes processing of one or more input vector operands to be skipped. In one implementation, the threshold is user-configurable. In another implementation, the user sets the threshold based on error tolerance in the underlying neural network. If the number of non-zero values ​​is greater than or equal to the threshold (condition box 815, "No" branch), the lossy sparse loading unit causes instructions to be issued to execute for one or more input vector operands (box 825). After boxes 820 and 825, method 800 ends. It should be noted that method 800 is repeated for each lossy sparse loading instruction received by the lossy sparse loading unit.

[0043] Now for reference Figure 9 The diagram illustrates one implementation of method 900 for implementing a neural network. A computing system receives program code to implement the neural network (box 905). The program code includes multiple instructions for implementing the neural network. In one implementation, the computing system includes multiple computing units, where each computing unit includes a lossy sparse loading unit.

[0044] The system executes one or more first instructions from the program code on multiple computing units (box 910). Additionally, the system skips the execution of one or more second instructions from the program code on multiple computing units (box 915). The system implements the neural network by executing one or more first instructions and skipping the execution of one or more second instructions (box 920).

[0045] The system then uses a neural network to generate classifications for the first dataset (box 925). Method 900 ends after box 925. In one implementation, the first dataset is images, and the classification identifies the given category to which the images belong. In another implementation, the first dataset is videos, and the classification assigns the videos to a given category. In other implementations, the first dataset includes other types of data. It should be noted that method 900 can be implemented multiple times to generate classifications for any number of datasets.

[0046] Turn now Figure 10 The image shows examples of pseudocode for implementing inner and outer product matrix multiplication operations. A wide variety of Generalized Matrix Multiplication (GEMM) routines exist for implementing dense matrix multiplication on GPUs and other types of processing units. The optimal routine for a given scheme is determined by the size of the operand matrix, the size of local and global memory, and the available accelerator features for computation and compression. GEMM algorithms can be broadly distinguished based on whether they utilize inner or outer products. Figure 10 The top of the code displays pseudocode 1005 as an example of implementing inner product matrix multiplication. The `matrixMul` function performs dense matrix multiplication between an M x K matrix A and a K x P matrix B to produce an M x P matrix C. For the purposes of pseudocode 1005, it is assumed that matrix A is stored in row-major format and matrix B is stored in column-major format. The `matrixMul` function repeatedly calls the `innerProd` function to compute each element C[i][j] by performing an inner product between the i-th row of A and the j-th column of B. In a DNN implementation, matrix A corresponds to the output activation of the previous layer, matrix B corresponds to the weights of the current layer, and matrix C corresponds to the output activation of the current layer.

[0047] exist Figure 10The bottom shows pseudocode 1010 as an example of implementing outer product matrix multiplication. The `matrixMul` function repeatedly calls the `outerProd` function to compute a block of N x N values ​​in matrix C. The name of the `outerProd` function comes from the fact that it sums the outer product between the N columns of A[i:i+N][:] and the N rows of B[:][j:j+N]. The sparsity of matrices A and B makes the multiply-accumulate (MAC) operation performed in the `innerProd` and `outerProd` functions redundant. These two functions are typically parallelized in a GPU by SIMD units. Simply masking a single thread using redundant MAC in a SIMD unit would save power and energy, but saving execution time would require skipping the entire redundant wavefront. Therefore, in one implementation, the entire wavefront is eliminated if all threads of the wavefront load zero operand values ​​from A or B. In another implementation, the resilience of the DNN application is leveraged to increase the chance of encountering redundant wavefronts by recognizing them as redundant when most (but not all) of the values ​​loaded by the threads are zero. In one implementation, the number of values ​​that need to be zero for the wavefront to be characterized as redundant is user-configurable. In this implementation, the final application specifies the number of non-zero values ​​(i.e., the amount of attenuation) that can be maintained under acceptable degradation of the result.

[0048] Now for reference Figure 11 An example of pseudocode 1105 for implementing a vector lossy combined sparse load instruction is shown. Pseudocode 1105 is shown as an example for implementing inner product matrix multiplication using vector lossy combined sparse load (or V_LCSLD) instructions. In other implementations, pseudocode 1105 may include instructions of other types and / or arrangements. In one implementation, the primary goal of the V_LCSLD instruction is a simple inner and outer product implementation with a block size of 1 that iteratively loads one operand for each of matrices A and B before performing a multiplication MAC operation on the operands. The V_LCSLD instruction reads the two operands from memory in a combined form and returns the values ​​to the vector register file (VGPR) only if the number of non-zero values ​​in each operand exceeds a provided lossy threshold (thres). If the current load returns zero values ​​in a majority of threads, the V_LCSLD instruction automatically advances to the operand load for the next MAC.

[0049] In one implementation, before performing the MAC operation on the values ​​vA and vB, the innerProd function repeatedly calls the V_LCSLD function to load the values ​​vA and vB. The V_LCSLD function loads the operands A and B from all threads in the wavefront into vectors vA and vB, respectively. Next, the V_LCSLD function counts the number of non-zero values ​​in vA and vB. If the loop termination condition is not met (i.e., the number of non-zero values ​​is within NZThres, and the current loop index is less than maxIdx), the V_LCSLD function loads the next operand and increments the index (i.e., idx). Upon termination, the current values ​​of vA and vB, along with the current idx value, are returned to ensure the final innerProd function executes correctly.

[0050] Turn now Figure 12 A block diagram of one implementation of logic 1200 for implementing vector lossy combinational sparse load instructions. In one implementation, logic 1200 includes at least a cache 1205, zero-parity logic 1210, a next address generator 1215, and non-zero value counting logics 1220 and 1225 for input vector operands A and B, respectively. The number of non-zero values ​​in vector operands A and B is counted by logics 1220 and 1225, respectively. The number of non-zero values ​​is compared with thresholds for operands A and B, and the output of the comparison is fed to an OR gate. The output of the OR gate is an "IsRedundant" signal, which indicates whether the input vector operands are redundant. In other implementations, logic 1200 may include other components and / or be organized in other suitable ways.

[0051] Now for reference Figure 13 Example of pseudocode 1305 for implementing a vector-lossy single sparse load instruction is shown. Pseudocode 1305 is shown as an example for implementing outer product matrix multiplication using a vector-lossy single sparse load (or V_LSSLD) instruction. In other implementations, pseudocode 1305 may include instructions of other types and / or arrangements. The V_LSSLD instruction aims for an outer product implementation with block size N > 1. The V_LSSLD instruction successfully skips redundant MAC blocks after multiple operand load blocks. The V_LSSLD instruction counts the number of zeros among the values ​​loaded into VGPR and accumulates the count across the entire load block for operands. The S_Z_CHECK_BRANCH instruction at the end of the load compares the accumulated value with a provided lossy threshold. The comparison determines whether execution flows into the MAC or continues to the next load block if the current MAC block is redundant. The array accNZCount accumulates the number of non-zero values ​​encountered by each operand loaded by different threads in the load block.

[0052] Turn now Figure 14 Example of pseudocode 1405 for implementing vector lossy sparse loading and skip instructions is shown. Pseudocode 1405 is shown as an example for implementing outer product matrix multiplication using vector lossy sparse loading and skip (or V_LSLS) instructions. In other implementations, pseudocode 1405 may include instructions of other types and / or arrangements. The goal of the V_LSLS instructions is a block-based implementation of outer products that operate on matrices with sparse distribution. The V_LSLS instructions allow the GPU to skip a single redundant MAC (at the wavefront stage) in a MAC block following the operand loading block. The V_LSLS instructions are written to a skipQueue with a list of redundant MAC indices. The scheduler reads the head of the skipQueue before fetching or dispatching instructions to check if the current instruction is redundant. If the current instruction is redundant, the scheduler proceeds to fetch and dispatch the next instruction after removing the current instruction from the queue. Otherwise, the current instruction is executed normally.

[0053] In one implementation, the V_LSLS instruction counts the number of non-zero values ​​loaded into the VGPR. Whenever the number of non-zero values ​​in the loaded vector falls below a threshold, the V_LSLS instruction determines which instructions become redundant. This determination is passed to the skipQueue. The parameters r_size, r_offset1, r_offset2, and r_base of the v_LSLS instruction allow the V_LSLS instruction to automatically generate program counter offsets for redundant MACs to be stored in the skipQueue. In one implementation, the parameters r_size, r_offset1, r_offset2, and r_base are programmed by the compiler based on the structure of the GEMM code. In one implementation, the skipQueue is implemented in a buffer within the LSLU. The size of the skipQueue determines the maximum number of MACs that can be skipped. Therefore, in one implementation, the size of the skipQueue is set to accommodate the maximum block size expected in the outer product implementation.

[0054] In one implementation, the DNN utilizes low-precision operands. Utilizing low-precision operands allows threads in the wavefront to load more operand values ​​and perform more MACs in a single instruction. For example, reducing the precision from 32 bits to 8 bits allows threads to load more than four times the matrix values ​​according to the load instruction and perform operations on four of those values ​​instead of a single value according to the MAC instruction. A MAC becomes redundant if and only if all low-precision values ​​in the MAC operation are zero, but the probability of encountering such a case can be very low. Therefore, in one implementation, a new threshold, "intraMAC," is introduced to determine whether the values ​​in the MAC operation are characterized as redundant. The new threshold "intraMAC" allows for some overhead within a single MAC instruction in each thread, and MAC instructions can be skipped if the number of low-precision non-zero values ​​is less than intraMAC. The sparse load instruction family described in this paper can be easily extended to implements with reduced precision.

[0055] In various implementations, program instructions from a software application are used to implement the methods and / or mechanisms described herein. For example, program instructions intended to be executable by a general-purpose or special-purpose processor. In various implementations, such program instructions are represented by a high-level programming language. In other implementations, the program instructions are compiled from a high-level programming language into binary, intermediate, or other forms. Alternatively, program instructions describing the behavior or design of the hardware are written. Such program instructions are represented by a high-level programming language (such as C). Alternatively, a hardware design language (HDL), such as Verilog, is used. In various implementations, the program instructions are stored on any of a variety of non-transitory computer-readable storage media. The storage media is accessible by the computing system during use to provide the computing system with the program instructions for program execution. Generally, such a computing system includes at least one or more memories and one or more processors configured to execute the program instructions.

[0056] It should be emphasized that the implementation described above is merely a non-limiting example. Many variations and modifications will become apparent to those skilled in the art upon a full understanding of the above disclosure. The following claims are intended to cover all such variations and modifications.

Claims

1. A system comprising: Multiple processing elements; as well as Control logic; The control logic is configured as follows: Load multiple input vector operands from memory; Before loading into the vector register file, determine whether the number of non-zero values ​​among the plurality of input vector operands is greater than or equal to a first threshold; as well as In response to the determination that the number of non-zero values ​​among the plurality of input vector operands is greater than or equal to the first threshold: Provide the plurality of input vector operands to the vector register file; as well as This causes the plurality of processing elements to process the plurality of input vector operands.

2. The system of claim 1, wherein the system is configured to reduce at least one of power consumption and execution time when implementing the neural network by skipping processing associated with the plurality of input vector operands of one or more layers of the neural network.

3. The system of claim 2, wherein the non-zero value is an absolute value greater than the second threshold.

4. The system of claim 3, wherein the first threshold and the second threshold are programmable based on the error tolerance of the neural network.

5. The system of claim 3, wherein the system is configured to set the first threshold and the second threshold to values ​​based on the error tolerance of the neural network.

6. The system of claim 1, wherein in response to the number of non-zero values ​​being less than the first threshold, the control logic is configured to cause the processing of the plurality of input vector operands to be skipped.

7. The system of claim 2, wherein the control logic is further configured to: in response to determining that the number of non-zero values ​​in the given input vector operands is less than the first threshold, store the program counter offset of the instruction processing the given input vector operands in a skip queue.

8. A method comprising: Load multiple input vector operands from memory; Before being loaded into the vector register file, control logic connected to multiple processing elements determines whether the number of non-zero values ​​in the multiple input vector operands is greater than or equal to a first threshold. as well as In response to the determination that the number of non-zero values ​​among the plurality of input vector operands is greater than or equal to the first threshold: Provide the plurality of input vector operands to the vector register file; as well as This causes the plurality of processing elements to process the plurality of input vector operands.

9. The method of claim 8, further comprising reducing at least one of power consumption and execution time when implementing the neural network by skipping processing associated with the plurality of input vector operands of one or more layers of the neural network.

10. The method of claim 9, wherein the non-zero value is an absolute value greater than the second threshold.

11. The method of claim 10, wherein the first threshold and the second threshold are programmable based on the error tolerance of the neural network.

12. The method of claim 10, wherein in response to determining that the number of non-zero values ​​is less than the first threshold, the method includes causing the plurality of processing elements to skip processing of the plurality of input vector operands.

13. The method of claim 8, further comprising performing matrix multiplication by a plurality of processing elements to multiply a first set of input vector operands by a second set of input vector operands.

14. The method of claim 9, further comprising: In response to determining that the number of non-zero values ​​in a given input vector operand is less than the first threshold, the program counter offset of the instruction that processes the given input vector operand is stored in a skip queue.

15. An apparatus comprising: Multiple processing elements; Scheduler; as well as Lossy sparse loading units; The lossy sparse loading unit is configured as follows: Load multiple input vector operands from memory; Determine how many non-zero values ​​are included in the plurality of input vector operands; as well as In response to the determination that the number of non-zero values ​​among the plurality of input vector operands is greater than or equal to a first threshold: Provide the plurality of input vector operands to the vector register file; as well as This causes the plurality of processing elements to process the plurality of input vector operands.

16. The device of claim 15, wherein the device is configured to reduce at least one of power consumption and execution time when implementing the neural network by skipping processing associated with the plurality of input vector operands of one or more layers of the neural network.

17. The device of claim 16, wherein the non-zero value is an absolute value greater than the second threshold.

18. The device of claim 17, wherein the first threshold and the second threshold are programmable based on the error tolerance of the neural network.

19. The device of claim 17, wherein the device is configured to set the first threshold and the second threshold to values ​​based on the error tolerance of the neural network.

20. The apparatus of claim 15, wherein the processing performed by the plurality of processing elements includes matrix multiplication operations for multiplying a first set of input vector operands with a second set of input vector operands.

Images