Method and system for providing vector sparsification in neural networks

Abstract

The present disclosure relates to a method and system for providing vector sparsification in a neural network. In some embodiments, an exemplary method for providing vector sparsification in a neural network includes partitioning a matrix associated with the neural network into a plurality of vectors, selecting a first subset of non-zero elements from the plurality of vectors to form a pruned matrix, and outputting the pruned matrix and performing the neural network using the pruned matrix. This method can address the irregular workload as a bottleneck for performing most of the existing neural networks.

Description

[0001] This disclosure claims priority to U.S. Provisional Application No. 62 / 893,768, filed on August 29, 2019, which is incorporated herein by reference in its entirety. Background Art

[0002] Today's neural networks often contain multiple nodes and layers, which reduces efficiency and increases latency. Therefore, input sparsity, output sparsity, weight sparsity, or a combination of these has been proposed for neural networks to improve efficiency and reduce latency. Indeed, the sparsity properties of artificial neural networks more accurately reflect how neurons in the human brain process information.

[0003] However, in the prior art, the sparsity of artificial neural networks often provides overall poor speedup (e.g., the irregular workload of sparse neural networks is a bottleneck for executing most existing neural networks) or lacks accuracy, so that hardware such as central processing units (CPUs) or graphics processing units (GPUs) still remains inefficient. Summary of the Invention

[0004] In some embodiments, an exemplary method for providing vector sparsification in a neural network includes: partitioning a matrix associated with the neural network into a plurality of vectors; selecting a first subset of non-zero elements from the plurality of vectors to form a pruned matrix; and outputting the pruned matrix, and executing the neural network using the pruned matrix.

[0005] In some embodiments, an exemplary system for providing vector sparsification in a neural network includes: at least one memory for storing instructions; and at least one processor for executing the instructions to cause the system to: partition a matrix associated with the neural network into a plurality of vectors; select a first subset of non-zero elements from the plurality of vectors to form a pruned matrix; and output the pruned matrix, and execute the neural network using the pruned matrix.

[0006] In some embodiments, a non-transitory computer-readable storage medium stores a set of instructions executable by at least one processor to cause a computer to perform a method for providing vector sparsification in a neural network, the method comprising: partitioning a matrix associated with the neural network into a plurality of vectors; selecting a first subset of non-zero elements from the plurality of vectors to form a pruned matrix; and outputting the pruned matrix, and executing the neural network using the pruned matrix.

[0007] In some embodiments, an exemplary processor for executing a neural network includes: a memory for storing offsets of respective elements in a compressed vector relative to the original vector; and at least one core for decoding the offsets of respective elements in a first plurality of compressed vectors and performing matrix multiplication of a first operand and a second operand based on the decoded offsets, wherein the first operand includes the first plurality of compressed vectors.

[0008] The additional features and advantages of the present disclosure will be explained in part in the following detailed description, and in part they will be apparent from the description or learned through practice of the present disclosure. These features and advantages of the present disclosure may be realized and obtained by the elements and combinations particularly pointed out in the appended claims.

[0009] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings, which constitute a part of the specification, illustrate several embodiments together with the description and, together with the description, explain the principles and features of the embodiments of the present disclosure. In the drawings:

[0011] Figure 1 is a schematic diagram of a neural network according to an embodiment of the present disclosure;

[0012] Figure 2A is a schematic diagram of an exemplary neural network accelerator architecture according to an embodiment of the present disclosure;

[0013] Figure 2B is a schematic diagram of an exemplary cloud system including a neural network accelerator according to an embodiment of the present disclosure;

[0014] Figure 3A This is a diagram of generic sparsifying a matrix.

[0015] Figure 3B This is a schematic diagram of unified sparsifying a matrix;

[0016] Figure 4 is a schematic diagram of vector division of a matrix according to an embodiment of the present disclosure;

[0017] Figure 5 is a schematic diagram of an exemplary matrix coding according to an embodiment of the present disclosure;

[0018] Figure 6Ais a flowchart of an exemplary method for providing vector-wise sparsity in a neural network according to an embodiment of the present disclosure;

[0019] Figure 6B is a flowchart of another exemplary method for providing vector-wise sparsity in a neural network according to an embodiment of the present disclosure;

[0020] Figure 7 is a schematic diagram of an exemplary vector-matrix multiplication according to an embodiment of the present disclosure;

[0021] Figure 8 is a schematic diagram of an exemplary processor according to an embodiment of the present disclosure;

[0022] Figure 9 is a schematic diagram of mapping matrix multiplication to worktuples in a warp according to an embodiment of the present disclosure;

[0023] Figure 10 is a schematic diagram of processing elements by a thread group in an exemplary existing dense mode and a vector sparse mode according to an embodiment of the present disclosure;

[0024] Figure 11 is a schematic diagram of an exemplary processor architecture according to an embodiment of the present disclosure;

[0025] Figure 12 is a schematic diagram of an exemplary execution flow of warp-based sparse matrix multiplication (SWMM) and warp-based sparse matrix multiply accumulation (SWMMA) according to an embodiment of the present disclosure;

[0026] Figure 13 is a schematic diagram of accuracy and sparsity in a weight matrix in a convolutional neural network (CNN) workload according to an embodiment of the present disclosure;

[0027] Figure 14 is a schematic diagram of Bilingual Evaluation Undergraduate (BLEU) scores and sparsity in a weight matrix of a recurrent neural network (RNN) load according to an embodiment of the present disclosure;

[0028] Figure 15 1 is a schematic diagram of normalized acceleration on a dense neural network-based unified computing device architecture (CUDA) core according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] Reference will now be made to exemplary embodiments in detail, some examples of which are illustrated in the accompanying drawings. In the following description, like numerals in the various figures represent the same or similar elements unless otherwise indicated. The exemplary embodiments listed below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely exemplary of apparatuses, systems, and methods consistent with aspects of the present disclosure as described in the appended claims.

[0030] Existing weight sparsification techniques include universal sparsification and unified sparsification. Universal sparsification reduces all elements in the weight matrix except the four elements (or any other predetermined number of elements) with the largest absolute value to zero. However, in a neural network after universal sparsification, the workload may be irregular because the positions of non-zero elements can be selected from anywhere within the weight matrix. Uniform sparsification reduces all elements in the weight matrix that are not in one or more selected spaces to zero based on the L1 (level 1) or L2 (level 2) norm of the selected space. Different unified sparsification techniques have different spatial constraints, for example, column-level, row-level, block-level, filter-level, channel-level, or similar spatial constraints. However, the output accuracy of unified sparsification may be significantly reduced.

[0031] Furthermore, generalized matrix multiplication (GEMM) continues to advance, as it is a fundamental primitive in many neural networks. Tensor Cores have been developed to serve as hardware accelerators for 16×16 matrix multiplication operations. Unfortunately, since Tensor Cores only focus on accelerating dense matrix multiplication, sparse GEMM cannot take advantage of this hardware accelerator.

[0032] Some embodiments of the present disclosure relate to a system and method for providing vector sparsification in a neural network. In addition, some embodiments of the present disclosure relate to a processor for executing a neural network. Advantageously, exemplary embodiments can improve speed and power efficiency by reducing the mathematical operations and memory transfers required to execute the neural network. In addition, embodiments of the present disclosure can be used on any processor, such as a central processing unit (CPU) or any other general or dedicated processing circuit. In some embodiments, speed can be further improved by implementing the embodiments using a graphics processing unit (GPU).

[0033] The devices, systems, and methods disclosed herein can be used for various neural network architectures, such as deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), etc., and can be various configurations, such as CPUs, GPUs, neural network processing units (NPUs), field programmable gate arrays (FPGAs), tensor processing units (TPUs), application-specific integrated circuits (ASICs), etc.

[0034] Figure 1 is a schematic diagram of a neural network (NN) 100. Figure 1 As shown, the neural network 100 may include an input layer 120 for receiving inputs such as inputs 110-1 to 110-m. The inputs may include images, text, or any other structured or unstructured data to be processed by the neural network 100. In some embodiments, the neural network 100 may accept multiple inputs simultaneously. For example, in Figure 1 In the example of FIG1 , neural network 100 may receive up to m inputs simultaneously. Additionally or alternatively, input layer 120 may receive up to m inputs in rapid succession, e.g., input 110 - 1 is an input received by input layer 120 in one cycle, a second input is an input received by input layer 120 in a second cycle, and in the second cycle, input layer 120 pushes data from input 110 - 1 to the first hidden layer, and so on. Any number of inputs may be input simultaneously, in rapid succession, or the like.

[0035] The input layer 120 may include one or more nodes, e.g., node 120-1, node 120-2, ..., node 120-a. Each node may apply an activation function to a corresponding input (e.g., one or more of inputs 110-1 to 110-m) and weight the output of the activation function by a specific weight associated with the node. The activation function may include a Heaviside step function, a Gaussian function, a multiquadratic function, an inverse multiquadratic function, a sigmoidal function, or the like. The weights may include positive values ​​between 0.0 and 1.0, or any other value that allows the outputs of some nodes in a layer to be larger or smaller than the outputs of other nodes in the layer.

[0036] like Figure 1 As further described, the neural network 100 may include one or more hidden layers, such as hidden layers 130-1 to 130-n. Each hidden layer may include one or more nodes. For example, Figure 1 , hidden layer 130-1 includes nodes 130-1-1, 130-1-2, 130-1-3, ..., 130-1-b, and hidden layer 130-n includes nodes 130-n-1, 130-n-2, 130-n-3, ..., 130-nc. Similar to the nodes of input layer 120, the nodes of the hidden layer can apply an activation function to the outputs of the connected nodes of the previous layer and weight the outputs of the activation function by specific weights associated with the nodes.

[0037] like Figure 1As further described in

[15] , the neural network 100 may include an output layer 140 for completing outputs (e.g., output 150-1, output 150-2, ..., output 150-d). The output layer 140 may include one or more nodes, for example, node 140-1, node 140-2, ..., node 140-d. Similar to the nodes of the input layer 120 and the hidden layer, the nodes of the output layer 140 may apply an activation function to the outputs of the connected nodes of the previous layer and weight the output of the activation function by a specific weight associated with the node.

[0038] Despite Figure 1 Although the full connection is described in the figure, the various layers 100 of the neural network can use any connection scheme. For example, one or more layers (e.g., input layer 120, hidden layer 130-1, ..., hidden layer 130-n, output layer 140, etc.) can be connected using a convolution scheme, a sparse connection scheme, or a similar connection scheme. Such an embodiment uses a ratio between the current layer and the previous layer. Figure 1 Fewer connections shown.

[0039] In addition, despite the Figure 1 Although a feedforward network is described in detail, neural network 100 may additionally or alternatively use backpropagation (e.g., by using long-short-term memory nodes or the like). Thus, although neural network 100 is depicted as being similar to a convolutional neural network (CNN), neural network 100 may include a recurrent neural network (RNN) or any other neural network.

[0040] Figure 2A 2 is a schematic diagram of an exemplary neural network accelerator architecture according to an embodiment of the present disclosure. In the context of the present disclosure, a neural network accelerator may also be referred to as a machine learning accelerator or a deep learning accelerator. In some embodiments, the accelerator architecture 200 may be referred to as an NPU architecture 200. Figure 2A As shown, the accelerator architecture 200 includes multiple cores 202, a command processor 204, a direct memory access (DMA) unit 208, a joint test action group (JTAG) / test access port (TAP) controller 210, a peripheral device interface 212, a bus 214, etc.

[0041] It will be appreciated that the core 202 may perform algorithmic operations based on the communication data. The core 202 may include one or more processing elements, which may include a single instruction multiple data (SIMD) architecture contained in one or more processing units, which are configured to perform one or more operations (e.g., multiplication, addition, multiplication-accumulation (MAC), etc.) based on commands received from the command processor 204. In order to perform operations on communication data packets, the core 202 may include one or more processing elements for processing information in the data packets. Each processing element may include any number of processing units. According to an embodiment of the present disclosure, the accelerator architecture 200 may include multiple cores 202, for example, four cores. In some embodiments, the multiple cores 202 may be communicatively coupled to each other. For example, the multiple cores 202 may be coupled to a unidirectional ring bus and may support efficient pipelines for large neural network models.

[0042] The command processor 204 can interact with the host unit 220 and transmit relevant commands and data to the corresponding core 202. In some embodiments, the command processor 204 can interact with the host unit under the supervision of a kernel mode driver (KMD). In some embodiments, the command processor 204 can modify the relevant commands given to each core 202 so that each core 202 can work in parallel as much as possible. The modified commands can be stored in an instruction buffer. In some embodiments, the command processor 204 can be configured to coordinate one or more cores 202 to execute in parallel.

[0043] The DMA unit 208 can facilitate data transfer between the host memory 221 and the accelerator architecture 200. For example, the DMA unit 208 can facilitate loading data or instructions from the host memory 221 into local memory within a core 202. The DMA unit 208 can also facilitate data transfer between multiple accelerators. The DMA unit 208 can allow off-chip devices to access on-chip and off-chip memory without interrupting the host CPU. Furthermore, the DMA unit 208 can facilitate data transfer between components of the accelerator architecture 200. For example, the DMA unit 208 can facilitate data transfer between multiple cores 202 or within each core. The DMA unit 208 can also generate memory addresses and initiate memory read and write cycles. The DMA unit 208 can also contain multiple hardware registers that can be written and read by one or more processors, including a memory address register, a byte count register, one or more control registers, and other types of registers. These registers can specify some combination of the source, destination, transfer direction (reading from or writing to an I / O device), the transfer unit size, and the number of bytes transferred in a burst. It is understood that the accelerator architecture 200 may include a second DMA unit for transferring data between other accelerator architectures to allow direct communication between multiple accelerator architectures without involving the host CPU.

[0044] The JTAG / TAP controller 210 may designate a dedicated debug port to implement a serial communication interface (e.g., a JTAG interface) for low-overhead access to the accelerator without requiring external direct access to the system address and data buses. The JTAG / TAP controller 210 may also include an on-chip test access interface (e.g., a TAP interface) that implements a protocol for accessing a set of test registers that represent various portions of the chip's logic level and device capabilities.

[0045] If present, the peripheral interface 212 (eg, a PCIe interface) typically serves as an inter-chip bus to provide communication between the accelerator and other devices.

[0046] The bus 214 (e.g., an I2C bus) includes an intra-chip bus and an inter-chip bus. The intra-chip bus connects all internal components to each other as required by the system architecture, but not all components are connected to any other component, and each component is connected to some other components with which it needs to communicate. The inter-chip bus connects the accelerator to other devices, such as off-chip memory or peripherals. For example, the bus 214 can provide high-speed communication across cores, and can also connect the core 202 to other units, such as off-chip memory or peripherals. Typically, if a peripheral interface 212 (e.g., an inter-chip bus) is present, the bus 214 is only related to the intra-chip bus, although in some implementations, it may still involve some dedicated inter-bus communication.

[0047] The accelerator architecture 200 can also communicate with a host unit 220. The host unit 220 can be one or more processing units (e.g., an X86 central processing unit). Figure 2A As shown, the host unit 220 can be associated with a host memory 221. In some embodiments, the host memory 221 can be an integrated memory associated with the host unit 220 or an external memory. In some embodiments, the host memory 221 can include a host disk, which is an external memory used to provide additional memory for the host unit 220. The host memory 221 can be a double data rate synchronous dynamic random access memory (e.g., DDRSDRAM) or similar memory. The host memory 221 can be used to store large amounts of data, and its access speed is slower than the on-chip memory integrated into the accelerator chip, thereby acting as a higher level cache. The data stored in the host memory 221 can be transmitted to the accelerator architecture 200 for executing the neural network model.

[0048] In some embodiments, the host system having the host unit 220 and the host memory 221 may include a compiler (not shown). A compiler is a program or computer software that converts computer code written in a programming language into instructions for the accelerator architecture 200 to create an executable program. In machine learning applications, the compiler can perform various operations, such as preprocessing, lexical analysis, parsing, semantic analysis, conversion of input programs to intermediate representations, initialization of neural networks, code optimization, and code generation, or a combination thereof. For example, a compiler can compile a neural network to generate static parameters, such as the connections between neurons and the weights of neurons.

[0049] In some embodiments, a host system including a compiler may push one or more commands to the accelerator architecture 200. As described above, these commands may be further processed by the command processor 204 of the accelerator architecture 200, temporarily stored in an instruction buffer (not shown) of the accelerator architecture 200, and assigned to corresponding one or more cores (e.g., Figure 2A Some of these commands may instruct a DMA unit (e.g., Figure 2A DMA unit 208) from host memory (e.g., Figure 2A The host memory 221) loads instructions and data into the accelerator architecture 200. The loaded instructions are then distributed to each core (e.g., Figure 2A 's core 202), which processes the instructions.

[0050] It will be appreciated that the first few instructions received by the core 202 may instruct the core 202 to load / store data from the host memory 221 into one or more local memories within the core (e.g., a local memory of the core). Each core 202 may then initiate an instruction pipeline, which includes fetching instructions from an instruction buffer (e.g., via a sequencer), decoding instructions (e.g., via a processor), and executing the instructions in the instruction pipeline. Figure 2A DMA unit 208), generates a local memory address (e.g., corresponding to an operand), reads the source data, performs or load / store operation, and then writes back the result.

[0051] According to some embodiments, the accelerator architecture 200 may further include a global memory (not shown) having memory blocks (e.g., four 8GB second-generation high-bandwidth memory (HBM2) blocks) as main memory. In some embodiments, the global memory may store instructions and data obtained from the host memory 221 via the DMA unit 208. The instructions may then be distributed to the instruction buffer of each core assigned a corresponding task, and each core may process the instructions accordingly.

[0052] In some embodiments, the accelerator architecture 200 may further include a memory controller (not shown) for managing the reading or writing of data from a specific memory block (e.g., HBM2) of the global memory. For example, the memory controller may manage reading and writing data from another accelerator core (e.g., from DMA unit 208 or a DMA unit corresponding to another accelerator) or from core 202 (e.g., from the local memory of core 202). It should be understood that multiple memory controllers may be provided in the accelerator architecture 200. For example, in the global memory, each memory block (e.g., HBM2) has a memory controller.

[0053] A memory controller can generate memory addresses and initiate memory read or write cycles. A memory controller can contain several hardware registers that can be written to and read from by one or more processors. These registers include a memory address register, a byte count register, one or more control registers, and other types of registers. These registers can specify the source, destination, direction of the transfer (reading from or writing to an I / O device), the size of the transfer unit, the number of bytes to be transferred in one burst, or other typical characteristics of a memory controller.

[0054] I understand. Figure 2AThe accelerator architecture 200 can be used for various neural networks, such as DNNs (e.g., CNNs, RNNs, etc.). In addition, in some embodiments, it can be configured for various processing architectures, such as NPUs, GPUs, FPGAs, TPUs, ASICs, and any other type of heterogeneous accelerator processing units (HAPUs), etc.

[0055] Figure 2B FIG is a schematic diagram of an exemplary cloud system including a neural network accelerator 200 according to an embodiment of the present disclosure. Figure 2B As shown, the cloud system 230 can provide cloud services with artificial intelligence (AI) capabilities and can include multiple computing servers (e.g., 232 and 234). In some embodiments, the computing server 232 can include, for example, Figure 2A The neural network architecture 200 is shown. For simplicity and clarity, Figure 2B The neural network accelerator architecture 200 is shown in a simplified manner.

[0056] With the neural network accelerator architecture 200, the cloud system 230 can provide multiple extended AI capabilities, such as image recognition, facial recognition, translation, and 3D modeling. It is understood that the neural network accelerator architecture 200 can be deployed on computing devices in other forms. For example, the neural network accelerator architecture 200 can also be integrated into computing devices such as smartphones, tablets, and wearable devices.

[0057] Furthermore, although neural network accelerator architectures such as Figures 2A-2B It is not shown, but it will be understood that any accelerator that provides parallel computing capabilities may be used.

[0058] Figure 3A General sparsification 300 is used to represent a matrix (e.g., a weight matrix) 301 associated with a neural network. For example, general sparsification 300 can reduce matrix 301 to a sparse matrix 305 to reduce the amount of computation required to execute the neural network. Although matrix 301 is depicted as a 4×4 matrix, matrix 301 can be of any size.

[0059] Therefore, if Figure 3A As shown, general sparsification 300 may include selecting one or more elements from matrix 301, such as elements 303a, 303b, 303c, and 303d. Although described as selecting four elements, general sparsification 300 may use any predetermined number of elements. Considering that elements 303a, 303b, 303c, and 303d have the four largest absolute values, elements 303a, 303b, 303c, and 303d are selected. As shown in sparse matrix 305, general sparsification 300 may also include zeroing out non-selected elements. Thus, as shown in Figure 3AAs shown, general sparsification 300 performs 75% sparsification on matrix 301. Furthermore, the sparsity level may depend on the predetermined number of elements and the size of matrix 301.

[0060] Figure 3B A uniform sparsification 350 is used to represent a matrix (e.g., a weight matrix) 351 associated with a neural network. For example, uniform sparsification 350 can reduce matrix 351 to a sparse matrix 355 to reduce the amount of computation required to execute the neural network. Although matrix 351 is depicted as a 4×4 matrix, matrix 351 can be of any size.

[0061] Therefore, if Figure 3B As shown, uniform sparsification 350 may include selecting one or more elements from matrix 351, such as elements 353a, 353b, 353c, and 353d. Although depicted as selecting four elements, uniform sparsification 350 may use any predetermined number of elements. Elements 353a, 353b, 353c, and 353d may be selected because they are within a selected column. Although depicted as selecting a column, uniform sparsification 350 may select any predetermined number of spaces, such as columns, rows, blocks, filters, channels, and the like. As shown in sparse matrix 355, uniform sparsification 350 may also include zeroing out non-selected elements. Thus, as shown in Figure 3B As shown, uniform sparsification 350 applies 75% weight sparsity to matrix 351. Furthermore, the sparsity level may depend on a predetermined number of columns and the size of matrix 351.

[0062] However, general sparsification 300 may not provide spatial predictability for the selection of elements that do not return to zero, because the elements with the largest absolute value may be distributed anywhere in the matrix 301. Therefore, the software / hardware used for sparse matrix multiplication must deal with the randomness of the positions of the selected elements in the general sparsification matrix, which can lead to a huge performance overhead. Therefore, for large matrices 301, tracking the multiplication of the simplified matrix 301 with the corresponding sparse input requires a lot of memory. However, unified sparsification 350 may not provide an acceptable level of accuracy in a neural network. Some embodiments of the present disclosure achieve similar speedups as unified sparsification 350 without similar loss of accuracy. For example, some embodiments of the present disclosure may partition the matrix into vectors (such as Figure 4 ), and enforce sparsity on each vector (as Figure 5 shown).

[0063] Figure 44 is a schematic diagram of vector partitioning 400 of a matrix 401 according to an embodiment of the present disclosure. Matrix 401 is related to a neural network. For example, matrix 401 can be a weight matrix, an input matrix, an output matrix, or the like. For example, vector partitioning 400 can partition matrix 401 into vectors of size L ("L-dimensional vectors"). Matrix 401 is depicted as an M×N matrix. In some embodiments, if the M×N matrix does not partition evenly into the L-dimensional vectors, vector partitioning 400 can include padding one or more residual vectors with zeros so that each vector has the same size.

[0064] like Figure 4 As shown. By dividing the matrix 401 into vectors (e.g., 401-0, 401-1, etc.), each vector can be identified by a coordinate relative to the matrix 401 (e.g., V(0,0), V(0,1), V(1,0), etc.), and each element within a vector can be identified by an offset relative to the vector (e.g., E(0), E(1), ..., E(L-1)).

[0065] Figure 5 5 is a schematic diagram of encoding 500 a matrix 501 according to an exemplary embodiment of the present disclosure. The matrix 501 is related to a neural network. For example, the matrix 501 can be a weight matrix, an input matrix, an output matrix, etc. Figure 5 As shown, matrix 501 can be divided into four vectors (e.g., V0, V1, V2, and V3) of size 8. For each of these four vectors V0, V1, V2, and V3, at most two (or any predetermined number) non-zero elements can be selected and encoded as a compact vector in a coding matrix 502, which is associated with their offsets in the offset matrix 503. These vectors V0, V1, V2, and V3 can be encoded as vectors of the same length K (e.g., Figure 5 , where K=2). If a vector (e.g., V1) has fewer non-zero elements than the length K of the compressed vector, the corresponding compressed vector may be padded with zeros. Although depicted in the figure using a 4×8 matrix 501, the 8-dimensional vectors V0, V1, V2, and V3 (e.g., L=8) and the two-dimensional compressed vectors (e.g., K=2), the matrix 501, the vectors V0, V1, V2, and V3, and the corresponding compressed vectors may be of any size.

[0066] In some embodiments, the encoding of an M×N matrix may include dividing the M×N matrix into L-dimensional vectors. Thus, the M×N matrix may have M*[N / L] vectors. If the M×N matrix is ​​not evenly divided into L-dimensional vectors, the encoding may include filling one or more residual vectors with zeros so that each vector is the same size. The encoding may also include selecting a predetermined number K (K<=L) of non-zero elements in each vector and encoding the selected non-zero elements into corresponding K-dimensional compressed vectors, with the corresponding K-dimensional compressed vectors having the corresponding offsets in the original vector. If a vector has fewer non-zero elements than K, the empty elements may be filled with zeros in the corresponding K-dimensional compressed vector. The encoding may also include counting the number NNZ of non-zero elements in each vector and setting K to the maximum value in the NNZ of all vectors in the matrix. Thus, in such an embodiment, all non-zero elements in the matrix can be encoded into compressed vectors.

[0067] It is understood that although Figures 4 and 5 The illustrated embodiment and many other embodiments use row vectors, but the vector partitioning and encoding described herein can be performed on a row or column basis.

[0068] In theory, it may take log2 L bits to encode each offset index in an L-dimensional vector. Therefore, the overall compression ratio of the encoding is:

[0069]

[0070] Where P represents the number of bits used to store the value of the element. Figure 5 In an exemplary embodiment of encoding a 4×8 16-bit floating point (FP16) matrix 501, where L=8 and K=2, since the offset is in the range [0, 7], each element of the offset matrix 503 can be represented by 3 bits, and thus the compression ratio is 3.37. If K is set to the maximum number of non-zero elements in a vector, the encoding can achieve an enhanced compression ratio when all vectors have the same number of non-zero elements.

[0071] Figures 6A-6B 6 and 650 are flowcharts of exemplary methods 600 and 650 for providing vector sparsity in a neural network according to embodiments of the present disclosure. The exemplary methods 600 and 650 may be performed by at least one processor (e.g., Figures 2A-2B The accelerator architecture 200 or host unit 220 shown, Figure 8 The processor 800 shown, Figure 11In addition, methods 600 and 650 can also be implemented by a computer program product, which is embodied in a computer-readable medium and includes computer-executable instructions. The computer-executable instructions can be executed by, for example, Figures 2A-2B 、 Figure 8 and Figure 11 The system or architecture shown is implemented by a computer.

[0072] like Figure 6A As shown, in step 601, at least one processor divides a matrix associated with a neural network into a plurality of vectors. The matrix can be a weight matrix, an input matrix, an output matrix, etc. For example, Figure 4 As explained, at least one processor divides the M×N matrix into a plurality of L-dimensional vectors. In some embodiments, if the M×N matrix is ​​not evenly divided into the L-dimensional vectors, method 600 may include padding one or more residual vectors with zeros so that each vector is of the same size L. It will be appreciated that the division in step 601 may be by column or by row.

[0073] In step 603, at least one processor may select a subset of non-zero elements from a plurality of vectors to form a pruned matrix. For example, for each L-dimensional vector, at least one processor may select K non-zero elements. K is a predetermined number and is less than or equal to L. In some embodiments, K may be set to the maximum number of non-zero elements in all vectors in the matrix. Thus, all non-zero elements in the matrix are selected to form the pruned matrix. Alternatively, fewer non-zero elements than all non-zero elements are selected in the matrix according to a predetermined criterion. For example, K non-zero elements with the largest absolute values ​​may be selected to form the pruned matrix. In some embodiments, the selection may include zeroing the unselected non-zero elements in each of the plurality of vectors to form the pruned matrix.

[0074] Optionally, in step 605, the at least one processor may encode the selected subset of non-zero elements into a compressed vector having associated offsets of the selected subset of non-zero elements in the plurality of vectors. If a vector has fewer non-zero elements than the length of the compressed vector, the corresponding compressed vector may be padded with zeros.

[0075] Optionally, at step 605, the at least one processor encodes the selected subset of non-zero elements into a compressed vector, the compressed vector having associated offsets corresponding to the selected subset of non-zero elements in the corresponding vector. If the vector has fewer non-zero elements than the length of the compressed vector, the corresponding compressed vector may be padded with zeros.

[0076] Method 600 may further include additional steps. For example, at step 607, at least one processor may output a pruning matrix and execute a neural network using the pruning matrix. The at least one processor may execute a neural network having a compressed vector and an associated offset ( Figure 6A For example, the at least one processor may extract the relevant offsets and perform a multiplication of the compressed vector and the matrix by multiplying the elements in the compressed vector that are relevant to the extracted offsets with the elements in the matrix that correspond to the extracted offsets. Additionally or alternatively, the at least one processor may use a pruning matrix ( Figure 6A The neural network may be retrained (not shown). The retraining may include at least one of the following: modifying one or more elements of the pruning matrix; and modifying the activation function of one or more nodes of the neural network.

[0077] Figure 6B FIG. 6 is a flow chart of another exemplary method 650 for providing vector sparsification in a neural network according to an embodiment of the present disclosure. Figure 6B As shown, in step 651, the at least one processor divides a matrix associated with the neural network into a plurality of vectors. In some embodiments, the matrix may be a weight matrix. For example, Figure 4 As explained, an M×N matrix is ​​divided into a plurality of L-dimensional vectors. In some embodiments, if the M×N matrix is ​​not evenly divided into the L-dimensional vectors, method 650 may include padding one or more residual vectors with zeros so that each vector has the same size L. It may be noted that this division may be by column or by row.

[0078] In step 653, at least one processor may select a subset of non-zero elements from one or more vectors of the plurality of vectors to form a pruning matrix. For example, for each L-dimensional vector, at least one processor may select K non-zero elements. K is a predetermined value and is less than or equal to L. In some embodiments, K may be set to the maximum number of non-zero elements of all vectors in the matrix. Thus, all non-zero elements in the matrix are selected to form the pruning matrix. Alternatively, fewer non-zero elements than all non-zero elements are selected from the matrix according to a predetermined criterion. For example, K non-zero elements with the largest absolute values ​​may be selected to form the pruning matrix. In some embodiments, the selection may include returning the unselected non-zero elements in each of the plurality of vectors to zero to form the pruning matrix.

[0079] In step 655, at least one processor may retrain the neural network using the pruned matrix. For example, the matrix may be a weight matrix, and the retraining may include applying a test dataset or a training dataset to the neural network, the neural network using the pruned weight matrix obtained via steps 651 and 653, and retraining may also include modifying the neural network accordingly to reduce one or more associated loss functions calculated after applying the test dataset or the training dataset. In some embodiments, the retraining may include modifying one or more elements of the weight matrix, or modifying one or more activation functions of one or more nodes of the neural network.

[0080] At step 657, at least one processor may determine whether the retrained neural network has converged. For example, convergence may occur if a desired level of sparsity has been achieved, if the accuracy of the neural network has dropped below a threshold, or if any other numerical value associated with the neural network has reached or exceeded a predetermined threshold. If convergence has occurred, then method 650 may end; if not, then method 650 may continue to iterate, as described in Figure 6B shown.

[0081] Some embodiments of the present disclosure may provide some flexibility in specifying convergence conditions (e.g., acceptable error rates), which typically vary across different applications. For example, if an application is more sensitive to latency than accuracy, the maximum accuracy drop can be set higher to achieve more sparsity. Alternatively, the maximum accuracy drop can be set small enough to ensure accuracy.

[0082] Although described above with respect to a single iteration, method 650 may further include iterative selection: in step 653, further iteratively selecting a second subset of non-zero elements from one or more vectors in the plurality of vectors to form a second pruned matrix, and retraining the neural network using the second pruned matrix in step 655. In some embodiments, selecting the second subset of non-zero elements may include reducing a predetermined number used in the first iteration, and selecting the largest non-zero element from each of the plurality of vectors according to the reduced predetermined number.

[0083] This iteration may be terminated after a desired level of sparsity is reached or the accuracy of the neural network falls below a threshold. In such embodiments, the neural network obtained in the final iteration (or, in embodiments where the accuracy has fallen below the threshold, the neural network obtained in the penultimate iteration) may be executed or stored, or otherwise finalized. Thus, as described with respect to step 657, if the accuracy of the retrained neural network falls below the threshold, the iteration may be terminated.

[0084] Optionally, at least one processor may encode the retrained subset of non-zero elements into a compressed vector having associated offsets of the selected subset of non-zero elements in the plurality of vectors in step 659. If a vector has fewer non-zero elements than the length of the compressed vector, the corresponding compressed vector may be padded with zeros.

[0085] Method 650 may further include additional steps. For example, at least one processor may output the retrained pruned matrix or the compressed vector with the associated offset (if an encoding step is present) to use the pruned matrix ( Figure 6B Additionally or alternatively, the at least one processor may execute a neural network having compressed vectors and associated offsets ( Figure 6B (Not shown). For example, at least one processor may retrieve the offset and perform a multiplication of the compressed vector and the matrix by multiplying the element of the compressed vector associated with the extracted offset with the element of the matrix corresponding to the extracted offset.

[0086] Figure 7 is a schematic diagram of an exemplary vector matrix multiplication 700 according to an embodiment of the present disclosure. In some embodiments, the matrix multiplication 700 can be Figure 5 Code 500, Figure 6A Method 600 or Figure 6B For example, the matrix multiplication 700 can be combined with the method 650 of Figure 5 Code 500, Figure 6A Method 600 or Figure 6B The encoding matrix generated by method 650.

[0087] Matrix multiplication 700 may include three matrices: matrix A, dense matrix B, and dense matrix C, where C = A × B. The sizes of matrices A, B, and C are 4 × 8, 8 × 6, and 4 × 6, respectively. In traditional dense matrix multiplication, sparsity is not considered, and the product of each row of matrix A and each column of matrix B needs to be calculated. Therefore, each row of matrix C requires 48 (48 = 8 × 6) multiplications. Figure 7 As shown, after pruning and encoding according to some embodiments of the present disclosure, the 4×8 matrix A becomes a 4×2 encoding matrix A 701, which has an associated offset matrix 703 of the same size. Therefore, the matrix multiplication 700 may only require a subset of the elements of matrix B 705 to compute the product matrix C 707. For example, since the first row of the encoding matrix A 701 has non-zero elements at offsets 1 and 6 (e.g., columns 1 and 6), the first row c0 in matrix C is equivalent to NZ0*b1+NZ1*b6, where b1 and b6 represent the corresponding rows 1 and 6 in matrix B. Therefore, in Figure 7In the example of , only 12 (12=2×6) multiplications are performed, thereby reducing the multiplication calculations by 75%. It should be understood that any level of multiplication reduction can be achieved using the vector sparsification technique described herein.

[0088] Figure 8 is a schematic diagram of an exemplary processor 800 according to an embodiment of the present disclosure. The processor 800 may execute Figure 4 The division of 400, Figure 5 Code 500, Figure 6A Method 600, Figure 6B Method 650, Figure 7 Matrix multiplication 700 or other embodiments of the present disclosure. The processor 800 may be a CPU (e.g., Figures 2A-2B host unit 220), GPU, NPU (e.g., Figures 2A-2B accelerator architecture 200), TPU, FPGA, ASIC, etc., or part of the above processing unit.

[0089] like Figure 8 As shown, the processor 800 may include a level 1 (L1) instruction cache 801, one or more sub-cores 803, and a memory IO 805. Figure 8 Four sub-cores 803-1, 803-2, 803-3, and 803-4 are shown in the figure, but any number of sub-cores 803 may be included in the processor 800. In some embodiments, the sub-cores may include a warp scheduler 8031, a single instruction, multiple data (SIMD) scheduling unit 8032, one or more tensor cores 8033, a processor array for multiple data types (e.g., a Compute Unified Device Architecture (CUDA) core) and a special function unit (SFU) 8034, a load / store unit (LD / ST) 8035, and a register file 8036. The tensor core 8033 can provide multiple execution modes: floating point 16 (FP16) mode, mixed precision mode, FP32 mode, and FP64 mode. In FP16 / FP32 / FP64 mode, all matrix elements are FP16 / FP32 / FP64. In mixed precision mode, the tensor core 8033 uses floating point 32 (FP32) accumulators and writes the results back to FP32 matrices. During computation, the tensor cores 8033 (eg, two tensor cores) in the sub-core 803 may be used simultaneously by warps.

[0090] The memory IO 805 may include a unified level 1 (L1) data cache, a shared memory 8051 , and a texture memory 8052 . The level 1 (L1) data cache and the shared memory 8051 may be shared among the plurality of sub-cores 803 .

[0091] WMMA can calculate the multiplication and accumulation of three matrices A, B, and C to obtain matrix D, where D = A × B + C. To perform WMMA, multiple threads in a thread warp are divided into thread groups. In addition, to better reuse data, many thread groups can work together as a work tuple. For example, assuming that A, B, C, and D are 16×16 matrices, the 32 threads in a thread warp are divided into 8 thread groups to perform WMMA. All threads in a thread group calculate the multiplication of 4×4×4 slices together. Two thread groups work together as a work tuple. Work tuple i includes thread group i and thread group i+4.

[0092] Figure 9 is a schematic diagram of mapping matrix multiplication to work tuples in a warp according to an embodiment of the present disclosure. In some embodiments, the mapping may be performed by Figures 2A-2B The accelerator architecture 200 shown, Figure 8 The processor 800 shown or Figure 11 The processor 1100 is shown to implement or utilize. Although matrices A, B, C, and D are Figure 9 A 16x16 matrix is ​​shown, but it will be appreciated that these matrices may be of other sizes.

[0093] Specifically, Figure 9 Depicts the element-wise processing of each work tuple in a WMMA operation. Matrix A can be associated with four work tuples 0-3 in the row direction, and matrix B can be associated with four work tuples 0-3 in the column direction. Matrices C and D can be evenly mapped to the four work tuples 0-3 in a tile-wise manner. Therefore, the computation of the product matrix D is evenly divided among the four work tuples. Each work tuple is responsible for computing an 8×8 slice of D. For example, work tuple 0 computes [0:7, 0:7] of D. To do this, work tuple 0 needs to multiply A[0:7,0:15] and B[0:15,0:7], add the 8×8 product slice to C[0:7,0:7], and save the result to [0:7,0:7] of D.

[0094] Figure 10 1001 (left) and vector sparse mode 1002 (right) according to an exemplary embodiment of the present disclosure. In dense mode 1001, during compilation, WMMA operations may be decomposed into four sets of machine-level HMMA (half-precision matrix multiply-accumulate) instructions. For two thread groups, a work tuple may calculate 8×8×4 matrix multiplications in one set of HMMA instructions. A set of HMMA instructions for mixed precision mode may include four HMMA instructions (such as Figure 10(shown as 1001 on the left).

[0095] In vector sparse mode 1002, instead of four HMMA instructions in a group as in dense mode 1001, a group of sparse HMMA (SHMMA) instructions may include:

[0096] SHMMA.FETCHIDX RO;

[0097] SHMMA.EXEC.F32.F32 RD,RA,RB,RC.

[0098] The instruction SHMMA.FETCHIDX can retrieve the offset index of a row of elements of matrix A from RO and store it in the offset register. The instruction SHMMA.EXEC.F32.F32 can decode the offset register to determine which rows of matrix B to retrieve from RB, calculate the multiplication of matrix A and matrix B, and accumulate the result with matrix C. It is worth noting that although the SHMMA instruction is described, in fact, any suitable matrix multiplication and accumulation instruction on any hardware accelerator can be used to implement the exemplary processing described herein.

[0099] For sparse WMMA (SWMMA) operations, there can be multiple sets of SHMMA instructions. Figure 10 As shown on the right, a SWMMA operation can include four groups of SHMMA instructions: Group 0, Group 1, Group 2, and Group 3. Each thread group computes eight columns of matrix B using one group of SHMMA instructions. Unlike dense mode 1001, in vector sparse mode 1002, two thread groups can compute the same row of matrix A. Row i of D can be computed by multiplying the four non-zero elements in row i of matrix A with the corresponding four rows of matrix B, and then accumulating the results with row i of C. The four rows of matrix B to be multiplied are determined by the four offset indices stored in the offset registers.

[0100] Figure 11 is a schematic diagram of an exemplary processor architecture 1100 according to an embodiment of the present disclosure. Figure 11 Only a portion of the processor 1100 is shown. In some embodiments, the processor 1100 may be used to Figure 8 The sub-core 803. Figure 11As shown, the processor 1100 may include a memory (e.g., a register file 1101), multiple operand buses 1103 (e.g., operand bus 1103-1, operand bus 1103-2, and operand bus 1103-3), and at least one core 1105 (e.g., core 1105-1 and core 1105-2). The memory may include the register file 1101, and the register file 1101 may include an offset register 11011. In some embodiments, the offset register 11011 may be a dedicated offset register and is implicitly accessed by the SHMMA instruction.

[0101] Core 1105 (e.g., a tensor core) can decode an offset stored in memory (e.g., decode offset register 11011) and perform a matrix multiplication on a first operand from a first operand bus (e.g., operand bus 1103-1) and a second operand from a second operand bus (e.g., operand bus 1103-2). In some embodiments, the first operand can be a matrix A, which can be processed in any of the methods disclosed herein (e.g., Figure 5 Code 500, Figure 6A Method 600 or Figure 6B The method 650) is encoded, and the matrix A includes a plurality of compressed vectors, and the corresponding offsets of the elements of the compressed vectors within the original vectors are stored in memory (for example, in the offset register 11011). The second operand can be another matrix B. Decoding the offsets can include determining which rows or columns of the matrix B to extract. Therefore, referring to Figure 7 As shown, core 1105 may perform multiplications of the plurality of compressed vectors of the encoding matrix A with corresponding rows or columns extracted from matrix B. In addition, core 1105 may accumulate the matrix multiplication result of matrices A and B with a third operand from a third operand bus (e.g., operand bus 1103-3).

[0102] In some embodiments, core 1105 may include one or more groups of processing units. Each group of processing units corresponds to a thread group. For example, Figure 11 As shown, core 1105 - 1 may include two octet units 0 and 1. Octet unit 0 includes two sets of four dot product (DP) units, one set corresponding to thread group 0 and the other set corresponding to thread group 4.

[0103] The core 1105 may also include multiple buffers. Figure 11As shown, buffer A can be connected to operand bus 1103-1 and to a group of processing units (e.g., DP units) corresponding to thread group 0 or thread group 4. Buffer B can be connected to operand bus 1103-2 and to at least one group of processing units, for example, two groups of processing units corresponding to thread group 0 and thread group 4. In some embodiments, buffer B can be a ping-pong buffer comprising a first portion for storing data to be read by a processing unit and a second portion for loading data from a memory (e.g., register file 11011). Core 1105 may further include a multiplexer connected between operand bus 1103-2 and buffer B. In addition, core 1105 may further include a third buffer (e.g., an accumulator buffer) connected to operand bus 1103-3 and to a group of processing units, for example, a group of DP units corresponding to thread group 0 or thread group 4. Thus, the core 1105 may accumulate the result of the matrix multiplication of the first operand from the operand bus 1103-1 and the second operand from the operand bus 1103-2 with the third operand from the operand bus 1103-3. After the calculation, the core 1105 writes back the calculation result.

[0104] In some embodiments, SWMMA operations may be performed by Figure 11 The processor 1100 is implemented as shown in FIG. Each core 1105 includes two octet units. Within an octet unit (e.g., octet unit 0, octet unit 1, octet unit 2, or octet unit 3), there are eight DP units, each of which computes a 4-dimensional vector dot product per cycle. During execution, a work tuple can be mapped to one octet unit, so each thread group has four DP units. The octet unit can have an operand buffer to provide data to the work tuple when executing a set of SHMMA instructions. Each thread group can have operand buffers for both operand A and operand C, for example, buffer A and an accumulator buffer. In addition, an operand buffer for operand B (e.g., buffer B) can store data shared by two thread groups in the same work tuple. When the octet unit loads 4×8 tiles from each set of HMMA instructions, the existing buffer B only needs to store 4×8 FP16 values. In some embodiments, the size of buffer B can be doubled to accommodate four rows of operand B. Buffer B can be a ping-pong buffer, a portion of which can be read by the DP and the other portion of which is loaded from memory (e.g., register file 11011). In some embodiments, operand A in buffer A can be broadcast to the four DP units to which it is connected, so that all DP units in an octet unit can read the same row in operand A. In this way, the thread group can calculate the dot product of one row of operand A and four columns of operand B per clock cycle.

[0105] Figure 12 An exemplary SWMMA execution flow according to an embodiment of the present disclosure is shown. The exemplary SWMMA execution flow can be Figure 8 The processor 800 shown or Figure 11 The illustrated processor 1100 implements the SWMMA execution flow which may be executed using SHMMA instructions.

[0106] like Figure 12 As shown, matrix A can be a vector-encoded 16*4 matrix with an associated 16×4 offset matrix (L=16, K=4). The four rows of encoding matrix A can be mapped to a work tuple. Work tuple i computes rows 4i to 4i+3. For example, when executing instructions in group 0, the first row of the offset matrix, which contains four elements: 4, 6, 8, and 9, can be extracted and decoded. The corresponding rows in dense matrix B (e.g., rows 4, 6, 8, and 9) can be extracted and multiplied with the first row of encoding matrix A. The multiplication results can be accumulated with the first row of dense matrix C. Since all offsets in the offset matrix are in the range [0, 15], each offset may require 4 bits of storage space. Therefore, each row of encoding matrix A may require 16 bits of storage space to store four offsets, which can be stored in a register.

[0107] Give an example.

[0108] To demonstrate the potential efficiency gains achieved using the disclosed techniques for executing neural networks, we developed and performed multiple simulations on five popular neural networks from three domains: image classification, image captioning, and machine translation. The neural networks were trained using both existing common sparsification methods and the disclosed vector sparsification method on a single DGX-1 machine with four NVIDIA Tesla V100 GPUs. Matrix multiplication was implemented using CUTLASS, an open-source, high-performance GEMM template library.

[0109] For the image classification application, four CNNs, AlexNet, VGG-16, ResNet-18, and ResNet-50, based on the 2012 ImageNet Large Scale Visual Recognition Challenge (ImageNetILSVRC-2012) dataset, were selected. For the image captioning application, the Show and Tell model was used, which consists of an Inception V3 model with a long short-term memory (LSTM) layer attached to the last layer of the CNN. By default, the LSTM layer has 512 units. The Show and Tell model has been applied to The Objects in Context (MSCOCO) dataset was used with a minimum batch size of 64. For machine translation applications, a Neural Machine Translation (NMT) model used an architecture consisting of a 2-layer LSTM encoder, a 4-layer LSTM decoder, and an attention module. The NMT model used the English-German dataset from the 2016 Conference on Machine Translation (WMT 16). The Bilingual Evaluation Understudy (BLEU) score was used as a metric for the NMT model.

[0110] In this example, the workload is first trained using the default training method to achieve the accuracy of the reference model. Vector sparsification is then applied to the reference dense model. Weights, activations, and gradients during training, as well as inference kernels based on CUDA cores, all use FP32. For Tensor Core-based kernels, the FP32 weights and input activations of each layer are dynamically downscaled to FP16 to avoid loss of accuracy. Output activations remain in FP32.

[0111] In the vector sparsification method according to an embodiment of the present disclosure, a vector size L is introduced. Table 1 shows the accuracy comparison between CNN and RNN between generic sparsification and vector sparsification, where Generic refers to generic sparsification and VectorSparse refers to vector sparsification of an embodiment of the present disclosure.

[0112] Table 1

[0113]

[0114]

[0115] As shown in Table 1, each weight matrix is ​​sparsified by four L configurations: 8, 16, 32, and 64. Although only results for 75% sparsity are shown, various sparsities are explored without loss of generality. The results show that larger vector sizes (e.g., L=16) have a small impact on accuracy regardless of the sparsity. In some cases, smaller vector sizes (such as L=8) may result in a significant decrease in accuracy. Since accuracy is not sensitive to larger vector sizes, in some embodiments, L=16 may be selected as a more preferred vector size, which requires 4 bits to store the offset index and supports a finer-grained tiling strategy. In these examples, when the sparsity is 75% and L=16, the accuracy of vector sparsification drops to within 1%.

[0116] Figure 13 is a graphical depiction of accuracy and sparsity in a weight matrix of a convolutional neural network (CNN) load according to an embodiment of the present disclosure. Figure 14 is a graphical depiction of Bilingual Evaluation Undergraduate (BLEU) scores and sparsity in a weight matrix of a recurrent neural network (RNN) load according to an embodiment of the present disclosure. Figure 13-14 In

[15] , CNN and RNN models are sparsified using vector sparsification method with L = 8 and 16, universal sparsification method and unified sparsification method respectively.

[0117] like Figure 13 As shown in , all CNN models can maintain accuracy when vectors are sparsely populated with L = 16 until the sparsity reaches 80%. Similarly, Figure 14 As shown in Figure 3, when the sparsity does not exceed 75%, the accuracy of the RNN is comparable to the reference model. If the vector size L is set to 8, the model accuracy decreases faster than when L = 16. At a sparsity of 75%, the accuracy loss of the scheme with L = 8 may exceed 2%, because L = 8 may impose too many spatial constraints on deleting elements. Figure 13-14 It is shown that RNNs are more resilient to pruning with L=8 than CNNs. However, their accuracy when sparsifying with L=8 is still not comparable to that when sparsifying with L=16. Therefore, a trade-off between sparsity and accuracy can be demonstrated. For a given vector size L, higher sparsity can achieve better performance at the expense of accuracy. It is understood that this trade-off can be made depending on the specific application. In addition, compared with uniform sparsification, vector sparsification has more flexibility in pruning weights, so when L is not very small, vector sparsification can produce a similar topology as universal sparsification. For example, when sparsifying with L=16, a sparsity of 75% is sufficient to guarantee accuracy. Therefore, 4 non-zero elements are retained in each 16-dimensional vector. Since the thread warp size 32 is 16 5 Multiples of L, L = 16 is also beneficial to CUDA cores.

[0118] Figure 15 Schematic diagram of normalized acceleration on a CUDA core based on a dense neural network according to an embodiment of the present disclosure. Figure 15 According to the embodiments of the present disclosure (e.g. Figure 111105), CUDA CoreDense refers to the baseline dense neural network running on CUDA Core, CUDA Core Generic refers to the neural network after generic sparsification running on CUDA Core, CUDA Core Unified refers to the neural network after unified sparsification running on CUDA Core, CUDA Core Vector-wise Sparse refers to the neural network after vector sparsification running on CUDA Core, Tensor Core Dense refers to the dense neural network running on Tensor Cores, and Tensor Core Vector-wise Sparse refers to the neural network after vector sparsification running on Tensor Cores. The neural network after vector sparsification has 75% sparsity, the neural network after generic sparsification has 96% sparsity, and the neural network after unified sparsification has 50% sparsity. All results are normalized to CUDA Core Dense.

[0119] like Figure 15 As shown, Tensor Core Dense is faster than CUDA Core Dense, while CUDA Core Generic is 20% slower than CUDA Core Dense. Alternatively, CUDA Core Unified is 1.50 times faster than CUDA Core Dense on average. On average, Tensor Core Vector-wise Sparse is 2.65 times faster than the baseline Tensor Core Dense. This performance improvement may be due to the following reasons: first, under relaxed spatial constraints, vector-sparse neural networks benefit from high sparsity, resulting in a 64% performance gain for CUDA Core Vector-wise Sparse over the baseline. Second, based on the disclosed SHMMA instructions and architecture, these vector-sparse neural networks can leverage the powerful Tensor Cores, resulting in an additional 61% performance improvement compared to CUDA Core Vector-wise Sparse. Furthermore, Tensor Core Vector-wise Sparse achieves a 1.49x speedup over Tensor Core Dense.

[0120] In vector sparse mode, the kernel according to an embodiment of the present disclosure (e.g., Figure 11Tensor Core 1105) may need to provide a 4*large value buffer (e.g., buffer B) for operand B to hold 4×16 values ​​of FP16 and enable ping-pong buffers. In addition, an offset register can be added for each octet unit. The existing size of buffer B in each octet unit is 512B (512=4×8×16), and each core needs to have two octet units, so the buffer size is 1KB. Therefore, vector sparsity mode may require a 4KB buffer B. If the processor (e.g., Figure 8 Processor 800 or Figure 11 If the processor 1100 has 8 cores, a 4KB buffer may be required.

[0121] Table 2

[0122]

[0123] Table 2 shows the cost analysis of the design using CACTI 7. 4KB of static random access memory (SRAM) occupies 0.019mm on a 22nm process node. 2 The 0.4ns cycle time is smaller than the nominal cycle time of the NVIDIA Tesla V100 GPU (0.65ns at 1530MHz), so there is no time overhead. Since the V100 GPU uses a 12nm process, the area is reduced to 0.007mm 2 . In addition, if a kernel (e.g. Figure 11 The tensor core 1105 of the V100 GPU requires two additional registers as two-octet offset registers, so the processor needs 16 additional offset registers to extract operands to buffer B. If the area of ​​the V100 GPU is 815mm 2 , then the total area overhead can be neglected.

[0124] The embodiments of the present disclosure can be applied to many products, environments, and scenarios. For example, some embodiments of the present disclosure can be applied to processors (e.g., CPUs), Alibaba NPUs (e.g., Hanguang NPUs), Alibaba Cloud, Alibaba PIM-AI (Processor in Memory for AI), Alibaba DPUs (Database Acceleration Units), Alibaba AI platforms, GPUs, TPUs, and the like.

[0125] The following sentences may be used to further describe the embodiments of the present disclosure.

[0126] 1. A method for providing vector sparsification in a neural network, comprising:

[0127] Divide the matrix associated with the neural network into multiple vectors;

[0128] selecting a first subset of non-zero elements from the plurality of vectors to form a pruning matrix; and

[0129] The pruning matrix is ​​output, and the neural network is executed using the pruning matrix.

[0130] 2. The method according to claim 1, further comprising padding residual vectors in the plurality of vectors with zeros.

[0131] 3. The method according to claim 1 or 2, further comprising: retraining the neural network using the pruning matrix.

[0132] 4. The method according to claim 3, wherein the retraining comprises at least one of the following: modifying one or more elements of a pruning matrix, or modifying one or more activation functions of one or more nodes of a neural network.

[0133] 5. The method according to claim 3 or 4, further comprising the following steps which are iteratively performed:

[0134] selecting a second subset of non-zero elements from the plurality of vectors to form a second pruning matrix; and

[0135] Retrain the neural network using the second pruned matrix.

[0136] 6. The method of claim 5, wherein selecting the first subset of non-zero elements further comprises selecting a predetermined number of largest non-zero elements from each of the plurality of vectors.

[0137] 7. The method of claim 6, wherein selecting the second subset of non-zero elements further comprises:

[0138] reduce the predetermined amount; and

[0139] The largest non-zero elements in each of the plurality of vectors are selected according to the reduced predetermined number.

[0140] 8. The method according to any one of claims 5 to 7, further comprising: stopping the iteration if the accuracy of the retrained neural network is lower than a threshold.

[0141] 9. The method of any one of claims 1 to 8, wherein selecting the subset of non-zero elements comprises zeroing out unselected non-zero elements in each of the plurality of vectors.

[0142] 10. The method according to any one of claims 1 to 9, further comprising encoding the selected subset of non-zero elements into a compressed vector having associated offsets of the selected subset of non-zero elements within the one or more vectors.

[0143] 11. The method of claim 10, wherein the compressed vector has a predetermined length, encoding the selected subset of non-zero elements further comprising padding the elements of the compressed vector other than the selected non-zero elements with zeros if the non-zero elements selected for the compressed vector are fewer than the predetermined length.

[0144] 12. The method according to claim 10 or 11, further comprising:

[0145] Extract offset;

[0146] Multiplication of the compressed vector with a matrix is ​​performed by multiplying the elements of the compressed vector associated with the extracted offsets with the elements of the matrix corresponding to the offsets.

[0147] 13. The method according to any one of claims 1 to 12, wherein the matrix is ​​a weight matrix.

[0148] 14. A system for providing vector sparsification in a neural network, comprising:

[0149] at least one memory for storing instructions; and

[0150] At least one processor, configured to execute the instructions to cause the system to perform the following steps:

[0151] Divide the matrix associated with the neural network into multiple vectors;

[0152] selecting a first subset of non-zero elements from the plurality of vectors to form a pruning matrix; and

[0153] A pruning matrix is ​​output, and the neural network is executed using the pruning matrix.

[0154] 15. The system of claim 14, wherein the at least one processor further executes instructions to cause the system to perform padding of residual vectors in the plurality of vectors with zeros.

[0155] 16. The system of claim 14 or 15, wherein the at least one processor further executes instructions to cause the system to retrain the neural network using the pruning matrix.

[0156] 17. The system of claim 16, wherein the at least one processor further executes instructions to cause the system to modify one or more elements of a pruning matrix or modify one or more activation functions of one or more nodes of a neural network.

[0157] 18. The system of claim 16 or 17, wherein the at least one processor further executes instructions to cause the system to iteratively perform the following steps:

[0158] selecting a second subset of non-zero elements from the plurality of vectors to form a second pruning matrix; and

[0159] Retrain the neural network using the second pruned matrix.

[0160] 19. The system of claim 18, wherein the at least one processor further executes instructions to cause the system to perform selecting a predetermined number of largest non-zero elements from each of the plurality of vectors.

[0161] 20. The system of claim 19, wherein the at least one processor further executes instructions to cause the system to perform the following steps:

[0162] reduce the predetermined amount; and

[0163] The largest non-zero elements in each of the plurality of vectors are selected according to the reduced predetermined number.

[0164] 21. The system of any one of claims 18-10, wherein the at least one processor further executes instructions to cause the system to stop iterating if the accuracy of the retrained neural network is below a threshold.

[0165] 22. The system of any one of claims 14-21, wherein the at least one processor further executes instructions to cause the system to perform zeroing of unselected non-zero elements in each of the plurality of vectors.

[0166] 23. The system of any one of claims 14-22, wherein the at least one processor further executes instructions to cause the system to perform encoding of the selected subset of non-zero elements into a compressed vector having associated offsets of the selected subset of non-zero elements in one or more vectors.

[0167] 24. The system of claim 23, wherein the compressed vector has a predetermined length, and wherein the at least one processor further executes instructions to cause the system to perform the following steps:

[0168] If the non-zero elements selected for the compressed vector are fewer than a predetermined length, the elements other than the selected non-zero elements in the compressed vector are padded with zeros.

[0169] 25. The system of claim 23 or 24, wherein the at least one processor executes instructions to cause the system to:

[0170] Extract offset;

[0171] Multiplication of the compressed vector with a matrix is ​​performed by multiplying the elements of the compressed vector associated with the extracted offsets with the elements of the matrix corresponding to the offsets.

[0172] 26. The system of any one of claims 14-25, wherein the matrix is ​​a weight matrix.

[0173] 27. A non-transitory computer-readable storage medium storing a set of instructions, wherein execution of the instructions by at least one processor causes a computer to perform a method for providing vector sparsification in a neural network, the method comprising:

[0174] Divide the matrix associated with the neural network into multiple vectors;

[0175] selecting a first subset of non-zero elements from the plurality of vectors to form a pruning matrix; and

[0176] The pruning matrix is ​​output, and the neural network is executed using the pruning matrix.

[0177] 28. The non-transitory computer-readable storage medium of claim 27, wherein the set of instructions, when executed by at least one processor, causes the computer to further perform: padding residual vectors in the plurality of vectors with zeros.

[0178] 29. The non-transitory computer-readable storage medium of claim 27 or 28, wherein execution of the set of instructions by the at least one processor causes the computer to further perform: retraining a neural network using a pruning matrix.

[0179] 30. The non-transitory computer-readable storage medium of claim 29, wherein the retraining comprises at least one of modifying one or more elements of a pruning matrix, or modifying one or more activation functions of one or more nodes of a neural network.

[0180] 31. The non-transitory computer-readable storage medium of claim 29 or 30, wherein execution of the set of instructions by the at least one processor causes the computer to further perform:

[0181] selecting a second subset of non-zero elements from the plurality of vectors to form a second pruning matrix; and

[0182] Retrain the neural network using the second pruned matrix.

[0183] 32. The non-transitory computer-readable storage medium of claim 31, wherein selecting a first subset of non-zero elements further comprises selecting a predetermined number of largest non-zero elements from each of the plurality of vectors.

[0184] 33. The non-transitory computer-readable storage medium of claim 32, wherein selecting the second subset of non-zero elements further comprises:

[0185] reduce the predetermined amount; and

[0186] The largest non-zero elements in each of the plurality of vectors are selected according to the reduced predetermined number.

[0187] 34. The non-transitory computer-readable storage medium of any one of claims 31-33, wherein execution of the set of instructions by the at least one processor causes the computer to further perform:

[0188] If the accuracy of the retrained neural network falls below a threshold, the iteration is stopped.

[0189] 35. The non-transitory computer-readable storage medium of any one of claims 27-34, wherein selecting the subset of non-zero elements comprises zeroing unselected non-zero elements in each of the plurality of vectors.

[0190] 36. The non-transitory computer-readable storage medium of any one of claims 27-35, wherein execution of the set of instructions by the at least one processor causes the computer to further perform:

[0191] The selected subset of non-zero elements is encoded as a compressed vector having associated offsets of the selected subset of non-zero elements in the one or more vectors.

[0192] 37. The non-transitory computer-readable storage medium of claim 36, wherein the compressed vector has a predetermined length, and encoding the selected subset of non-zero elements further comprises: if the non-zero elements selected for the compressed vector are fewer than the predetermined length, padding the elements in the compressed vector other than the selected non-zero elements with zeros.

[0193] 38. The non-transitory computer-readable storage medium of claim 36 or 37, wherein the set of instructions executable by the at least one processor causes the computer to further perform the following steps:

[0194] Extract offset;

[0195] Multiplication of the compressed vector with a matrix is ​​performed by multiplying the elements of the compressed vector associated with the extracted offsets with the elements of the matrix corresponding to the offsets.

[0196] 39. The non-transitory computer-readable storage medium of any one of claims 27-38, wherein the matrix is ​​a weight matrix.

[0197] 40. A processor for executing a neural network, comprising:

[0198] A memory for storing the relative offset of each element in the compressed vector in the original vector; and

[0199] At least one core is configured to decode offsets of respective elements in a first plurality of compressed vectors and perform matrix multiplication of a first operand and a second operand based on the decoded offsets, wherein the first operand comprises the first plurality of compressed vectors.

[0200] 41. The processor of claim 40, wherein the at least one core is configured to determine which rows or columns to extract from the second operand.

[0201] 42. The processor of claim 41, wherein the at least one core is further configured to multiply the plurality of packed vectors of the first operand with the extracted corresponding rows or columns of the first operation.

[0202] 43. The processor of claim 40, further comprising a plurality of operand buses, the plurality of operand buses comprising a first operand bus connected to the at least one core to provide the first operand and a second operand bus connected to the at least one core to provide the second operand.

[0203] 44. The processor of claim 43, wherein each of the at least one core comprises:

[0204] One or more groups of processing units, each group of processing units corresponds to a thread group;

[0205] a first buffer connected to the first operand bus and the first group of processing units;

[0206] A second buffer is connected to the second operand bus and at least one of the one or more groups of processing units.

[0207] 45. The processor of claim 44, wherein the second buffer comprises a first portion and a second portion, the first portion being used to store data read by the processing unit, and the second portion being used to load data from a register file.

[0208] 46. ​​The processor of claim 44, wherein the processing unit is a dot product (DP) unit.

[0209] 47. The processor of claim 44, wherein each of the at least one core further comprises:

[0210] A multiplexer is connected between the second operand bus and the second buffer.

[0211] 48. The processor of claim 44, wherein each of the at least one core comprises a third buffer connected to a third operand bus and the first group of processing units, the at least one core being configured to accumulate a result of the matrix multiplication of the first operand and the second operand with a third operand from the third operand bus.

[0212] Various exemplary embodiments are described herein in the context of method steps or processes, which, on the one hand, can be implemented by a computer program product, embodied in a computer-readable medium, and including computer-executable instructions of program code, for example, executed by a computer in a network environment. Computer-readable media can include removable and non-removable storage devices, including but not limited to read-only memory (ROM), random access memory (RAM), compact discs (CD), digital versatile discs (DVD), etc. Generally, program modules can include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of program code that is used to perform the steps of the methods disclosed herein. A specific sequence of executable instructions or associated data structures represents an example of corresponding operations for implementing the functions described in the above steps or processes.

[0213] The above description is for illustrative purposes only. However, this description is not exhaustive, and the present disclosure is not limited to the specific forms or embodiments disclosed herein. Modifications and adaptations of the embodiments will be apparent from consideration of the description and practice of the disclosed embodiments. For example, the described implementation includes hardware, but systems and methods consistent with the present disclosure may be implemented using both hardware and software. Furthermore, although certain components are described as coupled to each other, these components may be integrated with each other or distributed in any suitable manner.

[0214] Furthermore, although exemplary embodiments are described herein, the scope thereof shall include any and all equivalent elements, modifications, omissions, combinations (e.g., aspects of the various embodiments), adaptations, or selections based on the present disclosure. The elements of the claims are to be interpreted broadly based on the language used in the claims and are not limited to the examples described in this specification or during the application process, which are to be construed as non-exclusive. Furthermore, the steps of the disclosed methods may be modified in any manner, including reordering steps and / or inserting or deleting steps.

[0215] Based on the description herein, the features and advantages of the present disclosure are apparent, and thus the appended claims are intended to cover all systems and methods that fall within the true spirit and scope of the present application. As used herein, the indefinite articles "a" and "an" mean "one or more." Similarly, the use of plural terms does not necessarily indicate a plurality unless the context clearly indicates otherwise. In addition, several modifications or adaptations will readily occur upon study of the present disclosure, and thus it is not desired to limit the present disclosure to the exact construction and operation shown and described, and accordingly, all suitable modifications and equivalents within the scope of the present disclosure are intended to fall within the scope of the present disclosure.

[0216] As used herein, unless specifically stated otherwise, the term "or" includes all possible combinations unless not feasible. For example, if it is stated that a component may include A or B, then unless explicitly stated otherwise or not feasible, the component may include A, B, or A and B. As a second example, if it is stated that a component may include A, B, or C, then unless explicitly stated otherwise or not feasible, the component may include A, B, or C, or A and B, or A and C, or B and C, or A, B, and C.

[0217] Other embodiments will be apparent from consideration of the disclosed specification and practice of the various embodiments. That is, the specification and examples are to be considered as exemplary only, with the true scope and spirit of the disclosed embodiments being indicated by the following claims.

Claims

1. A method for providing vector sparsification in a neural network, comprising: training the neural network by applying a training data set to the neural network, wherein the neural network includes an input layer and is configured to perform image recognition or facial recognition using input received at the input layer; dividing a matrix associated with the trained neural network into a plurality of vectors; selecting a first subset of non-zero elements from the plurality of vectors to form a pruning matrix by selecting a predetermined number of non-zero elements in each of the vectors; retraining the trained neural network using the pruning matrix to reduce one or more associated loss function values ​​by modifying one or more elements of the pruning matrix or modifying one or more activation functions of one or more nodes of the neural network; and In response to reaching a desired sparsity level, outputting the pruned matrix and executing the retrained neural network using the pruned matrix includes encoding a selected subset of non-zero elements as a compressed vector, the compressed vector having associated offsets of the selected subset of non-zero elements within one or more vectors, outputting the pruned matrix and the compressed vectors with the associated offsets by a processor, decoding the offsets of respective elements in a first plurality of compressed vectors by at least one core of the processor, and performing a matrix multiplication of a first operand and a second operand based on the decoded offsets, accumulating the matrix multiplication result with a third operand to execute the neural network using the pruned matrix, wherein the first operand includes the first plurality of compressed vectors, and the at least one core is configured to determine a row or column to extract from the second operand.

2. The method according to claim 1, further comprising: Residual vectors in the plurality of vectors are padded with zeros.

3. The method according to claim 1 , further comprising iteratively performing: selecting a second subset of non-zero elements from the plurality of vectors to form a second pruning matrix; and The trained neural network is retrained using the second pruning matrix.

4. The method according to claim 3, wherein: Selecting the first subset of non-zero elements further includes selecting a predetermined number of largest non-zero elements from each vector of the plurality of vectors.

5. The method according to claim 4, wherein Selecting the second subset of non-zero elements also includes: reduce the scheduled amount; and The largest non-zero elements in each of the plurality of vectors are selected according to the reduced predetermined number.

6. The method according to claim 4, further comprising: If the accuracy of the neural network retrained using the second pruning matrix is ​​lower than a threshold, the iteration is stopped.

7. A system for providing vector sparsification in a neural network, comprising: at least one memory for storing instructions; and At least one processor, configured to execute the instructions to cause the system to perform the following steps: training the neural network by applying a training data set to the neural network, wherein the neural network includes an input layer and is configured to perform image recognition or facial recognition using input received at the input layer; dividing a matrix associated with the trained neural network into a plurality of vectors; selecting a first subset of non-zero elements from the plurality of vectors to form a pruning matrix by selecting a predetermined number of non-zero elements in each vector; retraining the trained neural network using the pruning matrix to reduce one or more associated loss function values ​​by modifying one or more elements of the pruning matrix or modifying one or more activation functions of one or more nodes of the neural network; and In response to reaching a desired sparsity level, outputting the pruned matrix and executing the retrained neural network using the pruned matrix includes encoding a selected subset of non-zero elements into a compressed vector, wherein the compressed vector has associated offsets of the selected subset of non-zero elements within one or more vectors, outputting the pruned matrix and the compressed vector with the associated offsets by a processor, and decoding the offsets of each element in a first plurality of compressed vectors by at least one core of the processor, and performing a matrix multiplication of a first operand and a second operand based on the decoded offsets, accumulating the matrix multiplication result with a third operand to execute the neural network using the pruned matrix, wherein the first operand includes the first plurality of compressed vectors, and the at least one core is configured to determine a row or column to extract from the second operand.

8. The system according to claim 7, wherein At least one processor includes one or more sub-cores, and a unified level 1 data cache and a shared memory shared among the one or more sub-cores to divide the matrix into a plurality of vectors.

9. A non-transitory computer-readable storage medium storing a set of instructions, the set of instructions being executable by at least one processor to cause a computer to perform a method for providing vector sparsification in a neural network, the method comprising: training the neural network by applying a training data set to the neural network, wherein the neural network includes an input layer and is configured to perform image recognition or facial recognition using input received at the input layer; dividing a matrix associated with the trained neural network into a plurality of vectors; selecting a first subset of non-zero elements from the plurality of vectors to form a pruning matrix by selecting a predetermined number of non-zero elements in each of the vectors; retraining the trained neural network using the pruning matrix to reduce one or more associated loss function values ​​by modifying one or more elements of the pruning matrix or modifying one or more activation functions of one or more nodes of the neural network; and In response to reaching a desired sparsity level, outputting the pruned matrix and executing the retrained neural network using the pruned matrix includes encoding a selected subset of non-zero elements as a compressed vector, the compressed vector having associated offsets of the selected subset of non-zero elements within one or more vectors, outputting the pruned matrix and the compressed vectors with the associated offsets by a processor, and decoding the offsets of each element in a first plurality of compressed vectors by at least one core of the processor, performing a matrix multiplication of a first operand and a second operand based on the decoded offsets, accumulating the matrix multiplication result with a third operand to execute the neural network using the pruned matrix, wherein the first operand includes the first plurality of compressed vectors, and the at least one core is configured to determine a row or column to extract from the second operand.

10. The non-transitory computer-readable storage medium of claim 9, wherein: The set of instructions is executed by the at least one processor to cause the computer to iteratively perform: selecting a second subset of non-zero elements from the plurality of vectors to form a second pruning matrix; and The trained neural network is retrained using the second pruning matrix.

11. The non-transitory computer-readable storage medium of claim 10, wherein: The selecting a first subset of non-zero elements further comprises: selecting a predetermined number of largest non-zero elements from each of the plurality of vectors, and wherein the selecting a second subset of non-zero elements further comprises: reduce the scheduled amount; and The largest non-zero elements in each of the plurality of vectors are selected according to the reduced predetermined number.

12. The non-transitory computer-readable storage medium of claim 11, wherein: The set of instructions is executed by the at least one processor to cause the computer to further perform: If the accuracy of the neural network retrained using the second pruning matrix is ​​lower than a threshold, the iteration is stopped.

Images