Techniques for removing masks from pruned neural networks

Abstract

This application provides a technique for removing masks from a pruned neural network. A demasking engine removes masks from a pruned neural network represented by a node graph. The demasking engine analyzes the tensor and mask associated with a given node in the node graph to determine the portions of the tensor that are zeroed out by the mask. The demasking engine then removes these portions from the tensor to generate a dense tensor with a dimension smaller than that of the original tensor. The function associated with the node can be evaluated faster based on the dense tensor compared to the original tensor. The demasking engine adds a scattering operation after the node to scale the dimension of the dense tensor to the dimension associated with the original tensor.

Description

Background Art

[0001] A "tensor" is a mathematical construct commonly used in linear algebra applications such as machine learning and artificial intelligence. Scalars, vectors, and matrices are examples of tensors. A neural network typically includes one or more tensors that are processed during the execution of the neural network to perform one or more operations. The value of a given tensor contained in the neural network is modified through a training process so that one or more current outputs of the neural network are close to one or more target outputs. After training is completed, some or all of the tensors contained in the neural network may be very large. Operations associated with large tensors cannot usually be performed quickly. Therefore, traditional neural networks are generally not suitable for real-time, low-latency applications, such as automatic vehicle control. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] In order that the manner in which the above-described features of various embodiments may be understood in detail, the inventive concept briefly summarized above may be described in more detail by reference to various embodiments, some of which are shown in the accompanying drawings. However, it should be noted that the drawings illustrate only typical embodiments of the inventive concept and are therefore not to be construed as limiting the scope in any way, and that there are other equally effective embodiments.

[0003] Figure 1 A system configured to implement one or more aspects of the various embodiments is shown.

[0004] Figure 2 Shown is a graphical representation of a neural network in accordance with various embodiments.

[0005] Figure 3 Shown are examples of how a node may evaluate a function based on an input tensor to generate an output tensor according to various embodiments.

[0006] Figure 4 Shown are examples of how a node may evaluate a function based on a masked input tensor to generate an output tensor in accordance with various embodiments.

[0007] Figure 5 Shown are examples of how a node may evaluate a function based on a dense version of an input tensor in accordance with various embodiments.

[0008] Figure 6 Neighboring nodes are shown over which a scatter operation may be propagated according to various embodiments.

[0009] Figure 7 shows how the dispersion operation according to various embodiments may be performed Figure 6 propagates between adjacent nodes.

[0010] Figure 8is a flow chart of method steps for removing masks from a neural network according to various embodiments.

[0011] Fig. 9 is a block diagram illustrating a computer system configured to implement one or more aspects of the various embodiments.

[0012] Fig.10 According to various embodiments, the Fig. 9 Block diagram of a parallel processing unit (PPU) in a parallel processing subsystem of FIG.

[0013] Fig.11 According to various embodiments, the Fig.10 Block diagram of a general processing cluster (GPC) in a parallel processing unit (PPU). DETAILED DESCRIPTION

[0014] In the following description, numerous specific details are set forth to provide a more thorough understanding of various embodiments. However, it will be apparent to one skilled in the art that the inventive concept may be practiced without one or more of these specific details.

[0015] As described above, a neural network may include one or more tensors that are processed during execution of the artificial neural network to perform one or more operations. Tensors may be very large and therefore cannot be processed quickly in some cases. To address this issue, the artificial neural network may be "pruned" to increase the speed at which the corresponding tensors are processed.

[0016] Pruning a neural network typically involves generating one or more masks that zero out elements of one or more tensors included in the neural network. Although pruning reduces the overall size of the neural network, pruning the neural network does not significantly and / or reliably speed up the inference time of the neural network. In particular, operations involving tensors of the pruned neural network (at least partially zeroed by the mask) are still performed, even if the results produced by these operations do not affect the final inference output of the neural network.

[0017] Various embodiments disclosed herein include a technique for removing masks from a pruned neural network, wherein the pruned neural network is represented by a graph of nodes. A given node in the graph includes at least one tensor (W) and a corresponding mask (M). The operation M·W clears the elements of W to zero. A first function associated with the given node is evaluated based on M·W to produce an output tensor X for the node. The output tensor X is provided as input to one or more subsequent nodes in the graph.

[0018] In one embodiment, to remove the mask M applied to a given node, M·W is replaced by a dense version of the tensor W represented by w. The dimension of the tensor w is smaller than that of W, and therefore, operations involving w are executed much faster than those involving the tensor W. In particular, the evaluation speed of the first function based on w is faster than the evaluation speed of the first function based on M·W. The evaluation of the first function based on the tensor w generates a smaller output tensor, represented as x. Because subsequent nodes in the neural network expect the given node to provide an output of a tensor X of larger dimension, a scatter operation is inserted in the subsequent node to add zeros to the tensor x, thereby expanding the tensor x to produce the tensor X (or a tensor of equivalent dimension). Since less data can be processed, the operations associated with the given node can be performed in a fast manner. The scatter operation can be propagated forward through the graph and propagated toward the output to accelerate other functions. The scatter operation can also be merged with other scatter operations and / or absorbed into subsequent nodes. Through the disclosed technology, a pruned neural network including one or more masks can be modified to produce an optimized neural network.

[0019] At least one technical advantage of the techniques described herein is that the optimized neural network performs inference operations faster than the original pruned neural network. Therefore, the optimized neural network is well suited for use in real-time applications (e.g., autonomous vehicles). Another advantage of the techniques described herein is that the optimized neural network can have a smaller memory footprint than the pruned neural network, thereby conserving memory resources. These technical advantages represent multiple technical advances over prior art methods.

[0020] System Overview

[0021] Figure 1 A system configured to implement one or more aspects of various embodiments is shown. As shown, in one embodiment, a neural network optimization pipeline 100 includes a training engine 110 , a pruning engine 120 , and a demasking engine 130 .

[0022] In one embodiment, the training engine 110 generates and trains the initial neural network 102 to generate the trained neural network 112. During training, the training engine 110 iteratively adjusts one or more tensors included in the initial neural network 102 based on the training data to cause the output of the initial neural network 102 to more closely match the target output. When the training is completed, the trained neural network 112 includes one or more tensors derived from the one or more tensors included in the initial neural network 102. The training engine 110 can implement any technically feasible training process to generate the trained neural network 112, including back propagation and / or gradient descent, etc. The trained neural network 112 is generally capable of performing inference operations to generate output data based on input data not included in the training data.

[0023] In one embodiment, pruning engine 120 prunes trained neural network 112 to generate masked neural network 122. During pruning, pruning engine 120 identifies redundant elements within one or more tensors included in trained neural network 112, and then generates one or more masks that cause those specific elements to be multiplied by zero (zeroed). When processing tensors, elements of those tensors that do not contribute to the output can be considered "redundant". Because the identified elements are redundant, those elements can be zeroed without adversely affecting the functional characteristics of masked neural network 122. Zeroing redundant elements in this manner can reduce the computational load associated with processing some or all of the tensors included in masked neural network 122.

[0024] In one embodiment, training engine 110 and pruning engine 120 interoperate with each other to simultaneously train initial neural network 102 and prune initial neural network 102. For example, training engine 110 may perform a first training to modify a portion of initial neural network 102, and then pruning engine 120 may perform pruning to insert one or more masks into at least a partially trained version of initial neural network 102. In this manner, training engine 110 and pruning engine 120 may coordinate operations to generate masked neural network 122.

[0025] In one embodiment, the demasking engine 130 performs a demasking process using the masked neural network 122 to generate an optimized neural network 132. During the demasking process, the demasking engine 130 removes one or more masks previously introduced by the pruning engine 120 from the masked neural network 122, applies various other modifications to the masked neural network 122 as described above, to produce the optimized neural network 132. The optimized neural network 132 has the same or similar functional characteristics as the masked neural network 122. However, the optimized neural network 132 can perform various processing operations faster than the masked neural network 122 can perform those processing operations, including inference operations, etc.

[0026] In one embodiment, any of the initial neural network 102, the trained neural network 112, the masked neural network 122, and the optimized neural network 132 can be represented by a graph of nodes coupled together by a set of edges. Each node can be associated with one or more tensors and one or more functions evaluated based on the one or more tensors. Figure 2 It shows that it can be expressed Figure 1 The node graph of any neural network shown in .

[0027] Example graph representation of a neural network

[0028] Figure 2 A graphical representation of a neural network according to various embodiments is shown. As shown, in one embodiment, the graphical representation 200 includes an input 210, a set of nodes N0, N1, and N2, and an output 220. Node N0 processes the input 210 to generate an output that is provided to node N1. Node N1 processes the received input to generate an output that is provided to node N2. Node N2 processes the received input to generate an output 220. Here, the graphical representation 200 is only one example of a node graph that can represent a neural network.

[0029] In one embodiment, each node included in the graphical representation 200 corresponds to a function and one or more tensors for a neural network. For example, node N0 may correspond to a convolution function, a concatenation function, a matrix multiplication function, an activation function, or a rectified linear unit (ReLU) function, etc. In addition, the function associated with node N0 may be evaluated based on one or more input tensors to generate one or more output tensors. In one embodiment, a given input tensor may be associated with an inbound edge of a given node, and a given output tensor may be associated with an outbound edge of a given node.

[0030] In one embodiment, the graphical representation 200 represents the initial neural network 102, and one or more nodes of the graphical representation 200 produce an output tensor that does not match the target output. Figure 1 During the training process discussed, the training engine 110 iteratively adjusts one or more input tensors associated with one or more nodes to make the output tensor more closely match the target output.

[0031] In one embodiment, the graphical representation 200 represents a trained neural network 112, and one or more nodes of the graphical representation correspond to at least partially redundant tensors. As referred to herein, a "redundant" element of a tensor is an element that does not contribute to the output of a function evaluated based on the tensor. Figure 1 During the pruning process discussed, the pruning engine 120 merges masks into one or more nodes in order to zero out redundant elements of the associated tensors, thereby generating a masked neural network 122. Figure 3-Figure 4 An example of how the pruning engine 120 may merge a mask into a node is shown.

[0032] In one embodiment, as described above, the graphical representation 200 represents a masked neural network 122, and one or more nodes of the graphical representation 200 include masks that zero out elements of input tensors associated with those nodes. Figure 1 During the demasking process discussed, the demasking engine 130 removes these masks and makes various other modifications to some or all of the nodes to generate an optimized neural network 132. The optimized neural network 132 includes tensors that are smaller in size than the corresponding tensors included in the masked neural network 122. As a result, the optimized neural network 132 may execute faster than the masked neural network 122. Figure 5-Figure 7 An example of how the demasking engine 130 may demask nodes and perform various other optimization operations is shown.

[0033] Example of the clipping and unmasking process

[0034] Figure 3 1 shows an example of how a node according to various embodiments evaluates a function based on an input tensor to generate an output tensor. As shown, in one embodiment, node 300 includes a tensor W, a function f1, and a tensor X. Node 300 may be included in a graphical representation of a trained neural network 112, etc. In the above combination Figure 1-Figure 2 During the training process discussed, the training engine 110 generates the node 300. When the node 300 is executed, the function f1 is evaluated based on the tensor W to produce the tensor X. The function f1 can be any technically feasible function configured to operate on one or more input tensors to generate one or more output tensors.

[0035] In one embodiment, the pruning engine 120 may modify the node 300 to simplify the evaluation of the function f1, as follows in conjunction with Figure 4 Described in more detail.

[0036] Figure 4 An example of how a node according to various embodiments evaluates a function based on a masked input tensor to generate an output tensor is shown. As shown, in one embodiment, node 400 includes a tensor W, a mask M, a function f1, and a tensor X. Node 400 can be included in a graphical representation of a trained neural network 112. In the above, Figure 1-Figure 2 During the pruning process discussed, the pruning engine 120 is based on Figure 3 The node 300 shown generates the node 400. In particular, the pruning engine 120 identifies the elements of W that do not contribute to the tensor X, and then generates a mask M to zero these elements, thereby reducing the computational burden of evaluating the function f1. When the node 400 is executed, the function f1 is evaluated based on the tensor W and the mask M to generate the tensor X.

[0037] In one embodiment, evaluating f1(W·M) may be computationally more efficient than evaluating f1(W). However, in practice, implementing mask M to zero the redundant elements of tensor W may not significantly improve computational efficiency because these zeroed elements are still processed during the evaluation of f1(W·M). To address this issue, the demasking engine 130 may perform techniques for removing masks and performing other optimizations, such as the following in conjunction with Figure 5-Figure 7 Described in more detail.

[0038] Figure 5 5 shows an example of how a node according to various embodiments evaluates a function based on a dense version of an input tensor. As shown, in one embodiment, a node 500 includes a tensor w, a function f1, a tensor x, a scatter operation S1, and a tensor X. Figure 1-Figure 2 During the demasking process discussed, the demasking engine 130 performs the Figure 4 Node 400 generates node 500. In particular, demasking engine 130 replaces tensor W with tensor w. Tensor w is a denser version of tensor W and has a smaller dimension than tensor W. To generate tensor w, demasking engine 130 identifies a portion of tensor W that is Figure 4 The mask M of tensor W is zeroed, and then these portions of tensor W are removed to produce tensor w. The remaining portions of tensor w are complementary to the zeroed portions of tensor W and can be said to correspond thereto. When node 500 is executed, function f1 is evaluated based on tensor w instead of tensor W. Since tensor w has a smaller dimension than tensor W, f1(w) is evaluated significantly faster than f1(W).

[0039] In one embodiment, evaluating f1(w) produces a tensor x whose dimension is less than Figure 3-Figure 4In some cases, a node that follows node 500 may expect as input a tensor with the same dimensions as tensor X. To address this issue, the demasking engine 130 generates a scatter operation S1 to expand tensor x to generate tensor X (or a tensor of equivalent dimensions). The scatter operation S1 inserts zeros into tensor x corresponding to those elements in tensor W that were previously cleared to zero by mask M and subsequently removed, thereby restoring the dimensions of tensor x to the dimensions associated with tensor X. Therefore, the node that follows node 500 receives as input a tensor with the expected dimensions of tensor X.

[0040] In one embodiment, instead of or in addition to inserting zeros, the scatter operation S1 may insert any technically feasible value into a given tensor. For example, suppose that the function f1 is a sigmoid function that maps zeros contained in the tensor W to zeros contained in the tensor X. The scatter operation S1 may insert 1 into the tensor x to compensate for the zeros removed from the tensor W.

[0041] In one embodiment, the demasking engine 130 performs the above-described techniques on each node included in the graph representation 200, thereby incorporating one or more scatter operations into the graph representation. Figure 6-Figure 7 As described in more detail, the demasking engine 130 iteratively propagates one or more of these scatter operations to the output of the graphical representation 200 .

[0042] Figure 6 1 shows neighboring nodes on which a scatter operation may be propagated according to various embodiments. As shown, Figure 5 The node 500 is adjacent to the subsequent node 600. The node 600 includes a function f2 and a tensor Y. The function f2 is evaluated based on the tensor X to produce the tensor Y. The demasking engine 130 propagates the scatter operation S1 from the node 500 to the node 600 to reduce the computational burden of evaluating the function f2(X), as shown below in conjunction with Figure 7 Describe in more detail.

[0043] Figure 7 shows how the dispersion operation according to various embodiments may be performed Figure 6 700 is propagated between adjacent nodes. As shown, node 700 includes tensor w, function f1, and tensor x. Node 700 includes the same elements as node 500, except that scatter operation S1 and tensor X are omitted. Node 710 includes function f2, tensor y, scatter operation S2, and tensor Y. The demasking engine 130 generates scatter operation S2 by forwarding scatter operation S1 to node 710. Therefore, function f2 can be evaluated based on tensor x instead of the larger tensor X. Since the dimension of tensor x is smaller than the dimension of output tensor X, f2(x) can be evaluated faster than f2(x).

[0044] In one embodiment, evaluating f2(x) produces an output tensor y with a ratio Figure 6 710. However, subsequent nodes may expect inputs with the dimensions of Y. To address this issue, scatter operation S2 expands tensor y to output tensor Y (or a tensor of equivalent dimensions). Scatter operation S2 inserts zeros into tensor y corresponding to any elements of W and / or X that are zeroed by masking. Thus, nodes subsequent to node 710 receive input tensors with the expected dimensions associated with Y.

[0045] In one embodiment, the demasking engine 130 combines the scatter operation S1 with any scatter operations previously associated with the node 710. Figure 6 The scatter operation S1 is propagated forward to the node 710. For example, the node 710 may include the demasking engine 130 combined with the above Figure 5 The demasking engine 130 combines the scatter operation S1 with any such pre-existing scatter operations associated with the node 710. In general, any two or more scatter operations may be combined when the zero values ​​inserted by those scatter operations are aligned along the same dimension. For example, two scatter operations that insert zeros along different rows may be combined to form a scatter operation that inserts zeros along both rows.

[0046] In one embodiment, the demasking engine 130 will Figure 6 The scatter operation S1 of 710 is propagated forward to the node 710, and then the scatter operation S1 is stacked with any scatter operations previously associated with the node 710. When the two scatter operations insert zeros along different dimensions, the demasking engine 130 can propagate the given scatter operation and then stack it adjacent to another scatter operation. For example, the scatter operation S1 can be propagated forward to insert a row of zeros into the output tensor y, and the demasking engine 130 can stack the scatter operation S1 adjacent to another scatter operation that inserts a column of zeros into the output tensor y.

[0047] In one embodiment, the demasking engine 130 propagates the scatter operation S1 forward and causes the node 710 to absorb the scatter operation. Figure 6 , assume that a scatter operation S1 inserts a column of zeros into tensor x to generate tensor X. Also assume that function f2 is a matrix multiplication operation that multiplies tensor X by the input tensor. Because the column of zeros inserted into X multiplies the corresponding row of the input tensor, the scatter operation S1 can be deleted as long as the corresponding row of the input tensor is also deleted. This approach does not change the dimensionality of the output tensor, but eliminates the need for the scatter operation S1.

[0048] In one embodiment, in addition to generating scatter operations, or in lieu of generating scatter operations, the demasking engine 130 generates a gather operation. For example, the demasking engine 130 can generate a gather operation that resides after a node and selects a subset of the node's output tensors to pass to subsequent nodes. Because the subsequent nodes only receive a subset of the output tensors, calculations involving the subset can be performed faster than equivalent calculations performed using the output tensors. The demasking engine 130 can also propagate gather operations toward the inputs of the graphical representation 200, and combine, stack, and / or absorb gather operations similar to how the demasking engine 130 combines, stacks, and / or absorbs scatter operations.

[0049] Generally speaking Figure 3-Figure 7 In various embodiments, the techniques described in conjunction with these figures may be advantageously applied to generate an optimized neural network 132. The disclosed techniques may be applied to any technically feasible neural network and / or any technically feasible graphical representation thereof. The disclosed techniques may also be applied to any portion of a neural network, including one or more layers, components, or elements, among others. The optimized neural network 132 may perform inference operations significantly faster than the masked neural network 122, while retaining the functional characteristics of the masked neural network 122. Thus, the disclosed techniques represent a significant improvement over prior art techniques that fail to provide similar improvements in inference speed.

[0050] The process of unmasking a masked neural network

[0051] Figure 8 is a flow chart of method steps for removing a mask from a neural network according to various embodiments. Figure 1-Figure 7 The method steps are described with reference to a system, but those skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the present embodiment.

[0052] As shown, method 800 begins at step 802, where the demasking engine 130 identifies a first node included in the graphical representation of the masked neural network. Figure 4 As shown, a graphical representation of a masked neural network can be configured with a set of masked tensors Figure 2 A graphical representation of 200.

[0053] At step 804, the demasking engine 130 determines a first tensor, a first mask, and a first function included in the first node. In one embodiment, the demasking engine 130 traverses the graphical representation of the neural network nodes node by node and iteratively processes each node. The demasking engine 130 may analyze the first node by parsing the program code associated with the first node to extract the first tensor, the first mask, and the first function.

[0054] At step 806, the demasking engine 130 removes the first mask from the first node. In one embodiment, the first mask zeroes out elements of the first tensor that do not contribute to the output of the first function. These elements can be considered redundant and can be safely deleted without adversely affecting the output of the first function. Figure 1 The pruning engine 120 may generate a first mask through the above pruning process.

[0055] At step 808, the demasking engine 130 replaces the first tensor with a dense version of the first tensor. As referred to herein, the term "densified" refers to a denser form of a tensor. In one embodiment, the demasking engine 130 generates a dense version of a given tensor by analyzing a mask associated with the tensor and identifying portions of the tensor that are zeroed by the mask. The demasking engine 130 can then remove these portions from the tensor to produce a smaller, denser version of the tensor. A given function evaluated based on a smaller, denser version of a tensor can be evaluated faster than a given function evaluated based on the original larger tensor.

[0056] At step 810, the demasking engine 130 adds a first scatter operation near a first node following a first function. Because the first function receives a dense version of a tensor as input, when the first function is evaluated, the first function may produce a smaller, denser output than the output produced when the first function is evaluated based on the original larger tensor. In various embodiments, the first scatter operation expands that smaller, denser output to have a dimension associated with a previous output of the first function. Thus, downstream nodes that expect inputs having a particular dimension are provided with input data having that dimension.

[0057] In step 812, the demasking engine 130 propagates the first scatter operation toward the output of the graphical representation. The demasking engine 130 may propagate the first scatter operation via one or more different techniques. In one embodiment, the demasking engine 130 removes the first scatter operation from the position after the first node and generates a second scatter operation at the position after the second node. The second node then receives a smaller, denser output of the first function as input and can therefore be evaluated faster. The second scatter operation expands the output of the second function to make it consistent with the expected dimension associated with the subsequent node. In another embodiment, the demasking engine 130 combines the first scatter operation with at least one other scatter operation when those scatter operations insert zeros along the same axis. In another embodiment, the demasking engine 130 stacks the first scatter operation with at least one other scatter operation when those scatter operations insert zeros along different axes. In yet another embodiment, the demasking engine 130 causes the subsequent node to absorb the scatter operation by modifying the input tensor processed by the subsequent node.

[0058] Overall reference Figure 1-Figure 8 , those skilled in the art will appreciate that the disclosed technology can be implemented by any technically feasible combination of computer hardware and / or software. Figure 9-11 An example computer system configured to execute neural network optimization pipeline 100 and / or optimized neural network 132 is described in further detail.

[0059] Example Hardware Architecture

[0060] Fig. 9 900 is a block diagram illustrating a computer system 900 configured to implement one or more aspects of various embodiments. In some embodiments, the computer system 900 is a server machine operating in a data center or cloud computing environment that provides scalable computing resources as a service over a network. In one embodiment, the various elements of the computer system 900 execute Figure 1 The neural network optimization pipeline 100 and / or the optimized neural network 132.

[0061] In various embodiments, computer system 900 includes, but is not limited to, a central processing unit (CPU) 902 and a system memory 904, which is coupled to a parallel processing subsystem 912 via a memory bridge 905 and a communication path 913. Memory bridge 905 is also coupled to an I / O (input / output) bridge 907 via a communication path 906, and I / O bridge 907 is in turn coupled to a switch 916.

[0062] In one embodiment, the I / O bridge 907 is configured to receive user input information from an optional input device 908 such as a keyboard or mouse, and forward the input information to the CPU 902 via the communication path 906 and the memory bridge 905 for processing. In some embodiments, the computer system 900 may be a server machine in a cloud computing environment. In such an embodiment, the computer system 900 may not have an input device 908. Instead, the computer system 900 may receive equivalent input information by receiving a command in the form of a message sent over the network and received via the network adapter 918. In one embodiment, the switch 916 is configured to provide a connection between the I / O bridge 907 and other components of the computer system 900 (e.g., the network adapter 918 and various plug-in cards 920 and 921).

[0063] In one embodiment, the I / O bridge 907 is coupled to a system disk 914, which can be configured to store content and applications and data for use by the CPU 902 and the parallel processing subsystem 912. In one embodiment, the system disk 914 provides non-volatile storage for applications and data and can include a fixed or removable hard drive, a flash memory device, and a CD-ROM (Compact Disc Read Only Memory), DVD-ROM (Digital Versatile Disc-ROM), Blu-ray, HD-DVD (High Definition DVD), or other magnetic, optical, or solid-state storage device. In various embodiments, other components such as a universal serial bus or other port connection, an optical drive, a digital versatile disk drive, a movie recording device, etc. can also be connected to the I / O bridge 907.

[0064] In various embodiments, memory bridge 905 may be a northbridge chip, and I / O bridge 907 may be a southbridge chip. Additionally, communication paths 906 and 913, as well as other communication paths within computer system 900, may be implemented using any technically suitable protocol, including but not limited to AGP (Accelerated Graphics Port), HyperTransport, or any other bus or point-to-point communication protocol known in the art.

[0065] In some embodiments, parallel processing subsystem 912 includes a graphics subsystem that transmits pixels to optional display device 910, which can be any conventional cathode ray tube, liquid crystal display, light emitting diode display, etc. In such embodiments, parallel processing subsystem 912 includes circuits optimized for graphics and video processing, including, for example, video output circuits. Figure 8 and Fig. 9Described in more detail, such circuitry may be incorporated across one or more parallel processing units (PPUs), also referred to herein as parallel processors, included within the parallel processing subsystem 912. In other embodiments, the parallel processing subsystem 912 includes circuitry optimized for general and / or computational processing. Again, such circuitry may be incorporated across one or more PPUs included within the parallel processing subsystem 912, which are configured to perform such general and / or computational operations. In other embodiments, one or more PPUs included within the parallel processing subsystem 912 may be configured to perform graphics processing, general processing, and computational processing operations. The system memory 904 includes at least one device driver configured to manage processing operations of one or more PPUs within the parallel processing subsystem 912.

[0066] In various embodiments, the parallel processing subsystem 912 may be coupled to Fig. 9 One or more other elements of the CPU 902 may be integrated together to form a single system. For example, the parallel processing subsystem 912 may be integrated with the CPU 902 and other connection circuits on a single chip to form a system on a chip (SoC).

[0067] In one embodiment, CPU 902 is the main processor of computer system 900, which controls and coordinates the operation of other system components. In one embodiment, CPU 902 issues commands that control the operation of the PPU. In some embodiments, communication path 913 is a serial bus (PCI Express) link, as is known in the art, in which a dedicated channel is allocated to each PPU. Other communication paths may also be used. The PPU advantageously implements a highly parallel processing architecture. The PPU can be equipped with any number of local parallel processing memories (PP memories).

[0068] It will be appreciated that the systems shown herein are illustrative and that variations and modifications are possible. The connection topology, including the number and arrangement of bridges, the number of CPUs 902, and the number of parallel processing subsystems 912, may be modified as desired. For example, in some embodiments, system memory 904 may be directly connected to CPU 902 rather than through memory bridge 905, and other devices may communicate with system memory 904 via memory bridge 905 and CPU 902. In other embodiments, parallel processing subsystems 912 may be connected to I / O bridge 907 or directly to CPU 902 rather than to memory bridge 905. In other embodiments, I / O bridge 907 and memory bridge 905 may be integrated into a single chip rather than existing as one or more discrete devices. Finally, in some embodiments, there may be no parallel processing subsystems 912. Fig. 9For example, the switch 916 may be omitted, and the network adapter 918 and the add-in cards 920, 921 may be directly connected to the I / O bridge 907.

[0069] Fig.10 According to various embodiments, the Fig. 9 1002 in the parallel processing subsystem 912 of FIG. Fig.10 One PPU 1002 is depicted, but as described above, the parallel processing subsystem 912 may include any number of PPUs 1002. As shown, the PPU 1002 is coupled to a local parallel processing (PP) memory 1004. The PPU 1002 and the PP memory 1004 may be implemented using one or more integrated circuit devices (e.g., a programmable processor, an application specific integrated circuit (ASIC), or a memory device) or in any other technically feasible manner.

[0070] In some embodiments, the PPU 1002 includes a graphics processing unit (GPU), which can be configured to implement a graphics rendering pipeline to perform various operations related to generating pixel data based on graphics data provided by the CPU 902 and / or the system memory 904. When processing graphics data, the PP memory 1004 can be used as a graphics memory, which stores one or more conventional frame buffers and, if necessary, one or more other rendering targets. Among them, the PP memory 1004 can be used to store and update pixel data, and transmit the final pixel data or display frame to the optional display device 910 for display. In some embodiments, the PPU 1002 can also be configured for general processing operations and computing operations. In some embodiments, the computer system 900 can be a server machine in a cloud computing environment. In such an embodiment, the computer system 900 may not have a display device 910. Instead, the computer system 900 can generate equivalent output information by sending a command in the form of a message over the network via the network adapter 918.

[0071] In some embodiments, CPU 902 is the main processor of computer system 900, which controls and coordinates the operation of other system components. In one embodiment, CPU 902 issues commands that control the operation of PPU 1002. In some embodiments, CPU 902 writes a command stream for PPU 1002 into a data structure (in Fig. 9 or Fig.101002), which may be located in system memory 904, PP memory 1004, or another storage location accessible to both CPU 902 and PPU 1002. A pointer to the data structure is written to a command queue, also referred to herein as a push buffer, to initiate processing of the command data stream in the data structure. In one embodiment, PPU 1002 reads the command stream from the command queue and then executes the commands asynchronously with respect to the operation of CPU 902. In embodiments where multiple push buffers are generated, the application may specify an execution priority for each push buffer through a device driver to control the scheduling of different push buffers.

[0072] In one embodiment, the PPU 1002 includes an I / O (input / output) unit 1005 that communicates with the rest of the computer system 900 via the communication path 913 and the memory bridge 905. In one embodiment, the I / O unit 1005 generates packets (or other signals) for transmission on the communication path 913, and also receives all incoming packets (or other signals) from the communication path 913, directing the incoming packets to the appropriate components of the PPU 1002. For example, commands related to processing tasks may be directed to the host interface 1006, while commands related to memory operations (e.g., reads from or writes to the PP memory 1004) may be directed to the crossbar unit 1010. In one embodiment, the host interface 1006 reads each command queue and sends the command stream stored in the command queue to the front end 1012.

[0073] As above combined Fig. 9 As described above, the connection of the PPU 1002 to the rest of the computer system 900 may vary. In some embodiments, the parallel processing subsystem 912 including at least one PPU 1002 is implemented as an add-in card that can be inserted into an expansion slot of the computer system 900. In other embodiments, the PPU 1002 may be integrated on a single chip with a bus bridge (e.g., a memory bridge 905 or an I / O bridge 907). Again, in other embodiments, some or all of the elements of the PPU 1002 may be included with the CPU 902 in a single integrated circuit or system on a chip (SoC).

[0074] In one embodiment, the front end 1012 sends the processing tasks received from the host interface 1006 to the work distribution unit (not shown) within the task / work unit 1007. In one embodiment, the work distribution unit receives a pointer to the encoded processing task as task metadata (TMD) and stores it in memory. The pointer to the TMD is contained in a command stream, which is stored as a command queue and received by the front end unit 1012 from the host interface 1006. The processing task that can be encoded as a TMD includes an index associated with the data to be processed and a state parameter and command that defines how to process the data. For example, the state parameter and command can define a program to be executed on the data. Also, for example, the TMD can specify the number and configuration of a set of CTAs. Typically, each TMD corresponds to a task. The task / work unit 1007 receives tasks from the front end 1012 and ensures that the GPC 1008 is configured to a valid state before starting the processing task specified by each TMD. A priority can be specified for each TMD used to schedule the execution of the processing task. Processing tasks can also be received from the processing cluster array 1030. Optionally, the TMD may include a parameter that controls whether the TMD is added to the head or tail of a list of processing tasks (or a list of pointers to processing tasks), thereby providing another level of control over execution priority.

[0075] In one embodiment, PPU 1002 implements a highly parallel processing architecture based on a processing cluster array 1030, which includes a set of C general processing clusters (GPCs) 1008, where C ≥ 1. Each GPC 1008 is capable of executing a large number of threads (e.g., hundreds or thousands) in parallel, where each thread is an instance of a program. In various applications, different GPCs 1008 can be assigned to process different types of programs or perform different types of calculations. The allocation of GPCs 1008 can vary depending on the workload generated by each program or calculation type.

[0076] In one embodiment, the memory interface 1014 includes a set of D partition units 1015, where D ≥ 1. Each partition unit 1015 is coupled to one or more dynamic random access memories (DRAMs) 1020 residing within the PPM memory 1004. In some embodiments, the number of partition units 1015 is equal to the number of DRAMs 1020, and each partition unit 1015 is coupled to a different DRAM 1020. In other embodiments, the number of partition units 1015 may be different from the number of DRAMs 1020. It will be appreciated by those of ordinary skill in the art that the DRAMs 1020 may be replaced with any other technically suitable storage device. In operation, various rendering targets, such as texture maps and frame buffers, may be stored on the DRAMs 1020, thereby allowing the partition units 1015 to write portions of each rendering target in parallel to efficiently use the available bandwidth of the PP memory 1004.

[0077] In one embodiment, a given GPC 1008 may process data to be written to any DRAM 1020 within PP memory 1004. In one embodiment, crossbar unit 1010 is configured to route the output of each GPC 1008 to the input of any partition unit 1015 or to any other GPC 1008 for further processing. GPCs 1008 communicate with memory interface 1014 via crossbar unit 1010 to read from or write to various DRAMs 1020. In some embodiments, crossbar unit 1010 has connections to I / O unit 1005 in addition to connections to PP memory 1004 via memory interface 1014, thereby enabling processing cores within different GPCs 1008 to communicate with system memory 904 or other memory that is not local to PPU 1002. In Fig.10 In the embodiment of the present invention, the crossbar unit 1010 is directly connected to the I / O unit 1005. In various embodiments, the crossbar unit 1010 can use virtual channels to separate the traffic flow between the GPC 1008 and the partition unit 1015.

[0078] In one embodiment, the GPCs 1008 may be programmed to perform processing tasks associated with a wide variety of applications, including, but not limited to, linear and nonlinear data transformations, filtering of video and / or audio data, modeling operations (e.g., applying physical laws to determine the position, velocity, and other properties of an object), image rendering operations (e.g., tessellation shaders, vertex shaders, geometry shaders, and / or pixel / fragment shader programs), general computing operations, etc. In operation, the PPU 1002 is configured to transfer data from the system memory 904 and / or the PP memory 1004 to one or more on-chip memory units, process the data, and write the resulting data back to the system memory 904 and / or the PP memory 1004. The resulting data may then be accessed by other system components (including the CPU 902, another PPU 1002 in the parallel processing subsystem 912, or another parallel processing subsystem 912 in the computer system 900).

[0079] In one embodiment, any number of PPUs 1002 may be included in the parallel processing subsystem 912. For example, multiple PPUs 1002 may be provided on a single plug-in card, or multiple plug-in cards may be connected to the communication path 913, or one or more PPUs 1002 may be integrated into a bridge chip. The PPUs 1002 in a multi-PPU system may be the same or different from each other. For example, different PPUs 1002 may have different numbers of processing cores and / or different numbers of PP memories 1004. In embodiments where there are multiple PPUs 1002, those PPUs may run in parallel to process data at a higher throughput (than is possible using a single PPU 1002). Systems containing one or more PPUs 1002 may be implemented in a variety of configurations and forms, including but not limited to desktops, laptops, handheld personal computers or other handheld devices, servers, workstations, game consoles, embedded systems, and the like.

[0080] Fig.11 According to various embodiments, the Fig.10 1 is a block diagram of a general processing cluster (GPC) 1008 in a parallel processing unit (PPU) 1002 of FIG. As shown, the GPC 1008 includes, but is not limited to, a pipeline manager 1105, one or more texture units 1115, a pre-raster (preROP) unit 1125, a work distribution crossbar 1130, and an L1.5 cache 1135.

[0081] In one embodiment, GPC 1008 can be configured to execute a large number of threads in parallel to perform graphics, general processing and / or computing operations. As used herein, "thread" refers to an instance of a specific program executed for a specific set of input data. In some embodiments, single instruction multiple data (SIMD) instruction issuance technology is used to support the parallel execution of a large number of threads without providing multiple independent instruction units. In other embodiments, a public instruction unit configured to issue instructions to a group of processing engines in GPC 1008 is used, and single instruction multiple thread (SIMT) technology is used to support the parallel execution of a large number of generally synchronized threads. Different from the SIMD execution mechanism in which all processing engines usually execute the same instruction, SIMT execution allows different threads to more easily follow different execution paths through a given program. It will be understood by those of ordinary skill in the art that the SIMD processing mechanism represents a functional subset of the SIMT processing mechanism.

[0082] In one embodiment, the operation of GPC 1008 is controlled via pipeline manager 1105, which distributes processing tasks received from a work distribution unit (not shown) within task / work unit 1007 to one or more streaming multiprocessors (SMs) 1110. Pipeline manager 1105 may also be configured to control work distribution crossbar 1130 by specifying the destination of processed data output by SM 1110.

[0083] In various embodiments, the GPC 1008 includes a set of M SMs 1110, where M ≥ 1. Furthermore, each SM 1110 includes a set of function execution units (not shown), such as execution units and load-store units. Processing operations specific to any function execution unit can be pipelined, which enables a new instruction to be issued for execution before the previous instruction has completed execution. Any combination of function execution units within a given SM 1110 can be provided. In various embodiments, the function execution units can be configured to support a variety of different operations, including integer and floating point arithmetic (e.g., addition and multiplication), comparison operations, Boolean operations (AND, OR, 5OR), shifts, and operations of various algebraic functions (e.g., plane interpolation and trigonometric functions, exponential functions and logarithmic functions, etc.). Advantageously, the same function execution unit can be configured to perform different operations.

[0084] In various embodiments, each SM 1110 includes multiple processing cores. In one embodiment, SM 1110 includes a large number (e.g., 128, etc.) of different processing cores. Each core may include a fully pipelined, single-precision, double-precision, and / or mixed-precision processing unit, including a floating-point arithmetic logic unit and an integer arithmetic logic unit. In one embodiment, the floating-point arithmetic logic unit implements the IEEE 754-2008 standard for floating-point arithmetic. In one embodiment, the core includes 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0085] In one embodiment, the tensor cores are configured to perform matrix operations, and in one embodiment, one or more tensor cores are included in these cores. In particular, the tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations for neural network training and inference. In one embodiment, each tensor core operates on 4×4 matrices and performs matrix multiplication and accumulation operations D=A×B+C, where A, B, C, and D are 4×4 matrices.

[0086] In one embodiment, the matrix multiplication inputs A and B are 16-bit floating-point matrices, while the accumulation matrices C and D can be either 16-bit floating-point or 32-bit floating-point matrices. The tensor cores operate on 16-bit floating-point input data using 32-bit floating-point accumulation. The 16-bit floating-point multiplication requires 64 operations and produces a full-precision product, which is then accumulated with other intermediate products using 32-bit floating-point addition to perform a 4×4×4 matrix multiplication. In practice, the tensor cores are used to perform larger two-dimensional or higher-dimensional matrix operations that are constructed from these smaller elements. APIs such as the CUDA9 C++ API expose specialized matrix loads, matrix multiplications and accumulations, and matrix store operations to efficiently use tensor cores from CUDA-C++ programs. At the CUDA level, the warp-level interface assumes that a 16×16-sized matrix spans all 32 threads of the warp.

[0087] Neural networks rely heavily on matrix math operations, and complex multi-layer networks require a large amount of floating point performance and bandwidth for efficiency and speed. In various embodiments, the SM 1110 has thousands of processing cores, optimized for matrix math operations, and provides tens to hundreds of TFLOPS of performance, providing a computing platform capable of delivering the performance required for deep neural network-based artificial intelligence and machine learning applications.

[0088] In various embodiments, each SM 1110 may also include a plurality of special function units (SFUs) that perform special functions (e.g., attribute evaluation, reciprocal square root, etc.). In one embodiment, the SFU may include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, the SFU may include a texture unit configured to perform texture map filtering operations. In one embodiment, the texture unit is configured to load a texture map (e.g., a 2D array of texture pixels) from a memory and sample the texture map to generate sampled texture values ​​for use in a shader program executed by the SM. In various embodiments, each SM 1110 also includes a plurality of load / store units (LSUs) that implement load and store operations between a shared memory / L1 cache and a register file internal to the SM 1110.

[0089] In one embodiment, each SM 1110 is configured to process one or more thread groups. As used herein, a "thread group" or "thread bundle" refers to a group of threads that execute the same program simultaneously on different input data, and a thread in the group is assigned to a different execution unit within the SM 1110. A thread group may include fewer threads than the number of execution units within the SM 1110, in which case some executions may be idle during the cycle processing the thread group. A thread group may also include more threads than the number of execution units within the SM 1110, in which case processing can be performed in consecutive clock cycles. Since each SM 1110 can support up to G thread groups simultaneously, up to G*M thread groups can be executed in the GPC 1008 at any given time.

[0090] Additionally, in one embodiment, multiple related thread groups may be simultaneously active (at different stages of execution) in an SM 1110. Such a collection of thread groups is referred to herein as a "cooperative thread array" ("CTA") or "thread array". The size of a particular CTA is equal to m*k, where k is the number of threads executing simultaneously in the thread group, typically an integer multiple of the number of execution units within the SM 1110, and m is the number of simultaneously active thread groups within the SM 1110. In some embodiments, a single SM 1110 may simultaneously support multiple CTAs, where such CTAs are at the granularity of assigning work to the SM 1110.

[0091] In one embodiment, each SM 1110 contains a level 1 (L1) cache, or uses space in a corresponding L1 cache external to the SM 1110 to support (or otherwise) load and store operations performed by the execution units. Each SM 1110 may also access a level 2 (L2) cache (not shown) that is shared among all GPCs 1008 in the PPU 1002. The L2 cache may be used to transfer data between threads. Finally, the SM 1110 may also access off-chip "global" memory, which may include PP memory 1004 and / or system memory 904. It should be understood that any memory external to the PPU 1002 may be used as global memory. In addition, as Fig.11 As shown, a level 1.5 (L1.5) cache 1135 may be included in GPC 1008 and is configured to receive and store data requested by SM 1110 from memory via memory interface 1014. Such data may include, but is not limited to, instructions, uniform data, and constant data. In embodiments having multiple SMs 1110 within GPC 1008, multiple SMs 1110 may beneficially share common instructions and data cached in L1.5 cache 1135.

[0092] In one embodiment, each GPC 1008 may have an associated memory management unit (MMU) 1120 that is configured to map virtual addresses to physical addresses. In various embodiments, the MMU 1120 may reside within the GPC 1008 or within the memory interface 1014. The MMU 1120 includes a set of page table entries (PTEs) that are used to map virtual addresses to physical addresses of tiles or memory pages, and optionally to cache line indices. The MMU 1120 may include an address translation lookaside buffer (TLB) or cache that may reside within the SM 1110, within one or more L1 caches, or within the GPC 1008.

[0093] In one embodiment, in graphics and compute applications, GPC 1008 may be configured such that each SM 1110 is coupled to a texture unit 1115 for performing texture mapping operations, such as determining texture sample locations, reading texture data, and filtering texture data.

[0094] In one embodiment, each SM 1110 sends the processed tasks to the work distribution crossbar 1130 so that the processed tasks are provided to another GPC 1008 for further processing, or the processed tasks are stored in an L2 cache (not shown), parallel processing memory 1004, or system memory 904 via the crossbar unit 1010. In addition, a pre-raster operation (preROP) unit 1125 is configured to receive data from the SM 1110, direct the data to one or more raster operation (ROP) units within the partition unit 1015, perform color blending optimization, organize pixel color data, and perform address translation.

[0095] It will be appreciated that the architecture described herein is illustrative and that variations and modifications are possible. Among other things, any number of processing units may be included within GPC 1008, such as SM 1110, texture unit 1115, or preROP unit 1125. In addition, as described above in conjunction with Figure 6 As described, a PPU 1002 may include any number of GPCs 1008 configured to be functionally similar to one another, such that execution behavior does not depend on which GPC 1008 receives a particular processing task. Furthermore, each GPC 1008 operates independently of the other GPCs 1008 in the PPU 1002 to execute tasks for one or more application programs.

[0096] In summary, the demasking engine removes masks from a pruned neural network represented by a node graph. The demasking engine analyzes a tensor and a mask associated with a given node in the node graph to determine portions of the tensor that are zeroed by the mask. The demasking engine then removes these portions from the tensor to generate a dense tensor with a smaller dimension than the original tensor. Functions associated with the node can be evaluated faster based on the dense tensor than on the original tensor. The demasking engine adds a scatter operation after the node to scale the dimensionality of the dense tensor to the dimensionality associated with the original tensor.

[0097] At least one technical advantage of the techniques described herein is that the optimized neural network performs inference operations faster than the original pruned neural network. Therefore, the optimized neural network is well suited for use in real-time applications (e.g., autonomous vehicles). Another advantage of the techniques described herein is that the optimized neural network can have a smaller memory footprint than the pruned neural network, thereby saving memory resources. These technical advantages represent multiple technical advances over prior art methods.

[0098] 1. Some embodiments include a computer-implemented method comprising: causing an unmasked output of a first neural network portion to be generated based at least in part on a masked output of the first neural network portion, wherein the unmasked output has a dimension less than a dimension of the masked output, causing the unmasked output to replace the masked output, causing a scatter operation to be performed to expand the dimension of the unmasked output to a dimension corresponding to the masked output.

[0099] 2. A computer-implemented method according to clause 1, wherein the unmasked output is associated with a first tensor and the masked output is associated with a second tensor.

[0100] 3. A computer-implemented method according to any of clauses 1-2, wherein generating the unmasked output includes: determining a first portion of the first tensor, the first portion of the first tensor corresponding to one or more zeros included in a first mask, wherein the masked output is derived based on the first tensor and the first mask, generating the second tensor based on the first portion of the first tensor, and evaluating a first function based on the second tensor to generate the unmasked output.

[0101] 4. A computer-implemented method according to any of clauses 1-3, wherein the first mask clears the first portion of the first tensor to zero, and wherein the first function is evaluated based on the first tensor to produce a first result that is independent of the first portion of the first tensor.

[0102] 5. A computer-implemented method according to any of clauses 1-4, wherein the second tensor comprises only a second portion of the first tensor.

[0103] 6. A computer-implemented method as described in any of clauses 1-5, wherein a processor evaluates the first function based on the second tensor faster than a processor evaluates the first function based on the first tensor.

[0104] 7. A computer-implemented method according to any of clauses 1-6, wherein causing the unmasked output to replace the masked output comprises: replacing the first tensor with the second tensor, wherein the dimension of the second tensor is smaller than the dimension of the first tensor.

[0105] 8. The computer-implemented method of any of clauses 1-7, wherein causing performance of the scatter operation comprises inserting one or more zeros into the unmasked output.

[0106] 9. The computer-implemented method of any of clauses 1-8, further comprising: combining the scatter operation with one or more additional scatter operations associated with one or more neural network layers.

[0107] 10. The computer-implemented method of any one of clauses 1-9, further comprising: absorbing the decentralized operation into a second neural network portion located after the first neural network portion.

[0108] 11. Some embodiments include a non-transitory computer-readable medium storing program instructions that, when executed by at least one processor, cause the at least one processor to at least: cause an unmasked output of the first neural network layer to be generated based at least in part on a masked output of the first neural network layer, wherein a dimensionality of the unmasked output is different from a dimensionality of the masked output, cause the unmasked output to replace the masked output, and cause a first operation to be performed to scale the dimensionality of the unmasked output to a dimensionality corresponding to the masked output.

[0109] 12. The non-transitory computer-readable medium of clause 11, wherein the first operation comprises a first scatter operation performed to expand a dimensionality of the unmasked output to a dimensionality corresponding to the masked output.

[0110] 13. The non-transitory computer-readable medium of any of clauses 11-12, further comprising merging the first scatter operation with a second scatter operation associated with a second neural network layer that is located after the first neural network layer in a sequence of neural network layers.

[0111] 14. The non-transitory computer-readable medium of any of clauses 11-13, wherein the first operation comprises a first gather operation performed to reduce a dimensionality of the unmasked output to a dimensionality corresponding to the masked output.

[0112] 15. The non-transitory computer-readable medium of any of clauses 11-14, further comprising: merging the first aggregation operation with a second aggregation operation associated with a second neural network layer, the second neural network layer being located before the first neural network layer in a sequence of neural network layers.

[0113] 16. A non-transitory computer-readable medium according to any of clauses 11-15, wherein generating the unmasked output includes: determining a first portion of a first tensor, the first portion of the first tensor corresponding to one or more zeros included in a first mask, wherein the masked output is derived based on the first tensor and the first mask, generating a second tensor based on the first portion of the first tensor, and evaluating a first function based on the second tensor to generate the unmasked output.

[0114] 17. A non-transitory computer-readable medium according to any one of clauses 11-16, wherein the second tensor includes only the second portion of the first tensor and does not include the first portion of the first tensor, and wherein the first function is evaluated faster based on the second tensor than the first function is evaluated based on the first tensor.

[0115] 18. Some embodiments include a system comprising: a memory storing one or more instructions, and a processor executing the instructions to at least: cause an unmasked output of the first neural network layer to be generated based at least in part on a masked output of the first neural network layer, wherein the dimension of the unmasked output is less than the dimension of the masked output, cause the unmasked output to replace the masked output, and cause a scatter operation to be performed to expand the dimension of the unmasked output to a dimension corresponding to the masked output.

[0116] 19. The system of clause 18, wherein the processor further executes the instructions to: combine the scatter operation with one or more scatter operations, wherein the one or more scatter operations include at least one dimension aligned with a corresponding dimension associated with the scatter operation.

[0117] 20. The system of any of clauses 18-19, wherein the processor further executes the instructions to: stack the scatter operation adjacent to one or more scatter operations, wherein the one or more scatter operations include at least one dimension that is not aligned with a corresponding dimension associated with the scatter operation.

[0118] Any and all combinations of any claim elements recited in any claim and / or any elements described in this application, in any manner, are within the intended scope of the disclosure and protection.

[0119] Descriptions of various embodiments have been given for purposes of illustration, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0120] Aspects of the present embodiment may be embodied as a system, method, or computer program product. Therefore, aspects of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.), or an embodiment that combines software and hardware aspects, which may generally all be referred to herein as a "module" or "system." In addition, aspects of the present disclosure may take the form of a computer program product embodying one or more computer-readable media (having computer-readable program code embodied thereon).

[0121] Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer-readable storage media will include the following: an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this article, a computer-readable storage medium may be any tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, a device, or an apparatus.

[0122] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It will be understood that each frame of the flowchart illustration and / or block diagram and the combination of frames in the flowchart illustration and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine. When the processor of a computer or other programmable data processing device executes instructions, these instructions enable the function / action specified in the flowchart and / or block diagram box to be implemented. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a processor specific to an application or a field programmable gate array.

[0123] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of the architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, segment or portion of a code, which includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative embodiments, the functions indicated in the box may not occur in the order indicated in the figure. For example, depending on the functions involved, two boxes shown in succession may actually be executed substantially simultaneously, or sometimes these boxes may be executed in reverse order. It should also be noted that each box illustrated in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart may be implemented by a dedicated hardware-based system that performs a specified function or action, or by a combination of dedicated hardware and computer instructions.

[0124] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof, and the scope of the disclosure is determined by the claims that follow.

Claims

1. A processor, comprising: One or more circuits for using one or more neural networks to generate one or more first tensors based at least in part on one or more second tensors, the one or more second tensors having a greater sparsity and size than the one or more first tensors.

2. A computer-implemented method comprising: One or more neural networks are used, by one or more processors, to generate one or more first tensors based at least in part on one or more second tensors, the one or more second tensors having a greater sparsity and size than the one or more first tensors.

3. The computer-implemented method of claim 2, further comprising the steps of: causing an unmasked output of the first neural network portion to be generated based at least in part on a masked input used to produce a masked output of the first neural network portion, wherein the dimensionality of the unmasked output is less than the dimensionality of the masked output; causing the unmasked output to replace the masked output; A scatter operation is performed to expand the dimension of the unmasked output to a dimension corresponding to the masked output.

4. The computer-implemented method of claim 3, wherein the unmasked output is associated with a second tensor and the masked output is associated with a first tensor.

5. The computer-implemented method of claim 4, wherein causing generation of the unmasked output comprises: determining a first portion of the first tensor, the first portion of the first tensor corresponding to one or more zeros included in a first mask, wherein the first mask zeroes elements of a masked input, wherein the masked output is derived based on the first tensor and the first mask; generating the second tensor based on the first portion of the first tensor; as well as A first function is evaluated based on the second tensor to generate the unmasked output.

6. The computer-implemented method of claim 5, wherein the first mask zeroes the first portion of the first tensor, and wherein the first function is evaluated based on the first tensor to produce a first result that is independent of the first portion of the first tensor.

7. A computer-implemented method according to claim 5 or 6, wherein the second tensor includes only a second portion of the first tensor, and wherein the one or more processors evaluate the first function based on the second tensor faster than the one or more processors evaluate the first function based on the first tensor.

8. The computer-implemented method of claim 5 or 6, wherein: Causing the unmasked output to replace the masked output includes replacing the first tensor with the second tensor, wherein the dimension of the second tensor is smaller than the dimension of the first tensor.

9. The computer-implemented method of claim 5 or 6, wherein causing the scatter operation to be performed comprises: One or more zeros are inserted into the unmasked output.

10. The computer-implemented method of claim 5 or 6, further comprising: The scatter operation is combined with one or more additional scatter operations associated with one or more neural network layers.

11. The computer-implemented method of claim 5 or 6, further comprising: The distributed operation is absorbed into a second neural network part located after the first neural network part.

12. A non-transitory computer readable medium storing program instructions which, when executed by at least one processor, cause the at least one processor to at least: The one or more neural networks are caused to generate one or more first tensors based at least in part on one or more second tensors, the one or more second tensors having a greater sparsity and size than the one or more first tensors.

13. The non-transitory computer readable medium of claim 12, the instructions further causing the processor to: causing an unmasked output of the first neural network layer to be generated based at least in part on a masked input used to produce a masked output of the first neural network layer, wherein the dimensionality of the unmasked output is different from the dimensionality of the masked output; causing the unmasked output to replace the masked output; A first operation is performed to scale the dimension of the unmasked output to a dimension corresponding to the masked output.

14. The non-transitory computer-readable medium of claim 13, wherein the first operation comprises: a first scatter operation performed to expand the dimensionality of the unmasked output to the dimensionality corresponding to the masked output, or A first clustering operation is performed to reduce the dimensionality of the masked output to the dimensionality corresponding to the unmasked output.

15. The non-transitory computer readable medium of claim 14, further comprising: merging the first scatter operation with a second scatter operation associated with a second neural network layer that follows the first neural network layer in the sequence of neural network layers, or The first aggregation operation is merged with a second aggregation operation associated with a second neural network layer that precedes the first neural network layer in the sequence of neural network layers.

16. The non-transitory computer-readable medium of any one of claims 13 to 15, wherein causing the generation of the unmasked output comprises: determining a first portion of a first tensor, the first portion of the first tensor corresponding to one or more zeros included in a first mask, wherein the masked output is derived based on the first tensor and the first mask; generating a second tensor based on the first portion of the first tensor; as well as A first function is evaluated based on the second tensor to generate the unmasked output.

17. The non-transitory computer-readable medium of claim 16, wherein the second tensor includes only the second portion of the first tensor and does not include the first portion of the first tensor, and wherein evaluating the first function based on the second tensor is faster than evaluating the first function based on the first tensor.

18. A system comprising: a memory storing one or more instructions; as well as a processor that executes the instructions to at least: The one or more neural networks are caused to generate one or more first tensors based at least in part on one or more second tensors, the one or more second tensors having a greater sparsity and size than the one or more first tensors.

19. The system of claim 18, wherein the processor executes the instructions to further: Causing one or more neural networks to generate one or more first tensors based at least in part on one or more second tensors, the one or more second tensors having a greater sparsity and size than the one or more first tensors, comprising the steps of: causing an unmasked output of the first neural network layer to be generated based at least in part on a masked input used to produce a masked output of the first neural network layer, wherein the dimensionality of the unmasked output is less than the dimensionality of the masked output, causing the unmasked output to replace the masked output, and A scatter operation is performed to expand the dimension of the unmasked output to a dimension corresponding to the masked output.

20. The system of claim 19, wherein the processor further executes the instructions to: combining the scatter operation with one or more scatter operations, wherein the one or more scatter operations include at least one dimension that is aligned with a corresponding dimension associated with the scatter operation, or The scatter operation is stacked adjacent to one or more scatter operations, wherein the one or more scatter operations include at least one dimension that is not aligned with a corresponding dimension associated with the scatter operation.

Images