Efficient data encoding for deep neural network training

Abstract

Functions are added to a deep neural network ("DNN") computation graph to encode data structures during a forward training pass of the DNN and to decode previously encoded data structures during a backward training pass of the DNN. The functions added to the DNN computation graph can be selected based on a particular pair of layers specified in the DNN computation graph. Once the modified DNN computation graph is generated, the modified DNN computation graph can be used to train the DNN. The functions added to the modified DNN computation graph can reduce memory utilization during training of the DNN.

Description

Background Art

[0001] The availability of powerful computing resources has enabled the emergence of new deep neural networks ("DNNs") to solve previously intractable problems such as image classification, translation, and speech processing. These DNNs are trained by repeatedly iterating over a dataset.

[0002] The widely used DNN training process has large computational and memory requirements and therefore typically uses a graphics processing unit ("GPU") as its primary computing platform. However, as DNNs have become larger and deeper, the size of available GPU main memory has become a serious bottleneck. This limits the size of DNNs that can be trained, and therefore restricts DNNs from solving even more complex problems.

[0003] It is with respect to these and other technical challenges that the disclosure herein is presented. Summary of the Invention

[0004] Disclosed herein are techniques for efficient data encoding for deep neural network training. Among other technical advantages, the disclosed techniques can reduce DNN memory utilization during training with minimal impact on performance. By reducing the DNN's memory footprint during training, the disclosed techniques enable larger amounts of training data to be stored in memory for training very deep networks. Other technical benefits can be realized through implementation of the disclosed techniques.

[0005] To provide the above and potentially other technical advantages, a schedule builder performs analysis on a DNN computation graph of a DNN to identify and select data structures to use during training of the DNN. The selection of data structures can be based on layer pairs specified in the DNN computation graph.

[0006] Based on the analysis, the schedule builder creates a modified DNN computation graph by adding nodes to the original DNN computation graph. The newly added nodes can define functions for encoding data structures during the forward training pass of the DNN. The new nodes can also define decoding functions for decoding previously encoded data structures during the backward training pass of the DNN. The functions added to the DNN computation graph can be selected based on the specific layer pairs specified in the DNN computation graph.

[0007] Once the modified DNN computation graph is generated, the modified DNN computation graph can be used to train the DNN. Including the function in the modified DNN computation graph can reduce memory utilization during training of the DNN.

[0008] When a DNN computation graph defines a DNN that includes a layer pair having a rectified linear unit ("ReLU") layer and a pooling layer, a node defining an encoding function can be added to the DNN computation graph to generate a positive value map ("PVM") during the forward training pass of the DNN. The PVM indicates whether the value in the input feature map to the ReLU layer of the DNN is positive. The ReLU layer can be modified to operate directly on the PVM during the backward training pass of the DNN.

[0009] When a DNN computation graph defines a layer pair including a ReLU layer and a pooling layer, a node for generating a data structure (which may be referred to herein as a "Y2 to X2 map") during the forward training pass of the DNN can be added to the DNN computation graph. The Y2 to X2 map defines the mapping between the input feature map to the pooling layer and the output feature map generated by the pooling layer. The max pooling layer can be modified to operate directly on the Y2 to X2 map during the reverse training pass of the DNN.

[0010] When a DNN computation graph defines a DNN that includes a layer pair of ReLU layers and convolutional layers, a node is added to the DNN computation graph for generating a memory-efficient sparse representation of the output feature map generated by the ReLU layer during the forward training pass of the DNN. The output feature map of the ReLU layer is also the input feature map to the convolutional layer. A node is also added to the DNN computation graph for performing backward pass computations using the output and input feature maps in their original dense format, thereby retaining the performance benefits of highly optimized dense computations while leveraging sparsity to achieve a significant reduction in memory usage.

[0011] When a DNN computation graph defines a DNN that includes a layer pair with a max pooling layer and a convolutional layer, a node is added to the DNN computation graph for generating a memory-efficient sparse representation of the output feature map generated by the max pooling layer during the forward training pass of the DNN. The output feature map of the max pooling layer is also the input feature map to the convolutional layer. A node is also added to the DNN computation graph for performing a backward pass computation using the output and input feature maps in their original dense format, thereby retaining the performance benefits of highly optimized dense computation while leveraging sparsity to achieve a significant reduction in memory usage.

[0012] In some configurations, nodes may be added to the DNN computation graph to reduce the precision of feature maps when they are no longer needed during the forward training pass of the DNN. For example, and not limitation, the precision of the input feature maps and the output feature maps may be reduced when they are no longer needed during the forward training pass. The reduced precision feature maps may be used during the reverse training pass instead of the original feature maps. This enables the memory used to store the original feature maps to be freed after the feature maps have been utilized during the forward pass, thereby saving memory. This mechanism may be referred to herein as delayed precision reduction ("DPR").

[0013] It should be understood that the above subject matter can be implemented as a computer-controlled apparatus, a computer-implemented method, a computing device, or an article of manufacture such as a computer-readable medium. These and various other features will become clear by reading the following "Detailed Description" and examining the associated drawings.

[0014] This Summary is provided to present a brief description of some aspects of the disclosed technology in a simplified form, which is further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a computational architecture diagram illustrating aspects of the configuration of one system disclosed herein for efficient data encoding for deep neural network training;

[0016] Figure 2 is a timing diagram illustrating how implementations of the disclosed technology can be used to reduce memory utilization during DNN training;

[0017] Figures 3A-3E is a DNN layer diagram illustrating aspects of several mechanisms disclosed herein for reducing memory utilization during DNN training;

[0018] Figure 4 is a flowchart showing a routine illustrating aspects of an illustrative computer-implemented process for efficient data encoding for deep neural network training;

[0019] Figure 5 is a computer architecture diagram showing an illustrative computer hardware and software architecture for a computing device that may implement aspects of the techniques presented herein; and

[0020] Figure 6is a network diagram illustrating a distributed computing environment in which aspects of the disclosed technology may be implemented. DETAILED DESCRIPTION

[0021] The following detailed description relates to techniques for efficient data encoding for deep neural network training. Among other technical advantages, the disclosed techniques can reduce memory utilization during DNN training with minimal impact on performance. By reducing the memory footprint of DNNs during training, the disclosed techniques enable large amounts of training data to be stored in memory for training very deep networks. Other technical benefits not specifically identified herein may also be realized through implementation of the disclosed techniques.

[0022] Before describing the disclosed techniques for efficient data encoding for deep neural network training, a brief overview of DNNs and DNN training will be provided. DNNs are typically composed of a series of different types of layers (e.g., convolutional, ReLU, fully connected, and pooling layers). DNNs are typically trained using labeled datasets (e.g., a set of images labeled with data describing the content in the images). DNN training typically utilizes GPUs as the computing platform.

[0023] The DNN is trained over multiple epochs. During each epoch, the DNN is trained on all the training data in the training dataset in multiple steps. In each step, the DNN first makes predictions on a subset of the training data (referred to herein as a "mini-batch" or "batch"). Training on mini-batches has been shown to achieve better accuracy and better hardware utilization, as opposed to training on individual instances of the training data (e.g., individual images). This step is often referred to as the "forward pass" (also referred to herein as the "forward training pass").

[0024] To make a prediction, the input data from the mini-batch is fed into the first layer of the DNN, which is often called the "input layer." Each layer of the DNN then computes a function on its input, typically using learned parameters or "weights," to produce the input for the next layer. The output of the last layer (often called the "output layer") is a class prediction. Based on the labels predicted by the DNN and the actual labels for each instance of the training data, the output layer computes a "loss" or error function.

[0025] In the "backward pass" of a DNN (which may also be referred to herein as the "backward training pass"), each layer of the DNN computes the error of the previous layer and calculates the gradients or updates of the weights of the layer that move the DNN's predictions toward the desired output. The result of training a DNN is a set of weights or "kernels" that represent a transformation function that can be applied to the input, resulting in a classified or semantically labeled output.

[0026] The DNN training process described above has significant computational and memory requirements. Most of the memory required during DNN training is occupied by data structures (e.g., weights that change during training, weight gradients, intermediate layer outputs or "feature maps" that need to be stored during the forward pass for use in the corresponding backward pass, and reverse gradient maps). As DNNs have become larger and deeper, the size of these data structures has increased significantly. As a result, GPU main memory has become a serious bottleneck. This bottleneck limits the size of DNNs that can be trained, and as a result, limits DNNs from solving more complex problems. The technology disclosed herein addresses these and potentially other considerations.

[0027] Referring now to the drawings, in which like reference numerals represent like elements throughout the several figures, various aspects of various techniques for efficient data encoding for deep neural network training will be described. In the following detailed description, reference is made to the accompanying drawings which form a part hereof, and which show by way of illustration specific configurations or examples.

[0028] Figure 1 FIG1 is a computational architecture diagram illustrating various aspects of a configuration of a system disclosed herein for efficient data encoding for deep neural network training. In one configuration, a DNN computation graph 104A comprising nodes 106 and edges 108 is utilized, with the computation graph 104A defining the DNN. In other configurations, other representations of the DNN may be used.

[0029] The DNN computation graph 104A is provided to a schedule builder 102. The schedule builder 102 is a software or hardware component that performs static analysis on the DNN computation graph 104A of the DNN to identify and select data structures 122 for use during training of the DNN. As will be described in more detail below, the selection of the data structures 122 can be based on a layer pair defined by the DNN computation graph 104A (i.e., two adjacent layers of the DNN).

[0030] Based on the analysis, the schedule builder 102 creates a modified DNN computation graph 104B by adding nodes 106 or other types of data to the original DNN computation graph 104A. The newly added nodes 106 can define an encoding function 110 for encoding a data structure 122 during the forward training pass of the DNN. The new nodes 106 can also define a decoding function 112 for decoding the previously encoded data structure 122 during the backward training pass of the DNN. The functions added to the DNN computation graph 104A to create the modified DNN computation graph 104B can be selected based on the specific layer pairs defined by the DNN computation graph 104A.

[0031] Schedule builder 102 also performs static liveness analysis on the affected feature maps stored during the training of the DNN defined by DNN computation graph 104A and the encoded / decoded representations of the feature maps generated by encoding function 110 and decoding function 112. The static liveness analysis performed by schedule builder 102 can identify the lifetime of data structures 122 used during the training of the DNN (i.e., the time from when data structure 122 is created to when it is destroyed). Schedule builder 102 passes data 116 identifying the lifetime 116 of DNN data structures to static memory allocator 118.

[0032] The static memory allocator 118 is a software or hardware component that utilizes data 116 defining the lifetime of DNN data structures and data identifying the sizes of the data structures to generate an efficient memory allocation strategy 120 for use by the DNN runtime 114 for training the DNN. In one configuration, the static memory allocator 118 creates groups of data structures 122 whose lifetimes do not overlap during DNN training and, therefore, can share the same memory space. The amount of memory required by the data structures in each group is the size of the largest member of the group. The static memory allocator 118 sorts the data structures 122 based on size and then forms groups so that larger data structures can share the same memory space.

[0033] At the end of the above process, the static memory allocator 118 has identified multiple sets of data structures 122 used during the training of the DNN that are either dominated by feature maps stored for the reverse training pass of the DNN or immediately consumed by feature maps or gradient maps. As will be discussed in more detail below, reducing the lifetime of feature maps stored during the forward training pass and used during the reverse training pass creates an opportunity for memory sharing, which results in a smaller memory footprint during DNN training.

[0034] Once the modified DNN computation graph 104B and the memory allocation policy 120 have been generated and provided to the DNN runtime 114, the DNN defined by the modified DNN computation graph 104B can be trained. Including the encoding function 110 and the decoding function 112 in the modified DNN computation graph 104A can reduce memory utilization during training of the DNN. Additional details regarding this process are provided below.

[0035] Figure 2 is a timing diagram illustrating how an implementation of the disclosed technology can be used to reduce memory utilization during DNN training. Figure 2In the example shown, the DNN includes at least two layers 202A and 202B. The forward training pass of the DNN begins at time T1, and data structure 122 (e.g., output feature map) is generated by layer 202A at time T2. Data structure 122 is then stored in memory for use during the backward training pass. However, during the backward training pass, data structure 122 is not utilized again until time T3. As a result, memory is utilized to store data structure 122 from time T2 until time T3, even though data structure 122 is not used during this time period.

[0036] Using the disclosed techniques, the amount of memory used between time T2 and time T3 can be reduced and used to store other data structures. In particular, data structure 122 can be kept in its original format as long as it is needed for immediate forward use. Data structure 122 can then be encoded and stored for use during the reverse training pass of the DNN. The original data structure 122 can be discarded. Then, when the reverse training pass requires it (i.e., during Figure 2 At time T3 in the example shown, the encoded data structure may be decoded.

[0037] As will be described in more detail below, data structures 122 used during training of the DNN (such as input and output feature maps) can be stored using efficient encodings between the time they are no longer needed during the forward training pass and until they are needed during the backward training pass. Furthermore, if layer types and interactions are taken into account, efficient layer-specific encodings can be utilized, thereby saving additional memory during DNN training. Figures 3A-3E We describe in more detail two example layer-specific lossless encoding functions and one lossy encoding function that quickly and efficiently reduce the memory footprint during DNN training with minimal impact on DNN training accuracy.

[0038] Figure 3A is a DNN layer diagram illustrating various aspects of the utilization of data structure 122 during typical DNN training. Figure 3A In the example shown, layer 202C of the DNN may take input feature map 302A as input during a forward training pass of the DNN.

[0039] Layer 202C can perform its assigned function (e.g., ReLU or convolution) and generate an output feature map 304A. Then, during the forward training pass, the output feature map 304A is provided to layer 202D as its input. In this regard, it should be understood that the output feature map 304A and the input feature map 302B are equivalent. Layer 202D performs its assigned function to generate the output feature map 304B.

[0040] exist Figure 3A During the backward training pass of the example DNN shown, layer 202D generates and stores a gradient map 306B. Gradient map 306B is a function of input feature map 302B (X2), output feature map 304B (Y2), and gradient map 306A (DY2) generated by the next layer in the DNN (i.e., DX2 = F(X2, Y2, DY2)). Therefore, conventionally, to generate gradient map 306B, input feature map 302B and output feature map 304B are stored in GPU memory from the time they are created during the forward pass until they are used by layer 202D during the backward pass.

[0041] Layer 202D provides a gradient map 306B (DX2) equivalent to gradient map 306C (DY1) to layer 202C. Layer 202C then generates gradient map 306D (DX1) using input feature map 302A (X1), output feature map 304A (Y1), and gradient map 306C (DY1) (i.e., DX1 = F (X1, Y1, DY1)). Thus, conventionally, input feature map 302A and output feature map 304A are stored in GPU memory from the time they are generated during the forward training pass until they are consumed by layer 202C. Using the techniques disclosed herein, the amount of memory required to store these data structures during their two temporary usage periods (i.e., during the forward pass and the backward pass) is reduced. Additional details regarding this process are provided below.

[0042] Figure 3B An example of layer-specific encoding and decoding of the data structure 122 during training of the DNN is shown. In this example, the DNN computation graph 104A defines a DNN having a layer pair including a ReLU layer 202E and a max pooling layer 202F. In a conventional implementation, the ReLU layer 202E stores the input feature map 302C and the output feature map 304C during the forward training pass to find the location of the maximum value during the backward pass.

[0043] In order to reduce the DNN training process Figure 3B Memory consumption of the ReLU / max pooling layer pair is shown, and a node 106 defining an encoding function 110 can be added to the DNN computation graph 104A to generate a positive value map ("PVM") 308 during the forward training pass of the DNN. The PVM 308 indicates whether the value in the input feature map 302C to the ReLU layer of the DNN is positive.

[0044] The PVM 308 can be implemented as a data structure that stores a bit for each entry in the input feature map 302C, indicating whether each entry is positive. A node 106 defining a decoding function 112 can also be added to the DNN computation graph 104A to enable the ReLU layer 202E to utilize the PVM 308 during the reverse training pass of the DNN (i.e., DX1=F(PVM, DY1)). Therefore, the input feature map 302C and the output feature map 304C can be discarded after being utilized in the forward training pass.

[0045] When a DNN computation graph defines something like Figure 3B When the layer pair shown includes a ReLU layer and a max pooling layer, a node 106 defining an encoding function 110 for generating a data structure 310 (which may be referred to herein as a "Y2 to X2 mapping 310") during a forward training pass of the DNN may also be added to the DNN computation graph 104A.

[0046] The Y2 to X2 map 310 defines the mapping between the input feature map 302D to the max pooling layer 202F and the output feature map 304D generated by the max pooling layer 202F. In one configuration, the Y2 to X2 map 310 has as many elements as the output feature map 304D, where each element is stored using 4 bits. The max pooling layer 203F can use the Y2 to X2 map 310 and the gradient map 306E (DY2) to calculate the gradient map 306F (DX2) (i.e., DX2 = F (Y2 to X2 map, DY2).

[0047] The storage and utilization of the Y2 to X2 mapping 310 eliminates the dependency of the max pooling layer 202F on its input feature map 302D and its output feature map 304D during the backward training pass, and thus these data structures can be discarded after being consumed in the forward training pass. The ReLU layer 202E and the max pooling layer 202F can be modified to operate directly on the PVM 308 and the Y2 to X2 mapping 310, respectively, during the backward training pass of the DNN.

[0048] Figure 3C and 3D Several additional examples of layer-specific encoding and decoding of data structure 122 during training of a DNN are shown. Figure 3C and 3DIn the example shown, sparse storage and dense computation ("SSDC") encoding is utilized. SSDC isolates computation and storage, facilitating storage of data structures in sparse format and computation on them in dense format. As described in more detail below, SSDC stores data structures in sparse format for most of their lifetime and converts them back to dense format just before computation requires them. This mechanism can significantly reduce memory usage while retaining the performance benefits of a highly optimized DNN runtime.

[0049] exist Figure 3C In the example shown, for example, the DNN computation graph 104A defines a DNN that includes a layer pair having a ReLU layer 202G and a convolutional layer 202H. In this configuration, a node 106 for an encoding function 110 can be added to the DNN computation graph 104A that generates a memory-efficient sparse representation 312 (Y1') of the output feature map 304E (Y1) generated by the ReLU layer 202G during the forward training pass of the DNN. As described above, the output feature map 304E of the ReLU layer is the same as the input feature map 302F to the convolutional layer. For clarity, the memory-efficient sparse representation 314 (X2') of the input feature map 302F is shown in FIG. Figure 3C However, it should be understood that sparse representation 312 and sparse representation 314 are identical, and therefore, only one sparse representation is computed and stored during the forward training pass.

[0050] Nodes for performing backward pass computation using the output feature map 304E (Y1) and input feature map 302F (X2) in their original dense format are also added to the DNN computation graph 104A, thereby retaining the performance advantages of highly optimized dense computation while exploiting sparsity to achieve a significant reduction in memory usage. For example, Figure 3C As shown, the sparse representation 314 can be converted back to a dense input feature map 302F to be used by the convolutional layer 202H during the backward training pass. Similarly, the sparse representation 312 can be converted to the original dense output feature map 304E before being utilized by the ReLU layer 202G during the backward pass. In other configurations, the ReLU layer 202G and the convolutional layer 202H operate directly on the sparse representations 312 and 314, respectively (i.e., DX1=F(Y1′, DY1) and DX2=F(X2′, DY2)).

[0051] exist Figure 3DIn the example shown, the DNN computation graph 104A defines a DNN that includes a layer pair having a max pooling layer 202I and a convolutional layer 202J. In this configuration, a node 106 for an encoding function 110 can be added to the DNN computation graph 104A that generates a memory-efficient sparse representation 312 (Y1') of the output feature map 304G (Y1) generated by the max pooling layer 202I during the forward training pass of the DNN. As in the example above, the output feature map 304G of the max pooling layer 202I is the same as the input feature map 302H to the convolutional layer 202J. For clarity, the memory-efficient sparse representation 314 (X2') of the input feature map 302H is shown in FIG. Figure 3D However, it should be understood that sparse representation 312 and sparse representation 314 are identical, and therefore, only one sparse representation is computed and stored during the forward training pass.

[0052] Nodes for performing backward pass computation using the output feature map 304G (Y1) and input feature map 302H (X2) in their original dense format are also added to the DNN computation graph 104A, thereby retaining the performance advantages of highly optimized dense computation while exploiting sparsity to achieve a significant reduction in memory usage. For example, Figure 3D As shown, the sparse representation 314 (X2') can be converted back to a dense input feature map 302H to be used by the convolution layer 202J during the backward training pass. Similarly, the sparse representation 312 can be converted to the original dense output feature map 304G before being utilized by the max pooling layer 202I during the backward pass. In other configurations, the max pooling layer 202I and the convolution layer 202J operate directly on the sparse representations 312 and 314, respectively (i.e., DX1=F(Y1', DY1) and DX2=F(X2', DY2)).

[0053] In some configurations, a compressed sparse row ("CSR") format is used to store the sparse representations 312 and 314. This format stores the non-zero values of the corresponding feature map, as well as an array of elements that holds the column indices of the non-zero values in each row. Two-dimensional ("2D") data structures commonly used by DNN frameworks can be converted to the CSR format. In some configurations, narrow value optimization (NVR) can also be performed, in which the shape of the above-mentioned 2D matrix is adjusted and the number of columns is limited to 256, so that each column index requires only 1 byte. This can reduce the minimum sparsity requirement for effective compression from 50% to 20%, resulting in wider applicability and higher compression rates.

[0054] In such Figure 3EIn some configurations shown, a node can be added to the DNN computation graph 104A for reducing the mathematical precision of a feature map after the feature map is consumed during the forward training pass of the DNN. For example, and not limitation, the mathematical precision of the input feature map and the output feature map can be reduced when they are no longer needed during the forward training pass. Because the precision reduction is delayed in the forward training pass until the feature map is consumed, this mechanism may be referred to herein as delayed precision reduction ("DPR"). DPR allows GPU memory used to store the original feature map to be released after the feature map has been used during the forward pass, thereby saving memory.

[0055] exist Figure 3E In the example shown, the input feature map 302I has been converted to a reduced precision input feature map 316 (X1”). Similarly, the output feature map 304I has been converted to a reduced precision output feature map 318 (Y1”). The reduced precision input feature map 316 and the reduced precision output feature map 318 can be utilized during the backward training pass to calculate the gradient map 306N (i.e., DX1=F(X1”, Y1”, DY1)).

[0056] exist Figure 3E In the example shown, the input feature map 302J has been converted to a reduced precision input feature map 320 (X2”). Similarly, the output feature map 304J has been converted to a reduced precision output feature map 322 (Y2”). The reduced precision input feature map 320 and the reduced precision output feature map 322 can be utilized during the backward training pass to calculate the gradient map 306N (i.e., DX2=F(X2”, Y2”, DY2)).

[0057] It should be understood that Figure 3E In the example shown, the reduced precision feature maps are directly utilized during the backward training pass instead of the original feature maps. However, in other configurations, the reduced precision feature maps are converted back to their higher precision representations before being used in the backward training pass. It should also be understood that DPR is applicable to any layer combination. In addition to the SSDC encoding described above, DPR can also be applied to compress non-zero value arrays in CSR format.

[0058] In some configurations, 2, 3, and 4 values are packed into 4 bytes using three smaller representations of 16 bits, 10 bits, and 8 bits, respectively. To pack 3 values into 4 bytes, the maximum possible length is 10 bits (9 bits results in 5 bits unused, and 11 bits requires one more bit). For 16 bits, the IEEE half-precision floating point format (1 sign, 5 exponent, and 10 mantissa bits) can be utilized, which may be referred to as "FP16." For 8 bits ("FP8"), 1 bit can be used for the sign, 4 bits for the exponent, and 3 bits for the mantissa; for 10 bits ("FP10"), 1 sign bit, 5 exponent bits, and 4 mantissa bits can be used. In FP10, three 10-bit values can be stored in a 4-byte space, making 2 bits unnecessary. The normalized numbers can be ignored because their impact on the accuracy of the DNN is negligible. A round-to-nearest rounding strategy can be used for these conversions. Since the conversions can occur in parallel, DPR can minimize performance overhead.

[0059] Now refer to Figure 4 , a flowchart showing a routine 400 will be described that illustrates aspects of an illustrative computer-implemented process for efficient data encoding for DNN training. Figure 4 The logical operations described with respect to the other figures may be implemented as (1) a sequence of computer-implemented acts or program modules running on a computing device, and / or (2) interconnected machine logic circuits or circuit modules within a computing device.

[0060] The specific implementation of the technology disclosed herein is a matter of choice depending on the performance and other requirements of the computing device. Therefore, the logical operations described herein are variously referred to as states, operations, structural devices, actions, or modules. These states, operations, structural devices, actions, and modules can be implemented with hardware, software, firmware, dedicated digital logic, and any combination thereof. It should be understood that more or fewer operations than those shown in the figures and described herein can be performed. These operations can also be performed in an order different from the order described herein.

[0061] The routine 400 begins at operation 402, where the schedule builder 102 receives the DNN computation graph 104A. The routine 400 then proceeds to operation 404, where the DNN computation graph 104A identifies edges in the DNN computation graph 104A where encoding and decoding will reduce memory utilization. The routine 400 then proceeds from operation 404 to operation 406.

[0062] At operation 406, the schedule builder 102 generates a modified computation graph 104B using the encoding and decoding functions for reducing memory utilization. The routine 400 then proceeds to operation 408, where the schedule builder 102 calculates the DNN data structure lifetime in the manner described above. Then, at operation 410, the data 116 describing the DNN data structure lifetime is provided to the static memory allocator 118.

[0063] The routine 400 then proceeds from operation 410 to operation 412, where the static memory allocator 118 calculates a memory allocation policy 120 to be used by the DNN runtime 114. Once the memory allocation policy 120 has been calculated, the DNN runtime 114 can train the DNN using the modified DNN computation graph 104B and the memory allocation policy 120. The routine 400 then proceeds from operation 412 to operation 414, where it ends.

[0064] Figure 5 is a computer architecture diagram showing an illustrative computer hardware and software architecture for a computing device that can implement the various techniques presented herein. In particular, Figure 5 The illustrated architecture may be used to implement a server computer, mobile phone, e-reader, smartphone, desktop computer, alternate reality or virtual reality (“AR / VR”) device, tablet computer, laptop computer, or other type of computing device.

[0065] Although the subject matter described herein is presented in the general context of a server computer performing DNN training, those skilled in the art will recognize that other implementations may be performed in conjunction with other types of computing systems and modules. Those skilled in the art will also recognize that the subject matter described herein may be practiced with other computer system configurations, including handheld devices, multi-processor systems, microprocessor-based or programmable consumer electronics, computing or processing systems embedded in devices (such as wearable computing devices, automobiles, home automation, etc.), minicomputers, mainframe computers, etc.

[0066] Figure 5The illustrated computer 500 includes one or more central processing units 502 ("CPUs"), one or more GPUs 530, system memory 504 (which includes random access memory 506 ("RAM") and read-only memory ("ROM") 508), and a system bus 510 that couples the memory 504 to the CPU 502. A basic input / output system ("BIOS" or "firmware"), containing basic routines that help transfer information between elements within the computer 500, such as during startup, may be stored in the ROM 508. The computer 500 also includes a mass storage device 512 for storing an operating system 522, application programs, and other types of programs. The mass storage device 512 may also be configured to store other types of programs and data, such as the DNN computation graph 104A, the schedule builder 102, the modified DNN computation graph 104B, the DNN data structure lifetime 116, the static memory allocator 118, the memory allocation policy 120, the DNN runtime 114, and the data structures 122.

[0067] The mass storage device 512 is connected to the CPU 502 through a mass storage controller (not shown) connected to the bus 510. The mass storage device 512 and its associated computer-readable media provide non-volatile storage for the computer 500. Although the descriptions of computer-readable media contained herein refer to mass storage devices, such as a hard disk, a CD-ROM drive, a DVD-ROM drive, or a USB memory key, those skilled in the art will understand that computer-readable media can be any available computer storage media or communication media that can be accessed by the computer 500.

[0068] Communication media includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any delivery media. The term "modulated data signal" refers to a signal that has one or more of its characteristics changed or set in such a way as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency, infrared, and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.

[0069] By way of example and not limitation, computer storage media may include volatile and nonvolatile removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. For example, computer storage media include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technology, CD-ROM, digital versatile disks ("DVD"), HD-DVD, BLU-RAY or other optical storage devices, cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computer 500. In the claims, the phrase "computer storage media" and its variations do not include waves or signals themselves or communication media.

[0070] According to various configurations, the computer 500 can operate in a networked environment using logical connections to remote computers via a network such as the network 520. The computer 500 can be connected to the network 520 via a network interface unit 516 connected to the bus 510. It should be understood that the network interface unit 516 can also be used to connect to other types of networks and remote computer systems. The computer 500 can also include an input / output controller 518 for receiving and processing input from many other devices, including a keyboard, a mouse, a touch input, an electronic pen ( Figure 5 Similarly, the input / output controller 518 may provide output to a display screen or other type of output device (also shown in FIG. Figure 5 not shown).

[0071] It should be understood that the software components described herein, when loaded into the CPU 502 and executed, can transform the CPU 502 and the entire computer 500 from a general-purpose computing device into a specialized computing device customized to facilitate the functions described herein. The CPU 502 can be constructed from any number of transistors or other discrete circuit elements that can individually or collectively assume any number of states. More specifically, in response to the executable instructions contained within the software modules disclosed herein, the CPU 502 can operate as a finite state machine. These computer-executable instructions can transform the CPU 502 by specifying how the CPU 502 transitions between states, thereby transforming the transistors or other discrete hardware elements that make up the CPU 502.

[0072] Encoding the software modules presented herein may also transform the physical structure of the computer-readable medium presented herein. In different implementations of this specification, the specific transformation of the physical structure depends on various factors. Examples of such factors include, but are not limited to, the technology used to implement the computer-readable medium, whether the computer-readable medium is characterized as primary storage or secondary storage, etc. For example, if the computer-readable medium is implemented as a semiconductor-based memory, the software disclosed herein can be encoded on the computer-readable medium by transforming the physical state of the semiconductor memory. For example, the software can transform the state of transistors, capacitors, or other discrete circuit elements that make up the semiconductor memory. The software can also transform the physical state of these components in order to store data thereon.

[0073] As another example, the computer storage media disclosed herein can be implemented using magnetic or optical technology. In such an implementation, when software is encoded therein, the software proposed herein can change the physical state of the magnetic or optical medium. These transformations can include changing the magnetic properties of specific locations within a given magnetic medium. These transformations can also include changing the physical features or characteristics of specific locations within a given optical medium to change the optical properties of these locations. Other transformations of the physical medium are possible without departing from the scope and spirit of this specification, and the foregoing examples are provided only to facilitate this discussion.

[0074] In view of the above, it should be understood that many types of physical transformations occur in the computer 500 in order to store and execute the software components presented herein. Figure 5 The architecture shown, or similar architectures, may be used to implement other types of computing devices, including handheld computers, video game devices, embedded computer systems, mobile devices such as smartphones, tablet computers, and AR / VR devices, as well as other types of computing devices known to those skilled in the art. It is also contemplated that the computer 500 may not include Figure 5 All components shown can be included in Figure 5 Other components not explicitly shown in the figure, or components that may be used with Figure 5 The architecture shown is a completely different architecture.

[0075] Figure 6 6 is a network diagram illustrating a distributed network computing environment 600 according to various configurations presented herein in which aspects of the disclosed technology may be implemented. Figure 6As shown, one or more server computers 600A can be interconnected with many client computing devices (such as but not limited to tablet computers 600B, game consoles 600C, smart watches 600D, phones such as smart phones 600E, personal computers 600F and AR / VR devices 600G) via a communication network 520 (which can be any one or combination of fixed-line or wireless LAN, WAN, intranet, extranet, peer-to-peer network, virtual private network, the Internet, Bluetooth communication network, proprietary low-voltage communication network or other communication networks).

[0076] For example, in a network environment where the communication network 520 is the Internet, the server computer 600A may be a dedicated server computer operable to process and communicate data with the client computing devices 600B-600G via any of a variety of known protocols, such as Hypertext Transfer Protocol ("HTTP"), File Transfer Protocol ("FTP"), or Simple Object Access Protocol ("SOAP"). Additionally, the network computing environment 600 may utilize various data security protocols, such as Secure Sockets Layer ("SSL") or Pretty Good Privacy ("PGP"). Each client computing device 600B-600G may be equipped with an operating system operable to support one or more computing applications or terminal sessions, such as a web browser ( Figure 6 Not shown) or other graphical user interface ( Figure 6 Not shown) or a mobile desktop environment ( Figure 6 ), to access the server computer 600A.

[0077] The server computer 600A can be communicatively coupled to other computing environments ( Figure 6 ), and receives data about the interaction / resource network of participating users. In illustrative operation, the user ( Figure 6 ) can interact with computing applications running on client computing devices 600B-600G to obtain desired data and / or execute other computing applications.

[0078] Data and / or computing applications may be stored on one or more servers 600A and transmitted to collaborating users via client computing devices 600B-600G via the exemplary communication network 520. Participating Users ( Figure 6 Client computing devices 600B-600G may request access to specific data and applications hosted in whole or in part on server computer 600A. Such data may be transferred between client computing devices 600B-600G and server computer 600A for processing and storage.

[0079] The server computer 600A can host computing applications, processes, and applets for data and application generation, authentication, encryption, and communication, and can communicate with other server computing environments ( Figure 6 Not shown), third-party service providers ( Figure 6 (not shown), network attached storage ("NAS"), and storage area network ("SAN") work together to enable application / data transactions.

[0080] It should be understood that for the sake of discussion, Figure 5 The computing architecture shown and Figure 6 The distributed network computing environment shown has been simplified. It should also be understood that the computing architecture and distributed computing network may include and utilize many more computing components, devices, software programs, network devices, and other components not specifically described herein.

[0081] The disclosure presented herein also encompasses the subject matter set forth in the following Examples:

[0082] Example A: A computer-implemented method comprising: performing analysis on a DNN computation graph of a DNN to identify one or more data structures created during training of the DNN; selecting a data structure from the one or more data structures to be encoded during training of the DNN based on the analysis; creating a modified DNN computation graph by adding at least one node to the DNN computation graph, the at least one node defining an encoding function for encoding the selected data structure during a forward pass of the DNN when training the DNN; and causing the DNN to be trained using the modified DNN computation graph.

[0083] Example B. The computer-implemented method of Example A, wherein the selected data structure and the encoding function are selected based on a layer in a layer pair of the DNN.

[0084] Example C: A computer-implemented method according to any one of Examples A to B, wherein the first layer in the layer pair includes a rectified linear unit (ReLU) layer, and wherein the second layer in the layer pair includes a pooling layer.

[0085] Example D: A computer-implemented method according to any one of Examples A to C, wherein the selected data structure includes a positive map (PVM), which indicates whether the value in the input feature map of the ReLU layer of the DNN is positive.

[0086] Example E: A computer-implemented method according to any one of Examples A to D, wherein the selected data structure includes a mapping between an output feature map generated by the pooling layer and an input feature map to the pooling layer.

[0087] Example F: A computer-implemented method according to any one of Examples A to E, wherein creating the modified DNN further comprises adding at least one node defining a decoding function, the decoding function being used to decode the selected data structure during a backward pass of the DNN when training the DNN.

[0088] Example G: A computer-implemented method according to any one of Examples A to F, wherein the first layer in the layer pair comprises a rectified linear unit (ReLU) layer, and wherein the second layer in the layer pair comprises a convolutional layer.

[0089] Example H: A computer-implemented method according to any one of Examples A to G, wherein the selected data structure includes an output feature map generated by a ReLU layer.

[0090] Example I: A computer-implemented method according to any one of Examples A to H, wherein the selected data structure includes an input feature map consumed by a convolutional layer.

[0091] Example J: A computer-implemented method according to any one of Examples A to I, wherein the selected data structure includes an input feature map to a layer of the DNN, and wherein the encoding function causes the precision of the input feature map to be reduced during training of the DNN.

[0092] Example K: A computer-implemented method according to any one of Examples A to J, wherein the selected data structure includes output feature maps generated by a layer of the DNN, and wherein the encoding function causes the precision of the input feature maps to be reduced during training of the DNN.

[0093] Example L: A computing device comprising: one or more processors; and at least one computer storage medium having computer-executable instructions stored thereon, the computer-executable instructions, when executed by the one or more processors, causing the computing device to: execute a schedule builder before training a DNN, the schedule builder being configured to analyze a DNN computation graph of the DNN to select data structures to be encoded during the training of the DNN based on layers in layer pairs of the DNN, create a modified DNN computation graph by adding at least one encoding function for encoding the selected data structures during a forward training pass of the DNN, and determine a lifetime of the selected data structures during the training of the DNN; execute a static memory allocator before training the DNN, the static memory allocator being configured to generate a memory allocation policy based on the lifetime of the selected data structures; and train the DNN using the modified DNN computation graph, wherein the memory allocation policy is utilized during the training of the DNN to allocate and deallocate memory for storing the selected data structures.

[0094] Example M: A computing device according to Example L, wherein the first layer in the layer pair includes a rectified linear unit (ReLU) layer, and wherein the second layer in the layer pair includes a pooling layer.

[0095] Example N: A computing device according to any of Examples L to M, wherein the first layer in the layer pair includes a rectified linear unit (ReLU) layer, and wherein the second layer in the layer pair includes a convolutional layer.

[0096] Example O: A computing device according to any one of Examples L to N, wherein the selected data structure includes an input feature map to a layer of the DNN, and wherein the encoding function causes the accuracy of the input feature map or the output feature map to be reduced during the training of the DNN.

[0097] Example P: A computer storage medium having computer-executable instructions stored thereon, which, when executed by one or more processors of a computing device, will cause the computing device to: analyze the DNN to select a data structure to be encoded during training of the DNN based on a layer in a layer pair of the DNN; create a modified DNN by adding at least one encoding function to the DNN for encoding the selected data structure during a forward training pass; determine a lifetime of the selected data structure during training of the modified DNN; generate a memory allocation policy based on the lifetime of the selected data structure; and cause the modified DNN to be trained using the memory allocation policy.

[0098] Example Q: The computer storage medium of Example P, wherein the first layer in the layer pair comprises a rectified linear unit (ReLU) layer, and wherein the second layer in the layer pair comprises a pooling layer.

[0099] Example R: A computer storage medium according to any of Examples P to Q, wherein the first layer in the layer pair comprises a rectified linear unit (ReLU) layer, and wherein the second layer in the layer pair comprises a convolutional layer.

[0100] Example S: A computer storage medium according to any one of Examples P to R, wherein the selected data structure includes an input feature map to a layer of the DNN, and wherein the encoding function causes the precision of the input feature map or the output feature map to be reduced during the training of the DNN.

[0101] Example T: The computer storage medium of any of Examples P to S, wherein creating the modified DNN further comprises adding at least one decoding function for decoding the selected data structure during a backward training pass.

[0102] Based on the foregoing, it should be understood that efficient data encoding techniques for deep neural network training have been disclosed herein. Although the subject matter presented herein has been described in language specific to computer structural features, methods, and conversion actions, specific computing machines, and computer-readable media, it should be understood that the subject matter set forth in the appended claims is not necessarily limited to the specific features, actions, or media described herein. Rather, the specific features, actions, and media are disclosed as example forms of implementing the claimed subject matter.

[0103] The subject matter described above is provided by way of example only and should not be construed as limiting. Various modifications and changes may be made to the subject matter described herein without following the example configurations and applications shown and described, and without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A computer-implemented method for data encoding for deep neural network (DNN) training, comprising: performing, by a schedule builder, analysis of a DNN computation graph of a DNN to identify one or more data structures created during training of the DNN, the DNN computation graph specifying a plurality of layer pairs of the DNN; selecting, by the schedule builder, from the one or more data structures to be encoded during training of the DNN based on the analysis; creating, by the schedule builder, a modified DNN computation graph by adding at least one node to the DNN computation graph, the at least one node defining an encoding function for encoding the selected data structure during a forward pass of the DNN while training the DNN; determining, by the schedule builder, a lifetime of the selected data structure; generating, by a static memory allocator, a memory allocation policy to be used by a DNN runtime based on the lifetime, wherein the memory allocation policy is utilized during the training of the DNN to allocate and deallocate memory for storing the selected data structure; as well as The DNN is trained by the DNN runtime using the modified DNN computation graph and the memory allocation policy, wherein the selected data structure and the encoding function are selected based on a layer in a layer pair of the DNN.

2. The computer-implemented method of claim 1 , wherein the first layer of the layer pair comprises a rectified linear unit layer, and wherein the second layer of the layer pair comprises a pooling layer.

3. The computer-implemented method of claim 2 , wherein the selected data structure comprises a positive map that indicates whether a value in an input feature map to the rectified linear unit layer of the DNN is positive.

4. The computer-implemented method of claim 2, wherein the selected data structure comprises a mapping between an output feature map generated by the pooling layer and an input feature map to the pooling layer.

5. The computer-implemented method of claim 1 , wherein creating the modified DNN further comprises adding at least one node defining a decoding function for decoding the selected data structure during a backward pass of the DNN while training the DNN.

6. A computing device for data encoding for deep neural network (DNN) training, comprising: one or more processors; as well as at least one computer storage medium having stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing device to: analyzing, via a schedule builder, a DNN computation graph for a DNN to identify one or more data structures created during training of the DNN, the DNN computation graph specifying a plurality of layer pairs of the DNN; selecting, via the schedule builder, a data structure from the one or more data structures to be encoded during training of the DNN based on the analysis; creating, via the schedule builder, a modified DNN computation graph by adding at least one node to the DNN computation graph, the at least one node defining an encoding function for encoding the selected data structure during a forward pass of the DNN while training the DNN; determining, via the schedule builder, a lifetime of the selected data structure; generating, via a static memory allocator, a memory allocation policy to be used by a DNN runtime based on the lifetime, wherein the memory allocation policy is utilized during the training of the DNN to allocate and deallocate memory for storing the selected data structure; as well as The DNN is trained via the DNN runtime using the modified DNN computation graph and the memory allocation policy, wherein the selected data structure and the encoding function are selected based on a layer in a layer pair of the DNN.

7. The computing device of claim 6, wherein the first layer of the layer pair comprises a rectified linear unit layer, and wherein the second layer of the layer pair comprises a pooling layer.

8. The computing device of claim 6, wherein the first layer of the layer pair comprises a rectified linear unit layer, and wherein the second layer of the layer pair comprises a convolutional layer.

9. The computing device of claim 6 , wherein the selected data structure comprises an input feature map to a layer of the DNN, and wherein the encoding function causes the precision of the input feature map or the output feature map to be reduced during the training of the DNN.

10. A computer storage medium having stored thereon computer-executable instructions that, when executed by one or more processors of a computing device, cause the computing device to: analyzing, via a schedule builder, a DNN computation graph for a deep neural network (DNN) to identify one or more data structures created during training of the DNN, the DNN computation graph specifying a plurality of layer pairs of the DNN; selecting, via the schedule builder, a data structure from the one or more data structures to be encoded during training of the DNN based on the analysis; creating, via the schedule builder, a modified DNN computation graph by adding at least one node to the DNN computation graph, the at least one node defining an encoding function for encoding the selected data structure during a forward pass of the DNN while training the DNN; determining, via the schedule builder, a lifetime of the selected data structure; generating, via a static memory allocator, a memory allocation policy to be used by a DNN runtime based on the lifetime, wherein the memory allocation policy is utilized during the training of the DNN to allocate and deallocate memory for storing the selected data structure; as well as The DNN is trained via the DNN runtime using the modified DNN computation graph and the memory allocation policy, wherein the selected data structure and the encoding function are selected based on a layer in a layer pair of the DNN.

11. The computer storage medium of claim 10, wherein the first layer of the layer pair comprises a rectified linear unit layer, and wherein the second layer of the layer pair comprises a pooling layer.

12. The computer storage medium of claim 10, wherein the first layer of the layer pair comprises a rectified linear unit layer, and wherein the second layer of the layer pair comprises a convolutional layer.

13. The computer storage medium of claim 10, wherein the selected data structure comprises an input feature map to a layer of the DNN, and wherein the encoding function causes the precision of the input feature map or the output feature map to be reduced during the training of the DNN.

14. The computer storage medium of claim 10, wherein creating the modified DNN further comprises adding at least one decoding function for decoding the selected data structure during a backward training pass.

Images