ReLU compression for reducing GPU memory

Abstract

We present a method for compressing rectified linear unit (ReLU) function data on a graphics processing unit (GPU) used in the training process of deep neural networks. The method includes converting an initial data structure containing both nonzero and zero data into a compressed data structure containing only the nonzero data of the initial data structure as compressed data by generating a nonzero data bitmap region, generating a nonzero data count table region using a parallel reduction algorithm, calculating a block-wise nonzero data array index region for all blocks from the nonzero data count table region using a parallel prefix sum scan algorithm, allocating a buffer for the compressed data, and copying the nonzero data from the initial data structure to the nonzero data array region in a compressed data format in parallel.

Description

[Technical Field]

[0001] The present invention relates generally to compressing graphics processing unit (GPU) memory, and more particularly to reducing GPU memory through rectified linear unit (ReLU) compression techniques. [Background technology]

[0002] A device can have one or more central processing units (CPUs) and one or more graphics processing units (GPUs). Each CPU is a general-purpose processing device, such as a microprocessor or other type of processor, used for the device's general computing tasks. Each GPU is a specialized electronic circuit designed to rapidly manipulate and modify memory to speed up the creation of images in a frame buffer for display output. Recently, these GPUs have been used for general-purpose computing on the GPU for computer tasks. General-purpose computing on the GPU refers to utilizing a GPU, which typically handles computer graphics-only computations, to perform computations traditionally handled by the CPU. For example, a GPU offering a functional set of operations that operate on arbitrary bits can be used for general-purpose computing on the GPU.

[0003] However, GPU memory is generally smaller than CPU memory, and GPUs may be insufficient for large neural network models. Data transfer between the CPU and GPU is relatively slow compared to the GPU's computational speed. Therefore, it may be difficult for GPUs to effectively compress data. Compression requires a large amount of computation, and it may be difficult to achieve sufficient parallelism on GPUs. Summary of the Invention

[0004] According to one embodiment, a method for compressing rectified linear unit (ReLU) function data on a graphics processing unit (GPU) used in a training process of a deep neural network is provided. The method includes converting an initial data structure containing non-zero and zero data into a compressed data structure containing only the non-zero data of the initial data structure as compressed data by generating a "non-zero data bitmap" region, generating a "non-zero data count table" region using a parallel reduction algorithm, calculating a "block-wise non-zero data array index" region for every block from the "non-zero data count table" region using a parallel prefix sum scan algorithm, allocating a buffer for the compressed data, and copying the non-zero data from the initial data structure to the "non-zero data array" region in parallel.

[0005] A non-transitory computer-readable storage medium is presented, which includes a computer-readable program for compressing rectified linear unit (ReLU) function data on a graphics processing unit (GPU) used in the training process of a deep neural network. The computer-readable program, when executed on a computer, causes the computer to perform the steps of converting an initial data structure containing non-zero data and zero data into a compressed data structure containing only the non-zero data of the initial data structure as compressed data, the converting being performed by generating a "non-zero data bitmap" region, generating a "non-zero data count table" region using a parallel reduction algorithm, calculating a "block-wise non-zero data array index" region for every block from the "non-zero data count table" region using a parallel prefix sum scan algorithm, allocating a buffer for the compressed data, and copying the non-zero data from the initial data structure to the "non-zero data array" region in compressed data format in parallel.

[0006] A system for compressing rectified linear unit (ReLU) function data on a graphics processing unit (GPU) used in the training process of a deep neural network is presented. The system includes a memory and one or more processors in communication with the memory, the one or more processors configured to perform a conversion of an initial data structure containing nonzero and zero data into a compressed data structure containing only the nonzero data of the initial data structure as compressed data, by generating a "nonzero data bitmap" region, generating a "nonzero data count table" region using a parallel reduction algorithm, calculating a "block-wise nonzero data array index" region for every block from the "nonzero data count table" region using a parallel prefix sum scan algorithm, allocating a buffer for the compressed data, and copying the nonzero data from the initial data structure to the "nonzero data array" region in parallel.

[0007] It should be noted that exemplary embodiments are described with reference to different subject matters. In particular, some embodiments are described with reference to method-type claims, and other embodiments are described with reference to apparatus-type claims. However, those skilled in the art will understand from the above and below description that, unless otherwise specified, any combination of features belonging to one type of subject matter, as well as any combination between features relating to different subject matters, in particular between features of a method-type claim and a feature of an apparatus-type claim, is also considered to be described within this document.

[0008] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings.

[0009] The present invention is provided in more detail in the following description of preferred embodiments with reference to the following drawings. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 illustrates a graphics processing unit (GPU) architecture versus a central processing unit (CPU) architecture, according to one embodiment of the present invention. [Figure 2] FIG. 2 illustrates an exemplary GPU memory, according to one embodiment of the present invention. [Figure 3] FIG. 1 illustrates an exemplary convolutional neural network (CNN) with a rectified linear unit (ReLU) layer, according to one embodiment of the present invention. [Figure 4] FIG. 1 illustrates an exemplary deep learning CNN, according to one embodiment of the present invention. [Figure 5] FIG. 2 illustrates an exemplary data structure of an uncompressed data input, according to one embodiment of the present invention. [Figure 6] FIG. 2 illustrates an exemplary data structure of compressed data, according to one embodiment of the present invention. [Figure 7] FIG. 2 illustrates an exemplary compression algorithm methodology, according to one embodiment of the present invention. [Figure 8] FIG. 2 illustrates an exemplary restoration algorithm methodology, according to one embodiment of the present invention. [Figure 9] FIG. 2 is a block / flow diagram of a compression method for reducing GPU memory via rectified linear unit (ReLU) compression techniques, according to one embodiment of the present invention. [Figure 10] FIG. 1 is a block / flow diagram of a decompression method for reducing GPU memory via rectified linear unit (ReLU) compression techniques, according to one embodiment of the present invention. [Figure 11] FIG. 1 is a block / flow diagram of a computational method for reducing GPU memory through rectified linear unit (ReLU) compression techniques, according to one embodiment of the present invention. [Figure 12] FIG. 1 is a block / flow diagram of an exemplary cloud computing environment, according to one embodiment of the present invention. [Figure 13] FIG. 2 is a schematic diagram of an exemplary abstraction model layer, according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] Throughout the drawings, the same or similar reference numbers represent the same or similar elements.

[0012] An exemplary embodiment of the present invention provides compression of graphics processing unit (GPU) memory in deep learning (DL) models. DL models include several layers, such as convolutional layers, rectified linear unit (ReLU) layers, pooling layers, fully connected (FC) layers, and loss layers (during the training process). The exemplary embodiment of the present invention uses a compression / decompression technique for the ReLU layer of a DL model. Specifically, approximately half of the floating-point numbers in the output of the ReLU layer of a DL model are zero. ReLU outputs "zero" for all negative inputs. The exemplary embodiment of the present invention aims to eliminate "zero" data in the output of the ReLU layer and provide a bitmap that records the locations of the "zero" data. Thus, the compression / decompression technique focuses on eliminating such "zero data," where "zero data" refers to data designated as "0."

[0013] Exemplary embodiments of the present invention disclose methods and systems for advantageously reducing GPU memory. In one exemplary method, ReLU compression is used to reduce GPU memory by compressing feature maps generated by ReLU. In another embodiment, ReLU compression is used in conjunction with a recalculation method to compress GPU memory by compressing and storing feature maps generated by ReLU layers for the starting point of recalculation and discarding other feature maps. In yet another embodiment, ReLU compression is used in conjunction with a data swapping method to compress GPU memory by compressing feature maps generated by ReLU layers on the GPU before transferring them to CPU memory. This effectively reduces data transfer time from the GPU to the CPU.

[0014] An exemplary embodiment of the present invention further includes a compression system that generates a "non-zero data bitmap" in parallel, generates a "non-zero data count table" using a parallel reduction algorithm, calculates a "non-zero data array index per block" for every block from the "non-zero data count table" using a parallel prefix sum scan algorithm, allocates buffers for the compressed data, and copies non-zero data from the original data to the "non-zero data array" in parallel.An exemplary embodiment of the present invention further includes a restoration system that allocates buffers from the original data, clears the buffers, calculates a "non-zero data array index per block" for every block from the "non-zero data count table" using a parallel prefix sum scan algorithm, and copies non-zero data from the original data to the "non-zero data array" in parallel according to the "non-zero data bitmap" data.

[0015] While the present invention is described with respect to a given exemplary architecture, it should be understood that other architectures, structures, substrate materials, and process features and steps / blocks may be modified within the scope of the present invention. Note that for clarity, certain features may not be shown in every figure. This is not intended to be construed as limiting the scope of the particular embodiments or examples or the claims.

[0016] Various illustrative embodiments of the present invention are described below. For clarity, not all features of an actual implementation are described herein. It will, of course, be understood that in developing such an actual embodiment, numerous implementation-specific decisions must be made to achieve the particular goals of the developer, such as compliance with system-related and business-related constraints, which will vary from implementation to implementation. It will further be understood that such a development effort may be complex and time-consuming, but would nevertheless be a routine undertaking for those of ordinary skill in the art having the benefit of this invention.

[0017] FIG. 1 is a diagram illustrating a graphics processing unit (GPU) architecture versus a central processing unit (CPU) architecture, according to one embodiment of the present invention.

[0018] The CPU 10 works in conjunction with the GPU 20 to increase data throughput and the number of simultaneous calculations within an application. While GPUs were originally designed to create images for computer graphics and video game consoles, GPUs can also be used to accelerate calculations involving large amounts of data (for example, in DL models).

[0019] A GPU cannot completely replace a CPU. The GPU complements the CPU architecture by allowing iterative calculations within an application to be performed in parallel while the main program continues to run on the CPU. The CPU can be thought of as the system-wide taskmaster coordinating a wide range of general-purpose computing tasks, while the GPU performs a narrower range of more specialized tasks (usually mathematical). By harnessing the power of parallel processing, the GPU can complete more work in the same amount of time compared to the CPU.

[0020] The primary difference between the CPU 10 and GPU architecture 20 is that the CPU 10 is designed to process a wide range of tasks quickly (as measured by CPU clock speed), but with a limited degree of concurrency in the tasks it can perform. The GPU 20 is designed to rapidly render high-resolution images and video in parallel.

[0021] Because GPUs can perform parallel operations on multiple data sets, they are also commonly used for non-graphical tasks such as machine learning and scientific computing. GPUs are designed to have thousands of processor cores running simultaneously, allowing for massively parallel processing, where each core is focused on performing an efficient calculation.

[0022] Although GPUs can process data orders of magnitude faster than CPUs through massive parallelism, GPUs are not as versatile as CPUs. CPUs have large and extensive instruction sets that manage all of the computer's inputs and outputs, something GPUs do not have. While individual CPU cores are faster (as measured by CPU clock speed) and smarter (as measured by the available instruction set) than individual GPU cores, the sheer number of GPU cores and the enormous amount of parallelism they provide more than compensate for the difference in clock speed and limited instruction set of a single core.

[0023] 1, CPU 10 may include dynamic random access memory (DRAM) 12, cache 14, control unit 16, and several, e.g., four, arithmetic logic units (ALUs) 18. In contrast, GPU 20 may include DRAM 12, several caches 24, several control units 26, and multiple ALUs 28. Thus, GPU architecture 20 differs from CPU architecture 10 in that multiple caches 24 and control units 26 are employed in GPU 20, and a significantly larger number of ALUs 28 are employed in GPU 20.

[0024] FIG. 2 illustrates an exemplary GPU memory according to one embodiment of the present invention.

[0025] GPUs have evolved into general-purpose parallel execution machines. The general model of stream computing is that each element in a set of streams, such as an ordered set of data, is processed by the same kernel, e.g., a set of functions, producing one or more streams as output. Each kernel is distributed to a set of GPU cores in the form of threads, each running the same program on a different set of data simultaneously. Threads are grouped into blocks and executed synchronously. In the event of a branch at runtime, the block is split into two, with all threads in the first branch executing in parallel, and eventually the same is done for all threads in the second branch. This general model of parallel execution is often referred to as single-instruction multiple-thread (SIMT) or single-program multiple-data (SPMD). Compared to the older single instruction, multiple data (SIMD), it allows for increased flexibility in the flow of different threads.

[0026] The GPU memory hierarchy is different from the CPU memory hierarchy. Generally, the GPU memory space can be categorized into the following groups: registers, constant memory, shared memory, texture memory, local memory, and global memory. In most GPU architectures, the on-board memory (also called device memory) is organized in the following hierarchy: global memory 50 accessible by all running threads 34, 44; shared memory 32, 42; faster cache memory dedicated to each single-thread block 30, 40; and local memory 36, 46 and / or registers dedicated to each thread 34, 44.

[0027] FIG. 3 illustrates an exemplary convolutional neural network (CNN) with a rectified linear unit (ReLU) layer, according to one embodiment of the present invention.

[0028] Convolutional neural networks (CNNs) are used in situations where data can be represented as a "map" where the proximity between two data points indicates their association. Images are such maps, which is why CNNs are used in the context of image analysis. If an image were taken and all of its pixels were randomly rearranged, the image would become unrecognizable. However, the relative positions of pixels to one another, i.e., their order, is important.

[0029] However, a CNN takes an image 62, represented as an array of numbers, applies a series of operations to that array, and finally returns the probability that an object in the image belongs to a particular object class. A CNN contains one or more of the following layers: a convolutional layer 64, a rectified linear unit (ReLU) layer 66, a pooling layer, a fully connected (FC) layer, and a loss layer (during the training process). A CNN processes an image 62 using a matrix of weights called a filter (or feature or feature map) that detects specific attributes, such as diagonal edges or vertical edges. Furthermore, as the image 62 passes through each layer, the filter can recognize more complex attributes.

[0030] A ReLU layer 66 typically follows the convolutional layer 64. The addition of the ReLU layer 66 allows the neural network to consider nonlinear relationships; for example, the ReLU layer 66 allows the CNN to consider situations where the relationship between pixel value input and CNN output is not linear. Note that the convolution operation is a linear operation; the output of a feature map is simply the sum of the weights of a given filter multiplied by the input pixel value.

[0031] The purpose of applying the ReLU function 66 is to enhance the nonlinearity in the image 62. The reason for doing so is that the image 62 is inherently nonlinear. When a person looks at an image, they notice that the image contains several nonlinear features (e.g., pixel transitions, boundaries, colors, etc.). The rectifier 66 serves to further resolve the linearity to compensate for any linearity that may be imposed on the image when a convolution operation is applied.

[0032] FIG. 4 is a diagram illustrating an exemplary deep learning CNN, according to one embodiment of the present invention.

[0033] CNN is sometimes called ConvNet, and its name comes from the "convolution operation." In the case of ConvNet, the "convolution" 64 is used to extract features from the input image 62. The convolution 64 preserves the spatial relationship between pixels by learning the image features of the input data. The output of the convolution layer 64 is a feature map.

[0034] The pooling layer 68 contributes to the CNN's ability to find features regardless of where they are located in the image. Specifically, the pooling layer 68 makes the CNN less sensitive to small changes in feature location; for example, the pooling layer 68 gives the CNN translation invariance, in that the output of the pooling layer 68 remains the same even if the feature moves slightly. Pooling also reduces the size of the feature map, thereby simplifying computations in later layers. There are several ways to implement pooling, but max pooling is the most effective in practice.

[0035] Flattening is the process of converting all the resulting two-dimensional arrays into a single long continuous linear vector.

[0036] The fully connected (FC) layer 70 is a conventional multilayer perceptron that uses a softmax activation function in the output layer. The term "fully connected" suggests that every neuron in the previous layer is connected to every neuron in the next layer. The FC layer 70 includes an input layer 72, a hidden layer 74, and an output layer 76. The outputs from the convolutional and pooling layers represent high-level features of the input image. The purpose of the FC layer 70 is to use these features to classify the input image 62 into various classes based on the training dataset. An exemplary embodiment aims to compress GPU memory by using compression techniques in the ReLU layer or rectifier 66.

[0037] FIG. 5 is a diagram illustrating an exemplary data structure of an uncompressed data input, according to one embodiment of the present invention.

[0038] Data structure 80 is an exemplary data structure presented for illustrative purposes only and is not intended to limit the present invention in any way. Data structure 80 may include, for example, two columns, which are divided into three blocks, for example, block 82, block 84, and block 86. Each row of the columns contains floating-point numbers. Half of the numbers may be zero and the other half may be non-zero. Each column may hold, for example, four bytes of data. Data structure 80 may be the original or initial uncompressed data structure.

[0039] Thus, in an exemplary embodiment of the present invention, assume that the output of a ReLU layer is assumed as input, and that an array of single-precision (e.g., 4-byte) floating-point numbers may be used. The size of the floating-point numbers may be, for example, 1, 2, 4, or 8 bytes. Due to the nature of the ReLU layer, approximately half of the floating-point numbers are (exact) zeros, and the other half are non-zero floating-point numbers. The goal of the exemplary embodiment is to eliminate or remove "0"s in the output and have a bitmap that records such locations. Thus, as described in more detail below with reference to FIG. 6, compression focuses on eliminating or removing such "zero data."

[0040] FIG. 6 is a diagram illustrating an exemplary data structure of compressed data, according to one embodiment of the present invention.

[0041] Before describing the modified data structure 90 with compressed data, we make a few observations. As shown in Figures 3 and 4 above, ReLU is a standard activation function for deep learning models. Most deep learning models have many ReLU functions 66. Half of the floating-point numbers in the output of a ReLU are zero. ReLU outputs "zero" for all negative inputs. While GPUs are a common architecture for deep learning, they have special limitations for algorithms that utilize GPU cores. To run effectively in practice, a high degree of parallelism is required. All threads must operate on the same program (or algorithm). Synchronization between threads and blocks must be minimized. Generally speaking, it is difficult for GPUs to compress data effectively because of the difficulty of parallelizing the construction of the compression dictionary.

[0042] The motivation for the exemplary embodiment of the present invention is to introduce lossless compression on the GPU that is specific to the output of the ReLU (as opposed to images or text) by eliminating or removing "0"s in the output and having a bitmap that records their location. The strategy is to design compression and decompression algorithms that include highly parallel algorithms for the GPU. The focus is on compressing / decompressing the output of the ReLU. The exemplary embodiment introduces algorithms that take advantage of the GPU architecture by combining several parallel algorithms used on the GPU.

[0043] 6, modified, adjusted, or compressed data structure 90 includes an "Original Data Length" block or region 92, a "Non-Zero Data Count Table" region 94, a "Non-Zero Data Bitmap" region 96, and a "Non-Zero Data Array Index Per Block" region 98 having block areas 100, 102, and 104. "Non-Zero Data Count Table" region 94 includes region 94' (the number of non-zeros in area 100), region 94'' (the number of non-zeros in area 102), region 94''' (the number of non-zeros in area 104), etc.

[0044] Data is compressed and decompressed using a combination of a parallel reduction algorithm and a parallel prefix-sum scan algorithm, allowing for efficient use of GPU cores. The data structure uses a simple, lossless compression format specific to the ReLU output (it does not support images or text). This constraint simplifies the compression algorithm and data structure, allowing a highly parallel algorithm to be used for compression. As a result, the compressed data structure 90 includes the following fields: an original data length 92 (optional), which is the length of the original data; a nonzero data count table 94, which is the number of nonzero data in each block; a nonzero data bitmap 96, which is a bitmap indicating the positions of nonzero data; and a nonzero data array 98, which is an array of nonzero data in the original data or uncompressed data.

[0045] The GPU cores are utilized by using a parallel reduction algorithm to generate a table of the number of non-zeros per block and a parallel prefix sum scan algorithm to calculate the starting index of the non-zero data array for every block. The parallel reduction algorithm can be, for example, an algorithm provided by NVIDIA(R). The parallel reduction works by using half the number of threads for the elements in the dataset. Every thread calculates the minimum between its element and the other elements. The resulting element is forwarded to the next round. The number of threads is then reduced by half, and the process is repeated until only one element remains, which becomes the result of the operation.

[0046] Regarding the compression algorithm, this method first generates a "non-zero data bitmap" region in parallel. Then, this method generates a "non-zero data count table" region using a parallel reduction algorithm. Next, this method calculates a "block-wise non-zero data array index" region for every block from the "non-zero data count table" using a parallel prefix sum scan algorithm. Then, this method allocates a buffer for the compressed data. Finally, this method copies non-zero data from the original data or initial input data to the "non-zero data array" region in parallel. Parallel prefix sum, also known as parallel scan, is a useful building block for many parallel algorithms, including sorting and constructing data structures. Both the parallel reduction algorithm and the parallel prefix sum scan algorithm are compatible with CUDA, a parallel computing platform and application programming interface model created by NVIDIA®. CUDA enables software developers and software engineers to use CUDA-enabled graphics processing units for general-purpose processing, an approach known as GPGPU.

[0047] Regarding the restoration algorithm, this method first allocates a buffer for the original data and then clears the buffer. Next, this method calculates the "non-zero data array index per block" field for all blocks from the "non-zero data count table" field using a parallel prefix sum scan algorithm. Finally, this method copies the "non-zero data array" field to the original data in parallel according to the "non-zero data bitmap" field.

[0048] Additionally, a "Non-Zero Data Bitmap" region 96 appears immediately above the "Non-Zero Data Array Index Per Block" region 98 within the compressed data structure 90. A "Non-Zero Data Count Table" region 94 appears immediately above the "Non-Zero Data Bitmap" region 96 within the compressed data structure 90. The "Non-Zero Data Bitmap" region 96 also sequentially displays the uncompressed data 200 (FIGS. 7 and 8) in binary format (1s and 0s). The sequence of binary digits extends horizontally from left to right within the "Non-Zero Data Bitmap" region 96. Additionally, blocks 94', 94'', and 94''' appear in a single row of the "Non-Zero Data Count Table" region 94. The areas or blocks 100, 102, and 104 can have different sizes, including rectangular and non-rectangular sizes.

[0049] FIG. 7 illustrates an exemplary compression algorithm methodology according to one embodiment of the present invention.

[0050] When the compression algorithm is executed, the uncompressed data 200 is converted into a bitmap 210, generating a "number of non-zero data" table 214, as indicated by arrow 225. The non-zero data in the uncompressed data 200 is labeled 202. The bitmap 210 represents the uncompressed data 200 in binary format, with "0" corresponding to all "zero data" and "1" corresponding to all positive non-zero data. The bitmap 210 is then mapped to a "non-zero data bitmap" 216. The "number of non-zero data" table 214 is formed by a single row and contains several boxes or areas, each of which indicates the number of non-zero data in each block. The first area 214' indicates four non-zero data in block #0, the second area 214'' indicates five non-zero data in block #1, and the third area 214'''' is being processed to determine the exact number of non-zero data in block #2. This is achieved using a parallel reduction algorithm. In the next step (step 3), a "non-zero data array index per block" 218 is calculated for every block from the "non-zero data count" table 214 using a parallel prefix sum scan algorithm. A buffer is then allocated for the compressed data. Finally, in step 5, the non-zero data 202 is copied from the original data (in the uncompressed data 200) to the "non-zero data array" 218. The compressed data structure further includes an "original data length" field 212 at the top of the compressed data structure. Field 212 can be optional.

[0051] FIG. 8 illustrates an exemplary restoration algorithm methodology, according to one embodiment of the present invention.

[0052] When the decompression algorithm is executed, a buffer is allocated from the original data and then cleared. In the next step, a parallel prefix sum scan algorithm is used to calculate the "non-zero data array index per block" for all blocks from the "non-zero data count table." Finally, in step 3, the "non-zero data array" is copied in parallel to the original data according to the "non-zero data bitmap" data. This is indicated by arrow 235. The first area 214' indicates four non-zeros (or non-zero entries) in block #0, the second area 214'' indicates five non-zeros (or non-zero entries) in block #1, and the third area 214'''' is being processed to determine the exact number of non-zeros in block #2. As a result, the data indicated as "zero data" or "0" is removed when assembling the compressed data structure shown in Figures 6-8.

[0053] In this way, the exemplary embodiment of the present invention supports both compression and decompression of the output of the ReLU layer, which is used in most DL models. The target hardware of the exemplary method is a GPU, which supports both the training phase and the inference shape. Furthermore, the exemplary method provides lossless compression and preserves the original DL model. Therefore, the exemplary method can better utilize the GPU with higher parallelism due to the nature of the output of the ReLU layer.

[0054] FIG. 9 is a block / flow diagram of a compression method for reducing GPU memory via rectified linear unit (ReLU) compression techniques, according to one embodiment of the present invention.

[0055] In block 910, "non-zero data bitmap" regions are generated in parallel.

[0056] At block 920, a parallel reduction algorithm is used to generate a "non-zero data count table."

[0057] In block 930, the "non-zero data array index per block" field of every block is calculated from the "non-zero data count table" using a parallel prefix sum scan algorithm.

[0058] At block 940, a buffer is allocated for the compressed data.

[0059] In block 950, the non-zero data is copied in parallel from the original data to a "non-zero data array" in the compressed data format.

[0060] FIG. 10 is a block / flow diagram of a decompression method for reducing GPU memory via rectified linear unit (ReLU) compression techniques, according to one embodiment of the present invention.

[0061] Block 1010 allocates a buffer from the original data and clears the buffer.

[0062] In block 1020, the "non-zero data array index per block" field of every block is calculated from the "non-zero data count table" using a parallel prefix sum scan algorithm.

[0063] In block 1030, the "non-zero data array" is copied in parallel from the original data according to the "non-zero data bitmap."

[0064] Therefore, in conclusion, a system is presented where the compressed data contains the output of the ReLU function on a GPU, including the "original data length", a "non-zero data count table" indicating the number of non-zero data per block, a "non-zero data bitmap" indicating the non-zero data locations, and a "non-zero data array" of non-zero data from the original data.

[0065] The compression system generates a "non-zero data bitmap" in parallel, generates a "non-zero data count table" using a parallel reduction algorithm, calculates "block-wise non-zero data array indexes" for all blocks from the non-zero data count table using a parallel prefix sum scan algorithm, allocates buffers for compressed data, and copies non-zero data from the original data to the "non-zero data array" in parallel.

[0066] The restoration system allocates a buffer for the original data, clears the buffer, calculates the "non-zero data array index per block" for every block from the "non-zero data count table" using a parallel prefix sum scan algorithm, and copies the "non-zero data array" from the original data in parallel according to the "non-zero data bitmap" data.

[0067] As a result, exemplary embodiments of the present invention reduce GPU memory, and several methods can be selected independently for each layer. One method uses ReLU compression to reduce GPU memory by compressing the feature maps generated by the ReLU layer. Another exemplary embodiment uses ReLU compression in conjunction with a recalculation method to compress GPU memory by compressing and storing the feature maps generated by the ReLU layer for the starting point of recalculation and discarding the other feature maps. This method can reduce the GPU memory size of the stored feature maps using the ReLU compression method. In yet another exemplary embodiment, ReLU compression is used in conjunction with a data swapping method to compress GPU memory by compressing the feature maps generated by the ReLU layer on the GPU before transferring them to CPU memory. This effectively reduces the data transfer time from the GPU to the CPU. This exemplary method transfers data from the CPU to the GPU and then restores the data.

[0068] FIG. 11 is a block / flow diagram of a computational method for reducing GPU memory via rectified linear unit (ReLU) compression techniques, according to one embodiment of the present invention.

[0069] 11 shows a block diagram of an apparatus 1100 for implementing one or more of the methodologies presented herein.

[0070] The apparatus 1100 includes a computer system 1110 and removable media 1150. The computer system 1110 includes a CPU device and a GPU device, collectively referred to as 1120, a network interface 1125, memory 1130, a media interface 1135, and an optional display 1140. The network interface 1125 allows the computer system 1110 to connect to a network, and the media interface 1135 allows the computer system 1110 to interact with media, such as a hard drive or removable media 1150.

[0071] The CPU / GPU 1120 can be configured to implement the methods, steps, and functions disclosed herein. The memory 1130 can be distributed or local, and the processor CPU / GPU 1120 can be distributed or unitary. The memory 1130 can be implemented as electrical, magnetic, or optical memory, or any combination of these or other types of storage devices. Furthermore, the term "memory" should be interpreted broadly enough to encompass any information that can be read from or written to an address within an addressable space accessed by the CPU / GPU 1120. By this definition, information on a network accessible via the network interface 1125 remains in the memory 1130 because the processor unit 1120 can retrieve that information from the network. Note that each distributed processor comprising the CPU / GPU 1120 generally includes its own addressable memory space. Note also that part or all of the computer system 1110 can be incorporated into an application-specific integrated circuit or a general-purpose integrated circuit.

[0072] Optional display 1140 is any type of display suitable for interacting with a human user of device 1100. Typically, display 1140 is a computer monitor or other similar display.

[0073] FIG. 12 is a block / flow diagram of an exemplary cloud computing environment according to one embodiment of the present invention.

[0074] Although the present invention includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings recited herein is not limited to cloud computing environments. Rather, embodiments of the present invention may be implemented in conjunction with any other type of computing environment now known or later developed.

[0075] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with a service provider. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0076] The features are as follows:

[0077] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, automatically as needed, without requiring human interaction with the provider of the service.

[0078] Broad Network Access: Capabilities are available over the network and are accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0079] Resource Pooling: Pooling a provider's computing resources to serve multiple consumers using a multi-tenant model where various physical and virtual resources are dynamically allocated and reallocated according to demand. The consumer generally has no control over or knowledge of the exact location of the resources provided, although there is a sense of location independence in that the location may be identifiable at a higher level of abstraction (e.g., country, state, or data center).

[0080] Rapid Elasticity: Capacity can be rapidly and elastically provisioned, sometimes automatically, to quickly scale out and rapidly release to quickly scale in. To the consumer, provisionable capacity often appears unlimited and can be purchased in any quantity at any time.

[0081] Metered Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, managed, and reported to provide transparency to both providers and consumers of utilized services.

[0082] The service model is as follows:

[0083] Software as a Service (SaaS): The consumer is offered the ability to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application features, with the possible exception of limited user-specific application configuration settings.

[0084] Platform as a Service (PaaS): The ability offered to consumers is to deploy applications they create or acquire, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but does control the deployed applications and, in some cases, the application hosting environment configuration.

[0085] Infrastructure as a Service (IaaS): The ability offered to consumers is to provision processing, storage, network, and other basic computing resources onto which they can deploy and run any software, which can include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they do have control over the operating systems, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0086] The deployment model is as follows:

[0087] Private Cloud: Cloud infrastructure is operated exclusively for an organization. It can be managed by the organization or a third party and can reside on-premise or off-premise.

[0088] Community Cloud: Cloud infrastructure is shared by several organizations to support a specific community with common concerns (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by the organization or a third party and can reside on-premise or off-premise.

[0089] Public Cloud: Cloud infrastructure is made available to the general public or large industry groups and is owned by an organization that sells cloud services.

[0090] Hybrid Cloud: A cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain unique entities but are joined by standardized or proprietary technologies that allow for data and application portability (e.g., cloud bursting for load balancing between clouds).

[0091] Cloud computing environments are service-oriented with an emphasis on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0092] Referring now to FIG. 12 , an exemplary cloud computing environment 1250 for implementing use cases of the present invention is shown. As shown, the cloud computing environment 1250 includes one or more cloud computing nodes 1210, with which local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or mobile phone 1254A, a desktop computer 1254B, a laptop computer 1254C, or an automotive computer system 1254N, or combinations thereof, can communicate. The nodes 1210 can communicate with each other. They can be grouped physically or virtually in one or more networks (not shown), such as, for example, the private, community, public, or hybrid clouds described above, or combinations thereof. This enables the cloud computing environment 1250 to provide infrastructure-as-a-service, platform-as-a-service, or software-as-a-service, or combinations thereof, without requiring cloud consumers to maintain resources on their local computing devices. It should be understood that the types of computing devices 1254A-N shown in FIG. 12 are intended to be exemplary only, and that the computing node 1210 and cloud computing environment 1250 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0093] Figure 13 is a schematic diagram of exemplary abstraction model layers according to one embodiment of the present invention. It should be understood in advance that the components, layers, and functions illustrated in Figure 13 are intended to be exemplary only, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0094] Hardware and software layer 1360 includes hardware and software components. Examples of hardware components include mainframe 1361, RISC (reduced instruction set computer) architecture-based server 1362, server 1363, blade server 1364, storage device 1365, and network and networking components 1366. In some embodiments, software components include network application server software 1367 and database software 1368.

[0095] The virtualization layer 1370 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 1371, virtual storage 1372, virtual networks including virtual private networks 1373, virtual applications and operating systems 1374, and virtual clients 1375.

[0096] In one example, management layer 1380 may provide the following functionality: Resource provisioning 1381 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 1382 provides cost tracking as resources are utilized within the cloud computing environment and accounting or billing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal 1383 provides consumers and system administrators with access to the cloud computing environment. Service level management 1384 provides allocation and management of cloud computing resources so that requested service levels are met. Service level agreement (SLA) planning and fulfillment 1385 provides advance arrangement and procurement of cloud computing resources anticipated for future requests according to SLAs.

[0097] The workload layer 1390 provides examples of functionality that can utilize a cloud computing environment. Examples of workloads and functions that can be provided from this layer include mapping and navigation 1391, software development and lifecycle management 1392, virtual classroom instruction delivery 1393, data analytics processing 1394, transaction processing 1395, and ReLU feature map compression 1396 on cloud servers.

[0098] As used herein, the terms "data," "content," "information," and similar terms may be used interchangeably to refer to data that may be captured, transmitted, received, displayed, or stored, or any combination thereof, according to various exemplary embodiments. Accordingly, the use of any such terms should not be construed to limit the scope of the present disclosure. Furthermore, when a computing device is described herein as receiving data from another computing device, the data may be received directly from the other computing device or may be received indirectly through one or more intermediate computing devices, such as one or more servers, relays, routers, network access points, base stations, or the like, or any combination thereof.

[0099] To provide for user interaction, embodiments of the subject matter described herein can be implemented on a computer having, for example, a display device such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0100] The present invention may be a system, a method, and / or a computer program product, which may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0101] A computer-readable storage medium may be any tangible device capable of retaining and storing instructions for use by an instruction-execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick®, floppy® disk, mechanically encoded devices such as punch cards or grooved ridge structures having instructions recorded thereon, and any suitable combination thereof. Computer-readable storage medium, as used herein, should not be construed as a transitory signal per se, such as, for example, radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through fiber optic cable), or electrical signals transmitted over wires.

[0102] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network can include copper transmission cables, fiber optic transmission cables, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0103] The computer-readable program instructions for carrying out the operations of the present invention may be source or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, C++, and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the last scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may provide a connection to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can be personalized by utilizing state information of the computer readable program instructions to execute the computer readable program instructions to perform aspects of the present invention.

[0104] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0105] These computer-readable program instructions can be provided to at least one processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when the instructions are executed by the processor of the computer or other programmable data processing apparatus, means are generated for implementing the functions / acts specified in one or more blocks or modules of the flowcharts and / or block diagrams. The computer-readable program instructions can also be stored on a computer-readable storage medium capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions that implement aspects of the functions / acts specified in one or more blocks or modules of the flowcharts and / or block diagrams.

[0106] Furthermore, computer-readable program instructions can be loaded into a computer, other programmable data processing apparatus, or other device and caused to execute a series of operational blocks / steps on the computer, other programmable apparatus, or other device to generate a computer-implemented process that, when executed on the computer, other programmable apparatus, or other device, implements the functions / acts specified in one or more blocks or modules of the flowcharts and / or block diagrams.

[0107] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order depicted. For example, depending on the functionality involved, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may possibly be executed in the reverse order. It will also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or acts or executes a combination of dedicated hardware and computer instructions.

[0108] References herein to "one embodiment" or "an embodiment" of the present principles, as well as other variations thereof, mean that the particular features, structures, characteristics, etc. described in connection with that embodiment are included in at least one embodiment of the present principles. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment," as well as any other variations thereof, appearing in various places throughout this specification are not necessarily all referring to the same embodiment.

[0109] For example, it should be understood that the use of " / ," "and / or," and "at least one of" in the cases of "A / B," "A and / or B," and "at least one of A and B" is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, in the cases of "A, B, and / or C" and "at least one of A, B, and C," such phrases are intended to encompass the selection of only the first listed alternative (A), the selection of only the second listed alternative (B), the selection of only the third listed alternative (C), the selection of only the first and second listed alternatives (A and B), the selection of only the first and third listed alternatives (A and C), the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A, B, and C). This can be expanded as many times as the number of listed items, as would be readily apparent to one of ordinary skill in this and related arts.

[0110] Having described preferred embodiments of a system and method for compressing rectified linear unit (ReLU) function data on a graphics processing unit (GPU) used in the training process of a deep neural network (which are intended to be illustrative and not limiting), it should be noted that modifications and variations will occur to those skilled in the art in light of the above teachings. It is therefore understood that changes may be made in the particular embodiments described that are within the scope of the invention as broadly set forth by the appended claims. Thus, having described aspects of the invention with the detail and particularity required by the Patent Laws, what is claimed and desired to be protected by Letters Patent is set forth in the appended claims.

Claims

1. 1. A method for compressing rectified linear unit (ReLU) function data on a graphics processing unit (GPU) used in a training process of a deep neural network, comprising: converting an initial data structure containing non-zero data and zero data into a compressed data structure containing only the non-zero data of the initial data structure as compressed data; wherein said converting comprises: generating a non-zero data bitmap region; generating non-zero data count table regions in parallel using a parallel reduction algorithm; calculating a non-zero data array index field for each block of all blocks from the non-zero data number table field in parallel using a parallel prefix sum scan algorithm; allocating a buffer for the compressed data; copying the non-zero data from the initial data structure in parallel to a non-zero data array region in compressed data format; A method carried out by.

2. 2. The method of claim 1, wherein the compressed data is restored by allocating the buffer from the non-zero data and zero data of the initial data structure and then clearing the buffer.

3. 3. The method of claim 2, wherein the restoring further comprises calculating the non-zero data array index areas for all blocks from the non-zero data number table in parallel processing using the parallel prefix sum scan algorithm.

4. 4. The method of claim 3, wherein said restoring further comprises copying said non-zero data array from said initial data structure to said non-zero data bitmap area in parallel.

5. 2. The method of claim 1, wherein the non-zero data bitmap area is displayed in the compressed data structure immediately above the per-block non-zero data array index area.

6. 2. The method of claim 1, wherein the non-zero data count table region is displayed in the compressed data structure immediately above the non-zero data bitmap region.

7. 2. The method of claim 1, wherein the non-zero data bitmap regions represent uncompressed data sequentially in binary format.

8. 1. A non-transitory computer-readable storage medium containing a computer-readable program for compressing rectified linear unit (ReLU) function data on a graphics processing unit (GPU) used in a training process of a deep neural network, the computer-readable program, when executed on a computer, causing the computer to: converting an initial data structure containing non-zero data and zero data into a compressed data structure containing only the non-zero data of the initial data structure as compressed data; and said converting comprises: generating a non-zero data bitmap region; generating non-zero data count table regions in parallel using a parallel reduction algorithm; calculating a non-zero data array index field for each block of all blocks from the non-zero data number table field in parallel using a parallel prefix sum scan algorithm; allocating a buffer for the compressed data; copying the non-zero data from the initial data structure in parallel to a non-zero data array region in compressed data format; A non-transitory computer-readable storage medium performed by

9. 9. The non-transitory computer-readable storage medium of claim 8, wherein the compressed data is restored by allocating the buffer from the non-zero data and zero data of the initial data structure and then clearing the buffer.

10. 10. The non-transitory computer-readable storage medium of claim 9, wherein the restoring further comprises calculating the per-block non-zero data array index areas of all blocks from the non-zero data number table in parallel processing using the parallel prefix sum scan algorithm.

11. 11. The non-transitory computer-readable storage medium of claim 10, wherein the restoring further comprises copying the non-zero data array from the initial data structure to the non-zero data bitmap area in parallel.

12. 9. The non-transitory computer-readable storage medium of claim 8, wherein the non-zero data bitmap area appears in the compressed data structure immediately above the per-block non-zero data array index area.

13. 9. The non-transitory computer-readable storage medium of claim 8, wherein the non-zero data count table region appears in the compressed data structure immediately above the non-zero data bitmap region.

14. 9. The non-transitory computer-readable storage medium of claim 8, wherein the non-zero data bitmap regions sequentially represent uncompressed data in binary format.

15. 1. A system for compressing rectified linear unit (ReLU) function data on a graphics processing unit (GPU) used in a training process of a deep neural network, comprising: Memory and one or more processors in communication with the memory; wherein the one or more processors: converting an initial data structure containing non-zero data and zero data into a compressed data structure containing only the non-zero data of the initial data structure as compressed data; wherein the converting comprises: generating a non-zero data bitmap region; generating non-zero data count table regions in parallel using a parallel reduction algorithm; calculating a non-zero data array index field for each block of all blocks from the non-zero data number table field in parallel using a parallel prefix sum scan algorithm; allocating a buffer for the compressed data; copying the non-zero data from the initial data structure in parallel to a non-zero data array region in compressed data format; The system is carried out by

16. 16. The system of claim 15, wherein the compressed data is restored by allocating the buffer from the non-zero data and zero data of the initial data structure and then clearing the buffer.

17. 17. The system of claim 16, wherein the restoring further comprises calculating the non-zero data array index areas for all blocks from the non-zero data number table in parallel processing using the parallel prefix sum scan algorithm.

18. 18. The system of claim 17, wherein the restoring further comprises copying the non-zero data array from the initial data structure to the non-zero data bitmap area in parallel.

19. 16. The system of claim 15, wherein the non-zero data bitmap area appears in the compressed data structure immediately above the per-block non-zero data array index area.

20. the non-zero data count table region is displayed in the compressed data structure immediately above the non-zero data bitmap region; the non-zero data bitmap region sequentially represents uncompressed data in binary format; 16. The system of claim 15.

Images