Sparsity-Based Feature Reordering for Improved Memory Compression and Transfer During Machine Learning Jobs

Abstract

A processing device for performing a machine learning neural network operation includes a memory and a processor configured to receive input data at a layer of the machine learning neural network operation, receive a plurality of sorted filters to be applied to the input data, apply the plurality of sorted filters to the input data to generate a plurality of distinct feature maps, compress the plurality of distinct feature maps according to sparsity of the feature maps, and store the plurality of distinct feature maps in a memory.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Patent Application No. 16 / 836,741, filed March 31, 2020, the contents of which are incorporated herein by reference. [Background technology]

[0002] Machine learning (e.g., deep learning) is widely used in various technologies (e.g., image classification) to make predictions or decisions to perform specific tasks (e.g., whether an image contains a specific object). Convolutional neural networks (CNNs) are a class of deep learning algorithms widely used in machine learning applications. These networks typically contain multiple layers. In each layer, a set of filters is applied to the output of the previous layer, and the output of each layer is known as activations or feature maps. The first and last layers in a network are known as the input layer and output layer, respectively, and the layers between the first and last layers are typically known as hidden layers.

[0003] In supervised learning, machine learning models are trained to make predictions or decisions to perform a specific task (e.g., whether an image contains a specific object). During training, the model is exposed to different data. At each layer, the model transforms the data and receives feedback on the accuracy of its operation. During the inference phase, the trained model is used to infer or predict outputs for test samples (e.g., input tensors).

[0004] A more detailed understanding may be had from the following description, given by way of example in conjunction with the accompanying drawings, in which: [Brief explanation of the drawings]

[0005] [Figure 1] FIG. 1 is a block diagram of an example device capable of implementing one or more features of the present disclosure. [Figure 2] FIG. 2 is a block diagram of the device of FIG. 1 showing further details. [Figure 3] FIG. 10 illustrates a storage layout of example activation tensor values ​​according to NHWC formatting before being sorted into memory, in accordance with aspects of the present disclosure. [Figure 4] FIG. 4 illustrates an exemplary sorting of the feature maps shown in FIG. 3 according to feature map similarity, and an exemplary memory layout of tensor values ​​stored in memory using NHWC formatting by sorting. [Figure 5] FIG. 1 is a flow diagram illustrating an example method for performing machine learning operations in accordance with aspects of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0006] The terms activation and feature map are used interchangeably in this disclosure. CNNs are used in different types of technical applications. For simplified explanation, the examples described herein include CNNs for image analysis.

[0007] The (full or partial) activations of a CNN model are written to and read from memory for each layer or for multiple layers, depending on the particular application. The output of each layer is, for example, a four-dimensional (4D) activation tensor containing a set of images divided into N batches of feature maps (i.e., channels), C, each representing an image and each having a size defined by height (H) and width (W). The activation tensor is subjected to an operation defined by the layer (e.g., a convolution kernel, a pooling operation), resulting in a new activation tensor for the next layer.

[0008] Deep learning models typically use large memory bandwidth, which can lead to bandwidth bottlenecks, adversely affect performance, and increase power consumption. The amount of memory used to store activation tensor data for different layers of a machine learning neural network is typically so large that, depending on the application, the activation tensor data cannot be stored in on-chip memory. Therefore, storing the activation tensor data involves transferring data to and from off-chip memory.

[0009] The sparsity of a group of elements (e.g., elements of a feature map) is measured by the amount of zero values ​​in the group of elements. Increasing data sparsity typically results in an increase in the data compression ratio (e.g., uncompressed data size / compressed data size) because zero values ​​of data can be transmitted with less information than non-zero values. The sparsity of data within the resulting feature map (i.e., channel) typically varies between feature maps. Thus, two adjacent channels may have different levels of sparsity. Typically, there is no inherent pattern of sparsity resulting in data for a typical machine learning neural network model.

[0010] The present application provides a processing device and method for efficiently compressing tensors for memory transfer during the inference stage of a machine learning model by applying filters to sorted input tensors during the training stage before the inference stage. The filters are sorted according to the sparsity of the feature map (i.e., the feature map resulting from applying the sorted filters to input data), so that the amount of consecutively stored values ​​(e.g., zero values ​​or small values ​​less than or equal to a threshold value) of interest in the feature map (i.e., stored in consecutive locations in memory) is greater than the amount of consecutively stored values ​​of interest in the feature map without sorting the filters according to sparsity.

[0011] The sparsity of a group of elements may be defined, for example, according to "full sparsity" (i.e., where each element in the group is zero-valued). For example, the filters may be ordered according to sparse groups (i.e., where each element in the group is zero-valued) and non-sparse groups (i.e., where the group contains at least one non-zero value).

[0012] Alternatively, the filters are sorted (i.e., ordered) according to their corresponding "partial sparsity." When partial sparsity is used, the sorting of the filters is determined, for example, according to the number (or range of numbers) of zero values ​​in a group of elements (e.g., a feature map), the number of consecutive zero values ​​in a group of elements, the percentage (or range of percentages) or ratio of zero values ​​to the total number of values ​​in a group of elements, or another metric. The sorting is also determined, for example, based on a comparison of the sparsity of the feature map with a sparsity threshold. The filters can also be ordered according to the level of sparsity in a group of elements.

[0013] Tensor data can be written to memory in different formats, such as NHWC (i.e., channel first) or NCHW (i.e., width first). In NHWC (or other channel-first memory layouts), collocated elements of co-located channels will be adjacent in memory.

[0014] The features of the present disclosure may be implemented via any of several different types of compression algorithms, such as, for example, delta-based compression algorithms, dictionary-based compression algorithms, frequency-based approaches, and run-length encoding.

[0015] For example, run-length encoding is a form of lossless data compression in which runs of data (sequences in which the same data value occurs in many consecutive data elements) are stored as a single data value and count rather than as the original run of data. Thus, data is compressed more efficiently if a greater number of zero values ​​are stored contiguously in memory because, by sorting, a greater number of zero values ​​are stored contiguously in memory, allowing a greater number of zero runs to be stored as a single data value, thereby reducing the number of memory transfers.

[0016] The filter can also sort by sparsity, based on the amount of values ​​in a group of elements that are less than or equal to a value threshold. For example, if lossy compression is used, small values ​​(i.e., values ​​less than or equal to a value threshold) are effectively treated as zero values.

[0017] A processing device for performing a machine learning neural network operation is provided, the processing device including a memory and a processor, wherein the processor is configured to: receive input data at a layer of the machine learning neural network operation, receive a plurality of sorted filters to be applied to the input data, apply the plurality of sorted filters to the input data to generate a plurality of distinct feature maps, compress the plurality of distinct feature maps according to sparsity of the feature maps, and store the plurality of distinct feature maps in a memory.

[0018] A machine learning processing method is provided, comprising: receiving, at a layer of a machine learning neural network, input data; receiving a plurality of sorted filters to be applied to the input data; applying the plurality of sorted filters to the input data to generate a plurality of distinct feature maps; compressing the plurality of distinct feature maps according to sparsity of the feature maps; and storing the plurality of distinct feature maps in a memory.

[0019] A non-transitory computer-readable storage medium including instructions stored thereon is provided, the instructions causing a computer to perform a machine learning processing method, the method including, at a layer of a machine learning neural network, receiving input data; receiving a plurality of sorted filters to be applied to the input data; applying the plurality of sorted filters to the input data to generate a plurality of distinct feature maps; compressing the plurality of distinct feature maps according to sparsity of the feature maps; and storing the plurality of distinct feature maps in a memory.

[0020] 1 is a block diagram of an example device 100 capable of implementing one or more features of the present disclosure. Device 100 may include, for example, a computer, a gaming device, a handheld device, a set-top box, a television, a mobile phone, or a tablet computer. Device 100 includes a processor 102, a memory 104, a storage device 106, one or more input devices 108, and one or more output devices 110. Device 100 may also optionally include an input driver 112 and an output driver 114. It should be understood that device 100 may include additional components not shown in FIG. 1 .

[0021] In various alternatives, processor 102 includes a central processing unit (CPU), a graphics processing unit (GPU), a CPU and a GPU located on the same die, or one or more processor cores, each of which may be a CPU or a GPU or a standalone accelerator. In various alternatives, memory 104 may be located on the same die as processor 102 or may be located separately from processor 102. Memory 104 may include volatile or non-volatile memory (e.g., random access memory (RAM), dynamic RAM, cache).

[0022] The storage devices 106 include fixed or removable storage devices (e.g., hard disk drives, solid state drives, optical disks, flash drives). The input devices 108 include, but are not limited to, a keyboard, a keypad, a touch screen, a touch pad, a detector, a microphone, an accelerometer, a gyroscope, a biometric scanner, or a network connection (e.g., a wireless local area network card for transmitting and / or receiving wireless IEEE 802 signals). The output devices 110 include, but are not limited to, a display, a speaker, a printer, a haptic feedback device, one or more optics, an antenna, or a network connection (e.g., a wireless local area network card for transmitting and / or receiving wireless IEEE 802 signals).

[0023] Input driver 112 communicates with processor 102 and input device 108, allowing processor 102 to receive input from input device 108. Output driver 114 communicates with processor 102 and output device 110, allowing processor 102 to send output to output device 110. Note that input driver 112 and output driver 114 are optional components, and that device 100 may operate similarly without input driver 112 and output driver 114. Output driver 114 includes an accelerated processing device ("APD") 116 coupled to display device 118. The APD accepts computational and graphics rendering commands from processor 102, processes those computational and graphics rendering commands, and provides output to display device 118 for display. As described in more detail below, APD 116 includes one or more parallel processing units that perform computations according to the single-instruction-multiple-data ("SIMD") paradigm. Although various functions are described herein as being performed by or in conjunction with APD 116, in various alternatives, the functions described as being performed by APD 116 may additionally or alternatively be performed by other computing devices having similar capabilities that are not driven by a host processor (e.g., processor 102) to provide graphical output to display device 118. For example, it is contemplated that any processing system that performs processing tasks according to the SIMD paradigm may perform the functions described herein. Alternatively, it is contemplated that computing systems that do not perform processing tasks according to the SIMD paradigm may perform the functions described herein.

[0024] FIG. 2 is a block diagram of device 100, showing additional details regarding the execution of processing tasks on APD 116. Processor 102 maintains, within system memory 104, one or more control logic modules for execution by processor 102. The control logic modules include operating system 120, kernel-mode driver 122, and applications 126. These control logic modules control various aspects of the operation of processor 102 and APD 116. For example, operating system 120 communicates directly with hardware and provides an interface to the hardware for other software executing on processor 102. Kernel-mode driver 122 controls the operation of APD 116, for example, by providing an application programming interface (API) to software executing on processor 102 (e.g., applications 126) to access various features of APD 116. Kernel-mode driver 122 also includes a just-in-time compiler that compiles programs for execution by processing components of APD 116 (such as SIMD unit 138, described in more detail below).

[0025] APD 116 executes commands and programs for selected functions, such as graphics and non-graphics operations that may be suitable for parallel processing. APD 116 may be used to perform graphics pipeline operations, such as pixel operations, geometric calculations, and rendering of images to display device 118, based on commands received from processor 102. APD 116 also performs computational operations not directly related to graphics operations, such as operations related to video, physics simulations, computational fluid dynamics, or other tasks, based on commands received from processor 102.

[0026] The APD 116 includes a computation unit 132 that includes one or more SIMD units 138 that, at the request of the processor 102, perform operations in a parallel manner according to the SIMD paradigm. The SIMD paradigm allows multiple processing elements to share a single program control flow unit and program counter, thus executing the same program but with different data. In one example, each SIMD unit 138 includes 16 lanes, each of which executes the same instruction simultaneously with other lanes in the SIMD unit 138 but can execute the instruction with different data. Lanes can be predictively switched off if not all lanes need to execute a given instruction. Prediction can also be used to execute programs with branching control flow. More specifically, for programs with conditional branches or other instructions where control flow is based on calculations performed by individual lanes, prediction of lanes corresponding to currently unexecuted control flow paths and serial execution of different control flow paths allows for arbitrary control flow.

[0027] The basic unit of execution within the compute unit 132 is the work item. Each work item represents a single instantiation of a program executing in parallel on a particular lane. Work items can execute simultaneously as a “wavefront” on a single SIMD unit 138. One or more wavefronts are included in a “workgroup,” which contains a collection of work items designated to execute the same program. A workgroup can be executed by executing each of the wavefronts that make up the workgroup. Alternatively, a wavefront executes serially on a single SIMD unit 138, or partially or fully in parallel on different SIMD units 138. A wavefront can be thought of as the largest collection of work items that can execute simultaneously on a single SIMD unit 138. Thus, if a command received from the processor 102 indicates that a particular program is parallelized to an extent that it cannot be executed simultaneously on a single SIMD unit 138, the program is divided into wavefronts that are either parallelized on two or more SIMD units 138, or serialized on the same SIMD unit 138 (or both parallelized and serialized, as appropriate). The scheduler 136 performs operations related to scheduling the various wavefronts on the different compute units 132 and SIMD units 138 .

[0028] The parallel processing provided by the computation units 132 is well suited to graphics-related operations such as pixel value calculations, vertex transformations, and other graphics operations. Thus, in some cases, the graphics processing pipeline 134, which accepts graphics processing commands from the processor 102, provides computational tasks to the computation units 132 for execution in parallel.

[0029] Computation unit 132 is also used to perform computational tasks that are not related to graphics or that are not performed as part of the "normal" operation of graphics processing pipeline 134 (e.g., custom operations performed to supplement the operations performed on graphics processing pipeline 134). Applications 126 or other software executing on processor 102 send programs defining such computational tasks to APD 116 for execution.

[0030] The APD 116 is configured to execute machine learning models, including deep learning models. The APD 116 is configured to store activation tensor data for different layers of a machine learning neural network. At each layer, the APD 116 is configured to perform operations (e.g., convolution kernels, pooling operations) on input data (e.g., images, activation tensors) of the previous layer, apply filters to the input data, and provide tensor data for the next layer.

[0031] As described above, the amount of memory used to store activation tensor data for different layers of a neural network is typically so large (e.g., in early layers) that the activation tensor data cannot be stored in on-chip memory (e.g., memory of the APD 116). Therefore, storing the activation tensor data involves transferring data between the APD 116 and off-chip memory (e.g., memory 104) over a link (e.g., a bus). The APD 116 is configured to compress (e.g., save bandwidth) the data transferred to the off-chip memory.

[0032] APD 116 is configured to compress tensor values ​​by changing the order in which tensor values ​​are stored according to any of a plurality of feature map sparsity metrics, using any of a plurality of different types of memory formats having a first configuration of channels, and using any of a plurality of types of compression algorithms. For simplified explanation, the embodiments described herein include delta-based compression of 4D tensor values ​​by changing the order in which tensor values ​​are written to memory according to an NHWC (i.e., channel-first) format based on the sparsity of the feature maps.

[0033] FIG. 3 is a diagram illustrating the storage layout of an example 4D activation tensor value in NHWC formatting before sorting in memory in accordance with features of the present disclosure.

[0034] In NHWC, activation tensors (e.g., 4D activation tensors) are first stored in channels. For example, a 4D activation tensor is written to memory by mapping each 4D tensor value through an offset function that inputs a logical index (n, h, w, c) and returns the address displacement where each value is located. Thus, two tensor values ​​stored adjacently in memory share the same primary indexes n, h, w, but a different index w (e.g., the index w of the second tensor value differs from the first tensor value by 1). Uppercase letters represent the four dimensions of the activation tensor (i.e., N, H, W, C), and lowercase letters represent the indexes for each dimension (i.e., n, h, w, c).

[0035] For example, when the NHWC format is used, to store tensor values ​​for multiple feature maps, each representing an activation, the element in the first location (e.g., the value in row 1, column 1) of each feature map is stored in memory first, then the element in the second location (e.g., the value in row 1, column 2) of each feature map is stored, and so on until each of the elements for each batch has been stored in memory.

[0036] The activation tensor shown in Figure 3 includes eight feature maps 302 (i.e., eight channels), each of which is a 2x2 element matrix. The dimensions of the feature maps shown in Figure 3 are merely examples. Features of the present disclosure may be implemented using any number of feature maps (i.e., channels) with dimensions different from those shown in Figure 3 (i.e., rows of width W and columns of height H).

[0037] Each feature map 302 is a different representation of the input tensor to which a different filter (e.g., weight) is applied. For example, the input tensor is subjected to an operation (e.g., a convolution kernel, pooling operation) using a first filter to produce a first feature map 302 (C0) containing element values ​​00, 01, 02, and 03. The input tensor is then subjected to an operation using a second filter to produce a second feature map 302 (C1) containing element values ​​04, 05, 06, and 07. The process continues with different filters to produce each of the feature maps 302 (C0-C7).

[0038] 3 also illustrates an exemplary memory layout showing where each element value is stored in accordance with NHWC formatting in memory portion 304 without sorting (i.e., reordering) the element values ​​in memory in accordance with features of the present disclosure. As illustrated, a first element 00 of a first feature map 302(C0) is stored in a first location in memory portion 304. A co-located first element 04 in a second feature map 302(C1) is then stored in a second location in memory portion 304 adjacent to the first element 00 of the first feature map 302(C0).

[0039] After each of the collocated first elements (i.e., 08, 12, 16, 20, 24, 28) of the remaining feature maps 302 (C2-C7) are stored in the next location in the memory portion 304, the second element 01 (along the width W from element 00) of the first feature map 302 (C0) is stored, followed by the collocated second element 05 of the second feature map 302 (C1).

[0040] After each of the collocated second elements (i.e., 09, 13, 17, 21, 25, 29) of the remaining feature maps 302 (C2-C7) are stored in the next location in memory portion 304, element 02 (along height H from element 00) of the first feature map 304 (C0) is stored, followed by collocated element 06 of the second feature map 302 (C1), and then each of the collocated elements (i.e., 10, 14, 18, 22, 26, 30) of the remaining feature maps 302 (C2-C7) are stored in the next location in memory portion 304.

[0041] After element 30 is stored, element 03 of the first feature map 304 (C0) is stored, followed by co-located element 07 of the second feature map 302 (C1), followed by the remaining co-located elements (11, 15, 19, 23, 27, 31) in the memory portion 304.

[0042] As mentioned above, the efficiency of tensor value compression (e.g., delta-based compression) depends on the sparsity between adjacent data stored in memory, for example.

[0043] 4 shows an example of how the feature map 302 shown in FIG. 3 is sorted according to the sparsity of the feature map during the training phase, and an example of the memory layout of element values ​​stored in memory using the NHWC format according to the sorting. That is, the channels (i.e., filters) are sorted according to the sparsity of the feature map during the training phase, so that the amount of consecutively stored zero values ​​in the feature map (i.e., stored in consecutive locations in memory) is greater than the amount of consecutively stored zero values ​​in the feature map without sorting the channels according to sparsity. Because the channels are sorted according to sparsity, the number of memory transfers performed to execute the model during the inference phase is reduced (i.e., memory bandwidth is reduced).

[0044] After or during training (i.e., before the inference stage), the data for different feature maps 302 (i.e., channels) is examined to determine the sparsity of the feature maps 302. Based on the results, it is determined that each of multiple filters applied to the activations produces a new activation tensor that can be evaluated based on the level of sparsity of the feature maps.

[0045] Table 1 below shows example filter information determined during the training phase, including the sparsity of different feature maps 302 (C0-C7) resulting from eight different filters applied to an input tensor subject to an operation (e.g., a convolution kernel, a pooling operation). For example, the sparsity of each feature map 302 (C0-C7) is determined during model training, which may include many iterations of applying different filters to the input tensor. In the example shown in Table 1, the sparsity is defined by the number of zero values ​​for each feature map 302 (C0-C7). However, as noted above, the number of zero values ​​is an example of sparsity. Alternatively, the sparsity may be defined according to, for example, the number of consecutive zero values ​​for a group of elements, the percentage (or range of percentages) or ratio of zero values ​​to the total number of values ​​for a group of elements, or the amount of values ​​below a value threshold.

[0046] [Table 1]

[0047] For example, as shown in Table 1, a first filter applied to the input tensor results in a first feature map (C0) having a sparsity value of 1 (i.e., one out of four elements has a zero value), a second filter applied to the input tensor results in a second feature 302 (C1) having a sparsity value of 2 (i.e., two out of four elements have a zero value), a third filter applied to the input tensor results in a third feature 302 (C2) having a sparsity value of 4 (i.e., each of the four elements has a zero value), and a fourth filter applied to the input tensor results in a fourth feature 302 (C3) having a sparsity value of 1 (i.e., one out of four elements has a zero value). , a fifth filter applied to the input tensor results in a first feature map (C4) having a sparsity value of 3 (i.e., three out of four elements have a zero value); a sixth filter applied to the input tensor results in a second feature map 302 (C5) having a sparsity value of 0 (i.e., none of the four elements have a zero value); a seventh filter applied to the input tensor results in a third feature map 302 (C6) having a sparsity value of 1 (i.e., one out of four elements have a zero value); and an eighth filter applied to the input tensor results in a fourth feature map 302 (C7) having a sparsity value of 3 (i.e., three out of four elements have a zero value).

[0048] Based on the filter information (e.g., the information shown in Table 1), the neural network is reconfigured by shuffling the filters and reordering (i.e., sorting) the output channels. For example, eight filters are applied to the input tensor data in an order different from that shown in FIG. 3, and the filters are applied according to the sparsity of the feature maps 302 (e.g., the number of zero values ​​in each feature map 302 (C0-C7)) using the NHWC format without resorting. The element values ​​are then stored in memory using the NHWC format.

[0049] For example, the filters are applied to the input tensor data in a different order than the order of the feature maps 302 shown in Figure 3. That is, the filters are sorted according to the sparsity of the feature maps using the predetermined sparsity levels shown in Table 1. Thus, as shown in Figure 4, the first element 20 of feature map 302(C5) is stored in a first location in memory portion 402. The co-located first element 00 of feature map 302(C0) is then stored in a second location in memory portion 402, adjacent to the first element 20 of feature map 302(C5). After each of the collocated first elements (i.e., 12, 24, 04, 16, 28, 08) of the remaining feature maps 302 (C3, C6, C1, C4, C7, C2) are stored in the next location in memory portion 402, the second element 21 (along width W from element 20) of feature map 302 (C5) is stored, followed by the second collocated element 01 of feature map 302 (C0).

[0050] After each of the collocated second elements (i.e., 13, 25, 05, 17, 29, 09) of the remaining feature maps 302 (C3, C6, C1, C4, C7, C2) are stored in the next location in memory portion 402, element 22 (along height H from element 20) of feature map 402 (C5) is stored, followed by collocated element 02 of feature map 302 (C0). After each of the collocated elements (i.e., 14, 26, 06, 18, 30, 10) of the remaining feature maps 302 (C3, C6, C1, C4, C7, C2) are stored in the next location in memory portion 402, element 23 of feature map 402 (C5) is stored, followed by collocated elements 03, 15, 27, 07, 19, 31, 11 of the remaining feature maps 302 (C3, C6, C1, C4, C7, C2).

[0051] Using a model developed during training that includes the above-described sorted filter, the sorted filter is applied to input tensors during the inference phase of the machine learning model. Because the amount of consecutive zero values ​​(i.e., stored in consecutive locations in memory) in a sorted feature map is greater than the amount of consecutive zero values ​​in a feature map that is not sorted according to space, the tensor data is compressed more efficiently during the inference phase of running the model. When a larger amount of zero values ​​is stored consecutively in memory by sorting, the data is compressed more efficiently (e.g., using run-length encoding). For example, run-length encoding is a form of lossless data compression in which runs of data (arrays in which the same data value occurs in many consecutive data elements) are stored as a single data value and count rather than as the original run of data. Therefore, because a larger amount of zero values ​​is stored consecutively in memory by sorting, a larger number of zero runs can be stored as a single data value, thereby reducing the number of memory transfers.

[0052] FIG. 5 is a flow diagram illustrating an example method for improving compression during the inference stage that performs machine learning operations according to features of the present disclosure.

[0053] At block 502, the method 500 includes receiving an input tensor. For example, the input tensor may be received (e.g., by a processor) at a layer of a CNN during an inference stage.

[0054] At block 504, the method 500 includes receiving a plurality of sorted filters to be applied to the input tensors during the inference phase. The sorted filters received during the inference phase may be filters that were sorted prior to the inference phase (e.g., during training), such as the sorted filters shown in FIG.

[0055] As indicated by the dashed line in block 506, when an input tensor is read from memory in a compressed format, the input tensor is decompressed. For example, the input tensor of a layer is decompressed by the processor so that the tensor can be subjected to an operation (e.g., a convolution kernel, a pooling operation) that results in a new activation tensor for the next layer. In some embodiments, the input tensor is written to memory in a compressed format, and the uncompressed input tensor is stored locally (e.g., local to the processor) and used as the next input data for the next layer of the machine learning neural network. If the input tensor is not read from memory in a compressed format, the method proceeds to block 508.

[0056] At block 508, the method 500 includes applying a plurality of sorted filters to the input tensor received at block 504. For example, the plurality of sorted filters are filters sorted according to the sparsity of each feature map 302.

[0057] At block 510, the method 500 includes compressing the tensor data (e.g., the resulting feature maps 302). For example, the tensor data is compressed and transmitted over a link (e.g., a bus) to a non-local memory (e.g., an off-chip memory). By sorting by sparsity, the data is compressed more efficiently (e.g., using run-length encoding) because a greater amount of zero values ​​are stored contiguously in memory than in feature maps that are not sorted by sparsity.

[0058] At block 512, the method 500 includes storing the tensor data. For example, the tensor data is stored in memory using the NHWC format. Because the channels are sorted according to sparsity, the number of memory transfers performed to run the model during the inference phase is reduced (i.e., memory bandwidth is reduced).

[0059] It should be understood that many variations are possible based on the disclosure herein, and although features and elements are described above in particular combinations, each feature or element can be used alone without other features and elements, or in various combinations with or without other features and elements.

[0060] The various functional units illustrated in the figures and / or described herein (including, but not limited to, the processor 102, input drivers 112, input devices 108, output drivers 114, output devices 110, acceleration processing device 116, scheduler 136, graphics processing pipeline 134, computational unit 132, and SIMD unit 138) may be implemented as a general-purpose computer, processor, or processor core, or as a program, software, or firmware stored on a non-transitory computer-readable storage medium or another medium executable by a general-purpose computer, processor, or processor core. The provided methods may be implemented in a general-purpose computer, processor, or processor core. Suitable processors include, by way of example, a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), and / or a state machine. Such a processor may be fabricated by configuring a manufacturing process using the results of processed hardware description language (HDL) instructions and other intermediate data, such as a netlist (such instructions may be stored on a computer-readable medium). The result of such processing may be a maskwork that is then used in a semiconductor manufacturing process to produce a processor that implements features of the present disclosure.

[0061] The methods or flow diagrams provided herein may be implemented in a computer program, software, or firmware embodied in a non-transitory computer-readable storage medium for execution by a general-purpose computer or processor. Examples of non-transitory computer-readable storage media include read-only memory (ROM), random-access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media (e.g., internal hard disks and removable disks), magneto-optical media, and optical media (e.g., CD-ROM disks and digital versatile disks (DVDs)).

Claims

1. 1. A processing device for executing a machine learning neural network, comprising: Memory and a processor, The processor: receiving input data at a layer of the machine learning neural network; receiving a plurality of sorted filters to be applied to the input data, the plurality of sorted filters being sorted based on sparsity of feature maps obtained during training; and applying the plurality of sorted filters to the input data to generate a plurality of different feature maps; compressing the plurality of different feature maps according to sparsity of the feature maps; storing the plurality of different feature maps in the memory; configured to: Processing device.

2. the machine learning neural network is executed during an inference phase, and the sorted filters are sorted during training prior to executing the machine learning neural network during the inference phase. The processing device of claim 1 .

3. the processor is configured to store the plurality of different feature maps in the memory using an NHWC format; The processing device of claim 1 .

4. the processor is configured to compress the plurality of different feature maps using run-length encoding. The processing device of claim 1 .

5. The sparsity of the different feature maps is determined according to a sparse feature map, where each element in the feature map has a zero value, and a non-sparse feature map, where each element in the feature map has at least one non-zero value. The processing device of claim 1 .

6. The sparsity is determined according to any of the number of zero values ​​in the feature map, the number of consecutive zero values ​​in the feature map, and the ratio of the zero values ​​to the total number of values ​​in the feature map. The processing device of claim 1 .

7. The sparsity is determined based on the amount of sub-threshold values ​​in the feature map. The processing device of claim 1 .

8. the processor is configured to store the compressed distinct feature maps in the memory according to the sparsity by transferring the compressed distinct feature maps over a link; an amount of memory transfer used to store the compressed distinct feature maps resulting from the sorted filter is less than an amount of memory transfer used to store the compressed distinct feature maps resulting from an unsorted filter; The processing device of claim 1 .

9. the processor is configured to decompress the input data if the input data is read from the memory in a compressed format; The processing device of claim 1 .

10. the processor is configured to write the input data to the memory in the compressed format and use the input data in an uncompressed format as next input data for a next layer of the machine learning neural network. The processing device of claim 9.

11. A machine learning processing method, comprising: receiving input data at a layer of a machine learning neural network; receiving a plurality of sorted filters to be applied to the input data, the plurality of sorted filters being sorted based on sparsity of feature maps obtained during training; and applying the plurality of sorted filters to the input data to generate a plurality of different feature maps; compressing the plurality of different feature maps according to sparsity of the feature maps; and storing the plurality of different feature maps. Machine learning processing methods.

12. The machine learning neural network is executed during an inference phase, and the sorted filters are sorted during training prior to executing the machine learning neural network during the inference phase. The method of claim 11.

13. storing the plurality of different feature maps using an NHWC format; The method of claim 11.

14. compressing the plurality of different feature maps using run-length encoding. The method of claim 11.

15. The sparsity of the different feature maps is determined according to a sparse feature map, where each element in the different feature map has a zero value, and a non-sparse feature map, where each element in the different feature map has at least one non-zero value. The method of claim 11.

16. The sparsity is determined according to any of the number of zero values ​​in the feature map, the number of consecutive zero values ​​in the feature map, and the ratio of the zero values ​​to the total number of values ​​in the feature map. The method of claim 11.

17. The sparsity is determined based on the amount of sub-threshold values ​​in the feature map. The method of claim 11.

18. decompressing the input data if the input data was read from memory in a compressed format; The method of claim 11.

19. writing the input data to the memory in the compressed format and using the input data in an uncompressed format as next input data for a next layer of the machine learning neural network.

20. The method of claim 18.

20. A computer-readable storage medium having instructions stored thereon, comprising: the instructions cause a computer to perform a machine learning processing method; The machine learning processing method includes: receiving input data at a layer of a machine learning neural network; receiving a plurality of sorted filters to be applied to the input data, the plurality of sorted filters being sorted based on sparsity of feature maps obtained during training; and decompressing the input data; applying the plurality of sorted filters to the input data to generate a plurality of different feature maps; compressing the plurality of different feature maps according to sparsity of the feature maps; and storing the plurality of different feature maps. A computer-readable storage medium.

Images