Neural network acceleration and embedding compression system and method using activation sparsification

Abstract

Systems, methods, and computer-readable media for (i) accelerating the inference speed of deep neural networks (DNNs) and (ii) compressing vector representations generated by DNNs from various input data (such as images, audio, video, and text). The method embodiments take a neural network architecture and a task-related loss function as input, measure the performance of the neural network on a training data set, and output a deep neural network with sparse neuron activations. The process of the present invention enhances the existing training objective function of the DNN using a regularization term, promotes sparse activations of neurons, and compresses the DNN by using various algorithms to solve optimization problems. The present disclosure also shows how to exploit sparsification of activations during DNN inference, thereby proportionally reducing the number of arithmetic operations, and how to use the sparse representations generated by the DNN to build an efficient search engine.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Non-Provisional Application No. 16 / 446,938, filed on June 20, 2019, entitled “Neural Network Acceleration and Embedding Compression Systems and Methods with Activation Sparsification,” and claims priority to and the benefit of U.S. Provisional Application No. 62 / 688,891, filed on June 22, 2018, entitled “Neural Network Acceleration and Embedding Compression via Activation Sparsification,” the entire disclosures of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to artificial intelligence, and more particularly to methods for faster computing achieved by reducing the number of operations by reducing the activation of neurons in deep neural networks. Background Art

[0004] Deep neural networks (DNNs) have become the most widely used approach in the field of artificial intelligence (AI) for extracting high-level information from lower-level data (e.g., images, videos, audio, and text). However, the expensive computational cost of DNNs has hindered their use in applications with tighter budgets in terms of energy consumption, storage space, or latency tolerance, especially on edge devices such as mobile phones and surveillance cameras.

[0005] The computational cost of DNNs comes from multiple sources. First, DNN model parameters are typically in the millions or tens of millions, resulting in huge storage costs and preventing model parameters from being placed on smaller but faster storage devices in the memory hierarchy. Second, the number of neurons in the computation graph of a DNN takes up a lot of memory space and often requires billions of arithmetic operations at runtime. Third, search engines based on vector representations generated by neural networks (e.g., face comparison engines) are typically much more computationally expensive than traditional text-based search engines, partly due to the high-dimensional dense vector representations (embeddings) produced by DNNs.

[0006] In recent years, ongoing research efforts have focused on reducing the computational cost of DNN inference. However, some of the conventional approaches focus on pruning DNN models, including (i) reducing the number of non-zero parameters (connections between neurons) in DNN filters, (ii) pruning some parts of the network, e.g., channels of neurons or columns / rows of filters, and (iii) quantizing the value ranges of parameters and neurons to reduce the number of bits used to represent these values.

[0007] Therefore, it is desirable to have methods and systems that can provide a more efficient DNN model that reduces high computational intensity. Summary of the invention

[0008] Embodiments of the present disclosure relate to methods, computer program products, and computer systems that are complementary to the above methods, which train (or fine-tune) a neural network to prevent the activation of neurons in a DNN so that during reasoning, when running on different input data, most (different) neurons are not activated. Each input data activates a small number of neurons, thereby reducing the number of operations required during reasoning and reducing the storage required for the vector representation (embedding) generated by the DNN for each input data in proportion to the number of neurons that are inactivated.

[0009] In summary, a method for reducing the computational cost of deep neural network (DNN) inference includes the following steps: determining a loss function based on a specific task, wherein the loss function is capable of measuring the prediction performance of the deep neural network; retrieving an initial deep neural network having a DNN computational graph and related parameter values; iteratively sparsifying the number of active neurons of the deep neural network until a sparsity ratio converges; and generating a deep neural network with sparse activations as an output.

[0010] The structure and method of the present disclosure will be disclosed in detail in the following description. The present disclosure should not be limited by the present disclosure. The present disclosure is defined by the claims. These and other embodiments, features, aspects and advantages of the present disclosure will be better understood based on the following description, the attached claims and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Specific embodiments of the present disclosure will be described with reference to the accompanying drawings, in which:

[0012] Figure 1 is a system diagram showing the overall software architecture of an activation compressor according to the present disclosure.

[0013] Figure 2 is a flow chart illustrating the input and output of an activation compressor according to the present disclosure.

[0014] Figure 3is a flow chart illustrating one embodiment of activating a compressor according to the present disclosure.

[0015] Figure 4 is a sequence diagram illustrating the interaction between multiple software modules for an embodiment of activating a compressor according to the present disclosure.

[0016] Figure 5 is a block diagram illustrating the inputs and outputs of an activated compressor system according to the present disclosure.

[0017] Figure 6-9 is a schematic diagram illustrating a commercial application of an activated compressor system according to the present disclosure.

[0018] Fig.10 is a block diagram illustrating an example of a computer device on which computer-executable instructions according to the present disclosure for performing the methods discussed herein may be installed and executed. DETAILED DESCRIPTION

[0019] refer to Figure 1-10 To provide a description of the structural embodiments and methods of the present disclosure. It should be understood that the present disclosure should not be limited to the specific disclosed embodiments, but the present disclosure can be implemented using other features, elements, methods and embodiments. The same elements in the various embodiments are generally referred to by the same reference numerals.

[0020] The following definitions apply to the elements and steps described herein. These terms can also be expanded.

[0021] Speedup - changing the DNN so that fewer arithmetic operations are required to compute the DNN function.

[0022] Activation Compressor - A software system that takes a DNN as input and outputs a sparsified DNN, obtained by iteratively adjusting the parameters of the DNN in a direction (gradient) that reduces both the loss function and the activation regularizer.

[0023] Activation Regularization Term - A function that measures the sparsity of activations in a DNN for a particular training dataset. The sparser the DNN, the lower the value of the regularization term.

[0024] Deep Neural Network (DNN) - A Deep Neural Network (DNN) is a composite function consisting of several layers of elementary functions, where each function takes the output of the previous layer as input and computes a new set of outputs. The output of each layer is called a neuron, and if a neuron has a non-zero value, it is activated (or active), otherwise it is deactivated (or inactive).

[0025] Loss function - used to measure the performance of a DNN in performing a specific task on a specific training dataset. The better the performance of the DNN, the lower its loss function value.

[0026] Sparsification (Sparsify) - A method of training a DNN (i.e., adjusting the parameters of the DNN) so that the number of activated neurons is significantly less than the total number of neurons (e.g., at least less than half, and typically less than a quarter).

[0027] Figure 1 1 is a system diagram showing the overall software architecture of the activation compressor 100. The activation compressor 100 includes a network module 110, a loss function module (or "loss function") 120, an activation regularization term module (or "activation regularization term") 130, a data reader module (or "data reader") 140, an optimizer module (or "optimizer") 150, and a synthesizer module (or "synthesizer") 160. The network model module 110 includes a set of elementary operations (i.e., numerical functions) and corresponding derivatives, and includes a complex function (i.e., DNN) composed of elementary functions. In addition, the network model module 110 includes a set of variables that parameterize the complex function. The complex function is modularized into several layers, each of which takes the output of the previous layer as input and calculates a new set of outputs. The network model module 110 calculates the activation value of the neuron (i.e., the output of each layer), and the calculation of the prediction value (i.e., the output of the last layer). Formally, let: X (0) :N×D 1 (0) ···×D P (0) ×K 0 is the input tensor, where N is the number of samples (or batch size), and D 1 (0) ,...,D P (0) ,K 0 is the dimension of the signal. DNN usually consists of several layers. Let j=1…J be the layer index. DNN can be represented as a combination of a series of functions, as shown in the following formula (1):

[0028]

[0029] in The function representing layer (j) is called filter W (j) Tensor parameterization: K (j-1) ×C 0 (j) ···×C P (j) ×K (j), and X (j-1) :N×D 1 (j-1 )···×D P (j-1) is the input tensor of the (j)th layer. Commonly used functions are, for example, (i) convolutional layer with Relu activation:

[0030]

[0031] Where * is a p-dimensional convolution operator, (ii) a fully connected layer with ReLU activation:

[0032] [σ W (X)] i，k :=[X i ; W ：，k ] + ,

[0033] And some other common operations, such as max-pooling, zero-padding, and reshaping. Figure 1 As shown, in the software architecture of the activation compressor 100 , the applicable functions and their numerical derivatives are implemented in the network model module 110 .

[0034] The loss function module 120 is configured to provide the definition and calculation of a penalty value that measures the difference between the predicted value and the ground-truth answer. The loss function module 120 also calculates the derivative of the penalty value with respect to the predicted value. Formally, let X (J) :N×K is the output of the last layer of the DNN, which provides K scores (i.e., logit) for each sample associated with the label of the target task Y:N×K. The loss function associated with the task is:

[0035]

[0036] Measuring a given training dataset The output of DNN prediction and training set The difference between the correct output given by . An example of a loss function is the regression loss, where

[0037]

[0038] Where x is the K-dimensional predicted score vector and y is the ground truth correct output. Another example of a loss function is the cross entropy loss, where

[0039]

[0040] The activation regularization term module 130 is configured to provide the definition and calculation of a penalty value (and its derivative) that measures the activation level of all neurons in the network by minimizing the number of neurons that can sparse the number of active neurons in the network. There are two embodiments of the activation regularization term 130. The first embodiment of the activation regularization term 130 directly regularizes the value of the activation. The second embodiment of the activation regularization term 130 regularizes the parameters that control the sparsity level of the activation of each layer.

[0041] An example of a first embodiment of activating the regularization term 130 is in represents tuning hyper parameter, and ‖X‖ 1 (the sum of all absolute values ​​of X) is a convex surrogate function of X, which approximately measures how many non-zero components there are in X. Note that X (j) are the parameters associated with all layers before layer j (i.e. {W (j′) l} j′≤j ) function.

[0042] The second embodiment of the activation regularizer 130 is used with an activation function having a parameter that controls the sparsity level of the activation. An example of such an activation function is shown below:

[0043]

[0044] It only retains the values ​​of the top-r elements with the largest values ​​and suppresses the values ​​of the remaining elements to 0. This special type of activation function is used instead of the DNN layer Other activation functions used in . Since the activation function (2) is not Lipschitz continuous, in practice, we use a smoothed version of the function in the following form:

[0045]

[0046] The symbol represents the Hadamard (element-wise) product, and the term Represents a convex set C r : = {q|0≤q i ≤1,∑ i q i ≤r}. In this example, the symbol “r” represents the control α 0r (X) is a control parameter for the sparsification level. A simple example of activating the regularization term 130 is where r l,l=1…J represents the control parameters of each layer.

[0047] The data reader module 140 includes a pipeline for obtaining input data for the DNN from a data source. The pipeline includes chopping, subsampling (or upsampling), batching, and whitening (normalization). The optimizer module 150 is configured to adjust the values ​​of the variables in the network model so as to reduce the objective function including the loss function 120 and the activation regularization term 130. In one embodiment, the adjustment is determined by the derivative of the loss function 120 and the activation regularization term 130 with respect to the variable. The synthesizer 160 is a high-level component of the software that plans and organizes the interactions between the other components and determines the relative strength between the loss function 120 and the activation regularization term 130.

[0048] Figure 2 2 is a flow chart showing the input and output of compression method 200. This process has two inputs: (i) in step 210, a training dataset of correct input-output pairs for a DNN, which measures and improves the prediction performance of the DNN on a specific task with a specific loss function; and (ii) in step 220, the architecture (computational graph) of the DNN with or without pre-trained parameter values. In step 230 ("Compression Process"), the compressor 100 is activated to iteratively sparsify the DNN until convergence. In step 240, compression method 200 generates an output of a compressed DNN with the same architecture but different parameter values. Compression method 200 compresses the DNN so that it has sparse activations: each layer X (j) The number of non-zero elements (activated neurons) of is small.

[0049] Figure 3 320 and the parameter update of step 330 until the termination criteria of step 340 of the optimizer 150 are met (i.e., the target change based on a predetermined threshold or the gradient magnitude is sufficiently small). (j) l} j＝1…J represents the set of parameters of all layers. The compression process 230 minimizes the following objective function:

[0050]

[0051] The term L(X (J) ) represents the loss function (depending only on the output of the last layer of the DNN), and the term R(X (j)) represents the activation regularization term 130 applied to the output of the jth layer. Next, in step 320, the compression process 230 evaluates the gradient of the target by backpropagation through the network model 110 and the data reader 140. The compression process 230 calculates the derivatives of F(W) for all parameters W, and in step 330, the parameter update changes the value of W according to the obtained gradient and the update equation determined by the optimizer. In step 340, the activation compressor 340 determines whether the target has converged. If the target has not converged, the activation compressor 100 continues to iteratively sparse the DNN through the gradient evaluation of step 320 and the processing of the parameters of step 330. After the target converges, the activation compressor 100 returns the process to step 240.

[0052] Figure 4 4 is a sequence diagram showing the interaction between multiple software modules for computing gradient evaluations at step 320 and for parameter updating at step 330. Steps 461, 462, and 463 depict the activation of the regularization term R(X (j) ). In steps 461, 462 and 463, the optimizer 130 is configured to request activation of the regularization term 130 with respect to the gradient of the parameter W (i.e., Then, the activation regularization term module 130 is configured to request the network model module 110 to calculate the activation values ​​{X (j)} j＝1…J (462), which requires the data reader module 140 to provide input data from the input preprocessing pipeline 463.

[0053] Steps 471, 472, and 473 show the loss function L(X (J) ). In steps 471, 472 and 473, the optimizer 150 is configured to request the gradient of the loss function with respect to the parameter W (i.e. The loss function module 120 is configured to request the network model module 110 to calculate the prediction (output) layer X (J) value (472), which further requires the data reader module 140 to provide the input data from the input preprocessing pipeline 473 to the network model module 110 and provide the correct label to the loss function module 120.

[0054] Referring to the first embodiment of the activation regularization term described in paragraph

[0027] , the gradient evaluation of the objective function F(W) is in the form of augmented backpropagation, where the DNN X is calculated by a forward pass. (J) The output:

[0055]

[0056] Then the gradient Backpropagate from the regularization term to the parameters of each layer with enhanced gradients:

[0057]

[0058] For layer j = (JI)...I, the term (. indicates the operation of this layer The second term ρ j sign(X (j) represents the key to the compression process, which enhances the back-propagated information by reducing the neuron activation value to zero momentum, and can be obtained from X based on the same rules of back-propagation in the standard back-propagation process. (j-1) and Get the parameters W of each layer (j) gradient.

[0059] Referring to the second embodiment of the activation regularization term described in paragraph

[0028] , the gradient evaluation of the objective function F(W) first performs a forward propagation similar to equation (1) but using a parameterized activation function such as equation (2) or equation (3), thereby obtaining an activation number X controlled by certain parameters. (j) The key difference during backpropagation is the following steps: relative to the activation output Propagate the gradient backward to obtain the value of The gradient of , which can be calculated by:

[0060]

[0061] For the non-smooth sparse activation function in formula (3), the mathematical expression is as follows:

[0062]

[0063] For the smoothed version, the mathematical expression is as follows:

[0064]

[0065] Among them Represents projection onto the convex set C r :={q|0≤qi≤1,∑ i q i ≤r}. Since q(X) is a very sparse vector in either case, the intermediate gradient in (4) Sparsified during back-propagation, it can be used to achieve significant speedups in operations such as convolution and matrix multiplication in DNNs.

[0066] The process update variables in step 481 implement an optimization algorithm that determines how to update the values ​​of the DNN parameters based on the gradients calculated from the back-propagation process. In one embodiment, this part uses a standard first-order optimization algorithm that uses only the gradient (and its history during the iteration process) to determine the update direction of the parameter W, such as stochastic gradient descent (SGD), stochastic variance reduction gradient method (SVRG), adaptive gradient method (Adagrad), root mean square propagation (RMSprop), adaptive moment estimation method (ADAM).

[0067] Figure 5 500 is a diagram showing how sparse activations as input to each layer can significantly improve the efficiency of a DNN. Specifically, in one embodiment of computing convolutions (usually the largest amount of computation in a DNN), it accesses activated neurons and accesses the connecting edges between neurons with non-zero weights. The method of the present disclosure can generally be integrated with other techniques that sparse the non-zero connections (filter weights) between neurons.

[0068] In the following examples, it is assumed that both neuron activations and (non-zero weight) neuron connections are sparse. Take two-dimensional convolution as an example. Those skilled in the art will recognize that two-dimensional convolution can be extended to multi-dimensional convolution without departing from the spirit of the present disclosure. Input neuron X: D 1 ×D 2 ×K 0 and filter W:K 0 ×C 1 ×C 2 ×K 1 The convolution between them produces a 3D tensor Z:×(D 1 -C 1 +1)×(D 2 -C2+1)×K 1 , which is related to the following formula:

[0069]

[0070] The symbol * indicates the convolution operation. Assume that X is stored as a sparse list of index-value pairs, and each input channel of the filter is also stored as a sparse list of index-value pairs, denoted as list X and listW c , to take advantage of the sparse structure of X and W, the implementation of convolution can be summarized as follows:

[0071]

[0072] The number of arithmetic operations used in the above process is proportional to the number of non-zero elements, not to the original shapes of W and X. Therefore, by Figure 4 The sparsification process introduced in can significantly reduce the computational burden. Note that since the fully connected layer in DNN can be converted into a special case of convolution, the above process can also be used for the fully connected layer.

[0073] In addition to improving inference speed, sparsification also compresses the embeddings (i.e., neuron activations in the last few layers) produced by DNNs. The embeddings produced by DNNs are often used as semantic vector representations of raw input data (e.g., images, audio, video, and text). Taking images as an example of input data, the similarity between two images can be measured by the inner product between the embeddings of the two images generated by DNNs. Then, a semantic search engine for images can be built on top of these embeddings. Formally, let {x 1 ,x 2 ,...,x N} is the set of images in the database, and {z 1 ,z 2 ,...,z N} are the embeddings of those images produced by the DNN. The search engine performs the following operations: (i) Given a query image x q , generate its embedding z through DNN q , (ii) return the embedding {z r 1,z r 2,...,z r m} in the image database with z q The list of images with the highest inner product is obtained by sparsifying the activations (100). In this application, sparsifying the activations (100) yields two advantages. First, by sparsifying the activations, the space required to store these embeddings is significantly reduced; second, the similarity computation can be significantly accelerated using the computational flow described in paragraph

[0037] ,

[0074] Figure 6 , Figure 7 , Figure 8 as well as Fig. 9 Four examples of commercial applications of the activation compressor 100 are shown, where the disclosed method improves inference efficiency for (i) facial recognition, (ii) image conversion, (iii) language translation, and (iv) speech recognition.

[0075] Figure 6510 is a diagram showing the application of the present disclosure to face recognition, where the activation compressor 100 reduces the number of arithmetic operations required to generate semantic embeddings of faces by DNN. By the similarity between two such face embeddings, it is possible to distinguish whether two images contain the same person.

[0076] Figure 7 5 is a diagram showing the application of the present disclosure to image conversion, where the DNN takes an image as input and outputs a different version of the image (e.g., with changed texture, brightened condition, etc.). The activation compressor 100 can be used to reduce the number of arithmetic operations required by the DNN to convert the image version.

[0077] Figure 8 530 is a diagram showing the application of the present disclosure to language translation, where a recurrent neural network (RNN, a special type of DNN) takes English word tokens as input and outputs Chinese characters. The activation compressor 100 reduces the number of active neurons in both the encoder and decoder parts of the RNN, thereby reducing the number of arithmetic operations.

[0078] Fig. 9 is a diagram 540 showing application of the activation compressor 100 to a DNN trained for speech recognition, where the DNN takes as input an audio wave signal and outputs human language words contained in the audio signal.

[0079] As discussed above, the various computer-based devices discussed in connection with the present invention may share similar attributes. Fig.10 An exemplary form of a computer system 600 is shown, in which a set of instructions can be executed to cause the computer system to perform any one or more of the methods discussed herein. The computer device 600 can represent any or all of the client, server, or network intermediate device discussed herein. In addition, although only a single machine is shown, the term "machine" should also be understood to include any collection of machines that individually or collectively execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein. The exemplary computer system 600 includes a processor 602 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), a main memory 604, and a static memory 606, which communicate with each other via a bus 608. The computer system 600 may also include a video display unit 610 (e.g., a liquid crystal display (LCD)). The computer system 600 also includes an alphanumeric input device 612 (e.g., a keyboard), a cursor control device 614 (e.g., a mouse), a disk drive unit 616, a signal generating device 618 (e.g., a speaker), and a network interface device 624.

[0080] The disk drive unit 616 includes a machine-readable medium 620 on which is stored one or more sets of instructions (e.g., software 622) embodying any one or more of the methods or functions described herein. The software 622 may also reside in whole or at least in part within the main memory 604 and / or within the processor 602. During execution, the computer system 600, the main memory 604, and the instruction storage portion of the processor 602 also constitute machine-readable media. The software 622 may also be sent or received over a network 626 via a network interface device 624.

[0081] Although the machine-readable medium 620 is shown as a single medium in the exemplary embodiment, the term "machine-readable medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database, and / or caches and servers associated therewith) to store one or more sets of instructions. The term "machine-readable medium" should also be considered to include any tangible medium that can store a set of instructions executed by a machine and cause the machine to perform any one or more of the methods of the present invention. Therefore, the term "machine-readable medium" should be considered to include, but is not limited to, solid-state memories and optical and magnetic media.

[0082] Certain portions of the detailed description herein are presented in terms of algorithms and symbolic representations of operations performed on data within a computer memory or other storage device. These algorithmic descriptions and representations are the means used by those skilled in the art of data processing to most effectively convey the substance of their work to others skilled in the art. Here, an algorithm is generally considered to be a self-consistent sequence of processing blocks that produce a desired result. Processing blocks are those blocks that require physical manipulation of physical quantities. Throughout the specification, discussions using terms such as "processing" or "calculating" or "calculating" or "determining" or "displaying" refer to the actions and processes of a computer system or similar electronic computing device for manipulating and converting data represented as physical (electronic) quantities in the computer system's registers and memories into other data similarly represented as physical quantities in the computer system's memories or registers or other such information storage, transmission or display devices.

[0083] The present invention also relates to an apparatus for performing the operations described herein. This apparatus may be specially constructed to achieve the desired purpose, or may include a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer-readable storage medium, such as, but not limited to, any type of disk, including a floppy disk, an optical disk, a CD-ROM, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a magnetic card or an optical card, an application-specific integrated circuit (ASIC), or any type of medium suitable for storing electronic instructions and coupled to a computer system bus. In addition, the computer and / or other electronic device referred to in the specification may include a single processor, or may be an architecture that uses multiple processors to increase computing power.

[0084] In addition, terms such as "request", "client request", "requested object", or "object" can be used interchangeably to refer to the action, object, and / or information requested by the client from a network device such as an intermediary device or a server. In addition, the term "response" or "server response" can be used interchangeably to refer to the corresponding action, object, and / or information returned from the network device. In addition, the terms "communication" and "client communication" can be used interchangeably to refer to the entire process of a client making a request and a network device responding to the request.

[0085] With respect to any of the above system, device or apparatus aspects, a method aspect comprising steps for performing system functions may also be provided.Additionally or alternatively, optional features may be found based on any one or more of the features described herein with respect to other aspects.

[0086] The present disclosure has been described in particular detail with respect to possible embodiments. Those skilled in the art will appreciate that the present disclosure may be practiced in other embodiments. The specific naming of components, capitalization of terms, attributes, data structures, or any other programming or structural aspects are not necessarily adopted or important, but the mechanisms that implement the present disclosure or its features may have different names, formats, or protocols. As described above, the system may be implemented by a combination of hardware and software, or entirely in hardware elements or entirely in software elements. The specific division of functions between the various system components described herein is merely exemplary and not necessarily adopted; functions performed by a single system component may alternatively be performed by multiple components, and functions performed by multiple components may alternatively be performed by a single component.

[0087] In various embodiments, the present disclosure may be implemented as a system or method for performing the above-mentioned techniques alone or in any combination. Combinations of any specific features described herein are provided even if not explicitly described. In another embodiment, the present disclosure may be implemented as a computer program product, which includes a computer-readable storage medium and a computer program code encoded on the medium, for causing a processor in a computing device or other electronic device to perform the above-mentioned techniques.

[0088] As used herein, any reference to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. The phrase "in one embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment.

[0089] It should be remembered, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise expressly noted in the discussion below, it should be understood that throughout the specification, discussions utilizing terms such as "processing" or "computing" or "calculating" or "displaying" or "determining" refer to the actions and processes of a computer system or similar electronic computing module and / or device for manipulating and transforming data represented as physical (electronic) quantities in a computer system memory or register or other such information storage, transmission or display device.

[0090] Certain aspects of the present disclosure include processing steps and instructions described herein in algorithmic form. It should be noted that the processing steps and instructions of the present disclosure may be embodied in software, firmware and / or hardware, and when embodied in software, may be downloaded to reside on and be used by various platforms used by various operating systems.

[0091] The algorithms and displays proposed herein do not have inherent relevance to any particular computer, virtualization system, or other device. According to the teachings herein, various general-purpose systems may also be used together with the program, or these systems may conveniently construct more specialized devices required for performing the required method steps. According to the description provided herein, the structures required for various these systems will be apparent. In addition, the present disclosure is not described with reference to any particular programming language. It should be understood that, as described herein, a variety of programming languages ​​may be used to implement the teachings of the present disclosure, and any reference to a particular language above is provided for disclosing implementations and best modes of the present invention.

[0092] In various embodiments, the present disclosure may be implemented as software, hardware, and / or other elements for controlling a computer system, a computing device, or other electronic device, or any combination or multiple thereof. Such electronic devices may include, for example, a processor, an input device (such as a keyboard, a mouse, a touchpad, a trackpad, a joystick, a trackball, a microphone, and / or any combination thereof), an output device (such as a screen, a speaker, and / or the like), a memory, a long-term memory (such as a magnetic memory, an optical memory, and / or the like), and / or a network connection, depending on the technology well known in the art. Such electronic devices may be portable or non-portable. Examples of electronic devices that may be used to implement the present disclosure include mobile phones, personal digital assistants, smart phones, self-service terminals, desktop computers, laptop computers, consumer electronic devices, televisions, set-top boxes, and the like. Electronic devices for implementing the present disclosure may use an operating system such as iOS available from Apple Inc. located in Cupertino, California, Android available from Google Inc. located in Mountain View, California, Microsoft Windows 10 available from Microsoft Corporation located in Redmond, Washington, or any other operating system suitable for the device. In some embodiments, electronic devices for implementing the present disclosure include functionality for communicating over one or more networks, including, for example, a cellular telephone network, a wireless network, and / or a computer network such as the Internet.

[0093] The words "coupled" and "connected" and their derivatives may be used to describe some embodiments. It should be understood that these terms should not be used as synonyms for each other. For example, the word "connected" may be used to describe some embodiments to indicate that two or more elements are in direct physical or electrical contact with each other. In another example, the word "coupled" may be used to describe some embodiments to indicate that two or more elements are in direct physical or electrical contact; however, the word "coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other. The embodiments are not limited to this.

[0094] As used herein, the terms "comprises," "comprising," "includes," "including," "has," "having," or any other variation thereof, shall encompass a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. In addition, unless expressly specified to the contrary, "or" refers to an inclusive "or" and not to an exclusive "or." For example, condition A or B is satisfied by any of the following conditions: A is true (or exists) and B is false (or does not exist), A is false (or does not exist) and B is true (or exists), and both A and B are true (or exists).

[0095] As used herein, the term "a" or "an" is defined as one or more. As used herein, the term "plurality" is defined as two or more. As used herein, the term "another" is defined as at least a second or more.

[0096] A skilled artisan does not require any additional instructions in developing the methods and systems described herein, but may find some potentially useful guidance in the preparation of these methods and systems by examining standardized reference works in the relevant art.

[0097] Although the present disclosure has been described with respect to a limited number of embodiments, those skilled in the art having the benefit of the above description will appreciate that other embodiments may be devised without departing from the scope of the present disclosure as described herein. It should be noted that the language used in the specification is selected primarily for readability and instructional purposes, and may not be selected to describe or limit the subject matter of the present invention. The terms used should not be interpreted as limiting the present disclosure to the specific embodiments disclosed in the specification and claims, but rather the terms should be interpreted to include all methods and systems operating in accordance with the claims described below. Therefore, the present disclosure is not limited to what is disclosed, and its scope should be determined entirely by the appended claims.

Claims

1. A method for reducing the computational cost of deep neural network (DNN) inference, comprising: (a) determining a loss function using N input samples and corresponding expected outputs, wherein the loss function measures the prediction performance of a deep neural network for a given task for a given training dataset; (b) retrieving an initial deep neural network having a deep neural network computation graph and associated parameter values, wherein the deep neural network is modularized into a plurality of layers, each layer taking outputs of one or more previous layers as input data and computing a set of new outputs; (c) enhancing the loss function with an activation regularization term, wherein the activation regularization term measures the activation level of a neuron activation matrix formed by all neurons of the deep neural network on the input sample, thereby sparsely sparsely making the number of active neurons in the deep neural network when the deep neural network is minimized; (d) iteratively sparsifying the number of active neurons of the deep neural network until the number of active neurons converges, wherein the number of active neurons is reduced by a compressor in response to one or more input samples from the loss function on the neuron activation matrix, the neuron activation matrix having a first matrix dimension equal to the number of samples and a second matrix dimension equal to the total number of neurons, and any non-zero value in the neuron activation matrix indicates an active neuron; and (e) Generate and output a deep neural network with sparse activations, The input data includes at least one input sample, and the at least one input sample includes at least one of an image, an audio wave, a video file, and a text.

2. The method according to claim 1, wherein: The sparsification step involves augmenting the loss function by replacing the original activation function with a set of new sparsity-induced activation functions to form the objective function.

3. The method according to claim 1, wherein: The sparsification step includes enhancing the loss function with one or more activation regularization terms to form an objective function.

4. The method according to claim 3, wherein: The sparsification step involves evaluating the gradient of the objective function by back-propagation with an enhanced activation regularization term.

5. The method according to claim 3, wherein: The sparsification step involves evaluating the gradient of the objective function by back-propagation using the sparsification-induced activation function.

6. The method according to claim 4, wherein: The sparsifying step includes iteratively adjusting one or more parameters associated with the deep neural network based on the gradient.

7. The method according to claim 6, wherein: The thinning step includes determining whether the deep neural network has converged by checking whether a change in the number of active neurons is less than a predetermined threshold.

8. The method according to claim 1, wherein: The number of active neurons in the last layer of the deep neural network is reduced to obtain a compressed representation of the vector generated by the deep neural network.

9. A computer-implemented activation compressor comprising: a network model module configured to calculate activation values ​​of a plurality of neurons and calculate prediction values, the network model module comprising a set of variables that parameterize a deep neural network (DNN), the deep neural network being modularized into a plurality of layers, each layer taking outputs of one or more previous layers as input data and calculating a set of new outputs; a loss function module, communicatively coupled to the network model module and configured to calculate a first penalty value, the first penalty value measuring the difference between the predicted value and the ground truth answer, the loss function module being configured to determine a loss function using N input samples and corresponding expected outputs, the loss function measuring the prediction performance of the deep neural network in performing a given task for a given training data set; an activation regularization term, communicatively coupled to the network model module and configured to calculate a second penalty value, the second penalty value measuring the activation level of all neurons of the network model module, the activation regularization term comprising a sum of a plurality of penalty functions, each penalty function being calculated on a neuron activation matrix, the neuron activation matrix having a first matrix dimension equal to the number of samples and a second matrix dimension equal to the total number of neurons, the activation regularization term enhancing the loss function to measure the activation level of the neuron activation matrix formed by all neurons of the deep neural network on the input sample, thereby sparseening the number of active neurons in the deep neural network when the deep neural network is minimized; a data reader module, communicatively coupled to the network model module, comprising a pipeline for obtaining input data for the deep neural network from a plurality of data sources; as well as an optimizer, communicatively coupled to the loss function module and the activation regularization term, and configured to adjust values ​​of variables in the network model module to reduce an objective function including the loss function module and the activation regularization term, The input data includes at least one input sample, and the at least one input sample includes at least one of an image, an audio wave, a video file, and a text.

10. The activation compressor according to claim 9, wherein: The neurons include one or more outputs in each layer.

11. The activation compressor according to claim 9, wherein: The predicted values ​​include one or more outputs of the last layer.

12. The activation compressor according to claim 9, wherein: The loss function module calculates a derivative of the first penalty value with respect to the predicted value.

13. The activation compressor according to claim 9, wherein: The loss function module includes a task-related loss function, which is used to measure the difference between the output predicted from the DNN on the given training data set and the given correct output on the training set.

14. The activation compressor according to claim 9, wherein: The activation regularization term computes the sum of a penalty function across the network minimizing the number of active neurons in which the compressor reduces the number of active neurons.

15. The activation compressor according to claim 9, wherein: Each penalty function in the activation regularization term measures the activation level of an N x K neuron activation matrix of the deep neural network, where N represents the number of input samples and K represents the number of neurons in the deep neural network.

16. The activation compressor according to claim 15, wherein: The activation regularization term directly regularizes the activation values.

17. The activation compressor according to claim 9, wherein: The activation regularization term regularizes multiple parameters that control the sparsity level of the activations of each layer.

18. The activation compressor according to claim 17, wherein: The activation regularization term is used with an activation function having a parameter that controls the sparsification level of the activation, which retains only the values ​​of the top-r elements with the largest values ​​and suppresses the rest to zero.

19. The activation compressor according to claim 9, wherein: The data reader module includes a pipeline for obtaining input data for the deep neural network from multiple data sources, the pipeline including at least one of cropping, subsampling, upsampling, batching, and whitening and normalization.

20. The activation compressor of claim 9, wherein: The optimizer is configured to adjust values ​​of variables in the network model module to reduce an objective function including the loss function and the activation regularization term.

21. A computer-implemented method for efficient computation of a deep neural network (DNN), comprising: (a) providing a network model module for calculating activation values ​​of multiple neurons and calculating prediction values, wherein the network model module includes a set of variables that parameterize a deep neural network, wherein the deep neural network is modularized into multiple layers, each layer taking the output of one or more previous layers as input data and calculating a set of new outputs; (b) evaluating the gradient of an objective function including an activation regularization term through back propagation of the network model module and the data reader, wherein evaluating the gradient includes requesting the gradient of the activation regularization term with respect to the parameter, wherein the activation regularization term includes a sum of a plurality of penalty functions, each penalty function being calculated on a neuron activation matrix, wherein the neuron activation matrix has a first matrix dimension equal to the number of samples and a second matrix dimension equal to the total number of neurons in the deep neural network, wherein the gradient specifies a direction of change of a weight value to sparse the number of non-zero entries in the neuron activation matrix; (c) updating the DNN parameters based on the gradient by an optimizer to adjust the values ​​of a set of variables in the network model module to reduce the objective function; (d) determining by the optimizer based on a predetermined threshold whether the target change or gradient size is small enough to indicate that the target has converged, the optimizer module being configured to adjust the value of the variable in the network model module to reduce the target function including the loss function and the activation regularization term, When the objective has not converged, iteratively repeating steps (b) and (c), and When the target converges, the deep neural network is activated to generate output. The input data includes at least one input sample, and the at least one input sample includes at least one of an image, an audio wave, a video file, and text.

22. The method according to claim 21, wherein: The gradient evaluation step includes requesting the network model module to calculate the activation values ​​of all layers {X (j) } j＝1…J .

23. The method according to claim 21, wherein: The step of updating the DNN parameters includes requesting the gradient of the loss function with respect to the parameter W from the loss function module, i.e.

24. The method according to claim 23, wherein: The step of updating the DNN parameters includes requesting the network model module to calculate the value of the prediction output layer, wherein input data is provided to the network model module from the input preprocessing pipeline and the correct label is provided to the loss function module.

25. The method according to claim 23, wherein: The sparsifying step includes iteratively adjusting one or more parameters associated with the deep neural network based on the gradient.

26. The method according to claim 25, wherein: The sparsification step includes determining whether the deep neural network converges by checking whether a change in the number of active neurons on the neuron activation matrices of all layers becomes smaller than a predetermined threshold.

Images