Systems and methods for balanced pruning of weights of a deep neural network

Abstract

A system and method for providing balanced pruning of weights of a deep neural network, wherein, in the system and method, the weights of the DNN are divided into multiple groups, the count of the number of non-zero weights is determined in each group, the variance of the count of the weights in each group is determined, the Lagrange multiplier is used to minimize the loss function of the DNN with the constraint that the variance of the count of the weights in each group is equal to 0, and the weights and the Lagrange multiplier are retrained through backpropagation.

Description

[0001] This application claims priority to U.S. Provisional Patent Application No. 62 / 735,846, filed on September 24, 2018, and assigned to the U.S. Patent and Trademark Office, and U.S. Non - Provisional Patent Application No. 16 / 186,470, filed on November 9, 2018, and assigned to the U.S. Patent and Trademark Office. The entire contents of the U.S. Provisional Patent Application and the U.S. Non - Provisional Patent Application are hereby incorporated by reference in their entirety. Technical Field

[0002] The subject matter disclosed herein relates to neural networks. More specifically, the subject matter disclosed herein relates to systems and methods for providing balanced pruning of the weights of a deep neural network (DNN). Background Art

[0003] Neural network hardware is designed to accelerate the computations of deep neural networks. However, hardware accelerators are not efficient in processing sparse parameters, and balancing the sparsity of neural network parameters to ensure that parallel processing elements have the same amount of workload is a difficult problem. Sparsity is used to reduce the parameter size and improve the computational efficiency of deep neural networks. Summary of the Invention

[0004] Example embodiments provide a system for providing balanced pruning of the weights of a deep neural network (DNN). The system may include: a weight masker and a loss determiner. The weight masker may mask the weights in each of a plurality of weight groups of the DNN; the loss determiner may determine the loss of the DNN based on the network loss of the DNN minus the variance of the count of non - zero weights in the plurality of weight groups. In one embodiment, the loss determiner may also determine the loss of the DNN based on the network loss of the DNN minus the variance of the count of non - zero weights in the plurality of weight groups multiplied by a Lagrange multiplier. In another embodiment, the weight masker may multiply the weights in each of the weight groups by a masking function. In another embodiment, the masking function may be differentiable.

[0005] An exemplary embodiment provides a method for providing balanced pruning of weights of a deep neural network (DNN). The method may include: dividing the weights of the DNN into multiple groups; applying a masking function to the weights in each group through a weight masker; after the masking function is applied, determining a count of the number of non-zero weights in each group; determining a variance of the count of the non-zero weights in each group; using a Lagrange multiplier by a loss determiner to minimize a loss function of the DNN with a constraint that the variance of the count of the non-zero weights in each group is equal to 0; and retraining the weights and the Lagrange multiplier through backpropagation. In one embodiment, the Lagrange multiplier may be used to multiply the variance of the count of the non-zero weights in each group. In another embodiment, post-processing may include: determining a pattern of the retrained weights; if a group among the multiple groups has more non-zero weights after balanced pruning, adjusting the masking function applied to the group to 0 until the count of the non-zero weights in the group is equal to the pattern of the retrained weights; and if a group among the multiple groups has fewer non-zero weights after balanced pruning, adjusting the masking function applied to the group to 1 until the count of the non-zero weights in the group is equal to the pattern of the retrained weights. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] In the following sections, aspects of the subject matter disclosed herein will be described with reference to exemplary embodiments shown in the drawings, where:

[0007] Figure 1 A graph depicting an example threshold function according to the subject matter disclosed herein;

[0008] Figure 2 is a flowchart of a method for balanced self-pruning of a DNN according to the subject matter disclosed herein;

[0009] Figure 3 is a flowchart of an exemplary embodiment of a post-processing method according to the subject matter disclosed herein;

[0010] Figure 4 Describes an exemplary embodiment of a balanced self-pruner according to the subject matter disclosed herein. DETAILED DESCRIPTION

[0011] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, those skilled in the art will understand that aspects of the disclosure may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the subject matter disclosed herein. Additionally, the described aspects may be implemented in any imaging device or system (including but not limited to smart phones, user equipment (UE), and / or laptop computers).

[0012] References to "one embodiment" or "an embodiment" in the present specification mean that the particular features, structures, or characteristics described in connection with the embodiment may be included in at least one embodiment disclosed herein. Thus, the appearances of the phrases "in one embodiment," "in an embodiment," "according to one embodiment" (or other phrases with similar meanings) in various places throughout this specification are not necessarily all referring to the same embodiment. Additionally, in one or more embodiments, the particular features, structures, or characteristics may be combined in any suitable manner. In this regard, as used herein, the word "exemplary" means "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" should not be construed as necessarily being preferred or advantageous over other embodiments. Further, in the context of the discussion herein, singular terms may include the corresponding plural forms, and plural terms may include the corresponding singular forms. Similarly, hyphenated terms (e.g., "two-dimensional," "pre-determined," "specific-pixel," etc.) may be used interchangeably with their corresponding non-hyphenated versions (e.g., "two dimensional," "pre determined," "specific pixel," etc.) on occasion, and capitalized terms (e.g., "Counter Clock," "Row Select," "PIXOUT," etc.) may be used interchangeably with their corresponding non-capitalized versions (e.g., "counter clock," "row select," "pixout," etc.). Such occasional interchangeable use should not be considered inconsistent with each other.

[0013] It should also be noted that the various figures (including component diagrams) shown and discussed herein are for illustrative purposes only and are not drawn to scale. Similarly, the various waveforms and timing diagrams are shown for illustrative purposes only. For example, for clarity, the dimensions of some elements may be exaggerated relative to other elements. Additionally, if deemed appropriate, reference numerals are repeated in the figures to indicate corresponding and / or similar elements.

[0014] The terms used herein are for the purpose of describing some example embodiments only and are not intended to limit the claimed subject matter. The singular forms as used herein are also intended to include the plural forms unless the context clearly dictates otherwise. It will also be understood that when the terms "comprises" and / or "comprising" are used in this specification, it indicates the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Unless clearly defined otherwise, the terms "first", "second", etc. as used herein serve as labels for the nouns following them and do not denote any type of order (e.g., spatial, temporal, logical, etc.). In addition, the same reference numerals may be used between two or more drawings to represent components, elements, blocks, circuits, units, or modules having the same or similar functions. However, such usage is only for the purpose of simplifying the description and facilitating discussion and does not mean that the construction or structural details of such components or units are the same among all embodiments or that such commonly referenced components / modules are the only way to implement some of the example embodiments disclosed herein.

[0015] It will be understood that when an element or layer is referred to as being "on", "connected to", or "coupled to" another element or layer, it can be directly "on", directly connected to, or directly coupled to the other element or layer, or intervening elements or layers may be present. In contrast, when an element or layer is referred to as being "directly on", "directly connected to", or "directly coupled to" another element or layer, no intervening elements or layers are present. The same reference numerals always denote the same elements. As used herein, the term "and / or" includes any combination and all combinations of one or more of the associated listed items.

[0016] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this subject matter belongs. It will also be understood that terms, such as those defined in a general dictionary, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless clearly so defined herein.

[0017] As used herein, the term "module" refers to any combination of software, firmware, and / or hardware configured to provide the functionality described in connection with the module herein. The software may be implemented as a software package, code, and / or instruction set or instructions, and the term "hardware" as used in any of the embodiments described herein may separately include, for example, hardwired circuitry, programmable circuitry, state machine circuitry, and / or firmware storing instructions executed by the programmable circuitry, or may include any combination of hardwired circuitry, programmable circuitry, state machine circuitry, and / or the firmware. The modules may be implemented jointly or separately as circuitry forming part of a larger system (e.g., but not limited to, an integrated circuit (IC), a system on a chip (SoC), etc.). The various components and / or functional blocks disclosed herein may be implemented as modules that may include software, firmware, and / or hardware providing the functionality described in connection with the various components and / or functional blocks.

[0018] The subject matter disclosed herein relates to methods and systems for sparsifying the weights of a deep neural network (DNN). The sparsity of a DNN model processed using the subject matter disclosed herein has the same number of non-zero weights for each group. The groups may be created based on the DNN hardware architecture and may depend on how activations and weights are fetched from memory to processing elements. During the training phase, the DNN model weights and weight threshold parameters are trained simultaneously such that the DNN model can achieve maximum training accuracy and the minimum number of non-zero weights. At the same time, by forcing the variance of the number of non-zero weights between groups to be close to zero, the number of non-zero weights within each group is approximately the same. At the end of training, a post-processing procedure may be applied to achieve the result that each group has exactly the same number of non-zero weights as the best balanced sparsity.

[0019] The subject matter disclosed herein provides an automatic pruning method, wherein the automatic pruning method trains parameters simultaneously, reduces the number of non-zero parameters, and balances the number of non-zero parameters of hardware processing elements to increase the utilization rate of the processing elements of the system. The post-processing step is a process at the end of training for adjusting the weights such that all groups have exactly the same number of weights to achieve the perfect balance of non-zero weights.

[0020] The training of a DNN model can be a numerical optimization problem that seeks a set of weight parameters that minimizes the difference between the prediction and the ground truth. The minimization problem can be written as:

[0021] argmin w (L net ) (1)

[0022] where L net is the network error (i.e., the difference between the prediction and the ground truth t), i.e.,

[0023] Lnet = [y (L) - t] 2 (2)

[0024] where y (L) is the output from the last layer and L is the number of layers.

[0025] To perform sparsification, a pruning function can be applied to the weights such that weights with a magnitude below a threshold are set to 0 and weights with a magnitude above the threshold are unaffected. To have a trainable pruning function, the function needs to be first-order differentiable. In one embodiment, a threshold function or pruning function that can be used is

[0026]

[0027] where w is the weight value of the neural network, α is a parameter that controls the sharpness of the threshold function h(w), and β is a parameter that controls the distance between the first and second boundaries of the threshold (or in other words, the width of the threshold).

[0028] Figure 1 A graph depicting an example threshold function of Equation (3) is described. The parameter α in Equation (3) controls the sharpness of boundaries 101a and 101b of the threshold function h(w), that is, the parameter α controls the rate of change of the boundaries 101a and 101b between h(w) = 0 and h(w) = 1. The parameter β in Equation (3) controls the width 102 between the two boundaries 101a and 101b at h(w) = 0.5. As Figure 1 described, the threshold function h(w) has a value of 0 for a first set 103 of continuous weight values centered at 0. Additionally, the threshold function h(w) has a value of 1 for a second set 104 of continuous weight values greater than the first set 103 of continuous weight values and has a value of 1 for a third set 105 of continuous weight values less than the first set 103 of continuous weight values. The first boundary 101a of the threshold function h(w) is between the first set 103 of continuous weight values and the second set 104 of continuous weight values. The second boundary 101b is between the first set 103 of continuous weight values and the third set 105 of continuous weight values. In the region of the first boundary 101a and the second boundary 101b, the value of the threshold function h(w) transitions between 0 and 1. When the threshold function h(w) transitions between 0 and 1, the sharpness of each of the first boundary 101a and the second boundary 101b is based on the value of the parameter α (i.e., the sharpness of the boundaries 101a and 101b of the threshold function h(w)), and the distance 102 between the first boundary 101a and the second boundary 101b at h(w) = 0.5 is based on the value of the parameter β.

[0029] A regularization term can be introduced to increase the loss function

[0030] L = L net + ρυ 2 (4)

[0031] Where ρ is a regularization parameter selected based on experience. The loss function L will be minimized to optimize the balance or sparsification of the pruning of the DNN model.

[0032] Figure 2 is a flowchart of method 200 for balanced self - pruning of a DNN according to the subject matter disclosed herein. At 201, method 200 starts. To assign sparse weights to multiple processing elements (PEs), at 202, the weights of the DNN are divided into multiple groups.

[0033] Within each group, the number of non - zero weights should be the same. An analytical masking function m(w) having a form similar to equation (3) can be applied to the weights of the DNN. For example, the masking function of equation (5) can be used.

[0034]

[0035] If the magnitude of the weight is greater than the threshold, the masking function of equation (5) returns 1, and if the magnitude of the weight is less than the threshold, the masking function of equation (5) returns 0. The functions of equation (3) and equation (5) are differentiable, and thus, the function parameters are trainable. After the application of the masking function at 203, at 204, the count of the number of non - zero weights within each group can be determined, where the count of the number of non - zero weights within each group is the sum of the return values of the masking function:

[0036] N ≠0 = ∑ i m(w i ) (6)

[0037] where i is the index of the weights within the group.

[0038] To make all groups have the same number of non - zero weights, the variance of N (≠0) should be zero. At 205, the variance of the count of non - zero weights between multiple groups is determined. At 206, the training problem can be formulated as a minimization problem with a hard constraint:

[0039] argmin w，β {L}, s.t. Var{N ≠0}= 0 (7)

[0040] The Lagrange multipliers, the method of dual ascent, or the alternating direction method of multipliers (ADMM) can be used to solve this minimization problem with hard constraints. In essence, the Lagrange multipliers can be used to connect the two terms in equation (7):

[0041] argmin w，β，λ {L + λVar(N ≠0 )} (8)

[0042] At 207, the three parameters w, β, and λ are trainable and updated during backpropagation. Gradient descent techniques can be used to update the parameters w and β. However, gradient ascent techniques can be used to update the Lagrange parameter (or Lagrange multiplier) λ, for example:

[0043] λ t+1 ← λ t + ηVar(N ≠0 ) t (9)

[0044] Since the total loss in equation 8 is decreasing and the Lagrange multiplier λ is increasing, this strictly enforces the hard constraint in equation 7. At 208, method 200 ends. Using the above method 200, the training will eventually converge to a minimum with an approximate balance of sparsity between groups.

[0045] To achieve a truly balanced sparsity, a post - processing method can be applied. After balanced self - pruning (i.e., Figure 2 ), the number of non - zero weights may have a partial sparse pattern. Post - processing can be applied to selectively set the mask of fringe weights to 0 or 1 to achieve a truly sparse balance.

[0046] Figure 3 is a flowchart of an exemplary embodiment of a post - processing method 300 according to the subject matter disclosed herein. At 301, method 300 starts, where 301 can be at the end 208 of method 200 in Figure 2 . For example, at 302, consider: at 202 in Figure 2 , the total number of weights for each group is equal to 1024. The result after balanced self - pruning at 207 can be a pattern where most groups have 500 non - zero weights, while in some groups the weights may all be zero.

[0047] At 303, to achieve a truly balanced sparsity, if a group has more non - zero weights than the pattern, the mask of the minimum non - zero weights of that group will be set to 0 until the number of non - zero weights of that group is equal to the number of non - zero weights of the pattern.

[0048] At 304, if the group has fewer non - zero weights than the pattern, the maximum (masked) zero weights are reset (i.e., unmasked) to non - zero until the number of non - zero weights in the group equals the number of non - zero weights in the pattern. Since the weights are masked using a soft - masking function (such as the soft - masking function in Equation (5)), the post - processing is feasible. At 305, method 300 ends.

[0049] Figure 4 Describe an example embodiment of a balanced self - pruner according to the subject matter disclosed herein. The balanced self - pruner 400 may include a memory 401, a masker 402, and a loss determiner 403. In some embodiments, the balanced self - pruner 400 may include a controller 404 that can provide control and iterative - type functions. In another embodiment, the balanced self - pruner 400 may be implemented as a state machine. In one embodiment, the balanced self - pruner 400 may be implemented as a minimization module, where the minimization module can be any combination of software, firmware, and / or hardware configured to provide the minimization and arithmetic functions of the balanced self - pruner 400. In another embodiment, one or more of the functional blocks of the balanced self - pruner 400 may be modules, in which case such modules can be any combination of software, firmware, and / or hardware configured to provide the functions described in connection with a particular module herein.

[0050] In addition to the intermediate values generated by the masker 402 and the loss determiner 403, the memory 401 may also store the weights w of the DNN to be self - pruned. In the memory 401, the weights w may be grouped. The masker 402 may include a first multiplier 405 that receives the weights w and a masking function m(w) as inputs. The output of the first multiplier 405 is summed in a first adder 406 to determine the count of non - zero weights in the group. The output of the masker 402 may be stored in the memory 401. The masker 402 may operate in an iterative manner such that the masking function m(w) is applied to each weight in each group of weights.

[0051] The loss determiner 403 may include a second adder 407, a second multiplier 408, and a third adder 409. The second adder 407 receives the count of non - zero weights in the group as an input and subtracts the average count value AVE of all groups. N≠0 The output of the second adder 407 is input to the second multiplier 408. The second multiplier 408 squares the output of the second adder 407 and multiplies the product by the Lagrange multiplier λ. The third adder 409 subtracts the output of the second adder 408 from the loss L of the network, and the output of the third adder 409 is stored in the memory 401 for evaluation to minimize the loss L. Here, the output of the third adder 409 can be used as the loss L. Post - processing can utilize the masker 402 to achieve a definite sparsity balance.

[0052] As those skilled in the art will recognize, the innovative concepts described herein can be modified and changed over a wide range of applications. Accordingly, the scope of the claimed subject matter should not be limited to any of the specific exemplary teachings discussed above, but is defined by the claims.

Claims

1. A system for providing balanced pruning of weights of a deep neural network, the system comprising: A plurality of parallel processing elements; A memory storing weights of a deep neural network, wherein in the memory, the weights of the deep neural network are divided into a plurality of weight groups; A weight masker masking the weights in each weight group of the plurality of weight groups of the deep neural network; and A loss determiner determining the loss of the deep neural network based on the variance of the count of non-zero weights in the masked plurality of weight groups subtracted from the network loss of the deep neural network, wherein the loss of the deep neural network is minimized by the loss determiner using a Lagrange multiplier with the constraint that the variance of the count of non-zero weights in the plurality of weight groups is equal to 0, wherein the plurality of weight groups and the Lagrange multiplier are retrained by backpropagation, wherein a post-processing procedure is applied to achieve that each weight group has exactly the same number of non-zero weights to balance the number of non-zero weights to be assigned to the plurality of parallel processing elements, wherein the plurality of weight groups having the same number of non-zero weights are assigned to the plurality of parallel processing elements to ensure that the plurality of parallel processing elements have the same amount of workload.

2. The system according to claim 1, wherein, The loss determiner also determines the loss of the deep neural network based on the variance of the count of non-zero weights in the plurality of weight groups multiplied by the Lagrange multiplier subtracted from the network loss of the deep neural network.

3. The system according to claim 2, wherein The weight masker multiplies the weights in each weight group by a masking function.

4. The system according to claim 3, wherein The masking function is differentiable.

5. The system according to claim 4, wherein The sharpness of the masking function is selectively controllable, and the width of the masking function is selectively controllable.

6. A method for providing balanced pruning of weights of a deep neural network, the method comprising: Storing the weights of the deep neural network by a memory; Dividing the weights of the deep neural network into a plurality of groups in the memory; Applying a masking function to the weights in each group by a weight masker; After the masking function is applied, determining the count of the number of non-zero weights in each group; Determining the variance of the count of non-zero weights in the plurality of groups; Minimizing the loss function of the deep neural network by a loss determiner using a Lagrange multiplier with the constraint that the variance of the count of non-zero weights in the plurality of groups is equal to 0; Retraining the plurality of groups of weights and the Lagrange multiplier by backpropagation, wherein a post-processing procedure is applied to achieve that each group has exactly the same number of non-zero weights to balance the number of non-zero weights to be assigned to a plurality of parallel processing elements, wherein the plurality of groups having the same number of non-zero weights are assigned to the plurality of parallel processing elements to ensure that the plurality of parallel processing elements have the same amount of workload.

7. The method according to claim 6, wherein The Lagrange multiplier is used to multiply the variance of the count of non-zero weights in the plurality of groups.

8. The method according to claim 6, further comprising: Determining the pattern of the retrained weights; If the first of the multiple groups has more non - zero weights than the pattern, adjust the masking function applied to the first group to 0 until the count of non - zero weights in the first group equals the count of non - zero weights in the pattern of the retrained weights; And If the second of the multiple groups has fewer non - zero weights than the pattern, adjust the masking function applied to the second group to 1 until the count of non - zero weights in the second group equals the count of non - zero weights in the pattern of the retrained weights.

9. The method according to claim 6, wherein, The masking function is differentiable.

10. The method according to claim 9, wherein, The sharpness of the masking function is optionally controllable, and the width of the masking function is optionally controllable.

11. The method according to claim 6, wherein, The step of retraining the weights and the Lagrange multiplier also includes: retraining the weights, the width of the masking function, and the Lagrange multiplier through backpropagation.

12. The method according to claim 11, wherein, Use gradient descent techniques to update the weights and the width of the masking function, and use gradient ascent techniques to update the Lagrange multiplier.

Images