Group balanced sparse activation feature map for neural network models

Abstract

Embodiments disclose a group-balanced sparse activation neural network model and a method of generating a group-balanced sparse activation neural network model. According to one embodiment, a neural network sparsification engine determines a first deep neural network (DNN) model having two or more hidden layers. The engine determines a group size, a group layout, and a target sparsity. The engine partitions activation feature maps into a plurality of groups based on the group size and the group layout. The engine generates a second DNN model by increasing a sparsity of each activation feature map group based on the target sparsity, wherein the second DNN model is used for inference.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. nonprovisional patent application No. 17 / 038557, filed on September 30, 2020, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The embodiments of this disclosure generally relate to artificial intelligence. More specifically, embodiments of this disclosure relate to methods and systems for group-balanced sparse activation feature maps for deep neural network models. Background Technology

[0004] Deep neural networks (DNNs) have become a widely used method in artificial intelligence (AI) for extracting high-level information from low-level data such as images, videos, audio, and text. However, the high computational cost of DNNs hinders their use in applications with tight budgets in terms of energy consumption, storage space, or latency tolerance, particularly on edge devices such as mobile phones and surveillance cameras.

[0005] The computational cost of DNN models stems from several factors. First, the number of parameters in a DNN model can be in the millions or tens of millions, leading to enormous storage costs and hindering the storage of these parameters in memory. Second, the sheer number of neurons in a DNN model consumes a significant amount of memory and may require billions of arithmetic operations during runtime. Third, search engines based on vector representations generated by neural networks, such as face comparison engines, can have very high computational costs, partly due to the high-dimensional, dense vector representations (embeddings) of DNNs. Attached Figure Description

[0006] Embodiments of this disclosure are illustrated by way of example rather than limitation in the accompanying drawings, wherein the same reference numerals denote the same elements.

[0007] Figure 1 This is a block diagram illustrating an example of a deep neural network (DNN) model with group-balanced sparse activation feature maps according to one embodiment.

[0008] Figures 2A to 2B This is a block diagram illustrating a neural network sparsity engine according to one embodiment.

[0009] Figure 3 This is a block diagram illustrating the layers of a DNN model according to one embodiment.

[0010] Figure 4 This is a block diagram illustrating activation feature map sparsification performed by updating the weights of layers in a DNN model according to one embodiment.

[0011] Figure 5A This is a block diagram illustrating the group-balanced sparse activation tensor of a layer in a DNN according to one embodiment.

[0012] Figure 5B This is a block diagram illustrating another example of a group-balanced sparse activation tensor for a layer of a DNN according to one embodiment.

[0013] Figure 6 This is a block diagram illustrating a two-dimensional activation tensor according to one embodiment.

[0014] Figure 7 This is a block diagram illustrating three examples of group arrangements with three-dimensional activation tensors according to some embodiments.

[0015] Figure 8 This is a flowchart illustrating the process of generating a group-balanced sparse DNN according to one embodiment.

[0016] Figure 9 This is a flowchart illustrating a method according to one embodiment. Detailed Implementation

[0017] Various embodiments and aspects of this disclosure will be described with reference to the details discussed below, and the accompanying drawings will illustrate various embodiments. The following description and drawings are illustrative of the disclosure and should not be construed as limiting the disclosure. Numerous specific details are described to provide a thorough understanding of various embodiments of this disclosure. However, in some cases, well-known or conventional details have not been described for the sake of a concise discussion of embodiments of this disclosure.

[0018] The phrase "an embodiment" or "one embodiment" as used in this specification means that a particular feature, structure, or characteristic described together with that embodiment may be included in at least one embodiment of this disclosure. The phrase "in one embodiment" appearing throughout the specification does not necessarily refer to the same embodiment.

[0019] The following definitions apply to the elements and operations described herein. These definitions are helpful in describing embodiments of this disclosure, but are not intended to be construed as limiting.

[0020] Acceleration refers to modifying the DNN to reduce the number of arithmetic operations required to compute its functions.

[0021] Group-balanced sparse weights are a sparse pattern balanced across one or more weight banks, such that the number of non-zero weight elements is the same in each weight bank. Group-balanced sparsity is applicable to the weights of layers in DNN models.

[0022] Group-balanced sparse activation is a sparse pattern balanced across one or more feature map groups, such that the number of non-zero output activation elements is the same in each feature map group. Group-balanced sparsity is applicable to the output activation feature maps of layers in DNN models.

[0023] A Deep Neural Network (DNN) model is a composite function consisting of several layers of elementary functions. Each function takes the outputs of the preceding layers as input and calculates a new set of outputs. The output of each layer is called a neuron. If a neuron has a non-zero value, it is activated (or active); otherwise, it is deactivated (or inactive).

[0024] The objective function is a function that is optimized during training to measure the performance of a DNN on a specific task associated with a specific training dataset.

[0025] Sparsity refers to the methodology of training a DNN (i.e., adjusting the parameters of the DNN) so that the number of activated neurons is significantly smaller than the total number of neurons (i.e., at least less than half, usually less than a quarter).

[0026] A tensor is a data structure used to store the parameters of a neural network, such as a vector or matrix of any dimension.

[0027] The embodiments disclose a group-balanced sparse activation neural network model and a method for generating such a model. According to one aspect, a neural network sparsification engine determines a first deep neural network (DNN) model with two or more hidden layers. The engine determines a group size, group layout, and a target sparsity. The engine segments activation feature maps into multiple groups based on the group size and group layout. The engine generates a second DNN model by increasing the sparsity of each activation feature map group based on the target sparsity, wherein the second DNN model is used for inference.

[0028] According to a second aspect, a group-balanced sparse activation deep neural network (DNN) model is disclosed. The group-balanced sparse activation DNN model includes an input layer, one or more hidden layers, and an output layer, wherein the one or more hidden layers are disposed between the input layer and the output layer, and each hidden layer is for outputting a group-balanced sparse activation feature map. The group-balanced sparse activation DNN model is generated by: determining a first DNN model with one or more hidden layers; determining a group size, group layout, and target sparsity; dividing the activation feature map into multiple groups based on the group size and group layout; and generating a second DNN model as the group-balanced sparse activation DNN model by increasing the sparsity of each activation feature map group based on the target sparsity, wherein the second DNN model is used for inference.

[0029] Figure 1 This is a block diagram illustrating an example of a deep neural network (DNN) model with group-balanced sparse activation feature maps according to one embodiment. (See also...) Figure 1 The DNN model 100 includes a deep neural network model having one or more inner / hidden layers 101-103 positioned between an input layer 111 and an output layer 113. Each inner layer 101-103 may have one or more tensors with activation functions (not shown) and weights / biases (not shown), which generate output activation feature maps 123-125, 113 for each layer when multiplied by the input feature maps 111, 123-125. Here, one or more output activation feature maps are group-balanced sparse feature maps (group-balanced sparse feature maps are discussed further below). Types of DNN models include, but are not limited to, convolutional NNs where each layer is fully connected to the previous and next layers, partially connected DNNs, recurrent NNs, Long Short-Term Memory (LSTM) DNN models, etc. Although it is possible to use... Figure 1 The DNN is used to describe some implementation examples, but the range of DNN models should not be understood as limited.

[0030] Figures 2A to 2B This is a block diagram illustrating a neural network sparsity engine according to one embodiment. (See reference...) Figures 2A to 2B The neural network sparsity engine 200 can generate group-balanced sparse DNN models. In one embodiment, the neural network sparsity engine 200 includes the following inputs: training data and evaluation metrics (e.g., an objective function), a pre-trained DNN model (optional), the number of groups, and a sparsity rate, and the neural network sparsity engine 200 includes the following output: a DNN model with group-balanced sparse activation feature maps. In one embodiment, the engine 200 can generate a pre-trained DNN model based on the input training data and evaluation metrics (e.g., an objective function).

[0031] In one embodiment, group size, group layout, and sparsity are set with default or initial values, such as group size = 2, 4, 8, 16; group layout = columns, rows, channels; sparsity = 20%, 40%, 60%, etc. In another embodiment, for a device / system with engine 200, group size, group layout, and sparsity can be configured by the user. In yet another embodiment, block size / layout and sparsity can be automatically configured based on one or more hyperparameters / types / tensor dimensions of the neural network. For example, for a natural language processing DNN, the tensor dimension of the neural network could be the sentence length obtained through word embeddings, where each word embedding is a vector representing each word with its associated sentiment.

[0032] Here, the number of groups in the first dimension can be determined based on the sentence length dimension; for example, the number of groups equals the sentence length divided by val, where val is a numeric data type representing the group size. The number of groups in the second dimension can be determined based on the word embedding dimension; for example, the number of groups equals the word embedding divided by val, where val is a numeric data type representing the group size. Figure 5A An example of group-balanced activation with a group size of 4 is shown in the figure, while Figure 5B An example of group-balanced activation with a group size of 2 is shown. The sparsity rate can be set to a default value. For example, the sparsity rate can be set to 50%. In another embodiment, the sparsity rate can be automatically configured to increase as long as the evaluation metric (objective function) converges and the weights / biases of the DNN model are not adjusted, for example, the output is within a predetermined threshold of a predetermined number of iterations.

[0033] Reference Figure 2B In one embodiment, the neural network sparsity engine 200 can be implemented as part of a data processing system, server, computing system, edge device, etc. Engine 200 can generate group-balanced sparse deep neural network models, such as... Figure 1 Model 100. (Refer to...) Figure 2B Engine 200 may include modules such as DNN generator 201, group number determiner 202, target sparsity determiner 203, activation group determiner 204, DNN trainer 205, and sparsity generator 206.

[0034] In one embodiment, the DNN generator 201 can generate a DNN model based on one or more training datasets and an objective function. The training data may include input data and output labels for a specific task (e.g., classification, object detection, face recognition, language processing, etc.). In one embodiment, the DNN generator 201 can generate a sparse DNN model based on a pre-trained DNN model. A group number determiner 202 can determine the value / number of groups for the output activation feature map groups, so these groups can be retrieved based on the group number. A target sparsity determiner 203 can determine the target sparsity of the sparse DNN model (e.g., 20%, 40%, 60%, 80%, etc.). An activation group determiner 204 can group / divide the feature maps into several groups based on group size / layout. It should be noted that each neural network layer outputs an activation feature map, and the output activation feature map is the input to the next layer.

[0035] DNN trainer 205 can train or retrain a DNN model through forward and backward propagation. Sparser 206 can apply sparsity to the DNN model by sparsifying the weights or activation feature maps of any layer. It should be noted that some or all of modules 201-206 can be implemented in software, hardware, or a combination of both. For example, these modules can be installed in storage device 212, loaded into memory 211, and executed by one or more processors 210. Some of modules 201-206 can be integrated together to form an integrated module.

[0036] Figure 3 This is a block diagram illustrating the layers of a DNN model according to one embodiment. Layer 300 can be a layer of a deep neural network, such as... Figure 1 Layer 101. (Refer to...) Figure 3 Layer 300 can be represented by y = activation_fn(Wx + b), where x represents the input tensor, y represents the output tensor (or output activation feature map), W represents the weight tensor, b represents the bias tensor, and activation_fn represents the activation function. Examples of activation functions include the Rectified Linear Array (ReLU), the sigmoid function, and the hyperbolic tangent function (tanh).

[0037] Figure 4 This is a block diagram illustrating the sparsification of activation feature maps of a DNN layer by updating the layer weights according to one embodiment. Layer 400 can represent... Figure 3 Layer 300, but with activated pruning (or activated sparsification). See reference... Figure 4 To enable activation pruning, one or more activation feature maps (e.g., output y) are set to zero based on the target sparsity. For example, y = activation_fn(Wx+b)·t, where t is the mask of the output tensor of activation_fn(Wx+b) for that layer, "·" is the element-wise product operation, and y is the output. Here, the weight value 401 can be updated, for example, W → W', where W' is the updated weight tensor. Based on the new weight tensor, the activation feature map is: y' = activation_fn(W'x+b), where y' is the output activation. Here, the activation-pruned DNN model, for example, a DNN model with pruned activation tensors (i.e., feature maps) for one or more layers, uses fewer computations (e.g., multiplication and / or addition) for inference, resulting in a faster and more efficient DNN model. It should be noted that the mask t is a binary vector that is dynamically derived based on the inputs x and t, where t is a function of activation_fn(Wx+b).

[0038] An example of a mask t can be obtained by selecting the top k (top-k) items of each group's tensor activation_fn(Wx+b), setting the element corresponding to the top k items of each group to 1, and setting the element corresponding to the other items to 0. This results in the total number of non-zero elements being k times the number of groups.

[0039] In one embodiment, for a DNN model with L layers, it has an output feature map at the i-th layer. In one embodiment, activating pruning can be represented as a mask: in i =1,...,L-1; Given that the output feature map is a three-dimensional (3D) tensor, c i n and d i It represents the size of each dimension of the 3D output feature map.

[0040] Figure 5A This is a block diagram illustrating the group-balanced sparse activation tensor of a layer in a DNN according to one embodiment. (See also...) Figure 5A Layer 500 can represent Figure 4 Layer 400. In one embodiment, layer 500 outputs a dense activation matrix Y1, where Y1 represents the activation feature map tensor 501 before activation sparsification. For an example, for group size = 4, group layout = rows, and target sparsity = 50%, the sparsification engine (e.g., ...) Figures 2A-2B Engine 200 can generate a group-balanced sparse feature map 502 based on feature map 501. It should be noted that sparse activation can provide hardware acceleration.

[0041] In one embodiment, feature map 501 is generated by determining the dimensions used to apply sparsification. (See reference...) Figure 5A In this example, Engine 200 can be automatically configured to apply sparsity to the row dimension as the group layout. Here, Engine 200 divides each row of the feature map according to the group size; in this case, a group size of 4 results in Group 1 and Group 2. For each group, Engine 200 sorts the values ​​from largest to smallest. For example, Group 1 has the sorted values ​​0.8, 0.2, 0.1, and -0.4. Based on the target sparsity, the largest values ​​are selected as activation values, while the remaining values ​​are set to zero. Here, 0.8 and 0.2 are selected as activation values ​​for Group 1, while the remaining values ​​are set to zero (i.e., deactivated) to achieve the target sparsity of 50%.

[0042] Engine 200 applies this selection process to other groups of layer 500 to generate feature map 502. In one embodiment, the selection is performed stepwise and / or iteratively, for example, zeroing out individual activation values ​​from a single group one at a time. In another embodiment, activation values ​​are zeroed based on a threshold, for example, an activation value in a group is set to zero if it is less than a threshold. In one embodiment, each group has a different threshold.

[0043] Figure 5B This is a block diagram illustrating the group-balanced sparse activation tensor of a layer in a DNN according to another embodiment. (See also...) Figure 5B Tensor 503 illustrates group-balanced sparsity applied to the column dimensions of tensor 501 (here, engine 200 can be configured by the user to apply sparsity to the column dimensions). See reference... Figure 5B Based on group size / layout, each column is divided into a top group (group 1) and a bottom group (group 2), for example, group size = 2 and group layout = column. For each group, the activation values ​​can be sorted, and the top k largest activation values ​​are retained, while the remaining activation values ​​are set to zero.

[0044] For example, based on a target sparsity rate of 50%, each group with two activation values ​​is reduced to one activation value (50% of the target rate) to generate a group-balanced sparse feature map 503. In one embodiment, the dimension (or component) to be sparsified, the group size / layout, and / or the sparsity rate can be configured by an operator using the accelerator / processing system / server / edge device of engine 200. In another embodiment, the dimension, group size / layout, and / or sparsity rate can be automatically configured based on the dimension / components or hyperparameters of tensor 501.

[0045] Figure 6 This is a block diagram illustrating a two-dimensional (2D) activation tensor according to one embodiment. (See also...) Figure 6 Layer 600 can represent Figure 5A Layer 500, while tensor 601 can represent Figure 5A Tensor 501. (Refer to...) Figure 6 In one embodiment, layer 600 outputs a 2D activation tensor 601. In this case, tensor 601 can be the activation tensor of a natural language processing deep neural network, and tensor 601 includes the following dimensions: sentence length dimension and hidden dimension. The hidden dimension can include the word embedding dimension, or any other dimension value of the natural language processing deep neural network.

[0046] Figure 7 This is a block diagram illustrating an activation tensor with three dimensions (3D) according to one embodiment. (See also...) Figure 7 Layer 700 can represent Figure 5A Layer 500, while tensor 701 can represent Figure 5ATensor 501. (Refer to...) Figure 7 In one embodiment, layer 700 outputs a 3D activation tensor 701. In this case, tensor 701 can be the activation tensor of an image recognition / classification deep neural network, and tensor 701 includes the following dimensions: height, width, and channels. For example, a red-green-blue (RGB) image includes a red channel, a green channel, and a blue channel, and each channel includes an image of size height × width. For an RGB image with depth, the channels can include red, green, blue, and depth. It should be noted that different layers of the neural network can generate activation outputs with different numbers of channel dimensions, height dimensions, and / or width dimensions.

[0047] Reference Figure 7 In one embodiment, groups can be automatically configured based on the number of color channels in each activation feature map. Here, each channel (e.g., channel 711) can be considered a group, and the top k largest values ​​from each group can be selected based on a target sparsity rate to generate group-balanced sparse activation. In another embodiment, groups can be configured automatically or by an operator. In yet another embodiment, a channel can be divided into several groups (e.g., channel 712 is divided into 4 groups). Next, the top k largest values ​​from each group can be selected based on a target sparsity rate to generate group-balanced sparse activation. In yet another embodiment, several channels can be merged into one group (e.g., group 713 includes two or more channels). Similarly, the top k largest values ​​from each group can be selected based on a target sparsity rate to generate group-balanced sparse activation. Although two-dimensional and three-dimensional activation tensors are shown, activation tensors can be of any dimension, such as four-dimensional, five-dimensional, six-dimensional, etc.

[0048] Figure 8 This is a flowchart illustrating a process for generating a group-balanced sparse DNN according to one embodiment. Process 800 can be executed by processing logic, which may include software, hardware, or a combination of software and hardware. For example, process 800 may be executed by… Figure 2A The neural network sparsity engine 200 executes this, for example, by an accelerator, processor, server, edge device, or host system having engine 200. (See also...) Figure 8 In block 801, the processing logic receives training data and an objective function for a neural network model. The training data may include one or more datasets used for training the neural network. The training data may be a set of examples used to fit the parameters of the neural network model (e.g., the weights and bias connections between neurons in the neural network model). The objective function describes the inference objective (e.g., classification, object recognition tasks) and may include evaluation criteria to determine how well the network model parameters fit the training data for a specific objective.

[0049] In block 802, the processing logic obtains a DNN model (e.g., a first DNN model) and checkpoints. The DNN model can be obtained by training a target function using training data, or it can be a pre-trained DNN model. Checkpoints can include those that demonstrate a percentage of accuracy (e.g., 89%) for a specific task when a particular training dataset is applied to the DNN model. In one embodiment, the target sparsity can be automatically reconfigured based on checkpoints. For example, the target sparsity can initially be set to 80%, and if the sparsed DNN model generated based on this target sparsity maintains the percentage of accuracy for a specific task within a predetermined threshold of the checkpoints, the target sparsity is automatically increased to generate the next sparsed DNN model. This automatic reconfiguration of the target sparsity can be repeated. In this way, a target sparsity can be automatically selected for the DNN to achieve optimal speedup.

[0050] In block 803, the processing logic obtains one or more group sizes / layouts (to derive the number of groups) and a target sparsity value. The group size / layout / target sparsity can be specified individually for each layer of the DNN, or the same group size / layout / sparseness can be applied to every layer of the DNN model. In one embodiment, the group size / layout yields the number of groups, which is divisible by the tensor dimension of the activation tensors of the DNN model. The target sparsity value can be a percentage value, such as 40%, 50%, 60%, etc. In block 803, the processing logic further divides the activation feature maps (e.g., the layer outputs) into several groups based on the group size / layout. The processing logic then determines the sparsity rate for each group.

[0051] In block 804, the processing logic evaluates whether the target sparsity has been achieved based on the sparsity rate of each group. If yes, the process continues to block 805, and the DNN model is output as a group-balanced sparse activation DNN model (e.g., a second DNN model). If not, the processing logic continues to block 806. In block 806, the processing logic determines whether the objective function has converged. In one embodiment, the objective function converges if the change in the last predetermined number of iterations is within a predetermined threshold. For example, in the last ten iterations, the output generated by the modified DNN model is within the 0.1% threshold range. If the objective function output is within the predetermined threshold range, e.g., the objective function output has converged, then the processing logic continues to block 807. If the objective function has not converged, then the processing logic continues to block 808.

[0052] In block 807, the processing logic increases the sparsity of the activation feature maps at each layer. Here, the sparsity of the activation feature maps is increased by zeroing or removing activation values ​​from the group. For example, refer to... Figure 5AGroup 1, row 1, has activation values ​​of 0.2, 0.1, -0.4, and 0.8. For a sparsity of 75%, sparsity is increased by removing -0.4. In one embodiment, activation values ​​are removed from a group in a layer while keeping the layer balanced across groups. In one embodiment, a threshold can be set for the group, and this threshold can be progressively adjusted to progressively remove activation values ​​smaller than the threshold. In one embodiment, sparsity is increased by removing activation values ​​from each group in each layer of the DNN model. Once sparsity is increased, the processing logic proceeds to block 804 and determines whether the target sparsity for that layer / DNN model has been achieved.

[0053] In block 808, if the objective function fails to converge (e.g., the change in the last predetermined number of iterations is outside a predetermined threshold range), the processing logic forward-propagates the DNN model based on the objective function. Here, input data (e.g., a subset of the training data) is fed forward through the DNN model. Each hidden layer receives the input data, processes it, and passes the output to the subsequent layers. In one embodiment, the weights / biases and outputs computed for each layer are stored for backpropagation.

[0054] In block 809, the processing logic backpropagates the DNN model. Here, the processing logic backpropagates through the neural network to compute layer output values ​​in order to calculate gradients (e.g., the differences from the output values ​​of the forward propagation). In one embodiment, the processing logic loops through each layer in a reverse topological manner, starting from the last layer, to compute the derivative of the last layer's output with respect to each layer. In this way, the processing logic can determine which weight / bias value is causing the majority of the error and can update the weights / bias based on this determination. Once the weights / bias values ​​have been updated, the processing logic continues to block 806. In one embodiment, as shown in blocks 804 and blocks 806 through 809, the processing logic iteratively or incrementally sparsifies the DNN model until the DNN model reaches the target sparsity.

[0055] Figure 9 This is a flowchart illustrating a method according to one embodiment. Process 900 can be executed by processing logic, which may include software, hardware, or a combination of software and hardware. For example, process 900 can be executed by... Figure 2A The neural network sparsity engine 200 executes this, or it is executed by an accelerator, processor, server, edge device, or host system having engine 200. (See also...) Figure 9In block 901, the processing logic determines a first deep neural network (DNN) model with one or more hidden layers. In block 902, the processing logic determines the group size, group layout, and target sparsity. In block 903, the processing logic divides the activation feature maps into groups based on the group size and group layout. In block 904, the processing logic generates a second DNN model by increasing the sparsity of each activation feature map group based on the target sparsity, where the second DNN model is used for inference.

[0056] In one embodiment, the second DNN model is generated by progressively increasing the sparsity of each set of activation feature maps until a target sparsity is reached, and then retraining the second DNN model to ensure that the second DNN model converges in response to the increased sparsity of the activation feature map sets. In one embodiment, grouping the activation feature maps includes: if the activation tensor is a two-dimensional tensor, dividing the activation feature maps into multiple groups based on a group layout, such as the row or column dimensions of the activation tensor of a layer in the first DNN model; and if the activation tensor is a three-dimensional or multi-dimensional tensor, dividing the activation feature maps into multiple groups based on the channel, row, or column dimensions (e.g., group layout) of the activation tensor of a layer in the first DNN model.

[0057] In one embodiment, retraining the second DNN model includes computing the output of the objective function through forward propagation and updating the weight parameters of each hidden layer of the DNN model through backpropagation based on the computed output. In one embodiment, the second DNN model is a group-balanced sparse DNN model.

[0058] In one embodiment, the second DNN model is a dynamic DNN model, wherein the weight parameters of the second DNN model used for the first inference are different from the weight parameters used for the second inference. In one embodiment, the sparsity of each activated feature map patch is progressively increased by retaining the largest k values ​​of each group and zeroing the remaining values ​​of that group, where k is equal to or greater than 1.

[0059] In one embodiment, a group-balanced sparse activation deep neural network (DNN) model includes an input layer, one or more hidden layers, and an output layer. One or more hidden layers are positioned between the input and output layers, and each hidden layer outputs a group-balanced sparse activation feature map. The group-balanced sparse activation DNN model is generated by: determining a first DNN model with one or more hidden layers; determining the group size, group layout, and target sparsity; dividing the activation feature maps into multiple groups based on the group size and group layout; and generating a second DNN model as the group-balanced sparse activation DNN model by increasing the sparsity of each activation feature map group based on the target sparsity, wherein the second DNN model is used for inference.

[0060] In one embodiment, the second DNN model is generated by progressively increasing the sparsity of each set of activation feature maps until a target sparsity is achieved, and then retraining the second DNN model to ensure that it converges in response to the increased sparsity of the activation feature map sets. In one embodiment, grouping the activation feature maps includes: if the activation tensor is a two-dimensional tensor, dividing the activation feature maps into multiple groups based on the row or column dimensions of the activation tensor of the layers of the first DNN model; and if the activation tensor is a three-dimensional or more-dimensional tensor, dividing the activation feature maps into multiple groups based on the channel, row, or column dimensions of the activation tensor of the layers of the first DNN model.

[0061] In one embodiment, retraining the second DNN model includes computing the output of the objective function via forward propagation and updating the weight parameters of each hidden layer of the DNN model via backpropagation based on the computed output. In one embodiment, the second DNN model is a group-balanced sparse DNN model.

[0062] In one embodiment, the second DNN model is a dynamic DNN model, wherein the weight parameters of the second DNN model used for the first inference are different from the weight parameters used for the second inference. In one embodiment, the sparsity of each activated feature map patch is progressively increased by retaining the top k values ​​of each group and progressively zeroing out the remaining values ​​of that group, where k is equal to or greater than 1.

[0063] It should be noted that some or all of the components shown and described above can be implemented in software, hardware, or a combination of software and hardware. For example, such components can be implemented as software installed and stored in persistent storage, which can be loaded into memory by a processor (not shown) and executed in memory to perform the processes or operations described throughout this application. Alternatively, such components can be implemented as executable code programmed or embedded in dedicated hardware, such as integrated circuits (e.g., application-specific integrated circuits or application-specific integrated circuits), digital signal processors (DSPs), or field-programmable gate arrays (FPGAs), and the executable code can be accessed from an application by a corresponding driver and / or operating system. Furthermore, these components can be implemented as specific hardware logic within a processor or processor core, which is part of an instruction set accessible by the software components via one or more specific instructions.

[0064] Some of the parts described in detail above have been presented in terms of the symbolic representation of algorithms and bitwise operations in computer memory. These algorithmic descriptions and representations are the methods by which those skilled in the art of data processing most effectively communicate their work to others skilled in the art. Generally speaking, an algorithm is considered to be a self-consistent sequence of operations that leads to a desired result. These operations are those that require physical manipulation of physical quantities.

[0065] However, it should be remembered that all these and similar terms are associated with appropriate physical quantities and are merely convenient labels applied to those quantities. Unless explicitly stated otherwise from the foregoing discussion, it is to be understood that throughout the description, discussion using terms such as those listed in the appended claims refers to the actions and processes of a computer system or similar electronic computing device that manipulates and converts data represented as physical (electronic) quantities in the registers and memory of the computer system into other data similarly represented as physical quantities in the computer system's memory or registers or other such information storage, transmission, or display devices.

[0066] Embodiments of this disclosure also relate to an apparatus for performing the operations described herein. Such a computer program is stored in a non-transitory computer-readable medium. A machine-readable medium includes any component for storing information in a machine-readable (e.g., computer-readable) form. For example, machine-readable (e.g., computer-readable) media include machine-readable storage media (e.g., read-only memory (ROM), random access memory (RAM), disk storage media, optical storage media, flash memory devices).

[0067] The processes or methods described in the foregoing figures can be executed by processing logic, which includes hardware (e.g., circuits, special-purpose logic, etc.), software (e.g., embodied in a non-transitory computer-readable medium), or a combination of both. Although the processes or methods described above are based on some sequential operations, it should be understood that some of the described operations can be performed in a different order. Furthermore, some operations can be performed in parallel rather than sequentially.

[0068] The embodiments disclosed herein are not described with reference to any particular programming language. It should be understood that various programming languages ​​can be used to implement the teachings of the embodiments of this disclosure described herein.

[0069] In the foregoing specification, embodiments of this disclosure have been described with reference to specific exemplary embodiments thereof. It is obvious that various modifications may be made thereto without departing from the broader spirit and scope of this disclosure as set forth in the appended claims. Therefore, this specification and the accompanying drawings should be considered illustrative rather than restrictive.

Claims

1. A method for generating a deep neural network model, wherein the tensor of the deep neural network model includes the sentence length dimension and the hidden dimension of the natural language processing deep neural network, comprising: Determine the first DNN model with one or more hidden layers; Determine group size, group layout, and target sparsity; Based on the group size and the group layout, multiple activation feature maps are divided into multiple groups; and A second DNN model is generated by increasing the sparsity of each group of activated feature maps based on the target sparsity, wherein the second DNN model is used for inference; Generate the second DNN model based on the first DNN model; Retraining the second DNN model includes: Determine the output tensor corresponding to the activation feature map of the hidden layer of the second DNN model; The activation feature maps are divided into multiple activation feature map groups based on group size and group layout; Determine the dynamic mask of the output tensor; Perform an element-wise product operation between the dynamic mask and the output tensor; and The sparsity of the activation feature map groups is gradually increased until the activation feature map groups reach the target sparsity, so as to ensure that the second DNN model converges in response to the increase of the sparsity of the activation feature map groups and that the number of non-zero elements in each activation feature map group is consistent.

2. The method according to claim 1, wherein, Generating the second DNN model includes: For each set of activated feature maps, the sparsity of the activated feature map set is gradually increased until the activated feature map set reaches the target sparsity.

3. The method according to claim 1, wherein, The step of dividing multiple activation feature maps into multiple groups includes: If the activation tensor is a two-dimensional tensor, then based on the row or column dimension of the activation tensor of the first DNN model's layer, the activation feature map is divided into multiple groups, or If the activation tensor is a three-dimensional or more-dimensional tensor, the activation feature map is divided into multiple groups based on the channel dimension, row dimension, or column dimension of the activation tensor of the first DNN model layer.

4. The method according to claim 2, wherein, Retraining the second DNN model includes: The output of the objective function is calculated through forward propagation; and Based on the calculated output, the weight parameters of each hidden layer of the DNN model are updated via backpropagation.

5. The method according to claim 1, wherein, The second DNN model is a group-balanced sparse DNN model.

6. The method according to claim 1, wherein, The second DNN model is a dynamic DNN model, wherein the weight parameters of the second DNN model used for the first inference are different from the weight parameters used for the second inference.

7. The method according to claim 1, wherein, The sparsity of each activated feature map block is gradually increased by retaining the top k values ​​of each group and progressively zeroing out the remaining values ​​of the group, where k is equal to or greater than 1.

8. A non-transitory machine-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

9. A method for generating a group-balanced sparse activation deep neural network model, wherein the tensor of the group-balanced sparse activation deep neural network model includes the sentence length dimension and the hidden dimension of a natural language processing deep neural network, including: Input layer; One or more hidden layers; and An output layer, wherein the one or more hidden layers are disposed between the input layer and the output layer, and each hidden layer outputs multiple sets of balanced sparse activation feature maps. The group-balanced sparse activation DNN model is generated in the following way: Determine the first DNN model with one or more hidden layers. Determine the group size, group layout, and target sparsity. Based on the group size and the group layout, multiple activation feature maps are divided into multiple groups; and A second DNN model is generated as the group-balanced sparse activation DNN model by increasing the sparsity of each group of activation feature maps based on the target sparsity, wherein the second DNN model is used for inference. Generate the second DNN model based on the first DNN model; Retraining the second DNN model includes: Determine the output tensor corresponding to the activation feature map of the hidden layer of the second DNN model; The activation feature maps are divided into multiple activation feature map groups based on group size and group layout; Determine the dynamic mask of the output tensor; Perform an element-wise product operation between the dynamic mask and the output tensor; and The sparsity of the activation feature map groups is gradually increased until the activation feature map groups reach the target sparsity, so as to ensure that the second DNN model converges in response to the increase of the sparsity of the activation feature map groups and that the number of non-zero elements in each activation feature map group is consistent.

10. The method according to claim 9, wherein, Generating the second DNN model includes: For each set of activated feature maps, the sparsity of the activated feature map set is gradually increased until the activated feature map set reaches the target sparsity.

11. The method according to claim 9, wherein, The step of dividing multiple activation feature maps into multiple groups includes: If the activation tensor is a two-dimensional tensor, then based on the row or column dimension of the activation tensor of the first DNN model's layer, the activation feature map is divided into multiple groups, or If the activation tensor is a three-dimensional or more-dimensional tensor, the activation feature map is divided into multiple groups based on the channel dimension, row dimension, or column dimension of the activation tensor of the first DNN model layer.

12. The method according to claim 10, wherein, Retraining the second DNN model includes: The output of the objective function is calculated through forward propagation; and Based on the calculated output, the weight parameters of each hidden layer of the DNN model are updated via backpropagation.

13. The method according to claim 9, wherein, The second DNN model is a group-balanced sparse DNN model.

14. The method according to claim 9, wherein, The second DNN model is a dynamic DNN model, wherein the weight parameters of the second DNN model used for the first inference are different from the weight parameters used for the second inference.

15. The method according to claim 9, wherein, The sparsity of each activated feature map block is gradually increased by retaining the top k values ​​of each group and progressively zeroing out the remaining values ​​of the group, where k is equal to or greater than 1.

Images