A neural network pruning method and device, and a storage medium

Abstract

The embodiment of the application provides a neural network pruning method and device, and a storage medium, the method comprises the following steps: obtaining a generator network according to a gating matrix of a target neural network, the generator network comprising the gating matrix and the target neural network; performing generative adversarial training on the generator network by using a multimedia data set, obtaining a trained generator network, and determining a sparse neural network corresponding to the target neural network according to the trained generator network.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of electronic applications, and in particular to a neural network pruning method and device, and a storage medium. BACKGROUND

[0002] Deep learning algorithms represented by Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), etc. have been widely applied in multimedia fields such as machine vision, natural language processing, autonomous driving, robots, data analysis, etc. However, the complex structure of these algorithms requires a large amount of power consumption and CPU, memory, storage space and other computing resources even during inference, which severely limits the deployment of such technologies on mobile and embedded platforms with limited power consumption and resources. Therefore, these algorithms must be compressed to some extent before deployment.

[0003] Currently, some parameters with lower weights can be systematically pruned from a target neural network (i.e., the target neural network is pruned), thereby reducing the number of parameters of the neural network and achieving the benefit of reducing resource occupation while minimizing the loss of accuracy. However, the current pruning method for neural networks still needs to further improve the accuracy of identifying parameters with lower weights. SUMMARY

[0004] The embodiments of the present application provide a neural network pruning method and device, and a storage medium, which are beneficial to accurately pruning parameters with lower weights in a neural network.

[0005] The technical solution of the present application is implemented as follows:

[0006] In a first aspect, the embodiments of the present application provide a neural network pruning method, which comprises:

[0007] obtaining a generator network according to a gating matrix of a target neural network, wherein the generator network comprises the gating matrix and the target neural network;

[0008] performing generative adversarial training on the generator network using a multimedia data set to obtain a trained generator network, and determining a sparse neural network corresponding to the target neural network according to the trained generator network.

[0009] In a second aspect, the embodiments of the present application provide a neural network pruning device, which comprises:

[0010] The creating unit is configured to obtain a generator network according to a gating matrix of the target neural network, the generator network comprising the gating matrix and the target neural network.

[0011] The training unit is configured to perform generative adversarial training on the generator network by using a multimedia data set, to obtain a trained generator network, and to determine a sparse neural network corresponding to the target neural network according to the trained generator network.

[0012] In a third aspect, an embodiment of the present application provides a neural network pruning device, which comprises a processor and a memory.

[0013] In a fourth aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method according to any one of the preceding aspects.

[0014] The embodiments of the present application provide a neural network pruning method and device and a storage medium. The method comprises the following steps: obtaining a generator network according to a gating matrix of a target neural network, the generator network comprising the gating matrix and the target neural network; performing generative adversarial training on the generator network by using a multimedia data set, to obtain a trained generator network, and determining a sparse neural network corresponding to the target neural network according to the trained generator network. By using the implementation scheme, a generator network is reconstructed based on the gating matrix of the target neural network, and the generator network is trained by using the multimedia data set to train the gating matrix representing the importance of weights in the generator network. The neural network pruning method provided by the embodiments of the present application can model the pruning as an unsupervised learning problem based on a model and data, and can automatically learn the optimal pruning strategy in the current scenario, so as to improve the accuracy and expression ability of the sparse neural network, and thus to prune the parameters with low weights in the neural network more accurately. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 A schematic diagram of a neural network pruning method Figure One ;

[0016] Figure 2 A schematic diagram of a neural network pruning method Figure Two ;

[0017] Figure 3 A flowchart of a neural network pruning method provided by an embodiment of the present application;

[0018] Figure 4 An exemplary neural network pruning method provided by an embodiment of the present application;

[0019] Figure 5A structure diagram of a neural network pruning device 1 provided for an embodiment of the present application Figure One ;

[0020] Figure 6 A structure diagram of a neural network pruning device 1 provided for an embodiment of the present application Figure Two . DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are merely exemplary of the present application. Therefore, the scope of the present application should be only defined by the appended claims.

[0022] It is a long-term research topic of pruning technology to accurately identify and prune the weights that are indeed redundant in the neural network. The so-called "pruning" specifically refers to evaluating the importance of the parameters in the neural network to the neural network, and deleting the parameters with low importance from the neural network under the premise of not affecting the network performance as much as possible, so as to reduce the parameter amount of the neural network. For example, in the neural network, there are a large number of redundant parameters from the convolutional layer to the fully connected layer, and most of the output values of the neurons after being activated are 0, that is, even if these neurons are removed, the model features of the neural network can still be expressed, and the neural network pruning can remove the neurons with the output value of 0 after being activated from the neural network.

[0023] In the embodiment of the present application, when the element value in the gating matrix is 0, it indicates that the output value of the corresponding neuron after being activated in the neural network is 0, at this time, through the multiplication operation of the gating matrix and the weight matrix in the neural network, the neuron can be removed from the neural network, and thus the operation of neural network pruning is completed.

[0024] For example, there are at least the following two neural network pruning methods, specifically;

[0025] The first one is to use the process based on the gradient descent method in the neural network training process, to update the weights by gradually propagating the error in reverse direction, according to the experience that "the less significant the gradient descent is, the less active the weight is, and the lower the importance is", the weights (or weight groups, such as weight strips, filters in CNN, feature maps, etc.) with gradient values lower than a certain threshold are pruned. In this way, the gradient values calculated in the training process itself can be reused, thereby reducing the calculation amount related to the pruning operation. Specifically, as shown in Figure 1 , first, train the target neural network structure and store the gradient; identify and prune the weights with small gradient; then continue to train the sparse network, and after the training is completed, judge whether the precision meets the standard or whether the maximum number of steps is reached, if the precision meets the standard or the maximum number of steps is reached, the training is ended, if the precision does not meet the standard and the maximum number of steps is not reached, the training of the sparse network is continued.

[0026] However, with the increasing depth of neural networks, inactive weights are not necessarily unimportant. Secondly, for DNNs with high structural complexity, because they contain higher-dimensional weight matrices than early shallow neural networks, according to mathematical derivation, to ensure accuracy during pruning, the gradient cannot be used directly, but large matrix operations such as inverse matrix and Hessian matrix calculation are required, which is too large in calculation and is not suitable for the current trend of deep learning. Thirdly, for the target neural network that has completed training and has not stored the gradient data of the final few steps, usually only the final weight data is available. This method needs to restart training on the already converged target neural network using the original data set and code to obtain gradient information. On the one hand, these original data and code may be difficult to obtain or need to be manually reproduced, which is difficult to ensure accuracy and consistency. On the other hand, training on an already converged network can easily lead to overfitting, and the pruning effect is poor.

[0027] Secondly, according to the empirical rule that "the smaller the weight value, the less important it is", weights with small absolute values (e.g. below a certain threshold, or in the later position of the descending order) or L0, L1 and / or L2 norm in the entire network are considered unimportant and are pruned. This type of method has a smaller calculation amount and is easy to understand, with lower development difficulty. Specifically, as shown in FIG. 2, first, the target neural network structure is trained and the gradient is stored; weights with small norm are identified and pruned; then the sparse network is further trained, and after the training is completed, it is determined whether the accuracy meets the standard or whether the maximum number of steps is reached. If the accuracy meets the standard or the maximum number of steps is reached, the training is ended. If the accuracy does not meet the standard and the maximum number of steps is not reached, the sparse network is further trained. Figure 2

[0028] However, the numerical value and the importance cannot be equated. One possible case is that some weights have large absolute values, but the signs are opposite, and in fact, the positive and negative cancel out and are not important. In addition, the threshold is determined according to experience. Therefore, this technique cannot always prune the least important weights.

[0029] To solve the above problems, an embodiment of the present application provides a neural network pruning method, as shown in FIG. 3, which can include the following steps. Figure 3

[0030] S101, obtaining a generator network according to a gating matrix of a target neural network, wherein the generator network includes the gating matrix and the target neural network.

[0031] The neural network pruning method provided in the embodiment of the present application supports neural network pruning scenarios in any field, any structure, such as image, voice, natural language processing, control, etc.

[0032] ​​In an embodiment of the present application, the target neural network can be a CNN, a DNN, an RNN, etc., which can be selected according to actual conditions, and the present application does not make specific limitations.

[0033] In an embodiment of the present application, the target neural network can be a model with floating-point parameters, or an arbitrary-bit-width fixed-point model, such as a 16-bit, 12-bit, 11-bit, 10-bit, 8-bit, 7-bit, 4-bit, 2-bit fixed-point model, or an integer model or a binary model. The specific selection can be made according to actual conditions, and the present application does not make specific limitations.

[0034] It should be noted that in order to optimize the pruning strategy of the neural network, the optimization of the pruning strategy is modeled as a machine learning problem, that is, for each layer of weights in the target neural network, a memory area with a dimension equal to the weight is used to store a gating matrix indicating whether to subtract each weight, wherein the element value in the gating matrix is usually a floating-point number between 0-1, which can represent the importance index of the corresponding weight, for example, 0 represents the least important, and 1 represents the most important. That is, each layer of the target neural network corresponds to a matrix representing the importance of the weights of the layer, and obtaining the gating matrix of the neural network means obtaining multiple matrices in the neural network.

[0035] It should be noted that the target neural network can include a multi-layer structure of convolutional layers, pooling layers, and / or fully connected layers, etc., wherein each layer of the target neural network corresponds to a set of parameters for implementing the function of the layer, and each parameter corresponds to a weight. For each set of parameters of each layer, a gating matrix corresponding to the set of weights of the set of parameters is set, wherein the element value in the gating matrix is used to represent the importance of the corresponding weight.

[0036] It should be noted that the number of weights corresponding to the elements in the gating matrix is related to the granularity of pruning, such as one weight for one element in the gating matrix when the granularity of pruning is one weight; a set of weights for one element in the gating matrix when the granularity of pruning is a structured set of weights, such as a weight strip, a filter, a feature map, etc.; and a set of weights for one element in the gating matrix when the granularity of pruning is a plurality of different granularities of weights, and the number of weights in different sets of weights can be different. The elements in the gating matrix can be determined by the actual determined granularity of pruning, and the present application does not make specific limitations.

[0037] In the field of artificial intelligence, pruning with a single weight is collectively referred to as unstructured pruning; pruning with a set of weights is collectively referred to as structured pruning; and pruning with multiple weights of different granularities is collectively referred to as heterogeneous pruning. It should be noted that all three pruning granularities are applicable to the neural network pruning method provided in the embodiments of this application.

[0038] In this embodiment of the application, the generator network can be obtained based on the gating matrix. For example, it can be obtained in at least the following two ways:

[0039] The first method involves copying the target neural network and its weight matrix to obtain a copy of the neural network; then inserting a gating matrix into the copy to obtain the generator network.

[0040] Specifically, a copy of the neural network and weight matrix is ​​made in memory to obtain a copy of the neural network. A trainable gating matrix is ​​then inserted into the copy of the neural network, that is, a gating matrix corresponding to each layer is inserted into each layer of the copy of the neural network. The computation graph of the neural network is then modified as necessary based on the gating matrix to obtain a generator network. The computation graph provides a graphical representation of the computational logic of the neural network. At this point, the output of each layer in the generator network is multiplied by the gating matrix of that layer.

[0041] The second method involves adjusting the values ​​of the gating matrix to preset values ​​to obtain the generator network.

[0042] Specifically, provided the programming language and / or deep learning (DL) framework support it, the values ​​of the gating matrices for each layer are dynamically specified to obtain the generator network. For example, setting all the values ​​of the gating matrix to 1 results in a neural network, while setting the values ​​of the gating matrix to preset values ​​results in the generator network. In this way, the operations of both the neural network and the generator network do not exist simultaneously in memory, thus saving memory during the training process.

[0043] It should be noted that the two methods for obtaining generator networks based on gating matrices mentioned above are only two methods proposed in this application. This application is not limited to the above methods. Specific methods can be obtained based on gating matrices according to actual conditions, which will not be elaborated here.

[0044] In this embodiment of the application, when creating the generator network, the initial weights of the generator network (that is, the initial values ​​of each matrix element in the weight matrix) can be recovered from the target neural network or can be re-initialized. The specific choice depends on the actual situation, and this embodiment of the application does not make specific limitations.

[0045] S102. Use a multimedia dataset to perform generative adversarial training on the generator network to obtain the trained generator network, and determine the sparse neural network corresponding to the target neural network based on the trained generator network.

[0046] After obtaining the generator network based on the gating matrix of the target neural network, a discriminator network is also created. This discriminator network can be created based on the network structure of the generator network. In this case, the network structure of the discriminator network is the same as or similar to that of the generator network.

[0047] For example, if the generator network is a U-NET (U-shaped network structure) network, the discriminator network also uses a U-NET network structure.

[0048] It is understandable that generator networks and discriminator networks have similar structures and similar expressive capabilities, and both can use similar training methods (learning rate, optimizer, etc.) to minimize the many training problems caused by the difference in expressive capabilities between discriminator networks and generator networks (including both the initial gap inherent due to structural differences and the gap that may gradually widen as training progresses—especially when discriminator networks and generator networks use training strategies that are not necessarily consistent).

[0049] Understandably, considering that many deep neural networks can be understood as an "encoder-decoder" architecture, shallower layers are responsible for extracting features from input samples, gradually increasing in order, while deeper layers are responsible for deriving features more relevant to the network's learning task, such as specific item categories in classification networks or character / complex shape output component features in image super-resolution / denoising networks. If two neural networks have similar structures and tasks that are somewhat similar but not entirely identical, such as in image denoising and super-resolution scenarios, they are highly likely to share some common detection features. Therefore, they can benefit from sharing the weights of certain layers (especially shallow layers) to achieve faster training and better accuracy. This also applies to schemes using discriminator and generator networks with similar structures. Especially in algorithms like generative adversarial networks (GANs), which are difficult to train, the initialization of weights is crucial. Using the same set of weights to initialize both the discriminator and generator networks ensures that both neural networks start learning from the same point, which is beneficial for stable training and convergence.

[0050] In this embodiment of the application, if the network structure of the discriminator network is the same as or similar to the network structure of the generator network, then the initial weights of the discriminator network are at least part of the initial weights of the generator network. That is, the discriminator network and the generator network can share a part or all of the initial weights.

[0051] In another alternative embodiment, a discriminator network, which may be simple or complex, similar to or unrelated to the generator network, can be constructed according to actual needs. Even other machine learning network models besides neural networks can be used, including but not limited to support vector machines, decision trees, and k-nearest neighbors. This application does not impose specific limitations on these models. In this case, the initial weights of the discriminator network and the generator network are generated using different initialization strategies.

[0052] After obtaining the discriminator network and the generator network, Generative Adversarial Learning (GAL) training is performed iteratively. After each iteration, it is checked whether the loop exit condition is met. If the loop exit condition is met, the trained generator network is obtained, which can be directly identified as a sparse neural network, thus completing the pruning operation of the neural network.

[0053] The target neural network in this embodiment can be used to process multimedia data, thus the generator network can be trained using a multimedia dataset for generative adversarial learning. Specifically, multimedia data from the multimedia dataset can be input into the target neural network and the generator network in batches, and the first sample corresponding to the target neural network and the second sample corresponding to the generator network can be output. That is, multimedia data is input into the target neural network and the first sample is output; multimedia data is input into the generator network and the second sample is output; the discriminator network is used to identify the first sample and the second sample to obtain the first identification result data; the discriminator network is trained based on the first identification result data, the category of the first sample and the category of the second sample to obtain the trained discriminator network; the generator network is trained based on the trained discriminator network and the second sample until the loop exit condition is met to obtain the trained generator network.

[0054] Optionally, the multimedia dataset may include at least one of the following: image dataset, audio dataset, text dataset, and video dataset. The specific selection depends on the application scenario corresponding to the target neural network, and this application embodiment does not impose any specific limitations.

[0055] It should be noted that the first sample is the real sample obtained by inputting into the target neural network, and the category of the first sample is real sample; the second sample is the deceptive sample obtained by inputting into the generator network, and the category of the second sample is deceptive sample. Since the discriminator network can accurately identify real samples but cannot identify deceptive samples when the system converges, exhibiting a 50% probability of classifying deceptive samples as real or false, the first identification result data can consist of three parts: classifying real samples as real, classifying deceptive samples as false, and classifying deceptive samples as real. This avoids the first identification result data being too homogeneous, thereby preventing the discriminator network from overfitting to any version of the generator network during training, which would affect the training of the generator network.

[0056] It should be noted that the first sample is the vector output by the target neural network, and the second sample is the vector output by the generator network. The output vector can be the output vector of the softmax layer containing the confidence of each class, or the output vector of the previous layer, etc., and is not just a scalar representing the final classification. The specific selection is based on the actual situation, and this application embodiment does not make specific limitations.

[0057] Specifically, the process of training the discriminator network based on the first recognition result data, the category of the first sample, and the category of the second sample to obtain the trained discriminator network includes: training the weights of the discriminator network based on the first recognition result data, the category of the first sample, and the category of the second sample to obtain the trained discriminator network.

[0058] It should be noted that during the training of the discriminator network, in addition to the error rate, other terms can be added to the loss function of the discriminator network, or they can be learned during the training process.

[0059] In one optional embodiment, the process of training the generator network based on the trained discriminator network and the second sample until the loop exit condition is met, and obtaining the trained generator network, includes: recognizing the second sample based on the trained discriminator network to obtain second recognition result data; training the gating matrix of the generator network based on the second recognition result data; and determining the generator network corresponding to the trained gating matrix as the trained generator network when the loop exit condition is met.

[0060] In this embodiment of the application, only the gating matrix of the generator network can be trained. After the training of the gating matrix is ​​completed, the generator network corresponding to the trained gating matrix is ​​determined as the trained generator network.

[0061] In another optional embodiment, the process of training the generator network based on the trained discriminator network and the second sample until the loop exit condition is met to obtain the trained generator network includes: recognizing the second sample based on the trained discriminator network to obtain the second recognition result data; training the gating matrix and weights of the generator network based on the second recognition result data, and obtaining the trained generator network when the loop exit condition is met.

[0062] In this embodiment of the application, the gating matrix and weights of the generator network can be trained simultaneously. After training the gating matrix and weights, the trained generator network can be obtained directly.

[0063] It should be noted that during the process of performing generative adversarial training on the discriminator network and the generator network in a loop, the multimedia data in the multimedia dataset is input into the target neural network and the generator network in batches in a loop. The process of inputting a batch of multimedia data into the target neural network and the generator network in a loop is called a step, and the process of inputting all the multimedia data in the entire multimedia dataset into the target neural network and the generator network in batches in one loop is called a generation.

[0064] In this embodiment of the application, the trained discriminator network identifies the second sample to obtain second identification result data. The second identification result data can be identifying the deceptive sample as real and the deceptive sample as fake. Then, the gating matrix of the generator network is trained based on the second identification result data (if necessary, the weights of the generator network can also be trained at the same time) one or more steps, or until the generator network converges.

[0065] In this embodiment of the application, during the training of the generator network based on the second recognition result data, the loss function of the generator network is Loss. G As shown in formula (1):

[0066] Loss G =μ×Loss data +Loss GAN +λ×Loss sparse +decay×Loss reg (1)

[0067] Among them, Loss data Loss represents the error between the corresponding real sample and the deceptive sample. GAN Loss is the accuracy of the discriminator network (since the generator network aims to maximize the error rate of the discriminator network). sparse Loss is the L1 regularization term of the gated matrix. regLet μ be the L2 regularization term for all weights and biases in the generator network, and μ, λ, and decay be the relative weights of each term in the overall loss function. Adding L1 and L2 regularization terms to the loss function ensures that a necessary number of elements in the gating matrix converge to 0.

[0068] It should be noted that the loss function of the generator network can include more or fewer terms as needed, such as perceptual loss, signal-to-noise ratio, color difference in other color gamuts, etc., or can be learned during training.

[0069] It should be noted that the weights of the generator network can be updated synchronously with or separately from the gating matrix, and the same or different loss functions can be used. The specific choice can be made according to the actual situation, and the embodiments of this application do not impose specific limitations.

[0070] In this embodiment, after each cycle of generative adversarial training of the discriminator network and the generator network, it is necessary to determine whether the cycle exit condition is met. The cycle exit condition is any one of the following: the number of cycles reaches a preset threshold, or the generator network converges. If any of the above cycle exit conditions are met, the sparse neural network corresponding to the target neural network is determined based on the trained generator network. If any of the above cycle exit conditions are not met, the next round of generative adversarial training is carried out using the trained generator network and discriminator network until the cycle exit condition is met, at which point the generative adversarial training process ends.

[0071] After obtaining the trained generator network, since the trained generator network already includes the operation of multiplying the gating matrix with the corresponding layer output, it can be directly identified as a sparse neural network for forward computation. The trained gating matrix corresponding to the trained generator network can also be determined. By multiplying the trained gating matrix with the weight matrix of the generator network and removing the trained gating matrix from the trained generator network, a sparse neural network is obtained. This sparse neural network is obtained by multiplying the trained gating matrix with the weight matrix of the generator network and removing the trained gating matrix, thus saving space occupied by the trained generator network. Furthermore, the sparse neural network can be trained further. The specific post-processing performed on the trained generator network can be selected according to the actual situation, and this application embodiment does not impose specific limitations.

[0072] It should be noted that when the gating matrix and weight matrix are multiplied after training, the weights corresponding to matrix elements with a value of 0 in the gating matrix are equivalent to being zeroed out and pruned. Optionally, the pruned weights can be simply zeroed out and stored in a normal dense matrix format; in the case of structured pruning, the pruned weights can be completely deleted and reassembled; in the case of unstructured pruning, the pruned weights can be deleted and then compressed and stored in a sparse format, and the corresponding optimized sparse operator can be used during the forward operation.

[0073] It should be noted that during the training of the sparse neural network, the loss function for network training is the loss function of the generator network. In this case, more sub-items can be selected or added, or the loss function can be learned during the training process. Alternatively, it can be a loss function that represents the error between the sparse neural network and the preset neural network. The preset neural network can be a neural network or other model, and the error between the sparse neural network and the preset neural network can be calculated by the output layer or intermediate layer of the neural network.

[0074] For example, such as Figure 4 As shown, the steps for pruning a neural network include:

[0075] 1. Copy the target neural network structure and weight matrix in memory to obtain a copy of the neural network;

[0076] 2. Insert trainable gating matrices into the neural network replicas and modify the computation graph so that the output of each layer is multiplied by the gating matrix of that layer, thus obtaining the generator network;

[0077] 3. Define another neural network as the discriminator network;

[0078] 4. Input a batch of data from the training dataset into the target neural network and the generator network to obtain real samples and deceptive samples respectively;

[0079] 5. Input real samples and deceptive samples into the discriminator network to train the discriminator network;

[0080] 6. Use the trained discriminator network to identify deceptive samples and train the gating matrix of the generator network;

[0081] 7. Determine whether the maximum number of iterations has been reached, or whether both the generator network and the discriminator network have converged;

[0082] 8. If the maximum number of iterations is reached, or if both the generator network and the discriminator network have converged, then post-process the trained generator network.

[0083] 9. If the maximum number of iterations has not been reached and the generator network and discriminator network have not converged, then determine the next batch of data from the training dataset and execute step 4.

[0084] Understandably, a generator network is reconstructed based on the gating matrix of the target neural network. Generative adversarial training is then performed on the generator network using a multimedia dataset to train the gating matrix representing the importance of weights within the generator network. The neural network pruning method provided in this application can model pruning as an unsupervised learning problem based on the model and data. It can automatically learn the optimal pruning strategy for the current scenario, thus improving the accuracy and expressive power of sparse neural networks, and consequently, enabling more accurate pruning of parameters with lower weights in the neural network.

[0085] Based on the above embodiments, it can be seen that the neural network pruning method proposed in this application has the following beneficial effects compared with the prior art:

[0086] 1. Since the generator network created in this application includes a gating matrix, the gating matrix can directly participate in the generative adversarial training of the generator network. The sparse neural network can be directly obtained from the trained generator network. Compared with the prior art, there is no need to train the sparse neural network separately, which simplifies the neural network pruning steps.

[0087] 2. The gating matrix participates in generative adversarial training, which enables the gating matrix to more accurately, flexibly and stably represent the importance of weights in the target neural network as training progresses, avoiding the binarization and one-time pruning strategies in existing technologies.

[0088] 3. This application utilizes a gating matrix to perform pruning operations on the target neural network. This pruning operation is only related to the gating matrix that represents the importance of the weights, thus decoupling it from the weights of the target neural network and facilitating the training and analysis of the generator network.

[0089] 4. In the embodiments of this application, the elements in the gate matrix correspond to the smallest unit of pruning. The smallest unit of pruning can be a single weight, a group of weights, or even a combination of multiple different pruning granularities. This enables heterogeneous pruning to be performed simultaneously at multiple granularities, reducing the accumulation of errors caused by performing pruning at different granularities multiple times.

[0090] 5. This application generates a generator network based on the gating matrix of the target neural network and performs generative adversarial training on the generator network. This enables the resulting sparse neural network to automatically learn the optimal pruning strategy for the current scene and the one that maximizes all layers. Compared with existing pruning strategies based on empirical assumptions, this eliminates the need to manually specify the pruning ratio, thus improving the accuracy and expressive power of the sparse neural network.

[0091] 6. In this application, a discriminator network can be created based on the network structure of the generator network, so that the network structure of the generator network and the network structure of the discriminator network are the same or similar. This can better control the progress and balance of generative adversarial training, making the network convergence more stable and reducing the probability of getting stuck in local optima in traditional generative adversarial training.

[0092] 7. The network structures of the generator network and the discriminator network are the same or similar, which allows the generator network and the discriminator network to share all or part of the weights. This is beneficial to the stability of adversarial training and can even be used for co-training. This can reduce the computational cost of training while improving the interpretability of sparse neural networks.

[0093] Based on the above embodiments, this application provides a neural network pruning device 1. For example... Figure 5 As shown, the device 1 includes:

[0094] Creation unit 10 is used to obtain a generator network based on the gating matrix of the target neural network, wherein the generator network includes the gating matrix and the target neural network;

[0095] The training unit 11 is used to perform generative adversarial training on the generator network using a multimedia dataset to obtain the trained generator network, and to determine the sparse neural network corresponding to the target neural network based on the trained generator network.

[0096] Optionally, the device further includes: an input unit and a recognition unit;

[0097] The input unit is used to cyclically input the multimedia data in the multimedia dataset into the target neural network and the generator network in batches, and output the first sample corresponding to the target neural network and the second sample corresponding to the generator network.

[0098] The identification unit is used to identify the first sample and the second sample using a discriminator network to obtain first identification result data;

[0099] The training unit 11 is further configured to train the discriminator network based on the first recognition result data, the category of the first sample, and the category of the second sample to obtain the trained discriminator network; and to train the generator network based on the trained discriminator network and the second sample until the loop exit condition is met to obtain the trained generator network.

[0100] Optionally, the apparatus further includes: a determining unit;

[0101] The training unit 11 is further configured to train the weights of the discriminator network based on the first recognition result data, the category of the first sample, and the category of the second sample, to obtain the trained discriminator network.

[0102] Optionally, the recognition unit is further configured to recognize the second sample based on the trained discriminator network to obtain second recognition result data;

[0103] The training unit 11 is further configured to train the gating matrix of the generator network based on the second recognition result data, and when the loop exit condition is met, determine the generator network corresponding to the trained gating matrix as the trained generator network.

[0104] Optionally, the recognition unit is further configured to recognize the second sample based on the trained discriminator network to obtain second recognition result data;

[0105] The training unit 11 is also used to train the gating matrix and weights of the generator network based on the second recognition result data, and obtain the trained generator network when the loop exit condition is met.

[0106] Optionally, the loop exit condition is any one of the following: the number of loops reaches a preset threshold, or the generator network converges.

[0107] Optionally, the apparatus further includes: a copying unit and an insertion unit;

[0108] The copying unit is used to copy the target neural network and the weight matrix of the target neural network to obtain a copy of the neural network; the weight matrix is ​​used to represent the weight of one or more elements in the target neural network; the gating matrix is ​​used to represent the importance index of one or more parameters in the weight matrix;

[0109] The insertion unit is used to insert the gating matrix into the copy of the neural network to obtain the generator network.

[0110] Optionally, the creation unit 10 is further configured to obtain the generator network by adjusting the value of the gating matrix of the target neural network to a preset value.

[0111] Optionally, the creation unit 10 is further configured to create a discriminator network, which is a network generated based on the network structure of the generator network.

[0112] Optionally, the initial weights of the discriminator network are at least partly the same as the initial weights of the generator network, and the initial weights of the generator network are recovered or initialized from the neural network.

[0113] Optionally, the apparatus further includes: a post-processing unit;

[0114] The post-processing unit is used to determine the trained gating matrix corresponding to the trained generator network; by multiplying the trained gating matrix and the weight matrix and removing the trained gating matrix, the sparse neural network is obtained.

[0115] Optionally, the post-processing unit is further configured to train the sparse neural network, wherein the loss function for network training is the loss function of the generator network.

[0116] Optionally, the multimedia dataset includes at least one of the following: image dataset, audio dataset, text dataset, and video dataset.

[0117] This application provides a neural network pruning device that obtains a generator network based on the gating matrix of the target neural network. The generator network includes the gating matrix and the target neural network. Generative adversarial training is performed on the generator network using a multimedia dataset to obtain a trained generator network. The sparse neural network corresponding to the target neural network is then determined based on the trained generator network. Therefore, the device proposed in this embodiment reconstructs a generator network based on the gating matrix of the target neural network, performs generative adversarial training on the generator network using a multimedia dataset, and trains the gating matrix representing the importance of weights in the generator network. The neural network pruning method provided in this application can model pruning as an unsupervised learning problem based on the model and data, and can automatically learn the optimal pruning strategy in the current scenario. Therefore, it is beneficial to improve the accuracy and expressive power of the sparse neural network, and thus facilitates the more accurate pruning of parameters with lower weights in the neural network.

[0118] Figure 6 A schematic diagram of the composition structure of a neural network pruning device 1 provided in this application embodiment. Figure Two In practical applications, based on the same disclosed concept of the above embodiments, such as Figure 6 As shown, the neural network pruning device 1 in this embodiment includes a processor 12, a memory 13, and a communication bus 14.

[0119] In a specific embodiment, the creation unit 10, training unit 11, input unit, partitioning unit, determination unit, copying unit, insertion unit, and post-processing unit described above can be implemented by a processor 12 located on the neural network pruning device 1. The processor 12 can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), CPU, controller, microcontroller, and microprocessor. It is understood that for different devices, the electronic device used to implement the above processor functions can also be other types; this embodiment does not specifically limit this.

[0120] In this embodiment, the communication bus 14 is used to establish a connection between the processor 12 and the memory 13; when the processor 12 executes the running program stored in the memory 13, it implements the following neural network pruning method:

[0121] A generator network is obtained based on the gating matrix of the target neural network, the generator network including the gating matrix and the target neural network; the generator network is subjected to generative adversarial training using a multimedia dataset to obtain a trained generator network, and the sparse neural network corresponding to the target neural network is determined based on the trained generator network.

[0122] Furthermore, the processor 12 is also configured to cyclically input multimedia data from the multimedia dataset into the target neural network and the generator network in batches, outputting a first sample corresponding to the target neural network and a second sample corresponding to the generator network; use a discriminator network to identify the first sample and the second sample to obtain first identification result data; train the discriminator network based on the first identification result data, the category of the first sample, and the category of the second sample to obtain a trained discriminator network; and train the generator network based on the trained discriminator network and the second sample until the loop exit condition is met to obtain a trained generator network.

[0123] Furthermore, the processor 12 is also used to train the weights of the discriminator network based on the first recognition result data, the category of the first sample, and the category of the second sample, to obtain the trained discriminator network.

[0124] Furthermore, the processor 12 is also used to identify the second sample based on the trained discriminator network to obtain second identification result data; to train the gating matrix of the generator network based on the second identification result data; and to determine the generator network corresponding to the trained gating matrix as the trained generator network when the loop exit condition is met.

[0125] Furthermore, the processor 12 is also used to identify the second sample based on the trained discriminator network to obtain second identification result data; to train the gating matrix of the generator network and the weights of the generator network based on the second identification result data, and to obtain the trained generator network when the loop exit condition is met.

[0126] Furthermore, the loop exit condition is any one of the following: the number of loops reaches a preset threshold, or the generator network converges.

[0127] Furthermore, the processor 12 is also used to copy the target neural network and the weight matrix of the target neural network to obtain a copy of the neural network; the weight matrix is ​​used to characterize the weight of one or more elements in the target neural network; the gating matrix is ​​used to characterize the importance index of one or more parameters in the weight matrix; the gating matrix is ​​inserted into the copy of the neural network to obtain the generator network.

[0128] Furthermore, the processor 12 is also used to obtain the generator network by adjusting the value of the gating matrix of the target neural network to a preset value.

[0129] Furthermore, the processor 12 is also used to create a discriminator network, which is a network generated based on the network structure of the generator network.

[0130] Furthermore, the initial weights of the discriminator network are at least partly the same as the initial weights of the generator network, and the initial weights of the generator network are recovered or initialized from the neural network.

[0131] Furthermore, the processor 12 is also used to determine the trained gating matrix corresponding to the trained generator network; by multiplying the trained gating matrix and the weight matrix of the generator network and removing the trained gating matrix from the trained generator network, the sparse neural network is obtained; the weight matrix is ​​used to characterize the weight of one or more elements in the neural network.

[0132] Furthermore, the processor 12 is also used to train the sparse neural network, wherein the loss function for network training is the loss function of the generator network.

[0133] Furthermore, the multimedia dataset includes at least one of the following: image dataset, audio dataset, text dataset, and video dataset.

[0134] This application provides a storage medium storing a computer program thereon. The computer-readable storage medium stores one or more programs, which can be executed by one or more processors and applied in a device. The computer program implements the neural network pruning method described above.

[0135] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause an image display device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0137] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A neural network pruning method, characterized in that, The method includes: A generator network is obtained based on the gating matrix of the target neural network, wherein the generator network includes the gating matrix and the target neural network; The generator network is trained using a multimedia dataset to perform generative adversarial training, resulting in a trained generator network. The sparse neural network corresponding to the target neural network is then determined based on the trained generator network. The multimedia dataset includes at least one of the following: image dataset, audio dataset, text dataset, and video dataset. The generator network is trained using a multimedia dataset to perform generative adversarial training, resulting in a trained generator network, including: The multimedia data in the multimedia dataset is input into the target neural network and the generator network in batches, and the first sample corresponding to the target neural network and the second sample corresponding to the generator network are output. The discriminator network is used to identify the first sample and the second sample to obtain the first identification result data. The discriminator network is trained based on the first identification result data, the category of the first sample and the category of the second sample to obtain the trained discriminator network. The generator network is trained based on the trained discriminator network and the second sample until the loop exit condition is met to obtain the trained generator network. The process of determining the sparse neural network corresponding to the target neural network based on the trained generator network includes: Determine the trained gating matrix corresponding to the trained generator network; obtain the sparse neural network by multiplying the trained gating matrix and the weight matrix of the generator network, and removing the trained gating matrix from the trained generator network; the weight matrix is ​​used to characterize the weights of one or more parameters in the neural network.

2. The method according to claim 1, characterized in that, The process of training the discriminator network based on the first identification result data, the category of the first sample, and the category of the second sample to obtain the trained discriminator network includes: The discriminator network is trained based on the first identification result data, the category of the first sample, and the category of the second sample to obtain the trained discriminator network.

3. The method according to claim 1, characterized in that, The step of training the generator network based on the trained discriminator network and the second sample until the loop exit condition is met, to obtain the trained generator network, includes: The second sample is identified based on the trained discriminator network to obtain the second identification result data; The gating matrix of the generator network is trained based on the second recognition result data. When the loop exit condition is met, the generator network corresponding to the trained gating matrix is ​​determined as the trained generator network.

4. The method according to claim 1, characterized in that, The step of training the generator network based on the trained discriminator network and the second sample until the loop exit condition is met, to obtain the trained generator network, includes: The second sample is identified based on the trained discriminator network to obtain the second identification result data; The gating matrix and weights of the generator network are trained based on the second recognition result data. The trained generator network is obtained when the loop exit condition is met.

5. The method according to claim 1, 3, or 4, characterized in that, The loop exit condition is any one of the following: the number of loops reaches a preset threshold, or the generator network converges.

6. The method according to claim 1, characterized in that, The process of obtaining the generator network based on the gating matrix of the target neural network includes: A copy of the target neural network and its weight matrix are obtained; the weight matrix is ​​used to characterize the weights of one or more parameters in the target neural network; the gating matrix is ​​used to characterize the importance index of one or more parameters in the weight matrix. The gating matrix is ​​inserted into the copy of the neural network to obtain the generator network.

7. The method according to claim 1, characterized in that, The process of obtaining the generator network based on the gating matrix of the target neural network includes: The generator network is obtained by adjusting the value of the gating matrix of the target neural network to a preset value.

8. The method according to claim 1, characterized in that, After obtaining the generator network based on the gating matrix of the target neural network, and before performing generative adversarial training on the generator network using a multimedia dataset to obtain the trained generator network, the method further includes: Create a discriminator network, which is a network generated based on the network structure of the generator network.

9. The method according to claim 8, characterized in that, The initial weights of the discriminator network are at least partly the same as the initial weights of the generator network, and the initial weights of the generator network are recovered or initialized from the neural network.

10. The method according to claim 1, characterized in that, After determining the sparse neural network corresponding to the target neural network based on the trained generator network, the method further includes: The sparse neural network is trained, wherein the loss function for network training is the same as the loss function for the generator network.

11. A neural network pruning device, characterized in that, The device includes: A creation unit is used to obtain a generator network based on the gating matrix of the target neural network, wherein the generator network includes the gating matrix and the target neural network; The training unit is used to perform generative adversarial training on the generator network using a multimedia dataset to obtain a trained generator network, and to determine the sparse neural network corresponding to the target neural network based on the trained generator network; the multimedia dataset includes at least one of the following: image dataset, audio dataset, text dataset, and video dataset. The input unit is used to cyclically input the multimedia data in the multimedia dataset into the target neural network and the generator network in batches, and output the first sample corresponding to the target neural network and the second sample corresponding to the generator network. The identification unit is used to identify the first sample and the second sample using a discriminator network to obtain first identification result data; The training unit is further configured to train the discriminator network based on the first recognition result data, the category of the first sample, and the category of the second sample to obtain the trained discriminator network; and to train the generator network based on the trained discriminator network and the second sample until the loop exit condition is met to obtain the trained generator network. The post-processing unit is used to determine the trained gating matrix corresponding to the trained generator network; by multiplying the trained gating matrix and the weight matrix of the generator network and removing the trained gating matrix, the sparse neural network is obtained.

12. A neural network pruning device, characterized in that, The device includes: a processor and a memory; when the processor executes the running program stored in the memory, it implements the method as described in any one of claims 1-10.

13. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-10.

Images