Pruning method and device for neural network model and electronic equipment

Abstract

The invention relates to the technical field of smart home, and discloses a pruning method for a neural network model, and the method comprises the steps: determining a first contribution degree of each network unit to the performance of the neural network model in a forward propagation stage; determining a second contribution degree of each network unit to the performance of the neural network model in a model training stage; according to the first contribution degree and / or the second contribution degree, the target contribution degree of each network unit to the performance of the neural network model is determined, and the target contribution degree represents the contribution degree of the network unit to the performance of the neural network model; and according to the target contribution degree of each network unit to the performance of the neural network model, determining a target network unit needing pruning, and performing pruning operation to obtain a new neural network model. According to the method, the model performance can be maintained and even optimized to the greatest extent. The invention further discloses a pruning device for the neural network model and electronic equipment.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, for example, to a pruning method and device for a neural network model, and an electronic device. Background Art

[0002] At present, the application of neural networks has been popularized in many areas of daily life, including visual recognition, speech recognition, natural language processing, etc., but at the same time, the requirements for computing resources have increased. In order to solve this problem, different model compression and optimization technologies have also emerged. Among them, pruning technology is one of the important methods. Pruning technology reduces the size of the model by removing irrelevant or unimportant neurons or connections in the neural network model.

[0003] In order to achieve compression and optimization of neural network models, the relevant technology discloses a structured pruning method for neural network models, including: sorting the network weights of all neurons according to a set sorting standard; determining the proportion of network weights based on the sorting results, and determining the weight with the smallest proportion; deleting the neurons corresponding to the weight with the smallest proportion to achieve a structured pruning operation.

[0004] In the process of implementing the embodiments of the present disclosure, it is found that there are at least the following problems in the related art:

[0005] Although related technologies can reduce the complexity of neural network models, the above structured pruning methods tend to treat all neurons or all network layers uniformly, which causes network layers or neurons that are critical to specific tasks to be pruned, resulting in a decrease in the performance of the neural network model. Therefore, structured pruning methods cannot achieve optimal global performance under different task requirements and computing resource constraints.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0007] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0008] The embodiments of the present disclosure provide a pruning method and device, and an electronic device for a neural network model, so as to accurately remove network units in the neural network model that have little contribution to performance, while retaining network units that have great contribution to performance, and to maximize the maintenance of model performance or even optimize model performance.

[0009] In some embodiments, the pruning method includes: determining a first contribution of each network unit to the performance of the neural network model in the forward propagation stage; determining a second contribution of each network unit to the performance of the neural network model in the model training stage; determining a target contribution of each network unit to the performance of the neural network model based on the first contribution and / or the second contribution, wherein the target contribution represents the size of the contribution of the network unit to the performance of the neural network model; determining the target network unit that needs to be pruned based on the target contribution of each network unit to the performance of the neural network model, and performing a pruning operation to obtain a new neural network model.

[0010] In some embodiments, based on the first contribution and / or the second contribution, the target contribution of each network unit to the performance of the neural network model is determined, including: multiplying the first contribution by the first proportional coefficient α to determine the first reference contribution; multiplying the second contribution by the second proportional coefficient β to determine the second reference contribution; determining the sum of the first reference contribution and the second reference contribution as the target contribution; wherein 0<α≤1, 0<β≤1.

[0011] In some embodiments, based on the target contribution of each network unit to the performance of the neural network model, the target network unit that needs to be pruned is determined, and a pruning operation is performed to obtain a new neural network model, including: determining that the network unit whose target contribution is less than a contribution threshold is the target network unit that needs to be pruned; removing the target network unit from the neural network model to obtain a new neural network model.

[0012] In some embodiments, when the network unit is a neuron, determining the first contribution of each network unit to the performance of the neural network model in the forward propagation stage includes: determining the evaluation index of each neuron as a weight; obtaining the weight corresponding to the i-th neuron and each neuron in the previous network layer, and constructing a weight vector Wi corresponding to the i-th neuron; wherein i represents the sequence number of the neuron in the neural network model, i=1,3,...,n, and n represents the total number of neurons; according to the weight vector Wi corresponding to the i-th neuron, determining the evaluation function f(i) of the i-th neuron in the forward propagation stage; wherein f(i)=||Wi||; normalizing the evaluation function f(i) to obtain the first contribution of the i-th neuron to the performance of the neural network model in the forward propagation stage.

[0013] Optionally, the evaluation function f(i) is standardized to obtain the first contribution of the i-th neuron to the performance of the neural network model in the forward propagation stage, including: F(i) = ||Wi|| / Σ||Wi||. Wherein, i = 1, 3, ..., n, n represents the total number of neurons, and F(i) represents the first contribution of the i-th neuron to the performance of the neural network model in the forward propagation stage.

[0014] In some embodiments, when the network units are neurons, determining the second contribution of each network unit to the performance of the neural network model during the model training stage includes: obtaining the current value of the neuron parameters of each neuron in the forward propagation stage, the neuron parameters including weights; obtaining the update amplitude of the neuron parameters of each network unit with the current value as the reference value in the back propagation stage; determining the average value of the update amplitude of the neuron parameters of each neuron based on the update amplitude of the neuron parameters of each neuron and the number of iterations; determining the average value of the update amplitude of each neuron parameter as the second contribution of each neuron to the performance of the neural network model during the model training stage.

[0015] In some embodiments, when the network unit is a network layer, determining the first contribution of each network unit to the performance of the neural network model in the forward propagation stage includes: determining the evaluation index of each network layer as the activation ratio; wherein the activation ratio represents the ratio of the number of neurons activated in the current network layer to the number of all neurons in the current network layer in the forward propagation stage; obtaining the number of neurons N(j) activated in the j-th network layer in the forward propagation stage; ratioing N(j) with SUM_N0(j) to obtain the activation ratio of the j-th network layer; wherein SUM_N0(j) represents the number of all neurons in the j-th network layer; determining the activation ratio of the j-th network layer as the first contribution of the j-th network layer to the performance of the neural network model in the forward propagation stage.

[0016] Optionally, obtaining the number of neurons N(j) activated in the j-th network layer in the forward propagation stage includes: obtaining the number of neurons Nl(t, j) activated in the j-th network layer in the t-th forward propagation stage; obtaining the total number of neurons SUM_Nl(j) activated in the j-th network layer in all forward propagation stages; wherein SUM_Nl(j) = ΣNl(t, j), t = 1, 2, ..., T, T1 represents the total number of forward propagations; and determining the number of neurons N(j) activated in the j-th network layer in the forward propagation stage by ratioing SUM_Nl(j) with the total number of forward propagations T1.

[0017] In some embodiments, when the network unit is a neuron, determining the second contribution of each network unit to the performance of the neural network model during the model training stage includes: obtaining the output value ela(s, i) of the ith neuron in the sth model training stage, where the output value represents the output value of the ith neuron with the sigmoid function as the activation function; obtaining the total number of output values ​​SUM_ela(i) of the ith neuron in all model training stages; wherein SUM_ela(i) = Σela(s, j), s = 1, 2, ..., T2, and T2 represents the total number of model trainings; ratioing SUM_ela(i) with the total number of model trainings T2 to obtain the average output value AVE(i) of the ith neuron; and determining the average output value AVE(i) of the ith neuron as the second contribution of the ith neuron to the performance of the neural network model during the model training stage.

[0018] Optionally, in the case where the network unit is a network layer, determining the second contribution of each network unit to the performance of the neural network model during the model training stage includes: obtaining the sum of the average output values ​​of each neuron of the j-th network layer, and determining the sum of the average output values ​​of each neuron of the j-th network layer as the second contribution of the j-th network layer to the performance of the neural network model during the model training stage.

[0019] In some embodiments, it also includes: after performing a pruning operation to obtain a new neural network model, inputting a training data set to the new neural network model for model training to obtain a fine-tuned neural network model; inputting a verification data set to verify the fine-tuned neural network model and obtaining a verification index; when the verification index indicates that it does not meet the expected model performance, returning to determine the first contribution and the second contribution of each network unit, and performing a pruning operation again based on the first contribution and the second contribution until the verification index meets the expected model performance.

[0020] In some embodiments, the device includes a processor and a memory storing program instructions, and the processor is configured to execute the pruning method for a neural network model as described above when running the program instructions.

[0021] In some embodiments, an electronic device includes: a device body; and a pruning device for a neural network model as described above, installed on the device body.

[0022] The pruning method, device and electronic device for a neural network model provided by the embodiments of the present disclosure can achieve the following technical effects:

[0023] The disclosed embodiments can implement an adaptive pruning strategy to accurately remove network units that contribute little to performance in a neural network model, while retaining network units that contribute greatly to performance. This can not only reduce model complexity and computing resource requirements, but also, by retaining network units that contribute greatly to performance, it can maintain or even optimize model performance to the greatest extent.

[0024] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:

[0026] Figure 1 is a schematic diagram of a pruning method for a neural network model provided by an embodiment of the present disclosure;

[0027] Figure 2 is a schematic diagram of another pruning method for a neural network model provided by an embodiment of the present disclosure;

[0028] Figure 3 is a schematic diagram of another pruning method for a neural network model provided by an embodiment of the present disclosure;

[0029] Figure 4 is a schematic diagram of another pruning method for a neural network model provided by an embodiment of the present disclosure;

[0030] Figure 5 is a schematic diagram of another pruning method for a neural network model provided by an embodiment of the present disclosure;

[0031] Figure 6 is a schematic diagram of another pruning method for a neural network model provided by an embodiment of the present disclosure;

[0032] Figure 7 is a schematic diagram of another pruning method for a neural network model provided by an embodiment of the present disclosure;

[0033] Figure 8 It is a schematic diagram of a pruning device for a neural network model provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0035] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0036] Unless otherwise stated, the term "plurality" means two or more.

[0037] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.

[0038] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0039] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

[0040] The method provided in this application is widely used in smart home (Smart Home), smart home, smart home device ecology, smart residential (Intelligence House) ecology and other whole-house intelligent digital control application scenarios. The above-mentioned network may include but is not limited to at least one of the following: wired network, wireless network. The above-mentioned wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the above-mentioned wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity, wireless fidelity), Bluetooth.

[0041] Combination Figure 1 As shown, the embodiment of the present disclosure provides a pruning method for a neural network model, comprising:

[0042] S01, the processor determines the first contribution of each network unit to the performance of the neural network model in the forward propagation stage.

[0043] In this step, the network unit includes a network layer or a neuron.

[0044] S02, the processor determines the second contribution of each network unit to the performance of the neural network model during the model training phase.

[0045] S03, the processor determines the target contribution of each network unit to the performance of the neural network model according to the first contribution and / or the second contribution, wherein the target contribution represents the contribution of the network unit to the performance of the neural network model.

[0046] S04, the processor determines the target network unit that needs to be pruned according to the target contribution of each network unit to the performance of the neural network model, and performs a pruning operation to obtain a new neural network model.

[0047] The pruning method for a neural network model provided by an embodiment of the present disclosure is adopted. The embodiment of the present disclosure determines the first contribution of each network unit to the performance of the neural network model in the forward propagation stage to quantify the contribution of each network unit to the neural network model in the forward propagation stage. The embodiment of the present disclosure also determines the second contribution of each network unit to the performance of the neural network model in the model training stage to obtain the contribution of each network unit to the neural network model during the model training process. Then, the target contribution of each network unit to the performance of the neural network model is comprehensively measured according to the first contribution and / or the second contribution, and the target network unit to be pruned is determined according to the target contribution, and a pruning operation is performed. The embodiment of the present disclosure can implement an adaptive pruning strategy to accurately remove network units in the neural network model that contribute little to performance, while retaining network units that contribute much to performance. Not only can the model complexity be reduced and the computing resource requirements be reduced, but also because the network units with large performance contributions are retained, the model performance can be maintained to the greatest extent and even optimized.

[0048] It should be noted that the embodiments of the present disclosure require fewer computing resources and are particularly suitable for performing deep learning tasks on embedded devices or edge devices. Therefore, the embodiments of the present disclosure can be widely used in scenarios where deep learning tasks are performed on the device side, such as the Internet of Things, mobile phone applications, or unmanned driving.

[0049] Optionally, combined Figure 2 As shown, the processor determines the target contribution of each network unit to the performance of the neural network model according to the first contribution and / or the second contribution, including:

[0050] S11, the processor multiplies the first contribution by the first proportionality coefficient α to determine a first reference contribution.

[0051] S12: The processor multiplies the second contribution by the second proportional coefficient β to determine a second reference contribution.

[0052] S13: The processor determines a sum of the first reference contribution and the second reference contribution as a target contribution.

[0053] Among them, 0<α≤1,0<β≤1.

[0054] In this way, the embodiment of the present disclosure multiplies the first contribution and the second contribution respectively according to the set proportional coefficient to determine the first reference contribution and the second reference contribution, and then sums the two to obtain the target contribution. This can comprehensively measure the contribution of each network unit to the model performance in the forward propagation stage and model training stage of the neural network model, and can maximize the prevention of pruning of network units that are significantly important to specific tasks, thereby improving the precision of the model pruning operation.

[0055] It should be noted that the first contribution represents the degree of influence of the state and inherent properties of the network unit itself on the entire neural network model, and the network units with high first contributions tend to be retained during the pruning process. By retaining network units with high first contributions, the accuracy of the neural network model can be improved. The second contribution is used to record the network units that actively learn and adjust to optimize the model during the model training process, and judge the future performance or potential of the network unit based on this. During the pruning process, the network units with high second contributions tend to be retained to expand the generalization ability of the model and improve the stability of model training. The above two factors are evaluated in combination, and in order to balance the trained situation and future needs, the first proportional coefficient and the second proportional coefficient are both set to non-zero values.

[0056] In addition, the numerical values ​​of the first proportional coefficient and the second proportional coefficient are configured by experimental data. Specifically, the network structure and task requirements of the neural network model are different, and the focus of the network units retained during pruning is also different. As an example, the optimal first proportional coefficient and the optimal second proportional coefficient are determined by setting a validation data set and / or cross-validation.

[0057] Optionally, the processor determines a target network unit to be pruned according to a target contribution of each network unit to the performance of the neural network model, and performs a pruning operation to obtain a new neural network model, including:

[0058] The processor determines that the network unit whose target contribution is less than the contribution threshold is the target network unit that needs to be pruned.

[0059] The processor removes the target network unit from the neural network model to obtain a new neural network model.

[0060] In this way, when it is determined that the target contribution is less than the contribution threshold, it indicates that the contribution of the network unit to the neural network model is small, and the importance of the network unit to the neural network model is low. At this time, the network unit whose target contribution is less than the contribution threshold is determined as the target network unit that needs to be pruned, and the target network unit is removed from the neural network model to obtain a new neural network model.

[0061] It can be understood that, when the network unit is a neuron, removing the target network unit from the neural network model includes: removing the target neuron from the neural network model. When the network unit is a network layer, removing the target network unit from the neural network model includes: removing the target network layer from the neural network model.

[0062] As an example, when α×F(i)+β×G(i)≤τ, the i-th network unit is removed.

[0063] Among them, i represents the serial number of the network unit in the neural network model, F(i) represents the first contribution, G(i) represents the second contribution, and τ represents the contribution threshold.

[0064] Optionally, combined Figure 3 As shown, when the network unit is a neuron, the processor determines the first contribution of each network unit to the performance of the neural network model in the forward propagation stage, including:

[0065] S21, the processor determines the evaluation index of each neuron as a weight.

[0066] S22, the processor obtains the weights corresponding to the i-th neuron and each neuron in the previous network layer, and constructs a weight vector Wi corresponding to the i-th neuron. Wherein, i represents the sequence number of the neuron in the neural network model, i=1,3,...,n, and n represents the total number of neurons. In this step, the previous network layer represents the previous network layer of the network layer where the i-th neuron is located.

[0067] S23, the processor determines the evaluation function f(i) of the i-th neuron in the forward propagation stage according to the weight vector Wi corresponding to the i-th neuron. Wherein, f(i) = ||Wi||, ||●|| represents the L1 norm, that is, the sum of the absolute values ​​of the elements of the vector.

[0068] S24, the processor normalizes the evaluation function f(i) to obtain the first contribution of the i-th neuron to the performance of the neural network model in the forward propagation stage.

[0069] Thus, in the case where the network unit is a neuron, the embodiment of the present disclosure first determines the evaluation index of each neuron as a weight, then obtains the weight corresponding to the ith neuron and each neuron in the previous network layer, and constructs the weight vector Wi corresponding to the ith neuron, and determines the evaluation function f(i) of the ith neuron in the forward propagation stage according to Wi. In order to more quickly and effectively compare the first contribution of all neurons, after obtaining the evaluation function f(i), the embodiment of the present disclosure performs a standardization process on it to obtain the first contribution of the ith neuron to the performance of the neural network model in the forward propagation stage, which can ensure the accuracy of the first contribution calculation, which is conducive to improving the accuracy of the target contribution calculation, thereby improving the accuracy of the model pruning operation, and optimizing the model performance through precise pruning.

[0070] Optionally, the processor performs normalization processing on the evaluation function f(i) to obtain the first contribution of the i-th neuron to the performance of the neural network model in the forward propagation stage, including:

[0071] F(i)=||Wi|| / Σ||Wi||.

[0072] Wherein, i=1,3,...,n, n represents the total number of neurons, and F(i) represents the first contribution of the i-th neuron to the performance of the neural network model in the forward propagation stage.

[0073] In this way, the unified standardization of the evaluation function f(i) can be achieved.

[0074] It should be noted that the above embodiment is a simplified and commonly used contribution evaluation example. In actual applications, the first contribution of each neuron to the performance of the neural network model in the forward propagation stage needs to be adaptively adjusted according to the specific task requirements of the neural network model.

[0075] Optionally, combined Figure 4 As shown, in the case where the network unit is a neuron, the processor determines the second contribution of each network unit to the performance of the neural network model during the model training phase, including:

[0076] S31, the processor obtains the current value of the neuron parameter of each neuron in the forward propagation stage, and the neuron parameter includes the weight.

[0077] S32, during the back propagation phase, the processor obtains the update amplitude of the neuron parameter of each network unit with the current value as the reference value.

[0078] S33, the processor determines the average value of the update amplitude of the neuron parameter of each neuron according to the update amplitude of the neuron parameter of each neuron and the number of iterations.

[0079] S34, the processor determines the average value of the update amplitude of each neuron parameter as the second contribution of each neuron to the performance of the neural network model during the model training phase.

[0080] In this way, the current value of the neuron parameter of each network unit can be the activation value of the neuron parameter of each network unit. In the case where the network unit is a neuron, if the activation value of the weight of a neuron in many samples is high and the weight update amplitude is also large, it means that the neuron has a significant contribution to the model performance and has a high degree of influence on the model performance. In this case, it should be retained as much as possible during the pruning process. On the contrary, if the activation value of the weight of a neuron in many samples is low or the weight update amplitude is small, it means that the neuron does not contribute much or even very little to the model performance, and has a low degree of influence on the model performance. In this case, deletion and removal are given priority during the pruning process. Based on the above considerations, the embodiment of the present disclosure determines the average value of the neuron parameter update amplitude of each neuron, and uses it as the second contribution of each neuron to the performance of the neural network model during the model training stage, which is used as a basic reference for model optimization and pruning strategy selection. The accuracy of the second contribution calculation can be guaranteed, which is conducive to improving the accuracy of the target contribution calculation, thereby improving the accuracy of the model pruning operation, and optimizing the model performance through precise pruning.

[0081] It should be noted that when the neural network model has a high complexity, each network layer contains a large number of neurons. At this time, the calculation of the second contribution can be further optimized based on the above embodiment, for example, by using a more complex evaluation standard to calculate the second contribution.

[0082] It can be understood that the above embodiment is a simplified and commonly used contribution evaluation example. In practical applications, other factors need to be considered, including bias terms and / or the activation state of each neuron.

[0083] As an example, in the case where the network unit is a neuron, the processor determines the second contribution of each network unit to the performance of the neural network model during the model training phase, including:

[0084] First, the processor obtains the current weight value wi(t-1) of the weight of the i-th neuron in the forward propagation stage.

[0085] Secondly, during the back propagation phase, the processor obtains the update amplitude Δwi(t) of the weight of each network unit with the current weight value wi(t-1) as the reference value. Among them, Δwi(t) = wi(t)-wi(t-1). wi(t) and wi(t-1) represent the weight value of the i-th neuron at time t and the weight value at time t-1 respectively.

[0086] Then, the processor determines the average value of the weight update amplitude of the i-th neuron according to the update amplitude Δwi(t) of the weight of the i-th neuron and the number of iterations T0. The determination of the average value of the weight update amplitude of the i-th neuron includes:

[0087] Finally, the processor determines the average value G(i) of the weight update amplitude of each neuron as the second contribution of each neuron to the performance of the neural network model during the model training phase.

[0088] Optionally, combined Figure 5 As shown, in the case where the network unit is a network layer, the processor determines the first contribution of each network unit to the performance of the neural network model in the forward propagation stage, including:

[0089] S41, the processor determines that the evaluation index of each network layer is the activation ratio. The activation ratio represents the ratio of the number of neurons activated in the current network layer to the number of all neurons in the current network layer in the forward propagation stage.

[0090] S42, the processor obtains the number N(j) of neurons activated in the j-th network layer in the forward propagation phase.

[0091] S43, the processor compares N(j) with SUM_N0(j) to obtain the activation ratio of the j-th network layer, where SUM_N0(j) represents the number of all neurons in the j-th network layer.

[0092] S44, the processor determines the activation ratio of the j-th network layer as the first contribution of the j-th network layer to the performance of the neural network model in the forward propagation stage.

[0093] In this way, when the network unit is a network layer, if the activation ratio of a certain network layer in many samples is relatively high, it means that the network layer has a significant contribution to the model performance and has a high degree of influence on the model performance. In this case, it should be retained as much as possible during the pruning process. On the contrary, if the activation ratio of a certain network layer in many samples is relatively low, it means that the contribution of the network layer to the model performance is not large or even very small, and the degree of influence on the model performance is low. Based on the above considerations, the embodiment of the present disclosure makes a ratio based on the number of activated neurons in each network layer and the number of all neurons in each network layer to obtain the activation ratio of each network layer, and uses it as the first contribution of each network layer to the performance of the neural network model in the forward propagation stage. The accuracy of the calculation of the first contribution can be guaranteed, which is conducive to improving the accuracy of the calculation of the target contribution, thereby improving the accuracy of the model pruning operation and optimizing the model performance through precise pruning.

[0094] Optionally, the processor obtains the number of neurons N(j) activated in the j-th network layer in the forward propagation phase, including:

[0095] The processor obtains the number of neurons Nl(t, j) activated in the jth network layer in the tth forward propagation stage.

[0096] The processor obtains the total number of neurons activated in the jth network layer in all forward propagation stages SUM_Nl(j), where SUM_Nl(j) = ΣNl(t, j), t = 1, 2, ..., T, and T1 represents the total number of forward propagations.

[0097] The processor compares SUM_Nl(j) with the total number of forward propagation times T1 to determine the number of neurons N(j) activated in the jth network layer during the forward propagation phase.

[0098] In this way, by performing multiple forward propagations and determining the number of neurons activated in each network layer during the forward propagation phase, the accurate activation ratio can be obtained, thereby improving the accuracy and reliability of the first contribution calculation.

[0099] Optionally, combined Figure 6 As shown, in the case where the network unit is a neuron, the processor determines the second contribution of each network unit to the performance of the neural network model during the model training phase, including:

[0100] S51, the processor obtains the output value ela(s, i) of the ith neuron in the sth model training phase, and the output value represents the output value of the ith neuron with the sigmoid function as the activation function. Among them, the output value of the ith neuron with the sigmoid function as the activation function is any real number greater than or equal to 0 and less than or equal to 1. In this step, ela(s, i) can be a continuous value or a discrete value. Whether ela(s, i) is a continuous value or a discrete value is determined by the specific requirements of the neural network model and / or the application scenario of the model.

[0101] S52, the processor obtains the total number of output values ​​of the i-th neuron in all model training stages SUM_ela(i), where SUM_ela(i)=Σela(s, j), s=1, 2, ..., T2, and T2 represents the total number of model training times.

[0102] S53, the processor compares SUM_ela(i) with the total number of model training times T2 to obtain the average output value AVE(i) of the i-th neuron.

[0103] S54, the processor determines the average output value AVE(i) of the ith neuron as the second contribution of the ith neuron to the performance of the neural network model during the model training phase.

[0104] In this way, by obtaining the total number of output values ​​based on ela(s, i), and comparing the total number of output values ​​of the i-th neuron in all model training stages with the total number of model training times, we can obtain the frequency of effective utilization of the i-th neuron during model training or the frequency of actively adjusting the model to optimize the results during model training. This method can enhance the flexibility of the model pruning strategy and better retain neurons that contribute greatly to model performance.

[0105] Optionally, when the network unit is a network layer, the processor determines a second contribution of each network unit to the performance of the neural network model in the model training phase, including:

[0106] The sum of the average output values ​​of each neuron in the j-th network layer is obtained, and the sum of the average output values ​​of each neuron in the j-th network layer is determined as the second contribution of the j-th network layer to the performance of the neural network model in the model training stage.

[0107] In this way, by obtaining the total number of output values ​​based on ela(s, i), and comparing the total number of output values ​​of the i-th neuron in all model training stages with the total number of model training times, we can obtain the frequency of effective utilization of the i-th neuron during model training or the frequency of actively adjusting the model to optimize the results during model training, and determine the second contribution of each network layer to the performance of the neural network model during the model training stage based on the box of the average output value of each neuron in each network layer. This method can enhance the flexibility of the model pruning strategy and better retain the network layers that contribute greatly to the model performance.

[0108] Combination Figure 7 As shown, the embodiment of the present disclosure also provides a pruning method for a neural network model, comprising:

[0109] S61, the processor determines the first contribution of each network unit to the performance of the neural network model in the forward propagation stage.

[0110] S62, the processor determines the second contribution of each network unit to the performance of the neural network model during the model training phase.

[0111] S63, the processor determines the target contribution of each network unit to the performance of the neural network model according to the first contribution and / or the second contribution, wherein the target contribution represents the contribution of the network unit to the performance of the neural network model.

[0112] S64, the processor determines the target network unit that needs to be pruned according to the target contribution of each network unit to the performance of the neural network model, and performs a pruning operation to obtain a new neural network model.

[0113] S65, after the processor performs pruning operation to obtain a new neural network model, the training data set is input into the new neural network model for model training to obtain a fine-tuned neural network model.

[0114] S66, the processor inputs a verification data set to verify the fine-tuned neural network model and obtains a verification indicator.

[0115] S67, when the verification indicator indicates that the expected model performance is not met, the processor returns to determine the first contribution and the second contribution of each network unit, and performs pruning operations based on the first contribution and the second contribution again until the verification indicator meets the expected model performance.

[0116] By adopting the pruning method for a neural network model provided by an embodiment of the present disclosure, the embodiment of the present disclosure can implement an adaptive pruning strategy to accurately remove network units in the neural network model that contribute little to performance, while retaining network units that contribute much to performance. The embodiment of the present disclosure uses a training data set to perform model training on a new neural network model, which can help the model find a new optimal solution and improve model performance. The fine-tuned neural network model is verified using a verification data set, mainly to check whether the various indicators of the model after pruning and fine-tuning (such as precision, recall, etc.) meet the indicator threshold, which helps to determine whether the pruning strategy is successful. If the verification indicator indicates that it does not meet the expected model performance, then return to execute to determine the first contribution and the second contribution and perform pruning operations based on the first contribution and the second contribution again until the verification indicator meets the expected model performance. The adaptive pruning strategy is thereby optimized to further optimize the model performance.

[0117] Combination Figure 8 As shown, the embodiment of the present disclosure provides a pruning device 70 for a neural network model, including a processor (processor) 700 and a memory (memory) 701. Optionally, the device 70 may also include a communication interface (Communication Interface) 702 and a bus 703. Among them, the processor 700, the communication interface 702, and the memory 701 can communicate with each other through the bus 703. The communication interface 702 can be used for information transmission. The processor 700 can call the logic instructions in the memory 701 to execute the pruning method for the neural network model of the above embodiment.

[0118] In addition, the logic instructions in the memory 701 described above may be implemented in the form of software functional units and when sold or used as independent products, may be stored in a computer-readable storage medium.

[0119] The memory 701 is a computer-readable storage medium that can be used to store software programs, computer executable programs, such as program instructions / modules corresponding to the methods in the embodiments of the present disclosure. The processor 700 executes the functional application and data processing by running the program instructions / modules stored in the memory 701, that is, the pruning method for the neural network model in the above-mentioned embodiment is implemented.

[0120] The memory 701 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 701 may include a high-speed random access memory and may also include a non-volatile memory.

[0121] The disclosed embodiment provides an electronic device 100, comprising: a device body, and the above-mentioned pruning device 70 for a neural network model. The pruning device 70 for a neural network model is installed on the device body. The installation relationship described here is not limited to placement inside the device body, but also includes installation connections with other components of the product 100, including but not limited to physical connections, electrical connections, or signal transmission connections. It can be understood by those skilled in the art that the pruning device 70 for a neural network model can be adapted to a feasible device body, thereby realizing other feasible embodiments.

[0122] It should be noted that the electronic device 100 includes a terminal device. The terminal device may be but is not limited to a PC, a mobile phone, a tablet computer, a smart air conditioner, a smart range hood, a smart refrigerator, a smart oven, a smart stove, a smart washing machine, a smart water heater, a smart washing device, a smart dishwasher, a smart projection device, a smart TV, a smart clothes drying rack, a smart curtain, a smart audio and video, a smart socket, a smart speaker, a smart fresh air device, a smart kitchen and bathroom device, a smart bathroom device, a smart sweeping robot, a smart window cleaning robot, a smart mopping robot, a smart air purification device, a smart steamer, a smart microwave oven, a smart kitchen treasure, a smart purifier, a smart water dispenser, a smart door lock, etc.

[0123] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned pruning method for a neural network model.

[0124] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, such as: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes.

[0125] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible changes. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0126] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods for each specific application to implement the described functions, but such implementations should not be considered to exceed the scope of the embodiments of the present disclosure. The technicians may clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above may refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0127] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

[0128] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A pruning method for a neural network model, characterized in that: include: Determine the first contribution of each network unit to the performance of the neural network model during the forward propagation phase; Determine the second contribution of each network unit to the performance of the neural network model during the model training phase; Determine the target contribution of each network unit to the performance of the neural network model according to the first contribution and / or the second contribution; wherein the target contribution represents the contribution of the network unit to the performance of the neural network model; According to the target contribution of each network unit to the performance of the neural network model, the target network unit that needs to be pruned is determined, and the pruning operation is performed to obtain a new neural network model.

2. The pruning method according to claim 1, characterized in that: Determining a target contribution of each network unit to the performance of the neural network model according to the first contribution and / or the second contribution, including: Multiplying the first contribution by the first proportionality coefficient α to determine a first reference contribution; Multiplying the second contribution by the second proportionality coefficient β to determine a second reference contribution; Determine the sum of the first reference contribution and the second reference contribution as the target contribution; Among them, 0<α≤1,0<β≤1.

3. The pruning method according to claim 1, characterized in that: According to the target contribution of each network unit to the performance of the neural network model, the target network unit that needs to be pruned is determined, and a pruning operation is performed to obtain a new neural network model, including: Determine the network unit whose target contribution is less than the contribution threshold as the target network unit that needs to be pruned; Remove the target network unit from the neural network model to obtain a new neural network model.

4. The pruning method according to claim 1, characterized in that: When the network unit is a neuron, the first contribution of each network unit to the performance of the neural network model in the forward propagation stage is determined, including: Determine the evaluation index of each neuron as weight; Obtain the weights corresponding to the i-th neuron and each neuron in the previous network layer, and construct the weight vector Wi corresponding to the i-th neuron; where i represents the sequence number of the neuron in the neural network model, i=1,3,...,n, and n represents the total number of neurons; According to the weight vector Wi corresponding to the i-th neuron, determine the evaluation function f(i) of the i-th neuron in the forward propagation stage; where f(i) = ||Wi||; The evaluation function f(i) is standardized to obtain the first contribution of the i-th neuron to the performance of the neural network model in the forward propagation stage.

5. The pruning method according to claim 4, characterized in that: In the case where the network unit is a neuron, determining the second contribution of each network unit to the performance of the neural network model during the model training phase includes: In the forward propagation phase, the current value of the neuron parameters of each neuron is obtained, and the neuron parameters include weights; In the back-propagation phase, the update amplitude of the neuron parameters of each network unit is obtained based on the current value as the reference value; Determine an average value of the update amplitude of the neuron parameter of each neuron according to the update amplitude of the neuron parameter of each neuron and the number of iterations; The average value of the update amplitude of each neuron parameter is determined as the second contribution of each neuron to the performance of the neural network model during the model training stage.

6. The pruning method according to claim 1, characterized in that: When the network unit is a network layer, determining the first contribution of each network unit to the performance of the neural network model in the forward propagation stage includes: Determine the evaluation index of each network layer as the activation ratio; wherein the activation ratio represents the ratio of the number of neurons activated in the current network layer to the number of all neurons in the current network layer during the forward propagation phase; Obtain the number of neurons N(j) activated in the jth network layer during the forward propagation phase; Ratio N(j) to SUM_N0(j) to obtain the activation ratio of the j-th network layer; where SUM_N0(j) represents the number of all neurons in the j-th network layer; Determine the activation ratio of the j-th network layer as the first contribution of the j-th network layer to the performance of the neural network model in the forward propagation stage.

7. The pruning method according to claim 6, characterized in that: In the case where the network unit is a neuron, determining the second contribution of each network unit to the performance of the neural network model during the model training phase includes: Obtain the output value ela(s, i) of the i-th neuron in the s-th model training stage, where the output value represents the output value of the i-th neuron with the sigmoid function as the activation function; Obtain the total number of output values ​​of the i-th neuron in all model training stages SUM_ela(i); where SUM_ela(i) = Σela(s, j), s = 1, 2, ..., T2, T2 represents the total number of model training times; Ratio SUM_ela(i) to the total number of model training times T2 to obtain the average output value AVE(i) of the i-th neuron; The average output value AVE(i) of the i-th neuron is determined as the second contribution of the i-th neuron to the performance of the neural network model during the model training phase.

8. The pruning method according to any one of claims 1 to 7, characterized in that: Also includes: After performing pruning operations to obtain a new neural network model, the training data set is input into the new neural network model for model training to obtain a fine-tuned neural network model; Input the validation data set to validate the fine-tuned neural network model and obtain validation indicators; When the verification indicator indicates that the expected model performance is not met, return to determine the first contribution and the second contribution of each network unit, and perform pruning operations based on the first contribution and the second contribution again until the verification indicator meets the expected model performance.

9. A pruning device for a neural network model, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute the pruning method for a neural network model as described in any one of claims 1 to 8 when running the program instructions.

10. An electronic device, characterized in that: include: Equipment body; The pruning device for a neural network model as described in claim 9 is installed on the device body.

Images