Model pruning method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose a model pruning method and device, electronic equipment and a storage medium. The method comprises: obtaining training data; obtaining a to-be-pruned model with set training parameters, wherein the training parameters comprise a target proportion of zero channel division, an initial value of a regularization degree coefficient, a total update period of the regularization degree coefficient and an update period of each time; based on the training data and the training parameters, dynamically adjusting the regularization degree coefficient until the relative standard deviation of all channel divisions corresponding to the to-be-pruned model after adjusting the regularization degree coefficient meets a target relative standard deviation, obtaining a target pruning model, and the target relative standard deviation is the relative standard deviation of all channel divisions corresponding to the target proportion; and pruning the target pruning model to obtain a target model. Through the above method, the occupation of GPU resources and time resources is reduced, and the efficiency of regularization degree coefficient adjustment is improved.

Description

Technical Field

[0001] This application belongs to the field of neural network technology, specifically relating to a model pruning method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the emergence of a range of embedded devices, performing inference in deep neural networks presents significant challenges due to their high computational and storage requirements. Pruning is a widely used method in model compression, which involves finding an effective evaluation mechanism to determine the importance of parameters and removing unimportant parameters to reduce model redundancy. However, related model pruning methods require experimenting with multiple regularization coefficients of varying magnitudes during model training. Then, while maintaining the model's predictive ability, the appropriate regularization coefficient is selected to meet the pruning requirements. This approach necessitates tuning the regularization coefficient through multiple training processes, which consumes excessive GPU, human, and time resources, resulting in a long workflow and low efficiency. Summary of the Invention

[0003] In view of the above problems, this application proposes a model pruning method, apparatus, electronic device, and storage medium to improve the above problems.

[0004] In a first aspect, embodiments of this application provide a model pruning method, the method comprising: acquiring training data; acquiring a model to be pruned with training parameters set, wherein the training parameters include a target proportion of zero channel scores, an initial value of a regularization coefficient, a total update cycle of the regularization coefficient, and an update cycle for each update; dynamically adjusting the regularization coefficient based on the training data and the training parameters until the relative standard deviation of all channel scores corresponding to the model to be pruned after adjusting the regularization coefficient satisfies a target relative standard deviation, thereby obtaining a target pruned model, wherein the target relative standard deviation is the relative standard deviation of all channel scores corresponding to the target proportion; and pruning the target pruned model to obtain a target model.

[0005] Secondly, embodiments of this application provide a model pruning device, the device comprising: a data acquisition unit for acquiring training data; a model acquisition unit for acquiring a model to be pruned with pre-set training parameters, wherein the training parameters include a target proportion of zero channel scores, an initial value of the regularization coefficient, a total update cycle of the regularization coefficient, and an update cycle for each update; an adjustment unit for dynamically adjusting the regularization coefficient until the relative standard deviation of all channel scores corresponding to the model to be pruned after adjusting the regularization coefficient satisfies the target relative standard deviation, thereby obtaining a target pruned model, wherein the target relative standard deviation is the relative standard deviation of all channel scores corresponding to the target proportion; and a pruning unit for pruning the target pruned model to obtain a target model.

[0006] Thirdly, embodiments of this application provide an electronic device, including one or more processors and a memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to perform the methods described above.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing program code, wherein the above-described method is executed when the program code is run.

[0008] This application provides a model pruning method, apparatus, electronic device, and storage medium. First, training data and a pre-configured model to be pruned are acquired. The training parameters include a target proportion of zero-channel scores, an initial value of the regularization coefficient, a total update cycle for the regularization coefficient, and an update cycle for each update. Then, based on the training data and training parameters, the regularization coefficient is dynamically adjusted until the relative standard deviation of all channel scores corresponding to the pruned model after adjusting the regularization coefficient meets the target relative standard deviation, thus obtaining the target pruned model. The target relative standard deviation is the relative standard deviation of all channel scores corresponding to the target proportion. Finally, the target pruned model is pruned to obtain the target model. This method makes the regularization coefficient an automatically adjustable parameter during model training. It can automatically adjust the regularization coefficient to meet the pruning proportion requirements while maintaining the model's predictive ability, reducing the consumption of GPU and time resources and improving the efficiency of regularization coefficient adjustment. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This illustration shows an application scenario diagram of a model pruning method proposed in an embodiment of this application;

[0011] Figure 2 This illustration shows an application scenario diagram of a model pruning method proposed in an embodiment of this application;

[0012] Figure 3 A flowchart of a model pruning method according to an embodiment of this application is shown;

[0013] Figure 4 A flowchart of a model pruning method according to another embodiment of this application is shown;

[0014] Figure 5 A flowchart of a model pruning method according to another embodiment of this application is shown;

[0015] Figure 6 This paper shows a structural block diagram of a model pruning device according to an embodiment of the present application;

[0016] Figure 7 A structural block diagram of an electronic device for performing a model pruning method according to an embodiment of this application is shown;

[0017] Figure 8 The illustration shows a storage unit for storing or carrying program code that implements the model pruning method according to the embodiments of the present application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0019] Currently, the industry commonly uses structured pruning techniques to prune the weights of the BN (Batch Normalization) layer or the channels corresponding to the convolution kernels of the convolutional layer for visual model pruning.

[0020] We often use the following two approaches to prune models using these common techniques:

[0021] The first approach involves prioritizing the importance of each channel in the model from low to high, and then pruning the model using a fixed pruning ratio. However, this forced pruning often results in an unbalanced model structure, and even with fine-tuning or retraining, the model's generalization and performance capabilities still decline significantly. Therefore, this method of forcibly setting pruning ratios is generally not used.

[0022] The second approach is to perform weight decay on the weights of the BN layer or the convolution kernels of the convolutional layer, so that the model generates a large number of BN weights close to zero or convolution kernels with an L2 norm close to zero. Then, the channels corresponding to these BN weights or convolution kernels with an L2 norm close to zero are clipped.

[0023] The inventors discovered in their research on related model pruning methods that, in the second approach, during model training, weight decay is applied to the weights of the BN layers or the kernels of the convolutional layers. This causes the L2 norm of some BN layer weights or convolutional layer kernels to decay to near zero. Therefore, these BN layer weights or convolutional layer kernels, along with their corresponding channels, can be physically removed from the model without compromising its structural consistency. This channel-based structured pruning weight decay can be considered a form of channel decay.

[0024] The channel decay method described above can often reduce model size and computational cost while maintaining good predictive performance. However, this method cannot accurately control the proportion of zero-weight or zero-kernel convolutions, resulting in inaccurate prediction of the compression degree of the pruned model. In practice, we often need to experiment with multiple regularization coefficients of different sizes, and then select the regularization coefficient that meets the pruning requirements while ensuring the model's predictive ability. This approach requires tuning the regularization coefficient by starting multiple training processes, which consumes excessive GPU, human, and time resources, resulting in a long workflow and low efficiency.

[0025] Therefore, the inventors have proposed the model pruning method, apparatus, electronic device, and storage medium of this application. First, training data and a model to be pruned with pre-set training parameters are acquired. These training parameters include the target proportion of zero-channel scores, the initial value of the regularization coefficient, the total update cycle of the regularization coefficient, and the update cycle for each update. Then, based on the training data and training parameters, the regularization coefficient is dynamically adjusted until the relative standard deviation of all channel scores corresponding to the pruned model after adjusting the regularization coefficient meets the target relative standard deviation, thus obtaining the target pruned model. The target relative standard deviation is the relative standard deviation of all channel scores corresponding to the target proportion. Finally, the target pruned model is pruned to obtain the target model. Through this method, the regularization coefficient becomes an automatically adjustable parameter during model training. While ensuring the model's predictive ability, the regularization coefficient can be automatically adjusted to meet the pruning proportion requirements, reducing the consumption of GPU and time resources and improving the efficiency of regularization coefficient adjustment.

[0026] The application scenarios involved in the embodiments of this application will be introduced below.

[0027] In this embodiment, the provided model pruning method can be executed by an electronic device. In this mode, all steps of the model pruning method provided in this embodiment can be performed by the electronic device. Alternatively, it can be executed by a server. Correspondingly, in this mode, all steps of the model pruning method provided in this embodiment can be performed by the server. Furthermore, it can be executed collaboratively by an electronic device and a server. In this mode, some steps of the model pruning method provided in this embodiment are performed by the electronic device, while other steps are performed by the server.

[0028] For example, such as Figure 1 As shown, the processor in the electronic device 100 can execute a model pruning method including: acquiring training data; acquiring a model to be pruned with pre-set training parameters, wherein the training parameters include a target proportion of zero channel scores, an initial value of the regularization coefficient, a total update cycle of the regularization coefficient, and an update cycle for each update; dynamically adjusting the regularization coefficient based on the training data and the training parameters until the relative standard deviation of all channel scores corresponding to the pruned model after adjusting the regularization coefficient satisfies the target relative standard deviation, thereby obtaining a target pruned model, wherein the target relative standard deviation is the relative standard deviation of all channel scores corresponding to the target proportion; and pruning the target pruned model to obtain a target model.

[0029] Furthermore, the model pruning method provided in this application embodiment can also be executed by a server (cloud). Correspondingly, in this server-executed method, the server can obtain the model to be pruned with its training parameters set in real time. The training parameters include the target proportion of zero-channel scores, the initial value of the regularization coefficient, the total update cycle of the regularization coefficient, and the update cycle for each update. Based on the training data and the training parameters, the regularization coefficient is dynamically adjusted until the relative standard deviation of all channel scores corresponding to the pruned model after adjusting the regularization coefficient meets the target relative standard deviation, thus obtaining the target pruned model. The target relative standard deviation is the relative standard deviation of all channel scores corresponding to the target proportion. The target pruned model is then pruned to obtain the target model.

[0030] Alternatively, the motion capture method can be executed collaboratively by an electronic device and a server. In this collaborative execution method, some steps of the motion capture method provided in this application are executed by the electronic device, while other steps are executed by the server.

[0031] For example, such as Figure 2 As shown, the electronic device 100 can perform a model pruning method including: acquiring training data; acquiring a model to be pruned with pre-set training parameters, wherein the training parameters include a target proportion of zero channel scores, an initial value of the regularization coefficient, a total update cycle of the regularization coefficient, and an update cycle for each update; dynamically adjusting the regularization coefficient based on the training data and the training parameters until the relative standard deviation of all channel scores corresponding to the pruned model after adjusting the regularization coefficient meets the target relative standard deviation, thereby obtaining a target pruned model, wherein the target relative standard deviation is the relative standard deviation of all channel scores corresponding to the target proportion; and then the server 200 performs pruning on the target pruned model to obtain the target model.

[0032] It should be noted that in this method where electronic devices and servers work together, the steps performed by the electronic devices and servers are not limited to those described in the examples above. In practical applications, the steps performed by the electronic devices and servers can be dynamically adjusted according to the actual situation.

[0033] It should be noted that, in addition to being able to... Figure 1 and Figure 2 In addition to smartphones, the devices shown can also be in-vehicle systems, wearable devices, tablets, laptops, smart speakers, etc. Server 120 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers.

[0034] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0035] Please see Figure 3 The model pruning method provided in this application embodiment is applied to, for example, Figure 1 or Figure 2 The method includes the following: (illustrated electronic device or server)

[0036] Step S110: Obtain training data.

[0037] In this embodiment, the training data can be of various types. The training data can be determined based on the function of the model to be pruned, with a one-to-one correspondence between the training data and the function of the model. For example, if the function of the model to be pruned is for face liveness detection, then the corresponding training data can be a set of face images; if the function of the model to be pruned is for action detection, then the corresponding training data can be a set of images including multiple actions or a set of videos containing multiple actions, etc., without specific limitations.

[0038] As one approach, training data can be pre-stored in a preset storage area. When training data is needed, it can be retrieved directly from the preset storage area. Furthermore, when storing training data in the preset storage area, a correspondence can be established between the training data and the model's performance. This allows the system to retrieve the corresponding training data from the preset storage area based on the current performance of the model when needed.

[0039] It is understood that the training data in the embodiments of this application can be data from some known datasets, or it can be data collected or acquired by electronic devices in real time, without any specific limitation here.

[0040] Step S120: Obtain the pruning model with the training parameters set, wherein the training parameters include the target proportion of zero channel scores, the initial value of the regularization coefficient, the total update cycle of the regularization coefficient, and the update cycle of each update.

[0041] In this embodiment of the application, the target proportion of zero channel score is the proportion of channels in the model with zero channel score when the model training ends. Channel score can be the absolute value of the weights of the BN layer of the model or the L2 norm of the convolution kernel. Zero channel score means that the absolute value of the weights of the BN layer of the model or the L2 norm of the convolution kernel is close to or equal to zero.

[0042] The regularization coefficient is a hyperparameter used to control the degree of regularization. The initial value of the regularization coefficient is a pre-set value. To ensure model convergence, the initial value of the regularization coefficient can be set to a very small value, for example, 10. -8 .

[0043] The total update cycle of the regularization coefficient refers to the total number of times the regularization coefficient is dynamically adjusted during model training. Generally, the total update cycle of the regularization coefficient is shorter than the total training cycle of the model. The total training cycle can be measured by the number of times data batches are fed into the model. For example, if a model requires 300,000 iterations to train, the end time of the total update cycle of the regularization coefficient can be set to the 100,000th or 200,000th iteration of the model.

[0044] The update cycle for each update refers to the time interval between each update of the regularization coefficient. Similarly, the update cycle for each update of the regularization coefficient can also be counted by the number of times the data batch is fed into the model. For example, the update cycle for each update of the regularization coefficient can be set to update once every 50 iterations or once every 100 iterations.

[0045] In this embodiment, the target proportion of zero-channel scores, the initial value of the regularization coefficient, the total update cycle of the regularization coefficient, and the update cycle of each update of the model to be pruned are all set before training the model. The target proportion of zero-channel scores, the initial value of the regularization coefficient, the total update cycle of the regularization coefficient, and the update cycle of each update can all be set by the relevant designers according to actual needs, and are not specifically limited here.

[0046] Of course, the model to be pruned in this application embodiment can be a model such as DenseNet (Densely connected convolutional networks), MobileNetV2, DepthNet, etc., and no specific limitation is made here.

[0047] Step S130: Based on the training data and the training parameters, dynamically adjust the regularization coefficient until the relative standard deviation of all channel scores of the model to be pruned after adjusting the regularization coefficient meets the target relative standard deviation, thereby obtaining the target pruned model. The target relative standard deviation is the relative standard deviation of all channel scores corresponding to the target ratio.

[0048] In this embodiment of the application, after obtaining the training data and setting the training parameters of the model to be pruned, the hyperparameter-regularization coefficient in the model to be pruned can be dynamically adjusted based on the training data and training parameters.

[0049] Specifically, during the training of the model to be pruned based on training data and training parameters, the value of the regularization coefficient is adjusted once every update cycle of the regularization coefficient. For example, when training the model to be pruned begins, the value of the regularization coefficient needs to be adjusted the first time an update cycle is reached. At this time, the initial value of the regularization coefficient is adjusted. When the next update cycle of the regularization coefficient is reached, the value of the regularization coefficient after the first adjustment is adjusted, and so on.

[0050] In the process of dynamically adjusting the regularization coefficient, before each adjustment of the current regularization coefficient, it is necessary to combine the relative standard deviation of all channel scores corresponding to the model to be pruned to determine whether it is necessary to continue adjusting the regularization coefficient.

[0051] The relative standard deviation of all channels can be calculated using the following formula:

[0052] in, Furthermore, f in the above formula Y (x;σ 2 ) represents the folded normal distribution of the standard normal distribution; σ 2 μ represents the variance of the standard normal distribution. Y α represents the expected value of the folded normal distribution, which is the expected value of the channel fractions that have not decayed to zero; α represents the proportion of non-zero channel fractions; σ r (α) represents the relative standard deviation of all channel scores; x represents the channel score of a channel.

[0053] It is understood that this application's embodiments propose an assumption that a deep CNN vision model should satisfy the following rule: if the weights of its Batch Normalization (BN) layer or the convolutional kernels of its convolutional layers undergo dynamic channel decay, then at the end of training, the distribution of its channel fractions satisfies a folded normal distribution of the standard normal distribution (a normal distribution with a mean of zero), as shown in the formula above. As shown.

[0054] Through formula It can be seen that the higher the proportion of zero channel scores (1-α) (i.e., the lower the proportion of non-zero channel scores (α)), the larger the relative standard deviation of all channel scores. Therefore, we can pre-construct a one-to-one mapping relationship between the proportion of zero channel scores (1-α) and the relative standard deviation of all channel scores. We can call this mapping relationship the "relative standard deviation hypothesis of channel scores".

[0055] Through calculation, a relational table as shown below was constructed:

[0056] (1-α)×100 0 1 2 3 4 5 6 7 8 9 relative standard deviation 0.756 0.766 0.776 0.787 0.798 0.808 0.819 0.830 0.841 0.852 (1-α)×100 10 11 12 13 14 15 16 17 18 19 relative standard deviation 0.863 0.875 0.886 0.898 0.909 0.921 0.933 0.945 0.957 0.969 (1-α)×100 20 21 22 23 24 25 26 27 28 39 relative standard deviation 0.982 0.994 1.007 1.020 1.033 1.046 1.060 1.073 1.087 1.101 (1-α)×100 30 31 32 33 34 35 36 37 38 39 relative standard deviation 1.115 1.130 1.145 1.160 1.175 1.190 1.206 1.222 1.238 1.255 (1-α)×100 40 41 42 43 44 45 46 47 48 49 relative standard deviation 1.272 1.289 1.307 1.325 1.344 1.362 1.382 1.401 1.422 1.442 (1-α)×100 50 51 52 53 54 55 56 57 58 59 relative standard deviation 1.463 1.485 1.507 1.530 1.554 1.578 1.603 1.629 1.655 1.683 (1-α)×100 60 61 62 63 64 65 66 67 68 69 relative standard deviation 1.711 1.740 1.770 1.801 1.834 1.868 1.903 1.939 1.977 2.017 (1-α)×100 70 71 72 73 74 75 76 77 78 79 relative standard deviation 2.058 2.102 2.147 2.195 2.245 2.299 2.355 2.414 2.478 2.546 (1-α)×100 80 81 82 83 84 85 86 87 88 89 relative standard deviation 2.618 2.696 2.780 2.871 2.969 3.078 3.197 3.329 3.477 3.644 (1-α)×100 90 91 92 93 94 95 96 97 98 99 relative standard deviation 3.835 4.056 4.317 4.630 5.018 5.515 6.186 7.167 8.806 12.493

[0057] As shown in the table above, each target proportion of zero-channel scores corresponds to a relative standard deviation. Therefore, achieving the target proportion of zero-channel scores can be successfully transformed into achieving the target value of the relative standard deviation of channel scores. Thus, during training, by automatically adjusting the regularization coefficient to control the relative standard deviation of channel scores, and ensuring the relative standard deviation reaches the target value before the model training ends, the target proportion of zero-channel scores can be achieved, thereby fulfilling the corresponding pruning objective. Optionally, in the process of automatically adjusting the regularization coefficient to control the relative standard deviation of channel scores, channel-based structured pruning regularization can be used to attenuate the channel scores, causing some channels' scores to decay to near zero, thus making the proportion of zero-channel scores approach the target proportion.

[0058] In this embodiment, the target relative standard deviation is the relative standard deviation of all channel scores corresponding to the target proportion of zero channel score, and it is also the relative standard deviation of all channel scores of the target pruning model when the pruning model training ends.

[0059] After obtaining the model to be pruned with the training parameters set, the target relative standard deviation can be determined by looking up the table above based on the target proportion of zero channel scores set in the training parameters. For example, if the target proportion of zero channel scores is set to 40%, the corresponding target relative standard deviation can be found to be 1.272 by looking up the table above.

[0060] Step S140: Prune the target pruning model to obtain the target model.

[0061] In this embodiment of the application, after obtaining the target pruning model, the channel scores corresponding to each channel in the target pruning model are sorted in order from high to low or from low to high to obtain a sorted channel sequence. Then, through structured pruning, without destroying the consistency of the model structure, the channels corresponding to zero channel scores in the sorted channel sequence are physically deleted according to the target proportion of zero channel scores to obtain the target model.

[0062] In this embodiment of the application, the process described in steps S110-S140 can be integrated into the pruning and compression tool, so that any electronic device or server can achieve dynamic channel attenuation through the pruning and compression tool.

[0063] This application provides a model pruning method that makes the regularization coefficient an automatically adjustable parameter during model training. It can automatically adjust the regularization coefficient to meet the pruning ratio requirements while ensuring the model's predictive ability, thereby reducing the consumption of GPU and time resources and improving the efficiency of regularization coefficient adjustment.

[0064] Please see Figure 4 The model pruning method provided in this application embodiment is applied to, for example, Figure 1 or Figure 2 The method includes the following: (illustrated electronic device or server)

[0065] Step S210: Obtain training data.

[0066] Step S220: Obtain the model to be pruned with the training parameters set, wherein the training parameters include the target proportion of zero channel scores, the initial value of the regularization coefficient, the total update cycle of the regularization coefficient, and the update cycle of each update.

[0067] Step S230: During the process of dynamically adjusting the regularization coefficient based on the training data and the training parameters, before the start of the current update cycle, the first relative standard deviation is obtained. The first relative standard deviation is the relative standard deviation of all channels of the model to be pruned at the end of the previous update cycle.

[0068] In this embodiment, after training of the model to be pruned is initiated, at the current update cycle of the regularization coefficient, the relative standard deviation of all channel scores is recalculated, i.e., the first relative standard deviation is calculated. The first relative standard deviation can be calculated based on the channel scores corresponding to each channel of the model to be pruned before the current update cycle, i.e., using the formula... formula and formula Calculated.

[0069] It can be seen that if the next update cycle is the first update cycle of the regularization coefficient, then the first relative standard deviation at this time refers to the relative standard deviation of all channels when the model to be pruned just started training.

[0070] Step S240: At the end of the current update cycle, obtain the second relative standard deviation, which is the relative standard deviation of all channels corresponding to the model to be pruned at the end of the current update cycle.

[0071] In this embodiment, at the arrival of the next update cycle, the relative standard deviation of all channel scores is recalculated, i.e., a second relative standard deviation is calculated. The second relative standard deviation can be calculated based on the channel scores corresponding to each channel of the model to be pruned before the next update cycle, i.e., using the formula... formula and formula Calculated.

[0072] Step S250: Based on the first relative standard deviation and the second relative standard deviation, at the end of the current update cycle, the current regularization coefficient is dynamically adjusted until the relative standard deviation of all channels corresponding to the model to be pruned after adjusting the regularization coefficient meets the target relative standard deviation, thus obtaining the target pruned model.

[0073] In this embodiment of the application, during the process of dynamically adjusting the regularization coefficient, it is necessary to statistically analyze the relative standard deviation of each two adjacent update cycles of the regularization coefficient. This allows for the determination of how to adjust the current regularization coefficient based on the relative standard deviation of each two adjacent update cycles of the regularization coefficient.

[0074] In one embodiment of this application, the step of dynamically adjusting the current regularization coefficient at the end of the current update cycle based on the first relative standard deviation and the second relative standard deviation includes: if the second relative standard deviation is greater than or equal to the target relative standard deviation before the end of the total update cycle of the regularization coefficient, the current regularization coefficient is adjusted to the minimum value at the end of the current update cycle.

[0075] Specifically, before the end of the total update cycle for the regularization coefficient, if the current relative standard deviation meets or even exceeds the target relative standard deviation, the current regularization coefficient is directly updated to the minimum value. This minimum value can be the same as the initial value of the regularization coefficient; that is, the minimum value can be 10. -8 .

[0076] As another approach, the dynamic adjustment of the current regularization coefficient based on the first relative standard deviation and the second relative standard deviation at the end of the current update cycle includes: after the end of the total update cycle of the regularization coefficient, if the second relative standard deviation is greater than or equal to the target relative standard deviation, the current regularization coefficient is adjusted to the minimum value; if the second relative standard deviation is less than the target relative standard deviation, the current regularization coefficient is adjusted to the first value.

[0077] Specifically, after the end of the total update cycle of the regularization coefficient, if the current relative standard deviation meets or even exceeds the target relative standard deviation, the current regularization coefficient is directly updated to the minimum value.

[0078] If the current relative standard deviation does not reach the target relative standard deviation, the current regularization coefficient is updated to the first value. The first value is the smaller of {2λ, 10}, where λ represents the current regularization coefficient.

[0079] In summary, since the total execution cycle of dynamic channel decay is shorter than the total training cycle of the model, the relative standard deviation of the channel scores will automatically drop to a value close to zero after reaching or exceeding the target value during the training process. Therefore, compared with the traditional decay method that keeps the value constant, dynamic channel decay is more conducive to optimizing the prediction accuracy in the later stages of model training.

[0080] Step S260: Prune the target pruning model to obtain the target model.

[0081] This application provides a model pruning method that, during the dynamic adjustment of the regularization coefficient, obtains the relative standard deviation before and after each adjustment in real time. Based on the standard deviation before and after each adjustment, the regularization coefficient is dynamically adjusted, making the regularization coefficient an automatically adjustable parameter during model training. While ensuring the model's predictive ability, the regularization coefficient can be automatically adjusted to meet the pruning ratio requirements, reducing the consumption of GPU and time resources and improving the efficiency of regularization coefficient adjustment.

[0082] Please see Figure 5 The model pruning method provided in this application embodiment is applied to, for example, Figure 1 or Figure 2 The method includes the following: (illustrated electronic device or server)

[0083] Step S310: Obtain training data.

[0084] Step S320: Obtain the pruning model with the training parameters set, wherein the training parameters include the target proportion of zero channel scores, the initial value of the regularization coefficient, the total update cycle of the regularization coefficient, and the update cycle of each update.

[0085] Step S330: During the process of dynamically adjusting the regularization coefficient based on the training data and the training parameters, before the start of the current update cycle, the first relative standard deviation is obtained. The first relative standard deviation is the relative standard deviation of all channels of the model to be pruned at the end of the previous update cycle.

[0086] Step S340: At the end of the current update cycle, obtain the second relative standard deviation, which is the relative standard deviation of all channels corresponding to the model to be pruned at the end of the current update cycle.

[0087] Step S350: Based on the first relative standard deviation and the second relative standard deviation, determine the first rate of change of the relative standard deviation.

[0088] In this embodiment, the first rate of change refers to the actual rate of change of the regularization coefficient. The first rate of change = (second relative standard deviation - first relative standard deviation) / update period corresponding to the second relative standard deviation.

[0089] Step S360: Based on the second relative standard deviation and the target relative standard deviation, determine the second rate of change of the relative standard deviation.

[0090] In this embodiment, the second rate of change refers to the expected rate of change of the relative standard deviation. The second rate of change = (target relative standard deviation - second relative standard deviation) / remaining update period corresponding to the second relative standard deviation. Wherein, remaining update period = total update period - already updated period.

[0091] Step S370: Before the end of the total update cycle of the regularization coefficient, if the second relative standard deviation is less than the target relative standard deviation and the first rate of change is less than one-third of the second rate of change, the current regularization coefficient is adjusted to the first value at the end of the current update cycle.

[0092] In this embodiment, after obtaining the first rate of change and the second rate of change, if the current relative standard deviation does not reach the target relative standard value and the first rate of change is less than one-third of the second rate of change before the end of the total update cycle of the regularization coefficient, it indicates that the current value of the regularization coefficient is small and needs to be increased to accelerate the rate of change of the relative standard deviation. At this time, the current regularization coefficient can be updated to the smaller value among {2λ, 10}.

[0093] Step S380: If the second relative standard deviation is less than the target relative standard deviation, and the first rate of change is greater than three times the second rate of change, at the end of the current update cycle, the current regularization coefficient is adjusted to the second value, wherein the first value is greater than the second value.

[0094] In this embodiment, before the end of the total update cycle of the regularization coefficient, if the current relative standard deviation has not reached the target relative standard value, and the first rate of change is greater than three times the second rate of change, it indicates that the current regularization coefficient is too large and needs to be reduced to slow down the rate of change of the relative standard deviation. At this time, the current regularization coefficient can be updated to {λ / 2, 10}. -8 The larger value in}.

[0095] Step S390: If the second relative standard deviation is less than the target relative standard deviation, and the first rate of change is greater than or equal to one-third of the second rate of change, and the first rate of change is less than or equal to three times the second rate of change, then at the end of the current update cycle, maintain the current regularization coefficient.

[0096] In this embodiment of the application, if the current relative standard deviation does not reach the target relative standard value before the end of the total update cycle of the regularization coefficient, and the first rate of change is greater than or equal to one-third of the second rate of change, and the first rate of change is less than or equal to three times the second rate of change, it indicates that the current value of the regularization coefficient is appropriate, and there is no need to adjust the current regularization coefficient.

[0097] Step S391: When the relative standard deviation of all channels in the model to be pruned after adjusting the regularization coefficient meets the target relative standard deviation, the target pruning model is obtained.

[0098] After training the model to be pruned using dynamic channel decay, we obtain the distribution of all channels whose relative standard deviation reaches the target relative standard deviation, which is the target pruned model.

[0099] Step S392: Prune the target pruning model to obtain the target model.

[0100] This application provides a model pruning method that, during the dynamic adjustment of the regularization coefficient, obtains the relative standard deviation before and after each adjustment in real time. Based on the standard deviation before and after each adjustment, the regularization coefficient is dynamically adjusted, making the regularization coefficient an automatically adjustable parameter during model training. While ensuring the model's predictive ability, the regularization coefficient can be automatically adjusted to meet the pruning ratio requirements, reducing the consumption of GPU and time resources and improving the efficiency of regularization coefficient adjustment.

[0101] Please see Figure 6 The present application provides a model pruning device 400, which operates as follows: Figure 1 or Figure 2 The electronic device or server shown, the device 400 includes:

[0102] The data acquisition unit 410 is used to acquire training data.

[0103] The model acquisition unit 420 is used to acquire the model to be pruned with the training parameters set. The training parameters include the target proportion of zero channel scores, the initial value of the regularization coefficient, the total update cycle of the regularization coefficient, and the update cycle of each update.

[0104] The adjustment unit 430 is used to dynamically adjust the regularization coefficient until the relative standard deviation of all channel scores of the model to be pruned after adjusting the regularization coefficient meets the target relative standard deviation, thereby obtaining the target pruning model. The target relative standard deviation is the relative standard deviation of all channel scores corresponding to the target ratio.

[0105] In one manner, the adjustment unit 430 is specifically used to, during the process of dynamically adjusting the regularization coefficient based on the training data and the training parameters, obtain a first relative standard deviation before the start of the current update cycle, wherein the first relative standard deviation is the relative standard deviation of all channel scores corresponding to the model to be pruned at the end of the previous update cycle; obtain a second relative standard deviation at the end of the current update cycle, wherein the second relative standard deviation is the relative standard deviation of all channel scores corresponding to the model to be pruned at the end of the current update cycle; and dynamically adjust the current regularization coefficient at the end of the current update cycle based on the first relative standard deviation and the second relative standard deviation.

[0106] Alternatively, the adjustment unit 430 is specifically used to adjust the current regularization coefficient to the minimum value at the end of the current update cycle if the second relative standard deviation is greater than or equal to the target relative standard deviation before the end of the total update cycle of the regularization coefficient.

[0107] Optionally, the adjustment unit 430 is specifically configured to: determine a first rate of change of the relative standard deviation based on the first relative standard deviation and the second relative standard deviation; determine a second rate of change of the relative standard deviation based on the second relative standard deviation and the target relative standard deviation; before the end of the total update cycle of the regularization coefficient, if the second relative standard deviation is less than the target relative standard deviation and the first rate of change is less than one-third of the second rate of change, adjust the current regularization coefficient to a first value at the end of the current update cycle; if the second relative standard deviation is less than the target relative standard deviation and the first rate of change is greater than three times the second rate of change, adjust the current regularization coefficient to a second value at the end of the current update cycle, wherein the first value is greater than the second value; if the second relative standard deviation is less than the target relative standard deviation and the first rate of change is greater than or equal to one-third of the second rate of change and less than or equal to three times the second rate of change, maintain the current regularization coefficient at the end of the current update cycle.

[0108] Optionally, the adjustment unit 430 is further configured to obtain the difference between the second relative standard deviation and the first relative standard deviation; and to determine a first rate of change of the relative standard deviation based on the difference and the current update cycle.

[0109] Optionally, the adjustment unit 430 is further configured to obtain the difference between the target relative standard deviation and the second relative standard deviation; and to determine the second rate of change of the relative standard deviation based on the difference and the current remaining update cycle.

[0110] Optionally, the adjustment unit 430 is further configured to, after the end of the total update cycle of the regularization coefficient, adjust the current regularization coefficient to the minimum value if the second relative standard deviation is greater than or equal to the target relative standard deviation; and adjust the current regularization coefficient to the first value if the second relative standard deviation is less than the target relative standard deviation.

[0111] The pruning unit 440 is used to prune the target pruning model to obtain the target model.

[0112] It should be noted that the device embodiments in this application correspond to the aforementioned method embodiments. The specific principles in the device embodiments can be found in the content of the aforementioned method embodiments, and will not be repeated here.

[0113] The following will combine Figure 7 This application describes an electronic device or server provided.

[0114] Please see Figure 7Based on the aforementioned model pruning method and apparatus, this application embodiment also provides another electronic device or server 800 capable of executing the aforementioned model pruning method. The electronic device or server 800 includes one or more (only one shown in the figure) processors 802, a memory 804, and a network module 806 coupled together. The memory 804 stores programs capable of executing the contents of the aforementioned embodiments, and the processor 802 can execute the programs stored in the memory 804.

[0115] The processor 802 may include one or more processing cores. The processor 802 connects to various parts of the electronic device or server 800 using various interfaces and lines. It executes various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 804, and by calling data stored in the memory 804. Optionally, the processor 802 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 802 may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 802 and may be implemented separately using a communication chip.

[0116] The memory 804 may include random access memory (RAM) or read-only memory (ROM). The memory 804 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 804 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created during the use of the electronic device or server 800 (such as phonebooks, audio and video data, chat log data, etc.).

[0117] The network module 806 is used to receive and transmit electromagnetic waves, realizing the mutual conversion between electromagnetic waves and electrical signals, thereby communicating with communication networks or other devices, such as electronic devices. The network module 806 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, SIM cards, memory, etc. The network module 806 can communicate with various networks such as the Internet, corporate intranets, and wireless networks, or communicate with other devices through wireless networks. The aforementioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. For example, the network module 806 can interact with base stations.

[0118] Please refer to Figure 8 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable storage medium 900 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0119] The computer-readable storage medium 900 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 900 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 900 has storage space for program code 910 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 910 may be compressed, for example, in a suitable form.

[0120] This application provides a model pruning method, apparatus, electronic device, and storage medium. First, training data and a pre-configured model to be pruned are acquired. The training parameters include a target proportion of zero-channel scores, an initial value of the regularization coefficient, a total update cycle for the regularization coefficient, and an update cycle for each update. Then, based on the training data and training parameters, the regularization coefficient is dynamically adjusted until the relative standard deviation of all channel scores in the pruned model after adjusting the regularization coefficient meets the target relative standard deviation, thus obtaining the target pruned model. The target relative standard deviation is the relative standard deviation of all channel scores corresponding to the target proportion. Finally, the target pruned model is pruned to obtain the target model. This method makes the regularization coefficient an automatically adjustable parameter during model training. It can automatically adjust the regularization coefficient to meet the pruning proportion requirements while maintaining the model's predictive ability, reducing the consumption of GPU and time resources and improving the efficiency of regularization coefficient adjustment.

[0121] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A model pruning method, characterized in that, The method includes: Obtain training data; Obtain the pruning model with the training parameters set, wherein the training parameters include the target proportion of zero channel scores, the initial value of the regularization coefficient, the total update cycle of the regularization coefficient, and the update cycle of each update. Based on the training data and the training parameters, the regularization coefficient is dynamically adjusted until the relative standard deviation of all channel scores of the model to be pruned after adjusting the regularization coefficient meets the target relative standard deviation, thus obtaining the target pruned model. The target relative standard deviation is the relative standard deviation of all channel scores corresponding to the target ratio. If the model to be pruned is a model for face liveness detection, the training data is a set of face images; if the model to be pruned is a model for action detection, the training data is a set of images or videos including multiple actions. The step of dynamically adjusting the regularization coefficient based on the training data and the training parameters includes: During the process of dynamically adjusting the regularization coefficient based on the training data and the training parameters, before the start of the current update cycle, the first relative standard deviation is obtained. The first relative standard deviation is the relative standard deviation of all channels of the model to be pruned at the end of the previous update cycle. At the end of the current update cycle, the second relative standard deviation is obtained. The second relative standard deviation is the relative standard deviation of all channels corresponding to the model to be pruned at the end of the current update cycle. Based on the first relative standard deviation and the second relative standard deviation, the current regularization coefficient is dynamically adjusted at the end of the current update cycle. The target pruning model is pruned to obtain a target model, which is then deployed to an embedded device so that the embedded device runs the target model to perform face liveness recognition or motion detection tasks.

2. The method according to claim 1, characterized in that, The step of dynamically adjusting the current regularization coefficient at the end of the current update cycle based on the first relative standard deviation and the second relative standard deviation includes: Before the end of the total update cycle of the regularization coefficient, if the second relative standard deviation is greater than or equal to the target relative standard deviation, the current regularization coefficient will be adjusted to the minimum value at the end of the current update cycle.

3. The method according to claim 1, characterized in that, The step of dynamically adjusting the current regularization coefficient at the end of the current update cycle based on the first relative standard deviation and the second relative standard deviation includes: Based on the first relative standard deviation and the second relative standard deviation, a first rate of change of the relative standard deviation is determined; Based on the second relative standard deviation and the target relative standard deviation, a second rate of change of the relative standard deviation is determined; Before the end of the total update cycle of the regularization coefficient, if the second relative standard deviation is less than the target relative standard deviation and the first rate of change is less than one-third of the second rate of change, the current regularization coefficient will be adjusted to the first value at the end of the current update cycle. If the second relative standard deviation is less than the target relative standard deviation, and the first rate of change is greater than three times the second rate of change, at the end of the current update cycle, the current regularization coefficient is adjusted to the second value, wherein the first value is greater than the second value. If the second relative standard deviation is less than the target relative standard deviation, and the first rate of change is greater than or equal to one-third of the second rate of change, and the first rate of change is less than or equal to three times the second rate of change, the current regularization coefficient is maintained at the end of the current update cycle.

4. The method according to claim 3, characterized in that, The step of determining the first rate of change of the relative standard deviation based on the first relative standard deviation and the second relative standard deviation includes: Obtain the difference between the second relative standard deviation and the first relative standard deviation; Based on the difference and the current update cycle, a first rate of change of the relative standard deviation is determined.

5. The method according to claim 3, characterized in that, The step of determining the second rate of change of the relative standard deviation based on the first relative standard deviation and the second relative standard deviation includes: Obtain the difference between the target relative standard deviation and the second relative standard deviation; Based on the difference and the current remaining update cycle, a second rate of change of the relative standard deviation is determined.

6. The method according to claim 2, characterized in that, The method further includes: After the end of the total update cycle of the regularization coefficient, if the second relative standard deviation is greater than or equal to the target relative standard deviation, the current regularization coefficient will be adjusted to the minimum value. If the second relative standard deviation is less than the target relative standard deviation, the current regularization coefficient is adjusted to the first value.

7. A model pruning device, characterized in that, The device includes: The data acquisition unit is used to acquire training data. The model acquisition unit is used to acquire the model to be pruned with the training parameters set. The training parameters include the target proportion of zero channel score, the initial value of the regularization coefficient, the total update cycle of the regularization coefficient, and the update cycle of each update. An adjustment unit is used to dynamically adjust the regularization coefficient until the relative standard deviation of all channel scores corresponding to the pruned model after adjusting the regularization coefficient meets the target relative standard deviation, thus obtaining the target pruned model. The target relative standard deviation is the relative standard deviation of all channel scores corresponding to the target ratio. Wherein, if the model to be pruned is a model for face liveness detection, the training data is a set of face images; if the model to be pruned is a model for action detection, the training data is a set of images or videos including multiple actions. The adjustment unit dynamically adjusts the regularization coefficient based on the training data and the training parameters. The adjustment includes: during the process of dynamically adjusting the regularization coefficient based on the training data and the training parameters, before the start of the current update cycle, obtaining a first relative standard deviation, which is the relative standard deviation of all channel scores corresponding to the model to be pruned at the end of the previous update cycle; at the end of the current update cycle, obtaining a second relative standard deviation, which is the relative standard deviation of all channel scores corresponding to the model to be pruned at the end of the current update cycle; and based on the first relative standard deviation and the second relative standard deviation, dynamically adjusting the current regularization coefficient at the end of the current update cycle. The pruning unit is used to prune the target pruning model to obtain a target model. The target model is used to deploy in an embedded device so that the embedded device runs the target model to perform face liveness recognition or motion detection tasks.

8. An electronic device, characterized in that, It includes one or more processors; one or more programs are stored in memory and configured to be executed by the one or more processors according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code, wherein the program code, when executed by a processor, performs the method according to any one of claims 1-6.

Images