Model pruning method and device, electronic equipment and program product

Abstract

The invention provides a model pruning method and device, electronic equipment and a program product, and the method comprises the steps: carrying out the iterative sparse processing of channels in each network layer in a pre-trained target model, and obtaining a target sparse model corresponding to the target model until the difference value between the model precision of the target model and the model precision of the sparse model corresponding to the target model in the current iteration is greater than or equal to a preset threshold value, and obtaining a sparse processing result of channels in each network layer in the target sparse model. And determining to-be-pruned channels in each network layer in the target model, and performing physical pruning on each to-be-pruned channel to remove the to-be-pruned channels in each network layer in the target model to obtain a pruned model of the target model. According to the model pruning method, the model precision of the target model after pruning can be effectively ensured on the premise of reducing the resource occupation amount of the target model after pruning.

Description

Technical Field

[0001] The present application relates to the field of model processing technology, and in particular to a model pruning method, device, electronic equipment and program product. Background Art

[0002] With the rapid development of intelligent technology and the increasing demand for deep learning models on the edge, deep learning models have been widely used in various fields. Among them, the significant increase in the accuracy of deep learning models in solving various problems is usually achieved through the high parameterization of deep learning models, which has caused the scale of deep learning models to continue to increase. However, the huge model scale requires more memory and computing resources for model deployment, which greatly limits the application of deep learning models. Therefore, pruning deep learning models is an important condition to ensure the effective application of deep learning models in resource-constrained environments.

[0003] At present, existing model pruning methods usually cause irreversible losses in the deep learning model during the pruning process, thereby failing to guarantee the model accuracy of the pruned deep learning model. Summary of the invention

[0004] Based on the above needs, the present application proposes a model pruning method, device, electronic device and program product to solve the problem in the prior art that the model accuracy of the deep learning model after pruning cannot be guaranteed.

[0005] In order to achieve the above technical objectives, this application specifically proposes the following technical solutions: The first aspect of the present application provides a model pruning method, comprising: Performing iterative sparse processing on the channels in each network layer of the pre-trained target model until the difference between the model accuracy of the target model and the model accuracy of the sparse model corresponding to the target model in the current iteration is greater than or equal to a predetermined threshold, so as to obtain a target sparse model corresponding to the target model; Based on the sparse processing results of the channels in each network layer in the target sparse model, the channels to be pruned in each network layer in the target model are determined, and each of the channels to be pruned is physically pruned to obtain a pruned model of the target model.

[0006] The second aspect of the present application provides a model pruning device, comprising: A first data processing module is used to perform iterative sparse processing on the channels in each network layer in the pre-trained target model until the difference between the model accuracy of the target model and the model accuracy of the sparse model corresponding to the target model in the current iteration is greater than or equal to a predetermined threshold, so as to obtain a target sparse model corresponding to the target model; The second data processing module is used to determine the channels to be pruned in each network layer in the target model based on the sparse processing results of the channels in each network layer in the target sparse model, and physically prune each of the channels to be pruned to obtain a pruned model of the target model.

[0007] A third aspect of the present application provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the model pruning method as described above is implemented.

[0008] A fourth aspect of the present application provides an electronic device, comprising: Memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the above-mentioned model pruning method by running the program in the memory.

[0009] In a fifth aspect, the present application proposes a computer program product, comprising computer program instructions, which, when executed by a processor, cause the processor to execute the model pruning method as described above.

[0010] The model pruning method proposed in the present application performs iterative sparse processing on the channels in each network layer in the pre-trained target model until the difference between the model accuracy of the target model and the model accuracy of the sparse model corresponding to the target model in the current iteration is greater than or equal to a predetermined threshold value, so as to obtain a target sparse model corresponding to the target model, thereby gradually performing sparse processing on the channels in each network layer in the target model through iterative sparse processing, thereby effectively avoiding irreversible loss of model accuracy caused by excessive sparseness in the process of sparse processing of the target model; at the same time, since the model structure will not be changed in the process of iterative sparse processing of the target model, the target model is iteratively sparse. The precision loss produced during the processing is relatively easy to recover, thereby effectively ensuring the model accuracy of the target sparse model; after obtaining the target sparse model corresponding to the target model, based on the sparse processing results of the channels in each network layer in the target sparse model, determine the channels to be pruned in each network layer in the target model, and physically prune each channel to be pruned to remove the channels to be pruned in each network layer in the target model to obtain a pruned model of the target model, thereby effectively avoiding the precision loss caused by model pruning that cannot be recovered, resulting in a significant reduction in the model accuracy of the target model after pruning, and then effectively ensuring the model accuracy of the target model after pruning while reducing the resource usage of the target model after pruning. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0012] Figure 1 A flowchart of a model pruning method provided in an embodiment of the present application.

[0013] Figure 2 A schematic diagram of the structure of a model pruning device provided in an embodiment of the present application.

[0014] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0015] The technical solution of the embodiment of the present application is suitable for application scenarios where model pruning is performed on deep learning models such as visual models. The technical solution of the embodiment of the present application can effectively avoid the situation where the accuracy loss caused by model pruning cannot be recovered, resulting in a significant reduction in the model accuracy after model pruning. Furthermore, the model accuracy of the pruned model can be effectively guaranteed while reducing the resource usage of the pruned model.

[0016] With the rapid development of intelligent technology and the increasing demand for deep learning models on the edge, deep learning models have been widely used in various fields. Among them, the significant increase in the accuracy of deep learning models in solving various problems is usually achieved through the high parameterization of deep learning models, which has caused the scale of deep learning models to continue to increase. However, the huge model scale requires more memory and computing resources for model deployment. Among them, the deployment of deep learning models on the edge is mainly restricted by the following aspects: 1) Model size. The powerful representation ability of deep learning models comes from their millions of trainable parameters. The parameters and network structure information need to be stored on disk and loaded into memory during the inference of the deep learning model. For example, storing a typical CNN (Convolutional Neural Network) on ImageNet (visual database) consumes more than 300MB of space, which is a huge resource burden for embedded devices. 2) Runtime memory. During the inference of a deep learning model, the intermediate activation / response of the deep learning model may take up more memory space than storing the model parameters, which is unaffordable for applications with low computing power. 3) The number of computational operations, such as convolution operations, is large on high-resolution images, and a large deep learning model may take several minutes to process an image on a mobile device. This shows that the huge model size has greatly limited the application of deep learning models.

[0017] Among them, model pruning can remove unimportant parameters in each network layer of the deep learning model, so that the pruned deep learning model can reduce the number of parameters and the amount of calculation, while trying to ensure that the accuracy of the pruned deep learning model is not affected. Therefore, pruning the deep learning model is an important condition to ensure that the deep learning model can be effectively applied in a resource-constrained environment.

[0018] At present, existing model pruning methods usually physically prune deep learning models according to a set fixed pruning rate. As a result, the deep learning model will suffer irreversible losses during the pruning process, and the model accuracy of the pruned deep learning model cannot be guaranteed.

[0019] Based on the above technical status, the present application proposes a new model pruning scheme, which iteratively sparses the channels in each network layer in a pre-trained target model until the difference between the model accuracy of the target model and the model accuracy of the sparse model corresponding to the target model in the current iteration is greater than or equal to a predetermined threshold, so as to obtain a target sparse model corresponding to the target model, thereby gradually performing sparse processing on the channels in each network layer in the target model through iterative sparse processing, thereby effectively avoiding irreversible loss of model accuracy caused by excessive sparseness in the process of sparse processing of the target model; at the same time, since the model structure will not be changed in the process of iterative sparse processing of the target model, The precision loss of the target model produced during the iterative sparse processing is relatively easy to recover, thereby effectively ensuring the model accuracy of the target sparse model; after obtaining the target sparse model corresponding to the target model, based on the sparse processing results of the channels in each network layer in the target sparse model, determine the channels to be pruned in each network layer in the target model, and physically prune each channel to be pruned to remove the channels to be pruned in each network layer in the target model to obtain a pruned model of the target model, thereby effectively avoiding the inability to recover the precision loss caused by model pruning, resulting in a greatly reduced model accuracy of the target model after pruning, and then effectively ensuring the model accuracy of the target model after pruning while reducing the resource occupation of the target model after pruning.

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] See also Figure 1 The embodiment of the present application first proposes a model pruning method, which can be executed by a processor and includes: S101. Perform iterative sparse processing on the channels in each network layer in a pre-trained target model until the difference between the model accuracy of the target model and the model accuracy of the sparse model corresponding to the target model in the current iteration is greater than or equal to a predetermined threshold, so as to obtain a target sparse model corresponding to the target model.

[0022] Specifically, the target model may be a deep learning model to be pruned, such as a convolutional neural network model, a recurrent neural network model, a self-attention mechanism model, etc. The target model may be a pre-trained model, that is, the target model is a model whose model accuracy meets the predetermined requirements.

[0023] The target model may include multiple network layers connected in sequence, such as an input layer, a convolution layer, a pooling layer, a fully connected layer, an activation function layer, a normalization layer, a recursive layer, an attention mechanism layer, a decoding layer, etc. For any one of the multiple network layers, the network layer may include multiple channels, and different channels may be used to capture and / or transmit different feature information.

[0024] During implementation, the channels in each network layer of the pre-trained target model can be iteratively sparsely processed based on the model accuracy. For example, the channels in each network layer of the target model can be iteratively sparsely processed until the iteration end condition is met, the iteration is stopped, and the sparse model corresponding to the target model in the last iteration is used as the target sparse model.

[0025] In any iteration, whether the iteration end condition is satisfied can be determined based on the model accuracy of the sparse model corresponding to the target model in the iteration. For example, when the model accuracy of the sparse model corresponding to the target model in the iteration is less than or equal to a predetermined accuracy, it can be determined that the iteration end condition is satisfied. It can also be determined that the iteration end condition is satisfied when the accuracy loss of the target model is greater than or equal to a predetermined threshold. The accuracy loss of the target model can be the difference between the model accuracy of the target model and the model accuracy of the sparse model corresponding to the target model in the iteration.

[0026] In this iteration, the sparse model corresponding to the target model is the model obtained by sparsely processing the current model in this iteration. The current model can be the model to be sparsely processed in this iteration. For example, when this iteration is the first iteration, the current model is the target model itself. When this iteration is any iteration after the first iteration, the current model is the training result of the sparse model corresponding to the target model in the previous iteration.

[0027] The model accuracy of the sparse model corresponding to the target model can be obtained by training the sparse model corresponding to the target model. For example, for any iteration, after sparse processing is performed on the channels in each network layer in the current model, the sparse model corresponding to the target model in this iteration can be trained, and the model accuracy of the sparse model at the end of the training is used as the model accuracy of the sparse model corresponding to the target model in this iteration. Therefore, by iteratively sparsely and training the target model, the target model can be sparsely processed to the greatest extent while ensuring the model accuracy of the target model after sparse processing.

[0028] Therefore, by iteratively sparse processing the channels in each network layer in the target model and real-time monitoring of the model accuracy during the iterative sparse processing, it is possible to effectively avoid irrecoverable loss of model accuracy caused by excessive sparseness during the sparse processing of the target model, thereby effectively ensuring the model accuracy of the target sparse model obtained by iterative sparse processing.

[0029] In addition, for any iteration, in the process of sparse processing of channels in each network layer in the current model, simulated pruning can be performed on the channels in each network layer in the current model, that is, in the process of sparse processing of channels in each network layer in the current model, the model structure of the current model remains unchanged, for example, the channels that need to be sparsely processed can be set to zero. Therefore, in the process of iterative sparse processing of the target model, the precision loss generated is relatively easy to recover, so that the model precision of the target sparse model corresponding to the target model can be effectively guaranteed.

[0030] S102: Based on the sparse processing results of the channels in each network layer in the target sparse model, determine the channels to be pruned in each network layer in the target model, and physically prune each of the channels to be pruned to obtain a pruned model of the target model.

[0031] Specifically, the sparsely processed channels in each network layer of the target sparse model can be used as channels to be pruned, and the channels to be pruned in each network layer of the target network model can be physically pruned, so as to remove the corresponding channels from each network layer of the target model through physical pruning to obtain a pruned model of the target model, thereby effectively reducing the amount of storage resources and computing resources occupied by the pruned model of the target model. The pruned model of the target model can be used for image processing.

[0032] At the same time, the process of physical pruning of the target model is performed based on the sparse processing results of the channels in each network layer in the target sparse model corresponding to the target model. Therefore, when the model accuracy of the target sparse model is high, the model accuracy of the pruned model obtained by physical pruning of the target model can be effectively guaranteed.

[0033] In addition, the physical pruning process does not need to rely on special software and / or hardware to obtain memory gains and reduce computational complexity, so that model pruning can be flexibly and efficiently performed on different processors, effectively improving the applicability of the model pruning method of the present application.

[0034] From the above introduction, it can be seen that the model pruning method proposed in the embodiment of the present application performs iterative sparse processing on the channels in each network layer in the pre-trained target model until the difference between the model accuracy of the target model and the model accuracy of the sparse model corresponding to the target model in the current iteration is greater than or equal to a predetermined threshold value, so as to obtain a target sparse model corresponding to the target model, thereby, through iterative sparse processing, the channels in each network layer in the target model can be gradually sparsely processed, thereby effectively avoiding the irreversible loss of model accuracy caused by excessive sparseness in the process of sparse processing of the target model; at the same time, since the model structure will not be changed in the process of iterative sparse processing of the target model, the accuracy loss of the target model in the iterative sparse processing process is reduced. The loss is relatively easy to recover, thereby effectively ensuring the model accuracy of the target sparse model; after obtaining the target sparse model corresponding to the target model, based on the sparse processing results of the channels in each network layer in the target sparse model, determine the channels to be pruned in each network layer in the target model, and physically prune each channel to be pruned to remove the channels to be pruned in each network layer in the target model to obtain the pruned model of the target model, thereby effectively avoiding the loss of accuracy caused by model pruning that cannot be recovered, resulting in a significant reduction in the model accuracy of the pruned target model, and then effectively ensuring the model accuracy of the pruned target model under the premise of reducing the resource occupation of the pruned target model, and realizing the effective trimming of redundant parameters and redundant calculation amount under the premise of almost lossless model accuracy.

[0035] In some embodiments, iterative sparse processing is performed on the channels in each network layer in the pre-trained target model until the difference between the model accuracy of the target model and the model accuracy of the sparse model corresponding to the target model in the current iteration is greater than or equal to a predetermined threshold, so as to obtain a target sparse model corresponding to the target model, including: In the current iteration, sparse processing is performed on the channels in each network layer in the current model based on the target data to obtain a sparse model corresponding to the target model in the current iteration; the current model is the training result of the sparse model corresponding to the target model or the target model in the previous iteration; Training the sparse model corresponding to the target model in the current iteration to obtain the model accuracy of the sparse model corresponding to the target model in the current iteration; If the difference between the model accuracy of the target model and the model accuracy of the sparse model corresponding to the target model in the current iteration is greater than or equal to a predetermined threshold, the sparse model corresponding to the target model in the current iteration is used as the target sparse model; If the difference between the model accuracy of the target model and the model accuracy of the sparse model corresponding to the target model in the current iteration is less than the predetermined threshold, the next iteration is performed.

[0036] Specifically, the current iteration may be any iteration in the process of iterative sparse processing of the channels in each network layer in the target model. The current model may be the model to be sparsely processed in the current iteration. For example, when the current iteration is the first iteration, the current model is the target model itself. When the current iteration is any iteration after the first iteration, the current model is the training result of the sparse model corresponding to the target model in the previous iteration. The sparse model corresponding to the target model in the previous iteration is the model obtained after the sparse processing is performed in the previous iteration.

[0037] The target data may include reference factors for determining the channels to be sparse in each network layer in the current model. For example, the target data may include the sparse factors corresponding to the channels in each network layer in the current model. The sparse factors may characterize the importance of the corresponding channels. For any channel in each network layer in the current model, the probability of the channel being sparsely processed may be negatively correlated with the importance of the channel, that is, the higher the importance of the channel, the lower the probability of the channel being sparsely processed. Conversely, the lower the importance of the channel, the higher the probability of the channel being sparsely processed. The sparse factor may be a predetermined value. For example, the sparse factor of each channel may be determined based on the importance of the channels in each network layer in the target model, and the sparse factors of each channel remain unchanged during the iterative sparse processing of the channels in each network layer in the target model. In addition, the sparse factors of each channel that has not been sparsely processed in the sparse model determined in the iteration may be dynamically updated in each iteration.

[0038] At the same time, the target data may also include a target sparsity rate in the current iteration, and the target sparsity rate in each iteration may be less than a predetermined sparsity rate, which may be a reference sparsity rate of the target sparse model. Thus, in the process of iterative sparse processing of the channels in each network layer in the target model, the model accuracy may be used as a constraint to gradually perform sparse processing on the channels in each network layer in the target model, thereby effectively avoiding the loss of model accuracy that is difficult to recover due to excessive sparseness in the process of sparse processing of the channels in each network layer in the target model.

[0039] It is understandable that the target sparsity rate in each iteration can be the same or different, and can be set according to actual needs. For example, the target sparsity rate in the current iteration can be determined based on the current rate of change of the model accuracy. The target sparsity rate can be negatively correlated with the current rate of change of the model accuracy, that is, the larger the current rate of change of the model accuracy, the smaller the target sparsity rate, and the smaller the current rate of change of the model accuracy, the larger the target sparsity rate, so that the sparse efficiency of the target model can be effectively improved under the premise of ensuring the model accuracy of the target model after sparse processing. In addition, as the number of iterations increases, the target sparsity rate can be gradually reduced according to a predetermined step size, so that the sparse efficiency of the target model can be effectively improved under the premise of ensuring the model accuracy of the target model after sparse processing.

[0040] In addition, the target data may also include associations between channels in each network layer in the current model, wherein channels with associations may be processed synchronously and sparsely.

[0041] Among them, in the current iteration, the channels to be sparsely processed in the current iteration can be determined based on the sparse factors corresponding to the channels in each network layer in the current model and the target sparse rate in the current iteration, and each channel to be sparsely processed is sparsely processed to obtain a sparse model corresponding to the target model in the current iteration. In addition, the channels to be sparsely processed in the current iteration can be determined based on the sparse factors corresponding to the channels in each network layer in the current model, the target sparse rate in the current iteration, and the correlation relationship between the channels in each network layer in the current model, and each channel to be sparsely processed is sparsely processed to obtain a sparse model corresponding to the target model in the current iteration.

[0042] During implementation, after obtaining the sparse model corresponding to the target model in the current iteration, the sparse model can be trained, and the model accuracy of the sparse model corresponding to the target model in the current iteration can be determined based on the training results. Thus, whether to stop the iteration can be determined based on the model accuracy of the sparse model.

[0043] Among them, if the difference between the model accuracy of the target model and the model accuracy of the sparse model corresponding to the target model in the current iteration is greater than or equal to a predetermined threshold, it indicates that the loss of model accuracy is large during the iterative sparse processing of the channels in each network layer in the target model. At this time, the iteration can be stopped, and the sparse model corresponding to the target model in the current iteration is used as the target sparse model corresponding to the target model, thereby effectively ensuring the model accuracy of the target sparse model.

[0044] If the difference between the model accuracy of the target model and the model accuracy of the sparse model corresponding to the target model in the current iteration is less than the predetermined threshold, it indicates that the loss of model accuracy in the iterative sparse processing of the channels in each network layer in the target model is acceptable. At this time, the next iteration can be continued until the difference between the model accuracy of the target model and the model accuracy of the sparse model corresponding to the target model is greater than or equal to the predetermined threshold. This allows the channels in each network layer in the target model to be sparsely processed to the greatest extent possible while ensuring that the loss of model accuracy of the target model after sparse processing is controllable. Furthermore, in the process of physically pruning the target model according to the target sparse model corresponding to the target model, the resource usage of the target model after pruning can be effectively reduced while ensuring the model accuracy of the pruned target model.

[0045] In some embodiments, sparse processing is performed on the channels in each network layer in the current model based on the target data to obtain a sparse model corresponding to the target model in the current iteration, including: Based on the target data, sparse processing is performed on the to-be-sparsed channels corresponding to each network layer in the current model in turn to obtain a sparse model corresponding to the target model in the current iteration; The to-be-sparse channels corresponding to the network layer include the target sparse channels in the network layer and the channels associated with the target sparse channels in each network layer of the current model.

[0046] Specifically, in the current iteration, the channels to be sparse corresponding to each network layer in the current model may be sparsely processed in turn based on the target data. For example, the sparse processing step may be repeated N times according to the connection relationship of each network layer in the current model, where N is the number of network layers in the current model, and each time the sparse processing step is executed, the channels to be sparse corresponding to the corresponding network layer in the current model may be sparsely processed, so that after repeating the sparse processing step N times, a sparse model corresponding to the target model in the current iteration is obtained.

[0047] In implementation, for any network layer in the current model, the to-be-sparse channels corresponding to the network layer may include the target sparse channels in the network layer, and may also include channels associated with each target sparse channel in each network layer of the current model.

[0048] Among them, the target sparse channel in the network layer may include the channels in the network layer that have not been sparsely processed and meet the sparse processing conditions in the current iteration. For example, for any channel in the network layer that has not been sparsely processed, it can be determined whether the channel meets the sparse processing conditions based on the target sparsity rate in the current iteration and the sorting result of the sparse factor corresponding to the channel.

[0049] For any target sparse channel in the network layer, the channels associated with the target sparse channel may include channels that are associated with the target sparse channel, that is, channels that need to be sparsely processed simultaneously with the target sparse channel.

[0050] Among them, the channels associated with the target sparse channel may include channels that have an intra-layer association relationship with the target sparse channel, that is, channels in the network layer that are associated with the target sparse channel. For example, if the input channel and the output channel in the network layer are interrelated, such as the network layer is a BN (Batch Normalization) layer, a ReLU (Rectified Linear Unit) layer, etc., then when the input channel in the network layer is sparse or pruned, the output channel in the network layer connected to the input channel also needs to be sparse or pruned, and when the output channel in the network layer is sparse or pruned, the input channel in the network layer connected to the output channel also needs to be pruned.

[0051] At the same time, the channels associated with the target sparse channel may also include channels that have an inter-layer association relationship with the target sparse channel, that is, channels associated with the target sparse channel in other network layers outside the network layer. For example, if the input channel and output channel in the network layer are independent of each other, such as the network layer is a Conv (convolution) layer, then when the output channel in the previous network layer of the network layer is sparse or pruned, the input channel connected to the output channel in the network layer also needs to be sparse or pruned, and when the input channel in the next network layer of the network layer is sparse or pruned, the output channel connected to the input channel in the network layer also needs to be sparse or pruned.

[0052] It can be understood that the channels associated with the target sparse channel may include channels directly associated with the target sparse channel and channels indirectly associated with the target sparse channel. The channels directly associated with the target sparse channel may include channels connected to and associated with the target sparse channel. The channels indirectly associated with the target sparse channel may include channels associated with channels directly associated with the target sparse channel.

[0053] For example, assuming that the current model includes a Conv1 layer, a BN layer, a ReLU layer, and a Conv2 layer connected in sequence, in the process of determining the channel to be sparse corresponding to the BN layer, if the output channel C1 with a dimension of 2C in the BN layer is determined to be the target sparse channel, the channels associated with the target sparse channel may include the input channel C2 associated with the output channel C1 in the BN layer, the output channel C3 associated with the input channel C2 in the Conv1 layer, and the input channel C4 associated with the output channel C1 in the Conv2 layer. During implementation, the target sparse channel and all channels associated with the target sparse channel can be formed into an association group, and each channel in the association group can be used as the channel to be sparse corresponding to the network layer to achieve synchronous sparse processing of each channel in the association group. Therefore, in the process of sparse processing of the channels in each network layer in the current model, the consistent sparsity of the channels associated between the network layers can be effectively taken into account, so that the accuracy loss caused by the model sparse process is easier to recover, thereby further improving the model accuracy of the target sparse model corresponding to the target model, and then in the process of model pruning based on the target sparse model, the accuracy loss caused by model pruning can be greatly reduced.

[0054] In some embodiments, the target data includes the sparse factors corresponding to each channel in each network layer in the current model, the association relationship between the channels in each network layer, and the target sparsity rate in the current iteration; the association relationship between the channels in each network layer includes intra-layer association and inter-layer association; The process of determining the to-be-sparse channels corresponding to each network layer in the current model includes: Based on the target sparsity rate in the current iteration and the sparsity factors corresponding to each channel in the current network layer, determining the target sparse channel from the channels in the current network layer that have not been sparsely processed; Based on the association relationship between the channels in each of the network layers, channels associated with the target sparse channels are determined from the channels of each of the network layers, and each of the target sparse channels and channels associated with each of the target sparse channels are used as the channels to be sparse corresponding to the current network layer.

[0055] Specifically, the target data may include the sparse factors corresponding to each channel in each network layer in the current model, the target sparsity rate in the current iteration, and the association relationship between the channels in each network layer of the current model. The association relationship between the channels in each network layer may include intra-layer association and inter-layer association.

[0056] In implementation, the target sparse channels can be determined from the channels that have not been sparsely processed in the current network layer based on the target sparse rate in the current iteration and the sparse factors corresponding to each channel in the current network layer. For example, the sparse factors corresponding to the channels that have not been sparsely processed in the current network layer can be sorted in order from large to small or from small to large, and the target number of target sparse channels can be determined based on the target sparse rate in the current iteration, so as to determine the target number of target sparse channels from the channels that have not been sparsely processed in the current network layer according to the sorting result.

[0057] After determining the target sparse channels in the current network layer, the channels associated with each target sparse channel can be determined from the channels of each network layer based on the association relationship between the channels in each network layer, and each target sparse channel and the channels associated with each target sparse channel are used as the channels to be sparse corresponding to the current network layer, so as to perform sparse processing on the channels to be sparse corresponding to the current network layer.

[0058] Among them, in the process of determining the target number of target sparse channels based on the target sparsity rate in the current iteration, the total number of required sparse channels of the current network layer in the current iteration can be determined based on the product of the number of channels that have not been sparsely processed in the current network layer and the target sparsity rate in the current iteration, and the target number of target sparse channels can be determined based on the difference between the total number of required sparse channels and the number of sparsely processed channels in the current network layer in the current iteration. It can be understood that the channels that have been sparsely processed in the current network layer in the current iteration may include: the channels that have been sparsely processed in the current network layer during the process of sparsely processing the channels to be sparse corresponding to the network layers before the current network layer.

[0059] Optionally, the target number of target sparse channels can be smaller than the difference, thereby effectively avoiding the situation in which the actual sparse rate of the current network layer is higher than the target sparse rate in the current iteration during the sparse processing of the channels to be sparse corresponding to the current network layer and the channels to be sparse corresponding to each network layer after the current network layer, causing the current network layer to be over-sparse. This can effectively avoid the loss of accuracy that is difficult to recover during the model sparse process.

[0060] For example, when the channels to be sparsed corresponding to the current network layer include channels that have an intra-layer association relationship with the target sparse channels in the current network layer, in the process of sparse processing the channels to be sparsed corresponding to the current network layer, the channels associated with the target sparse channels in the current network layer will be sparsely processed at the same time. At this time, if the target number of the target sparse channels is equal to the difference, the actual sparse rate of the current network layer will be higher than the target sparse rate in the current iteration. In addition, when the channels to be sparsed corresponding to the network layer after the current network layer include channels in the current network layer, in the process of sparse processing the channels to be sparse corresponding to the network layer after the current network layer, the channels in the current network layer will be sparsely processed. At this time, if the target number of the target sparse channels is equal to the difference, the actual sparse rate of the current network layer will be higher than the target sparse rate in the current iteration.

[0061] Therefore, through the method of the embodiment of the present application, in the process of model sparsification, the consistent sparsity of the channels associated with each network layer can be effectively taken into account, so that the precision loss caused by the model sparsification process is easier to recover, thereby further improving the model accuracy of the target sparse model corresponding to the target model, and then in the process of model pruning based on the target sparse model, the precision loss caused by model pruning can be greatly reduced.

[0062] In some embodiments, training the sparse model corresponding to the target model in the current iteration to obtain the model accuracy of the sparse model corresponding to the target model in the current iteration includes: Based on the target loss function, the sparse model corresponding to the target model is trained, and the model accuracy of the sparse model corresponding to the target model at the end of the training is used as the model accuracy of the sparse model corresponding to the target model in the current iteration; Among them, the target loss function includes a first loss function, and the first loss function is used to determine the first sparsity loss of the sparse model corresponding to the target model based on the association relationship between the channels in each of the network layers in the sparse model corresponding to the target model.

[0063] Specifically, in the process of iterative sparse processing of channels in each network layer in the target model, for any iteration, in the process of determining the model accuracy of the sparse model corresponding to the target model in this iteration, the sparse model corresponding to the target model in this iteration can be trained based on the target loss function, and the model accuracy of the sparse model at the end of training is used as the model accuracy of the sparse model corresponding to the target model in this iteration.

[0064] Among them, the target loss function can be used to determine the model loss of the sparse model corresponding to the target model. Therefore, the model parameters of the sparse model corresponding to the target model in this iteration can be adjusted in the direction of reducing the model loss.

[0065] In implementation, the target loss function may include a first loss function, which may determine the first sparsity loss of the sparse model corresponding to the target model based on the correlation relationship between the channels in each network layer in the sparse model corresponding to the target model, so as to constrain the sparsity of the associated channels in each network layer through the first sparsity loss. Thus, in the process of training the sparse model corresponding to the target model through the target loss function, the sparse model can effectively learn the correlation between the channels in each network layer, and then in the next iteration, the channels to be sparse corresponding to each network layer can be comprehensively and effectively determined.

[0066] For example, the first loss function Loss1 can be expressed as formula (1): (1) In the formula, is the first sparse penalty factor; L is the total number of network layers of the sparse model corresponding to the target model, For the The set of channels associated with each network layer, Each element in represents the network layer of the sparse model corresponding to the target model, and The channels associated with each channel in the network layer; for The elements in for The index is Elements of express of Order norm, The value can be set according to actual needs, for example, it can be set to 1 or 2.

[0067] In some embodiments, the target loss function also includes a second loss function, which is used to determine a second sparsity loss of the sparse model corresponding to the target model based on the sparse factors corresponding to the channels in each of the network layers in the sparse model corresponding to the target model.

[0068] Specifically, the target loss function may also include a second loss function, which may determine the second sparsity loss of the sparse model corresponding to the target model based on the sparsity factor corresponding to the channel in each network layer in the sparse model corresponding to the target model, so as to constrain the sparsity of each channel in each network layer through the second sparsity loss. Thus, in the process of training the sparse model corresponding to the target model through the target loss function, the sparse model can effectively learn the sparsity of each channel in each network layer, and then in the next iteration, the target sparse channel in each network layer can be accurately determined.

[0069] For example, the second loss function Loss2 can be expressed as formula (2): (2) In the formula, is the second sparse penalty factor; For the The set of sparse factors corresponding to the network layers, Each element in represents the The sparse factor corresponding to each channel in a network layer, wherein the sparse factor corresponding to each channel can change as the sparse model learns the sparsity of the channels in each network layer during the training process.

[0070] In addition, the target loss function may also include a third loss function, which may be used to determine the prediction loss of the sparse model corresponding to the target model based on the predicted value output by the sparse model corresponding to the target model and the label corresponding to the predicted value. Thus, in the process of training the sparse model corresponding to the target model through the target loss function, the model accuracy of the sparse model corresponding to the target model can be effectively improved.

[0071] For example, the third loss function Loss3 can be expressed as formula (3): (3) In the formula, x is the sample data in the training set, and y is the label corresponding to the sample data; The predicted value output by the sparse model corresponding to the target model based on the sample data x. is a loss function model. For example, the loss function model may be a cross entropy loss function model, a contrast loss function model, or the like.

[0072] In implementation, the first sparsity loss, the second sparsity loss and the prediction loss of the sparse model corresponding to the target model can be summed, and the summation result can be used as the model loss of the sparse model corresponding to the target model, so as to train the sparse model corresponding to the target model according to the model loss, so that in the process of training the sparse model corresponding to the target model through the target loss function, the prediction accuracy of the sparse model corresponding to the target model, the sparsity of the associated channels in each network layer in the model, and the sparsity of the channels in each network layer in the model can be effectively constrained, so that the trained sparse model can effectively learn the correlation between the channels in each network layer and the sparsity of the channels in each network layer while ensuring the model accuracy, and then in the next iteration, the channels to be sparse corresponding to each network layer can be comprehensively and accurately determined. Therefore, in the process of physically pruning the target model according to the final target sparse model, the target model can be pruned comprehensively and effectively under the premise of ensuring the model accuracy of the pruned target model, so as to minimize the resource occupation of the pruned target model.

[0073] In some embodiments, the process of determining the sparse factor corresponding to each channel in each network layer in the current model includes: Obtaining parameter values ​​of target parameters of each channel in the network layer in the current model; the parameter values ​​of the target parameters are used to characterize the importance of the corresponding channels; Based on the parameter value of the target parameter of each channel in the network layer, a sparse factor corresponding to each channel in the network layer is determined.

[0074] Specifically, for any channel in each network layer in the current model, the target parameter of the channel can be used to characterize the importance of the channel. For example, the target parameter of the channel may include the scaling factor of the channel. The larger the scaling factor, the higher the importance represented. Conversely, the smaller the scaling factor, the lower the importance represented. The target parameter of the channel may also include the weight of the channel. The larger the weight, the higher the importance represented. Conversely, the smaller the weight, the lower the importance represented. In addition, the target parameter of the channel may also include the statistical characteristics of the features generated by the channel, such as mean, variance, maximum value, minimum value, etc.

[0075] In implementation, in any iteration, the parameter values ​​of the target parameters of each channel in each network layer in the current model can be obtained. For any network layer, the sparse factors corresponding to each channel in the network layer can be determined based on the parameter values ​​of the target parameters of each channel in the network layer, that is, the sparse factors corresponding to each channel are continuously learned and updated during the training process of the current model. Therefore, in the process of iterative sparse processing of the channels in each network layer in the target model, by updating the sparse factors of the channels in each network layer in each iteration, the target model can be fully sparse while ensuring the model accuracy of the target sparse model. Then, in the process of physically pruning the target model according to the final target sparse model, the resource usage of the pruned target model can be minimized while ensuring the model accuracy of the pruned target model.

[0076] Among them, for any channel in the network layer, the sparse factor corresponding to the channel can be positively correlated with the importance represented by the target parameter of the channel, that is, the higher the importance represented by the target parameter of the channel, the larger the sparse factor corresponding to the channel, and the smaller the probability that the channel is sparsely processed in the model sparsification process. On the contrary, the lower the importance represented by the target parameter of the channel, the smaller the sparse factor corresponding to the channel, and the larger the probability that the channel is sparsely processed in the model sparsification process, thereby being able to quickly and accurately determine the sparse factors corresponding to each channel in each network layer in the current model.

[0077] In some embodiments, the process of determining the target sparsity rate in the current iteration includes: Based on the ratio of the predetermined sparsity rate of the target model to the predetermined number of iterations, a target sparsity rate in each iteration is determined.

[0078] Specifically, the predetermined sparsity rate of the target model may be a reference sparsity rate of the target sparse model, that is, a reference value of the overall sparsity rate in the process of iteratively sparsifying the target model to obtain the target sparse model. In implementation, the predetermined sparsity rate may be determined in response to a user input operation on the reference sparsity rate, and the predetermined sparsity rate may also be pre-configured in the program, and may be specifically set according to actual needs.

[0079] The predetermined number of iterations of the target model may be a reference number of iterations in the process of iterative sparse processing of the channels in each network layer in the target model. In implementation, the predetermined number of iterations may be determined by responding to a user input operation on the reference number of iterations, or may be pre-configured in a program, and may be specifically set according to actual needs.

[0080] It can be understood that, in the process of iteratively sparse processing of the channels in each network layer in the target model, the predetermined sparsity rate and the predetermined number of iterations may not be used as the basis for judging the iteration termination condition, that is, the predetermined sparsity rate and the predetermined number of iterations may only be used to determine the target sparsity rate in each iteration, so as to gradually perform sparse processing on the target model according to the target sparsity rate, so as to avoid one-time over-sparse processing of the target model, resulting in irrecoverable loss of model accuracy of the target model.

[0081] During implementation, the ratio of the predetermined sparsity rate of the target model to the predetermined number of iterations can be obtained, and the target sparsity rate in each iteration can be determined based on the ratio. For example, the target sparsity rate in each iteration can be the ratio, and the ratio can be used as a baseline value. As the number of iterations increases, the target sparsity rate can be gradually reduced according to a predetermined step size based on the ratio. The specific setting can be based on actual needs.

[0082] Corresponding to the above-mentioned model pruning method, the embodiment of the present application also provides a model pruning device, see Figure 2 As shown, the device comprises: The first data processing module 201 is used to iteratively sparsely process the channels in each network layer in the pre-trained target model until the difference between the model accuracy of the target model and the model accuracy of the sparse model corresponding to the target model in the current iteration is greater than or equal to a predetermined threshold, so as to obtain a target sparse model corresponding to the target model; The second data processing module 202 is used to determine the channels to be pruned in each network layer in the target model based on the sparse processing results of the channels in each network layer in the target sparse model, and physically prune each of the channels to be pruned to obtain a pruned model of the target model.

[0083] In a possible implementation manner, the first data processing module 201 is specifically configured to: In the current iteration, sparse processing is performed on the channels in each network layer in the current model based on the target data to obtain a sparse model corresponding to the target model in the current iteration; the current model is the training result of the sparse model corresponding to the target model or the target model in the previous iteration; Training the sparse model corresponding to the target model in the current iteration to obtain the model accuracy of the sparse model corresponding to the target model in the current iteration; If the difference between the model accuracy of the target model and the model accuracy of the sparse model corresponding to the target model in the current iteration is greater than or equal to a predetermined threshold, the sparse model corresponding to the target model in the current iteration is used as the target sparse model; If the difference between the model accuracy of the target model and the model accuracy of the sparse model corresponding to the target model in the current iteration is less than the predetermined threshold, the next iteration is performed.

[0084] In a possible implementation manner, the first data processing module 201 is specifically configured to: Based on the target data, sparse processing is performed on the to-be-sparsed channels corresponding to each network layer in the current model in turn to obtain a sparse model corresponding to the target model in the current iteration; The to-be-sparse channels corresponding to the network layer include the target sparse channels in the network layer and the channels associated with the target sparse channels in each network layer of the current model.

[0085] In a possible implementation, the target data includes the sparse factors corresponding to the channels in each network layer in the current model, the association relationship between the channels in each network layer, and the target sparse rate in the current iteration; the association relationship between the channels in each network layer includes intra-layer association and inter-layer association; The first data processing module 201 is also used for: Based on the target sparsity rate in the current iteration and the sparsity factors corresponding to each channel in the current network layer, determining the target sparse channel from the channels in the current network layer that have not been sparsely processed; Based on the association relationship between the channels in each of the network layers, channels associated with the target sparse channels are determined from the channels of each of the network layers, and each of the target sparse channels and channels associated with each of the target sparse channels are used as the channels to be sparse corresponding to the current network layer.

[0086] In a possible implementation manner, the first data processing module 201 is specifically configured to: Based on the target loss function, the sparse model corresponding to the target model is trained, and the model accuracy of the sparse model corresponding to the target model at the end of the training is used as the model accuracy of the sparse model corresponding to the target model in the current iteration; Among them, the target loss function includes a first loss function, and the first loss function is used to determine the first sparsity loss of the sparse model corresponding to the target model based on the association relationship between the channels in each of the network layers in the sparse model corresponding to the target model.

[0087] In one possible implementation, the target loss function also includes a second loss function, which is used to determine the second sparsity loss of the sparse model corresponding to the target model based on the sparse factors corresponding to the channels in each of the network layers in the sparse model corresponding to the target model.

[0088] In a possible implementation, a third data processing module is further included, and the third data processing module is used to: Obtaining parameter values ​​of target parameters of each channel in the network layer in the current model; the parameter values ​​of the target parameters are used to characterize the importance of the corresponding channels; Based on the parameter value of the target parameter of each channel in the network layer, a sparse factor corresponding to each channel in the network layer is determined.

[0089] In a possible implementation, a fourth data processing module is further included, and the fourth data processing module is used to: Based on the ratio of the predetermined sparsity rate of the target model to the predetermined number of iterations, a target sparsity rate in each iteration is determined.

[0090] The model pruning device provided in this embodiment belongs to the same application concept as the model pruning method provided in the above embodiments of this application, and can execute the model pruning method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the model pruning method. For technical details not fully described in this embodiment, please refer to the specific processing content of the model pruning method provided in the above embodiments of this application, and will not be repeated here.

[0091] The functions implemented by the above modules can be implemented by the same or different processors respectively, and the embodiments of the present application are not limited thereto.

[0092] It should be understood that each module in the above device can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the unit in the device can be implemented in the form of a hardware circuit, and the functions of some or all units can be realized by designing the hardware circuit. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of some or all of the above units. All units of the above device can be implemented in the form of a processor calling software, or in the form of a hardware circuit, or in part by a processor calling software, and the remaining part is implemented in the form of a hardware circuit.

[0093] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0094] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0095] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0096] Another embodiment of the present application also provides an electronic device, see Figure 3 As shown, the electronic device includes: Memory 200 and processor 210; The memory 200 is connected to the processor 210 and is used to store programs; The processor 210 is used to implement the model pruning method described in any of the above embodiments of this specification by running the program stored in the memory 200.

[0097] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0098] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus. A bus may include a pathway that transfers information between components of a computer system.

[0099] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0100] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0101] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0102] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0103] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0104] The communication interface 220 may include any device such as a transceiver to communicate with other devices or communication networks, such as Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc.

[0105] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of any model pruning method provided in the above embodiments of the present application.

[0106] The method in the present application can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions, and when the computer programs or instructions are loaded and executed on a computer, the process or function described in the present application is executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, a core network device, an OAM or other programmable device.

[0107] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0108] The computer program or instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0109] In addition, the embodiment of the present application may also be a storage medium on which a computer program is stored. The computer program is executed by a processor to execute the steps of the model pruning method described in any of the above embodiments of this specification, and specifically the following steps may be implemented: S101, performing iterative sparse processing on the channels in each network layer in the pre-trained target model until the difference between the model accuracy of the target model and the model accuracy of the sparse model corresponding to the target model in the current iteration is greater than or equal to a predetermined threshold, so as to obtain a target sparse model corresponding to the target model; S102: Based on the sparse processing results of the channels in each network layer in the target sparse model, determine the channels to be pruned in each network layer in the target model, and physically prune each of the channels to be pruned to obtain a pruned model of the target model.

[0110] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0111] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0112] The steps in the methods of the embodiments of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in the embodiments can be replaced or combined. The modules and submodules in the devices and terminals in the embodiments of the present application can be combined, divided and deleted according to actual needs.

[0113] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0114] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0115] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0116] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0117] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0118] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0119] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A model pruning method, characterized in that: include: Performing iterative sparse processing on the channels in each network layer of the pre-trained target model until the difference between the model accuracy of the target model and the model accuracy of the sparse model corresponding to the target model in the current iteration is greater than or equal to a predetermined threshold, so as to obtain a target sparse model corresponding to the target model; Based on the sparse processing results of the channels in each network layer in the target sparse model, the channels to be pruned in each network layer in the target model are determined, and each of the channels to be pruned is physically pruned to obtain a pruned model of the target model.

2. The method according to claim 1, characterized in that: Iterative sparse processing is performed on the channels in each network layer of the pre-trained target model until the difference between the model accuracy of the target model and the model accuracy of the sparse model corresponding to the target model in the current iteration is greater than or equal to a predetermined threshold, so as to obtain a target sparse model corresponding to the target model, including: In the current iteration, sparse processing is performed on the channels in each network layer in the current model based on the target data to obtain a sparse model corresponding to the target model in the current iteration; the current model is the training result of the sparse model corresponding to the target model or the target model in the previous iteration; Training the sparse model corresponding to the target model in the current iteration to obtain the model accuracy of the sparse model corresponding to the target model in the current iteration; If the difference between the model accuracy of the target model and the model accuracy of the sparse model corresponding to the target model in the current iteration is greater than or equal to a predetermined threshold, the sparse model corresponding to the target model in the current iteration is used as the target sparse model; If the difference between the model accuracy of the target model and the model accuracy of the sparse model corresponding to the target model in the current iteration is less than the predetermined threshold, the next iteration is performed.

3. The method according to claim 2, characterized in that Based on the target data, the channels in each network layer in the current model are sparsely processed to obtain a sparse model corresponding to the target model in the current iteration, including: Based on the target data, sparse processing is performed on the to-be-sparsed channels corresponding to each network layer in the current model in turn to obtain a sparse model corresponding to the target model in the current iteration; The to-be-sparse channels corresponding to the network layer include the target sparse channels in the network layer and the channels associated with the target sparse channels in each network layer of the current model.

4. The method according to claim 3, characterized in that The target data includes the sparse factors corresponding to each channel in each network layer in the current model, the correlation relationship between the channels in each network layer, and the target sparse rate in the current iteration; The association relationship of the channels in each of the network layers includes intra-layer association and inter-layer association; The process of determining the to-be-sparse channels corresponding to each network layer in the current model includes: Based on the target sparsity rate in the current iteration and the sparsity factors corresponding to each channel in the current network layer, determining the target sparse channel from the channels in the current network layer that have not been sparsely processed; Based on the association relationship between the channels in each of the network layers, channels associated with the target sparse channels are determined from the channels of each of the network layers, and each of the target sparse channels and channels associated with each of the target sparse channels are used as the channels to be sparse corresponding to the current network layer.

5. The method according to claim 4, characterized in that The sparse model corresponding to the target model in the current iteration is trained to obtain the model accuracy of the sparse model corresponding to the target model in the current iteration, including: Based on the target loss function, the sparse model corresponding to the target model is trained, and the model accuracy of the sparse model corresponding to the target model at the end of the training is used as the model accuracy of the sparse model corresponding to the target model in the current iteration; Among them, the target loss function includes a first loss function, and the first loss function is used to determine the first sparsity loss of the sparse model corresponding to the target model based on the association relationship between the channels in each of the network layers in the sparse model corresponding to the target model.

6. The method according to claim 5, characterized in that The target loss function also includes a second loss function, which is used to determine the second sparsity loss of the sparse model corresponding to the target model based on the sparse factors corresponding to the channels in each of the network layers in the sparse model corresponding to the target model.

7. The method according to any one of claims 4 to 6, characterized in that: The process of determining the sparse factors corresponding to each channel in each network layer in the current model includes: Obtaining parameter values ​​of target parameters of each channel in the network layer in the current model; the parameter values ​​of the target parameters are used to characterize the importance of the corresponding channels; Based on the parameter value of the target parameter of each channel in the network layer, a sparse factor corresponding to each channel in the network layer is determined.

8. The method according to any one of claims 4 to 6, characterized in that: The process of determining the target sparsity rate in the current iteration includes: Based on the ratio of the predetermined sparsity rate of the target model to the predetermined number of iterations, a target sparsity rate in each iteration is determined.

9. A model pruning device, characterized in that: include: A first data processing module is used to perform iterative sparse processing on the channels in each network layer in the pre-trained target model until the difference between the model accuracy of the target model and the model accuracy of the sparse model corresponding to the target model in the current iteration is greater than or equal to a predetermined threshold, so as to obtain a target sparse model corresponding to the target model; The second data processing module is used to determine the channels to be pruned in each network layer in the target model based on the sparse processing results of the channels in each network layer in the target sparse model, and physically prune each of the channels to be pruned to obtain a pruned model of the target model.

10. An electronic device, characterized in that: include: Memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the model pruning method according to any one of claims 1 to 8 by running the program in the memory.

11. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, enable the processor to perform the model pruning method according to any one of claims 1 to 8.

Images