Class adaptive model pruning method and device, electronic equipment and storage medium

Abstract

The application provides a class adaptive model pruning method and device, electronic equipment and a storage medium, and belongs to the technical field of model pruning. The method comprises the following steps: establishing an optimization model for each class respectively, determining a target pruning rate for each class based on the optimization model; performing sparse regularization training on an original convolutional neural network model to determine a scaling factor corresponding to each channel of each layer network in a backbone part; determining a target class attention coefficient of each layer network of the backbone part for each class based on a calibration data set for the original convolutional neural network model after sparse regularization training; determining an importance coefficient of each layer network of the backbone part for each class based on the scaling factor and the target class attention coefficient; and performing pruning processing on the original convolutional neural network model for each class based on the target pruning rate and the importance coefficient. The pruning model obtained by the method is convenient to deploy and has relatively high performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model pruning, in particular to a class adaptive model pruning method and device, electronic equipment and storage medium. BACKGROUND

[0002] In recent years, deep learning technology has developed rapidly and is widely used in many industries, such as natural language processing, computer vision, etc. However, with the continuous increase of model size, deploying these models on terminal devices faces great challenges: due to the limited computing resources of terminal devices, deploying large models may cause too long inference latency and too high inference energy consumption, which is particularly prominent in application scenarios with high requirements for terminal user experience. Therefore, model compression has become a popular research field, aiming to reduce the size of the model to a level that can be deployed on terminal devices while maintaining the performance of the model. One commonly used method is pruning, which aims to remove unimportant network structures to reduce the size of the model while preserving the inference accuracy of the model.

[0003] However, most pruning methods only focus on the inference accuracy of the model, ignoring some important factors when actually deploying on devices, including resource requirements, latency requirements, and class requirements, such as storage space and energy. For example, in emergency rescue scenarios, drones may be used to perceive disasters and provide information to rescue personnel, at this time, the rescue task has a high priority and accuracy requirement for the perception results of the drone, and the task of perceiving disaster victims has a high requirement for latency, and the power and storage resources of the drone may also change during flight. Therefore, ignoring the above important factors when pruning the model may result in the inability to deploy the pruned model on the terminal device or the inability to obtain the best performance of the pruned model. SUMMARY

[0004] The present application provides a class adaptive model pruning method, device, electronic equipment and storage medium, to solve the defect that the existing technology only focuses on the inference accuracy of the model when pruning the model, making it difficult to deploy or perform poorly after pruning, and to achieve a pruning method that considers multiple factors to enable the pruned model to be successfully deployed on the device and obtain a class pruning model with better performance.

[0005] The present application provides a class adaptive model pruning method, comprising:

[0006] Based on the storage requirements, inference energy consumption requirements, inference latency requirements, and inference accuracy requirements for each class, an optimization model is established for each class, and based on the optimization model, a target pruning rate is determined for each class, each class being each class for which the original convolutional neural network model performs classification and identification;

[0007] performing sparse regularization training on the original convolutional neural network model to determine a scaling factor of each layer network of a backbone part of the original convolutional neural network model corresponding to each channel respectively;

[0008] determining, based on a calibration data set, a target class attention coefficient of each layer network of the backbone part corresponding to each class of target respectively for the original convolutional neural network model after sparse regularization training;

[0009] determining, based on the scaling factor and the target class attention coefficient, an importance coefficient of each channel corresponding to each layer network of the backbone part respectively for each class;

[0010] performing pruning processing on the original convolutional neural network model for each class respectively based on the target pruning rate and the importance coefficient, to obtain a class pruning model for each class.

[0011] According to the class adaptive model pruning method provided by the application, the optimization model for each class is established based on the storage requirement, inference energy consumption requirement, inference time delay requirement and inference accuracy requirement for each class, and the target pruning rate for each class is determined based on the optimization model, which comprises:

[0012] establishing an optimization model for each class based on the storage requirement, inference energy consumption requirement, inference time delay requirement and inference accuracy requirement for each class;

[0013] For the original convolutional neural network model, an inference energy consumption prediction model, an inference time delay prediction model, a storage prediction model and an inference accuracy prediction model for each class are established, the inference energy consumption prediction model represents the relationship between the required inference energy consumption of the pruned model and the pruning rate, the inference time delay prediction model represents the relationship between the inference time delay of the pruned model and the pruning rate, the storage prediction model represents the relationship between the required storage space of the pruned model and the pruning rate, and the inference accuracy prediction model represents the relationship between the inference accuracy of the pruned model and the pruning rate;

[0014] Based on the inference accuracy prediction model, the inference energy consumption prediction model, the inference time delay prediction model and the storage prediction model, the pruning rate corresponding to the optimal solution of the optimization model for each class is determined, and the pruning rate corresponding to the optimal solution of the optimization model for each class is determined as the target pruning rate for each class respectively.

[0015] According to the class adaptive model pruning method provided by the application,

[0016]

[0017] Subject to conditions: Sprune ≤S budget

[0018]

[0019] E prune ≤E budget

[0020] 0≤p≤1

[0021] wherein, p represents a pruning rate, minimize represents minimization, T prune represents an inference time delay of a pruned model, S prune represents a storage space required by the pruned model, S budget represents a storage space threshold value, represents an inference accuracy of the pruned model for class n, represents an inference accuracy threshold value for class n, E prune represents an inference energy consumption required by the pruned model, E budget represents an inference energy consumption threshold value. According to the class adaptive model pruning method provided by the application, the inference energy consumption prediction model, the inference time delay prediction model, the storage prediction model and the inference accuracy prediction model for each class are established for the original convolutional neural network model, comprising:

[0022] An initial inference energy consumption prediction model is established, and the inference energy consumption prediction model for the original convolutional neural network model is determined based on the initial inference energy consumption prediction model;

[0023] For the original convolutional neural network model, the inference time delay prediction model is established based on the real sampling data of the inference time delay and the corresponding pruning rate;

[0024] For the original convolutional neural network model, the storage prediction model is established based on the real sampling data of the storage space and the corresponding pruning rate;

[0025] For the original convolutional neural network model, the inference accuracy prediction model for each class is established based on the real sampling data of the inference accuracy corresponding to each class and the corresponding pruning rate.

[0026] According to the class adaptive model pruning method provided by the application, the initial inference energy consumption prediction model is established, comprising:

[0027] An inference energy consumption dataset is established, and the inference energy consumption dataset includes the calculation amount, the parameter amount and the inference energy consumption corresponding to different convolutional neural networks;

[0028] The inference energy consumption dataset is divided into an inference energy consumption training set and an inference energy consumption verification set;

[0029] a plurality of different regression models are trained based on the inference energy training set;

[0030] The performance of the plurality of different regression models is verified based on a verification set, and the regression model with the optimal performance is determined as an initial inference energy prediction model.

[0031] According to the category adaptive model pruning method provided by the application, for the original convolutional neural network model trained based on sparse regularization, based on a calibration data set, the target class attention coefficient of each layer network of the backbone part for each category is determined, which comprises:

[0032] The standard data set is used as the input of the original convolutional neural network model trained based on sparse regularization, and the prediction frame and the prediction category corresponding to each calibration sample in the calibration data set are output;

[0033] The original convolutional neural network model trained based on sparse regularization is subjected to back propagation, and the gradient graph of each layer network of the backbone part for each prediction category is determined;

[0034] Based on the gradient graph of each layer network of the backbone part for each prediction category, the gradient class activation graph of each layer network of the backbone part for each calibration sample is determined;

[0035] Based on the prediction frame and the gradient class activation graph, the attention coefficient of each layer network of the backbone part for each calibration sample is determined;

[0036] Based on the calibration samples of the same category, the target class attention coefficient of each layer network of the backbone part for each category is determined according to the attention coefficient of each layer network of the backbone part for each calibration sample.

[0037] According to the category adaptive model pruning method provided by the application, the scaling factor and the target class attention coefficient are used to determine the channel corresponding to each layer network of the backbone part, and the importance coefficient of each category is determined, which comprises:

[0038] For each layer network of the backbone network, the scaling factor of each channel corresponding to the same layer network is multiplied by the target class attention coefficient of each category corresponding to the layer, and the importance coefficient of each channel corresponding to each layer network of the backbone part for each category is obtained.

[0039] The application also provides a category adaptive model pruning device, which comprises:

[0040] The construction module is configured to establish an optimization model for each category based on a storage requirement, an inference energy consumption requirement, an inference time delay requirement, and an inference accuracy requirement for each category, and determine a target pruning rate for each category based on the optimization model, each category being each category classified and recognized by the original convolutional neural network model.

[0041] The first determination module is configured to perform sparse regularization training on the original convolutional neural network model to determine a scaling factor of each layer network of the backbone part corresponding to each channel.

[0042] The second determination module is configured to determine a target category attention coefficient of each layer network of the backbone part for each category based on a calibration data set for the original convolutional neural network model after sparse regularization training.

[0043] The third determination module is configured to determine an importance coefficient of each layer network of the backbone part corresponding to each channel for each category based on the scaling factor and the target category attention coefficient.

[0044] The pruning module is configured to perform pruning processing on the original convolutional neural network model for each category based on the target pruning rate and the importance coefficient to obtain a category pruning model for each category.

[0045] The application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the class adaptive model pruning method according to any one of the above when executing the program.

[0046] The application further provides a non-transitory computer readable storage medium having a computer program stored thereon, and the computer program is executable on a processor to implement the class adaptive model pruning method according to any one of the above.

[0047] The application further provides a computer program product, including a computer program, and the computer program is executable on a processor to implement the class adaptive model pruning method according to any one of the above.

[0048] The category adaptive model pruning method, device, electronic equipment and storage medium provided by the application establish an optimization model based on storage requirements, inference energy consumption requirements, inference time delay requirements and inference accuracy requirements, and solve the optimization model corresponding to each category to obtain a target pruning rate that meets the storage requirements, inference energy consumption requirements, inference accuracy requirements and inference time delay requirements. Further, based on the target pruning rate, category-based adaptive pruning is performed, thereby achieving a balance between various requirements of the model in a dynamic environment, and improving the accuracy of the category pruning model for the target category, so that the pruned model is easy to deploy on a device and has high performance. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0050] Figure 1 is a flowchart of the category adaptive model pruning method provided by the application;

[0051] Figure 2 is a fitting curve diagram of the inference time delay prediction model provided by the application;

[0052] Figure 3 is a fitting curve diagram of the storage prediction model provided by the application;

[0053] Figure 4 is a fitting curve diagram of the inference accuracy prediction model provided by the application;

[0054] Figure 5 is a flowchart of collecting inference energy consumption data provided by the application;

[0055] Figure 6 is a fitting curve diagram of the inference energy consumption prediction model provided by the application;

[0056] Figure 7 is a pruning example diagram of the original convolutional neural network provided by the application;

[0057] Figure 8 is a structure diagram of the category adaptive model pruning device provided by the application;

[0058] Figure 9 is a structure diagram of the electronic equipment provided by the application. DETAILED DESCRIPTION

[0059] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in conjunction with the drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0060] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the present application. Figures 1 to 7 A class adaptive model pruning method is described in the present application, as shown in the figure, the class adaptive model pruning method comprises: Figure 1

[0061] S101: An optimization model for each class is established based on the storage requirement, the inference energy consumption requirement, the inference time delay requirement and the inference accuracy requirement for each class, and a target pruning rate for each class is determined based on the optimization model, each class being each class for which the original convolutional neural network model performs classification recognition;

[0062] S102: The original convolutional neural network model is subjected to sparse regularization training to determine a scaling factor of each network layer corresponding to each channel in the backbone part of the original convolutional neural network model;

[0063] S103: For the original convolutional neural network model subjected to sparse regularization training, a target class attention coefficient of each network layer of the backbone part for each class is determined based on a calibration data set;

[0064] S104: An importance coefficient of each network layer of the backbone part corresponding to each channel for each class is determined based on the scaling factor and the target class attention coefficient;

[0065] S105: The original convolutional neural network model is subjected to pruning processing for each class based on the target pruning rate and the importance coefficient, to obtain a class pruning model for each class.

[0066] Specifically, the target of class adaptive pruning is to improve the accuracy of the pruned model for a specific class, which is modeled as follows:

[0067]

[0068] As can be seen from the above formula, the target of class adaptive pruning is to minimize the loss of the pruned model N c,p (x; W p ; n) and the original model N(x; W; n) for a specific class n, wherein x represents the input of the model. By applying a pruning algorithm P c ​applied to the original model N(x; W; n) with weights W, to obtain the pruned model N c,p (x; W p ; n), thereby obtaining the pruned model weight W p for the specific class n. The pruning algorithm P c selects the most important weights in the original model for the specific class n based on some preset indicators and considering the specific class n, and removes the remaining weights.

[0069] It can be understood that the model is deployed on a terminal device, which has the characteristics of changing resources such as power and storage, and the parameters of the model need to be stored on the terminal device, and the inference process of the model consumes the power of the terminal device. Therefore, the above storage requirement and inference energy consumption requirement are set based on the power resources and storage resources of the terminal device. The inference delay requirement is set based on the urgency of the prediction task, and the inference accuracy requirement is set based on the accuracy requirement of the prediction task.

[0070] When establishing the optimization model, the importance of the storage requirement, the inference energy consumption requirement, the inference delay requirement, and the inference accuracy requirement for each class can be considered comprehensively to establish an optimization model for each class that can balance each requirement, such as establishing an optimization model for class A according to the storage requirement, the inference energy consumption requirement, the inference delay requirement, and the inference accuracy requirement for class A. The target pruning rate of pruning the original convolutional neural network model for each class is determined by solving the optimization model for each class. For example, when the device energy and storage resources are scarce, it is necessary to reduce the inference energy consumption and storage amount of model inference, and when the device resources are sufficient, more accurate inference results are required, that is, the inference accuracy of the model can be mainly considered.

[0071] The original convolutional neural network model is subjected to sparse regularization training to determine the scaling factor corresponding to each channel of each network layer layer k of the backbone part of the original convolutional neural network model. The backbone part includes a plurality of sequentially connected network layers layer k , and each network layer layer k includes a BN (Batch Normalization) layer and a convolution layer (conv). After the original convolutional neural network model is subjected to sparse regularization training, each BN (Batch Normalization) layer corresponds to a scaling factor corresponding to each channel, and the scaling factor corresponding to each channel of the BN layer of each network layer is the scaling factor corresponding to each channel of the network.

[0072] The BN layer in the convolutional neural network is used to normalize the data output by the previous conv layer, which adjusts the data input to the next conv layer to make the data distribution more stable. Specifically, the BN layer normalizes the input data of each batch to make the mean value of the data 0 and the standard deviation 1, which can eliminate the influence of each conv layer of the network model on the data distribution, make the network model more stable, improve the training speed of the network model, and make the network model more easily converge. The formula for normalization by the BN layer is as follows:

[0073]

[0074] wherein z in represents the output of the previous conv layer, represents the size of the batch data, and ∈ = 10 -6 , respectively represent the mean and variance of z in , denotes the normalized data, and z out denotes the feature map of the corresponding channel output by the BN layer, and γ and β are two learnable parameters, respectively, with the shape of z in channel size, wherein γ is a scaling factor set for adjusting the order of magnitude of the output data of each layer, so that the data between different layers can better cooperate, and β represents an offset, and γ and β are learned together during sparse regularization training. The scaling factor set includes multiple elements, and the multiple elements are scaling factors γ c of the corresponding channels of the BN layer.

[0075] For the original convolutional neural network model, when some scaling factors γ c tend to 0, the feature map z out output by the BN layer also tends to 0, and these scaling factors tend to 0. These channels contribute little to subsequent inference and can be removed. To obtain a model that can be pruned, an L1 (L1 = λ∑ γ∈Γ |γ|) regularization term is added to the scaling factor set γ during sparse regularization training, which is used to limit the sum of the absolute values of the parameters of the original convolutional neural network model. The original convolutional neural network model will automatically select important scaling factors and make the values of other scaling factors tend to 0. The loss function L with the regularization term added to the scaling factor is as follows:

[0076]

[0077] Here, x and t represent the input and corresponding true value of the original convolutional neural network model, respectively. W represents the trainable weights. l(f(x, W), y)) is the loss function for normal training of the original convolutional neural network. After sparse regularization training, the set of all scaling factors γ in the original convolutional neural network model generally shows a sparse trend. λ is the weight coefficient, and Γ represents the set of all scaling factors.

[0078] When performing sparse regularization training on the original convolutional neural network model, the training can be stopped after the number of training rounds meets the preset number of rounds, and the trained original convolutional neural network model is obtained. Then, the scaling factors of each channel corresponding to each layer of the backbone of the original convolutional neural network model can be determined.

[0079] When executing different types of tasks, each layer of the network has different importance for different categories. The same pruning method is used for all models corresponding to different categories, without considering the importance difference of each layer of the network for different categories. For the original convolutional neural network model after sparse regularization training, based on the calibration dataset, the importance of each layer of the backbone network is determined. k The target category attention coefficient for each category is respectively used. Among them, a data set can be determined and divided into a training set and a validation set. The training set is used to perform sparse regularization training on the original convolutional neural network model. M (M is greater than or equal to 1000) samples are extracted from the validation set as the calibration data set Dataset cal ={img i ,i∈[1,M]},img i Represents the calibration samples in the calibration dataset. The class of this calibration dataset is class={class j ,j∈[1,S]}, where S represents all categories corresponding to the calibration dataset. For example, if there are 100 calibration samples in the calibration dataset, 10 of them belong to the rabbit category, 25 belong to the dog category, 5 belong to the monkey category, 30 belong to the pig category, 12 belong to the cat category, and 18 belong to the chicken category, then S represents all categories corresponding to the calibration dataset. If there are 6 categories in total, S is 6. When the calibration dataset has 6 categories, each layer of the backbone network determines the target category attention coefficient for each category, that is, each layer of the network will determine 6 target category attention coefficients, and these 6 target category attention coefficients correspond to 6 categories respectively.

[0080] After determining the target category attention coefficient corresponding to each layer of the network and the scaling factor corresponding to each channel of each layer of the network, the scaling factor and the target category attention coefficient are comprehensively considered to determine the importance coefficient of each category for each channel corresponding to each layer of the backbone network.

[0081] After obtaining the importance coefficient, the original convolutional neural network model is pruned for each category to obtain a category pruning model for each category. For a certain category, pruning is performed from the channel with the smallest importance coefficient for the category until the target pruning rate for the category is reached, i.e., a category pruning model for the category is obtained. Compared with pruning considering overall accuracy, pruning considering a specific type can sacrifice the accuracy of other categories, retain parameters more important for a specific category, and make the pruned model have higher inference accuracy for the specific category.

[0082] The category adaptive model pruning method provided by the application establishes an optimization model based on storage requirements, inference energy consumption requirements, inference time delay requirements, and inference accuracy requirements, and solves the optimization model corresponding to each category to obtain a target pruning rate that meets the storage requirements, inference energy consumption requirements, inference accuracy requirements, and inference time delay requirements. Further, based on the target pruning rate, adaptive pruning based on categories is performed, thereby balancing the requirements in a dynamic environment and improving the accuracy of the category pruning model for the target category, so that the pruned model is easy to deploy on a device and has high performance.

[0083] In one embodiment, the optimization model for each category is established based on the storage requirements, inference energy consumption requirements, inference time delay requirements, and inference accuracy requirements for each category, and the target pruning rate for each category is determined based on the optimization model, comprising:

[0084] An optimization model for each category is established based on the storage requirements, inference energy consumption requirements, inference time delay requirements, and inference accuracy requirements for each category;

[0085] For the original convolutional neural network model, an inference energy consumption prediction model, an inference time delay prediction model, a storage prediction model, and an inference accuracy prediction model for each category are established. The inference energy consumption prediction model represents the relationship between the inference energy consumption of the pruned model and the pruning rate, the inference time delay prediction model represents the relationship between the inference time delay of the pruned model and the pruning rate, the storage prediction model represents the relationship between the storage space required by the pruned model and the pruning rate, and the inference accuracy prediction model represents the relationship between the inference accuracy of the pruned model and the pruning rate;

[0086] Based on the inference accuracy prediction model, the inference energy consumption prediction model, the inference time delay prediction model, and the storage prediction model, the pruning rate corresponding to the optimal solution of the optimization model for each category is determined, and the pruning rate corresponding to the optimal solution of the optimization model for each category is determined as the target pruning rate for each category.

[0087] Specifically, according to the importance of the storage requirement, the inference energy consumption requirement, the inference time delay requirement and the inference accuracy requirement of each category, an optimization model that can balance each requirement is established for each category.

[0088] For the original convolutional neural network model, an inference energy consumption prediction model, an inference time delay prediction model, a storage prediction model and an inference accuracy prediction model for each category are established. Since the inference accuracy prediction model can represent the relationship between the inference accuracy of the pruned model and the pruning rate, the inference energy consumption prediction model can represent the relationship between the required inference energy consumption of the pruned model and the pruning rate, the inference time delay prediction model can represent the relationship between the inference time delay of the pruned model and the pruning rate, and the storage prediction model can represent the relationship between the required storage space of the pruned model and the pruning rate. Therefore, the optimization model for each category established according to the storage requirement, the inference energy consumption requirement, the inference time delay requirement and the inference accuracy can be converted into an optimization model based on the pruning rate. Solving the optimal solution of each optimization model can determine the target pruning rate for each category.

[0089] In one embodiment, the optimization model is:

[0090]

[0091] Subject to the condition:

[0092] wherein p represents the pruning rate, minimize represents minimization, T prune represents the inference time delay of the pruned model, S prune represents the required storage space of the pruned model, S budget represents the storage space threshold, represents the inference accuracy of the pruned model for category n, represents the inference accuracy threshold for category n, E prune represents the required inference energy consumption of the pruned model, E budget represents the inference energy consumption threshold.

[0093] Specifically, the optimization model can be established according to the terminal resource condition and the requirements of the user for the inference time delay and the inference accuracy. Exemplarily, the optimization model can also be other types in addition to formula (4).

[0094] The optimization model comprehensively considers the importance of the storage requirement, the inference energy consumption requirement, the inference time delay requirement and the inference accuracy requirement, so that the pruned model is easy to deploy and the performance of the model is good.

[0095] In one embodiment, the establishing, for the original convolutional neural network model, an inference energy consumption prediction model, an inference time delay prediction model, a storage prediction model, and an inference accuracy prediction model for each category respectively, comprises:

[0096] An initial inference energy consumption prediction model is established, and the inference energy consumption prediction model for the original convolutional neural network model is determined based on the initial inference energy consumption prediction model.

[0097] For the original convolutional neural network model, the inference time delay prediction model is established based on the real sampling data of the inference time delay and the corresponding pruning rate.

[0098] For the original convolutional neural network model, the storage prediction model is established based on the real sampling data of the storage space and the corresponding pruning rate.

[0099] For the original convolutional neural network model, the inference accuracy prediction model for each category is established based on the real sampling data of the inference accuracy corresponding to each category and the corresponding pruning rate.

[0100] Specifically, an initial inference energy consumption prediction model is established, and the initial inference energy consumption model can represent the relationship between the parameter quantity, the calculation quantity, and the inference energy consumption. Further, the parameter quantity and the calculation quantity of the model are related to the pruning rate, so that the parameter quantity and the calculation quantity are represented based on the pruning rate, which can determine the inference energy consumption prediction model that can represent the relationship between the inference energy consumption and the pruning rate.

[0101] Considering the influence of different pruning rates on the inference time delay, for the original convolutional neural network model, a linear least squares method is used to fit the relationship between the pruning rate and the inference time delay in this embodiment to establish an inference time delay prediction model, which is represented as:

[0102] T prune = w0 + w1 x p (5)

[0103] where T prune represents the inference time delay, p represents the pruning rate, and the parameters w0 and w1 can be learned from the actual data pairs of the inference time delay and the pruning rate using the linear least squares method. The fitting curve of the inference time delay prediction model after fitting is shown in Figure 2 . Figure 2 In the "100-pruning rate%" in the above formula, "100-pruning rate%" means that when the pruning rate is 0%, it corresponds to 100 on the horizontal coordinate.

[0104] Considering the influence of different pruning rates on the storage space, for the original convolutional neural network model, a linear least squares method is used to fit the relationship between the pruning rate and the storage space in this embodiment to establish a storage prediction model, which is represented as:

[0105] Sprune = w2 + w3 x p (6)

[0106] where S prune denotes the storage space, p denotes the pruning rate, and the parameters w2 and w3 can be learned from the actual data pairs of the storage space and the pruning rate using the linear least squares method, and the fitting curve of the prediction model after fitting is shown in Figure 3 Figure 3 In the figure, “100-pruning rate %” means that when the pruning rate is 0%, it corresponds to 100 on the horizontal coordinate.

[0107] Considering the influence of different pruning rates on inference accuracy in each category, the embodiment establishes an inference accuracy prediction model by sampling the real inference accuracy and pruning rate data pairs in category n, determines the S-shaped function as the fitting function corresponding to the inference accuracy prediction model, and the obtained inference accuracy prediction model after fitting is as follows:

[0108]

[0109] wherein, is the predicted accuracy of category n, p is the pruning rate, e represents the natural constant, and the parameters a n and b n control the shape of the S-shaped function, the parameter a n controls the maximum value of the S-shaped function, and the parameter b n determines the midpoint of the curve corresponding to the S-shaped function, and the values of the parameters a n and b n are learned from the data pairs of inference accuracy and pruning rate, and are determined based on the nonlinear least squares regression method, Figure 4 The fitting curve diagram of the inference accuracy prediction model is shown in Figure 4 In the figure, “100-pruning rate %” means that when the pruning rate is 0%, it corresponds to 100 on the horizontal coordinate.

[0110] In one embodiment, the establishing an initial inference energy consumption prediction model comprises:

[0111] establishing an inference energy consumption dataset, the inference energy consumption dataset comprising the calculation amount, parameter amount and inference energy consumption corresponding to different convolutional neural networks;

[0112] dividing the inference energy consumption dataset into an inference energy consumption training set and an inference energy consumption verification set;

[0113] training a plurality of different regression models based on the inference energy consumption training set;

[0114] verifying the performance of the plurality of different regression models based on the verification set, and determining the regression model with the optimal performance as the initial inference energy consumption prediction model.

[0115] ​Specifically, the terminal device includes a mobile device with limited power, and therefore, the inference process of the model on the terminal device needs to consider the power resource consumption problem. If the inference energy consumption is too high, it may affect the performance of the terminal device or even cause the terminal device to shut down. However, due to the complexity of the model and the influence of various factors such as hardware platforms, input data, and environmental conditions, it is very difficult to estimate the inference energy consumption, and it is difficult to accurately describe it through mathematical modeling methods. In order to solve this problem, the embodiment collects inference energy consumption data of different models under different conditions, establishes an inference energy consumption dataset, and uses a data-driven method to establish an initial inference energy consumption prediction model.

[0116] The model inference energy consumption is usually related to the computational complexity and storage complexity of the model. The computational complexity can be measured by the number of floating-point operations (FLOPs), which reflects the number of mathematical operations required by the model during inference. The storage complexity can be measured by the number of parameters, which reflects the number of parameters required by the model during inference. Generally, the higher the computational complexity and storage complexity, the higher the inference energy consumption of the model, because more operations and memory access operations will consume more power resources. Based on this, an initial inference energy consumption prediction model can be established according to the relationship between inference energy consumption and computational complexity and storage complexity.

[0117] Exemplarily, the method for establishing an initial inference energy consumption prediction model includes:

[0118] In order to simulate convolutional neural network models with different computational complexity, different sizes of convolutional neural network models are generated by randomly combining three different types of modules: convolution (Conv), depthwise convolution (DWConv), and bottleneck module (BottleNeck). Exemplarily, a total of 5550 different convolutional neural network models are generated, with the number of layers of the convolutional neural network model varying between 10 and 50 layers, and the size of the input image varying from the largest 100x100x3 (pixels) to the smallest 20x20x3 (pixels). And each convolutional neural network model randomly uses 10 groups of input images with different sizes for inference. In order to reduce data collection errors, the final inference energy consumption result is the average of N (such as N=10) inference results, and data with inference energy consumption less than 0.1 joule is removed.

[0119] As Figure 5As shown, when collecting inference energy consumption, before each randomly generated convolutional neural network model starts inference, a sub-thread is started to monitor the GPU power of the terminal device, and the millisecond-level timestamp and the corresponding power are recorded. For example, the GPU power can be obtained based on the GPU bus interface provided by NVIDIA. For example, on the NVIDIA Jetson TX2, the current GPU power of the terminal device can be obtained in real time by monitoring the address / sys / bus / i2c / drivers / ina3221x / 0-0040 / iio:device0 / in_power0_input on the bus. After a single convolutional neural network model inference ends, the power data between the inference start and end is analyzed, and the energy consumption of an input sample for one inference is calculated by summing the product of the power and the time period during the inference. The average energy consumption of an input sample for N times of inference is the inference energy consumption for an input sample, where t i represents the time period of the inference process, P i represents the power during the inference process.

[0120] By collecting the inference energy consumption data of different convolutional neural network models, an inference energy consumption dataset can be established, which contains the computational complexity, parameter quantity and inference energy consumption of 5550 convolutional neural network models of different sizes. For example, the complete inference energy consumption dataset is randomly divided into an inference energy consumption training set and an inference energy consumption verification set according to a ratio of 9:1. The inference energy consumption training set is used to train multiple types of regression models, and the inference energy consumption verification set is used to compare the performance of different types of regression models in predicting inference energy consumption, such as accuracy and generalization. The regression model with the best performance is determined as the initial inference energy consumption prediction model. According to experiments, the regression model with the best performance is a linear regression model. Linear regression is a simple and reliable regression method that predicts target variables by fitting data points. It learns the rules in the training data to predict energy consumption. Linear regression models also have high sample efficiency. They do not need to generate multiple additional models to fit the data, reducing the consumption of computing resources. Moreover, this method can be directly written as an expression, so it is convenient for subsequent optimization and solution.

[0121] Table 1 gives the statistical information corresponding to the complete inference energy consumption dataset, the inference energy consumption training set and the inference energy consumption verification set, respectively. The statistical information includes the data quantity and the maximum, minimum and average values of the computational complexity, parameter quantity and inference energy consumption.

[0122] Table 1 Inference energy consumption dataset statistical information

[0123]

[0124] The initial inference energy consumption prediction model represents the relationship between inference energy consumption and parameter quantity and calculation quantity. After determining the linear regression model as the initial inference energy consumption prediction model, the parameter quantity and the calculation quantity are represented based on the pruning rate, so as to determine the inference energy consumption prediction model capable of representing the relationship between inference energy consumption and the pruning rate. The inference energy consumption prediction model is represented as follows:

[0125] E prune = (w2 + w3 x p) x a e + (a f + b f x p) x b e + c e (8)

[0126] wherein, E prune represents inference energy consumption, (w2 + w3 x p) represents the parameter quantity represented based on the pruning rate, (a f + b f x p) represents the calculation quantity represented based on the pruning rate, p represents the pruning rate, and parameters a f , b f , a e , b e and c e are the fitting coefficients of the relationship between the model calculation quantity and the pruning rate p, which can be obtained by fitting training based on the inference energy consumption training set. The fitting curve of the obtained inference energy consumption prediction model is shown in Figure 6 .

[0127] In an embodiment, the original convolutional neural network model trained for sparse regularization is based on a calibration data set to determine the target class attention coefficient of each layer network of the backbone part for each category, comprising:

[0128] The standard data set is used as the input of the original convolutional neural network model trained for sparse regularization, and the prediction frame and the prediction category corresponding to each calibration sample in the calibration data set are output;

[0129] The original convolutional neural network model trained for sparse regularization is subjected to back propagation to determine the gradient graph of each layer network of the backbone part for each prediction category;

[0130] Based on the gradient graph of each layer network of the backbone part for each prediction category, the gradient category activation graph of each layer network of the backbone part for each calibration sample is determined;

[0131] Based on the prediction frame and the gradient category activation graph, the attention coefficient of each layer network of the backbone part for each calibration sample is determined;

[0132] According to the attention coefficients of each of the calibration samples by the backbone network, the target class attention coefficients of each of the layers of the backbone network are determined for each of the calibration samples.

[0133] Specifically, the original convolutional neural network model Model = {layer k ,k∈[1,Y], Y represents the number of network layers layer k of the backbone of the original convolutional neural network model. k includes Conv k and BN k layer, so the backbone includes Conv = {Conv k ,k∈[1,Y]} layer and corresponding BN = {BN k ,k∈[1,Y]} layer, generally, the output of Conv k layer will be input to BN k .

[0134] The calibration dataset Dataset cal ={img i ,i∈[1,M]} is used as the input of the original convolutional neural network model after sparse regularization training, and the output is the prediction box bbox and the prediction class class pred corresponding to each calibration sample in the calibration data set:

[0135] bbox, class pred = Model(img i ) (9)

[0136] After the prediction box bbox and the prediction class class pred are output, the original convolutional neural network model after sparse regularization training is backpropagated to determine the gradient graph of each network layer layer k of the backbone for each prediction class class pred .

[0137]

[0138] wherein, the shape of CxHxW, C is the number of channels of the original convolutional neural network model, H is the number of pixels in height, W is the number of pixels in width, Backpropagation represents backpropagation, feat k represents the output feature of network layer layer k , the shape is the same as , which is CxHxW.

[0139] Summing up in spatial dimensions and taking average value to obtain average gradient

[0140]

[0141] wherein, represents gradient map gradient of pixel point on the upper image.

[0142] Based on average gradient layer layer k Gradient class activation map of class class pred

[0143]

[0144] wherein, shape of is CxHxW.

[0145] For object detection task, class class pred is class of prediction box bbox, by retaining gradient class activation value of each pixel point in bbox, and summing up gradient class activation value of each pixel point in bbox, layer layer pred k class class pred class attention coefficient

[0146]

[0147] wherein, represents activation map gradient class activation value of pixel point on the upper image, (h,w)in bbox) represents pixel point located in prediction box, represents pixel number on prediction box height, represents pixel number on prediction box width.

[0148] After obtaining layer layer k attention coefficient of all calibration samples in calibration data set After, based on each calibration sample with same prediction class, average value of layer layer k corresponding attention coefficient, to obtain target class attention coefficient of each layer network of original convolutional neural network backbone for each class For example, when calibration sample has 100 calibration samples, prediction class of 5 calibration samples is monkey, at this time, layer layer k ​​​Take the average of the attention coefficients of these five calibration samples to get layer k The target category attention coefficient.

[0149] In one embodiment, determining the channels corresponding to each layer of the backbone network based on the scaling factor and the target category attention coefficient, and respectively for each category importance coefficient, includes:

[0150] For each layer of the backbone network, the scaling factors of the channels corresponding to the same layer of the network are multiplied by the target category attention coefficients corresponding to the layer for each category, and the importance coefficients of the channels corresponding to each layer of the backbone network for each category are obtained.

[0151] Specifically, such as Figure 7 As shown, when each network layer is obtained k Target category attention coefficient for each category After that, use And the scaling factor γ of each channel corresponding to each network layer kc (k represents the corresponding network layer, c represents the corresponding channel) to complete the pruning of a specific category, specifically using the product of the two As the importance coefficient of each channel corresponding to each layer:

[0152]

[0153] right Sort by importance coefficient set α sort , for a certain target pruning rate The scaling factor threshold T can be calculated α :

[0154]

[0155] Among them, len represents the length of the set, mask k presentation layer k An index set for the index of each channel in the set, where each index is 0 or 1, 0 indicates that the channel needs to be removed, and 1 indicates that the channel needs to be retained.

[0156] Get the mask k After that, the layers of the original convolutional neural network model k Expressed as:

[0157]

[0158] Among them, Conv′ k To perform Conv based on the index setk obtained after channel pruning, BN' k is based on an index set of BN k obtained after channel pruning. When it is determined that the layer layer k corresponding to a channel needs to be removed, the layer layer k in Conv k and BN k corresponding to the channel need to be removed.

[0159] The category adaptive model pruning device provided by the present application is described below. The category adaptive model pruning device described below can be correspondingly referred to the category adaptive model pruning method described above.

[0160] As shown in FIG. 8, the category adaptive model pruning device 800 comprises a construction module 801, a first determination module 802, a second determination module 803, a third determination module 804 and a pruning module 805. Figure 8 The construction module 801 is configured to establish an optimization model based on a storage requirement, an inference energy consumption requirement, an inference time delay requirement and an inference accuracy requirement, and determine a target pruning rate based on the optimization model, each category being each category classified and recognized by the original convolutional neural network model.

[0161] The first determination module 802 is configured to perform sparse regularization training on the original convolutional neural network model to determine a scaling factor of each layer network in a backbone part of the original convolutional neural network model corresponding to each channel.

[0162] The second determination module 803 is configured to determine a target category attention coefficient of each layer network in the backbone part for each category based on a calibration data set for the original convolutional neural network model after sparse regularization training.

[0163] The third determination module 804 is configured to determine an importance coefficient of each channel corresponding to each layer network in the backbone part for each category based on the scaling factor and the target category attention coefficient.

[0164] The pruning module 805 is configured to perform pruning processing on the original convolutional neural network model for each category based on the target pruning rate and the importance coefficient to obtain a category pruning model for each category.

[0165]

[0166] ​The category adaptive model pruning device provided by the application balances the requirements of the model in a dynamic environment, and improves the accuracy of the category pruning model for target categories, so that the pruned model is convenient to deploy on a device and has high performance.

[0167] In one embodiment, the construction module 801 is specifically configured to:

[0168] establish an optimization model for each category according to the storage requirement, the inference energy consumption requirement, the inference time delay requirement, and the inference accuracy requirement for each category;

[0169] For the original convolutional neural network model, an inference energy consumption prediction model, an inference time delay prediction model, a storage prediction model, and an inference accuracy prediction model for each category are established, the inference energy consumption prediction model represents the relationship between the required inference energy consumption of the pruned model and the pruning rate, the inference time delay prediction model represents the relationship between the inference time delay of the pruned model and the pruning rate, the storage prediction model represents the relationship between the required storage space of the pruned model and the pruning rate, and the inference accuracy prediction model represents the relationship between the inference accuracy of the pruned model and the pruning rate;

[0170] Based on the inference accuracy prediction model, the inference energy consumption prediction model, the inference time delay prediction model, and the storage prediction model, the pruning rate corresponding to the optimal solution of the optimization model for each category is determined, and the pruning rate corresponding to the optimal solution of the optimization model for each category is determined as the target pruning rate for each category.

[0171] In one embodiment, the optimization model is:

[0172]

[0173] Subject to condition: S prune ≤S budget

[0174]

[0175] E prune ≤E budget

[0176] 0≤p≤1

[0177] Wherein, p represents the pruning rate, minimize represents minimization, T prunean inference latency of the pruned model, S prune a storage space required by the pruned model, S budget a storage space threshold value, an inference accuracy of the pruned model for class n, an inference accuracy threshold value for class n, E prune an inference energy consumption required by the pruned model, E budget an inference energy consumption threshold value.

[0178] In an embodiment, the constructing module 801 is specifically configured to:

[0179] establish an initial inference energy consumption prediction model, and determine, based on the initial inference energy consumption prediction model, an inference energy consumption prediction model for the original convolutional neural network model;

[0180] for the original convolutional neural network model, establish the inference latency prediction model based on real sampling data of inference latency and corresponding pruning rates;

[0181] for the original convolutional neural network model, establish the storage prediction model based on real sampling data of storage space and corresponding pruning rates;

[0182] for the original convolutional neural network model, respectively establish the inference accuracy prediction model for each class based on real sampling data of inference accuracy of each class and corresponding pruning rates.

[0183] In an embodiment, the constructing module 801 is specifically configured to:

[0184] establish an inference energy consumption dataset, the inference energy consumption dataset including calculation amount, parameter amount and inference energy consumption corresponding to different convolutional neural networks;

[0185] divide the inference energy consumption dataset into an inference energy consumption training set and an inference energy consumption verification set;

[0186] train a plurality of different regression models based on the inference energy consumption training set;

[0187] verify the performance of the plurality of different regression models based on the verification set respectively, and determine the regression model with the optimal performance as the initial inference energy consumption prediction model.

[0188] In an embodiment, the second determining module 803 is specifically configured to:

[0189] use the standard dataset as the input of the original convolutional neural network model after sparse regularization training, and output the prediction frame and the prediction class corresponding to each calibration sample in the calibration dataset respectively;

[0190] perform back propagation on the original convolutional neural network model trained by sparse regularization to determine gradient maps of each layer network of the backbone part for each prediction category;

[0191] determine gradient category activation maps of each layer network of the backbone part for each calibration sample based on the gradient maps of each layer network of the backbone part for each prediction category;

[0192] determine attention coefficients of each layer network of the backbone part for each calibration sample based on the prediction box and the gradient category activation map;

[0193] determine target category attention coefficients of each layer network of the backbone part for each category based on the attention coefficients of each layer network of the backbone part for each calibration sample according to each calibration sample of the same category.

[0194] In one embodiment, the third determination module 804 is specifically configured to:

[0195] for each layer network of the backbone network, multiply the scaling factor of each channel corresponding to the same layer network by the target category attention coefficient of each category corresponding to the layer to obtain an importance coefficient of each channel corresponding to each layer network of the backbone part for each category.

[0196] Figure 9 An example of an entity structure diagram of an electronic device is shown in Figure 9 As shown, the electronic device can include a processor 910, a communications interface 920, a memory 930, and a communications bus 940, wherein the processor 910, the communications interface 920, and the memory 1030 communicate with each other through the communications bus 940. The processor 910 can invoke the logic instructions in the memory 930 to execute a class adaptive model pruning method, which includes:

[0197] based on storage requirements, inference energy consumption requirements, inference latency requirements, and inference accuracy requirements for each category, an optimization model is established for each category, and based on the optimization model, a target pruning rate is determined for each category, each category being each category for which the original convolutional neural network model performs classification recognition;

[0198] perform sparse regularization training on the original convolutional neural network model to determine scaling factors of each layer network of the backbone part of the original convolutional neural network model corresponding to each channel;

[0199] For the original convolutional neural network model trained by sparse regularization, based on the calibration data set, the attention coefficient of each layer network of the backbone part for each category of target is determined respectively;

[0200] Based on the scaling factor and the target category attention coefficient, the corresponding channel of each layer network of the backbone part is determined respectively for each category, and the importance coefficient of each category is determined respectively;

[0201] Based on the target pruning rate and the importance coefficient, the original convolutional neural network model is pruned for each category respectively, and a category pruning model for each category is obtained.

[0202] In addition, the logical instructions in the memory 930 described above can be implemented in the form of a software function unit and sold or used as a separate product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the part of the prior art that contributes essentially or the part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various storage program codes.

[0203] On the other hand, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program is executed by a processor, so that the computer can execute the category adaptive model pruning method provided by the above-mentioned method, which comprises:

[0204] Based on the storage requirement, the inference energy consumption requirement, the inference time delay requirement and the inference accuracy requirement for each category, an optimization model for each category is established respectively, and based on the optimization model, a target pruning rate for each category is determined respectively, each category being each category classified and recognized by the original convolutional neural network model;

[0205] The original convolutional neural network model is trained by sparse regularization to determine the scaling factor of each layer network corresponding to each channel in the backbone part of the original convolutional neural network model;

[0206] For the original convolutional neural network model trained by sparse regularization, based on the calibration data set, the target class attention coefficient of each layer network of the backbone part is determined for each category respectively;

[0207] Based on the scaling factor and the target class attention coefficient, the importance coefficient of each category is determined for each channel corresponding to each layer network of the backbone part respectively;

[0208] Based on the target pruning rate and the importance coefficient, the original convolutional neural network model is pruned for each category respectively, and a category pruning model for each category is obtained.

[0209] In another aspect, the application also provides a non-transitory computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the category adaptive model pruning method provided by the above method, which comprises:

[0210] Based on the optimization model, the target pruning rate for each category is determined, and each category is each category classified and recognized by the original convolutional neural network model;

[0211] The original convolutional neural network model is trained by sparse regularization to determine the scaling factor of each channel corresponding to each layer network in the backbone part of the original convolutional neural network model;

[0212] For the original convolutional neural network model trained by sparse regularization, based on the calibration data set, the target class attention coefficient of each layer network of the backbone part is determined for each category respectively;

[0213] Based on the scaling factor and the target class attention coefficient, the importance coefficient of each category is determined for each channel corresponding to each layer network of the backbone part respectively;

[0214] Based on the target pruning rate and the importance coefficient, the original convolutional neural network model is pruned for each category respectively, and a category pruning model for each category is obtained.

[0215] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0216] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0217] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A class adaptive model pruning method, characterized in that, The method comprises the steps of: establishing an optimization model for each category based on storage requirements, inference energy consumption requirements, inference time delay requirements, and inference accuracy requirements for each category, and determining a target pruning rate for each category based on the optimization model, each category being each category classified and identified by the original convolutional neural network model; performing sparse regularization training on the original convolutional neural network model to determine a scaling factor of each network layer of the backbone part of the original convolutional neural network model corresponding to each channel; based on the calibration data set, determining a target class attention coefficient of each network layer of the backbone part for each category; the calibration data set is an image classification data set; based on the scaling factor and the target class attention coefficient, determining an importance coefficient of each channel corresponding to each network layer of the backbone part for each category; based on the target pruning rate and the importance coefficient, performing pruning processing on the original convolutional neural network model for each category to obtain a category pruning model for each category; the optimization model is: ； ； in, represents the pruning rate, represents minimization, represents the inference latency of the pruned model, represents the storage space required by the pruned model, Indicates the storage space threshold. Indicates that the category The inference accuracy of the pruned model, Indicates that the category The inference accuracy threshold, represents the inference energy consumption required by the pruned model, Indicates the inference energy consumption threshold.

2. The class adaptive model pruning method of claim 1, wherein, the optimization model is: the optimization model is: the optimization model is: the optimization model is:

3. The class adaptive model pruning method of claim 2, wherein, the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the optimization model is: the The inference accuracy prediction model for each category is respectively established based on the inference accuracy corresponding to each category and the real sampling data of the corresponding pruning rate.

4. The class adaptive model pruning method of claim 3, wherein, The initial inference energy consumption prediction model is established, including: An inference energy consumption dataset is established, which includes the calculation amount, parameter amount and inference energy consumption corresponding to different convolutional neural networks; The inference energy consumption dataset is divided into an inference energy consumption training set and an inference energy consumption verification set; A plurality of different regression models are trained based on the inference energy consumption training set; The performance of the plurality of different regression models is verified based on the verification set, and the regression model with the optimal performance is determined as the initial inference energy consumption prediction model.

5. The class adaptive model pruning method according to any one of claims 1 to 4, characterized in that, The target class attention coefficient of each layer network of the backbone part for each category is determined based on the calibration dataset, including: The calibration dataset is used as the input of the original convolutional neural network model after sparse regularization training, and the prediction box and the prediction category corresponding to each calibration sample in the calibration dataset are output; The gradient graph of each layer network of the backbone part for each prediction category is determined by performing back propagation on the original convolutional neural network model after sparse regularization training; Based on the gradient graph of each layer network of the backbone part for each prediction category, the gradient class activation graph of each layer network of the backbone part for each calibration sample is determined; Based on the prediction box and the gradient class activation graph, the attention coefficient of each layer network of the backbone part for each calibration sample is determined; Based on the calibration samples with the same category, the target class attention coefficient of each layer network of the backbone part for each category is determined according to the attention coefficient of each layer network of the backbone part for each calibration sample.

6. The class adaptive model pruning method according to any one of claims 1 to 4, characterized in that, The importance coefficient of each channel corresponding to each layer network of the backbone part for each category is determined based on the scaling factor and the target class attention coefficient, including: For each layer network of the backbone network, the scaling factor of each channel corresponding to the same layer network is multiplied by the target class attention coefficient of each category corresponding to the layer, to obtain the importance coefficient of each channel corresponding to each layer network of the backbone part for each category.

7. A class adaptive model pruning apparatus, characterized by, It includes: The construction module is used to establish an optimization model for each category based on the storage requirement, inference energy consumption requirement, inference time delay requirement and inference accuracy requirement for each category, and to determine the target pruning rate for each category based on the optimization model, each category being each category for classification and identification of the original convolutional neural network model; The first determination module is used to perform sparse regularization training on the original convolutional neural network model to determine the scaling factor of each layer network of the backbone part of the original convolutional neural network model corresponding to each channel; The second determination module is used to determine the target class attention coefficient of each layer network of the backbone part for each category based on the calibration dataset for the original convolutional neural network model after sparse regularization training. The calibration dataset is an image classification dataset; The third determination module is configured to determine the channels corresponding to each layer of the backbone part based on the scaling factor and the target class attention coefficient, and determine the importance coefficients of each class respectively; The pruning module is configured to perform pruning processing on the original convolutional neural network model for each class based on the target pruning rate and the importance coefficients, and obtain a class pruning model for each class. The optimization model is: ； ； in, represents the pruning rate, represents minimization, represents the inference latency of the pruned model, represents the storage space required by the pruned model, Indicates the storage space threshold. Indicates that the category The inference accuracy of the pruned model, Indicates that the category The inference accuracy threshold, represents the inference energy consumption required by the pruned model, Indicates the inference energy consumption threshold.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the class adaptive model pruning method according to any one of claims 1-6 when executing the program. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the class adaptive model pruning method according to any one of claims 1-6 when executed by the processor.

Images