A scene-adaptive model compression method and system

Abstract

The application provides a scene-adaptive model compression method and system. The method comprises the following steps: calculating a pixel vector entropy of a scene data set, and calculating a scene complexity of the scene data set based on the pixel vector entropy and by using a predefined scene complexity evaluation method; under the control of the scene complexity, calculating a weighted total importance of each channel in a channel feature map data set by using weight parameters and channel feature map data of a pre-trained model based on a mixed strategy of dynamic importance and static importance; converting a layer sparsity distribution problem of the pre-trained model into a single-target multi-constraint knapsack problem by using the weighted total importance of each channel in the channel feature map data set, solving the single-target multi-constraint knapsack problem by using a multi-constraint dynamic programming algorithm, and adaptively compressing the pre-trained model based on a solving result.

Description

Technical Field

[0001] This invention relates to the fields of Internet of Things and artificial intelligence technologies, and in particular to a scene-adaptive model compression method and system, electronic device and storage medium. Background Technology

[0002] With the development of the Internet of Things (IoT) and Artificial Intelligence (AI), the deployment and inference of intelligent models have gradually become a research focus. Today, a large number of smart surveillance cameras have become important IoT sensors. These smart cameras deployed in cities and residences have enabled many intelligent applications, such as crowd counting and behavior analysis. In the traditional computing paradigm, data collected by the cameras is transmitted to the cloud, and the model inferences the results and sends them back to the local device. Due to factors such as user privacy, latency requirements, and unstable network connections, deploying convolutional neural networks in embedded devices such as smart cameras has become a hot research topic. However, the scarce computing and storage resources of embedded devices make it difficult to directly deploy large pre-trained models on them. Although existing work has studied how to optimize the computational efficiency of models, such as model pruning, parameter quantization, knowledge distillation, input filtering, and hardware-level optimization, problems such as high computational and time costs and poor generalization ability of the deployed intelligent models still exist. Summary of the Invention

[0003] In view of the above problems, the present invention provides a scene-adaptive model compression method and system, electronic device and storage medium, in order to at least solve one of the above problems.

[0004] According to a first aspect of the present invention, a scene-adaptive model compression method is provided, comprising:

[0005] Calculate the pixel vector entropy of the scene dataset, and based on the pixel vector entropy, calculate the scene complexity of the scene dataset using a predefined scene complexity evaluation method, wherein the scene dataset includes images;

[0006] Based on a hybrid strategy of dynamic and static importance, under the control of scene complexity, the weighted total importance of each channel in the channel feature map dataset is calculated using the weight parameters of the pre-trained model and the channel feature map dataset. The channel feature map dataset is obtained by processing the scene dataset through the pre-trained model.

[0007] By utilizing the weighted total importance of each channel in the channel feature map dataset, the layer sparsity allocation problem of the pre-trained model is transformed into a single-objective multi-constraint knapsack problem. The single-objective multi-constraint knapsack problem is solved using a multi-constraint dynamic programming algorithm, and the pre-trained model is adaptively compressed based on the solution results.

[0008] According to an embodiment of the present invention, the above-described calculation of the pixel vector entropy of the scene dataset, and the calculation of the scene complexity of the scene dataset based on the pixel vector entropy using a predefined scene complexity evaluation method, includes:

[0009] Extract pixels at specific pixel locations from all images in the scene dataset, and flatten the extracted pixels into an intermediate vector;

[0010] The pixel vector entropy is obtained by calculating the intermediate vector using a predefined pixel vector entropy calculation method. The pixel vector entropy represents the amount of information in the image at a specific pixel location.

[0011] For different specific pixel locations in the scene dataset, the extraction, flattening and calculation operations are repeatedly performed to obtain multiple pixel vector entropies at different specific pixel locations.

[0012] The scene complexity of the scene dataset is obtained by averaging and normalizing the entropy of multiple pixel vectors and then calculating the entropy of the averaged and normalized pixel vectors using a predefined scene complexity evaluation method.

[0013] According to an embodiment of the present invention, the predefined pixel vector entropy calculation method described above is represented by formula (1):

[0014]

[0015] Where i represents n different pixel values, and H(c,w,h) reflects the amount of information in the dataset at position (c,w,h);

[0016] The predefined scenario complexity evaluation method is represented by formula (2):

[0017]

[0018] The value of β is in [0,1], and the maximum entropy max_entropy depends on the number of pixel values.

[0019] According to an embodiment of the present invention, the above-described hybrid strategy based on dynamic and static importance, under the control of scene complexity, utilizes the weight parameters of the pre-trained model and the channel feature map to calculate the weighted importance of each channel in the channel feature map, including:

[0020] The scene dataset is processed using a pre-trained model to obtain a channel feature map dataset, and the dynamic importance of each channel in the channel feature map dataset is calculated using a predefined dynamic importance calculation method.

[0021] The weight parameters of the pre-trained model are processed using a predefined static importance calculation method to obtain the static importance of each channel in the channel feature map dataset;

[0022] Based on a predefined scaling factor, the importance ratio is obtained using scene complexity. The dynamic and static importance of each channel in the channel feature map dataset is then weighted using the importance ratio to obtain the weighted total importance of each channel in the channel feature map dataset.

[0023] According to an embodiment of the present invention, the predefined dynamic importance calculation method described above is represented by formula (3):

[0024]

[0025] Among them, X i,w,h Let (w, h) represent the position (w, h) of the i-th feature map in the c-th channel feature map dataset, where b represents the number of images in the scene dataset, and w l h represents the feature map width of the l-th convolutional layer in the pre-trained model. l This represents the feature map height of the l-th convolutional layer in the pre-trained model;

[0026] The static importance of meaning is calculated using formula (4):

[0027]

[0028] Where ||*||1 represents the first-order norm, W l c This represents the channel weight parameters corresponding to the l-th convolutional layer of the pre-trained model.

[0029] According to an embodiment of the present invention, the above-described transformation of the layer sparsity allocation problem of the pre-trained model into a single-objective multi-constraint knapsack problem by utilizing the weighted total importance of each channel in the channel feature map dataset, the solution of the single-objective multi-constraint knapsack problem using a multi-constraint dynamic programming algorithm, and the adaptive compression of the pre-trained model based on the solution results include:

[0030] The channel retention coefficient is calculated based on the number of convolutional layers in the pre-trained model, the number of images in the scene dataset, the order of the current convolutional layer in the pre-trained model, and predefined constant terms.

[0031] Based on the channel weighted total importance, the channels belonging to the current convolutional layer in the channel feature map dataset are sorted, and the sorted channels of the current convolutional layer are grouped according to the channel retention coefficient to obtain the channel grouping results of the current convolutional layer.

[0032] The channels in the channel feature map dataset are parsed into value, parameter quantity, and computational cost. The value, parameter quantity, and computational cost of all channels in the current convolutional layer are summed to obtain the value, parameter quantity, and computational cost of the current convolutional layer.

[0033] Repeatedly perform sorting, grouping, and summation operations to obtain the value, parameter count, and computational cost of all convolutional layers in the pre-trained model. Then, use the value of all convolutional layers in the pre-trained model as the objective function and the parameter count and computational cost of all convolutional layers in the pre-trained model as constraints to construct a single-objective multi-constraint knapsack problem.

[0034] According to an embodiment of the present invention, the above-described transformation of the layer sparsity allocation problem of the pre-trained model into a single-objective multi-constraint knapsack problem using the weighted total importance of each channel in the channel feature map dataset, the solution of the single-objective multi-constraint knapsack problem using a multi-constraint dynamic programming algorithm, and the adaptive compression of the pre-trained model based on the solution results further include:

[0035] The computation scaling factor is obtained based on the weight parameters of the pre-trained model, the predefined computation threshold, and the predefined computation scaling constant. The parameter scaling factor is obtained based on the weight parameters of the pre-trained model, the predefined parameter threshold, and the predefined parameter scaling constant.

[0036] The objective function and constraints of the single-objective multi-constraint knapsack problem are scaled according to the computational scaling factor and the parameter scaling factor, and the scaled single-objective multi-constraint knapsack problem is solved using a multi-constraint dynamic programming algorithm to obtain the solution result.

[0037] The upper bound of the pruning channel ratio is controlled by scene complexity, and the solution results are used to reduce the channels of the convolutional layer of the pre-trained model, thereby achieving adaptive compression of the pre-trained model.

[0038] According to a second aspect of the present invention, a scene-adaptive model compression system is provided, comprising:

[0039] The scene complexity evaluation module is used to calculate the pixel vector entropy of the scene dataset and, based on the pixel vector entropy, calculate the scene complexity of the scene dataset using a predefined scene complexity evaluation method. The scene dataset includes images.

[0040] The channel importance evaluation module is used to calculate the weighted total importance of each channel in the channel feature map dataset based on a hybrid strategy of dynamic importance and static importance, under the control of scene complexity, using the weight parameters of the pre-trained model and the channel feature map dataset. The channel feature map dataset is obtained by processing the scene dataset through the pre-trained model.

[0041] The layer sparsity allocation module is used to transform the layer sparsity allocation problem of the pre-trained model into a single-objective multi-constraint knapsack problem by utilizing the weighted total importance of each channel in the channel feature map dataset. The single-objective multi-constraint knapsack problem is solved using a multi-constraint dynamic programming algorithm, and the pre-trained model is adaptively compressed based on the solution results.

[0042] According to a third aspect of the present invention, an electronic device is provided, comprising:

[0043] One or more processors;

[0044] Storage device for storing one or more programs.

[0045] Among them, when one or more programs are executed by one or more processors, a model compression method that makes the execution scenario of one or more processors adaptive.

[0046] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform a scenario-adaptive model compression method.

[0047] The scenario-adaptive model compression method provided by this invention achieves model compression without fine-tuning in relevant application scenarios, removes a large number of redundant model parameters to accelerate model inference, and achieves almost no loss of accuracy. The scenario-adaptive model compression method provided by this invention significantly reduces the end-to-end overhead of model deployment, including the time cost and energy consumption caused by data annotation and fine-tuning. In addition, the scenario-adaptive model compression method provided by this invention allows pre-trained models to be quickly deployed on a large number of heterogeneous devices and adapts to different scenario data. Attached Figure Description

[0048] Figure 1 This is a flowchart of a scene-adaptive model compression method according to an embodiment of the present invention;

[0049] Figure 2 This is a scene-adaptive, fine-tuning-free model pruning framework diagram according to an embodiment of the present invention.

[0050] Figure 3 This is a pseudocode diagram illustrating the channel importance calculation method according to an embodiment of the present invention;

[0051] Figure 4 This is a pseudocode diagram illustrating the initialization process of a single-objective, multi-constraint knapsack problem according to an embodiment of the present invention;

[0052] Figure 5 This is a pseudocode diagram illustrating the solution process of the single-objective multi-constraint knapsack problem according to an embodiment of the present invention;

[0053] Figure 6 This is a schematic diagram of the structure of a scene-adaptive model compression system according to an embodiment of the present invention;

[0054] Figure 7 A block diagram of an electronic device suitable for implementing a scene-adaptive model compression method according to an embodiment of the present invention is shown schematically. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0056] Due to the limited computing power and scarce storage resources of terminal devices (such as smart cameras and embedded devices), the large-scale deployment of intelligent models to these devices necessitates pruning to remove redundant model parameters that are less important for inference. Existing technologies typically employ a three-step process: parameter importance assessment, pruning, and fine-tuning. This requires collecting labeled data and incurring significant fine-tuning overhead. Taking the large-scale deployment of intelligent models to smart cameras as an example, on the one hand, performing these operations individually for each camera incurs enormous computational and time costs, even with high-end GPUs. On the other hand, training a unified intelligent model using all available data struggles to meet the accuracy and computational performance requirements of different scenarios and devices. Therefore, this invention aims to provide a scene-adaptive, fine-tuning-free model pruning method to accelerate the end-to-end model compression process.

[0057] Unlike existing pruning methods based on input data, weight magnitude, automatic learning, and hardware structure, this invention focuses on designing a fine-tuning-free pruning scheme, proposing F2Zip, a scene-adaptive, fine-tuning-free model pruning framework. This invention proposes a scene complexity measurement method, using pixel vector entropy to quantify the degree of scene variation. By analyzing scene complexity, F2Zip adaptively evaluates the importance of different channels and layers of the model using only a small amount (dozens) of unlabeled data. Furthermore, this invention designs a multi-constraint knapsack solver to prune scene-independent redundant channels.

[0058] It should be noted that the acquisition of the scene data involved in the technical solution of this invention has been authorized by the data owner, and the above-mentioned scene data is processed, applied and stored with the permission of the data owner. The relevant process complies with the requirements of laws and regulations and public order and good morals, and necessary confidentiality measures have been taken.

[0059] The purpose of this invention is to reduce the computational overhead of intelligent models in specific scenarios while maintaining accuracy, and to enable the pruned intelligent models to be used without fine-tuning. Since fine-tuning of intelligent models in traditional solutions requires significant data annotation and training costs, the method provided by this invention can significantly reduce the time and energy consumption for deploying intelligent models.

[0060] Figure 1 This is a flowchart of a scene-adaptive model compression method according to an embodiment of the present invention.

[0061] like Figure 1 As shown, the above-mentioned adaptive model compression method includes operations S110 to S130.

[0062] In operation S110, the pixel vector entropy of the scene dataset is calculated, and based on the pixel vector entropy, the scene complexity of the scene dataset is calculated using a predefined scene complexity evaluation method, wherein the scene dataset includes images.

[0063] Scene datasets can be obtained through terminal devices (such as smart cameras or embedded devices). Scene datasets can contain images or videos. It should be noted that the acquisition, processing, application, and storage of the above-mentioned scene datasets have been authorized and permitted by the data owner.

[0064] In operation S120, based on a hybrid strategy of dynamic importance and static importance, under the control of scene complexity, the weighted total importance of each channel in the channel feature map dataset is calculated using the weight parameters of the pre-trained model and the channel feature map dataset. The channel feature map dataset is obtained by processing the scene dataset through the pre-trained model.

[0065] The above operation S120 aims to obtain the dynamic importance based on the scene dataset and the static importance based on the weight parameters of the pre-trained model, and to sum the two types of importance by weight to obtain the total importance of each channel.

[0066] By using a pre-trained model to process each image in the scene dataset, multiple channel feature maps of each image can be obtained, thus obtaining a channel feature map dataset.

[0067] In operation S130, the layer sparsity allocation problem of the pre-trained model is transformed into a single-objective multi-constraint knapsack problem by utilizing the weighted total importance of each channel in the channel feature map dataset. The single-objective multi-constraint knapsack problem is solved using a multi-constraint dynamic programming algorithm, and the pre-trained model is adaptively compressed based on the solution results.

[0068] Operation S130 aims to prune convolutional layers in the pre-trained model based on the weighted total importance of channels, removing convolutional layers that are not important to the accuracy of the results during the inference process of the pre-trained model. To achieve this objective, the layer sparsity allocation problem is transformed into a single-objective multi-constraint knapsack problem, which is solved using a multi-constraint dynamic programming algorithm. The solution result is used to prune the structure of the pre-trained model without requiring fine-tuning of the model structure.

[0069] The scenario-adaptive model compression method provided by this invention achieves model compression without fine-tuning in relevant application scenarios, removes a large number of redundant model parameters to accelerate model inference, and achieves almost no loss of accuracy. The scenario-adaptive model compression method provided by this invention significantly reduces the end-to-end overhead of model deployment, including the time cost and energy consumption caused by data annotation and fine-tuning. In addition, the scenario-adaptive model compression method provided by this invention allows pre-trained models to be quickly deployed on a large number of heterogeneous devices and adapts to different scenario data.

[0070] The adaptive model compression method for the above-mentioned scenario will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0071] Figure 2 This is a scene-adaptive, fine-tuning-free model pruning framework diagram according to an embodiment of the present invention.

[0072] F2Zip is a scene-adaptive, no-fine-tuning model pruning framework, such as... Figure 2 As shown, F2Zip consists of the following modules: a scene complexity evaluation module, which calculates the scene complexity β based on pixel vector entropy given the input scene dataset X; a channel importance evaluation module, which evaluates the importance of each channel based on the scene complexity β, combined with the importance of dynamic feature maps and static parameters; and a layer sparsity allocation module, which transforms the importance and cost of channels into values ​​and constraints of a single-objective multi-constraint knapsack problem, and solves it using a multi-constraint dynamic programming algorithm.

[0073] According to an embodiment of the present invention, the above-mentioned calculation of pixel vector entropy of the scene dataset and the calculation of scene complexity of the scene dataset based on pixel vector entropy using a predefined scene complexity evaluation method include: extracting pixels at specific pixel positions of all images in the scene dataset and flattening the extracted pixels into intermediate vectors; calculating the intermediate vectors using a predefined pixel vector entropy calculation method to obtain pixel vector entropy, wherein pixel vector entropy represents the amount of information of the image at a specific pixel position; repeating the extraction operation, flattening operation and calculation operation for different specific pixel positions of the images in the scene dataset to obtain multiple pixel vector entropies at different specific pixel positions; averaging and normalizing the multiple pixel vector entropies, and calculating the averaged and normalized multiple pixel vector entropies using a predefined scene complexity evaluation method to obtain the scene complexity of the scene dataset.

[0074] According to an embodiment of the present invention, the predefined pixel vector entropy calculation method described above is represented by formula (1):

[0075]

[0076] Where i represents n different pixel values, and H(c,w,h) reflects the amount of information in the dataset at position (c,w,h);

[0077] The predefined scenario complexity evaluation method is represented by formula (2):

[0078]

[0079] The value of β is in [0,1], and the maximum entropy max_entropy depends on the number of pixel values.

[0080] The following detailed description of the scenario complexity calculation method provided by the present invention, with reference to specific embodiments, will further illustrate this method.

[0081] Assessing scene complexity is a prerequisite for designing scene adaptive pruning. Although the data perceived in a moving scene has a narrower distribution than in a typical dataset, this narrowness affects the optimization space (e.g., a high-speed moving camera can perceive a very broad distribution containing various backgrounds and objects). Therefore, this invention first needs to assess the complexity of the scene dataset. Many existing studies have explored dataset complexity to estimate the learning difficulty of the dataset. However, most existing works focus on classification problems and do not consider dataset variations. For example, they assess the complexity of a single image based on grayscale image entropy and then take the average, without considering the information interaction between different images. This is not suitable for the problem that this invention aims to solve. In the scenario of this invention, it is actually necessary to calculate the degree of variation of the entire dataset (e.g., a fixed proportion of the background, the density of moving objects, etc.), thus requiring attention to the global information of the dataset. Therefore, this invention proposes a scene complexity assessment method based on pixel vector entropy. That is, given a scene dataset X∈R b*c0*w0*h0 Where b is the total number of images, c is the number of channels, and the size is w0*h0, this invention extracts the pixels of all images at position (c,w,h) and flattens them into a vector of size b. Then the pixel vector entropy can be defined as: Where i represents n different pixel values, and H(c,w,h) reflects the amount of information in the dataset at position (c,w,h). After averaging and normalizing the different pixel positions, this invention can calculate the scene complexity, defined as follows: The value of β is in [0,1], and the maximum entropy max_entropy depends on the number of pixel values. The scene complexity will directly affect the design of the subsequent module pruning strategy.

[0082] According to an embodiment of the present invention, the above-mentioned hybrid strategy based on dynamic importance and static importance, under the control of scene complexity, calculates the weighted importance of each channel in the channel feature map using the weight parameters of the pre-trained model and the channel feature map, including: processing the scene dataset using the pre-trained model to obtain the channel feature map dataset, and calculating the dynamic importance of each channel in the channel feature map dataset using a predefined dynamic importance calculation method; processing the weight parameters of the pre-trained model using a predefined static importance calculation method to obtain the static importance of each channel in the channel feature map dataset; obtaining the importance ratio based on the scene complexity using a predefined scaling factor, and using the importance ratio to perform a weighted calculation of the dynamic importance and static importance of each channel in the channel feature map dataset to obtain the weighted total importance of each channel in the channel feature map dataset.

[0083] According to an embodiment of the present invention, the predefined dynamic importance calculation method described above is represented by formula (3):

[0084]

[0085] Among them, X i,w,h Let (w, h) represent the position (w, h) of the i-th feature map in the c-th channel feature map dataset, where b represents the number of images in the scene dataset, and w l h represents the feature map width of the l-th convolutional layer in the pre-trained model. l This represents the feature map height of the l-th convolutional layer in the pre-trained model;

[0086] The static importance of meaning is calculated using formula (4):

[0087]

[0088] Where ||*||1 represents the first-order norm, W l c This represents the channel weight parameters corresponding to the l-th convolutional layer of the pre-trained model.

[0089] The following specific embodiments, in conjunction with the appendix, demonstrate this process. Figure 3 The method for calculating the importance of channels provided by the present invention will be described in further detail below.

[0090] Figure 3 This is a pseudocode diagram illustrating the channel importance calculation method according to an embodiment of the present invention.

[0091] Based on weight parameters and channel feature map output, this invention proposes a hybrid strategy of dynamic importance and static importance to evaluate the importance of each channel under the control of scene complexity.

[0092] Dynamic importance depends on the scene dataset and changes dynamically with the scene. Different channels in the same layer of a convolutional neural network are often designed to extract features of different patterns, and there may be a large number of redundant channels used to extract background features. However, background features usually have little impact on the accuracy of the detection model (or may even have a negative impact). For a specific scene, channels that tend to extract background features are considered to have lower importance, while channels that tend to extract target features are considered to have higher importance. In mobile vision scenes, the dataset perceived by smart cameras contains many similar or identical backgrounds, and the target is often moving and changing. Therefore, this invention determines the importance of channels by analyzing the changes in the channel feature maps of different input images. Specifically, for a feature map... Definition of dynamic importance: Among them, X i,w,h Let (w, h) represent the position (w, h) of the i-th feature map in the c-th channel feature map dataset, where b represents the number of images in the scene dataset, and w lh represents the feature map width of the l-th convolutional layer in the pre-trained model. l This represents the feature map height of the l-th convolutional layer in the pre-trained model.

[0093] Static importance can be used to examine the generalization ability of a model. Dynamic importance can reflect the trend of a channel extracting features. However, evaluating importance based solely on the data input may result in poor generalization ability of the pruned model, only fitting the data used for evaluation (because fine-tuning is not used to restore accuracy). To preserve a certain level of generalization ability of the pre-trained model, the importance of model parameters is also considered. In summary, this invention aims to retain channels that can both extract target features and have good generalization ability. Many techniques exist for measuring the importance of model parameters; this invention uses parameter norms to define parameter importance. For parameter weights W, static importance can be defined as: Where ||*||1 represents the first-order norm, W l c This represents the channel weight parameters corresponding to the l-th convolutional layer of the pre-trained model.

[0094] After obtaining dynamic and static importance, this invention adjusts the proportion of different importance levels based on scene complexity. Intuitively, if the scene complexity is low, the requirement for generalization ability can be reduced accordingly. Conversely, if the scene complexity is high (i.e., closer to a general dataset), then we need to retain more of the model's generalization ability. Based on the above design, this invention presents Algorithm 1, namely... Figure 3 The pseudocode shown is as follows: First, obtain the feature map and parameters during the forward propagation process; then, calculate the importance ratio α using the scene complexity; finally, calculate the two importance scores and weight them to obtain the total importance of each channel.

[0095] According to an embodiment of the present invention, the above-described transformation of the layer sparsity allocation problem of the pre-trained model by utilizing the weighted total importance of each channel in the channel feature map dataset into a single-objective multi-constraint knapsack problem, solving the single-objective multi-constraint knapsack problem using a multi-constraint dynamic programming algorithm, and adaptively compressing the pre-trained model based on the solution results includes: calculating the channel retention coefficient based on the number of convolutional layers in the pre-trained model, the number of images in the scene dataset, the order of the current convolutional layer in the pre-trained model, and predefined constant terms; sorting the channels belonging to the current convolutional layer in the channel feature map dataset according to the weighted total importance of the channels, and grouping the sorted channels of the current convolutional layer according to the channel retention coefficient to obtain the channel grouping result of the current convolutional layer; parsing the channels in the channel feature map dataset into value, parameter quantity, and computational quantity, and summing the value, parameter quantity, and computational quantity of all channels in the channel grouping result of the current convolutional layer to obtain the value, parameter quantity, and computational quantity of the current convolutional layer; repeating the process... The algorithm performs sorting, grouping, and summation operations to obtain the value, parameter count, and computational cost of all convolutional layers in the pre-trained model. It then constructs a single-objective, multi-constraint knapsack problem using the value of all convolutional layers as the objective function and the parameter count and computational cost as constraints. A computational scaling factor is obtained based on the weight parameters, predefined computational cost threshold, and predefined computational cost scaling constant of the pre-trained model. Similarly, a parameter scaling factor is obtained based on the weight parameters, predefined parameter cost threshold, and predefined parameter cost scaling constant. The objective function and constraints of the single-objective, multi-constraint knapsack problem are scaled using these scaling factors, and a multi-constraint dynamic programming algorithm is used to solve the scaled problem. Finally, scene complexity is used to control the upper bound of the pruning channel ratio, and the solution results are used to reduce the number of channels in the convolutional layers of the pre-trained model, thus achieving adaptive compression of the pre-trained model.

[0096] The following specific embodiments and appendices Figure 4 and 5 The solution process for the single-objective multi-constraint knapsack problem provided by the present invention will be further explained in detail.

[0097] Figure 4 This is a pseudocode diagram illustrating the initialization process of a single-objective, multi-constraint knapsack problem according to an embodiment of the present invention.

[0098] Figure 5 This is a pseudocode diagram illustrating the solution process of a single-objective, multi-constraint knapsack problem according to an embodiment of the present invention.

[0099] The layer sparsity allocation problem essentially involves removing a portion of channels from each layer based on their importance. Existing techniques uniformly (in the same proportion) remove the least important channels from each layer according to a given pruning rate. However, uniform pruning does not simultaneously consider the computational and storage constraints of the device in a specific scenario. Furthermore, some preliminary experiments show that uniform pruning only maintains accuracy when a low pruning rate is set.

[0100] Furthermore, model layers of different depths can extract features from different levels of the image. In shallower layers, the model extracts more general features, while in deeper layers, it extracts more task-related features. This means that for specific scenarios, more depth parameters can be removed without affecting accuracy. Based on existing research, this invention transforms the layer sparsity allocation problem into a single-objective, multi-constraint knapsack problem. Each channel is considered to have a value of I. l,c The number of parameters is The computational cost is The knapsack constraint consists of the total Param(M) and FLOPs(M) of the pruned model. The goal is to maximize the total value of the items put into the knapsack, while also maximizing the total value of the remaining passageways.

[0101] This paper utilizes a multi-constraint dynamic programming algorithm to solve the single-objective multi-constraint knapsack problem and optimizes it for practical applications. First, constraints and costs are scaled proportionally. The size of the dynamic programming transition matrix depends on the values ​​of the constraints. Due to the large computational cost and number of parameters in the model, constructing the original transition matrix is ​​impossible. Therefore, we scale the constraints and costs proportionally and use rounding to obtain approximate solutions. Second, channels are grouped according to importance. Channels in the same layer are grouped, and each group's channel is treated as a new item with its own sum and cost. The problem is updated to address the knapsack problem with new items, further reducing time complexity. Finally, the upper bound of the pruned channel ratio is controlled based on scene complexity. Unconstrained knapsack algorithms may result in all channels in a layer not being included in the knapsack, leading to incorrect reasoning in the pruned model. Therefore, we need to control the proportion of channels entering the knapsack algorithm. The upper bound of the pruned channels is adjusted using scene complexity (e.g., allowing more depth channels to be removed for low-complexity scenarios).

[0102] Based on the above design, this invention proposes Algorithm 2 (i.e. Figure 4 The pseudocode shown) and Algorithm 3 (i.e. Figure 5 (The pseudocode shown). In Figure 4 In this invention, the total number of channels to be allocated in each layer is calculated. After the channels are grouped, the corresponding channel importance, computational complexity, and parameter count are transformed into the values ​​v and cost W of a knapsack problem. F W P . Figure 5A dynamic programming solver for a multi-constraint knapsack problem is shown. By scaling the constraints, an approximate solution that satisfies the constraints as much as possible can be found in a relatively fast time.

[0103] In implementing the above-mentioned scene-adaptive model compression method provided by the present invention, the following steps can be followed: S1: Collect data for each scene, allocate equipment, and determine the compression ratio of computational load and parameter quantity; S2: Calculate scene complexity and evaluate channel importance based on the scene dataset; S3: Allocate pruning rate to each layer according to the compression ratio of computational load and parameter quantity; S4: Generate pruned model and deploy it according to the pruning strategy of S2-S3.

[0104] Figure 6 This is a schematic diagram of the structure of a scene-adaptive model compression system according to an embodiment of the present invention.

[0105] like Figure 6 As shown, the above-mentioned scene-adaptive model compression system includes a scene complexity evaluation module 610, a channel importance evaluation module 620, and a layer sparsity allocation module 630.

[0106] The scene complexity evaluation module 610 is used to calculate the pixel vector entropy of the scene dataset and, based on the pixel vector entropy, calculate the scene complexity of the scene dataset using a predefined scene complexity evaluation method. The scene dataset includes images.

[0107] The channel importance evaluation module 620 is used to calculate the weighted total importance of each channel in the channel feature map dataset using a hybrid strategy based on dynamic importance and static importance, under the control of scene complexity, by utilizing the weight parameters of the pre-trained model and the channel feature map dataset. The channel feature map dataset is obtained by processing the scene dataset through the pre-trained model.

[0108] The layer sparsity allocation module 630 is used to transform the layer sparsity allocation problem of the pre-trained model into a single-objective multi-constraint knapsack problem by utilizing the weighted total importance of each channel in the channel feature map dataset. The single-objective multi-constraint knapsack problem is solved using a multi-constraint dynamic programming algorithm, and the pre-trained model is adaptively compressed based on the solution results.

[0109] Figure 7 A block diagram of an electronic device suitable for implementing a scene-adaptive model compression method according to an embodiment of the present invention is shown schematically.

[0110] like Figure 7As shown, an electronic device 700 according to an embodiment of the present invention includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0111] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 702 and / or RAM 703. It should be noted that programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in one or more memories.

[0112] According to an embodiment of the present invention, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0113] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0114] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0116] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A scene-adaptive model compression method, characterized in that, include: Calculate the pixel vector entropy of the scene dataset, and based on the pixel vector entropy, calculate the scene complexity of the scene dataset using a predefined scene complexity evaluation method, wherein the scene dataset includes images; Based on a hybrid strategy of dynamic and static importance, under the control of the scene complexity, the weighted total importance of each channel in the channel feature map dataset is calculated using the weight parameters of the pre-trained model and the channel feature map dataset. The channel feature map dataset is obtained by processing the scene dataset through the pre-trained model. The layer sparsity allocation problem of the pre-trained model is transformed into a single-objective, multi-constraint knapsack problem by utilizing the weighted total importance of each channel in the channel feature map dataset. A multi-constraint dynamic programming algorithm is used to solve the single-objective, multi-constraint knapsack problem, and the pre-trained model is adaptively compressed based on the solution results. This includes: calculating channel retention coefficients based on the number of convolutional layers in the pre-trained model, the number of images in the scene dataset, the order of the current convolutional layer in the pre-trained model, and predefined constant terms; sorting the channels belonging to the current convolutional layer in the channel feature map dataset according to the weighted total importance of the channels; and further refining the current convolutional layer according to the channel retention coefficients. The channels after sorting are grouped to obtain the channel grouping result of the current convolutional layer; the channels in the channel feature map dataset are parsed into value, parameter, and computational cost, and the value, parameter, and computational cost of all channels in the current convolutional layer's channel grouping result are summed to obtain the value, parameter, and computational cost of the current convolutional layer; the sorting, grouping, and summing operations are repeated to obtain the value, parameter, and computational cost of all convolutional layers in the pre-trained model, and the value of all convolutional layers in the pre-trained model is used as the objective function, and the parameter and computational cost of all convolutional layers in the pre-trained model are used as constraints to construct the single-objective multi-constraint knapsack problem.

2. The method according to claim 1, characterized in that, Calculating the pixel vector entropy of the scene dataset, and based on the pixel vector entropy, calculating the scene complexity of the scene dataset using a predefined scene complexity evaluation method includes: Extract pixels at specific pixel locations from all images in the scene dataset, and flatten the extracted pixels into an intermediate vector; The intermediate vector is calculated using a predefined pixel vector entropy calculation method to obtain the pixel vector entropy, wherein the pixel vector entropy represents the amount of information in the image at a specific pixel location; For different specific pixel positions in the image of the scene dataset, the extraction, flattening and calculation operations are repeatedly performed to obtain multiple pixel vector entropies at different specific pixel positions. The entropy of multiple pixel vectors is averaged and normalized, and the scene complexity of the scene dataset is calculated by using the predefined scene complexity evaluation method after averaging and normalization.

3. The method according to claim 2, characterized in that, The predefined pixel vector entropy calculation method is represented by formula (1): （1）， in, represent Different pixel values, Reflects the location of the dataset The amount of information at the location; The predefined scenario complexity evaluation method is represented by formula (2): （2）， in, The value is in [0,1], and the maximum entropy is... It depends on the number of pixel values.

4. The method according to claim 1, characterized in that, Based on a hybrid strategy of dynamic and static importance, under the control of the scene complexity, the weighted importance of each channel in the channel feature map is calculated using the weight parameters of the pre-trained model and the channel feature map, including: The scene dataset is processed using the pre-trained model to obtain the channel feature map dataset, and the dynamic importance of each channel in the channel feature map dataset is calculated using a predefined dynamic importance calculation method. The weight parameters of the pre-trained model are processed using a predefined static importance calculation method to obtain the static importance of each channel in the channel feature map dataset; Based on a predefined scaling factor, the importance ratio is obtained using the scene complexity, and the dynamic and static importance of each channel in the channel feature map dataset is weighted using the importance ratio to obtain the weighted total importance of each channel in the channel feature map dataset.

5. The method according to claim 4, characterized in that, The predefined dynamic importance calculation method is expressed by formula (3): （3）， in, Indicates the first In the feature map dataset of the nth channel, the nth channel... Each feature map location , This indicates the number of images in the scene dataset. The first term of the pre-trained model is represented by the second term. The feature map width of each convolutional layer The first term of the pre-trained model is represented by the second term. Feature map height of each convolutional layer; The static importance calculation method for the meaning is expressed by formula (4): （4）， in, Denotes the first-order norm. The first term of the pre-trained model is represented by the second term. The channel weight parameters corresponding to each convolutional layer.

6. The method according to claim 1, further comprising: The computation scaling factor is obtained based on the weight parameters of the pre-trained model, the predefined computation threshold, and the predefined computation scaling constant; and the parameter scaling factor is obtained based on the weight parameters of the pre-trained model, the predefined parameter threshold, and the predefined parameter scaling constant. The objective function and constraints of the single-objective multi-constraint knapsack problem are scaled according to the computational scaling factor and the parameter scaling factor, and the scaled single-objective multi-constraint knapsack problem is solved using the multi-constraint dynamic programming algorithm to obtain the solution result. The upper bound of the pruning channel ratio is controlled by the scene complexity, and the channel reduction of the convolutional layer of the pre-trained model is performed using the solution result, thereby achieving adaptive compression of the pre-trained model.

7. A scene-adaptive model compression system, characterized in that, include: The scene complexity evaluation module is used to calculate the pixel vector entropy of the scene dataset, and based on the pixel vector entropy, calculate the scene complexity of the scene dataset using a predefined scene complexity evaluation method, wherein the scene dataset includes images; The channel importance evaluation module is used to calculate the weighted total importance of each channel in the channel feature map dataset based on a hybrid strategy of dynamic importance and static importance, under the control of the scene complexity, using the weight parameters of the pre-trained model and the channel feature map dataset. The channel feature map dataset is obtained by processing the scene dataset through the pre-trained model. The layer sparsity allocation module is used to transform the layer sparsity allocation problem of the pre-trained model into a single-objective multi-constraint knapsack problem by utilizing the weighted total importance of each channel in the channel feature map dataset, solve the single-objective multi-constraint knapsack problem using a multi-constraint dynamic programming algorithm, and adaptively compress the pre-trained model based on the solution results. Specifically, the layer sparsity allocation problem of the pre-trained model is transformed into a single-objective multi-constraint knapsack problem by utilizing the weighted total importance of each channel in the channel feature map dataset. A multi-constraint dynamic programming algorithm is then used to solve the single-objective multi-constraint knapsack problem, and the pre-trained model is adaptively compressed based on the solution results. The channel retention coefficient is calculated based on the number of convolutional layers in the pre-trained model, the number of images in the scene dataset, the order of the current convolutional layer in the pre-trained model, and predefined constant terms. Based on the channel weighted total importance, the channels belonging to the current convolutional layer in the channel feature map dataset are sorted, and the sorted channels of the current convolutional layer are grouped according to the channel retention coefficient to obtain the channel grouping result of the current convolutional layer; The channels in the channel feature map dataset are parsed into value, parameter, and computational cost. The value, parameter, and computational cost of all channels in the current convolutional layer are summed to obtain the value, parameter, and computational cost of the current convolutional layer. Repeatedly perform sorting, grouping, and summation operations to obtain the value, parameter count, and computational cost of all convolutional layers in the pre-trained model. Then, use the value of all convolutional layers in the pre-trained model as the objective function and the parameter count and computational cost of all convolutional layers in the pre-trained model as constraints to construct the single-objective multi-constraint knapsack problem.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

Images