Visual image classification method and system based on pruning CNN
Through the sensitivity assessment and pruning of the CNN model's convolution nuclear, the depth CNN's computing complexity and large number of parameters on resource limited equipment were solved, and more efficient reasoning and storage were achieved.
Patent Information
- Application Number
- CN202411513729.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-28
- Publication Date
- 2025-01-28
AI Technical Summary
Existing deep convolutional neural networks (CNNs) have challenges in computing complexity, large parameters and high energy consumption, especially in resource-limited devices that are difficult to deploy and real-time inference.
Through the sensitivity assessment and pruning of the CNN model's convolution nuclear, reduce the unimportant weight and convolution kernel, thereby reducing the computing complexity and parameters of the model.
A significant reduction of the calculation complexity and parameters of the model, improving the efficiency and storage capacity of reasoning, and suitable for the environment with limited resources.
Smart Images
Figure CN119380098A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a visual image classification method and system based on pruned CNN, belonging to the technical field of computer vision. Background Art
[0002] In recent years, convolutional neural networks (CNNs) have been widely used in the field of computer vision, especially in image classification, object detection, semantic segmentation and other tasks. CNN effectively reduces the number of parameters through local perception mechanism and weight sharing, while capturing local features in the image, which makes it feasible to train CNN on large-scale datasets.
[0003] The convolution layer is the core of CNN, which extracts features from the input image through the convolution kernel, and as the depth of the network increases, CNN can extract more and more advanced features. However, as the depth and width of the network increase, the computational complexity and number of parameters of the CNN model also increase significantly, resulting in increasing overhead in computation and storage requirements. Especially on devices with limited resources such as mobile devices and embedded systems, traditional large CNN models are difficult to deploy, and often face the following challenges in practical applications:
[0004] High computational complexity: Deep convolutional neural networks usually require a large number of floating-point operations, which places strict requirements on real-time inference speed, especially in mobile and embedded devices, which cannot bear the burden of high computational complexity due to limited hardware resources.
[0005] Large number of model parameters: The depth and width of CNN models continue to increase, resulting in a continuous increase in the number of parameters. This not only increases the pressure on model storage, but also causes model overfitting, making it difficult to effectively train on small sample data sets.
[0006] High energy consumption: For large-scale CNN models, the inference process requires a lot of computing resources, which directly leads to high energy consumption. In some resource-sensitive scenarios (such as drones, IoT devices, etc.), high energy consumption has become a key bottleneck for deploying models. Summary of the invention
[0007] In order to solve the above problems existing in the prior art, the present invention proposes a visual image classification method and system based on pruned CNN.
[0008] The technical solution of the present invention is as follows:
[0009] On the one hand, the present invention provides a visual image classification method based on pruned CNN, comprising the following steps:
[0010] Construct a target classification image dataset and preprocess the target classification image dataset;
[0011] Construct a CNN-based visual image classification model, and train the visual image classification model using the preprocessed target classification image dataset to obtain an initial visual image classification model;
[0012] Perform sensitivity evaluation on all convolution kernels of the initial visual image classification model and obtain sensitivity evaluation results of all convolution kernels;
[0013] The convolution kernel is pruned according to the sensitivity evaluation result to obtain a pruned visual image classification model, and then the pruned visual image classification model is retrained and evaluated. If the evaluation passes, the final pruned visual image classification model is obtained, otherwise, the model is pruned again;
[0014] Perform image classification tasks through the final visual image classification model.
[0015] As a preferred embodiment of the present invention, the specific steps of sensitivity assessment are:
[0016] For each convolution kernel in any convolution layer, the weight importance evaluation and gradient importance evaluation are performed respectively, and the sensitivity of the current convolution kernel is obtained by weighted combination calculation, as shown in the following formula:
[0017] I k =α·I k，weight +(1-α)·I k，grad
[0018] Where: I k represents the sensitivity of the kth convolution kernel in the current convolution layer; α is a hyperparameter; I k，weight Represents the weight importance evaluation result of the kth convolution kernel in the current convolution layer; I k，grad Represents the gradient importance evaluation result of the kth convolution kernel in the current convolution layer.
[0019] As a preferred embodiment of the present invention, the specific steps of weight importance evaluation are:
[0020] According to the number of convolutional layers of the initial visual image classification model, select the L1 norm or L2 norm to evaluate the weight importance of the convolution kernel of the convolutional layer:
[0021] When the number of convolution kernels is greater than the preset threshold, the importance of the convolution kernel weight is evaluated by the L1 norm. For any convolution kernel of any convolution layer, there is:
[0022]
[0023] Where: ||W k||1 represents the weight importance evaluation result of the kth convolution kernel of the current convolution layer; i represents the height index of the convolution kernel; j represents the width index of the convolution kernel; l represents the input channel index of the convolution kernel; W k，i，j，l Represents the weight value of the kth convolution kernel of the current convolution layer at input channel l and position (i, j).
[0024] As a preferred embodiment of the present invention, when the number of convolution kernels is less than a preset threshold, the importance of the convolution kernel weight is evaluated by the L2 norm, and for any convolution kernel of any convolution layer, there is:
[0025]
[0026] Where: ||W k ||2 represents the weight importance evaluation result of the kth convolution kernel in the current convolution layer.
[0027] As a preferred embodiment of the present invention, the gradient importance evaluation calculation formula is shown as follows:
[0028]
[0029] Where: L represents the loss function of the initial visual image classification model.
[0030] As a preferred embodiment of the present invention, the specific steps of pruning are: presetting a pruning rate P, and pruning P% of the convolution kernels with the smallest weight values in each convolution layer.
[0031] As a preferred embodiment of the present invention, the evaluation steps of the pruned visual image classification model are:
[0032] Calculate the inference time FLOP of the pruned visual image classification model as shown below:
[0033]
[0034] Where: P v Indicates the pruning rate of the vth convolutional layer; FLOP v Indicates the number of floating-point operations of the vth convolutional layer;
[0035] Calculate the model size Size of the pruned visual image classification model as shown in the following formula:
[0036]
[0037] Where: Size v Represents the number of parameters of the vth convolutional layer;
[0038] Whether the pruned visual image classification model passes the evaluation is determined by whether the inference time of the pruned visual image classification model and the model size reach a preset threshold.
[0039] On the other hand, the present invention also provides a visual image classification system based on pruned CNN, comprising the following data acquisition module, initial classification model construction module, classification model evaluation module, classification model pruning module and image classification module:
[0040] The data acquisition module is used to construct a target classification image data set and preprocess the target classification image data set;
[0041] The initial classification model building module is used to build a visual image classification model based on CNN, and train the visual image classification model through the preprocessed target classification image data set to obtain an initial visual image classification model;
[0042] The classification model evaluation module is used to perform sensitivity evaluation on all convolution kernels of the initial visual image classification model and obtain sensitivity evaluation results of all convolution kernels;
[0043] The classification model pruning module is used to prune the convolution kernel according to the sensitivity evaluation result to obtain a pruned visual image classification model, and then retrain the pruned visual image classification model for evaluation. If the evaluation passes, the final pruned visual image classification model is obtained, otherwise, the model is pruned again;
[0044] The image classification module is used to perform image classification tasks through the final visual image classification model.
[0045] On the other hand, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any embodiment of the present invention when executing the program.
[0046] In yet another aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any embodiment of the present invention.
[0047] The present invention has the following beneficial effects:
[0048] 1. The present invention significantly reduces the computational complexity of the model by pruning unimportant weights and convolution kernels, which makes the model more efficient during reasoning and can better meet performance requirements in real-time applications, especially in resource-constrained environments.
[0049] 2. The present invention effectively reduces the number of model parameters and the storage requirements of the model during the pruning process; this enables the model to be deployed on devices with limited storage, reduces transmission and loading time, and improves the practicality of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION
[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0052] It should be understood that the step numbers used in this document are only for convenience of description and are not intended to limit the order in which the steps are executed.
[0053] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0054] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0055] The term "and / or" means and includes any and all possible combinations of one or more of the associated listed items.
[0056] Embodiment 1:
[0057] See also Figure 1 , a visual image classification method based on pruned CNN, comprising the following steps:
[0058] Construct a target classification image dataset and preprocess the target classification image dataset;
[0059] Construct a CNN-based visual image classification model, and train the visual image classification model using the preprocessed target classification image dataset to obtain an initial visual image classification model;
[0060] Perform sensitivity evaluation on all convolution kernels of the initial visual image classification model and obtain sensitivity evaluation results of all convolution kernels;
[0061] The convolution kernel is pruned according to the sensitivity evaluation result to obtain a pruned visual image classification model, and the pruned visual image classification model is retrained and evaluated. If the evaluation passes, the final pruned visual image classification model is obtained, otherwise, the model is pruned again;
[0062] Perform image classification tasks through the final visual image classification model.
[0063] In this embodiment, the target classification image dataset may be CIFAR-10, CIFAR-100 or ImageNet, etc.;
[0064] The preprocessing steps include data normalization: ensuring that the input image data is normalized to the same range, such as [0, 1] or [-1, 1].
[0065] Data enhancement: Use random cropping, flipping, rotation and other operations to improve the generalization ability of the model.
[0066] The initial visual image classification model can be:
[0067] ResNet: Residual network with good performance and scalability.
[0068] VGG: Simple and effective, suitable for structured pruning.
[0069] MobileNet: It is designed to be lightweight and suitable for mobile devices.
[0070] The model parameters are optimized through the cross entropy loss function:
[0071]
[0072] Where: N represents the number of samples in the target classification image dataset; C represents the number of categories; Represents the true label of the sample; is the predicted probability of the model;
[0073] When training the model, use optimizers such as SGD or Adam for optimization. The initial learning rate is 1e-3, which is gradually reduced according to the training situation. The early stopping method is used to prevent overfitting, or the learning rate decay is used to optimize the training process.
[0074] As a preferred implementation of this embodiment, the specific steps of the sensitivity assessment are:
[0075] For each convolution kernel in any convolution layer, the weight importance evaluation and gradient importance evaluation are performed respectively, and the sensitivity of the current convolution kernel is obtained by weighted combination calculation, as shown in the following formula:
[0076] I k =α·I k，weight +(1-α)·I k，grad
[0077] Where: I k represents the sensitivity of the kth convolution kernel in the current convolution layer; α is a hyperparameter; I k，weight Represents the weight importance evaluation result of the kth convolution kernel in the current convolution layer; I k，grad Represents the gradient importance evaluation result of the kth convolution kernel in the current convolution layer.
[0078] As a preferred implementation of this embodiment, the specific steps of weight importance evaluation are:
[0079] According to the number of convolutional layers of the initial visual image classification model, select the L1 norm or L2 norm to evaluate the weight importance of the convolution kernel of the convolutional layer:
[0080] When the number of convolution kernels is greater than the preset threshold, the importance of the convolution kernel weight is evaluated by the L1 norm. For any convolution kernel of any convolution layer, there is:
[0081]
[0082] Where: ||W k ||1 represents the weight importance evaluation result of the kth convolution kernel of the current convolution layer; i represents the height index of the convolution kernel; j represents the width index of the convolution kernel; l represents the input channel index of the convolution kernel; W k，i，j，l Represents the weight value of the kth convolution kernel of the current convolution layer at input channel l and position (i, j).
[0083] As a preferred implementation of this embodiment, when the number of convolution kernels is less than a preset threshold, the importance of the convolution kernel weight is evaluated by the L2 norm, and for any convolution kernel of any convolution layer, there is:
[0084]
[0085] Where: ||W k ||2 represents the weight importance evaluation result of the kth convolution kernel in the current convolution layer.
[0086] As a preferred implementation of this embodiment, the gradient importance evaluation calculation formula is shown as follows:
[0087]
[0088] Where: L represents the loss function of the initial visual image classification model;
[0089] As a preferred implementation of this embodiment, the specific steps of pruning are: presetting a pruning rate P, and pruning P% of the convolution kernels with the smallest weight values in each convolution layer.
[0090] As a preferred implementation of this embodiment, the evaluation steps of the pruned visual image classification model are:
[0091] Calculate the inference time FLOP of the pruned visual image classification model as shown below:
[0092]
[0093] Where: P v Indicates the pruning rate of the vth convolutional layer; FLOP v Indicates the number of floating-point operations of the vth convolutional layer;
[0094] Calculate the model size Size of the pruned visual image classification model as shown in the following formula:
[0095]
[0096] Where: Size v Represents the number of parameters of the vth convolutional layer;
[0097] Whether the pruned visual image classification model passes the evaluation is determined by whether the inference time of the pruned visual image classification model and the model size reach a preset threshold.
[0098] Embodiment 2:
[0099] A visual image classification system based on pruned CNN includes the following data acquisition module, initial classification model construction module, classification model evaluation module, classification model pruning module and image classification module:
[0100] The data acquisition module is used to construct a target classification image data set and preprocess the target classification image data set;
[0101] The initial classification model building module is used to build a visual image classification model based on CNN, and train the visual image classification model through the preprocessed target classification image data set to obtain an initial visual image classification model;
[0102] The classification model evaluation module is used to perform sensitivity evaluation on all convolution kernels of the initial visual image classification model and obtain sensitivity evaluation results of all convolution kernels;
[0103] The classification model pruning module is used to prune the convolution kernel according to the sensitivity evaluation result to obtain a pruned visual image classification model, and then retrain the pruned visual image classification model for evaluation. If the evaluation passes, the final pruned visual image classification model is obtained, otherwise, the model is pruned again;
[0104] The image classification module is used to perform image classification tasks through the final visual image classification model.
[0105] The system is used to implement the method in Example 1, which will not be described in detail here.
[0106] Embodiment three:
[0107] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any embodiment of the present invention when executing the program.
[0108] Embodiment 4:
[0109] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in any embodiment of the present invention is implemented.
[0110] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.
[0111] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0112] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0113] In several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), disk or optical disk, and other media that can store program codes.
[0114] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A visual image classification method based on pruned CNN, characterized in that: The following steps are involved: Construct a target classification image dataset and preprocess the target classification image dataset; Construct a CNN-based visual image classification model, and train the visual image classification model using the preprocessed target classification image dataset to obtain an initial visual image classification model; Perform sensitivity evaluation on all convolution kernels of the initial visual image classification model and obtain sensitivity evaluation results of all convolution kernels; The convolution kernel is pruned according to the sensitivity evaluation result to obtain a pruned visual image classification model, and then the pruned visual image classification model is retrained and evaluated. If the evaluation passes, the final pruned visual image classification model is obtained, otherwise, the model is pruned again; Perform image classification tasks through the final visual image classification model.
2. The visual image classification method based on pruned CNN according to claim 1, characterized in that: The specific steps of the sensitivity assessment are: For each convolution kernel in any convolution layer, the weight importance evaluation and gradient importance evaluation are performed respectively, and the sensitivity of the current convolution kernel is obtained by weighted combination calculation, as shown in the following formula: I k =α·I k，weight +(1-α)·I k，grad Where: I k represents the sensitivity of the kth convolution kernel in the current convolution layer; α is a hyperparameter; I k，weight Represents the weight importance evaluation result of the kth convolution kernel in the current convolution layer; I k，grad Represents the gradient importance evaluation result of the kth convolution kernel in the current convolution layer.
3. A visual image classification method based on pruned CNN according to claim 2, characterized in that: The specific steps of weight importance evaluation are: According to the number of convolutional layers of the initial visual image classification model, select the L1 norm or L2 norm to evaluate the weight importance of the convolution kernel of the convolutional layer: When the number of convolution kernels is greater than the preset threshold, the importance of the convolution kernel weight is evaluated by the L1 norm. For any convolution kernel of any convolution layer, there is: Where: ||W k ||1 represents the weight importance evaluation result of the kth convolution kernel of the current convolution layer; i represents the height index of the convolution kernel; j represents the width index of the convolution kernel; l represents the input channel index of the convolution kernel; W k，i，j，l Represents the weight value of the kth convolution kernel of the current convolution layer at input channel l and position (i, j).
4. The visual image classification method based on pruned CNN according to claim 3, characterized in that: When the number of convolution kernels is less than the preset threshold, the importance of the convolution kernel weight is evaluated by the L2 norm. For any convolution kernel of any convolution layer, there is: Where: ||W k ||2 represents the weight importance evaluation result of the kth convolution kernel in the current convolution layer.
5. The visual image classification method based on pruned CNN according to claim 3, characterized in that: The gradient importance evaluation calculation formula is as follows: Where: L represents the loss function of the initial visual image classification model.
6. The visual image classification method based on pruned CNN according to claim 3, characterized in that: The specific steps of pruning are: presetting a pruning rate P, and pruning P% of the convolution kernels with the smallest weight values in each convolution layer.
7. The visual image classification method based on pruned CNN according to claim 6, characterized in that: The evaluation steps of the pruned visual image classification model are: Calculate the inference time FLOP of the pruned visual image classification model as shown below: Where: P v Indicates the pruning rate of the vth convolutional layer; FLOP v Indicates the number of floating-point operations of the vth convolutional layer; Calculate the model size Size of the pruned visual image classification model as shown in the following formula: Where: Size v Represents the number of parameters of the vth convolutional layer; Whether the pruned visual image classification model passes the evaluation is determined by whether the inference time of the pruned visual image classification model and the model size reach a preset threshold.
8. A visual image classification system based on pruned CNN, characterized in that: It includes the following data acquisition module, initial classification model building module, classification model evaluation module, classification model pruning module and image classification module: The data acquisition module is used to construct a target classification image data set and preprocess the target classification image data set; The initial classification model building module is used to build a visual image classification model based on CNN, and train the visual image classification model through the preprocessed target classification image data set to obtain an initial visual image classification model; The classification model evaluation module is used to perform sensitivity evaluation on all convolution kernels of the initial visual image classification model and obtain sensitivity evaluation results of all convolution kernels; The classification model pruning module is used to prune the convolution kernel according to the sensitivity evaluation result to obtain a pruned visual image classification model, and then retrain the pruned visual image classification model for evaluation. If the evaluation passes, the final pruned visual image classification model is obtained, otherwise, the model is pruned again; The image classification module is used to perform image classification tasks through the final visual image classification model.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.