A model lightweight method based on slice down-sampling, device and storage medium
By constructing a lightweight model through slice downsampling and depthwise separable convolution, the problem of accuracy loss caused by model compression is solved, and a method for efficiently deploying deep learning models on edge computing devices is realized, thereby improving the training and inference speed of the model.
Patent Information
- Application Number
- CN202310002001.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-01-03
- Publication Date
- 2025-12-16
- Estimated Expiration
- 2043-01-03
AI Technical Summary
Existing model compression methods, while reducing the number of parameters in deep learning models, often result in a loss of model accuracy, making it difficult to effectively deploy and apply deep learning models on resource-constrained edge computing devices.
A slice-based downsampling method is adopted, which constructs a lightweight model through feature slicing and depthwise separable convolution to ensure the full inheritance of image feature information. The steps include image collection, data augmentation, non-nearest neighbor slice downsampling, and depthwise separable convolution. A slice downsampling module is constructed for feature extraction.
While significantly reducing the number of model parameters and computational cost, it maintains or slightly improves model accuracy, increases training and inference speed, and reduces device resource consumption and power consumption, making it suitable for resource-constrained edge computing devices.
Smart Images
Figure CN116090541B_ABST
Abstract
Description
Invention Field
[0001] This invention belongs to the field of computer science and involves fields such as artificial intelligence, computer vision, deep learning, and edge computing. Specifically, it relates to a method, device, and storage medium for lightweighting models based on slice downsampling. Background Technology
[0002] In the foreseeable future, the user data generated by the surge in mobile and IoT devices will grow exponentially. With the development of artificial intelligence technology, deep learning models are increasingly being applied to these network edge devices. However, deep learning models often require high-performance hardware, and resource-constrained edge computing servers struggle to meet the deployment environment requirements of large deep learning models. Therefore, deploying lightweight deep learning models on edge computing devices is gradually becoming a mainstream solution.
[0003] Researchers typically employ model compression methods such as pruning, quantization, knowledge distillation, and reparameterization to generate lightweight deep learning models. Pruning reduces the number of model parameters by removing unimportant weights; quantization simplifies high-precision 32-bit floating-point models into low-precision 8-bit fixed-point models for inference; knowledge distillation transfers knowledge learned from high-performance, large models to lightweight models for inference; and reparameterization simplifies multiple convolutional layers and batch normalization layers into a single convolutional layer to reduce redundant parameters. However, existing model compression methods often result in varying degrees of model accuracy loss while reducing the number of model parameters. Summary of the Invention
[0004] To address the aforementioned problems, this invention provides a model lightweighting method, device, and storage medium based on slice downsampling. According to a first aspect of the invention, a model lightweighting method based on slice downsampling is provided. This method first performs a downsampling operation on the model through feature slicing, followed by an appended depthwise separable convolution to extract high-level semantic information from the downsampled feature map layer. The method mainly includes the following steps:
[0005] Step 1, Image Collection and Data Augmentation. First, download or create a computer vision (CV) image dataset tailored to the specified task. Then, perform data augmentation operations on the CV image dataset using methods such as brightness transformation, scaling, rotation, masking, and mosaicking to increase the number and diversity of training samples, thereby improving the robustness of the deep learning model.
[0006] Step 2, non-neighbor slice-based down-sampling operation. The input layer of the deep learning model is set as a slice down-sampling layer to perform a down-sampling operation on the image data enhanced in step 1, and the down-sampling operation is performed in a non-neighbor slice-based down-sampling manner. The specific implementation is as follows: first, in each channel of the original image, all non-neighbor points with an interval of one pixel in the width dimension or the height dimension are divided into the same slice. For the R, G, and B channels of the original image, each channel down-sampling will obtain 4 non-overlapping slice subgraphs, and a total of C x 4 slice subgraphs will be generated (for a CV image, the number of channels C = 3). Then, all the subgraphs are spliced together according to the channel dimension to form a down-sampling feature mapping layer.
[0007] Step 3, constructing a slice down-sampling module. The down-sampling feature mapping layer obtained in step 2 is input into a depth separable convolution to construct a slice down-sampling module, which can enable the deep learning model to extract high-level semantic information with a lower number of parameters. The depth separable convolution is composed of a channel-by-channel convolution and a point-by-point convolution connected in series. The channel-by-channel convolution is a 3 x 3 convolution kernel with N in channels, and N in represents the number of channels of the input down-sampling feature mapping layer. The point-by-point convolution is composed of N out 1 x 1 convolution kernels with N in channels, and N out is the number of channels of the output feature mapping layer defined by the user. N in represents the number of channels of the input down-sampling feature mapping layer.
[0008] Step 4, building a lightweight deep learning model for a specified task. First, all down-sampling modules in the neural network use the slice down-sampling module constructed in step 3. The output of the down-sampling module will be input to the subsequent specified module (such as residual structure, cross-stage local network, and various convolution or Transformer modules) for feature extraction. Then, based on the slice down-sampling module constructed in step 3 and its subsequent specified module, N times (N is the number of down-sampling defined by the user) of down-sampling and multiple feature extraction operations are designed in the network architecture to complete the building of a lightweight deep learning model for a specified task.
[0009] Step 5, model lightweight effect verification. First, the image after data enhancement in step 1 is input into the deep learning model for the specified task built in step 4 for training. Then, the down-sampling layer in the neural network is replaced with a traditional convolution-based down-sampling layer (convolution kernel is 3x3, step is 2 convolution layer), and the image after data enhancement in step 1 is input into the deep learning model using the traditional convolution-based down-sampling layer for training for comparison experiment. Finally, based on the precision evaluation indexes widely used in the target detection field such as precision, recall and all-class average precision (mAP@0.5), the precision of the traditional down-sampling method and the method of the application is compared, and the parameter quantity and the calculation quantity are compared, and the whole process of the model lightweight method based on slice down-sampling is completed.
[0010] According to the second aspect of the application, a model lightweight device based on slice down-sampling is provided, which comprises a memory, a processor and a computer program stored in the memory and executable in the processor, wherein the processor executes the computer program to realize the steps of the model lightweight method based on slice down-sampling.
[0011] According to the third aspect of the application, a model lightweight storage medium based on slice down-sampling is provided, and the computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the steps of the model lightweight method based on slice down-sampling.
[0012] Compared with the prior art, the model lightweight method based on slice down-sampling ensures the full inheritance of image feature information in the down-sampling process, effectively maintains the model performance under the condition of significantly reducing the model parameter quantity, helps to improve the training and inference speed of the deep learning model, reduces the device resource occupation and reduces the power consumption. It provides a new reference and idea for the construction of lightweight deep learning model, and has important significance for deploying and applying deep learning model on resource-limited edge computing devices.
[0013] In addition, the application also provides a model lightweight device based on slice down-sampling and a model lightweight storage medium based on slice down-sampling, which can also realize the above technical effects, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS
[0014] Figure 1 The flowchart of the model lightweight method based on slice down-sampling of the application.
[0015] Figure 2 The schematic diagram of the down-sampling operation based on non-neighbor slice in the method of the application.
[0016] Figure 3This is a comparison chart of the prediction results of Embodiment 1 of the present invention on the COCO dataset. Figure 3 (a) Figure 3 (b) and Figure 3 (c) The prediction results of YOLOv5s, the prediction results of the method of this invention, and the ground truth values of the dataset are shown respectively.
[0017] Figure 4 This is a line graph showing the accuracy evaluation of Embodiment 1 of the present invention on the COCO dataset. Figure 4 (a) Figure 4 (b) and Figure 4 (c) respectively shows the precision, recall and average precision of the YOLOv5s method and the method of this invention on the COCO validation set during the model training process.
[0018] Figure 5 This is a comparison chart of the prediction results of Embodiment 2 of the present invention on the VOC dataset. Figure 5 (a) Figure 5 (b) and Figure 5 (c) The prediction results of YOLOv5s, the prediction results of the method of this invention, and the ground truth values of the dataset are shown respectively.
[0019] Figure 6 This is a line graph showing the accuracy evaluation of Embodiment 2 of the present invention on the VOC dataset. Figure 6 (a) Figure 6 (b) and Figure 6 (c) respectively shows the precision, recall and average precision of the YOLOv5s method and the method of this invention on the VOC validation set during the model training process. Detailed Implementation
[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings. (See attached drawings.) Figure 1 As shown, this invention provides a model lightweighting method based on slice downsampling.
[0021] In Example 1, this invention implements a lightweight end-to-end model based on slice downsampling on the typical object detection dataset MS COCO (Microsoft Common Objects in Context, or COCO for short) and the object detection model YOLOv5s, mainly including the following steps:
[0022] Step 1, image collection and data augmentation. First, download the image object detection dataset COCO. Then, perform data augmentation operations on the COCO dataset based on brightness transformation, scaling, rotation, masking, mosaicking, etc. to increase the number and diversity of training samples, thereby improving the robustness of the deep learning model.
[0023] Step 2, non-neighbor slice-based downsampling operation. Set the input layer of the YOLOv5s model to a slice downsampling layer for performing downsampling operations on the images augmented in step 1. The downsampling operation is performed using a non-neighbor slice-based downsampling method. The specific implementation is as follows: first, in each channel of the original image, all non-neighbor points with an interval of one pixel in the width or height dimension are divided into the same slice. For the R, G, and B channels of the original image, each channel downsampling will generate 4 non-overlapping slice subgraphs, a total of C x 4 slice subgraphs (for CV images, the number of channels C = 3). Then, all subgraphs are concatenated together according to the channel dimension to form a downsampling feature mapping layer. The non-neighbor slice-based downsampling operation diagram in the present method is shown in FIG. 1. Figure 2
[0024] Step 3, construct a slice downsampling module. Input the downsampling feature mapping layer obtained in step 2 into a depth separable convolution to construct a slice downsampling module, which can enable the deep learning model to extract high-level semantic information with a lower number of parameters. The depth separable convolution is composed of a channel-by-channel convolution and a point-by-point convolution connected in series. The channel-by-channel convolution is a 3 x 3 convolution kernel with a channel number of Nin, where Nin represents the number of channels of the input downsampling feature mapping layer. The point-by-point convolution is composed of Nout 1 x 1 convolution kernels with a channel number of Nin, where Nout is the number of channels of the output feature mapping layer defined by the user. Nin represents the number of channels of the input downsampling feature mapping layer.
[0025] Step 4, build a lightweight deep learning model for multi-class object detection task. First, replace all downsampling modules in the YOLOv5s network architecture with the slice downsampling module constructed in step 3. The output of the downsampling module will be input to the subsequent BottleNeckCSP and SPP modules for feature extraction. Finally, based on the slice downsampling module constructed in step 3 and its subsequent specified modules, design 5 times of downsampling and multiple times of feature extraction operations in the network architecture to complete the construction of a lightweight deep learning model for multi-class object detection task.
[0026] Step 5, model lightweight effect verification. First, the image after data enhancement in step 1 is input into the deep learning model for multi-class target detection task built in step 4 for training. Then, the down-sampling layer in the neural network is replaced with a traditional convolution-based down-sampling layer (convolution kernel is 3x3, step is 2 convolution layer), and the image after data enhancement in step 1 is input into the original YOLOv5s model using the traditional convolution-based down-sampling layer for training for comparison experiment. Finally, based on the three precision evaluation indexes widely used in the field of target detection, namely precision, recall and all-class average precision (mAP@0.5), the precision of the traditional down-sampling method and the method of the application is compared, and the parameter quantity and the calculation quantity are compared, to complete the whole process of a model lightweight method based on slice down-sampling.
[0027] Table 1 compares different down-sampling methods on the COCO dataset
[0028]
[0029] Table 1 compares the parameter quantity, calculation quantity and inference precision of the method of the application and the traditional convolution-based down-sampling method on the COCO dataset. From the table, it can be seen that YOLOv5s (using convolution-based down-sampling method) achieves detection precision of 65.96%, 51.67% and 55.62% in precision, recall and all-class average precision, respectively, and the method of the application based on slice down-sampling achieves detection precision of 64.47%, 52.38% and 55.71% in the above indexes, respectively. It is not difficult to see that compared with the traditional method, the method of the application achieves similar detection precision, and the precision in the most representative mAP@0.5 detection index is improved by nearly 0.1%. In addition, YOLOv5s has a model parameter quantity of 7.28MB and a calculation quantity of 17.1GFLOPS, while the method of the application has a parameter quantity of only 6.03MB and a calculation quantity of 14.0GFLOPS. Compared with the traditional method, the method of the application reduces the parameter quantity by 17.17% and the calculation quantity by 18.13%, which is more conducive to saving resource consumption on the device.
[0030] For the COCO dataset example image of embodiment 1 of the application, FIG. Figure 3 (a), FIG. Figure 3 (b) and FIG. Figure 3(c) respectively shows the prediction result graph of YOLOv5s, the prediction result graph of the method of the application and the label true value of the data set. As can be seen from the figure, for the target of the class bench, YOLOv5s and the method of the application both achieve good detection effect, and the confidence reaches 90%. For the target of fire hydront in the label, since its size is small and the similarity with person is large, YOLOv5s and the method of the application both incorrectly predict its class as person, but can correctly frame the outline of the target. And the position and size of the target frame predicted by YOLOv5s and the method of the application and the confidence are consistent, which shows similar prediction performance.
[0031] Figure 2 shows the prediction result of the method of the application on the VOC data set Figure 4 (a), Figure 2 shows the prediction result of the method of the application on the VOC data set Figure 4 (b) and Figure 2 shows the prediction result of the method of the application on the VOC data set Figure 4 (c) respectively shows the precision, recall and average precision of the YOLOv5s method and the method of the application on the COCO validation set during the model training process. The light-colored fold line represents the YOLOv5s method, and the dark-colored fold line represents the method of the application. As can be seen from the representative average precision fold line graph, during the model training process, the precision fold lines of the YOLOv5s method and the method of the application are basically consistent, and basically coincide in the first 100 training processes. After 100 rounds, the precision of the method of the application is slightly higher than that of the YOLOv5s method.
[0032] In embodiment 2, the application implements a model lightweight full link based on slice downsampling on a typical target detection data set PASCAL VOC (The PASCAL Visual Object Classes, referred to as VOC) and a target detection model YOLOv5s, mainly including the following steps:
[0033] Step 1, image collection and data enhancement. First, download the image target detection data set VOC. Then, based on brightness transformation, scaling, rotation, mask, inlay and other methods, perform data enhancement operations on the VOC data set to increase the number and diversity of training samples, and thus improve the robustness of the deep learning model.
[0034] Steps 2 to 5 adopt the same operation process as embodiment 1, only the data set used for training the neural network is different. The subsequent steps can refer to the related content of embodiment 1, which will not be described here.
[0035] Table 2 compares different downsampling methods on the VOC data set
[0036]
[0037] Table 2 compares the parameter quantity, computational quantity and inference accuracy of the VOC dataset of the method of the present application and the traditional convolution-based downsampling method. From the table, it can be seen that YOLOv5s (using a convolution-based downsampling method) achieves detection accuracies of 76.22%, 70.61% and 75.17% in the precision, recall and average precision per class indicators, respectively, and the model lightweight method based on slice downsampling of the present application achieves detection accuracies of 71.14%, 57.66% and 61.96% in the above indicators, respectively. It is not difficult to see that the method of the present application achieves detection accuracies very close to the traditional method in the Precision, Recall and mAP@0.5 indicators. In addition, YOLOv5s has a model parameter quantity of 7.11 MB and a computational quantity of 16.5 GFLOPS, while the method of the present application has a parameter quantity of only 5.87 MB and a computational quantity of 13.5 GFLOPS. Compared with the traditional method, the method of the present application reduces the parameter quantity by 17.44% and the computational quantity by 18.18%, which is more conducive to saving resource consumption on the device.
[0038] For the VOC dataset example image of Embodiment 2 of the present application, FIGS. Figure 5 (a), FIGS. Figure 5 (b) and FIGS. Figure 5 (c) respectively show the prediction result graph of YOLOv5s, the prediction result graph of the method of the present application and the label true value of the dataset. From the figures, it can be seen that for the car category target, YOLOv5s and the method of the present application both achieve good detection effects, and the confidence reaches 80%. The target box position and size predicted by YOLOv5s and the method of the present application both show consistent effects with the label true value, showing similar prediction performance.
[0039] FIGS. Figure 6 (a), FIGS. Figure 6 (b) and FIGS. Figure 6 (c) respectively show the precision, recall and average precision per class fold line of YOLOv5s method and the method of the present application on the VOC validation set during model training. The light-colored fold line represents the YOLOv5s method, and the dark-colored fold line represents the method of the present application. From the most representative average precision per class fold line graph, it can be seen that during the model training process, the precision fold lines of the YOLOv5s method and the method of the present application are basically consistent, and basically coincide in the first 100 training processes. After 100 rounds, the precision of the method of the present application is slightly lower than that of the YOLOv5s method in the Precision indicator, the precision of the method of the present application is slightly higher than that of the YOLOv5s method in the Recall indicator, and the precision of the method of the present application is basically consistent with that of the YOLOv5s method in the most representative mAP@0.5 indicator.
[0040] According to a second aspect of the present application, a model lightweight device based on slice down-sampling is provided, which mainly comprises a memory, a processor and a computer program stored in the memory and executable in the processor, wherein the processor implements the steps of the above-mentioned model lightweight method based on slice down-sampling when executing the computer program.
[0041] According to a third aspect of the present application, a model lightweight storage medium based on slice down-sampling is provided, wherein a computer program is stored in the computer readable storage medium, and the steps of the above-mentioned model lightweight method based on slice down-sampling are implemented when the computer program is executed by a processor.
[0042] In summary, the present application provides a model lightweight method, device and storage medium based on slice down-sampling, which can maintain or slightly improve the model accuracy while significantly reducing the parameter amount and calculation amount. Compared with the prior art, the above-mentioned model lightweight method based on slice down-sampling ensures the full inheritance of image feature information in the down-sampling process, effectively maintains the model performance while significantly reducing the model parameter amount, helps to improve the training and inference speed of the deep learning model, reduces the device resource occupation and reduces the power consumption. It provides a new reference and idea for the construction of lightweight deep learning model, and has important significance for deploying and applying the deep learning model on the resource-limited edge computing device.
[0043] The specific embodiments described herein are merely illustrative of the present application. Various modifications or changes in addition or substitution to the described specific embodiments can be made by those skilled in the art of the present application without departing from the spirit of the present application or exceeding the scope defined by the appended claims.
Claims
1. A model lightweighting method based on slice down-sampling, characterized in that, Comprising the following steps: Step 1, image collection and data enhancement; Step 2, based on the non-neighbor slice downsampling operation, the input layer of the deep learning model is set to the slice downsampling layer, which is used to perform the downsampling operation on the image data enhanced in step 1, and the downsampling operation is performed in a non-neighbor slice downsampling manner; the specific implementation is as follows: first, in each channel of the original image, all non-neighbor points with an interval of one pixel in the width or height dimension are divided into the same slice, for the R, G, B three channels of the original image, each channel downsampling will obtain 4 non-overlapping slice subgraphs, a total of Cx4 slice subgraphs, then all subgraphs are spliced together according to the channel dimension to form a downsampling feature mapping layer; Step 3, constructing a slice down-sampling module, inputting the down-sampling feature mapping layer obtained in step 2 into a depth separable convolution to construct a slice down-sampling module; wherein the depth separable convolution is jointly constituted by a channel-by-channel convolution and a point-by-point convolution connected in series, the channel-by-channel convolution is a 3x3 convolution kernel with a channel number N in , N in represents the channel number of the input down-sampling feature mapping layer, the point-by-point convolution is composed of N out 1x1 convolution kernels with a channel number N in , N out is a self-defined channel number of the output feature mapping layer, and N in represents the channel number of the input down-sampling feature mapping layer; Step 4, building a lightweight deep learning model for a specified task; first, all the downsampling modules in the neural network use the slice downsampling module constructed in step 3, and the output result of the downsampling module will be input to the subsequent feature extraction module, then based on the slice downsampling module and the subsequent feature extraction module constructed in step 3, N times of downsampling and multiple times of feature extraction operations are designed in the network architecture to complete the building of the lightweight deep learning model for the specified task.
2. The model lightweighting method based on slice down-sampling of claim 1, wherein: In step 1, the image is enhanced by methods based on brightness transformation, scaling, rotation, mask, and inlay.
3. The model lightweighting method based on slice down-sampling of claim 1, wherein: Also comprising step 5, model lightweight effect verification, the specific implementation is as follows: First, input the image data enhanced in step 1 into the deep learning model for a specified task built in step 4 for training, then replace the downsampling layer in the deep learning model with a traditional convolution-based downsampling layer, and input the image data enhanced in step 1 into the deep learning model using the traditional convolution-based downsampling layer for training for comparison experiment, finally, based on the precision, recall and average precision evaluation indexes widely used in the field of object detection, the precision of the traditional downsampling method and the method in steps 1-4 are compared, and the parameter quantity and calculation quantity are compared.
4. The model lightweighting method based on slice down-sampling of claim 3, wherein: The traditional convolution-based downsampling layer is a convolution layer with a 3x3 convolution kernel and a step of 2.
5. The model lightweighting method based on slice down-sampling of claim 1, wherein: The neural network in step 4 is a YOLOv5s model.
6. The model lightweighting method based on slice down-sampling of claim 1, wherein: In step 1, the images are collected from the data set MS COCO or PASCAL VOC.
7. A model light-weighting device based on slice down-sampling, comprising: Comprising: A memory, a processor, and a computer program stored in the memory and executable in the processor, wherein the processor executes the computer program to implement the steps of the model lightweight method based on slice downsampling according to any one of claims 1 to 4.
8. A computer-readable storage medium, characterized in that: The computer program is stored on the computer readable storage medium, and the computer program is executed by the processor to implement the steps of the model lightweight method based on slice downsampling according to any one of claims 1 to 4.