Convolutional neural network compression method based on pruning and distillation

A convolutional neural network and compression method technology, which is applied in the field of convolutional neural network compression based on pruning and distillation, can solve the problems of reducing the computational load of large models, and the performance of small models is not ideal, and achieves the effect of improving performance.

Active Publication Date: 2018-07-27
XILINX INC
View PDF5 Cites 42 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

This type of method can improve the performance of small models and reduce the computational load of large models, but the performance of small models after distillation is often not ideal

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Convolutional neural network compression method based on pruning and distillation
  • Convolutional neural network compression method based on pruning and distillation
  • Convolutional neural network compression method based on pruning and distillation

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0019] The accompanying drawings are only for illustrative purposes and should not be construed as limiting the patent; the technical solutions of the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0020] To compress the convolutional neural network model, we first prune the network. figure 1 is a schematic diagram of pruning a convolutional neural network.

[0021] When pruning, we set a pruning criterion, and for the parameters of each layer of the model, the elements or modules that meet the pruning criterion are deleted. The purpose of pruning is to retain more important connections or elements.

[0022] When pruning the elements of the network, the pruning criterion is generally the size of the absolute value of the element. Elements with a large absolute value are retained, and elements with a small absolute value are set to 0. Another pruning method is to prune the convolution kernel of the network, ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The disclosure provides a convolutional neural network compression method (400) based on pruning and distillation. The method comprises the steps of performing pruning on an original convolutional neural network model to obtain a pruned model (S401); fine-tuning parameters of the pruned model (S403); using the original convolutional neural network model as a teacher network of a distillation algorithm and the pruned model with fine-tuned parameters as a student network of the distillation algorithm, and according to the distillation algorithm, using the teacher network to guide the student network to perform training (S405); and using the student network trained by the distillation algorithm as a compressed convolutional neural network model (S407). The method of the disclosure more effectively compresses the convolutional neural network model by using two conventional network compression methods in combination.

Description

technical field [0001] The present invention relates to a convolutional neural network, and more particularly to a convolutional neural network compression method based on pruning and distillation. Background technique [0002] Convolutional Neural Network (CNN) is currently widely used in various fields of computer vision, such as image recognition, object detection, image segmentation, etc. It is a research hotspot and key technology in the field of computer vision and multimedia, and has important research significance and practical value. Convolutional neural network is one of the representative network structures in deep learning technology. It has achieved great success in the field of image processing. On the international standard ImageNet dataset, many successful models are based on convolutional neural network. Compared with the traditional image processing algorithm, the convolutional neural network avoids the complex preprocessing process of the image (extractin...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06N3/04G06N3/08
CPCG06N3/04G06N3/082
Inventor 江帆单羿
Owner XILINX INC
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap