Multi-iteration compression method for deep neural networks

A neural network, network technology, applied in neural learning methods, biological neural network models, neural architectures, etc., can solve problems such as large amount of calculation

Active Publication Date: 2018-02-09
XILINX TECH BEIJING LTD
View PDF5 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0030] Each gate has a weight matrix, and the input at time T and the output of T-1 pass through the gate with a large amount of calculation;

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Multi-iteration compression method for deep neural networks
  • Multi-iteration compression method for deep neural networks
  • Multi-iteration compression method for deep neural networks

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0058] Inventor's past research results

[0059] As in the inventor's previous article "Learning both weights and connections efficient neural networks", a method for compressing neural networks (eg, CNN) by pruning has been proposed. The method includes the following steps.

[0060] In the initialization step, the weights of the convolutional layer and the FC layer are initialized to random values, wherein a fully connected ANN is generated, and the connection has a weight parameter,

[0061] In the training step, the ANN is trained, and the weight of the ANN is adjusted according to the accuracy of the ANN until the accuracy reaches a predetermined standard. The training step adjusts the weight of the ANN based on the stochastic gradient descent algorithm, that is, randomly adjusts the weight value, and selects based on the accuracy change of the ANN. For an introduction to the stochastic gradient algorithm, see "Learning bothweights and connections for efficient neural ne...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The application discloses a multi-iteration compression method for a deep neural network in which weights among various neurons are represented by a plurality of matrices. The method includes a sensitivity analysis step of analyzing sensitivity of each of the plurality of matrices and determining the initial compression ratios of the respective matrices; a compression step of compressing the respective matrices based on the initial compression ratios to obtain a compressed neural network; and a recurrent training step for the recurrent training of the compressed neural network. The invention also discloses a device for compressing a neural network.

Description

[0001] This application claims priority to US Patent Application No. 15 / 242,622, filed August 22, 2016, and US Patent Application No. 15 / 242,624, filed August 22, 2016. field of invention [0002] The invention relates to a multiple iteration deep neural network compression method and device. Background technique [0003] Compression of Artificial Neural Networks [0004] Artificial Neural Networks (ANNs), also referred to as neural networks (NNs), is a mathematical computing model that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. In recent years, neural networks have developed rapidly and are widely used in many fields, such as image recognition, speech recognition, natural language processing, weather forecast, gene expression, content push and so on. [0005] In a neural network, there are a large number of nodes (also called "neurons") connected to each other. The neural network has two char...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06N3/04G06N3/08
CPCH04L63/10G06N3/04G06N3/063G06N3/08G10L15/063G10L15/16G06N3/045
Inventor 李鑫韩松孙世杰单羿
Owner XILINX TECH BEIJING LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap