Neural network sparsification device and method and corresponding product

A neural network model and sparse technology, which is applied in the field of sparse training of neural network models, can solve the problems of high output overhead, many methods, and unfriendly hardware access memory, so as to improve accuracy and reduce input/output overhead. Effect

Pending Publication Date: 2022-05-06
ANHUI CAMBRICON INFORMATION TECH CO LTD
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Although the existing fine-grained parameter sparse method model performs well, it is not friendly to hardware memory access, that is, the on-chip and off-chip input / output overhead is large, and the performance is low; on the other hand, the structured sparse method based on channels and convolution kernels Although the method improves the hardware performance, the loss of model accuracy is large; finally, most of the existing sparse algorithms are offline fine-tuning, that is, the pre-training model is sparse and then fine-tuned. The offline fine-tuning method has many restrictions and cannot be used in model training There are more substantial performance gains

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Neural network sparsification device and method and corresponding product
  • Neural network sparsification device and method and corresponding product
  • Neural network sparsification device and method and corresponding product

Examples

Experimental program
Comparison scheme
Effect test

Embodiment approach 1301

[0106] Embodiment 1301 only has a mask adjustment stage. Both the parameter initial value W0 and the mask tensor initial value M0 are randomly generated by the random generation module 61, or the mask tensor initial value M0 is determined based on the parameter initial value W0, and the training parameters are updated at the same time. Mask matrix to obtain the trained parameters Wf and the updated mask tensor Mf.

Embodiment approach 1302

[0107] Embodiment 1302 has only an unmasked phase and a masked adjustment phase. In the unmasked stage, only the parameters are trained, and the parameter initial value W0 is randomly generated by the random generation module 61, and the updated parameter W1 is obtained after training. In the mask adjustment stage, the training parameters update the mask matrix at the same time. The initial value of the parameter in this stage is the updated parameter W1, and the initial value M0 of the mask tensor is randomly generated by the random generation module 61, or the updated parameter W1 is used to generate Obtain the initial value M0 of the mask tensor, and finally obtain the trained parameter Wf and the updated mask tensor Mf.

Embodiment approach 1303

[0108] Embodiment 1303 only has a mask adjustment phase and a mask fixation phase. In the mask adjustment stage, the parameter initial value W0 and the mask tensor initial value M0 are randomly generated by the random generation module 61, or the mask tensor initial value M0 is determined based on the parameter initial value W0, and the training parameters update the mask matrix at the same time, To obtain the updated parameter W1 and the updated mask tensor Mf. In the mask fixing stage, the training is continued with the updated mask tensor Mf mask parameters. The initial value of the parameters in this stage is the updated parameter W1, and finally the trained parameter Wf is obtained.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a device, a board card and a method for sparse training of a neural network model and a readable storage medium, and the processing device is included in an integrated circuit device, and the integrated circuit device comprises a universal interconnection interface and a computing device. And the computing device interacts with the processing device to jointly complete the computing operation specified by the user. The integrated circuit device can further comprise a storage device, and the storage device is connected with the computing device and the processing device and used for data storage of the computing device and the processing device.

Description

technical field [0001] The present disclosure relates generally to the field of neural networks. More specifically, the present disclosure relates to a device, a board, a method and a readable storage medium for sparse training of a neural network model. Background technique [0002] In recent years, with the rapid development of deep learning, the performance of algorithms in a series of fields such as computer vision and natural language processing has made leaps and bounds. However, the deep learning algorithm is a computing-intensive and storage-intensive tool. With the increasing complexity of information processing tasks, the requirements for real-time and accuracy of the algorithm continue to increase, and the neural network is often designed deeper and deeper, making Its computing power and storage space requirements are increasing, making it difficult for existing artificial intelligence technology based on deep learning to be directly applied to mobile phones, sat...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06N3/08G06N3/063
CPCG06N3/082G06N3/063G06N3/084G06N3/047G06N3/045G06F18/285
Inventor 不公告发明人
Owner ANHUI CAMBRICON INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap