Machine learning program, machine learning method, and information processing device
By applying L1 regularization learning and threshold-based pruning, neural networks with attention structures are made more efficient by identifying minimal-impact channels for pruning, thus maintaining accuracy and reducing size effectively.
JP7885681B2Active Publication Date: 2026-07-07FUJITSU LTD
Patent Information
- Application Number
- JP2022212372
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-12-28
- Publication Date
- 2026-07-07
- Estimated Expiration
- 2042-12-28
Smart Images
Figure 0007885681000008 
Figure 0007885681000009 
Figure 0007885681000010
Abstract
To achieve weight reduction of a neural network including an attention structure.SOLUTION: A machine learning program causes a computer to execute processing of: for an element of each of a Q layer 161 outputting a Query and a K layer 162 outputting a Key as an arithmetic processing result on an input tensor in an attention structure in a trained machine learning model of a neural network 180 including an attention structure 160, deleting an element included in at least one of a tensor QT and a tensor KT such that elements having a same index are left in the tensors QT and KT from among one or more elements included in a tensor QT from a reduced Q layer in which the elements are reduced on the basis of a first reduction ratio and one or more elements included in a tensor KT from a reduced K layer in which the elements are reduced on the basis of a second reduction ratio.SELECTED DRAWING: Figure 19
Need to check novelty before this filing date? Find Prior Art
Citation Information
Patent Citations
Deep neural network optimization system for machine learning model scaling
US20220036194A1
Method for sparsification of feature maps in self-attention mechanisms
US20230028226A1