Mixed-update augmentation for a knowledge distillation framework
By mixing soft labels in the output of the teacher's neural network, simulated data is generated to train students' neural networks, the efficiency of speech recognition on edge devices and the complexity of information transmission in knowledge distillation is solved, and efficient speech recognition and knowledge transmission are achieved.
JP7649447B2Active Publication Date: 2025-03-21INTERNATIONAL BUSINESS MACHINE CORPORATION
1 Cites 0 Cited by
Patent Information
- Application Number
- JP2023534230
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-12-11
- Filing Date
- 2021-10-19
- Publication Date
- 2025-03-21
- Estimated Expiration
- 2041-10-19
AI Technical Summary
Technical Problem
The prior art is difficult to efficiently perform speech recognition on edge devices (such as smartphones, etc.), and knowledge distillation technology has problems of complexity and lack of information when transmitting teacher network information.
Method used
Mix-up augmentation technology is used to mix soft labels in the output of teacher neural networks to generate simulated data to train students' neural networks, thereby realizing knowledge distillation.
Benefits of technology
Through obfuscation enhancement technology, student neural networks can perform speech recognition on edge devices more efficiently, and can effectively convey the knowledge of teacher networks, improving the accuracy and efficiency of speech recognition.
✦ Generated by Eureka AI based on patent content.
Abstract
A method for training a student neural network is provided. The method includes feeding a dataset including a plurality of input vectors into a teacher neural network to generate a plurality of output values, and converting two of the plurality of output values from the teacher neural network for two corresponding input vectors into two corresponding soft labels. The method further includes combining the two corresponding input vectors to form a combined data vector and forming a masked soft label vector from the two corresponding soft labels. The method further includes feeding the combined data vector into the student neural network, using the masked soft label vector to determine errors for modifying weights of the student neural network, and modifying the weights of the student neural network.
Need to check novelty before this filing date? Find Prior Art