Neural network training method and device, equipment and computer storage medium

Abstract

The embodiment of the invention discloses a neural network training method and device, equipment and a computer storage medium. The neural network training method comprises the steps of executing a loop process until soft labels meeting a preset number of anchoring samples are obtained; training a neural network at least based on the soft labels of the preset number of anchoring samples and the preset number of anchoring samples, wherein a current anchoring sample and at least one knowledge transfer sample are determined from the current training sample set in each period of executing the loop process; based on the neural network, determining the similarity between the current anchoring sample and each knowledge transfer sample, and the prediction probability of the current anchoring sample and the prediction probability of each knowledge transfer sample; and determining a soft label of the current anchoring sample based on the similarity between the current anchoring sample and each knowledge transfer sample, the prediction probability of the current anchoring sample and the prediction probability of each knowledge transfer sample. Knowledge integration under a self-distillation algorithm is realized.

Description

technical field [0001] The present disclosure relates to the field of deep learning, and in particular to a neural network training method, device, device, and computer storage medium. Background technique [0002] In recent years, edge devices such as mobile phones, wearable devices, etc., need to process deep learning-related tasks locally, but edge devices are generally limited by limited resources and power consumption, as well as delay and cost. In order to recommend the application of deep learning-based products on edge devices, a related technology proposes a model compression method known as Knowledge Distillation (KD). [0003] Specifically, the model compression method based on knowledge distillation is to transfer the reasoning and prediction ability of the trained more complex "teacher" model to the simpler "student" model, that is, the "soft label" predicted by the "teacher" model is used as training supervision , to guide the training of the "student" model, ...

AI Technical Summary

Problems solved by technology

[0004] However, in order to further obtain more accurate "soft labels" to improve the network performance of the "student" model, related technologies often use the knowledge integration dis

Images