Speech translation model training method and system based on model enhancement and speech translation method and device

Abstract

The invention discloses a speech translation model training method and system based on model enhancement and a speech translation method and device, and the training method comprises the steps of collecting a speech translation data set which is composed of a plurality of speech-translation-transcription triples; training a speech recognition model by using the speech-transcription data pair in the speech translation data set, and training a machine translation model by using the transcription-translation data pair in the speech translation data set; initializing a coding layer of the speech translation model by using the speech recognition model, and initializing a decoding layer of the speech translation model by using the machine translation model; masking the output of a hidden layer of the speech translation model, and training the speech translation model by using the speech translation data set in combination with a loss function; and after the speech translation model is trained, removing the mask, and performing fine tuning on the trained speech translation model. According to the invention, the recognition performance of the speech translation model is improved, and the speech translation efficiency and quality can be effectively improved.

Description

technical field [0001] The invention belongs to the technical field of speech translation, and in particular relates to a speech translation model training method and system based on model enhancement, and a speech translation method and device. Background technique [0002] Speech translation is an end-to-end model that converts speech in one language into text in another language, that is, skips the step of converting the source language into text in the traditional model, and directly converts the speech in the source language into the target Language text is a hot research topic at present. In terms of model selection, the Transformer model proposed by Google (A.Vaswani, N.Shazeer, N.Parmar, J.Uszkoreit, L.Jones, A.N.Gomez, L.Kaiser, and I.Polosukhin, "Attention is all you need, "in Proc.NIPS, California, USA, 2017, pp.5998–6008.) Due to the effectiveness of its self-attention mechanism modeling, the efficiency of parallel processing and the simplicity of the model stru...

AI Technical Summary

Problems solved by technology

[0005] Aiming at the problem that the existing speech translation model requires a large amount of data, the present invention proposes a speech translation model training method, system, and speech translation method and device based on ModAugment, by increasing the alignment difficulty of the encoding and decoding parts , transforming the overfitting problem into an underfitting problem, after a longer training period, an overall more robust model can be obtained, and better results can be achieved with a smaller hyperparameter adjustment burden

Images