Speech recognition text enhancement system fused with multi-modal semantic invariance

Abstract

The invention provides a speech recognition text enhancement system fused with multi-modal semantic invariance. The system comprises an acoustic feature extraction module, an acoustic downsampling module, an encoder and a decoder fused with multi-modal semantic invariance; A method comprises the following steps: performing framing processing on the voice data by the acoustic feature extraction module, dividing the voice data into short-time audio frames with fixed lengths, extracting acoustic features from the short-time audio frames, and inputting the acoustic features into the acoustic downsampling module for downsampling to obtain acoustic representation; inputting the voice data into an existing voice recognition module to obtain input text data, and inputting the input text data into an encoder to obtain input text encoding representation; and inputting the acoustic representation and the input text coding representation into a decoder for fusion, and carrying out similarity constraint on the representation of the acoustic mode and the representation of the text mode to obtain a decoding representation. According to the method, by fusing the cross-modal semantic invariance constraint loss, the dependence of the model on data is reduced, the performance of the model is improved, and the method is suitable for Chinese-English mixed speech recognition.

Description

technical field [0001] This application relates to the field of text enhancement for Chinese-English mixed speech recognition, in particular to a text enhancement system for speech recognition that integrates multi-modal semantic invariance. Background technique [0002] The phenomenon of Chinese-English mixing refers to switching languages ​​during the speaking process, mainly including two types of inter-sentence transitions and intra-sentence transitions. This phenomenon poses a huge challenge to speech recognition technology. The main problems are accent problems caused by non-standard pronunciation of speakers; more and more complex modeling units; collaborative pronunciation of different languages; difficulties in data collection; difficulties in data labeling, etc. With the development of deep learning technology, especially end-to-end models, monolingual speech recognition technology has been greatly improved. However, the end-to-end model can only use speech-text ...

AI Technical Summary

Problems solved by technology

At the same time, due to the powerful modeling ability of the neural network, the existing end-to-end speech recognition text enhancement system is very easy to overfit, requires a large amount of training data, and does not integrate the semantic similarity between the acoustic mode and the text mode

Unable to apply to Chinese-English mixed speech recognition problems, model training is difficult, and the error rate is high

Images