Speech recognition method and system for dysarthria based on visual facial contour movement

Abstract

The invention discloses a method and system for speech recognition of dysarthria based on visual facial contour movement. The system includes multi-modal data acquisition, multi-modal fusion feature calculation, multi-modal speech recognition calculation and language model calculation modules; multi-modal The state data acquisition calculation module is used to obtain the facial contour motion video data of the dysarthria and the voice data synchronized with the video; the multi-modal fusion feature calculation module is used to fuse the facial contour motion features and speech acoustic features; multi-modal speech recognition The calculation module is used to obtain the mapping relationship from multimodal features to phoneme characters; the language model calculation module is used to obtain the mapping relationship from phoneme characters to Chinese sentences. The present invention obtains the fused multimodal features by fusing the speech acoustic feature parameters and the pronunciation actions of the dysarthria, and utilizes the fused multimodal features to perform dysarthria speech recognition, thereby effectively improving dysarthria speech recognition Accuracy.

Description

technical field [0001] The invention relates to the technical field of speech recognition for dysarthria, in particular to a method and a system for enhancing speech recognition for dysarthria based on the movement trajectory of visual facial contours, which can be applied to assist speech rehabilitation of dysarthria. Background technique [0002] Existing research shows that the proportion of dysarthria caused by stroke is 30%-40%, of which 15% of dysarthria caused by stroke cannot be completely recovered. The dysarthria seriously affects the speaker's ability to communicate, resulting in a decline in the quality of life, which brings both physical and psychological pain to the dysarthria. Therefore, improving the intelligibility of dysarthria speech and effectively improving the speech recognition ability of dysarthria has important social significance and practical value. [0003] At present, the mainstream practice in the field of dysarthria speech recognition technolo...

AI Technical Summary

Problems solved by technology

However, because dysarthria speech has many defects such as difficulty in collecting and confusion of vowels and consonants, simply using speech acoustic feature parameters as the basis for speech recognition, the accuracy of dysarthria speech recognition is not high, and it is difficult to meet the needs of speakers Needs in actual communication application scenarios

Images