Speech Emotion Recognition Method Based on Multimodal Feature Extraction and Fusion

Abstract

The invention discloses a speech emotion recognition method based on multimodal feature extraction and fusion. First, the eGeMAPS acoustic feature set is extracted from the input audio. The feature set defines 88 basic speech emotion analysis features and standardizes the low-level The operation method of hierarchical acoustic features solves the problem of poor interpretability; secondly, the present invention provides a method for extracting user features and audio content text features, and uses hierarchical self-attention to combine user features and text features with acoustic features Carry out fusion, make full use of the multi-modal information in the corpus database, carry out more comprehensive modeling to speech emotion, thus realize speech emotion recognition more effectively; Finally, the present invention utilizes a kind of hierarchical self-attention mechanism to The in-depth fusion of modal features and the alignment of features of different modalities in the same high-dimensional semantic space enhance the ability to express emotions, thereby improving the accuracy of speech emotion recognition.

Description

technical field [0001] The invention relates to the field of speech recognition, in particular to a speech emotion recognition method based on multimodal feature extraction and fusion. Background technique [0002] With the rapid development of information technology and the popularization of various smart devices, speech recognition is no longer limited to the simple recognition of a single phoneme and sentence in the voice of a specific speaker, such as remote call voice, children's voice, low-resource voice, The effective recognition of the voices of speech-impaired persons and emotional voices has also become some new hotspots in the field of speech recognition. Speech contains not only the text information that the speaker wants to express, but also the emotional information that the speaker wants to express. Effective recognition of emotion in speech can improve the intelligibility of speech. Therefore, some researchers are trying to develop human nature with human thi...

AI Technical Summary

Problems solved by technology

[0006] Aiming at the disadvantage that the existing speech emotion recognition method is only based on the feature extraction of a certain aspect of the audio, and cannot make the classification model comprehensively identify the emotion according to the characteristics of the audio, the present invention discloses a speech emotion recognition method based on multi-modal feature extraction and fusion. The recognition method first extracts the eGeMAPS acoustic feature set from the input audio, which defines 88 basic speech emotion analysis features, and standardizes the operation method for low-level acoustic features to solve the problem of poor interpretability; secondly, The invention provides a method for extracting user features and audio content text features, and uses hierarchical self-attention to fuse user features, text features and acoustic features, fully utilizes the multimodal information in the corpus database, and extracts voice emotion More comprehensive modeling for more effective speech emotion recognition

Images