A multimodal emotion recognition intelligent interaction system and performance optimization method

Abstract

This invention discloses a multimodal emotion recognition intelligent interaction system and its performance optimization method. The system adopts a hierarchical and progressive architecture consisting of a data acquisition layer, a preprocessing layer, an emotion recognition layer, a fusion and interaction layer, and a performance optimization layer. Multimodal data is simultaneously acquired via a camera and microphone. After preprocessing, an improved MobileNetV3 model and a CNN-LSTM model respectively complete image and speech emotion recognition. A confidence-based dynamic weight fusion algorithm outputs accurate emotion results. Personalized multi-turn interactions are achieved by combining a large language model and an emotion history queue. Full-process performance optimization is achieved through model quantization, pruning, multi-threaded parallelism, and hardware adaptation. This invention achieves an emotion recognition accuracy of over 93% and a processing latency of less than 80ms. It can run stably on embedded devices and is suitable for scenarios such as intelligent customer service, education, and rehabilitation, solving the problems of poor recognition, rigid interaction, and insufficient performance in existing technologies.

Images