An AI vision-based multi-modal emotion recognition method

By employing a multimodal emotion recognition method, which utilizes feature extraction and fusion of facial images, speech, and text data, the robustness of single-modal emotion recognition is insufficient, thereby improving the stability and adaptability of emotion recognition results and making it suitable for complex interactive scenarios.

CN121834758BActive Publication Date: 2026-06-05HEFEI UNIV OF TECH
View PDF 3 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
HEFEI UNIV OF TECH
Filing Date
2026-03-11
Publication Date
2026-06-05

AI Technical Summary

Technical Problem

Existing emotion recognition technologies have shortcomings in terms of robustness, real-time performance, and cross-scenario adaptability. Single-modal methods are easily affected by external interference, resulting in unstable recognition results and weak cross-group generalization ability, making it difficult to meet the needs of highly reliable interactive systems.

Method used

We employ an AI-based vision-based multimodal emotion recognition method. By extracting and fusing multimodal features from users' facial image data, voice data, and text data, and combining a temporal alignment mechanism and a majority voting strategy, we output continuous emotion dimension values, thereby reducing the impact of single-modal anomalies.

Benefits of technology

It improves the stability and robustness of emotion recognition results, adapts to complex interaction scenarios, has the ability to track emotional states, enhances the reliability and adaptability of the system, and reduces deployment costs and real-time performance issues.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121834758B_ABST
    Figure CN121834758B_ABST
Patent Text Reader

Abstract

The application discloses a kind of multi-modal sentiment recognition methods based on AI vision, comprising the following steps: obtaining the facial image data of user, speech data and text data and pre-processing;Using HOG algorithm to the facial image data after pre-processing is carried out feature extraction, obtains visual emotional characteristics;TEO method is used to the speech data after pre-processing is carried out feature extraction, obtains speech emotional characteristics;Based on pre-training language model, the text data after pre-processing is carried out feature extraction, obtains text emotional characteristics;Timing alignment is carried out, and timing alignment multi-modal feature sequence is obtained;Fusion multi-modal emotional characteristics are obtained;Improved VA emotional model is used, and the sentiment recognition result is output;Emotional state change sequence is obtained, realizes the stable identification and continuous tracking to user emotional state, improves the robustness and timing consistency of emotional recognition.
Need to check novelty before this filing date? Find Prior Art