Multi-modal emotion recognition device, method, and storage medium using artificial intelligence

Abstract

A multi-modal emotion recognition system is disclosed. The system includes a data input unit for receiving video data and voice data of a user, a data pre-processing unit including a voice pre-processing unit for generating voice feature data from the voice data and a video pre-processing unit for generating one or more face feature data from the video data, a preliminary inference unit for generating situation determination data as to whether or not the user's situation changes according to a temporal sequence based on the video data. The system further comprises a main inference unit for generating at least one sub feature map based on the voice feature data or the face feature data, and inferring the user's emotion state based on the sub feature map and the situation determination data.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS[0001]Any and all applications for which a foreign or domestic priority claim is identified in the Application Data Sheet as filed with the present application are hereby incorporated by reference under 37 CFR 1.57.BACKGROUNDTechnical Field[0002]Embodiments of the present disclosure relate to multi-modal emotion recognition device, method, and storage medium using artificial intelligence.Related Art[0003]A conventional technology recognizes occlusion and processes it as an error. Covering a mouth with a hand is important information and can determine the intensity of the emotion state. Recognition information can be insufficient with the occlusion problem in a static image simply.[0004]In addition, when a target person talks upon recognizing emotion with face expression, a false emotion recognition result is derived. In the emotion recognition through expression recognition, a mouth shape is very important information, but the mouth shape often...

AI Technical Summary

Benefits of technology

The present patent provides a multi-modal emotion recognition device, method, and storage medium that can recognize emotion states using face expression, voice information, and mouth shape information. By tracking the face or mouth shape and using temporal information, the device can accurately recognize emotion states when in conversation or performing expression occlusion with the hand. The method also includes a method for minimizing mouth shape information and increasing the weight of voice feature information to minimize errors and noise. The device can infer the user's emotion state based on the sub feature map, situation determination data, and location inference feature map. Overall, the patent provides a more accurate way to recognize emotion states using multiple modalities.

Problems solved by technology

Recognition information can be insufficient with the occlusion problem in a static image simply.

In addition, when a target person talks upon recognizing emotion with face expression, a false emotion recognition result is derived.

In the emotion recognition through expression recognition, a mouth shape is very important information, but the mouth shape often changes upon talking, such that the mouth shape such as surprise, anger, and laugh can come out, resulting in a false recognition result.

Images