Multi-modal emotion recognition method based on micro-expressions, body movements and voices

A body movement and emotion recognition technology, applied in character and pattern recognition, speech analysis, instruments, etc., can solve problems such as inability to fully reflect human emotions anyway, and achieve high recognition rate, strong robustness, and good recognition effect Effect

Active Publication Date: 2021-10-01
INST OF AUTOMATION CHINESE ACAD OF SCI
View PDF5 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0009] However, in real life, human emotions are subtle and complex. These classification-base

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Multi-modal emotion recognition method based on micro-expressions, body movements and voices
  • Multi-modal emotion recognition method based on micro-expressions, body movements and voices
  • Multi-modal emotion recognition method based on micro-expressions, body movements and voices

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0063] Such as figure 1 and figure 2 As shown, the emotion recognition method based on micro-expressions and body movements provided by the embodiment of the present application is realized by a camera, a microphone and an emotion processing unit, and the method includes:

[0064] S1: Micro-expression recognition method,

[0065] The camera collects the facial video data of the subject for emotional analysis and sends it to the micro-expression recognition unit to obtain an emotion recognition result based on the micro-expression;

[0066] S2: Recognition method of body movements,

[0067] The camera collects the body movement video data of the emotional analysis subject and sends it to the body movement recognition unit to obtain the emotion recognition result based on the body movement;

[0068] S3: voice recognition method,

[0069] The microphone collects the voice signal of the subject for emotional analysis and transmits it to the voice emotion recognition module t...

Embodiment 2

[0107] Such as figure 2 As shown, the micro-expression recognition method:

[0108] S11: Cut out the facial area image, and take the peak frame of the macro-expression and micro-expression, that is, the peak frame with the largest movement range, as part A of an expression sample; extract the distance between the starting frame and the vertex frame of the macro-expression and micro-expression The optical flow characteristics of , as part B of the expression sample;

[0109] In some embodiments,

[0110] S111: using the Dlib library of OpenCV to detect 68 facial feature points for each macro-expression and micro-expression sample;

[0111] S112: Obtain the facial area image according to 68 key points of the face, cut out the facial area image, and take the peak frame of the macro-expression and micro-expression, that is, the peak frame with the largest movement range, as part A of a sample; the macro-expression data set Randomly rotate the samples by 0°, 90°, 180° or 270° t...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a multi-modal emotion recognition method based on micro-expressions, limb actions and voice, which comprises the following steps: step 1, inputting a facial video of a subject receiving stimulation of a certain signal, and recognizing the micro-expressions; 2, inputting a body video of a subject receiving stimulation of a certain signal, and identifying limb actions; and 3, inputting an audio signal stimulated by a certain signal received by the subject, and recognizing the voice emotion. The micro-expression recognition result in the first step, the limb action recognition result in the second step and the voice emotion recognition result in the third step are fused, and the continuous emotion state of the current subject is judged. According to the method, the emotion recognized by the micro-expression is combined with the emotion recognized by the limb action and the speech emotion, so that the emotional state of the subject can be predicted more accurately. Compared with the prior art, the method has the beneficial effects that the real emotion of a person can be identified more accurately.

Description

technical field [0001] The invention relates to the field of image processing and pattern recognition, in particular to a multimodal emotion recognition method based on micro-expressions, body movements and language. Background technique [0002] With the development of technology, computers have become an indispensable part of life. Human beings will have emotional ups and downs such as joy, anger, sorrow and joy anytime and anywhere. How to make computers understand human emotions has become a research hotspot. Scientists endow the computer system with the ability to observe, recognize, understand, express and generate various emotional expressions similar to humans, so that the computer system has higher and more comprehensive intelligence, enabling it to behave naturally, vividly and emotionally like humans. Communicate and interact cordially. Emotion recognition is necessary in many scenarios. For example, in the process of human-computer interaction, if an intellige...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06K9/00G06K9/62G10L25/63
CPCG10L25/63G06F18/254
Inventor 陶建华张昊刘斌连政
Owner INST OF AUTOMATION CHINESE ACAD OF SCI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap