Psychological stress analysis method of multimodal fusion

Abstract

The present invention provides a multi-modal fusion psychological stress analysis method, including: cutting long audio and video into short audio and video with faces and voices, and performing frame processing on the short audio and video to obtain image sequences and voice signals; Perform facial feature extraction on the image sequence to obtain a facial frame sequence; apply the optical flow method to extract optical flow from adjacent frames of the facial frame sequence to obtain an optical flow sequence; fuse the facial frame sequence and the optical flow sequence to perform linear Mapping to obtain the facial embedding vector; extracting the region of interest from the image sequence to obtain the sequence of interest, performing linear mapping to obtain the physiological signal embedding vector; extracting the basic acoustic features of the speech signal in units of frames, and performing linear mapping to obtain the acoustic embedding vector; Emotional features are extracted from speech signals and image sequences; the above features are fused according to the time sequence of the frame sequence to obtain a spatiotemporal feature vector; the spatiotemporal feature vector is input into the model, and then the psychological stress level is obtained through softmax classification.

Description

technical field [0001] This application relates to the field of emotion recognition, and in particular to a method for analyzing psychological stress with multimodal fusion. Background technique [0002] Facial features include facial expression features and facial movement characteristics. Facial expressions and facial movements revealed by people inadvertently can express people's true psychological state. There is a correlation between changes in micro-expressions such as eyebrows, mouth, eyes, and forehead and their psychological stress. At the same time, facial movements when people speak are also correlated with their psychological stress. [0003] Voice features mainly include voice prosody features, voice spectrum features, and voice quality features; voice prosody features are acoustically expressed as fundamental frequency, duration, and energy parameters; voice spectrum features include spectrum, spectrum envelope, cepstral coefficient, formant, etc.; Voice quali...

AI Technical Summary

Problems solved by technology

[0008] Existing technologies mainly model and analyze psychological pressure on facial expressions, voice, and physiological signals, and then make fusion decisions to obtain psychological pressure prediction results. There is no way and method to integrate them. A certain physiological signal needs to be integrated and considered in order to make an accurate judgment

Images