Personality detection method based on multi-modal alignment and multi-vector representation

Abstract

The invention discloses a personality detection method based on multi-modal alignment and multi-vector representation. The personality detection method comprises the following steps: resampling voiceand video modal data according to each epoch; inputting the plurality of samples and the text modal data thereof into an intra-modal representation module for independent coding to obtain a voice sequence, a video sequence and a text sequence; inputting the voice sequence, the video sequence and the text sequence into an inter-modal alignment characterization module, and splicing after pairwise alignment and interaction to obtain enhanced voice characterization, video characterization and text characterization; respectively splicing all the voice representations, all the video representationsand all the text representations to obtain a voice vector, a video vector and a text vector, and inputting the voice vector, the video vector and the text vector into a convolutional neural network tobe converted into at least two types of personality vectors; and respectively linearizing the at least two types of personality vectors, and mapping through a sigmoid function to obtain prediction probabilities of the at least two types of personality characteristics. Modal representation is enhanced through pairwise interaction of the three modal data, the discrimination capability of the modelis improved, and a more accurate prediction result is obtained.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a personality detection method based on multi-modal alignment and multi-vector characterization. Background technique [0002] Some people use the two modal data of voice and video to predict the personality of characters. Specifically, the video and voice spectrum of a certain number of frames are obtained by random sampling of the original video. For each frame, the residual network is used to extract the features of the video, and the Fourier transform is used to extract the MFCC features of the speech spectrum. The video features of each frame and the MFCC features of the audio are spliced, and input into a multi-layer bidirectional LSTM network to jointly encode the video and audio features. After that, for the encoded vector of each frame, it is input into the linear layer, and the sigmoid function is used for regression. Finally, average pooling is used to obtain a 5-dimen...

AI Technical Summary

Problems solved by technology

Furthermore, the coding between different modalities in the prior art is independent, which limits the expressive ability of the model

Secondly, in the existing technology, the same sample is sampled once before training, and the entire training process only repeats the video and audio of a few frames obtained after this sampling, which lacks the problem of data volume

Then, the existing technology only learns one vector representation for each sample, and uses this vector representation to perform 5 regression tasks, which cannot distinguish 5 types of personality well

Images