Transform algorithm-based single-mode label generation and multi-mode emotion discrimination method

Abstract

The invention discloses a single-mode label generation and multi-mode emotion discrimination method based on a Transform algorithm, and the method comprises the steps: 1, obtaining a multi-mode non-aligned data set, and carrying out the preprocessing of the multi-mode non-aligned data set, and obtaining an embedded expression feature of a corresponding mode; 2, establishing an ITE network module, and extracting intra-modal features; 3, single-mode label prediction and multi-mode emotion decision discrimination label fusion generation are carried out; 4, establishing an inter-modal BTE network module and a modal enhancement MTE network module, and obtaining inter-modal features and modal enhancement features through a global self-attention STE network module; and 5, obtaining a label of multi-modal emotion deep prediction. According to the method, for the condition that a current multi-modal data set only has one multi-modal label, decision fusion is carried out through a self-supervised weighted voting mechanism to generate a single-modal label, and based on the use of various cross-modal TE, data between modals are fully interacted, so that the precision of multi-modal emotion discrimination can be improved.

Description

technical field [0001] The present invention relates to time series one-dimensional convolutional neural network Conv1D, BiLSTM, Transformer self-attention mechanism and multimodal interactive attention mechanism, involves different fusion strategies of modalities, and realizes multimodal (voice, text, video) emotion Evaluation, and using a self-supervised mechanism with weighted voting, the prediction of single-modal labels and the final multi-modal emotional discrimination are realized, which belongs to the field of multi-modal multi-task emotional computing. Background technique [0002] With the advent of the big data era, the data content is complicated and the data forms are also extremely rich. Human cognition of a certain event is a response to combining multiple modal information perceptions. It is difficult to fully interpret information using only a single modality, especially the judgment of human emotion. For example, the frowning person said to the escort rob...

AI Technical Summary

Problems solved by technology

However, there is a strong correlation between unimodal and multimodal, and manual labeling of unimodal labels is time-consuming and laborious. Therefore, how to use self-supervised methods to learn unimodal feature representations from multimodal features and shared labels, It is of great significance for in-depth understanding of the expression of multimodal emotion

Images