Multi-modal emotion recognition system suitable for uncertain modal deficiency

Abstract

The invention relates to a multi-mode emotion recognition system suitable for uncertain mode missing. Comprising a multi-modal information perception unit, a heterogeneous modal missing sample generation unit, a multi-granularity cross-modal interaction unit, a feature semantic consistency constraint unit, an adaptive feature enhancement unit, a selective feature filtering unit, a feature integration unit, an emotion recognition unit and an emotion display unit. Compared with the prior art, the method has the advantages that the natural correlation between elements in the missing mode can be deeply mined, and the potential capability of the missing mode can be motivated to the greatest extent, so that the robustness of the multi-mode emotion recognition method in the missing mode scene can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of multimodal emotion recognition, and in particular to a multimodal emotion recognition system suitable for uncertain modality loss. Background Art

[0002] With the increasing abundance of user-generated online content such as videos, multimodal sentiment analysis of speech has become a research hotspot. Different from traditional sentiment learning tasks performed on a single modality, multimodal sentiment analysis, as an important part of human-computer interaction behavior, aims to understand and interpret human emotions through multiple forms of human expressions (such as text, audio, and visual information), and is becoming a hot research field. Most multimodal sentiment analysis works are based on the assumption that all modalities are available in the training and testing phases. However, many unavoidable factors, such as privacy, equipment, or security constraints, may make the perception data incomplete, resulting in the failure of the assumption. Existing methods based on complete modality samples often give unsatisfactory results when faced with modality missing samples. In recent years, researchers have alleviated the impact of modality missing on multimodal emotion recognition to a certain extent by designing multimodal joint vector learning and missing feature reconstruction.

[0003] As an important part of human-computer interaction, multimodal emotion recognition aims to understand and interpret human emotions through multiple forms of human expressions, such as text, audio, and visual information. However, many unavoidable factors, such as privacy, equipment, or security constraints, may make the perceived data incomplete, which seriously affects the effect of multimodal recognition.

[0004] Current multimodal emotion recognition technologies based on missing modalities can be roughly divided into two categories: joint learning-based methods and reconstruction-based methods. Joint learning-based methods attempt to learn multimodal joint representations based on the relationship between different modalities. For example, Tran et al. discovered the widespread problem of modal missingness in multimodal data and proposed a cascaded residual autoencoder network to learn the complex relationship between different modalities. Zhao et al. proposed a Missing Modality Imagination Network (MMIN) to deal with the problem of uncertain modal missingness. The network learns a robust joint multimodal representation, which can predict the representation of any missing modality given the available modality under different missing modalities. Pham et al. proposed a method to learn multimodal joint representations by transforming the source modality to the target modality, while the cycle consistency loss is used to enhance modality translation. Zeng et al. proposed a label-assisted Transformer Encoder (TATE) network to deal with the problem of missing uncertain modalities, in which the label encoding module can cover both single-modal and multi-modal missing cases. Reconstruction-based methods learn to predict missing modalities with similar distributions to obey the distribution of available modalities. For example, Ma et al. proposed a learning strategy based on a meta-learning framework to solve the problem of severe modal missingness. Du et al. proposed a novel semi-supervised multi-view deep learning framework for multimodal emotion recognition under incomplete data.

[0005] Although multimodal sentiment analysis technology in uncertain modality-missing scenarios has made great progress, there are still some shortcomings and challenges that need to be addressed:

[0006] 1) Current research only performs insufficient interactions within samples and fails to transfer knowledge between different missing modal cases, resulting in an inability to fully utilize the potential of incomplete data to accurately reconstruct the semantics of the missing parts of the modality.

[0007] 2) The modality missing samples involved in the training process of current research are fixed, resulting in poor generalization and robustness of the model, which cannot be extended to complex and random situations in real-world scenarios.

[0008] 3) Most of the current studies only consider the pairwise interactions between independent modalities, while ignoring the complementary advantages of the three modalities. As a result, they are unable to effectively refine the semantics of the missing parts of the modality, nor can they make full use of the information of the available parts of the modality.

[0009] In summary, developing a multimodal emotion recognition system suitable for uncertain modality loss has become an urgent problem to be solved by technicians in this research field. Summary of the invention

[0010] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a multimodal emotion recognition system suitable for uncertain modality loss.

[0011] The purpose of the present invention can be achieved by the following technical solutions:

[0012] The present invention provides a multimodal emotion recognition system suitable for uncertain modality loss, comprising:

[0013] A multimodal information sensing unit, used for acquiring multimodal data;

[0014] A heterogeneous modality missing sample generation unit, used to generate two heterogeneous missing modality samples for each sample by randomly discarding intra-modality features and discarding complete modality between modalities;

[0015] A multi-granularity cross-modal interaction unit is used to deeply explore the natural relationships between elements of modalities by performing coarse-grained and fine-grained interactions between modalities to generate robust modal features;

[0016] The feature semantic consistency constraint unit is used to supervise the missing samples of two heterogeneous modalities to learn consistent modal features, fully explore the complementary semantics between modalities and effectively reconstruct the emotional semantics of the missing modality;

[0017] The adaptive feature enhancement unit fully models the incomplete modality to adaptively enhance the beneficial semantics in the incomplete modality and generate enhanced modality features;

[0018] A selective feature filtering unit is used to filter redundant information in incomplete modalities and generate refined modal features;

[0019] The feature integration unit is used to obtain multi-modal integrated features through modal feature fusion and temporal network modeling;

[0020] Emotion recognition unit, used to predict human emotion scores;

[0021] The emotion display unit is used to display the visual emotion recognition results.

[0022] Preferably, it is characterized in that the multimodal data includes a language modality obtained through pre-trained Glove word embedding processing, an audio modality obtained through the COVAREP tool, and a visual modality obtained through the Facet tool to complete facial action unit and intention information collection.

[0023] Preferably, it is characterized in that the heterogeneous modality missing sample generating unit specifically generates two heterogeneous missing modality samples for each sample by randomly discarding intra-modality features and discarding complete modality between modalities.

[0024] Preferably, it is characterized in that the random intra-modal feature discarding is specifically: discarding features on the modal sequence dimension with a random probability, and the random probability value is between 0.1 and 0.9.

[0025] Preferably, it is characterized in that the multi-granularity cross-modal interaction unit includes language, audio and visual feature sequence projection, language, audio and visual feature sequence alignment, coarse-grained cross-modal interaction, fine-grained cross-modal interaction, feature fusion and robust modal feature output stage.

[0026] Preferably, the feature semantic consistency constraint unit includes modal feature difference calculation and modal feature difference bridging stages, specifically: using the maximum average difference to achieve the measurement and calculation of the difference between modal feature distributions, and continuously bridging the difference between the modal features of two samples through continuous optimization of the model to achieve their semantic consistency supervision.

[0027] Preferably, the adaptive feature enhancement unit includes feature adaptive weight calculation, feature dynamic enhancement and modal enhancement feature output stages, specifically: the feature adaptive weight calculation adaptively learns the importance of the features of each modality in the time dimension by constructing a weight matrix of specified dimensions, then uses element multiplication to achieve feature dynamic enhancement, and finally generates enhanced modal features.

[0028] Preferably, the selective feature filtering unit includes redundant information location, redundant information filtering and modality refinement feature output stages.

[0029] Preferably, the feature integration unit includes modal feature fusion, temporal network modeling and multimodal integrated feature output stages.

[0030] Preferably, the emotion recognition unit of the continuous dimension is specifically: the multimodal integrated features are transformed into emotion score variables through a fully connected layer, and the mean square error loss is used to calculate the difference between the emotion score variable and the emotion label value to supervise the training of the model, and finally obtain the human emotion score prediction value.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] (1) Different from traditional modality missing methods that can only perform insufficient interactions within samples, this paper proposes a method for generating heterogeneous modality missing samples and supervises them to learn consistent feature distribution and semantic expression. By building a two-way knowledge transfer path between heterogeneous modality missing samples, the one-way dependency of features is effectively suppressed, and the potential of incomplete data can be fully explored and utilized, while restoring the missing information in the missing modality and strengthening the modality-specific features.

[0033] (2) Different from the fixed modality missing data in the traditional modality missing method, the present invention proposes a modality random missing strategy, which randomly performs intra-modality feature discarding and inter-modality entire modality discarding in an online manner during the training process, thereby simulating the random and complex modality missing conditions in real-world scenarios.

[0034] (3) Different from the traditional attention mechanism that only considers the pairwise interactions between independent modalities, the present invention proposes a multi-granularity cross-modal attention interaction unit. By performing coarse-grained and fine-grained cross-modal interactions, it deeply explores the natural correlations between elements in the missing modality and can maximize the potential capabilities of the missing modality. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A principle block diagram of a multimodal emotion recognition system applicable to uncertain modality loss provided by the present invention;

[0036] Figure 2 A principle block diagram of a multimodal information sensing unit provided by the present invention;

[0037] Figure 3 A principle block diagram of a heterogeneous modality missing sample generation unit provided by the present invention;

[0038] Figure 4 A principle block diagram of a multi-granularity cross-modal interaction unit provided by the present invention;

[0039] Figure 5 The principle block diagram of the feature semantic consistency constraint unit provided by the present invention;

[0040] Figure 6 A principle block diagram of the adaptive feature enhancement unit provided by the present invention;

[0041] Figure 7 A principle block diagram of a selective feature filtering unit provided by the present invention;

[0042] Figure 8 This is a principle block diagram of the characteristic integration unit provided by the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0045] Example

[0046] like Figure 1 As shown, the present invention constructs a multimodal emotion recognition system suitable for uncertain modal missing, including a multimodal information perception unit, a heterogeneous modal missing sample generation unit, a multi-granularity cross-modal interaction unit, a feature semantic consistency constraint unit, an adaptive feature enhancement unit, a selective feature filtering unit, a feature integration unit, a continuous dimension emotion recognition unit and an emotion display unit. Specifically, first, the multimodal information perception unit first automatically and in real time collects asynchronous multimodal sequences from humans, including language, audio and visual sequences. Feature embedding of different modalities is performed through preliminary feature preprocessing and modality-specific tools. Subsequently, two heterogeneous modal missing samples are generated for each sample by random intra-modal feature discarding and inter-modal complete modal discarding. Afterwards, the two heterogeneous modal missing samples will go through two branches in parallel: 1) The multi-granular cross-modal interaction unit realizes the mining of potential complementary relationships between elements of the missing modality, and the feature semantic consistency constraint unit performs consistency learning and constraints on the modal features of the heterogeneous modal missing samples; 2) The adaptive feature enhancement unit effectively models the missing modality to achieve adaptive enhancement of the beneficial emotional semantics, and the selective feature filtering unit retains the beneficial semantics while filtering the redundant information in the missing modality. Subsequently, all modal features obtained from the above two branches are integrated. Afterwards, the continuous dimension emotion recognition unit realizes the prediction of human emotion scores. Finally, the emotion recognition results can be presented through the emotion display unit.

[0047] Figure 2 The functional block diagram of the multimodal information perception unit provided by the present invention comprises three independent multimodal extraction units, namely language, audio and visual modes. In the language mode extraction unit, the system converts the transcript of the video into pre-trained Glove word embedding to obtain a 300-dimensional vector. In the audio mode extraction unit, the system uses the COVAREP toolkit to extract 74-dimensional low-level acoustic features, including 12 Mel-frequency cepstral coefficients, voiced sound segmentation features and glottal source parameters. In the visual mode extraction unit, the system uses Facet to represent 35 facial action units, obtains a 35-dimensional feature vector, and records facial muscle movements to express emotions. Through the above-mentioned feature extraction process, the system automatically completes the feature embedding of language, audio and vision.

[0048] Figure 3 This is a block diagram of the principle of the heterogeneous modality missing sample generation unit provided by the present invention, which includes two stages: intra-modality feature discarding and inter-modality complete modality discarding. First, the system discards the features on the modality sequence dimension with a random probability, and the random probability value is between 0.1 and 0.9; secondly, the system discards all features of certain modalities with a random probability, and the number of randomly discarded modalities is an integer between 0 and 2.

[0049] Figure 4 The principle block diagram of the multi-granularity cross-modal interaction unit provided by the present invention, the unit mainly includes the stages of language, audio and visual feature sequence projection, language, audio and visual feature sequence alignment, coarse-grained cross-modal interaction, fine-grained cross-modal interaction, feature fusion and modal Robin feature output. For multi-modal sequence projection, the system first completes the projection transformation of low-order features to high-order features for language, audio and visual sequences respectively through three independent fully connected layer models, so as to achieve better representation of modal features in the potential high-dimensional space. Afterwards, the system maps and aligns multi-modal feature sequences with different dimensions to the same feature dimension through three one-dimensional convolutional neural networks, where the convolution kernel size of text, audio and visual modalities is 3, and the dimension after alignment is 40. In the coarse-grained cross-modal interaction stage, one of the modalities is first projected into a specific target sequence space as an index query through a fully connected layer, where the number of layers of the fully connected layer is 1; then, the three modalities are concatenated in the sequence dimension, and they are projected into a specific source sequence space as index keys and index values ​​through two fully connected layers with different weights, where the number of layers of the two fully connected layers is 1; finally, the coarse-grained cross-modal interaction of target index query, source index key and source index value is performed to mine the deep correlation and complementary information between the missing modalities at a coarser granularity. In fine-grained cross-modal interaction, one of the modalities is first projected to a specific target sequence space as an index query through a fully connected layer, where the number of fully connected layers is 1; then, the other two modalities are independently projected to a specific source sequence space as index keys and index values ​​through two fully connected layers with different weights, where the number of layers of the two fully connected layers is 1; finally, the target index query and the fine-grained cross-modal interaction of the source index keys and source index values ​​of the other two modalities are executed in parallel to capture more refined inter-modal correlations at a finer granularity. In the feature fusion stage, the features obtained from the coarse-grained cross-modal interaction and the fine-grained cross-modal interaction are fused using element-level addition operations to finally obtain robust modal features.

[0050] Figure 5 This is a principle block diagram of the feature semantic consistency constraint unit provided by the present invention. The unit includes two stages: modal feature difference calculation and modal feature difference bridging. The system uses the maximum average difference with higher robustness as a non-parametric metric to measure and calculate the difference between two feature distributions. During the model optimization process, the model is guided to continuously narrow the difference between the two sample features through the two-way knowledge transfer between the missing sample features of the two heterogeneous modalities, so that the model enhances the available semantics and restores the missing semantics to produce more valuable modal features.

[0051] Figure 6This is a principle block diagram of the adaptive feature enhancement unit provided by the present invention. This unit includes three stages: feature adaptive weight calculation, feature dynamic enhancement, and modal enhancement feature output. The system first constructs an importance weight matrix of a specified dimension to adaptively learn the importance of each element of each modal feature in the time dimension; secondly, the normalization of feature weights in the sequence dimension is achieved through the Softmax function; then, the system performs element-wise multiplication on the modal features and the importance weight matrix to generate enhanced modal features.

[0052] Figure 7 This is a principle block diagram of the selective feature filtering unit provided by the present invention, which includes three stages: redundant information positioning, redundant information filtering, and modal refined feature output. The system calculates the contribution of each part of the feature to the judgment of the emotion recognition result, combines the preset modality-related threshold, realizes the positioning of feature redundant information, and implements the filtering of redundant information by replacing the zero vector to obtain the modal refined feature.

[0053] Figure 8 This is a functional block diagram of the feature integration unit provided in the present invention. The unit mainly includes three stages: modal feature fusion, temporal network modeling and multimodal integrated feature output. First, the system implements feature splicing operations on the features of the three modalities in the sequence dimension to perform joint feature modeling; secondly, the multimodal features are comprehensively and fully modeled using the temporal network Transformer to obtain multimodal integrated features, wherein the Transformer has 8 heads and 4 layers.

[0054] The continuous dimension emotion recognition unit provided by the present invention consists of three stages: multimodal integrated features, emotion score regression training and regression prediction output. The system transforms the multimodal integrated features into emotion score variables through a fully connected layer; then, the system uses mean square error loss to calculate the difference between the emotion score variable and the emotion label value to supervise the training of the model. Specifically, this difference is equal to the sum of the squares of the difference between the two; finally, the emotion prediction value is obtained.

[0055] After obtaining the emotion recognition result through the emotion recognition unit of the continuous dimension, the emotion recognition result is visually displayed and presented in the display unit.

[0056] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A multimodal emotion recognition system suitable for uncertain modality loss, characterized in that: include: A multimodal information sensing unit, used for acquiring multimodal data; A heterogeneous modality missing sample generation unit, used to generate two heterogeneous missing modality samples for each sample by randomly discarding intra-modality features and discarding complete modality between modalities; A multi-granularity cross-modal interaction unit is used to deeply explore the natural relationships between elements of modalities by performing coarse-grained and fine-grained interactions between modalities to generate robust modal features; feature The semantic consistency constraint unit is used to supervise the missing samples of two heterogeneous modalities to learn consistent modal features, fully explore the complementary semantics between modalities and effectively reconstruct the emotional semantics of the missing modality; The adaptive feature enhancement unit fully models the incomplete modality to adaptively enhance the beneficial semantics in the incomplete modality and generate enhanced modality features; A selective feature filtering unit is used to filter redundant information in incomplete modalities and generate refined modal features; The feature integration unit is used to obtain multi-modal integrated features through modal feature fusion and temporal network modeling; Emotion recognition unit, used to predict human emotion scores; The emotion display unit is used to display the visual emotion recognition results.

2. A multimodal emotion recognition system suitable for uncertain modality loss according to claim 1, characterized in that: The multimodal data includes the language modality obtained through the pre-trained Glove word embedding processing, the audio modality obtained through the COVAREP tool, and the visual modality obtained through the Facet tool to complete the facial action unit and intention information collection.

3. The multimodal emotion recognition system suitable for uncertain modality loss according to claim 1, characterized in that: The heterogeneous modality missing sample generating unit specifically generates two heterogeneous missing modality samples for each sample by randomly discarding intra-modality features and discarding complete modality between modalities.

4. A multimodal emotion recognition system suitable for uncertain modality loss according to claim 3, characterized in that: The random intra-modal feature discarding is specifically: discarding the features on the modal sequence dimension with a random probability, and the random probability value is between 0.1 and 0.

9.

5. The multimodal emotion recognition system suitable for uncertain modality loss according to claim 1, characterized in that: The multi-granularity cross-modal interaction unit includes language, audio and visual feature sequence projection, language, audio and visual feature sequence alignment, coarse-grained cross-modal interaction, fine-grained cross-modal interaction, feature fusion and robust modal feature output stages.

6. A multimodal emotion recognition system suitable for uncertain modality loss according to claim 1, characterized in that: The feature semantic consistency constraint unit includes modal feature difference calculation and modal feature difference bridging stages, specifically: using the maximum average difference to measure and calculate the difference between modal feature distributions, and continuously bridging the difference between the modal features of two samples through continuous optimization of the model to achieve semantic consistency supervision.

7. According to the multimodal emotion recognition system suitable for uncertain modal missing according to claim 1, the adaptive feature enhancement unit includes feature adaptive weight calculation, feature dynamic enhancement and modality enhancement feature output stages, specifically: the feature adaptive weight calculation adaptively learns the importance of each modality in the feature time dimension by constructing a weight matrix of specified dimensions, then uses element multiplication to realize feature dynamic enhancement, and finally generates enhanced modal features.

8. According to a multimodal emotion recognition system suitable for uncertain modality loss according to claim 1, the selective feature filtering unit includes redundant information positioning, redundant information filtering and modality refinement feature output stages.

9. According to the multimodal emotion recognition system suitable for uncertain modality loss described in claim 1, the feature integration unit includes modal feature fusion, temporal network modeling and multimodal integrated feature output stages.

10. According to the multimodal emotion recognition system suitable for uncertain modality loss described in claim 1, the emotion recognition unit of the continuous dimension is specifically: the multimodal integrated features are transformed into emotion score variables through a fully connected layer, and the mean square error loss is used to calculate the difference between the emotion score variable and the emotion label value to supervise the training of the model, and finally obtain the human emotion score prediction value.

Images