Emotion recognition method based on low-rank multi-modal cutting, splicing and fusion

Abstract

The invention provides an emotion recognition method based on low-rank multi-mode cutting, splicing and fusion, and belongs to the field of emotion recognition. According to the method, firstly, image features are extracted through a VGG19 model, text features are extracted through a BERT model, and audio features are obtained in combination with MFCC and 1D-CNN models. Then, a cross-modal fusion module is used to perform deep interaction and fusion on the features of different modals to obtain a multi-modal comprehensive representation; and finally, combining the fusion representation with the original features through an LMSSF method, and generating final output of emotion recognition. The method has the advantages that complementary information among different modes is utilized, interaction inside the modes and interaction among the modes are deeply mined, and circulation and transmission of emotional semantic features are promoted. According to the method, the accuracy rate of 97.3% in the sentiment analysis task is achieved, the advanced performance in the industry is achieved, and the accuracy and robustness of the model in sentiment analysis are improved.

Description

Technical Field

[0001] The present invention belongs to the field of emotion recognition and relates to an emotion recognition method based on low-rank multimodal cutting, splicing and fusion. Background Art

[0002] With the popularity of social media and the rapid development of smart devices and human-computer interaction technology, the demand for emotion recognition technology continues to grow. Emotion recognition aims to accurately identify and understand human emotional states by analyzing multimodal data such as human language, images, and audio through computers. This technology has important value in many application fields, including user experience optimization, market research, and mental health monitoring. However, existing emotion recognition methods face many challenges in multimodal data processing. The correlations between data modalities such as images, texts, and audios are complex, and traditional single-modal emotion analysis usually cannot fully capture these correlations, resulting in insufficient accuracy and comprehensiveness of recognition. Under the current technical background, single-modal emotion recognition models are difficult to meet the needs of comprehensive analysis of multi-dimensional emotional information. Therefore, there is an urgent need for an emotion recognition method that can be cross-modally integrated to effectively integrate multimodal data such as images, texts, and audios, thereby improving the comprehensiveness and accuracy of emotion analysis.

[0003] The present invention is proposed based on this demand, aiming to overcome the limitations of traditional methods and provide a more comprehensive and accurate emotion recognition technology. Summary of the invention

[0004] In view of this, the object of the present invention is to provide an emotion recognition method based on low-rank multimodal cutting, splicing and fusion. First, for the feature extraction of the image modality, the widely recognized VGG19 model is used; for the feature extraction of the text modality, a pre-trained language model based on the Transformer model called BERT is used; for the feature extraction of the speech modality, MFCC and 1D-CNN are used to obtain audio features. Secondly, the three features are cropped to separate the common features and the unique feature parts. The common features are cropped again to distinguish between trimodal and bimodal common features. The one-dimensional features of the three modalities are not processed. The bimodal features are fused into a two-dimensional form with the non-overlapping unique features of the other modality. The unique features of the three modalities are fused into a three-dimensional form. This completes the cross-modal fusion module. Finally, the obtained fused representation is combined with the original representation to obtain the output of the emotion.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A sentiment recognition method based on low-rank multimodal cutting, splicing and fusion, comprising:

[0007] S1: For feature extraction of image modality, the VGG19 model is used; for feature extraction of text modality, the pre-trained language model of the BERT Transformer model is used; for feature extraction of speech modality, MFCC and 1D-CNN are used to obtain audio features;

[0008] S2: Crop the three features of image modality, text modality and speech modality, separate the common features and unique features, and crop the three common features again to distinguish the trimodal common features from the bimodal common features. Do not process the one-dimensional features of the three modalities. Fuse the bimodal features with the non-overlapping unique features of the other modality and express them in a two-dimensional form. Fuse the unique features of the three modalities and express them in a three-dimensional form to complete the cross-modal fusion module.

[0009] S3: The obtained fusion representation is combined with the original representation to obtain the output of emotion.

[0010] Furthermore, in S1, for the feature extraction of the image modality, the facial expression image is directly used as the input and represented as x I ,Then 16 layers of convolution operations are performed on the features to extract the local features of adjacent audio elements. In this process, 5 layers of maximum pooling are interspersed to sample, compress features and remove redundant information. Finally, three fully connected layers are applied to flatten the feature map into a one-dimensional vector for classification.

[0011] Furthermore, in S1, for feature extraction of text modality, the sentiment text is represented as Y, and then the masked language model (MLM) technology is used to train the text modality model.

[0012] Furthermore, in S1, for the feature extraction of speech text, a 1D convolutional neural network 1DCNN is used to preprocess the audio modal features by obtaining mfc, represented by Z, and transfer the features through a 2-layer convolution operation to extract local features of adjacent audio elements; the maximum pooling is used to downsample the samples, compress the features, and remove redundant information.

[0013] Furthermore, in S2, the adversarial multimodal optimization module decomposes multiple modalities into two non-overlapping parts: public features and private features, and extracts the consistency and specificity of heterogeneous modalities.

[0014] Furthermore, in S2, in order to maintain the sufficiency of the three-modal fusion, three bimodal croppers are designed to combine the three modal features (Z IT , Z IA and Z TA )'s bimodal public feature part and unimodal feature private feature part (Z I0 , Z T0 and Z A0) is divided; a three-mode cutter is designed for the three common parts of the bimodal mode, and the common part Z is divided ITa , and update the public feature part of the bimodal mode and the private feature part of the unimodal mode to ensure that the features are not counted repeatedly.

[0015] Further, the common feature is represented by C {I，T，A} are encoded in a shared latent subspace, the generator G(,;θc) is encouraged to confuse the discriminator D(,;θ D ), cannot distinguish C {I，T，A} The source modality of ; the adversarial loss is as follows:

[0016]

[0017] Where L C Training with a gradient reversal layer that keeps the input unchanged during forward propagation and multiplies the gradient by -1 during backpropagation;

[0018] Private features are denoted as P {I，T，A} Indicates that the encoding is in different latent subspaces, which are often in different distributions; encourages the use of discriminators D(,; θ D ) to distinguish the source of modality; the private adversarial loss is as follows:

[0019]

[0020] In order to encode different aspects of multimodal data, C {I，T，A} and P {I、T、A} The remainder with orthogonal loss is penalized as follows:

[0021]

[0022] Further, in S2, the feature set representation [Z ITA , Z IT , Z IA , Z TA , Z' I* Z' T* Z' A ], which includes one one-dimensional feature, three two-dimensional features and one three-dimensional feature. In order to reduce the required computing resources, step-by-step calculation and low-rank tensor calculation are proposed. Only one-dimensional feature data is used to determine the sentiment classification. When the threshold is reached, the result is directly output. The specific calculation formula is as follows:

[0023]

[0024] Where Z 1D is a one-dimensional input, Z 2D is a two-dimensional input, Z3D is a three-dimensional input; F 1D is a one-dimensional feature, F 2D is a two-dimensional feature, F 3D For three-dimensional features, when the L value exceeding the threshold θ is calculated 1D Stop prediction and give output; otherwise, calculate L in sequence 2D and L 3D , until the threshold is reached or the calculation ends, stop predicting and give the output.

[0025] The beneficial effects of the present invention are: making full use of the complementary information between multimodal data, separating the unique features within the modality and the common features between modalities by cutting and splicing the image, text and audio features, thereby improving the comprehensiveness and accuracy of emotion recognition. At the same time, through the detailed division of trimodal and bimodal common features, and the two-dimensional and three-dimensional fusion processing of multimodal features, the model's refined recognition level and robustness are significantly enhanced, so that it can still maintain a high accuracy when facing complex emotional expressions or data noise. Experimental results show that the emotion recognition accuracy of this method reaches 97.3%, achieving advanced performance in the field of emotion recognition, and providing strong technical support for optimizing user experience, market analysis, and mental health monitoring.

[0026] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0028] Figure 1 It is an overall framework diagram of the emotion recognition method based on low-rank multimodal cutting, splicing and fusion in an embodiment of the present invention;

[0029] Figure 2 This is a low-rank multimodal shearing flow chart in an embodiment of the present invention;

[0030] Figure 3 Flow chart of multimodal feature splicing and fusion in an embodiment of the present invention;

[0031] Figure 4 The overall architecture diagram of LMSSF in an embodiment of the present invention;

[0032] Figure 5 The training-testing accuracy changes with the number of iterations;

[0033] Figure 6 is the change of loss with the number of iterations;

[0034] Figure 7 t-SNE visualization of audio features;

[0035] Figure 8 t-SNE visualization of fusion features;

[0036] Fig. 9 t-SNE visualization of image text features;

[0037] Fig.10 t-SNE visualization of image features;

[0038] Fig.11 t-SNE visualization for text features;

[0039] Fig.12 t-SNE visualization of speech audio features;

[0040] Fig.13 Visualize audio tag correlations;

[0041] Fig.14 Visualize label correlation for image-text fusion;

[0042] Fig.15 Visualize label correlations for text-speech fusion;

[0043] Fig.16 Visualize the fusion label correlation. DETAILED DESCRIPTION

[0044] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0045] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0046] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0047] like Figure 1 As shown, the present invention provides an emotion recognition method based on low-rank multimodal cutting, splicing and fusion. The overall architecture of the method is designed to integrate and utilize shared and unique features from multiple modalities. On the left side of the architecture, three feature clippers are used to separate shared and unique features between modalities. The three shared features are further clipped to distinguish trimodal shared features from bimodal shared features. However, the one-dimensional features from the three modalities are not processed at this stage. On the right side of the architecture, the bimodal features are fused with non-overlapping unique features of the other modality and expressed in a two-dimensional form. At the same time, the unique features of the three modalities are fused and expressed in a three-dimensional form. This fusion process enables the model to capture shared and modality-specific information. Below the fusion stage, the three-dimensional features are input into the feature splicer for step-by-step calculation. In order to reduce the computational complexity, the input is processed from low dimension to high dimension. The calculation will continue until the predicted value reaches a predefined threshold or the three-dimensional features are fully utilized. Once the threshold is reached or the three-dimensional features are fully utilized, the result of the LMSSF architecture will be output. This architecture effectively integrates multimodal information, enabling the model to capture and exploit shared and unique features from multiple modalities to improve performance in various tasks.

[0048] The multimodal emotion recognition method is described in detail step by step as follows:

[0049] S1: For feature extraction of the image modality, the widely recognized VGG19 model is used; for feature extraction of the text modality, a pre-trained language model based on the Transformer model called BERT is used; for feature extraction of the speech modality, MFCC and 1D-CNN are used to obtain audio features.

[0050] S2: Three feature clippers are used to separate the shared and unique features between the modalities. The three shared features are further clipped to distinguish the tri-modal shared features from the bi-modal shared features. However, the one-dimensional features from the three modalities are not processed at this stage.

[0051] S3: In the fusion part of the method, the bimodal features are fused with the non-overlapping unique features of the other modality and expressed in 2D. At the same time, the unique features of the three modalities are fused and expressed in 3D. This fusion process enables the model to capture both shared and modality-specific information.

[0052] S4: After the fusion stage, the 3D features are input into the feature splicer for step-by-step calculation. In order to reduce the computational complexity, the input is processed from low dimension to high dimension.

[0053] Figure 3 The multimodal feature concatenation in the above refers to the step-by-step concatenation, and the feature set representation [Z ITA , Z IT , Z IA , Z TA , Z' I *Z' T *Z' A ], which includes one one-dimensional feature, three two-dimensional features and one three-dimensional feature. To this end, step-by-step calculation and low-rank tensor calculation are proposed to reduce the required computing resources. Step-by-step calculation is to first use only one-dimensional feature data to determine the emotion classification. When the threshold is reached, the result will be directly output without further calculation. The specific formula is as follows:

[0054]

[0055] Among them, Z 1D is a one-dimensional input, Z 2D is a two-dimensional input, Z 3D is a three-dimensional input. F 1D is a one-dimensional feature, F 2D is a two-dimensional feature, F 3D is a three-dimensional feature. When calculating L 1D When , if it exceeds the threshold θ we designed, stop predicting and output the result. Otherwise, calculate L in sequence 2D and L 3D , until the threshold is reached or the calculation ends.

[0056] In order to evaluate the multimodal emotion recognition method proposed in this application, this embodiment uses two types of data sets: CK+ data set and RAVDESS data set. Among them, the CK+ data set is a commonly used emotion recognition data set for facial expression recognition and emotion analysis research. The data set contains 593 facial expression images from 123 participants. Each participant expressed seven different basic emotions, including anger, disgust, fear, happiness, sadness, surprise and calmness. The RAVDESS data set is a commonly used data set for emotional speech and song analysis. The data set contains about 1,500 audio samples from 24 actors (12 men and 12 women). Each actor expressed eight different emotional states through voice and song, including anger, contempt, disgust, fear, happiness, sadness and surprise. Here, the remaining seven emotional data in the data set except for no emotion are used. In order to accurately represent the characteristics of multimodal coexistence, the present invention combines images, audio and text as data sets for multimodal model training and verification. The basis of the combination is the same emotional attribute.

[0057] The main method of the present invention uses the pre-trained convolutional neural network VGG19 as the image feature extraction network, the BERT algorithm based on the Transformer network as the text feature extraction network, and the acoustic feature extraction algorithm for audio feature extraction. Experiments and evaluations are performed on the RAVDESS-CK+ joint dataset, which contains 981 samples in the test set, each of which contains data from three modalities (image, text, and audio). We directly use seven emotion categories: anger, neutral, disgust, fear, happiness, sadness, and surprise. We use accuracy and recall as evaluation metrics to evaluate the performance of our model in the emotion recognition task. During the training process, the cross entropy loss function is used as the objective function, and the stochastic gradient descent algorithm is used for parameter optimization. The initial learning rate is set to 0.01 and dynamically adjusted according to the performance on the validation set. The entire model training is performed on an NVIDIA GeForce RTX 4090 GPU. The reported accuracy shows that the accuracy rate exceeds the average level.

[0058] In order to verify the performance of the model, the experimental results obtained by the method of the present invention were compared with the best emotion recognition model in recent years that also includes picture, text, and speech modalities. The results are shown in Table 1.

[0059] Table 1 compares the results of the multimodal emotion recognition model of this application with other models.

[0060] Table 1

[0061]

[0062] Table 1 shows the comparison of our proposed method with the baseline methods in terms of accuracy. We can see that our model achieves 97.3% accuracy with only 180K parameters, reaching the state-of-the-art performance. Compared with unimodal methods, our multimodal network exceeds them by more than 30% in accuracy, indicating the importance of multimodal fusion. Compared with bimodal methods, our multimodal network exceeds them by more than 20% in accuracy, highlighting the importance of image, text, and audio modalities.

[0063] Compared with the fusion method, our method achieves nearly 10% improvement in accuracy, which shows that discovering the correlation between image, text and audio modalities at an early stage is a challenging task. We verify the importance of intermodal interactions between image, text and audio modalities for emotion recognition. This shows that LMSSF effectively exploits the complementary information between the three modalities and reduces redundant features.

[0064] To better understand LMSSF, we study the different components in the three main modules: multimodal feature cutting (MFS), multimodal splicing (MS), and step-wise splicing (SS), which are simplified as MFS, MS, and SS in the table. At the same time, in order to fully understand the three modalities of image, text, and audio, we use single-modality and dual-modality as input to verify the experimental data. The ablation results are shown in Table 2.

[0065] Table 2

[0066]

[0067] During the ablation experiment, ① and ⑥ are the worst, which verifies the importance of multimodal information and multimodal shearing. As the modality of information increases, ⑦ and ⑧ gradually increase, ② is worse than ③, which shows that when considering multimodal data fusion, attention should be paid to increasing modal differences to achieve complementarity, ④ and ⑤ are not as good as ⑧, which verifies that changing the order of multimodal fusion may lead to poor performance. ④ and ⑤ are better than ① and ②. This confirms the rationality and optimality of LMSSF.

[0068] In LMSSF, both the overall loss and the cross-output loss are optimized. Figure 5 and Figure 6 shown.

[0069] Figure 1 .Training-testing accuracy and loss as the number of iterations increases.

[0070] according to Figure 5 , the accuracy of LMSSF training and testing process is almost monotonically increasing and smoothly converging, without overfitting or underfitting. Figure 6As shown in , the overall loss is almost monotonically decreasing and converges smoothly, while the 2D feature loss and 3D feature loss also decrease monotonically and then tend to be stable. Figure 6 It can be seen that the second and third layers are used less, which saves a lot of computational resources and ensures that there is no underfitting when there is a lot of data. Eventually, it reaches a stable point with more than 6 times the performance.

[0071] In order to improve the visualization level of the experiment, this paper also uses t-SNE to study the effectiveness of multimodal shearing. The seven emotions learned are visualized, such as Figure 7 to Figure 12 shown.

[0072] The distributions of the seven emotions in the unimodal and bimodal conditions sometimes overlapped. Figure 8 The seven emotions in the dataset can be completely separated and the distributions of different modalities can be effectively aligned.

[0073] Visualize the learned label correlations to illustrate their interpretability, e.g. Figure 13 to Figure 16 As shown in Figure 2, the learned label correlations vary across data modalities, and multimodal data jointly enrich the semantic information.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. An emotion recognition method based on low-rank multimodal cutting, splicing and fusion, characterized by: The method comprises the following steps: S1: For feature extraction of image modality, the VGG19 model is used; for feature extraction of text modality, the pre-trained language model of the BERT Transformer model is used; for feature extraction of speech modality, MFCC and 1D-CNN are used to obtain audio features; S2: Crop the three features of image modality, text modality and speech modality, separate the common features and unique features, and crop the three common features again to distinguish the trimodal common features from the bimodal common features. Do not process the one-dimensional features of the three modalities. Fuse the bimodal features with the non-overlapping unique features of the other modality and express them in a two-dimensional form. Fuse the unique features of the three modalities and express them in a three-dimensional form to complete the cross-modal fusion module. S3: The obtained fusion representation is combined with the original representation to obtain the output of emotion.

2. The emotion recognition method based on low-rank multimodal cutting, splicing and fusion according to claim 1 is characterized in that: In S1, for feature extraction of image modality, facial expression images are directly used as input and expressed as x I ,Then 16 layers of convolution operations are performed on the features to extract the local features of adjacent audio elements. In this process, 5 layers of maximum pooling are interspersed to sample, compress features and remove redundant information. Finally, three fully connected layers are applied to flatten the feature map into a one-dimensional vector for classification.

3. The emotion recognition method based on low-rank multimodal cutting, splicing and fusion according to claim 1 is characterized in that: In S1, for feature extraction of text modality, the sentiment text is represented as Y, and then the masked language model (MLM) technology is used to train the text modality model.

4. The emotion recognition method based on low-rank multimodal cutting, splicing and fusion according to claim 1, characterized in that: In S1, for feature extraction of speech text, a 1D convolutional neural network 1DCNN is used to preprocess the audio modal features by obtaining the MFC, represented by Z, and the features are transferred through a 2-layer convolution operation to extract local features of adjacent audio elements; Use maximum pooling to downsample samples, compress features, and remove redundant information.

5. The emotion recognition method based on low-rank multimodal cutting, splicing and fusion according to claim 1, characterized in that: In S2, the adversarial multimodal optimization module decomposes multiple modalities into two non-overlapping parts: public features and private features, and extracts the consistency and specificity of heterogeneous modalities.

6. The emotion recognition method based on low-rank multimodal cutting, splicing and fusion according to claim 1, characterized in that: In S2, in order to maintain the sufficiency of the three-modal fusion, three bimodal croppers are designed to combine the three modal features (Z IT , Z IA and Z TA )'s bimodal public feature part and unimodal feature private feature part (Z I0 , Z T0 and Z A0 ) is divided; a three-mode cutter is designed for the three common parts of the bimodal mode, and the common part Z is divided ITa , and update the public feature part of the bimodal mode and the private feature part of the unimodal mode to ensure that the features are not counted repeatedly.

7. The emotion recognition method based on low-rank multimodal cutting, splicing and fusion according to claim 1 is characterized in that: The common feature is denoted as C {I，T，A} are encoded in a shared latent subspace, the generator G(,;θc) is encouraged to confuse the discriminator D(,;θ D ), cannot distinguish C {I，T，A} The source modality of ; the adversarial loss is as follows: Where L C Training with a gradient reversal layer that keeps the input unchanged during forward propagation and multiplies the gradient by -1 during backpropagation; Private features are denoted as P {I，T，A} Indicates that the encoding is in different latent subspaces, which are often in different distributions; encourage the use of discriminators D(,; θ D ) to distinguish the source of modality; the private adversarial loss is as follows: In order to encode different aspects of multimodal data, C {I，T，A} and P {I、T、A} The remainder with orthogonal loss is penalized as follows:

8. The emotion recognition method based on low-rank multimodal cutting, splicing and fusion according to claim 1, characterized in that: In S2, the feature set representation [Z ITA , Z IT , Z IA , Z TA , Z' I *Z' T *Z' A ], which includes one one-dimensional feature, three two-dimensional features and one three-dimensional feature. In order to reduce the required computing resources, step-by-step calculation and low-rank tensor calculation are proposed. Only one-dimensional feature data is used to determine the sentiment classification. When the threshold is reached, the result is directly output. The specific calculation formula is as follows: Where Z 1D is a one-dimensional input, Z 2D is a two-dimensional input, Z 3D is a three-dimensional input; F 1D is a one-dimensional feature, F 2D is a two-dimensional feature, F 3D For three-dimensional features, when the L value exceeding the threshold θ is calculated 1D Stop prediction and give output; otherwise, calculate L in sequence 2D and L 3D , until the threshold is reached or the calculation ends, stop predicting and give the output.

Images