Multi-modal emotion recognition method based on Transform intra-modal perception and inter-modal cross fusion

Abstract

The invention discloses a multi-modal emotion recognition method based on Transform intra-modal perception and inter-modal cross fusion, and the method comprises the steps: carrying out the coding of voice and text, extracting depth features, capturing the long-distance dependence relation in each modal based on a Transform intra-modal perception module provided by the invention, and carrying out the recognition of the long-distance dependence relation in each modal, and carrying out the recognition of the long-distance dependence relation in each modal. Local perception learning of emotion features is realized, and redundant information in deep features is reduced; secondly, in order to fuse unaligned multi-modal sequence information and fully utilize complementarity of different modal information, an inter-modal interactive fusion module based on Transform is proposed to capture information dependency relationships among different modals, and fused multi-modal global information is obtained; finally, an ablation experiment is carried out, and the effectiveness of the method is verified. According to the method, effective parallel computing of multi-modal emotion recognition is realized, and the recognition performance and generalization ability of a multi-modal emotion recognition system are further improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal emotion recognition, and specifically relates to a multimodal emotion recognition method based on Transformer intra-modal perception and inter-modal cross-fusion. Background Art

[0002] Emotion recognition is an important part of the field of artificial intelligence. It is mainly devoted to exploring how to use a series of mathematical processing methods to deeply analyze input data so that computers can accurately capture human emotional states. By building such an emotion recognition system, people will have a better chance of creating a natural and barrier-free human-computer interaction environment. Emotion recognition can generally be divided into two categories: direct emotion recognition and indirect emotion recognition. Direct emotion recognition mainly involves the use of multiple modal information, including text, voice, image, video, etc. Indirect emotion recognition mainly relies on monitoring human physiological reactions for implicit emotion recognition, including eye movement signals, body movement signals, EEG signals, ECG signals, etc. When human emotional states change, they are often accompanied by changes in multiple information. Information of different modalities is often highly correlated, thus acting comprehensively on human emotion recognition. Therefore, it is of great significance to study multimodal emotion recognition that integrates multiple modal emotion information for affective computing.

[0003] In the early days of emotion recognition, researchers mostly used unimodal emotion recognition technology to achieve human emotion recognition. Unimodal emotion recognition refers to the process of identifying and understanding emotions using only a single type of data source. Usually, this single data source can be speech, text, images, etc. Since emotions are expressed in a variety of ways, humans can perceive the emotions or intentions of others by integrating facial expressions, speech or other information. Therefore, the field of emotion recognition based on multimodal information has received increasing attention. In the early days of multimodal emotion recognition research, researchers mostly used traditional machine model methods for feature extraction, such as hidden Markov model (HMM), Gaussian mixture model (GMM), etc. However, these models can only study limited emotional context information and cannot fully utilize the characteristics of human emotions that change slowly and are highly dependent on context information.

[0004] In recent years, with the rapid development of deep learning algorithms, emotion recognition technology based on deep learning has shown new vitality. Deep learning technology enables researchers to extract complex modal information and subtle nuances from multimodal data, thereby promoting a deep understanding of complex emotional expressions. In 2021, Cao et al. constructed a multimodal emotion recognition system using the stacked network HNSD to better capture high emotion discrimination features. In 2023, Xie et al. proposed a multimodal emotion recognition method based on multi-task learning and attention mechanism, and obtained emotion recognition rates of 85.36% and 84.61% on the CMU-MOSI and CMU-MOSEI databases, respectively. In 2024, Li et al. proposed a multimodal shared network with cross-modal constraints to achieve continuous emotion recognition tasks.

[0005] As important manifestations of emotional information in people's daily lives, speech and text provide key judgment information for multimodal emotion recognition systems. Considering the heterogeneity of speech and text information in actual situations, whether multimodal emotion recognition technology can effectively obtain key emotional information and complementary information in speech and text data, and whether it can effectively integrate the extracted speech and text information, will affect the performance of multimodal emotion recognition systems. Summary of the invention

[0006] The purpose of the present invention is to address the defects and shortcomings of the above-mentioned prior art, and propose a multimodal emotion recognition method based on Transformer intra-modal perception and inter-modal cross-fusion. By introducing a Transformer-based intra-modal perception module, the long-distance dependencies within each modality are captured, local perception learning of emotional features is achieved, and redundant information in deep features is reduced. By introducing a Transformer-based inter-modal interactive fusion module to capture the information dependencies between different modalities, the fused multimodal global information is obtained, and the complementarity of different modal information is fully utilized. The present invention realizes effective parallel computing for multimodal emotion recognition, improves training speed and recognition performance, and has better generalization ability and practicality.

[0007] The technical solution adopted by the present invention to solve the technical problem is: a multimodal emotion recognition method based on Transformer intra-modal perception and inter-modal cross-fusion, the method comprising the following steps:

[0008] Step 1: Perform shallow feature extraction on the original input speech and text modalities respectively, and introduce the Mel-spectrogram-AlexNet branch to make up for the lack of spatial information in speech features;

[0009] Step 2: Input the extracted shallow features into the deep coding network to obtain deep features. For text modality, build a GRUs network for deep coding. For speech modality, build a CNNs network for deep coding. For Mel spectrogram, build a Mel spectrogram-AlexNet branch.

[0010] Step 3: Input the deeply encoded text features, multi-scale utterance-level speech features, and mel-spectrogram features into the Transformer-based intra-modal perception module to obtain the perceptual learning features of the key emotional information within the modality;

[0011] Step 4: Input the perceptual learning text features, perceptual learning multi-scale speech features and perceptual learning Mel spectrogram features into the Transformer-based inter-modal cross-fusion module, use Mel spectrogram as the reused data information for interactive fusion, and different modal information can complement each other. The results are input into the sentiment classifier for sentiment discrimination and classification;

[0012] Step 5: Perform a performance evaluation on the proposed multimodal emotion recognition method based on Transformer intra-modal perception and inter-modal cross-fusion.

[0013] Furthermore, the specific steps of step 1 include:

[0014] Step 1-1: Pass each piece of text information through the Word2vec sub-network to obtain a 300-dimensional utterance-level text feature vector;

[0015] Step 1-2: Preprocess each speech signal with different frame lengths, where the frame lengths are 256 and 512 respectively, and use librosa for feature extraction to fuse speech features of different scales to obtain 750-dimensional multi-scale speech-level speech features;

[0016] Step 1-3: Use librosa to generate a mel-spectrogram with a window size of 25ms and a step size of 10ms and crop it to a size of 224×224 to obtain an input of uniform size and obtain the time-frequency-spatial emotional information in the speech signal.

[0017] Furthermore, the specific steps of step 2 include:

[0018] Step 2-1: Input 300-dimensional Word2vec features into the GRUs network, and the output is the deep features extracted by 1 Masking layer, 2 GRU layers, and 1 Dropout layer;

[0019] Step 2-2: Input the 750-dimensional multi-scale utterance-level speech features into a CNN network consisting of two one-dimensional convolutional blocks, where each convolutional block contains one one-dimensional convolutional layer, one ReLu activation layer, and one maximum pooling layer;

[0020] Step 2-3: Input the 224×224 Mel-spectrogram into the pre-trained AlexNet network for deep encoding, set the number of filters in the last fully connected layer of the AlexNet network to 300, and obtain 300-dimensional Mel-spectrogram information.

[0021] Furthermore, the specific steps of step 3 include:

[0022] Step 3-1: Input the deeply encoded text features, multi-scale speech-level features, and Mel-spectrogram features into the Transformer module for operation, respectively, to improve the model's perceptual learning effect on intra-modal information;

[0023] Step 3-2: The feature information obtained after the Transformer module operation is subjected to nonlinear changes using the tanh layer to further increase the perception module's ability to capture data. At the same time, the data information from the Transformer module is subjected to a sigmoid function operation so that the probability of the feature data is mapped to the interval (0, 1);

[0024] Step 3-3: The two outputs of the above step 3-2 are multiplied element-by-element using the Hadamard product operation to realize the product of the feature data of the same dimension, and the output of the modal intra-aware learning is obtained;

[0025] Step 3-4: Considering the dimensional information of each modal data, add a Flatten layer to each modal branch to flatten the modal data.

[0026] Furthermore, in step 3-1, the specific method of Transformer module operation includes the following steps:

[0027] Step 3-1-1: Input the deeply encoded text features and the deeply encoded multi-scale utterance-level speech features into the Transformer_1 module for operation, where the number of layers of the Transformer_1 module is 4, the hidden unit dimension is set to 128, the dimension of the hidden layer in the feedforward neural network is 512, the number of heads of the multi-head attention mechanism is 8, the input shape is 300×256, and the maximum length of the positional encoding is 1000;

[0028] Step 3-1-2: The deeply encoded Mel-spectrogram features are input to the Transformer_2 module. Its parameters are basically the same as those of Transformer_1, except that the input shape of Transformer_2 is 300×512.

[0029] Furthermore, the specific steps of step 4 include:

[0030] Step 4-1: Input the information of different modalities into the multimodal gated fusion mechanism, use the Mel spectrogram as the multiplexed data information for interactive fusion, and obtain text-Mel spectrogram information and multi-scale speech-Mel spectrogram information;

[0031] Step 4-2: Input the output text-mel spectrogram information and multi-scale speech-mel spectrogram information into the Transformer module for feature calculation, where the number of layers of the Transformer module is 4, the hidden unit dimension is set to 128, the dimension of the hidden layer in the feedforward neural network is 512, the number of heads of the multi-head attention mechanism is 8, the input shape is 300×256, and the maximum length of the positional encoding is 1000;

[0032] Step 4-3: Calculate the weighted value through the sigmoid function to obtain the activation value of the Transformer feature information, and effectively screen the key emotional feature information in the sample data;

[0033] Step 4-4: Perform a Hadamard product operation on the activation value of the above feature information and the Transformer feature information, and obtain the final multimodal emotion feature representation through a Concat operation.

[0034] Furthermore, in step 5, a specific method for evaluating the performance of the proposed multimodal emotion recognition method based on Transformer intra-modality perception and inter-modality cross-fusion includes the following steps:

[0035] Step 5-1: Conduct a comparative experiment on the multimodal emotion recognition method based on Transformer intra-modal perception and inter-modal gating interaction and the multimodal emotion recognition method based on intra-modal perception and inter-modal gating interaction to verify the effectiveness of the Transformer intra-modal perception module;

[0036] Step 5-2: Conduct a comparative experiment on the multimodal emotion recognition method based on Transformer intra-modal perception and inter-modal cross-fusion and the multimodal emotion recognition method based on Transformer intra-modal perception and inter-modal gated interaction to verify the effectiveness of the Transformer inter-modal cross-fusion module;

[0037] Beneficial effects:

[0038] 1. The present invention proposes a Transformer-based intra-modal perception module, which uses the Transformer's self-attention mechanism to realize local learning of emotional features, reduce redundant information in deep features, and achieve effective feature screening.

[0039] 2. In order to fuse unaligned multimodal sequence information and make full use of the complementarity of different modal information, the present invention introduces a Transformer-based inter-modal interactive fusion module, which is used to explore the information dependency between different modalities and obtain the fused multimodal global information.

[0040] 3. The present invention can realize the modeling of information dependency within the sequence through the Transformer mechanism, realize complete parallel computing, improve the training speed and recognition performance, and have better generalization ability and practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the operation flow of the present invention.

[0042] Figure 2 This is a block diagram of the Transformer-based intra-modal perception module of the present invention.

[0043] Figure 3 This is a block diagram of the cross-fusion module based on Transformer modalities in the present invention.

[0044] Figure 4 This is a graph of ablation experiment results based on the Transformer intra-modality perception module and the Transformer inter-modality interaction fusion module of the present invention. DETAILED DESCRIPTION

[0045] The invention will be further described in detail below in conjunction with the accompanying drawings.

[0046] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in order to avoid blurring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions of the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0047] In addition, it should be noted that the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus.

[0048] like Figures 1 to 3 As shown, the present invention proposes a multimodal emotion recognition method based on Transformer intra-modal perception and inter-modal cross-fusion, applies deep neural network to multimodal emotion recognition, and is used to extract deep features, so as to build a higher performance multimodal emotion recognition system, which specifically includes the following steps:

[0049] Step 1: Perform shallow feature extraction on the speech and text modalities in the original input database, and introduce the Mel-spectrogram-AlexNet branch to make up for the lack of spatial information in the speech features. The dataset used in the present invention is the IEMOCAP emotion database, which consists of about 12 hours of audio and video content recorded by 10 professional performers (5 men and 5 women) in a professional studio. The corpus is divided into 5 parts, each of which includes two actors, one male and one female.

[0050] Step 1 specifically includes:

[0051] Step 1-1: Pass each piece of text information through the Word2vec sub-network to obtain a 300-dimensional utterance-level text feature vector;

[0052] Step 1-2: Preprocess each speech signal with different frame lengths, where the frame lengths are 256 and 512 respectively, and use librosa for feature extraction to fuse speech features of different scales to obtain 750-dimensional multi-scale speech-level speech features;

[0053] Step 1-3: Use librosa to generate a mel-spectrogram with a window size of 25ms and a step size of 10ms and crop it to a size of 224×224 to obtain an input of uniform size and obtain the time-frequency-spatial emotional information in the speech signal.

[0054] Step 2: Input the extracted shallow features into the deep coding network to obtain deep features. For the text modality, build a GRUs network for deep coding. For the speech modality, build a CNNs network for deep coding. For the Mel-spectrogram, build a Mel-spectrogram-AlexNet branch.

[0055] Step 2 specifically includes:

[0056] Step 2-1: Input 300-dimensional Word2vec features into the GRUs network, and the output is the deep features extracted by 1 Masking layer, 2 GRU layers, and 1 Dropout layer;

[0057] Step 2-2: Input the 750-dimensional multi-scale utterance-level speech features into a CNN network consisting of two one-dimensional convolutional blocks, where each convolutional block contains one one-dimensional convolutional layer, one ReLu activation layer, and one maximum pooling layer;

[0058] Step 2-3: Input the 224×224 Mel-spectrogram into the pre-trained AlexNet network for deep encoding, set the number of filters in the last fully connected layer of the AlexNet network to 300, and obtain 300-dimensional Mel-spectrogram information.

[0059] Step 3: Input the deeply encoded text features, multi-scale utterance-level speech features, and mel-spectrogram features into the Transformer-based intra-modal perception module to obtain perceptual learning features of key intra-modal sentiment information.

[0060] Step 3 specifically includes:

[0061] Step 3-1: Input the deeply encoded text features, multi-scale speech-level features, and Mel-spectrogram features into the Transformer module for operation, respectively, to improve the model's perceptual learning effect on intra-modal information;

[0062] Step 3-2: The feature information obtained after the Transformer module operation is subjected to nonlinear changes using the tanh layer to further increase the perception module's ability to capture data. At the same time, the data information from the Transformer module is subjected to a sigmoid function operation so that the probability of the feature data is mapped to the interval (0, 1);

[0063] Step 3-3: The two outputs of the above step 3-2 are multiplied element-by-element using the Hadamard product operation to realize the product of the feature data of the same dimension, and the output of the modal intra-aware learning is obtained;

[0064] Step 3-4: Considering the dimensional information of each modal data, add a Flatten layer to each modal branch to flatten the modal data.

[0065] Among them, step 3-1 specifically includes:

[0066] Step 3-1-1: Input the deeply encoded text features and the deeply encoded multi-scale utterance-level speech features into the Transformer_1 module for operation, where the number of layers of the Transformer_1 module is 4, the hidden unit dimension is set to 128, the dimension of the hidden layer in the feedforward neural network is 512, the number of heads of the multi-head attention mechanism is 8, the input shape is 300×256, and the maximum length of the positional encoding is 1000;

[0067] Step 3-1-2: The deeply encoded Mel-spectrogram features are input to the Transformer_2 module. Its parameters are basically the same as those of Transformer_1, except that the input shape of Transformer_2 is 300×512.

[0068] Step 4: Input the perceptual learning text features, perceptual learning multi-scale speech features and perceptual learning Mel spectrogram features into the Transformer-based inter-modal cross-fusion module, use Mel spectrogram as the reused data information for interactive fusion, realize the complementarity of different modal information, and input the results into the sentiment classifier for sentiment discrimination and classification;

[0069] Step 4 specifically includes:

[0070] Step 4-1: Input the information of different modalities into the multimodal gated fusion mechanism, use the Mel spectrogram as the multiplexed data information for interactive fusion, and obtain text-Mel spectrogram information and multi-scale speech-Mel spectrogram information;

[0071] Step 4-2: Input the output text-mel spectrogram information and multi-scale speech-mel spectrogram information into the Transformer module for feature calculation, where the number of layers of the Transformer module is 4, the hidden unit dimension is set to 128, the dimension of the hidden layer in the feedforward neural network is 512, the number of heads of the multi-head attention mechanism is 8, the input shape is 300×256, and the maximum length of the positional encoding is 1000;

[0072] Step 4-3: Calculate the weighted value through the sigmoid function to obtain the activation value of the Transformer feature information, so as to effectively screen the key emotional feature information in the sample data;

[0073] Step 4-4: Perform a Hadamard product operation on the activation value of the above feature information and the Transformer feature information, and obtain the final multimodal emotion feature representation through a Concat operation.

[0074] Step 5: Perform a performance evaluation on the proposed multimodal emotion recognition method based on Transformer intra-modal perception and inter-modal cross-fusion.

[0075] Step 5 specifically includes:

[0076] Step 5-1: Conduct a comparative experiment on the multimodal emotion recognition method based on Transformer intra-modal perception and inter-modal gating interaction and the multimodal emotion recognition method based on intra-modal perception and inter-modal gating interaction to verify the effectiveness of the Transformer intra-modal perception module;

[0077] Step 5-2: Conduct a comparative experiment on the multimodal emotion recognition method based on Transformer intra-modal perception and inter-modal cross-fusion with the multimodal emotion recognition method based on Transformer intra-modal perception and inter-modal gated interaction to verify the effectiveness of the Transformer inter-modal cross-fusion module.

[0078] In this embodiment, the experimental performance evaluation is to measure the model performance by using weighted recognition rate (WA), unweighted recognition rate (UA), F1 score and precision. The IEMOCAP database contains a variety of emotions, including angry, excited, happy, sad, neutral, frustrated, etc. The above six categories of emotional sentences are selected for the six-category multimodal emotion recognition task. At the same time, the IEMOCAP data set is tested for performance by five-fold cross validation. During the experiment, the computer model is Lenovo XiaoXinPro 16, the CPU processor is AMD Ryzen 7 6800H, the Tensorflow version number is 2.13.0, the Keras version number is 2.13.1, the speech emotion feature extraction library is Librosa 0.8.1, the optimizer of the network model is Adam, the initial learning rate is 0.01, the batch size is 32, the number of iterations is 100 times, and the patience value is 10.

[0079] Ablation experiments were conducted on the Transformer-based intra-modal perception module and the Transformer-based inter-modal interaction fusion module. Framework A represents the use of the intra-modal perception module and the inter-modal gated interaction module, framework B represents the use of the Transformer-based intra-modal perception module and the inter-modal gated interaction module, and framework C represents the use of the Transformer-based intra-modal perception module and the Transformer-based inter-modal interaction fusion module. The results are shown in Figure 2. Figure 4 shown.

[0080] We can see that when using framework B for emotion recognition tasks, the weighted emotion recognition rate of 6 categories is 73.45%, which is 1.99% higher than the baseline model A; the unweighted recognition rate of 6 categories is 74.65%, which is 1.5% higher than the unweighted recognition rate of framework A; at the same time, the F1 score of framework B is 73.25%, and the precision is 74.17%, which is 3.07% and 3.01% higher than the F1 score and precision of framework A, which only uses nonlinear calculation in the intra-modal perception module. It can be seen that the constructed Transformer-based intra-modal perception model has achieved certain improvements in various indicators in the 6-category multimodal emotion classification task. When using framework C for various index tests, the weighted recognition rate of the model is 73.89%, and the unweighted recognition rate is 75.02%, which are 2.43% and 1.87% higher than the baseline model framework A, respectively; the F1 score of the framework C model is 73.74%, which is 3.56% higher than that of framework A, and the precision is 3.84% higher than that of framework A. Compared with Framework B, the weighted recognition rate and non-weighted recognition rate of the multimodal emotion recognition model of Framework C increased by 0.44% and 0.37% respectively; at the same time, compared with Framework B, the F1 score and precision of Framework C increased by 0.49% and 0.83% respectively. It can be seen that when the Transformer-based inter-modal interaction fusion module is adopted, the various parameter indicators of the model have been improved to a certain extent.

[0081] In summary, this embodiment proposes a multimodal emotion recognition method based on Transformer intra-modal perception and inter-modal cross-fusion. It proposes a method that can improve the performance of the multimodal emotion recognition system from two modules: the Transformer intra-modal perception module and the Transformer inter-modal interactive fusion module. The experimental results verify the rationality and effectiveness of these two modules. Compared with other methods, the method of this embodiment is improved in multiple measurement indicators and has great reference significance in practical applications.

[0082] The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A multimodal emotion recognition method based on Transformer intra-modality perception and inter-modality cross-fusion, characterized in that: The method comprises the following steps: Step 1: Perform shallow feature extraction on the original input speech and text modalities respectively, and introduce the Mel-spectrogram-AlexNet branch to make up for the lack of spatial information in speech features; Step 2: Input the extracted shallow features into the deep coding network to obtain deep features. For text modality, build a GRUs network for deep coding. For speech modality, build a CNNs network for deep coding. For Mel spectrogram, build a Mel spectrogram-AlexNet branch. Step 3: Input the deeply encoded text features, multi-scale utterance-level speech features, and Mel-spectrogram features into the Transformer-based intra-modal perception module to obtain the perceptual learning features of the key intra-modal emotional information; Step 4: Input the perceptual learning text features, perceptual learning multi-scale speech features and perceptual learning Mel spectrogram features into the Transformer-based inter-modal cross-fusion module, use Mel spectrogram as the reused data information for interactive fusion, and different modal information can complement each other. The results are input into the sentiment classifier for sentiment discrimination and classification; Step 5: Perform a performance evaluation on the proposed multimodal emotion recognition method based on Transformer intra-modal perception and inter-modal cross-fusion.

2. According to claim 1, a multimodal emotion recognition method based on Transformer intra-modality perception and inter-modality cross-fusion is characterized in that: The specific steps of step 1 include: Step 1-1: Pass each piece of text information through the Word2vec sub-network to obtain a 300-dimensional discourse-level text feature vector; Step 1-2: Preprocess each speech signal with different frame lengths, where the frame lengths are 256 and 512 respectively, and use librosa for feature extraction to fuse speech features of different scales to obtain 750-dimensional multi-scale speech-level speech features; Step 1-3: Use librosa to generate a mel-spectrogram with a window size of 25ms and a step size of 10ms and crop it to a size of 224×224 to obtain an input of uniform size and obtain the time-frequency-spatial emotional information in the speech signal.

3. According to claim 1, a multimodal emotion recognition method based on Transformer intra-modality perception and inter-modality cross-fusion is characterized in that: The specific steps of step 2 include: Step 2-1: Input 300-dimensional Word2vec features into the GRUs network, and the output is the deep features extracted by 1 Masking layer, 2 GRU layers, and 1 Dropout layer; Step 2-2: Input the 750-dimensional multi-scale utterance-level speech features into a CNN network consisting of two one-dimensional convolutional blocks, where each convolutional block contains one one-dimensional convolutional layer, one ReLu activation layer, and one maximum pooling layer; Step 2-3: Input the 224×224 Mel-spectrogram into the pre-trained AlexNet network for deep encoding, set the number of filters in the last fully connected layer of the AlexNet network to 300, and obtain 300-dimensional Mel-spectrogram information.

4. According to claim 1, a multimodal emotion recognition method based on Transformer intra-modality perception and inter-modality cross-fusion is characterized in that: The specific steps of step 3 include: Step 3-1: Input the deeply encoded text features, multi-scale utterance-level speech features, and Mel-spectrogram features into the Transformer module for operation respectively; Step 3-2: The feature information obtained after the Transformer module operation is subjected to nonlinear changes using the tanh layer to further increase the perception module's ability to capture data. At the same time, the data information from the Transformer module is subjected to a sigmoid function operation so that the probability of the feature data is mapped to the interval (0, 1); Step 3-3: The two outputs of the above step 3-2 are multiplied element-by-element using the Hadamard product operation to realize the product of the feature data of the same dimension, and the output of the modal intra-aware learning is obtained; Step 3-4: Considering the dimensional information of each modal data, add a Flatten layer to each modal branch to flatten the modal data.

5. According to claim 4, a multimodal emotion recognition method based on Transformer intra-modality perception and inter-modality cross-fusion is characterized in that: In step 3-1, the specific method for implementing the Transformer module operation includes the following steps: Step 3-1-1: Input the deeply encoded text features and the deeply encoded multi-scale utterance-level speech features into the Transformer_1 module for operation, where the number of layers of the Transformer_1 module is 4, the hidden unit dimension is set to 128, the dimension of the hidden layer in the feedforward neural network is 512, the number of heads of the multi-head attention mechanism is 8, the input shape is 300×256, and the maximum length of the positional encoding is 1000; Step 3-1-2: The deeply encoded Mel-spectrogram features are input to the Transformer_2 module. Its parameters are basically the same as those of Transformer_1, except that the input shape of Transformer_2 is 300×512.

6. According to claim 1, a multimodal emotion recognition method based on Transformer intra-modality perception and inter-modality cross-fusion is characterized in that: The specific steps of step 4 include: Step 4-1: Input the information of different modalities into the multimodal gated fusion mechanism, use the Mel spectrogram as the multiplexed data information for interactive fusion, and obtain text-Mel spectrogram information and multi-scale speech-Mel spectrogram information; Step 4-2: Input the output text-mel spectrogram information and multi-scale speech-mel spectrogram information into the Transformer module for feature calculation, where the number of layers of the Transformer module is 4, the hidden unit dimension is set to 128, the dimension of the hidden layer in the feedforward neural network is 512, the number of heads of the multi-head attention mechanism is 8, the input shape is 300×256, and the maximum length of the positional encoding is 1000; Step 4-3: Calculate the weighted value through the sigmoid function to obtain the activation value of the Transformer feature information, and effectively screen the key emotional feature information in the sample data; Step 4-4: Perform a Hadamard product operation on the activation value of the above feature information and the Transformer feature information, and obtain the final multimodal emotion feature representation through a Concat operation.

7. The multimodal emotion recognition method based on Transformer intra-modality perception and inter-modality cross-fusion according to claim 1 is characterized in that: In step 5, a specific method for evaluating the performance of the proposed multimodal emotion recognition method based on Transformer intra-modality perception and inter-modality cross-fusion includes the following steps: Step 5-1: Conduct a comparative experiment on the multimodal emotion recognition method based on Transformer intra-modal perception and inter-modal gating interaction and the multimodal emotion recognition method based on intra-modal perception and inter-modal gating interaction to verify the effectiveness of the Transformer intra-modal perception module; Step 5-2: Conduct a comparative experiment on the multimodal emotion recognition method based on Transformer intra-modal perception and inter-modal cross-fusion with the multimodal emotion recognition method based on Transformer intra-modal perception and inter-modal gated interaction to verify the effectiveness of the Transformer inter-modal cross-fusion module.

Images