Context awareness and cross-modal attention sharing-based dialogue emotion recognition method

Abstract

The invention discloses a dialogue emotion recognition method based on context awareness and cross-modal attention sharing, and the method comprises the steps: collecting texts containing contexts and audio information in different dialogue scenes to form a data set, and completing the construction of the data set; preprocessing the constructed data set to extract corresponding modal features; the method comprises the following steps: respectively establishing an audio context sensing network, a cross-modal sharing attention network and an adaptive fusion network, and sequentially connecting the established networks to form an initial emotion recognition model; performing end-to-end training on the initial emotion recognition model, and storing the trained emotion classification model to form an emotion recognition model; performing feature extraction on the collected text and audio data by adopting a feature encoder to obtain text and audio features; and inputting the text and audio features into an emotion recognition model, and recognizing and outputting a predefined condition category by using the emotion recognition model. The method can effectively solve the problem of spatial heterogeneity, and has an excellent recognition effect.

Description

Technical Field

[0001] The present invention belongs to the field of computer natural language processing, and in particular relates to a conversation emotion recognition method based on context perception and cross-modal shared attention. Background Art

[0002] The goal of Emotion Recognition in Conversation (ERC) is to identify the emotion label of each utterance in a conversation. By providing context-rich emotional responses, ERC can significantly improve the user experience in dialogue systems. This capability plays a key role in multiple fields such as chatbots, recommendation systems, and medical systems. However, the ERC task faces several challenges. Relying solely on text data will lead to the problem of inaccurate recognition, as similar statements may convey completely different emotions. Therefore, adopting multiple modalities (such as text, audio, and video) is crucial to fully and accurately understand the emotional expression in the conversation.

[0003] According to the different input modalities, emotion recognition models can be divided into two categories: text-based methods and multimodal methods. Text-based emotion recognition models mainly focus on conversation context, speaker modeling, and common sense knowledge. Although such models can effectively capture text emotions in some scenarios, when encountering ambiguous or complex emotional expressions, text alone may not be able to fully understand the emotions. Multimodal emotion recognition models can more comprehensively understand emotional expressions in conversations by integrating text, audio, and visual information. Current models mainly focus on multimodal feature extraction, interaction, and fusion.

[0004] In terms of multimodal feature extraction, different types of data (such as text, audio, and video) need to be processed by different tools and models. For example, text features can be extracted by BERT or RoBERTa models, audio features can be extracted by Librosa or Wav2Vec2, and facial expressions and gesture features in videos can be extracted by tools such as MTCNN or OpenFace, which can effectively capture emotional expressions in different modalities. In terms of multimodal feature interaction, existing technologies use neural network methods such as RNN, LSTM, GRU, and GCN for feature interaction, but these existing works do not fully consider the heterogeneity of different modalities in the feature space. In the field of multimodal fusion, some methods integrate features through early fusion technology, that is, splicing the original features from different modalities together before processing and inputting them into a shared network. Other methods adopt a late fusion strategy, allowing independent modality-specific networks to process features separately, and then splicing their outputs in the later stage, but this simple splicing cannot effectively fuse multimodal features in different feature spaces. To this end, it is urgent to provide a conversation emotion recognition method based on context awareness and cross-modal shared attention. Summary of the invention

[0005] In view of the problems existing in the above-mentioned prior art, the present invention provides a method for dialogue emotion recognition based on context perception and cross-modal shared attention. The method can effectively solve the problems existing in the current emotion recognition methods based on various neural networks, such as insufficient consideration of the heterogeneity of different modalities in the feature space and inability to effectively exchange emotional information between multiple modalities, resulting in poor performance in the final fusion stage. At the same time, the method can effectively improve the robustness of the emotion recognition algorithm, and can effectively solve the problem of spatial heterogeneity, and has excellent recognition effect.

[0006] In order to solve the above technical problems, the present invention provides a conversation emotion recognition method based on context perception and cross-modal shared attention, comprising the following steps:

[0007] Step 1: Data acquisition;

[0008] Collect text and audio information containing context in different conversation scenarios to form a data set and complete the construction of the data set;

[0009] Step 2: Data preprocessing;

[0010] Preprocess the constructed dataset to extract the corresponding modal features; for text information, extract the context features in each conversation; for audio information, extract the audio features of each sentence;

[0011] Step 3: Model construction;

[0012] Build an audio context-aware network, a cross-modal shared attention network, and an adaptive fusion network respectively, and then connect the three networks in sequence to form an initial sentiment classification model;

[0013] Step 4: Model training;

[0014] S41: Set the maximum number of training rounds and other hyperparameters required for training;

[0015] S42: adjusting the data set in step 1, and dividing the adjusted data set into a training set, a validation set, and a test set according to a set ratio;

[0016] S43: Start end-to-end training of the initial sentiment classification model;

[0017] The following operations are performed in each round of training: determine whether the learning rate needs to be adjusted in the current round, input the text and audio feature data in the training set as input data to the initial sentiment classification model, predict and generate and output predefined sentiment categories through the initial sentiment classification model, and at the same time, calculate the error between the sentiment category predicted by the initial sentiment classification model and the original category, calculate the loss function and back-propagate to update the parameters of each layer, use the current initial sentiment classification model to make predictions on the validation set, record the current loss, and compare it with the minimum loss. When the minimum loss is updated, first save the initial sentiment classification model in the training process, and then conduct the next round of training. If the minimum loss is not updated, do not save the initial sentiment classification model in the training process, and directly conduct the next round of training; at the same time, calculate the difference between the loss function of the current round and the previous round, and compare the difference with the convergence threshold to determine whether it converges. When it converges, first save the initial sentiment classification model in the training process, and then conduct the next round of training. When it does not converge, do not save the initial sentiment classification model in the training process, and directly conduct the next round of training;

[0018] S44: When the loss function converges and reaches the maximum training round, the initial sentiment classification model is trained and a trained sentiment classification model is obtained;

[0019] Step 5: Save the model;

[0020] Save the trained sentiment classification model to form a sentiment recognition model;

[0021] Step 6: Emotion recognition;

[0022] S61: extracting features from the collected text and audio data using a feature encoder to obtain text and audio features;

[0023] S62: Inputting the text and audio features into the emotion recognition model, and using the emotion recognition model to recognize and output a predefined situation category;

[0024] Furthermore, in order to effectively improve the situation recognition performance of the emotion recognition model and the robustness of the emotion recognition model, in step three, the audio context perception network is composed of a position embedding layer and five encoding layers connected in sequence, wherein the encoding layer is composed of two residual structures connected in sequence, the first residual structure is composed of a multi-head attention network layer and a layer normalization network layer, and the second residual structure is composed of a feedforward neural network layer and a layer normalization network layer, so as to realize the aggregation of emotional information in the audio;

[0025] The cross-modal shared attention network is composed of 2 position embedding layers, 5 encoding layers and 5 decoding layers, wherein each encoder layer is connected in sequence by 2 residual structures, the first residual structure is composed of 1 multi-head attention network layer, and the second residual structure is composed of 1 feedforward neural network layer and 1 layer normalization network layer, so as to realize the emotion convergence of modality 1 signal; each decoding layer is connected in sequence by 3 residual structures, the first residual structure is composed of 1 multi-head attention network layer and 1 layer normalization network layer, the second residual structure is composed of 1 cross-modal multi-head attention network layer and 1 layer normalization network layer, so as to realize modality 2 to perceive the emotion information of modality 1, and the third residual structure is composed of 1 feedforward neural network layer and 1 layer normalization network layer, so as to realize the emotion information of modality 1 based on the convergence of emotion information of modality 2;

[0026] The adaptive fusion network includes an additive attention layer and a feedforward neural network layer, which is used to fuse the features of time scale.

[0027] Furthermore, in order to effectively improve the context perception ability of the sentiment classification model, in step one, in the process of constructing the dataset, text and audio information in different dialogue scenarios are collected in pairs, and the text and audio information in the scene are segmented at the sentence level to form a dataset with context.

[0028] Furthermore, in order to effectively extract text and audio features to ensure better training results in the later stage, in step 2, the RoBERTa pre-trained model is used to extract text features, and the Librosa sampling module with a sampling rate of 16000 and the Wav2vec2 pre-trained model are used to extract audio features.

[0029] Furthermore, in order to ensure the training effect, in step 4 S41, the maximum number of training rounds is set to be greater than 12 times.

[0030] Furthermore, in order to ensure that the trained sentiment classification model has good denoising performance, in step 4 S41, other hyperparameters required for training include the initial learning rate, where the initial learning rate is set to 7e-6; in step 4 S43, a dynamic learning rate warm-up and decay strategy is adopted during the training process.

[0031] In the present invention, during the data acquisition process, for different dialogue scenarios, text and audio data containing context are constructed to form a data set, which is conducive to effectively improving the model's processing capabilities for different dialogue scenarios in the subsequent training process and improving the generalization performance. By preprocessing the constructed data set to extract the features of the corresponding modality, the quality of the extracted feature data can be effectively improved. In the model construction process, an audio context-aware network is introduced into the constructed emotion classification model. In this way, when the audio features are input into the multi-head attention mechanism network layer contained in the audio context-aware network, the current audio features can be respectively passed into the scaled dot product network layer in the multi-head attention mechanism network layer using three fully connected mappings as query vectors (Q), key vectors (K), and value vectors (V), so that the audio features can perceive the emotional information in the context and converge, and then pass through the layer normalization network to normalize the audio features to the standard range to facilitate comparison with the original features, and then use residual connections to converge the audio of the context emotion. Features and original audio features, so that the model can more effectively learn the differences between the two features, combined with richer emotional information; through the feedforward neural network layer connected to the output of the residual structure, first use a fully connected layer to map the 768-dimensional audio features to a 3072-dimensional feature space, and then map the 3072-dimensional audio features to a 768-dimensional audio space, so that the model can learn potential emotional features under different feature spaces, and finally uniformly map them to a 768-dimensional feature space, and then pass through a layer normalization network layer to facilitate residual connection with the features before entering the feedforward neural network layer to focus on learning feature differences. Therefore, through the introduction of the audio context-aware network, the emotion recognition model can better gather emotional information in the context and improve the performance of the model. At the same time, a cross-modal shared attention network layer is introduced into the emotion recognition model, wherein the structure of the encoder is similar to that of the encoder in the audio context-aware network, and the differences of the decoder are as follows: a residual structure and a third residual structure in the decoder are also similar to the first residual structure and the second residual structure in the audio context-aware network, and the second residual structure in the decoder maps the features of the current modality 1 using two fully connected network layers as key vectors (K) and value vectors (V), respectively, and maps the features of the current modality 2 using a fully connected network layer as a query vector (Q), so that modality 2 perceives the emotional features in modality 1, and the two modalities are learned using the same set of parameters, so that the features of modality 1 and modality 2 are close to the same emotional features in the same feature space, and different emotional features remain independent, so that the model converges features of different modalities while retaining their independence, and through the stacking of encoders and decoders, the model can fuse features at different levels. In this way, the introduction of the audio context-aware network can effectively improve the robustness of the emotion recognition model.Finally, an adaptive fusion network layer is introduced into the emotion recognition model. First, the time scale features are fused through the additive attention mechanism, and then a fully connected network layer is used to identify the emotion of the data. In the process of model training, the error between the category predicted by the model and the original category is calculated in each round, the loss function is calculated and the parameters of each layer are updated by back propagation. The current model is used to predict on the validation data set, and the current loss is recorded and compared with the minimum loss. When the minimum loss is updated, the model in the training process is saved first, and then the next round of training is carried out. However, when the minimum loss is not updated, the model in the training process is not saved, and the next round of training is carried out directly until all epoch training is completed. In this way, the emotion recognition model saved at the end can have better emotion recognition performance.

[0032] This method uses attention mechanism and weight sharing to make the common features of the two modalities close to each other in the feature space, while keeping their own exclusive features unaffected. This method ensures the effective alignment and fusion of features, significantly improves the overall performance of the ERC task, and at the same time solves the problem that the current emotion recognition methods based on various neural networks do not adequately consider the heterogeneity of different modalities in the feature space and cannot effectively exchange emotional information between multiple modalities, resulting in poor performance in the final fusion stage. The present invention not only effectively improves the robustness of the emotion recognition algorithm, but also can effectively solve the problem of spatial heterogeneity, thereby having a very excellent recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of the present invention;

[0034] Figure 2 It is a framework diagram of the emotion recognition model in the present invention;

[0035] Figure 3 It is a structural diagram of the cross-modal shared attention network (CMS-Attention) in the present invention;

[0036] Figure 4 It is a structural diagram of the multi-head attention network in the present invention;

[0037] Figure 5 It is a structural diagram of the scaled dot product network in the present invention. DETAILED DESCRIPTION

[0038] The present invention will be further described below in conjunction with the accompanying drawings.

[0039] like Figures 1 to 5 As shown, the present invention provides a conversation emotion recognition method based on context perception and cross-modal shared attention, comprising the following steps:

[0040] Step 1: Data acquisition;

[0041] The text and audio information containing context in different dialogue scenarios are collected to form a data set, and the data set is constructed; the data set constructed in one embodiment of the present application is shown in Table 1;

[0042] Table 1: Datasets containing text and audio

[0043]

[0044] Step 2: Data preprocessing;

[0045] Preprocess the constructed dataset to extract the corresponding modal features; for text information, extract the context features in each conversation; for audio information, extract the audio features of each sentence;

[0046] Step 3: Model construction;

[0047] Build an audio context-aware network, a cross-modal shared attention network, and an adaptive fusion network respectively, and then connect the three networks in sequence to form an initial sentiment classification model;

[0048] Step 4: Model training;

[0049] S41: Set the maximum number of training rounds and other hyperparameters required for training;

[0050] S42: adjusting the data set in step 1, and dividing the adjusted data set into a training set, a validation set, and a test set according to a set ratio;

[0051] S43: Start end-to-end training of the initial sentiment classification model; during the training, the initial sentiment classification model inputs the features extracted by the text feature encoder and the audio feature encoder, and outputs the predefined sentiment category;

[0052] The following operations are performed in each round of training: determine whether the learning rate needs to be adjusted in the current round, input the text and audio feature data in the training set as input data to the initial sentiment classification model, predict and generate and output predefined sentiment categories through the initial sentiment classification model, and at the same time, calculate the error between the sentiment category predicted by the initial sentiment classification model and the original category, calculate the loss function and back-propagate to update the parameters of each layer, use the current initial sentiment classification model to make predictions on the validation set, record the current loss, and compare it with the minimum loss. When the minimum loss is updated, first save the initial sentiment classification model in the training process, and then conduct the next round of training. If the minimum loss is not updated, do not save the initial sentiment classification model in the training process, and directly conduct the next round of training; at the same time, calculate the difference between the loss function of the current round and the previous round, and compare the difference with the convergence threshold to determine whether it converges. When it converges, first save the initial sentiment classification model in the training process, and then conduct the next round of training. When it does not converge, do not save the initial sentiment classification model in the training process, and directly conduct the next round of training;

[0053] S44: When the loss function converges and reaches the maximum training round, the initial sentiment classification model is trained and a trained sentiment classification model is obtained;

[0054] Step 5: Save the model;

[0055] Save the trained sentiment classification model to form a sentiment recognition model;

[0056] Step 6: Emotion recognition;

[0057] S61: using a feature encoder to perform feature extraction on the collected text and audio data (the data in the divided test set may be used) to obtain text and audio features;

[0058] S62: Inputting text and audio features into the emotion recognition model, using the emotion recognition model for recognition, the recognition process is a forward propagation process, and finally outputting a predefined situation category;

[0059] After using the emotion recognition model of this application to identify the situation of the data set in Table 1, the output results are shown in Table 2.

[0060] Table 2: Results of emotion recognition on the text and audio data in Table 1

[0061]

[0062] In order to effectively improve the situation recognition performance of the emotion recognition model and the robustness of the emotion recognition model, in step three, the audio context perception network is named ACA, which is composed of a position embedding layer and 5 encoding layers connected in sequence, wherein the encoding layer is connected in sequence by two residual structures, the first residual structure is composed of a multi-head attention network layer and a layer normalization network layer, and the second residual structure is composed of a feedforward neural network layer and a layer normalization network layer, so as to realize the aggregation of emotional information in the audio;

[0063] The cross-modal shared attention network is named CMS-Attention, which consists of 2 position embedding layers, 5 encoding layers and 5 decoding layers, wherein each encoder layer is connected in sequence by 2 residual structures, the first residual structure is composed of 1 multi-head attention network layer, and the second residual structure is composed of 1 feedforward neural network layer and 1 layer normalization network layer, so as to realize the emotion convergence of modality 1 signal; each decoding layer is connected in sequence by 3 residual structures, the first residual structure is composed of 1 multi-head attention network layer and 1 layer normalization network layer, the second residual structure is composed of 1 cross-modal multi-head attention network layer and 1 layer normalization network layer, so as to realize modality 2 to perceive the emotion information of modality 1, and the third residual structure is composed of 1 feedforward neural network layer and 1 layer normalization network layer, so as to realize the emotion information of modality 1 based on the convergence of emotion information of modality 2;

[0064] The adaptive fusion network is named A-Fusion, which includes 1 additive attention layer and 1 feedforward neural network layer for fusing the features of time scale.

[0065] Since a scene contains many sentences, the sentiment of each sentence is different. In order to effectively improve the context perception ability of the sentiment classification model, in step 1, when constructing the dataset, the text and audio information in different dialogue scenarios are collected in pairs, and the text and audio information in the scene are segmented at the sentence level to form a dataset with context.

[0066] In order to effectively extract text and audio features to ensure better training results in the later stage, in step 2, the RoBERTa pre-trained model is used to extract text features, and the Librosa sampling module with a sampling rate of 16000 and the Wav2vec2 pre-trained model are used to extract audio features.

[0067] In order to ensure the training effect, in step 4 S41, the maximum number of training rounds is set to be greater than 12 times.

[0068] In order to ensure that the trained sentiment classification model has good denoising performance, in step 4 S41, other hyperparameters required for training include the initial learning rate, where the initial learning rate is set to 7e-6; in step 4 S43, a dynamic learning rate warm-up and decay strategy is adopted during the training process.

[0069] In the present invention, during the data acquisition process, for different dialogue scenarios, text and audio data containing context are constructed to form a data set, which is conducive to effectively improving the model's processing capabilities for different dialogue scenarios in the subsequent training process and improving the generalization performance. By preprocessing the constructed data set to extract the features of the corresponding modality, the quality of the extracted feature data can be effectively improved. In the model construction process, an audio context-aware network is introduced into the constructed emotion classification model. In this way, when the audio features are input into the multi-head attention mechanism network layer contained in the audio context-aware network, the current audio features can be respectively passed into the scaled dot product network layer in the multi-head attention mechanism network layer using three fully connected mappings as query vectors (Q), key vectors (K), and value vectors (V), so that the audio features can perceive the emotional information in the context and converge, and then pass through the layer normalization network to normalize the audio features to the standard range to facilitate comparison with the original features, and then use residual connections to converge the audio of the context emotion. Features and original audio features, so that the model can more effectively learn the differences between the two features, combined with richer emotional information; through the feedforward neural network layer connected to the output of the residual structure, first use a fully connected layer to map the 768-dimensional audio features to a 3072-dimensional feature space, and then map the 3072-dimensional audio features to a 768-dimensional audio space, so that the model can learn potential emotional features under different feature spaces, and finally uniformly map them to a 768-dimensional feature space, and then pass through a layer normalization network layer to facilitate residual connection with the features before entering the feedforward neural network layer to focus on learning feature differences. Therefore, through the introduction of the audio context-aware network, the emotion recognition model can better gather emotional information in the context and improve the performance of the model. At the same time, a cross-modal shared attention network layer is introduced into the emotion recognition model, wherein the structure of the encoder is similar to that of the encoder in the audio context-aware network, and the differences of the decoder are as follows: a residual structure and a third residual structure in the decoder are also similar to the first residual structure and the second residual structure in the audio context-aware network, and the second residual structure in the decoder maps the features of the current modality 1 using two fully connected network layers as key vectors (K) and value vectors (V), respectively, and maps the features of the current modality 2 using a fully connected network layer as a query vector (Q), so that modality 2 perceives the emotional features in modality 1, and the two modalities are learned using the same set of parameters, so that the features of modality 1 and modality 2 are close to the same emotional features in the same feature space, and different emotional features remain independent, so that the model converges features of different modalities while retaining their independence, and through the stacking of encoders and decoders, the model can fuse features at different levels. In this way, the introduction of the audio context-aware network can effectively improve the robustness of the emotion recognition model.Finally, an adaptive fusion network layer is introduced into the emotion recognition model. First, the time scale features are fused through the additive attention mechanism, and then a fully connected network layer is used to identify the emotion of the data. In the process of model training, the error between the category predicted by the model and the original category is calculated in each round, the loss function is calculated and the parameters of each layer are updated by back propagation. The current model is used to predict on the validation data set, and the current loss is recorded and compared with the minimum loss. When the minimum loss is updated, the model in the training process is saved first, and then the next round of training is carried out. However, when the minimum loss is not updated, the model in the training process is not saved, and the next round of training is carried out directly until all epoch training is completed. In this way, the emotion recognition model saved at the end can have better emotion recognition performance.

[0070] This method uses attention mechanism and weight sharing to make the common features of the two modalities close to each other in the feature space, while keeping their own exclusive features unaffected. This method ensures the effective alignment and fusion of features, significantly improves the overall performance of the ERC task, and at the same time solves the problem that the current emotion recognition methods based on various neural networks do not adequately consider the heterogeneity of different modalities in the feature space and cannot effectively exchange emotional information between multiple modalities, resulting in poor performance in the final fusion stage. The present invention not only effectively improves the robustness of the emotion recognition algorithm, but also can effectively solve the problem of spatial heterogeneity, thereby having a very excellent recognition effect.

Claims

1. A conversation emotion recognition method based on context awareness and cross-modal shared attention, characterized in that: The following steps are involved: Step 1: Data acquisition; Collect text and audio information containing context in different conversation scenarios to form a data set and complete the construction of the data set; Step 2: Data preprocessing; Preprocess the constructed dataset to extract corresponding modal features; For text information, extract the context features in each conversation; Audio information, extract the audio features of each sentence; Step 3: Model construction; Build an audio context-aware network, a cross-modal shared attention network, and an adaptive fusion network respectively, and then connect the three networks in sequence to form an initial sentiment classification model; Step 4: Model training; S41: Set the maximum number of training rounds and other hyperparameters required for training; S42: adjusting the data set in step 1, and dividing the adjusted data set into a training set, a validation set, and a test set according to a set ratio; S43: Start end-to-end training of the initial sentiment classification model; The following operations are performed in each round of training: determine whether the learning rate needs to be adjusted in the current round, input the text and audio feature data in the training set as input data to the initial sentiment classification model, predict and generate and output predefined sentiment categories through the initial sentiment classification model, and at the same time, calculate the error between the sentiment category predicted by the initial sentiment classification model and the original category, calculate the loss function and back-propagate to update the parameters of each layer, use the current initial sentiment classification model to make predictions on the validation set, record the current loss, and compare it with the minimum loss. When the minimum loss is updated, first save the initial sentiment classification model in the training process, and then conduct the next round of training. If the minimum loss is not updated, do not save the initial sentiment classification model in the training process, and directly conduct the next round of training; at the same time, calculate the difference between the loss function of the current round and the previous round, and compare the difference with the convergence threshold to determine whether it converges. When it converges, first save the initial sentiment classification model in the training process, and then conduct the next round of training. When it does not converge, do not save the initial sentiment classification model in the training process, and directly conduct the next round of training; S44: When the loss function converges and reaches the maximum training round, the initial sentiment classification model is trained and a trained sentiment classification model is obtained; Step 5: Save the model; Save the trained sentiment classification model to form a sentiment recognition model; Step 6: Emotion recognition; S61: extracting features from the collected text and audio data using a feature encoder to obtain text and audio features; S62: Input the text and audio features into the emotion recognition model, and use the emotion recognition model to identify and output predefined situation categories.

2. According to claim 1, a method for conversation emotion recognition based on context awareness and cross-modal shared attention is characterized in that: In step three, the audio context perception network is composed of a position embedding layer and five encoding layers connected in sequence, wherein the encoding layer is composed of two residual structures connected in sequence, the first residual structure is composed of a multi-head attention network layer and a layer normalization network layer, and the second residual structure is composed of a feedforward neural network layer and a layer normalization network layer, so as to realize the aggregation of emotional information in the audio; The cross-modal shared attention network is composed of 2 position embedding layers, 5 encoding layers and 5 decoding layers, wherein each encoder layer is connected in sequence by 2 residual structures, the first residual structure is composed of 1 multi-head attention network layer, and the second residual structure is composed of 1 feedforward neural network layer and 1 layer normalization network layer, so as to realize the emotion convergence of modality 1 signal; each decoding layer is connected in sequence by 3 residual structures, the first residual structure is composed of 1 multi-head attention network layer and 1 layer normalization network layer, the second residual structure is composed of 1 cross-modal multi-head attention network layer and 1 layer normalization network layer, so as to realize modality 2 to perceive the emotion information of modality 1, and the third residual structure is composed of 1 feedforward neural network layer and 1 layer normalization network layer, so as to realize the emotion information of modality 1 based on the convergence of emotion information of modality 2; The adaptive fusion network includes an additive attention layer and a feedforward neural network layer, which is used to fuse the features of time scale.

3. According to claim 1, a method for conversation emotion recognition based on context awareness and cross-modal shared attention is characterized in that: In step one, in the process of constructing a dataset, text and audio information in different dialogue scenarios are collected in pairs, and the text and audio information in the scene are segmented at the sentence level to form a dataset with context.

4. According to claim 1, a method for conversation emotion recognition based on context awareness and cross-modal shared attention is characterized in that: In step 2, the RoBERTa pre-trained model is used to extract text features, and the Librosa sampling module with a sampling rate of 16000 and the Wav2vec2 pre-trained model are used to extract audio features.

5. According to claim 1, a method for conversation emotion recognition based on context awareness and cross-modal shared attention is characterized in that: In step 4 S41, the maximum number of training rounds is set to be greater than 12 times.

6. According to claim 1, a method for conversation emotion recognition based on context awareness and cross-modal shared attention is characterized in that: In step 4 S41, other hyperparameters required for training include an initial learning rate, wherein the initial learning rate is set to 7e-6; in step 4 S43, a dynamic learning rate warm-up and decay strategy is adopted during the training process.

Images