Context-aware and cross-modal shared attention based dialogue sentiment recognition method

Abstract

A dialogue emotion recognition method based on context awareness and cross-modal shared attention, collects context-containing text and audio information under different dialogue scenarios to form a data set, and completes the construction of the data set; the constructed data set is preprocessed to extract corresponding modal features; an audio context awareness network, a cross-modal shared attention network and a self-adaptive fusion network are respectively built, and the built networks are sequentially connected to form an initial emotion recognition model; the initial emotion recognition model is trained end-to-end, and the trained emotion classification model is saved to form an emotion recognition model; the collected text and audio data are feature-extracted by using a feature encoder to obtain text and audio features; the text and audio features are input into the emotion recognition model, and the emotion recognition model is used to recognize and output a predefined condition category. The method can effectively solve the problem of spatial heterogeneity and has excellent recognition effect.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer natural language processing, and particularly relates to a dialogue emotion recognition method based on context perception and cross-modal shared attention. BACKGROUND

[0002] The goal of Emotion Recognition in Conversation (ERC) is to identify the emotional label of each utterance in a conversation. By providing contextually rich emotional responses, ERC can significantly enhance the user experience in conversational systems. This capability plays a key role in multiple domains such as chatbots, recommendation systems, and medical systems. However, the ERC task faces several challenges. Relying solely on textual data can lead to inaccurate recognition, as similar statements can convey completely different emotions. Therefore, employing multiple modalities (e.g., text, audio, and video) is crucial for a comprehensive and accurate understanding of emotional expressions in conversations.

[0003] According to the difference of input modalities, emotion recognition models can be divided into two categories: text-based methods and multi-modal methods. Text-based emotion recognition models mainly focus on dialogue context, speaker modeling, and common sense knowledge. Although this type of model can effectively capture text emotions in certain scenarios, it may not fully understand emotions when encountering ambiguous or complex emotional expressions. Multi-modal emotion recognition models integrate text, audio, and visual information to more comprehensively understand emotional expressions in conversations. Current models mainly focus on multi-modal feature extraction, interaction, and fusion.

[0004] In multimodal feature extraction, different types of data (such as text, audio, and video) require different tools and models for processing. For example, text features can be extracted using BERT or RoBERTa models, audio features can be extracted using Librosa or Wav2Vec2, and facial expressions and gestures in videos can be extracted using tools such as MTCNN or OpenFace. These tools can effectively capture emotional expressions across different modalities. Regarding multimodal feature interaction, existing technologies use neural network methods such as RNN, LSTM, GRU, and GCN for feature interaction; however, these existing works do not fully consider the heterogeneity of different modalities in the feature space. In the field of multimodal fusion, some methods integrate features through early fusion techniques, that is, concatenating the original features from different modalities before processing and inputting them into a shared network. Other methods employ late fusion strategies, allowing independent modality-specific networks to process features separately and then concatenating their outputs later. However, this simple concatenation cannot effectively fuse multimodal features located in different feature spaces. Therefore, there is an urgent need to provide a dialogue emotion recognition method based on context awareness and cross-modal shared attention. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a dialogue emotion recognition method based on context awareness and cross-modal shared attention. This method effectively solves the problems of insufficient consideration of the heterogeneity of different modalities in the feature space and the inability to effectively interact emotional information between multiple modalities in current emotion recognition methods based on various neural networks, resulting in poor performance in the final fusion stage. At the same time, this method can effectively improve the robustness of the emotion recognition algorithm and effectively solve the spatial heterogeneity problem, thus exhibiting excellent recognition results.

[0006] To address the aforementioned technical problems, this invention provides a dialogue emotion recognition method based on context awareness and cross-modal shared attention, comprising the following steps:

[0007] Step 1: Data Acquisition;

[0008] Collect contextual text and audio information from different dialogue scenarios to form a dataset, and complete the construction of the dataset;

[0009] Step 2: Data preprocessing;

[0010] The constructed dataset is preprocessed to extract the corresponding modal features; for text information, the context features of each dialogue are extracted; for audio information, the audio features of each sentence are extracted.

[0011] Step 3: Model Building;

[0012] An audio context perception network, a cross-modal shared attention network and an adaptive fusion network are built, and the three networks are connected in sequence to form an initial emotion classification model;

[0013] Step four: model training;

[0014] S41: setting the maximum number of training rounds and other hyperparameters required for training;

[0015] S42: adjusting the data set in step one, and dividing the adjusted data set into a training set, a validation set and a test set according to the set proportion;

[0016] S43: starting end-to-end training of the initial emotion classification model;

[0017] In each round of training, the following operations are performed: determining whether the learning rate needs to be adjusted in the current round, inputting the text and audio feature data in the training set as input data into the initial emotion classification model, generating and outputting the predefined emotion category through the initial emotion classification model, calculating the error between the emotion category generated by the initial emotion classification model and the original category, calculating the loss function and updating the parameters of each layer through back propagation, predicting the current initial emotion classification model on the validation set, recording the current loss, and comparing it with the minimum loss, when the minimum loss is updated, the initial emotion classification model in the training process is saved first, and then the next round of training is performed, if the minimum loss is not updated, the initial emotion classification model in the training process is not saved, and the next round of training is directly performed; at the same time, the difference between the loss functions of the current round and the last round is calculated, and the difference and the convergence threshold are compared to determine whether convergence is achieved, when convergence is achieved, the initial emotion classification model in the training process is saved first, and then the next round of training is performed, when convergence is not achieved, the initial emotion classification model in the training process is not saved, and the next round of training is directly performed;

[0018] S44: when the loss function converges and reaches the maximum training round, the initial emotion classification model training is completed, and a trained emotion classification model is obtained;

[0019] Step five: model saving;

[0020] The trained emotion classification model is saved to form an emotion recognition model;

[0021] Step six: emotion recognition;

[0022] S61: using a feature encoder to extract features from the collected text and audio data to obtain text and audio features;

[0023] S62: input the text and audio features into the emotion recognition model, recognize and output the predefined case category by using the emotion recognition model;

[0024] Further, in order to effectively improve the case recognition performance of the emotion recognition model, and at the same time, to effectively improve the robustness of the emotion recognition model, in step three, the audio context perception network is composed of one position embedding layer and five encoding layers connected in turn, wherein the encoding layer is connected by two residual structures in turn, the first residual structure is composed of one multi-head attention network layer and one layer normalization network layer, and the second residual structure is composed of one feedforward neural network layer and one layer normalization network layer, so as to realize the aggregation of emotional information in the audio.

[0025] The cross-modal shared attention network is composed of two position embedding layers and five encoding layers and five decoding layers, wherein each encoder layer is connected by two residual structures in turn, the first residual structure is composed of one multi-head attention network layer, and the second residual structure is composed of one feedforward neural network layer and one layer normalization network layer, so as to realize the aggregation of emotional information in the audio.

[0026] The adaptive fusion network includes one additive attention layer and one feedforward neural network layer, which is used for fusing the features of the time scale.

[0027] Further, in order to effectively improve the context perception ability of the emotion classification model, in step one, during the construction of the data set, the text and audio information in different dialogue scenes are collected in pairs, and the scene text and audio information are processed by sentence-level segmentation to form a data set with context.

[0028] Further, in order to effectively extract text and audio features to ensure better training effect in the later stage, in step two, RoBERTa pre-training model is used to extract text features, Librosa sampling module with sampling rate of 16000 and Wav2vec2 pre-training model are used to extract audio features.

[0029] Further, in order to ensure the training effect, in S41 of step four, the maximum number of training is set to be greater than 12 times.

[0030] Further, in order to ensure that the trained emotion classification model has good denoising performance, in S41 of step four, other hyperparameters required for training include an initial learning rate, wherein the initial learning rate is set to 7e-6; in S43 of step four, a dynamic learning rate warm-up and decay strategy is adopted in the training process.

[0031] In the present application, in the data acquisition process, different dialogue scenes are constructed to form a data set containing context text and audio data, which is beneficial to effectively improve the processing capacity of the model for different dialogue scenes and improve the generalization performance in the subsequent training process. By preprocessing the constructed data set to extract the corresponding modal features, the quality of the extracted feature data can be effectively improved. In the process of model construction, the audio context perception network is introduced into the constructed emotion classification model. When the audio features are input into the multi-head attention mechanism network layer contained in the audio context perception network, the current audio features are mapped as query vector (Q), key vector (K) and value vector (V) into the scaling dot product network layer in the multi-head attention mechanism network layer, so that the audio features can perceive the emotional information in the context and converge, and then pass through the layer normalization network to standardize the audio features to a standard range for easy comparison with the original features. Subsequently, the context emotional audio features and the original audio features are converged using a residual connection, so that the model can more effectively learn the differences between the two features and combine more rich emotional information. The output end of the residual structure is connected through a feedforward neural network layer. First, a fully connected layer is used to map the 768-dimensional audio features to a 3072-dimensional feature space, and then the 3072-dimensional audio features are mapped to a 768-dimensional audio space, so that the model can learn the latent emotional features in different feature spaces. Finally, the features are uniformly mapped to a 768-dimensional feature space, and then pass through the layer normalization network layer to facilitate residual connection learning of feature differences with the features before entering the feedforward neural network layer. Therefore, by introducing the audio context perception network, the emotion recognition model can better converge the emotional information in the context and improve the performance of the model. At the same time, a cross-modal shared attention network layer is introduced into the emotion recognition model. The structure of the encoder is similar to that of the audio context perception network. The differences between the decoder are as follows: the first and second residual structures in the decoder are similar to the first and second residual structures in the audio context perception network, and the second residual structure in the decoder maps the features of the current modal 1 as key vector (K) and value vector (V) using two fully connected network layers, and maps the features of the current modal 2 as query vector (Q) using a fully connected network layer, so that modal 2 can perceive the emotional features in modal 1, and the two modalities are learned using the same set of parameters, so that the same emotional features of modal 1 and modal 2 are close in the same feature space, and the different emotional features remain independent, so that the model can fuse different levels of features while preserving their independence. Through the stacking of the encoder and the decoder, the robustness of the emotion recognition model can be effectively improved by introducing the audio context perception network.Finally, an adaptive fusion network layer is introduced in the emotion recognition model. First, the features of the time scale are fused through the additive attention mechanism, and then a fully connected network layer is used to identify the emotion of the data. During the training process of the model, the error between the predicted class and the original class is calculated every round, the loss function is calculated and the parameters of each layer are updated by back propagation, the current model is used to predict on the validation dataset, and the current loss is recorded and compared with the minimum loss. When the minimum loss is updated, the model in the training process is saved first, and then the next round of training is performed, but when the minimum loss is not updated, the model in the training process is not saved, and the next round of training is directly performed, until all epoch training is completed. In this way, the last saved emotion recognition model can have better emotion recognition performance.

[0032] The method utilizes attention mechanism and weight sharing to make the common features of the two modalities close to each other in the feature space while keeping their respective exclusive features unaffected. This method ensures effective alignment and fusion of features, significantly improving the overall performance of the ERC task. At the same time, it solves the problem of insufficient consideration of heterogeneity in feature space for different modalities in current emotion recognition methods based on various neural networks, which cannot effectively interact emotional information between multiple modalities, resulting in poor performance in the final fusion stage. The present application not only effectively improves the robustness of the emotion recognition algorithm, but also effectively solves the problem of spatial heterogeneity, thereby having very excellent recognition effect. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a flowchart of the present application;

[0034] Figure 2 is a framework structure diagram of the emotion recognition model in the present application;

[0035] Figure 3 is a structure diagram of the cross-modal shared attention network (CMS-Attention) in the present application;

[0036] Figure 4 is a structure diagram of the multi-head attention network in the present application;

[0037] Figure 5 is a structure diagram of the scaled dot-product network in the present application. DETAILED DESCRIPTION

[0038] The present application will be further described below in conjunction with the accompanying drawings.

[0039] As shown in Figures 1 to 5 , the present application provides a dialogue emotion recognition method based on context perception and cross-modal shared attention, comprising the following steps:

[0040] Step one: data acquisition;

[0041] Collecting texts and audio information containing context under different dialogue scenarios to form a data set and complete the construction of the data set; the data set constructed by an embodiment in the present application is shown in Table 1;

[0042] Table 1: data set containing text and audio

[0043]

[0044] Step two: data preprocessing;

[0045] The constructed data set is preprocessed to extract corresponding modal features; for text information, the context features in each dialogue are extracted; for audio information, the audio features of each sentence are extracted;

[0046] Step three: model construction;

[0047] An audio context perception network, a cross-modal shared attention network and an adaptive fusion network are respectively built, and the three networks built are sequentially connected to form an initial emotion classification model;

[0048] Step four: model training;

[0049] S41: setting the maximum number of training rounds and other hyperparameters required for training;

[0050] S42: adjusting the data set in step one, and dividing the adjusted data set into a training set, a validation set and a test set according to a set proportion;

[0051] S43: starting to train the initial emotion classification model end to end; the initial emotion classification model inputs the features extracted by the text feature encoder and the audio feature encoder during training, and outputs the predefined emotion category;

[0052] In each round of training, the following operations are performed: determining whether the learning rate needs to be adjusted in the current round, inputting the text and audio feature data in the training set as input data into the initial emotion classification model, generating and outputting the predefined emotion category through the initial emotion classification model, calculating the error of the emotion category generated by the initial emotion classification model and the original category, calculating the loss function and updating the parameters of each layer through back propagation, predicting on the validation set using the current initial emotion classification model, recording the current loss, and comparing it with the minimum loss, when the minimum loss is updated, the initial emotion classification model in the training process is saved first, and then the next round of training is performed, if the minimum loss is not updated, the initial emotion classification model in the training process is not saved, and the next round of training is directly performed; at the same time, the difference between the loss functions of the current round and the last round is calculated, and the difference and the convergence threshold are compared to determine whether the convergence is reached, when the convergence is reached, the initial emotion classification model in the training process is saved first, and then the next round of training is performed, when the convergence is not reached, the initial emotion classification model in the training process is not saved, and the next round of training is directly performed;

[0053] S44: When the loss function converges and reaches the maximum training round, the initial emotion classification model training is completed, and a trained emotion classification model is obtained;

[0054] Step five: model saving;

[0055] The trained emotion classification model is saved to form an emotion recognition model;

[0056] Step six: emotion recognition;

[0057] S61: The feature encoder is used to extract features from the collected text and audio data (which can use the data in the divided test set), to obtain text and audio features;

[0058] S62: The text and audio features are input into the emotion recognition model, and the emotion recognition model is used for recognition. The recognition process is a forward propagation process, and finally the predefined situation category is output;

[0059] After the emotion recognition model of the present application is used to recognize the data set in Table 1, the output result is shown in Table 2.

[0060] Table 2: Result of emotion recognition on the text and audio data in Table 1

[0061]

[0062] In order to effectively improve the situation recognition performance of the emotion recognition model, and at the same time, effectively improve the robustness of the emotion recognition model, in step three, the audio context perception network is named ACA, which is composed of 1 position embedding layer and 5 encoding layers connected in turn, wherein the encoding layer is connected by 2 residual structures in turn, the first residual structure is composed of 1 multi-head attention network layer and 1 layer normalization network layer, and the second residual structure is composed of 1 feedforward neural network layer and 1 layer normalization network layer, so as to realize the aggregation of emotional information in the audio;

[0063] The cross-modal shared attention network is named CMS-Attention, which is composed of 2 position embedding layers and 5 encoding layers and 5 decoding layers, wherein each encoder layer is connected by 2 residual structures in turn, the first residual structure is composed of 1 multi-head attention network layer, and the second residual structure is composed of 1 feedforward neural network layer and 1 layer normalization network layer, so as to realize the aggregation of emotional information in the audio;

[0064] The adaptive fusion network is named A-Fusion, which includes 1 additive attention layer and 1 feedforward neural network layer, which is used for fusion of time scale features.

[0065] Since a scene contains many sentences, the emotion of each sentence is not the same, in order to effectively improve the context perception ability of the emotion classification model, in step one, during the construction of the data set, the text and audio information in different dialogue scenes are collected in pairs, and the text and audio information in the scene are processed by sentence-level segmentation, forming a data set with context.

[0066] In order to effectively extract text and audio features to ensure better training effect in the later stage, in step two, RoBERTa pre-training model is used to extract text features, Librosa sampling module with sampling rate of 16000 and Wav2vec2 pre-training model are used to extract audio features.

[0067] In order to ensure the training effect, in S41 of step four, the maximum number of training is set to be greater than 12 times.

[0068] In order to ensure that the trained emotion classification model has good denoising performance, in S41 of step four, other hyperparameters required for training include an initial learning rate, wherein the initial learning rate is set to 7e-6; in S43 of step four, a dynamic learning rate warm-up and decay strategy is adopted in the training process.

[0069] In the present application, in the data acquisition process, different dialogue scenes are constructed to form a data set containing context text and audio data, which is beneficial to effectively improve the processing capacity of the model for different dialogue scenes and improve the generalization performance in the subsequent training process. By preprocessing the constructed data set to extract the corresponding modal features, the quality of the extracted feature data can be effectively improved. In the process of model construction, the audio context perception network is introduced into the constructed emotion classification model. When the audio features are input into the multi-head attention mechanism network layer contained in the audio context perception network, the current audio features are mapped as query vector (Q), key vector (K) and value vector (V) into the scaling dot product network layer in the multi-head attention mechanism network layer, so that the audio features can perceive the emotional information in the context and converge, and then pass through the layer normalization network to standardize the audio features to a standard range for easy comparison with the original features. Subsequently, the context emotional audio features and the original audio features are converged using a residual connection, so that the model can more effectively learn the differences between the two features and combine more rich emotional information. The output end of the residual structure is connected through a feedforward neural network layer. First, a fully connected layer is used to map the 768-dimensional audio features to a 3072-dimensional feature space, and then the 3072-dimensional audio features are mapped to a 768-dimensional audio space, so that the model can learn the latent emotional features in different feature spaces. Finally, the features are uniformly mapped to a 768-dimensional feature space, and then pass through the layer normalization network layer to facilitate residual connection learning of feature differences with the features before entering the feedforward neural network layer. Therefore, by introducing the audio context perception network, the emotion recognition model can better converge the emotional information in the context and improve the performance of the model. At the same time, a cross-modal shared attention network layer is introduced into the emotion recognition model. The structure of the encoder is similar to that of the audio context perception network. The differences between the decoder are as follows: the first and second residual structures in the decoder are similar to the first and second residual structures in the audio context perception network, and the second residual structure in the decoder maps the features of the current modal 1 as key vector (K) and value vector (V) using two fully connected network layers, and maps the features of the current modal 2 as query vector (Q) using a fully connected network layer, so that modal 2 can perceive the emotional features in modal 1, and the two modalities are learned using the same set of parameters, so that the same emotional features of modal 1 and modal 2 are close in the same feature space, and the different emotional features remain independent, so that the model can fuse different levels of features while preserving their independence. Through the stacking of the encoder and the decoder, the robustness of the emotion recognition model can be effectively improved by introducing the audio context perception network.Finally, an adaptive fusion network layer is introduced in the emotion recognition model. First, the features of the time scale are fused through the additive attention mechanism, and then a fully connected network layer is used to identify the emotion of the data. During the model training process, the error between the predicted class and the original class is calculated every round, the loss function is calculated and the parameters of each layer are updated by back propagation. The current model is used to predict the validation dataset, and the current loss is recorded and compared with the minimum loss. When the minimum loss is updated, the model in the training process is saved first, and then the next round of training is performed. But when the minimum loss is not updated, the model in the training process is not saved, and the next round of training is directly performed. Until all the epoch training is completed, the last saved emotion recognition model can have better emotion recognition performance.

[0070] The method uses attention mechanism and weight sharing to make the common features of the two modalities close to each other in the feature space while keeping their respective exclusive features unaffected. This method ensures effective alignment and fusion of features, significantly improving the overall performance of the ERC task. At the same time, it solves the problem of insufficient consideration of heterogeneity in feature space for different modalities in current emotion recognition methods based on various neural networks, and cannot effectively interact emotional information between multiple modalities, resulting in poor performance in the final fusion stage. The present application not only effectively improves the robustness of the emotion recognition algorithm, but also effectively solves the problem of spatial heterogeneity, thereby having very excellent recognition effect.

Claims

1. A dialog sentiment recognition method based on context awareness and cross-modal shared attention, characterized in that, The method comprises the following steps: Step one: data acquisition; Collecting context-embedded texts and audio information in different dialogue scenarios to form a data set and complete the construction of the data set; Step two: data preprocessing; Preprocessing the constructed data set to extract corresponding modal features; for text information, extract the context features in each dialogue; for audio information, extract the audio features of each sentence; RoBERTa pre-training model is used to extract text features, Librosa sampling module with a sampling rate of 16000 and Wav2vec2 pre-training model are used to extract audio features; Step three: model construction; an audio context perception network, a cross-modal shared attention network and an adaptive fusion network are respectively built, and the three networks are connected in sequence to form an initial emotion classification model; the audio context perception network is composed of a position embedding layer and five encoding layers connected in sequence, wherein the encoding layer is composed of two residual structures connected in sequence, the first residual structure is composed of a multi-head attention network layer and a layer normalization network layer, and the second residual structure is composed of a feedforward neural network layer and a layer normalization network layer, so as to realize the aggregation of emotional information in audio; when the audio feature is input into the multi-head attention mechanism network layer contained in the audio context perception network, the current audio feature is mapped into three full connections as query vector Q, key vector K and value vector V, which are input into the scaling dot product network layer in the multi-head attention mechanism network layer, so that the audio feature can perceive the emotional information in the context and be gathered, and then pass through the layer normalization network, so that the audio feature is standardized to a standard range to facilitate comparison with the original feature, and then the context emotional audio feature and the original audio feature are gathered through residual connection, so that the model can more effectively learn the differences between the two features and combine more rich emotional information; the output end of the feedforward neural network layer is connected with the residual structure, a full connection layer is used to map the 768-dimensional audio feature to a 3072-dimensional feature space, and then the 3072-dimensional audio feature is mapped to a 768-dimensional audio space, so that the model can learn the potential emotional features in different feature spaces, and finally the features are uniformly mapped to a 768-dimensional feature space, and then pass through the layer normalization network layer to facilitate residual connection with the features before entering the feedforward neural network layer to focus on learning feature differences; ​ The cross-modal shared attention network is composed of 2 position embedding layers and 5 encoding layers and 5 decoding layers, wherein each encoder layer is sequentially connected by 2 residual structures, the first residual structure is composed of 1 multi-head attention network layer, and the second residual structure is composed of 1 feedforward neural network layer and 1 layer normalization network layer, so as to realize the emotion convergence of the mode 1 signal; each decoding layer is sequentially connected by 3 residual structures, the first residual structure is composed of 1 multi-head attention network layer and 1 layer normalization network layer, the second residual structure is composed of 1 cross-modal multi-head attention network layer and 1 layer normalization network layer, so as to realize the emotion information of mode 2 perceiving mode 1, and the third residual structure is composed of 1 feedforward neural network layer and 1 layer normalization network layer, so as to realize the emotion information of mode 2 perceiving mode 1 on the basis of converging the emotion information; the second residual structure in the decoder maps the current mode 1 feature into a key vector K and a value vector V by using two fully connected network layers respectively, and maps the current mode 2 feature into a query vector Q by using a fully connected network layer, so that mode 2 perceives the emotion feature in mode 1, and the two modes are learned by using the same set of parameters, so that the same emotion features of the features of mode 1 and mode 2 in the same feature space are close, and the different emotion features remain independent, so that the model converges different modal features while retaining their independence, and the stacking of the encoder and the decoder enables the model to fuse features at different levels; the adaptive fusion network comprises 1 additive attention layer and 1 feedforward neural network layer, the features of the time scale are fused by using the additive attention mechanism first, and then a fully connected network layer is used to identify the emotion of the data, and the features of the time scale are fused. Step four: model training; S41: set the maximum number of training rounds and other hyperparameters required for training; S42: adjust the data set in step one, and divide the adjusted data set into a training set, a validation set and a test set according to the set proportion; S43: start training the initial emotion classification model end to end; in each round of training, the following operations are performed: judge whether the learning rate needs to be adjusted in the current round, input the text and audio feature data in the training set as input data into the initial emotion classification model, generate and output the predefined emotion category through the initial emotion classification model, calculate the error of the emotion category generated by the initial emotion classification model and the original category, calculate the loss function and update the parameters of each layer by back propagation, use the current initial emotion classification model to predict on the validation set, record the current loss and compare it with the minimum loss, when the minimum loss is updated, save the initial emotion classification model in the training process first, and then perform the next round of training, if the minimum loss is not updated, do not save the initial emotion classification model in the training process, and directly perform the next round of training; at the same time, calculate the difference between the loss functions of the current round and the last round, and compare the difference with the convergence threshold to determine whether it converges, when it converges, save the initial emotion classification model in the training process first, and then perform the next round of training, when it does not converge, do not save the initial emotion classification model in the training process, and directly perform the next round of training; S44: when the loss function converges and reaches the maximum training round, the initial emotion classification model training is completed, and a trained emotion classification model is obtained; Step five: model saving; save the trained emotion classification model to form an emotion recognition model; Step six: emotion recognition; S61: use the feature encoder to extract features from the collected text and audio data to obtain text and audio features; S62: input the text and audio features into the emotion recognition model, and use the emotion recognition model to recognize and output the predefined situation category.

2. The dialog sentiment recognition method based on context awareness and cross-modal shared attention according to claim 1, characterized in that, In step one, during the construction of the data set, text and audio information under different dialogue scenes are collected in pairs, and the text and audio information in the scene are processed by sentence-level segmentation to form a data set with context.

3. The dialog sentiment recognition method based on context awareness and cross-modal shared attention according to claim 1, characterized in that, In S41 of step four, the maximum number of training rounds is set to more than 12 times.

4. The dialog sentiment recognition method based on context awareness and cross-modal shared attention according to claim 1, characterized in that, In S41 of step four, the other hyperparameters required for training include an initial learning rate, wherein the initial learning rate is set to 7e-6; in S43 of step four, a dynamic learning rate warm-up and decay strategy is used in the training process.

Images