Multi-modal emotion fusion and user emotion intention understanding method and system based on attention mechanism

Abstract

The invention provides a multi-modal emotion fusion and user emotion intention understanding method and system based on an attention mechanism, and relates to the technical field of multi-modal emotion recognition. According to the method, firstly, a multi-modal dialogue data set is constructed by collecting voice, text, image and physiological data, and data preprocessing is carried out; secondly, extracting and coding multi-modal emotion features by using an improved convolutional neural network (CNN) and a long-short term memory (LSTM) network, and generating a feature set; then, a dynamic modal weighting mechanism is adopted, the weight of each modal feature is adjusted in real time according to the current dialogue situation, and a feature set is optimized through an adaptive noise suppression strategy and a generative adversarial network; and finally, carrying out feature fusion through a mixed cross entropy attention mechanism, inputting the fused features into an emotion classifier, generating a probability distribution vector of an emotion category, and identifying a user intention so as to generate interaction feedback in real time and display the interaction feedback. According to the method, the sentiment analysis accuracy and response flexibility are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal emotion recognition, and in particular to a method and system for multimodal emotion fusion and user emotion intention understanding based on an attention mechanism. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, human-computer interaction (HCI) systems have been widely used in areas such as intelligent customer service, virtual assistants, and smart home control. These systems aim to enhance the naturalness and intelligence of interactions by sensing users' emotional states. However, traditional sentiment analysis methods often rely on data from a single modality (such as text, speech, or images), which limits the system's comprehensive understanding of emotions.

[0003] In multimodal sentiment analysis, data from different modalities, such as speech, text, and images, possess distinct feature representations and distributions. However, the heterogeneity between these modalities (e.g., differences in semantics and data structure) makes information fusion complex and challenging. For example, speech data emphasizes intonation and speech rate, text data contains semantic information, and images provide visual signals of emotional expression. Fusion of these modalities requires not only the effective extraction of their respective emotional features but also the overcoming of intermodal influences to ensure consistent and accurate sentiment analysis.

[0004] An existing patent (CN119293740B) proposes a multimodal conversational emotion recognition method that uses face recognition and gesture recognition to obtain expression timing features and gesture features. These features are then adaptively weighted and fused through an attention module to generate the final visual modality features, and cross-modal feature fusion is combined with text and speech modalities. This method effectively solves the problem of insufficient and inaccurate fusion caused by the heterogeneity between different modalities in traditional emotion recognition. However, although this method uses an attention mechanism to weight different emotional cues, the importance of different modal signals may change rapidly in scenarios with multiple rounds of conversation and multi-person interactions, and the current model is not able to respond to such rapid changes. In addition, for interactions involving multiple people, emotion recognition requires distinguishing and integrating emotional signals from different participants. However, the system fails to effectively solve the problem of noise information, which affects the accuracy of emotion recognition. Summary of the Invention

[0005] In order to overcome the shortcomings of the existing technology, the purpose of the present invention is to provide a multimodal emotion fusion and user emotion intention understanding method and system based on attention mechanism.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A multimodal emotion fusion and user emotional intention understanding method based on attention mechanism, including:

[0008] Collect multimodal conversation datasets;

[0009] performing data preprocessing on the multimodal conversation dataset to obtain a preprocessed conversation dataset;

[0010] Based on the improved convolutional neural network and the improved long short-term memory network, the emotional features of each modality in the preprocessed dialogue dataset are extracted and encoded to obtain a multimodal feature set;

[0011] Based on the current conversation context, a dynamic modality weighting mechanism is used to adjust the weights of each modality feature in the multimodal feature set;

[0012] Adopting an adaptive noise suppression strategy and utilizing a generative adversarial network to obtain an enhanced feature set based on the multimodal feature set;

[0013] According to the hybrid cross entropy attention mechanism, the weights corresponding to each modality in the current multimodality are used to perform feature fusion to obtain a fused feature;

[0014] Input the fused features into the sentiment classifier to generate the probability distribution vector of the user's sentiment category and identify the user's intention;

[0015] Generate and display interactive feedback in real time based on user intent.

[0016] Preferably, the multimodal dialogue dataset includes:

[0017] Voice, physiological data, text and image data of user interaction.

[0018] Preferably, the performing data preprocessing on the multimodal conversation dataset to obtain a preprocessed conversation dataset includes:

[0019] Performing noise elimination and segmentation slicing processing on the voice data to obtain preprocessed voice data;

[0020] Perform word segmentation, stop word removal, and word embedding processing on the text data to obtain a text vector representation;

[0021] Performing image scaling and standardization on the image data and using data enhancement technology to obtain pre-processed image data;

[0022] performing standardization and normalization processing on the physiological data to obtain preprocessed physiological data;

[0023] A preprocessed dialogue data set is determined based on the preprocessed speech data, the preprocessed image data, the preprocessed physiological data and the text vector representation.

[0024] Preferably, the improved convolutional neural network and the improved long short-term memory network are used to extract and encode the emotional features of each modality in the preprocessed conversation dataset to obtain a multimodal feature set, including:

[0025] Build improved convolutional neural networks and improved long short-term memory networks;

[0026] Inputting the preprocessed speech data and the preprocessed image data into the improved convolutional neural network to obtain speech emotion features and image emotion features, wherein the improved convolutional neural network includes: a first input layer, a bidirectional convolution layer, a pooling layer, a global pooling average layer, a fully connected layer and a first output layer;

[0027] Inputting the preprocessed physiological data and the text vector representation into the improved long short-term memory network to obtain text emotion features and physiological emotion features, wherein the improved long short-term memory network includes: a second input layer, an embedding layer, a bidirectional LSTM layer, a feature extraction layer, and a second output layer;

[0028] The speech emotion features, image emotion features, text emotion features and physiological emotion features are tokenized and input into MLP for encoding processing to obtain a multimodal feature set.

[0029] Preferably, the expression of the multimodal feature set is:

[0030]

[0031] Among them, E m is a multimodal feature set, To extract the speech emotion features, To extract the emotional features of the image, is the extracted text sentiment feature, is the extracted physiological emotion feature, Γ is the feature tokenization layer, MLP is the multi-layer perceptron encoder, and ‖ is the feature concatenation.

[0032] Preferably, the method of adjusting the weight of each modal feature in the multimodal feature set based on the current conversation context by using a dynamic modal weighting mechanism includes:

[0033] Determining the importance score of each modality in the multimodal feature set based on an attention mechanism;

[0034] Determine the context vector based on the current conversation situation;

[0035] The dynamic weight of each modality is calculated through the context vector and importance score.

[0036] Preferably, the expression of the dynamic weight is:

[0037]

[0038] Among them, w m is the dynamic weight corresponding to the modality m, c is the context vector, α m is the modal importance adjustment factor, e m is the characteristic representation of mode m, d k is the feature dimension.

[0039] Preferably, the expression of the fusion feature is:

[0040]

[0041] Among them, F fusion is the fusion feature, m is the modality identifier, a, i, t and p are speech, image, text and physiology respectively, W m is the learnable projection matrix, σ is the activation function, is the unimodal auxiliary loss, β is the loss sensitivity factor, and GRU is the gated recurrent unit network.

[0042] A multimodal emotion fusion and user emotion intention understanding system based on attention mechanism, including:

[0043] The acquisition module is used to collect multimodal dialogue datasets;

[0044] A preprocessing module, configured to perform data preprocessing on the multimodal conversation dataset to obtain a preprocessed conversation dataset;

[0045] The feature extraction module is used to extract and encode the emotional features of each modality in the preprocessed conversation dataset based on the improved convolutional neural network and the improved long short-term memory network to obtain a multimodal feature set;

[0046] An adjustment module, which is used to adjust the weights of each modal feature in the multimodal feature set based on the current conversation context using a dynamic modality weighting mechanism;

[0047] An enhancement module, configured to adopt an adaptive noise suppression strategy and utilize a generative adversarial network to obtain an enhanced feature set based on the multimodal feature set;

[0048] A fusion module is used to perform feature fusion using the weights corresponding to each modality in the current multimodality according to a hybrid cross entropy attention mechanism to obtain a fused feature;

[0049] The recognition module is used to input the fusion features into the sentiment classifier to generate the probability distribution vector of the user's sentiment category and identify the user's intention;

[0050] The display module is used to generate and display interactive feedback in real time based on user intention.

[0051] The present invention discloses the following technical effects:

[0052] The present invention provides a method and system for multimodal emotion fusion and user emotion intention understanding based on an attention mechanism. The method comprises: collecting a multimodal conversation dataset; preprocessing the multimodal conversation dataset to obtain a preprocessed conversation dataset; extracting and encoding emotion features of each modality in the preprocessed conversation dataset based on an improved convolutional neural network and an improved long short-term memory network to obtain a multimodal feature set; adjusting the weights of each modal feature in the multimodal feature set based on the current conversation context using a dynamic modal weighting mechanism; obtaining an enhanced feature set based on the multimodal feature set using an adaptive noise suppression strategy and a generative adversarial network; fusing features using the weights corresponding to each modality in the current multimodal context using a hybrid cross-entropy attention mechanism to obtain fused features; inputting the fused features into an emotion classifier to generate a probability distribution vector of the user emotion category and identify the user's intention; and generating and displaying interactive feedback in real time based on the user's intention. Through the method for multimodal emotion fusion and user emotion understanding, the present invention endows the system with stronger data processing capabilities and environmental adaptability, overcoming the shortcomings of existing technologies in terms of dynamic changes and noise interference. This will greatly enhance the intelligence and naturalness of the human-computer interaction system, giving it higher practical value and market competitiveness in application areas such as intelligent customer service, virtual assistants, and smart homes. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 A flowchart of a method for multimodal emotion fusion and user emotion intention understanding based on an attention mechanism provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0056] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] like Figure 1 As shown, the present invention provides a multimodal emotion fusion and user emotion intention understanding method based on attention mechanism, including:

[0058] Step 100: Collect a multimodal conversation dataset;

[0059] Specifically, the specific process of collecting multimodal dialogue datasets is as follows:

[0060] Define a multimodal dataset: Determine the types of data that need to be collected, including user voice, text messages, images (such as facial expressions), and physiological data (such as heart rate, galvanic skin response, etc.).

[0061] Data collection: Use audio recording equipment to collect users' voice conversations.

[0062] Configure the text input interface to record the user's text input content.

[0063] An image capturing device (such as a camera) is used to obtain image data of the user's facial expression.

[0064] The user's physiological data is collected through physiological signal sensors (such as a heart rate monitor).

[0065] Data annotation: Annotate the collected multimodal data and associate emotional states (such as happiness, anger, sadness, etc.) with each data sample to support subsequent model training.

[0066] Step 200: performing data preprocessing on the multimodal conversation dataset to obtain a preprocessed conversation dataset;

[0067] Step 300: Extracting and encoding the emotional features of each modality in the preprocessed conversation dataset based on the improved convolutional neural network and the improved long short-term memory network to obtain a multimodal feature set;

[0068] Step 400: Based on the current conversation context, a dynamic modality weighting mechanism is used to adjust the weight of each modality feature in the multimodal feature set;

[0069] Step 500: Adopting an adaptive noise suppression strategy and utilizing a generative adversarial network to obtain an enhanced feature set based on the multimodal feature set;

[0070] Specifically, we construct a GAN, including a generator and a discriminator, to optimize the representation of the feature set so that it has better emotional expression capabilities.

[0071] The imperfect modality features are enhanced by the generator, and an enhanced feature set is obtained at the output layer.

[0072] Step 600: According to the hybrid cross entropy attention mechanism, the weights corresponding to each modality in the current multimodality are used to perform feature fusion to obtain a fused feature;

[0073] Step 700: Input the fused features into the emotion classifier to generate a probability distribution vector of the user emotion category and identify the user intention;

[0074] Step 800: Generate interactive feedback in real time according to user intention and display it.

[0075] Specifically, establish a sentiment classification model: build a neural network model for sentiment classification (such as a fully connected network) and use the cross-entropy loss function for training.

[0076] Probability distribution generation: The fused features are input into the sentiment classifier, and the probability distribution vector of the user's sentiment category (such as the scores of N sentiment categories) is output.

[0077] Intent recognition: Select the emotion class with the highest probability based on the probability distribution vector and further identify the user's specific intention.

[0078] Interactive feedback generation: Generate corresponding feedback content based on the identified user intent, such as text reply, speech synthesis or action instructions.

[0079] Display results: Dynamically display the generated interactive feedback through the user interface to achieve natural and smooth human-computer interaction.

[0080] Furthermore, the multimodal dialogue dataset includes:

[0081] Voice, physiological data, text and image data of user interaction.

[0082] Furthermore, the performing data preprocessing on the multimodal conversation dataset to obtain a preprocessed conversation dataset includes:

[0083] Performing noise elimination and segmentation slicing processing on the voice data to obtain preprocessed voice data;

[0084] Perform word segmentation, stop word removal, and word embedding processing on the text data to obtain a text vector representation;

[0085] Performing image scaling and standardization on the image data and using data enhancement technology to obtain pre-processed image data;

[0086] performing standardization and normalization processing on the physiological data to obtain preprocessed physiological data;

[0087] A preprocessed dialogue data set is determined based on the preprocessed speech data, the preprocessed image data, the preprocessed physiological data and the text vector representation.

[0088] Furthermore, the improved convolutional neural network and the improved long short-term memory network are used to extract and encode the emotional features of each modality in the preprocessed conversation dataset to obtain a multimodal feature set, including:

[0089] Build improved convolutional neural networks and improved long short-term memory networks;

[0090] Inputting the preprocessed speech data and the preprocessed image data into the improved convolutional neural network to obtain speech emotion features and image emotion features, wherein the improved convolutional neural network includes: a first input layer, a bidirectional convolution layer, a pooling layer, a global pooling average layer, a fully connected layer and a first output layer;

[0091] Inputting the preprocessed physiological data and the text vector representation into the improved long short-term memory network to obtain text emotion features and physiological emotion features, wherein the improved long short-term memory network includes: a second input layer, an embedding layer, a bidirectional LSTM layer, a feature extraction layer, and a second output layer;

[0092] The speech emotion features, image emotion features, text emotion features and physiological emotion features are tokenized and input into the MLP for encoding processing to obtain a multimodal feature set;

[0093] Specifically, the improved convolutional neural network structure includes:

[0094] First input layer: For speech data, the input is the preprocessed Mel-spectrogram; for image data, the input is the adjusted image.

[0095] Bidirectional Convolutional Layer:

[0096] Convolutional layer 1: uses 32 3x3 convolution kernels with a stride of 1 and a padding of 1. The output of this layer is used to extract basic features.

[0097] Add bidirectional convolution, that is, output the forward and reverse feature maps, and integrate information from both directions.

[0098] Convolutional layer 2: uses 64 3x3 convolution kernels with a stride of 1 and a padding of 1 to extract more complex features.

[0099] It also implements bidirectional convolution and combines the forward and reverse convolution outputs to capture richer contextual information into the model.

[0100] Max Pooling Layer: A 2x2 max pooling layer is added after each convolutional layer to reduce the feature map size and increase the spatial invariance of the model to the input.

[0101] Global Average Pooling Layer: A global average pooling layer is added after the convolutional layer to reduce the feature value of each channel to a single average value, which is used to reduce model parameters and highlight the main features.

[0102] Fully Connected Layer: Use a fully connected layer to map the globally averaged pooled features to the emotion categories (output emotion features).

[0103] First output layer (OutputLayer): The Softmax layer outputs the probability distribution of emotion categories, corresponding to the extracted speech emotion features and image emotion features respectively.

[0104] The improved neural network has the following advantages:

[0105] Enhanced context capture capability: Bidirectional convolution improves the ability to capture contextual information, especially for time-series data such as speech, which can make features more robust; Powerful feature representation capability: Allows the network to integrate feature representations of more dimensions, making emotional feature extraction more accurate and improving performance; Adaptability to multimodal data: The improved network can not only effectively extract speech emotional features, but can also easily adapt to the processing of image data, thereby improving the overall application effect.

[0106] The improved convolutional neural network structure includes:

[0107] Second input layer: text input: preprocessed text data, shape is [batch_size, seq_length], it is embedded into a vector, shape is [batch_size, seq_length, embedding_dim].

[0108] Physiological data input: Normalized physiological data, shape is [batch_size, seq_length, feature_dim].

[0109] Embedding Layer: Embeds text input using pre-trained word vectors or randomly initialized word embeddings with a dimension of embedding_dim.

[0110] Bidirectional LSTM layer: The first bidirectional LSTM layer: the input is the embedded text data, and the output is a bidirectional hidden state with a shape of [batch_size, seq_length, hidden_dim*2]; the second bidirectional LSTM layer: the input is the output of the first layer, and the output is also a bidirectional hidden state.

[0111] Feature extraction layer: Extracts the hidden state of the last time step from the output of the bidirectional LSTM, which are the emotional features of text and physiological data respectively.

[0112] Second output layer: finally outputs two features:

[0113] Text sentiment features: hidden states corresponding to LSTM from text input.

[0114] Physiological emotion features: hidden states corresponding to LSTM derived from physiological inputs.

[0115] The improved LSTM structure has the following advantages:

[0116] Use of bidirectional LSTM: Bidirectional LSTM can simultaneously capture contextual information of the sequence, improving the model's understanding of time series data. This effectively captures long-term and short-term dependencies between text and physiological data. Separate LSTM layers for text and physiological data: Designing separate LSTM layers for text and physiological data allows the model to independently learn and extract features from different types of data, helping to improve its ability to represent emotional features. Extraction of the last time step: Directly using the output of the last time step as the emotional feature allows the model to focus on the final state of the sequence, highlighting the final emotional expression.

[0117] Furthermore, the expression of the multimodal feature set is:

[0118]

[0119] Among them, E m is a multimodal feature set, To extract the speech emotion features, To extract the emotional features of the image, is the extracted text sentiment feature, is the extracted physiological emotion feature, Γ is the feature tokenization layer, MLP is the multi-layer perceptron encoder, and ‖ is the feature concatenation.

[0120] Specifically, after extracting modality-specific features through the CNN / LSTM heterogeneous network, a tokenization layer is used to unify the feature space to overcome the dimensionality mismatch problem caused by simple splicing in traditional methods; the MLP encoder introduces a channel attention mechanism to achieve adaptive feature compression rather than fixed-dimensional mapping; the speech / image CNN features (spatial domain) and the text / physiological LSTM features (temporal domain) are respectively subjected to spatiotemporal normalization before splicing.

[0121] Furthermore, the dynamic modality weighting mechanism is used to adjust the weight of each modality feature in the multimodal feature set based on the current conversation context, including:

[0122] Determining the importance score of each modality in the multimodal feature set based on an attention mechanism;

[0123] Determine the context vector based on the current conversation situation;

[0124] The dynamic weight of each modality is calculated through the context vector and importance score.

[0125] Furthermore, the expression of the dynamic weight is:

[0126]

[0127] Among them, w m is the dynamic weight corresponding to modality m, reflecting the importance of the modality in the current conversation context, and c is the context vector, representing the state of the current conversation, including user emotions, historical interaction information, etc. The context vector can help the model understand the features related to the current context, α m is the modality importance adjustment factor, which is used to dynamically adjust the weight of a specific modality. It is calculated through the attention mechanism and reflects the relative contribution of the modality to sentiment analysis. m is the characteristic representation of mode m, d k Is the feature dimension, representing the dimension size of the modal feature, and divide the dot product result by To prevent excessive dot product values from causing gradient vanishing problems, ensure numerical stability.

[0128] Specifically, the dynamic weight adjustment factor model can adaptively adjust the importance of different modalities according to the current dialogue context, thereby obtaining a more accurate emotional feature representation; by utilizing the information of the context vector, the model's ability to respond to real-time changing environments is enhanced, making the learning process more efficient, especially in dialogue scenarios with drastic changes.

[0129] Furthermore, the expression of the fusion feature is:

[0130]

[0131] Among them, Ffusion is the fusion feature, m is the modality identifier, a, i, t and p are speech, image, text and physiology respectively, W m is the learnable projection matrix, σ is the activation function, is the unimodal auxiliary loss, β is the loss sensitivity factor, GRU is the gated recurrent unit network, where the learnable projection matrix is generated by the dynamic modality weighting mechanism (Dynamic ModalityWeighting):

[0132] W m =Attention(Q context , K modm , V modm );

[0133] Among them, Q contex is the query vector of the current dialogue context, K modm , V modm are modal feature key-value pairs, and high-dimensional features The projection is a scalar weight that reflects the importance of the modality in the current context.

[0134] Among them, y is the real emotion label (such as happy, angry, etc.), Classifier m It is a unimodal classifier with the following structure:

[0135] The weight of the classifier is used to constrain the quality of each modality feature, avoid the feature deviation from the semantics after GAN enhancement, and determine The smaller the value, the more reliable the modal prediction is, which is used to correct the attention weight.

[0136] The loss sensitivity factor is a learnable parameter with an initial value of 1.0. It is automatically optimized through backpropagation to control the penalty intensity of the cross entropy loss on the attention weight.

[0137] Furthermore, the structural formula of the gated recurrent unit is:

[0138]

[0139] Among them, z t To update the gate, control the current moment feature and historical characteristics The fusion ratio;

[0140] The gated recurrent unit has the following functions:

[0141] Timing alignment: Solve the problem of multimodal timing misalignment (such as pronunciation delay between speech and text).

[0142] Noise suppression: Filter out abnormal fluctuations in timing (such as burst noise in physiological signals) through a gating mechanism.

[0143] Long-range dependency modeling: Captures long-term sentiment changes in conversations more efficiently than ordinary LSTM.

[0144] This implementation also provides a multimodal emotion fusion and user emotion intention understanding system based on the attention mechanism, including:

[0145] The acquisition module is used to collect multimodal dialogue datasets;

[0146] A preprocessing module, configured to perform data preprocessing on the multimodal conversation dataset to obtain a preprocessed conversation dataset;

[0147] The feature extraction module is used to extract and encode the emotional features of each modality in the preprocessed conversation dataset based on the improved convolutional neural network and the improved long short-term memory network to obtain a multimodal feature set;

[0148] An adjustment module, which is used to adjust the weights of each modal feature in the multimodal feature set based on the current conversation context using a dynamic modality weighting mechanism;

[0149] An enhancement module, configured to adopt an adaptive noise suppression strategy and utilize a generative adversarial network to obtain an enhanced feature set based on the multimodal feature set;

[0150] A fusion module is used to perform feature fusion using the weights corresponding to each modality in the current multimodality according to a hybrid cross entropy attention mechanism to obtain a fused feature;

[0151] The recognition module is used to input the fusion features into the sentiment classifier to generate the probability distribution vector of the user's sentiment category and identify the user's intention;

[0152] The display module is used to generate and display interactive feedback in real time based on user intention.

[0153] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0154] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A multimodal emotion fusion and user emotion intention understanding method based on attention mechanism, characterized by: include: Collect multimodal conversation datasets; performing data preprocessing on the multimodal conversation dataset to obtain a preprocessed conversation dataset; Based on the improved convolutional neural network and the improved long short-term memory network, the emotional features of each modality in the preprocessed dialogue dataset are extracted and encoded to obtain a multimodal feature set; Based on the current conversation context, a dynamic modality weighting mechanism is used to adjust the weights of each modality feature in the multimodal feature set; Adopting an adaptive noise suppression strategy and utilizing a generative adversarial network to obtain an enhanced feature set based on the multimodal feature set; According to the hybrid cross entropy attention mechanism, the weights corresponding to each modality in the current multimodality are used to perform feature fusion to obtain a fused feature; Input the fused features into the sentiment classifier to generate the probability distribution vector of the user's sentiment category and identify the user's intention; Generate and display interactive feedback in real time based on user intent.

2. The method for multimodal emotion fusion and user emotional intention understanding based on attention mechanism according to claim 1 is characterized in that: The multimodal dialogue dataset includes: Voice, physiological data, text and image data of user interaction.

3. The method of multimodal emotion fusion and user emotional intention understanding based on attention mechanism according to claim 2 is characterized in that: The performing data preprocessing on the multimodal dialogue dataset to obtain a preprocessed dialogue dataset includes: Performing noise elimination and segmentation slicing processing on the voice data to obtain preprocessed voice data; Perform word segmentation, stop word removal, and word embedding processing on the text data to obtain a text vector representation; Performing image scaling and standardization on the image data and using data enhancement technology to obtain pre-processed image data; performing standardization and normalization processing on the physiological data to obtain preprocessed physiological data; A preprocessed dialogue data set is determined based on the preprocessed speech data, the preprocessed image data, the preprocessed physiological data and the text vector representation.

4. The method of multimodal emotion fusion and user emotional intention understanding based on attention mechanism according to claim 3 is characterized in that: The improved convolutional neural network and the improved long short-term memory network extract the emotional features of each modality in the preprocessed dialogue dataset and encode them to obtain a multimodal feature set, including: Build improved convolutional neural networks and improved long short-term memory networks; Inputting the preprocessed speech data and the preprocessed image data into the improved convolutional neural network to obtain speech emotion features and image emotion features, wherein the improved convolutional neural network includes: a first input layer, a bidirectional convolution layer, a pooling layer, a global pooling average layer, a fully connected layer and a first output layer; Inputting the preprocessed physiological data and the text vector representation into the improved long short-term memory network to obtain text emotion features and physiological emotion features, wherein the improved long short-term memory network includes: a second input layer, an embedding layer, a bidirectional LSTM layer, a feature extraction layer, and a second output layer; The speech emotion features, image emotion features, text emotion features and physiological emotion features are tokenized and input into MLP for encoding processing to obtain a multimodal feature set.

5. The method for multimodal emotion fusion and user emotional intention understanding based on attention mechanism according to claim 3 is characterized in that: The expression of the multimodal feature set is: Among them, E m is a multimodal feature set, To extract the speech emotion features, To extract the emotional features of the image, is the extracted text sentiment feature, is the extracted physiological emotion feature, Γ is the feature tokenization layer, MLP is the multi-layer perceptron encoder, and ‖ is the feature concatenation.

6. The method for multimodal emotion fusion and user emotional intention understanding based on attention mechanism according to claim 5 is characterized in that: The dynamic modality weighting mechanism is used to adjust the weight of each modality feature in the multimodal feature set based on the current conversation context, including: Determining the importance score of each modality in the multimodal feature set based on an attention mechanism; Determine the context vector based on the current conversation situation; The dynamic weight of each modality is calculated through the context vector and importance score.

7. The method for multimodal emotion fusion and user emotional intention understanding based on attention mechanism according to claim 6 is characterized in that: The expression of the dynamic weight is: Among them, w m is the dynamic weight corresponding to the modality m, c is the context vector, α m is the modal importance adjustment factor, e m is the characteristic representation of mode m, d k is the feature dimension.

8. The method for multimodal emotion fusion and user emotional intention understanding based on attention mechanism according to claim 6 is characterized in that: The expression of the fusion feature is: Among them, F fusion is the fusion feature, m is the modality identifier, a, i, t and p are speech, image, text and physiology respectively, W m is the learnable projection matrix, σ is the activation function, is the unimodal auxiliary loss, β is the loss sensitivity factor, and GRU is the gated recurrent unit network.

9. A multimodal emotion fusion and user emotion intention understanding system based on attention mechanism, characterized by: include: The acquisition module is used to collect multimodal dialogue datasets; A preprocessing module, configured to perform data preprocessing on the multimodal conversation dataset to obtain a preprocessed conversation dataset; The feature extraction module is used to extract and encode the emotional features of each modality in the preprocessed conversation dataset based on the improved convolutional neural network and the improved long short-term memory network to obtain a multimodal feature set; An adjustment module, which is used to adjust the weights of each modal feature in the multimodal feature set based on the current conversation context using a dynamic modality weighting mechanism; An enhancement module, configured to adopt an adaptive noise suppression strategy and utilize a generative adversarial network to obtain an enhanced feature set based on the multimodal feature set; A fusion module is used to perform feature fusion using the weights corresponding to each modality in the current multimodality according to a hybrid cross entropy attention mechanism to obtain a fused feature; The recognition module is used to input the fusion features into the sentiment classifier to generate the probability distribution vector of the user's sentiment category and identify the user's intention; The display module is used to generate and display interactive feedback in real time based on user intention.

Images