Multi-modal fine-grained emotion recognition method oriented to human-computer interaction and based on large model

Abstract

According to the man-machine interaction-oriented multi-modal fine-grained emotion recognition method based on the large model provided by the invention, cross-modal alignment from coarse granularity to fine granularity is realized through an attention pairing interaction module (APIM) on the basis of an aspect-driven vision-text alignment and fusion network (AVTAF); emotion-related visual features (such as facial expressions and gestures) in a robot scene can be accurately captured, and environmental noise is inhibited; meanwhile, the RD-GAT is enhanced, and the reasoning ability of a large model on multi-modal emotion semantics is improved by integrating external emotion knowledge (such as SenticNet). The technology provides a new normal form for intelligent upgrading of robot emotion interaction and multi-modal understanding of a large model, and is expected to promote breakthrough application in the fields of family service robots, medical accompanying assistants, multi-modal content generation and the like.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a multimodal fine-grained emotion recognition method based on a large model for human-computer interaction. Background Technology

[0002] With the rapid development of artificial intelligence technology, robotic systems and large-scale multimodal models (such as GPT-4 and CLIP) are increasingly being used in fields such as intelligent interaction, service robots, and emotional companionship. Robots need to perceive user intentions and emotional states in real time through multimodal data (such as speech, vision, and text) to provide natural and personalized responses; while large models need to overcome the limitations of single modalities and achieve deep alignment and semantic understanding of cross-modal information. However, existing technologies still face significant challenges:

[0003] 1. Coarse-grained problem in multimodal emotion understanding: Traditional robotic systems rely on processing data from each modality independently (such as analyzing facial expressions or speech separately), lacking fine-grained alignment with cross-modal emotional cues, leading to misjudgments of complex user emotions (such as sarcasm or contradictory emotions).

[0004] 2. Visual noise interference: In dynamic environments (such as homes and public places), images captured by robots often contain irrelevant backgrounds or occlusions. Existing models have difficulty distinguishing between emotion-related visual features (such as facial expression details) and noise, affecting the accuracy of interaction.

[0005] 3. Bottleneck of multimodal fusion in large models: Although large models perform well in single-modal tasks, their multimodal interaction mechanisms (such as simple splicing or single attention) are difficult to achieve accurate semantic matching between text and images, especially in aspect-level sentiment analysis, where they are easily interfered with by cross-modal redundant information.

[0006] For example, service robots need to combine user language ("This function is too slow") with frowning expressions to determine their dissatisfaction and optimize response strategies. However, existing methods may fail due to visual noise (such as cluttered backgrounds) or text-image misalignment (such as contradictory descriptions and expressions). In addition, large models often lack fine-grained alignment mechanisms when generating multimodal content, leading to semantic inconsistencies in the generated results (such as describing a "happy scene" but using conflicting images). Summary of the Invention

[0007] In view of this, the present invention proposes a multimodal fine-grained emotion recognition method based on a large model for human-computer interaction, which effectively alleviates the above problems through fine-grained visual-text alignment and emotion information fusion.

[0008] A multimodal emotion recognition method includes the following steps:

[0009] Step 1: Extract multimodal features from the input text-image pair: Generate image description C and facial expression description D based on the visual features V of the image; then convert the text features, including text, image description and facial expression description, into text embeddings respectively.

[0010] Step 2: Capture pairwise local interactions between text features and image features using a bilinear attention network. This includes using a bilinear interaction graph to capture pairwise attention weights, and utilizing the interaction graph in a variant of the multimodal residual network to selectively fuse text and visual features for hierarchical alignment. Specifically, this includes:

[0011] The multimodal features obtained in step 1 are input into the BART encoder to obtain the multimodal hidden states, which are: the hidden states corresponding to the text embedding T. The hidden state corresponding to visual feature V The image describes the corresponding hidden state. Hidden states corresponding to facial expression descriptions ,as well as ;

[0012] Based on hidden state Hidden state The method utilizes a bilinear interaction graph to obtain the single-head pairwise interaction matrix between the two elements; a bilinear pooling layer is introduced to obtain the attention map F; and a residual layer is then introduced to integrate multiple bilinear attention maps to obtain the enhanced text features. ;

[0013] Based on hidden state Hidden state The algorithm utilizes a bilinear interaction graph to obtain the single-head pairwise interaction matrix between the two entities; introduces a bilinear pooling layer to obtain an attention map F; and then introduces a residual layer to integrate multiple bilinear attention maps, gradually fusing fine-grained visual semantics of facial expressions, ultimately generating a locally fine-grained aligned joint representation. ;

[0014] Finally, learn the parameters To selectively fuse features and To obtain the final text representation: ;Will and The images were stitched together to generate image-text features. ;

[0015] Step 3: Weightedly fuse the sentiment scores from SenticNet with the multimodal features obtained from the attention pairing interaction module to construct a multimodal dependency graph, and aggregate sentiment information through a graph attention network, specifically including:

[0016] For a word in the input text Retrieve its sentiment score from SenticNet And project it onto multimodal features To obtain common-sense sentiment features within the same dimensional space. Then, this emotional characteristic Output integrated into APIM middle:

[0017]

[0018]

[0019]

[0020] in, and These are learnable parameters. This indicates features that include emotional knowledge; Representing image text features The feature representation of the i-th node; the hidden state rich in common-sense emotional features is obtained through the above formula. ,in, for The feature representation of the i-th node;

[0021] A global attention mechanism is used to obtain the dependency matrix between words in the input text. A multi-head attention mechanism is used to obtain the dependencies between visual features. And the dependency between visual features and text features. and The final dependency matrix D is obtained:

[0022] ;

[0023] Using graph attention network graph structure data Learning about the importance of different nodes allows for richer emotional information output. ;

[0024] Step 4: BART decoder reception , and the output of the previous decoder The predicted label probability distribution is as follows:

[0025]

[0026]

[0027]

[0028]

[0029] in, and There are two hyperparameters. express The text portion, W represents the text embedding representation of step one; This indicates the processing of the BART decoder. express Function processing of data within parentheses Embedded representation of corresponding sentiment tags; Indicates the decoder at time step The hidden state; Indicates at time step The generated predicted sentiment label probability distribution.

[0030] Preferably, in step 2, the single-head pairwise interaction matrix :

[0031]

[0032] in, and It is a learnable weight matrix for text and subtitle representation. It is a learnable weight vector. It is a fixed vector of all 1s. denoted by ; I represents the interaction strength between each word or subphrase in the text and the image description pair.

[0033] Ideally, enhanced text features are obtained. Specific methods include:

[0034] Introduce a bilinear pooling layer to obtain a bilinear joint representation. Specifically, the k-th element of F′ is calculated as follows:

[0035]

[0036]

[0037] Represents the elements in I; Concatenating along the feature dimensions yields ,right A linear transformation is performed to obtain the attention map F;

[0038] in, and represent the k-th column in the text representation weight matrix U and the image description weight matrix V, respectively;

[0039] Introduce a residual layer to integrate multiple bilinear attention maps:

[0040]

[0041] These are the input features of the i-th layer in the residual module, and their initial values ​​are... Original text features , This represents the i-th attention map. This represents the i-th bilinear attention module. This represents the transpose of a vector consisting entirely of 1s. This represents the output feature map of the (i+1)th layer, i.e., through... Calculate the weighted interaction between textual and visual features to generate joint features. Then, after batch normalization and linear transformation, combine the joint features with the original features. Add them together to get the updated result. .

[0042] Better learning parameters Represented as:

[0043]

[0044] It is a weight matrix used to perform a linear transformation on the input features to aid in the calculation. ; It is a weight matrix used to weight the input features. Perform a linear transformation; It is a weight matrix used to weight the input features. Perform a linear transformation; It is a bias vector used to add an offset to the result after linear transformation, thereby increasing the model's fitting ability.

[0045] Ideally, obtain the dependency matrix. The methods include: using the Stanza parser to parse the text into a dependency tree G, and then converting the dependency tree G into a fixed induced syntactic graph. This graph is fully connected and uses a symmetric adjacency matrix. This represents the distance between the i-th and j-th words;

[0046]

[0047] This represents the maximum possible distance between two nodes in the tree; by adjusting the parameters... To control the range of distance attenuation;

[0048] Assign a numeric identifier to the type and convert G to a matrix derived from the adjacency matrix. Inductive syntactic graph of representation Then initialize the type feature matrix. Where U represents the total number of dependency types, and D represents the dimension of the feature matrix set by the user. As the transposed query vector; then the softmax function is used to derive the attention weight matrix, and the gather method is applied; extracting and... The corresponding weights; From the initial adjacency matrix, we finally obtain the type dependency matrix. :

[0049]

[0050] Finally, the final submatrix is ​​obtained by adding the two together. Its expression is:

[0051] .

[0052] Ideally, obtain the dependency matrix. Methods include: employing multi-head attention mechanisms, from Hidden features obtained from visual features The final submatrix Represented as:

[0053]

[0054] yes Column elements in It is a weight matrix used to weight features Projecting onto the query space is a key parameter for calculating the query vector in multi-head attention mechanisms; Similarly, it's a weight matrix, responsible for weighting features. Projecting onto the key space is used to calculate the key vectors, which are also parameters optimized during training. Their role is to help the model extract information from different perspectives from the input features for attention calculation.

[0055] Ideally, obtain the dependency matrix. and The methods include: from Hidden features obtained from visual and textual features and The final submatrix and Represented as:

[0056]

[0057] .

[0058] Preferably, graph attention network graph structure data is used. Learning about the importance of different nodes allows for richer emotional information output. The methods include:

[0059] Will As the initial node representation in the graph, the attention coefficient between nodes i and j is first calculated:

[0060]

[0061] in and Let be the hidden states of nodes i and j, respectively, where 'a' represents the learnable parameters, and '||' represents the vector concatenation operation. This represents the correlation between node i and node j;

[0062]

[0063]

[0064] in, Is using The weight matrix obtained after masking the attention coefficients is further processed to obtain... ;

[0065] (

[0066] Where K represents the number of attention heads. , It is a learnable weight matrix. The outputs of multiple attention heads are concatenated and then subjected to a linear transformation to obtain the final node feature representation. .

[0067] Furthermore, it also includes optimizing the model using the cross-entropy loss function and generating joint prediction results of aspect terms and their sentiment polarities by weighting the contributions of multiple modules through hyperparameters.

[0068] A multimodal emotion recognition system, implemented using the AVTAF model, includes the following modules: a feature extraction module, an attention pairing interaction module APIM, an enhanced dependency graph attention network RD-GAT, and a prediction and optimization module;

[0069] The feature extraction module is used to implement the method in step 1;

[0070] The attention pairing interaction module APIM is used to implement the method in step 2;

[0071] The Enhanced Dependency Graph Attention Network (RD-GAT) is used to implement the method in step 3;

[0072] The prediction and optimization module is used to implement the method in step 4.

[0073] The present invention has the following beneficial effects:

[0074] This invention proposes a multimodal fine-grained emotion recognition method for human-computer interaction based on a large model. Utilizing an aspect-driven visual-text alignment and fusion network (AVTAF), it achieves cross-modal alignment from coarse to fine granular through an attention pairing interaction module (APIM), accurately capturing emotion-related visual features (such as facial expressions and gestures) in robotic scenarios while suppressing environmental noise. Simultaneously, an enhanced dependent graph attention network (RD-GAT) integrates external emotional knowledge (such as SenticNet) to improve the large model's reasoning ability for multimodal emotional semantics. This technology provides a new paradigm for the intelligent upgrading of robot emotional interaction and the multimodal understanding of large models, and is expected to drive breakthrough applications in fields such as home service robots, medical companions, and multimodal content generation. Attached Figure Description

[0075] Figure 1 This is a schematic diagram of the AVTAF framework of the present invention.

[0076] Figure 2 Note the structure diagram of the Pairing Interaction Module (APIM).

[0077] Figure 3 This is a visualization of the AVTAF model in a multimodal sentiment analysis task. Detailed Implementation

[0078] 1. System Architecture

[0079] like Figure 1 As shown, the AVTAF model constitutes a multimodal sentiment analysis framework. Its core objective is to improve the accuracy of sentiment analysis through fine-grained alignment and fusion of visual and textual information. The system architecture of this model mainly includes the following modules: feature extraction module, attention pairing interaction module (APIM), augmented dependency graph attention network (RD-GAT), and prediction and optimization module.

[0080] The operating mechanism of the AVTAF model can be broken down into several steps:

[0081] First, feature extraction is performed, choosing to interact with the frozen image encoder for visual feature extraction. To establish coarse-grained alignment between the global image and text, the image captioning tool CATR is applied, which generates high-quality captions for the scene. Simultaneously, to extract fine-grained emotional visual information, facial expression description templates are used to generate facial descriptions. For text input, context-aware text features are generated through the BART model.

[0082] Subsequently, the design utilizes the APIM module. First, a bilinear attention network is used to capture the pairwise local interactions between text features and image features. A residual layer is introduced to integrate multiple bilinear attention layers. Coarse-grained and fine-grained alignments are selectively fused between global image and text features to obtain the final image-text features aligned from coarse to fine-grained, generating a joint representation.

[0083] Furthermore, sentiment information is aggregated by integrating SenticNet sentiment knowledge using the RD-GAT module, constructing a multimodal dependency matrix, and aggregating node information through a graph attention network to output a feature representation rich in sentiment information.

[0084] Finally, the outputs of APIM and RD-GAT are input into the BART decoder to generate aspect-sentiment pair sequences. The model is then optimized using the cross-entropy loss function, and hyperparameters are adjusted to balance the contributions of each module, thus completing the prediction and optimization.

[0085] The AVTAF model can analyze multimodal information posted by users and identify sentiment tendencies. It can be widely used in market research, e-commerce review analysis, intelligent customer service and human-computer interaction to improve the personalization of services and the level of intelligent interaction.

[0086] 2. Feature Extraction Module

[0087] 2.1 Image Representation

[0088] 2.1.1 Generate descriptive captions

[0089] This invention employs an image encoder to extract visual features. It takes an image as input, pre-trains it to extract general and stable visual features as output, and eliminates the need for retraining after freezing parameters, saving computational resources and time. It also avoids overfitting on small datasets. Furthermore, the extracted features exhibit strong transferability. ,..., The visual features are represented by}. Furthermore, this invention employs an image captioning model to establish a close connection between the visual and textual modalities, conveying the semantic essence of visual content on a broader level.

[0090] CATR (Cross-Attention Transformer) is a Transformer-based cross-attention model that fuses visual and linguistic modalities across attention layers to accurately locate and segment image regions corresponding to natural language descriptions. To achieve coarse-grained alignment between global images and text, this invention uses this tool to generate descriptive captions for the scene, represented as follows:

[0091]

[0092] 2.1.2 Generating Facial Descriptions

[0093] Facial expressions, as a direct and intuitive way for humans to convey emotions, play a crucial role in accurately identifying object-level emotions in images. Analysis of the TWITTER-2015 and TWITTER-2017 datasets shows that a significant portion of the images contain facial expressions.

[0094] Yang et al. proposed a facial expression description framework, a deep learning-based multimodal sentiment analysis system, designed to more accurately identify and understand facial expressions by combining visual information and textual descriptions. The framework first utilizes convolutional neural networks (CNNs) to extract high-level visual features from facial images, capturing detailed information about the expression. Simultaneously, natural language processing (NLP) techniques are used to generate textual descriptions related to the expression, such as "smiling" or "frowning." Next, a cross-modal attention mechanism or a Transformer architecture is employed to align and fuse the visual features and textual descriptions to capture the semantic relationships between them. Finally, based on the fused multimodal features, sentiment classification (e.g., happiness, sadness, anger) or sentiment intensity regression is performed. The core advantage of this framework lies in its ability to more comprehensively understand the semantic and emotional connotations of facial expressions through collaborative analysis of visual and textual information, significantly improving the accuracy and robustness of sentiment analysis.

[0095] In order to capture subtle emotional visual cues, this invention applies the aforementioned facial expression description framework to generate detailed facial descriptions.

[0096]

[0097] 2.2 Text Representation

[0098] For text input (e.g., sentence text, image-generated captions and Face_Descriptions), BART Embedding can transform high-dimensional, discrete data into low-dimensional, continuous vectors, yielding initial word embeddings. The initial word embedding for sentence text is represented as follows: ,in This refers to the features of the j-th word in the text. The initial word embedding of the Caption is represented as follows: ,in This refers to the feature of the j-th word in the Caption. The initial word embedding of the Face Description is represented as... This refers to the feature of the j-th word in the Face Description.

[0099] 3.2.1 BART-based generative framework

[0100] 3.2.1.1 Encoder

[0101] The encoder in this invention employs a multi-layer bidirectional transformer, enabling simultaneous processing of contextual information. To distinguish inputs from different modalities, special markers are added: `` and `` are placed before and after visual features to mark their boundaries, while `<bos>` and `<eos>` are used to indicate the start and end of text, caption, and face description features, respectively. This invention concatenates multimodal features as input X to the BART encoder, whose output is the multimodal hidden state. as well as and ,in , , , .

[0102] 3.2.1.2 Decoder

[0103] The decoder in this invention employs a multi-layer transformer, but unlike the encoder, which generates output bidirectionally, the decoder generates output unidirectionally. This invention introduces a special marker `<bos>` to indicate the start of generation. The decoder interacts with the encoder's output through self-attention and cross-attention mechanisms to generate the predicted word for the current time step. This generated word is then used as input for the next step, and this process is repeated until an end marker is generated, ultimately producing the complete target sequence.

[0104] 3. Pay attention to the pairing interaction module (APIM).

[0105] 3.1 Module Composition

[0106] Bilinear Attention Networks (BANs) effectively extend single-attention networks using bilinear attention maps for multimodal learning. They evaluate each pair of multimodal input channels, such as image regions and text words, to learn interactive representations. Compared to applying a single attention mechanism to multimodal data, BANs provide richer joint information while maintaining similar computational costs.

[0107] This invention designs an Attention Pairing Interaction Module (APIM), such as Figure 2 As shown, this invention uses a Bilinear Interaction Graph (BAN) to capture local pairwise interactions between text and image features. The module consists of two parts: (I) a bilinear interaction graph for capturing pairwise attention weights; and (II) a variant of a multimodal residual network to efficiently utilize the interaction graph.

[0108] 3.2 Bilinear Interaction Graph

[0109] 3.2.2 Bilinear Interaction Feature Capture

[0110] Given a text hidden state representation encoded by a BART encoder And the subtitle hidden status Let N and Z represent the number of features in the text and subtitles, respectively, and R represent the extended set of real numbers. Then, the bilinear interaction graph can be used to obtain a single-head pairwise interaction matrix. :

[0111]

[0112] in, and It is a learnable weight matrix for text and subtitle representation. It is a learnable weight vector. It is a fixed vector of all 1s. Represents the Hadamard (element-wise) product. I represents the interaction strength between each word (or subphrase) in the text and the caption pair.

[0113] To better understand the bilinear interaction, element I in equation (3) can also be expressed as:

[0114]

[0115] in, express The i-th column, express The j-th column represents the i-th and j-th feature representations of the text and subtitle, respectively. Therefore, this invention can view the bilinear interaction as first representing the text... Subtitle features Map to a common feature space with weight matrices U and V, and then learn the interaction between the Hadamard (element-wise) product and the weights of vector q.

[0116] 3.2.3 Bilinear Pooling Sharing

[0117] Introduce a bilinear pooling layer to obtain a bilinear joint representation. Specifically, the k-th element of F′ is calculated as follows:

[0118]

[0119]

[0120] This formula calculates text features using bilinear pooling. Image caption features The interactive representation generates the bilinear joint feature of the k-th channel. Its core is through attention weights Element-wise multiplication of weighted text and image features captures fine-grained correlations between modalities. Concatenating along the feature dimensions yields ,right A linear transformation is performed to obtain the attention map F.

[0121] in, and Let U and V represent the k-th columns of the weight matrices U and V, respectively. To reduce the number of model parameters and prevent overfitting, this invention shares these two weight matrices between the current interaction layer and the previous interaction layer. This sharing strategy helps improve the performance and stability of the model.

[0122] 3.3 Variations of Multimodal Residual Networks

[0123] This invention introduces a residual layer to integrate multiple bilinear attention maps:

[0124]

[0125] These are the input features of the i-th layer in the residual module, and their initial values ​​are... Original text features , Let represent the i-th attention map. The residual layer adds this attention-enhanced feature to the original input feature using formula (6). This represents the i-th bilinear attention module. This represents the transpose of a vector consisting entirely of 1s. This represents the output feature map of the (i+1)th layer, i.e., through... Calculate the weighted interaction between textual and visual features to generate joint features. Then, after batch normalization and linear transformation, combine the joint features with the original features. Add them together to get the updated result. .

[0126] Input text features and image caption features The attention map I of the text and image captions is calculated using formula (3), and then joint features are generated by bilinear pooling according to formula (5). The enhanced text features are obtained by iteratively updating the residual network using formula (6). .

[0127] Text features are calculated using a bilinear attention mechanism. Facial descriptive features The fine-grained interaction weight matrix I is used to map the interacted features into a local joint representation using bilinear pooling. Furthermore, the text features are iteratively updated through a multi-layer residual network, gradually fusing fine-grained visual semantics of facial expressions, and finally generating a locally fine-grained aligned joint representation. Its core is to accurately capture emotional cues (such as facial expression details) related to textual aspects through cross-modal alignment driven by facial description, thereby reducing the interference of irrelevant visual noise.

[0128] Finally, learn the parameters To selectively fuse features and To obtain the final text representation. .

[0129]

[0130]

[0131] It is a weight matrix used to perform a linear transformation on the input features to aid in the calculation. ; It is a weight matrix used to weight the input features. Perform a linear transformation; It is a weight matrix used to weight the input features. Perform a linear transformation; It is a bias vector used to add an offset to the result after linear transformation, thereby increasing the model's fitting ability.

[0132] With the help of the matching interaction module, precise semantic matching is gradually achieved between images and text through a visual text alignment process that progresses from coarse-grained to fine-grained. This multi-layered alignment method... and By splicing the images together, a rich image-text feature was ultimately generated. .

[0133] As shown in Table 1, compared to the full model, disabling the Attention Paired Interaction Module (APIM) resulted in an accuracy decrease of approximately 1.9% and 0.7% on the two datasets, respectively. This highlights that the APIM module effectively coordinates different modalities, and this coarse-to-fine image-text alignment approach better enables multimodal feature fusion.

[0134] Table 1

[0135]

[0136] 4. Enhanced Dependency Graph Attention Network (RD-GAT)

[0137] 4.1 Extracting Emotional Features

[0138] SenticNet is a sentiment computing knowledge base designed to support sentiment analysis in Natural Language Processing (NLP). It assigns sentiment polarity (e.g., positive, negative, neutral) and sentiment intensity to a large number of concepts by combining semantic and psychological methods, and considers multidimensional representations of sentiment (e.g., pleasantness, attention).

[0139] To effectively capture sentiment information from multimodal data, this invention develops an Enhanced Dependency Graph Attention Network (RD-GAT). First, this invention integrates common-sense knowledge related to sentiment concepts into multimodal features. In other words, for a single word in the input text... This invention retrieves sentiment scores from SenticNet. And project it onto multimodal features To obtain common-sense sentiment features within the same dimensional space. Then, the present invention will incorporate this emotional characteristic. Output integrated into APIM middle:

[0140]

[0141]

[0142]

[0143] in, and These are learnable parameters. This indicates features that include emotional knowledge; Representing image text features The feature representation of the i-th node. The above equation yields the hidden state rich in common-sense emotional features. ,in, for The feature representation of the i-th node.

[0144] 4.2 Calculate the dependency matrix

[0145] 4.2.1 Distance Importance Calculation

[0146] This invention designs a dependency matrix A to establish the relationship between images and text. The submatrices of this matrix... This invention describes the relationships between words, primarily considering two aspects: the importance of the distance between words and the importance of their dependency types. To generate this matrix, the invention uses the Stanza parser (a powerful tool capable of word segmentation, part-of-speech tagging, and syntactic analysis for multiple languages). It parses the text into a dependency tree G, a special graph structure that shows the directed relationships between words in a sentence (e.g., which depends on which). To simplify modeling, the invention transforms directed dependencies into undirected edges, ensuring that any two words in a sentence are connected. Next, the invention calculates the minimum tree distance between words, which is the number of edges on the shortest path connecting them. Finally, the invention transforms the dependency tree G into a fixed induced syntactic graph. This graph is fully connected and uses a symmetric adjacency matrix. It means that, among them [i, j] reflects the distance between the i-th and j-th words. In simple terms, this invention quantifies the distance and relationship between words by parsing text, constructing a dependency tree, and ultimately generating a matrix, thereby helping to establish connections between images and text.

[0147] Considering that edge weights are typically inversely proportional to distance, this invention innovatively introduces a distance importance calculation method based on the inverse hyperbolic tangent function to more intuitively map the impact of distance on weights. Ultimately, an enhanced distance dependency matrix is ​​generated. .

[0148]

[0149] This represents the maximum possible distance between two nodes in the tree. By adjusting the parameters... This controls the range of distance decay. The inverse hyperbolic tangent function can smoothly adjust the weights of distant nodes and provide higher weights when the distance is close to zero, making it suitable for hierarchical data structures.

[0150] 4.2.2 Calculation of the Importance of Type Edges

[0151] To calculate the importance weights of type edges, this invention employs a global attention mechanism. Specifically, this invention first assigns numerical identifiers to types and transforms G into a matrix composed of adjacency matrices. Inductive syntactic graph of representation Then initialize the type feature matrix. Where U represents the total number of dependency types, and D represents the dimension of the feature matrix set by the user. This is used as the transposed query vector. The attention weight matrix is ​​then derived using the softmax function, and the gather method (an operation that collects elements from a tensor or array based on a specified index) is applied to extract the values. The corresponding weights. From the initial adjacency matrix, we finally obtain the type dependency matrix. :

[0152] (13)

[0153] Finally, the final submatrix is ​​obtained by adding the two together. Its expression is:

[0154]

[0155] 4.2.3 Obtaining Dependencies

[0156] This invention employs a multi-head attention mechanism to obtain the dependencies between visual features. And the dependency between visual features and text features. and In multi-head attention mechanisms, attention weights are a key output. They determine the degree to which each query vector "pays attention" to different key vectors. Attention weights can be understood as a probability distribution representing which key vectors each query vector should obtain more information from. First, this invention obtains the attention weights. It represents the degree of mutual attention between different locations within the same mode. Hidden features obtained from visual features The final submatrix Represented as:

[0157]

[0158] yes Column elements in It is a weight matrix used to weight features Projecting onto the query space is a key parameter for calculating the query vector in multi-head attention mechanisms; Similarly, it's a weight matrix, responsible for weighting features. Projecting onto the key space is used to calculate the key vectors, which are also parameters optimized during training. Their role is to help the model extract information from different perspectives from the input features for attention calculation.

[0159] 4.2.4 Calculate the dependency matrix

[0160] To establish element-level (word-level / region-level) interactions between visual and textual features, a cross-attention mechanism is employed to obtain attention weights as the visual-text / text-visual dependency matrix. Hidden features obtained from visual and textual features and The final submatrix and Represented as:

[0161]

[0162]

[0163] Attention weight Attention weights represent the degree of attention a location in the visual modality (e.g., image features) gives to a query vector (e.g., word representation) in the text modality. This indicates the degree of attention a query vector (e.g., a word representation) in the text modality pays to a specific location (e.g., an image feature) in the visual modality. Combined with... , , and The present invention yields the final dependency matrix D.

[0164]

[0165] Let the dependency matrix D be used as the adjacency matrix A of this invention.

[0166] 4.3 RD-GAT Principle

[0167] Graph Attention Network (GAT) is a model that combines graph neural networks and attention mechanisms, aiming to improve the quality of node representations by adaptively aggregating information from neighboring nodes. This method learns the importance of different nodes in graph-structured data, thereby effectively capturing the features of the graph.

[0168] It is a multimodal feature rich in common-sense emotional information. The Reinforced Dependency Graph Attention Network (RD-GAT) will... As the initial node representation in the graph, the attention coefficient between nodes i and j is first calculated:

[0169]

[0170] in and Let be the hidden states of nodes i and j, respectively; let 'a' represent the learnable parameters; and let || represent the vector concatenation operation. This represents the correlation between node i and node j.

[0171] (20)

[0172] (twenty one)

[0173] in, Is using The weight matrix obtained after masking the attention coefficients is further processed to obtain... .

[0174] ( (twenty two)

[0175] Where K represents the number of attention heads. , It is a learnable weight matrix. The outputs of multiple attention heads are concatenated and then subjected to a linear transformation to obtain the final node feature representation. The final output is obtained through the above formula. The output integrates information from image-text pairs and is rich in emotional information.

[0176] 4.4 Ablation Experiment

[0177] RD-GAT ablation experiments: As shown in the table below, disabling RD-GAT significantly reduced the overall performance of the model. This is because this invention incorporates common-sense knowledge related to sentiment concepts into RD-GAT, utilizing external sentiment information to assist sentiment prediction and improve prediction accuracy. After removing the external sentiment information, the F1 scores on the two datasets decreased by approximately 1.8% and 0.9%, respectively, highlighting the module's role in enhancing the model's sentiment understanding ability. Similarly, disabling the RD-GAT module resulted in a decrease in F1 scores on the two datasets of approximately 1.3% and 2.3%, respectively, indicating that this module is crucial for maintaining model performance. This underscores the effectiveness of RD-GAT in aggregating sentiment information from multimodal features.

[0178]

[0179] 5. Prediction and Optimization

[0180] This invention proposes a multi-task prediction and optimization framework based on the BART decoder. Its core lies in unifying complex tasks into index generation tasks, thereby achieving collaborative reasoning and end-to-end optimization across modal information. Specifically, the BART decoder dynamically receives three inputs during the generation process: (1) the output of APIM. (2) Output of RD-GAT (3) The output of the previous decoder To maintain causal consistency in the task sequence, the probability distribution of the predicted tokens is as follows:

[0181]

[0182]

[0183]

[0184]

[0185] in, and These are hyperparameters that control the contributions of the two modules. express In the text portion, W represents the embedded representation of the input token. Corresponding sentiment tags (such as [positive, neutral, negative, ...)) <eos>Embedded representation of ]).

[0186] The loss function is defined as follows:

[0187]

[0188] Where O = 2M + 2N + 2 is the length of Y, and X represents the multimodal input.

[0189] APIM and RD-GAT joint ablation experiments: As shown in the table below, compared with the full model, disabling both the Attention Pairing Interaction Module (APIM) and the Enhanced Dependency Graph Attention Network (RD-GAT) resulted in a decrease in recall of approximately 1.4% and 2.3% on the two datasets, respectively. This highlights the effectiveness of the model of this invention in mitigating visual noise and emotional interference between different aspects.

[0190] Experimental results from AVTAF on the TWITTER-2015 and TWITTER-2017 datasets demonstrate that it significantly outperforms existing methods in multimodal sentiment analysis tasks, particularly in addressing visual noise and inter-aspect sentiment interference.

[0191]

[0192]

[0193] The specific embodiments described above only illustrate the design principles of the present invention. The shapes and names of the components in this description may differ and are not limited. Therefore, those skilled in the art can modify or make equivalent substitutions to the technical solutions described in the foregoing embodiments; and these modifications and substitutions do not depart from the inventive spirit and technical solutions of the present invention, and should all fall within the protection scope of the present invention.< / eos>

Claims

1. An emotion recognition method, characterized in that, Includes the following steps: Step 1: Extract multimodal features from the input text-image pair: Generate image description C and facial expression description D based on the visual features V of the image; then convert the text features, including text, image description and facial expression description, into text embeddings respectively. Step 2: Capture pairwise local interactions between text features and image features based on a bilinear attention network, including using a bilinear interaction graph to capture pairwise attention weights, and using a variant of the multimodal residual network to selectively fuse the interaction graph to achieve hierarchical alignment of text and visual features. Step 3: Weigh and fuse the sentiment scores from SenticNet with the multimodal features obtained from the attention pairing interaction module to construct a multimodal dependency graph, and aggregate sentiment information through a graph attention network; Step 4: The BART decoder receives image text features. Emotional information and the predicted tag probability output by the previous decoder Predict the probability distribution of the markers.

2. The emotion recognition method as described in claim 1, characterized in that, Specifically, the following steps are included: Step 1: Extract multimodal features from the input text-image pair: Generate image description C and facial expression description D based on the visual features V of the image; then convert the text features, including text, image description and facial expression description, into text embeddings respectively. Step 2: Capture pairwise local interactions between text features and image features using a bilinear attention network. This includes capturing pairwise attention weights using a bilinear interaction graph, and selectively fusion using a variant of the multimodal residual network to achieve hierarchical alignment of text and visual features. Specifically, this includes: The multimodal features obtained in step 1 are input into the BART encoder to obtain the multimodal hidden states, which are: the hidden states corresponding to the text embedding T. The hidden state corresponding to visual feature V The image describes the corresponding hidden state. Hidden states corresponding to facial expression descriptions ,as well as ; Based on hidden state Hidden state The method utilizes a bilinear interaction graph to obtain the single-head pairwise interaction matrix between the two elements; a bilinear pooling layer is introduced to obtain the attention map F; and a residual layer is then introduced to integrate multiple bilinear attention maps to obtain the enhanced text features. ; Based on hidden state Hidden state The algorithm utilizes a bilinear interaction graph to obtain the single-head pairwise interaction matrix between the two entities; introduces a bilinear pooling layer to obtain an attention map F; and then introduces a residual layer to integrate multiple bilinear attention maps, gradually fusing fine-grained visual semantics of facial expressions, ultimately generating a locally fine-grained aligned joint representation. ; Finally, learn the parameters To selectively fuse features and To obtain the final text representation: ;Will and The images were stitched together to generate image-text features. ; Step 3: Weightedly fuse the sentiment scores from SenticNet with the multimodal features obtained from the attention pairing interaction module to construct a multimodal dependency graph, and aggregate sentiment information through a graph attention network, specifically including: For a word in the input text Retrieve its sentiment score from SenticNet And project it onto multimodal features To obtain common-sense sentiment features within the same dimensional space. Then, this emotional characteristic Output integrated into APIM middle: in, and These are learnable parameters. This indicates features that include emotional knowledge; Representing image text features The feature representation of the i-th node; the hidden state rich in common-sense emotional features is obtained through the above formula. ,in, for The feature representation of the i-th node; A global attention mechanism is used to obtain the dependency matrix between words in the input text. A multi-head attention mechanism is used to obtain the dependencies between visual features. And the dependency between visual features and text features. and The final dependency matrix D is obtained: ； Using graph attention network graph structure data Learning about the importance of different nodes allows for richer emotional information output. ; Step 4: BART decoder reception , and the output of the previous decoder The predicted label probability distribution is as follows: in, and There are two hyperparameters. express The text portion, W represents the text embedding representation of step one; Indicates the processing of the BART decoder express Function processing of data within parentheses Embedded representation of corresponding sentiment tags; Indicates the decoder at time step The hidden state; Indicates at time step The generated probability distribution of predicted sentiment labels.

3. The emotion recognition method as described in claim 2, characterized in that, In step 2, the single-head pairwise interaction matrix : ； in, and It is a learnable weight matrix for text and subtitle representation. It is a learnable weight vector. It is a fixed vector of all 1s. denoted by ; I represents the interaction strength between each word or subphrase in the text and the image description pair.

4. The emotion recognition method as described in claim 2, characterized in that, Enhanced text features Specific methods include: Introduce a bilinear pooling layer to obtain a bilinear joint representation. Specifically, the k-th element of F′ is calculated as follows: ； Represents the elements in I; Concatenating along the feature dimensions yields ,right A linear transformation is performed to obtain the attention map F; in, and represent the k-th column in the text representation weight matrix U and the image description weight matrix V, respectively; Introduce a residual layer to integrate multiple bilinear attention maps: ； These are the input features of the i-th layer in the residual module, and their initial values ​​are... Original text features , This represents the i-th attention map. This represents the i-th bilinear attention module. This represents the transpose of a vector consisting entirely of 1s. This represents the output feature map of the (i+1)th layer, i.e., through... Calculate the weighted interaction between textual and visual features to generate joint features. Then, after batch normalization and linear transformation, combine the joint features with the original features. Add them together to get the updated result. .

5. The emotion recognition method as described in claim 2, characterized in that, Learning parameters Represented as: ； It is a weight matrix used to perform a linear transformation on the input features to aid in the calculation. ; It is a weight matrix used to weight the input features. Perform a linear transformation; It is a weight matrix used to weight the input features. Perform a linear transformation; It is a bias vector used to add an offset to the result after linear transformation, thereby increasing the model's fitting ability.

6. The emotion recognition method as described in claim 2, characterized in that, Obtain the dependency matrix The methods include: using the Stanza parser to parse the text into a dependency tree G, and then converting the dependency tree G into a fixed induced syntactic graph. This graph is fully connected and uses a symmetric adjacency matrix. This represents the distance between the i-th and j-th words; ； This represents the maximum possible distance between two nodes in the tree; by adjusting the parameters... To control the range of distance attenuation; Assign a numeric identifier to the type and convert G to a matrix derived from the adjacency matrix. Inductive syntactic graph of representation Then initialize the type feature matrix. Where U represents the total number of dependency types, and D represents the dimension of the feature matrix set by the user. As the transposed query vector; then the softmax function is used to derive the attention weight matrix, and the gather method is applied; extracting and... The corresponding weights; From the initial adjacency matrix, we finally obtain the type dependency matrix. : ； Finally, the final submatrix is ​​obtained by adding the two together. Its expression is: 。 7. The emotion recognition method as described in claim 2, characterized in that, Obtain the dependency matrix Methods include: employing multi-head attention mechanisms, from Hidden features obtained from visual features The final submatrix Represented as: yes Column elements in It is a weight matrix used to weight features Projecting onto the query space is a key parameter for calculating the query vector in multi-head attention mechanisms; Similarly, it's a weight matrix, responsible for weighting features. Projecting onto the key space is used to calculate the key vectors, which are also parameters optimized during training. Their role is to help the model extract information from different perspectives from the input features for attention calculation.

8. The emotion recognition method as described in claim 2, characterized in that, Obtain the dependency matrix and The methods include: from Hidden features obtained from visual and textual features and The final submatrix and Represented as: 。 9. The emotion recognition method as described in claim 2, characterized in that, Using graph attention network graph structure data Learning about the importance of different nodes allows for richer emotional information output. The methods include: Will As the initial node representation in the graph, the attention coefficient between nodes i and j is first calculated: ； in and Let be the hidden states of nodes i and j, respectively; let 'a' represent the learnable parameters; and let || represent the vector concatenation operation. This represents the correlation between node i and node j; ； ； in, Is using The weight matrix obtained after masking the attention coefficients is further processed to obtain... ; ( ； Where K represents the number of attention heads. , It is a learnable weight matrix. The outputs of multiple attention heads are concatenated and then subjected to a linear transformation to obtain the final node feature representation. .

10. The emotion recognition method as described in claim 2, characterized in that, It also includes optimizing the model using the cross-entropy loss function and generating joint prediction results of aspect terms and their sentiment polarity by weighting the contributions of multiple modules through hyperparameters.

Images