A multimodal sentiment analysis method based on tensor dynamic interaction and decoupling

Abstract

This invention discloses a multimodal sentiment analysis method based on tensor dynamic interaction and decoupling, belonging to the field of multimodal information processing. The method includes the following steps: Feature extraction, where the original modal data is input into different encoders for feature extraction to generate latent feature representations for the multimodal modes; Feature interaction and decoupling, where extended concatenation is performed and interactive features are generated using Cartesian product and Einstein summation conventions; Convolution and pooling, where interactive features are progressively extracted through convolution operations to extract higher-dimensional features, flattening the feature map into a one-dimensional feature vector, and processing each modal feature using a Transformer encoder; Step S4, Sentiment prediction, where the decoupled feature vector is input and sentiment label prediction is performed using a multilayer perceptron, with the prediction accuracy optimized using the mean absolute error loss function. This invention addresses the shortcomings of existing multimodal sentiment analysis models in handling intermodal interactions and feature decoupling.

Description

Technical Field

[0001] This invention relates to the field of multimodal information processing technology, and in particular to a multimodal sentiment analysis method based on tensor dynamic interaction and decoupling. Background Technology

[0002] In interpersonal communication, emotion plays a crucial role. Emotion is not only a reflection of an individual's inner experience but also an important signal of social interaction. With the continuous development of artificial intelligence technology, multimodal emotion analysis has gradually become a research hotspot, aiming to identify and understand human emotional states by analyzing various modalities such as speech, facial expressions, and text. Research in this field has not only promoted the naturalization and intelligence of human-computer interaction but also played an important role in multiple fields such as affective computing, social signal processing, and mental health monitoring. However, multimodal emotion recognition faces many challenges. First, the subjectivity of emotion means that different individuals may react very differently to the same emotional stimulus, making dataset labeling difficult. Second, the heterogeneity between different modalities leads to differences in feature distribution, complicating information fusion between modalities. Furthermore, how to design an effective fusion mechanism to fully utilize the complementarity of multimodal information is also a problem that urgently needs to be solved in this field.

[0003] In recent years, although some progress has been made in the field of multimodal sentiment recognition (e.g., extracting and learning multimodal features through deep learning models, a method for constructing a multimodal sentiment analysis model, analysis model and method (CN118468138A), a multimodal sentiment analysis method based on a multi-view bipolar similarity fusion network (CN117933265A), a multimodal sentiment analysis method based on similarity modality completion (CN117540007A), a Mongolian multimodal sentiment analysis method based on a pre-trained model and a high-resolution network (CN116738359A), a multimodal sentiment analysis method and system based on an attention mechanism (CN116563751A), and a multimodal sentiment analysis method based on dynamic gradient and multi-view collaborative attention (CN116204850A)), the main technical means of these multimodal sentiment analysis methods are: acquiring image, text and audio modal features, obtaining multimodal features through feature fusion, and finally using deep neural networks for sentiment analysis. These methods often rely on the feature representation capabilities of the feature extractor, ignoring the interaction of multimodal features, resulting in insufficient robustness of the extracted feature representations and limiting model performance. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal sentiment analysis method based on tensor dynamic interaction and decoupling. By extracting features from multimodal data such as text, audio and images, and dynamically interacting and decoupling tensor features, the method fully explores the interaction features between modalities, improves the accuracy and robustness of sentiment analysis, and solves the shortcomings of existing multimodal sentiment analysis models in handling intermodal interactions and feature decoupling.

[0005] To achieve the above objectives, this invention provides a multimodal sentiment analysis method based on tensor dynamic interaction and decoupling, comprising the following steps:

[0006] Step S1, Feature Extraction: Collect a multimodal sentiment intensity dataset, and input the collected raw modal data into different encoders for feature extraction to generate multimodal latent feature representations. The text modality is processed using a BERT encoder, while the audio and image modalities are processed using a recurrent neural network (RNN) and a long short-term memory network (LSTM) to capture temporal dependencies and generate feature vectors for each modality. Fully connected layers are used to map audio and visual features to a unified dimensional space, simplifying the complexity of feature processing.

[0007] Step S2, Feature Interaction: Based on the feature extraction, a bias term is added to each modal feature matrix. The modal features are linearly transformed by the parameter matrix obtained through training. The three modal features are combined pairwise and mapped to three dimensions by expansion and concatenation. The interaction relationship between the three dimensions is captured by the Cartesian product and Einstein summation convention to generate interactive features.

[0008] Step S3, Feature Decoupling: The Transformer encoder processes each modal feature to capture the information of each modality and obtain unique features. The contrastive loss function is used to maximize the difference between modal features and interaction features. The reconstruction loss function is used to decouple the unique features and interaction features and recover information from the original features.

[0009] Step S4, Convolution and Pooling: The calculated interaction features are processed through convolution to gradually extract higher-dimensional features, and pooling is used to reduce redundant information, flattening the feature map into a one-dimensional feature vector.

[0010] Step S5, Sentiment Prediction: The decoupled interaction features and feature-related features are concatenated and input, and sentiment label prediction is performed through a multilayer perceptron (MLP). The prediction accuracy is optimized by the mean absolute error (MAE) loss function.

[0011] Preferably, in step S1, a multimodal sentiment intensity dataset A is collected, where each data sample has a corresponding real number score y∈[-3.0,3.0], representing sentiment intensity: strong negative, negative, weak negative, neutral, weak positive, positive, and strong positive;

[0012] The original modal sequence X = {X1, X2, ..., X} L}={X t ,X a ,X v The text modality X is fed into different encoders for feature extraction. t The text is processed using a BERT encoder, which dynamically assigns weights to each word, capturing the complex relationships between words and generating the final text feature representation vector.

[0013] Audio Modality X a and image modality X v Feature extraction employs a recurrent neural network (RNN), which uses its recurrent structure to pass information from the previous time step to the current time step, capturing temporal dependencies in the sequence, processing dynamic changes in audio signals and the motion trajectories of objects in image sequences, and generating audio feature representation vectors. Image feature representation vector Key information is stored and retrieved using memory units of a single-layer unidirectional long short-term memory network (LSTM). The degree of attention to input information is dynamically adjusted through a gating mechanism, thereby capturing time dependence.

[0014] Using fully connected layers, visual and audio features are mapped to a unified dimension. The formula is as follows:

[0015]

[0016]

[0017] K t =FC(X) t );

[0018] Where m∈{a,v,t}, a,v,t represent audio, image, and text, respectively. and K represents the parameters of the Long Short-Term Memory (LSTM) network model related to the visual and audio modalities, respectively. m ∈R n×d R represents the projection result of the modal feature representation after LSTM encoding. The superscript n in R indicates that the size of the obtained feature dimension is n, d represents the unified feature dimension, and K represents the feature obtained after LSTM encoding.

[0019] Preferably, in step S2, features from different modalities are concatenated to fuse information from each pair of modalities, resulting in a fused feature representation:

[0020]

[0021]

[0022]

[0023] Features from various modalities are fused into the same dimensional space, and each concatenated feature contains information from both modalities.

[0024] Each concatenated feature matrix F x F y F z Add a column of all 1s to the end to expand the feature matrix of each dimension to 129 dimensions, as shown in the following formula:

[0025]

[0026]

[0027]

[0028] Among them 1 n A column vector consisting entirely of 1s, with dimensions n×1;

[0029] Define the learnable matrix for training and Applied to the expanded dimension matrix F x '、F y 'and F z A linear transformation is performed on the surface to generate a new feature representation, thereby extracting higher-dimensional features. The formula is as follows:

[0030]

[0031]

[0032]

[0033] New interactive features are generated by calculating the Cartesian product between dimensions. First, the feature matrices F of the two dimensions are... lx F ly Expand to fit the shape of the outer product so that they interact element-wise, as shown in the following formula:

[0034]

[0035]

[0036] Using Einstein's summation convention einsum, the Cartesian product is calculated, excluding repeated indices:

[0037] fusionx,y =einsum('nxt,nty→nxy',F lx ',F ly ');

[0038]

[0039] Dimension F l ' z Extended to fit the result of Cartesian product fusion a,v The interactive shapes are fused using the Einstein summation convention einsum to convert the Cartesian product result. a,v and dimension F l ' z The features are fused together, combining the three dimensions, using the following formula:

[0040]

[0041] fusion x,y,z =einsum('nxt,nty→nxy',fusion x,y ,F lz ');

[0042]

[0043] Preferably, in step S3, for each modal feature input into the Transformer encoder, the contextual information of the features within each modality is captured through a self-attention mechanism, generating a latent feature representation K. m ∈{K a ,K v ,K t}; Use a contrastive loss function to maximize the latent feature representation K m ∈{K a ,K v ,K t The distance between the fusion features and the modality allows each modality to independently express its information, as shown in the following formula:

[0044]

[0045] Among them, L spc This represents the loss function for unique features, where s is the distance threshold;

[0046] Based on Euclidean distance, we define the reconstruction information loss function L. rec By minimizing the distance between similar samples, the alignment and learning of reconstructed information can be achieved.

[0047]

[0048] Where: yij The labels are generated based on a similarity metric, d ij This represents the similarity between samples, where k is a preset threshold.

[0049] Preferably, in step S4, for three-dimensional tensor fusion x,y,z To address the problem of high-dimensional data processing and feature extraction, a 3×3×3 convolution kernel is used for multiple convolution operations to progressively extract higher-level features and reduce data dimensionality. After each convolution operation, max pooling is used to further reduce the dimensionality of the feature map. After convolution and pooling, the output feature map H is obtained. m Flattened into a one-dimensional vector V, the flattened vector V is connected to a fully connected layer for nonlinear transformation and feature integration, outputting a one-dimensional feature representation F. final The size of the output feature maps of convolutional and pooling layers is calculated using the following formula:

[0050]

[0051]

[0052]

[0053] F final =w·V+b;

[0054] In the formula, W is the width of the 3D tensor F, K is the size of the convolution kernel, P is the padding size, S is the stride, and P0 is the depth of the convolution kernel. p P' is the pooling window size, P' is the pooling fill size, and S is the pooling fill size. p Here, F is the pooling size, H is the feature vector size, w is the weight matrix, and b is the bias term; Output Size represents the output features, Pooled Size represents the pooled features, and F is the pooled size. final This represents the final characteristic obtained;

[0055] Use the flattening operation to transform the original three-dimensional tensor of the i-th sample into a one-dimensional tensor:

[0056]

[0057] Preferably, in step S5, the decoupled feature vector F task ∈{K m ,F con The input multimodal sentiment analysis task module uses a multilayer perceptron (MLP) architecture to predict sentiment labels. The output of the task module is represented as follows:

[0058]

[0059] The prediction error loss term L generated by the task module taskThe loss function, calculated using the Mean Absolute Error (MAE), penalizes the absolute difference between the predicted and the true label. The formula is as follows:

[0060]

[0061] In the formula, For the predicted sentiment label, y i This is a real label;

[0062] The sum of all loss terms constitutes the global loss function L for model training. total The calculation formula is as follows:

[0063] L total =αL spc +βL rec +L task ;

[0064] Here, α and β are hyperparameters used to control the weights of each loss term in the overall loss function.

[0065] Therefore, the multimodal sentiment analysis method based on tensor dynamic interaction and decoupling described above, as used in this invention, has the following beneficial effects:

[0066] This invention significantly improves the accuracy of emotion recognition and the generalization ability of the model by comprehensively analyzing information from multiple modalities, including text, audio, and images. The invention extracts features from the input raw modal data using different encoders: the text modality uses a BERT encoder, while the audio and image modalities are processed using recurrent neural networks (RNNs) and long short-term memory networks (LSTMs) to capture temporal dependencies and generate feature vectors for each modality. To simplify the complexity of subsequent feature fusion, audio and image features are mapped to a unified dimensional space. Then, through an extended concatenation operation, the features of the three modalities are combined pairwise and mapped to three dimensions. The Cartesian product and Einstein summation convention are used to capture the interaction relationships between these dimensions, generating high-dimensional interactive features, including modality-specific features, pairwise modal interaction features, and trimodal interaction features. These interactive features undergo convolution and pooling operations to further extract high-level features and reduce redundant information, ultimately flattening them into a one-dimensional feature vector. Subsequently, a Transformer encoder is used to process each modal feature using a self-attention mechanism to capture contextual information between modalities. A contrastive loss function is then used to maximize the difference between modal features and shared features, ensuring that each modality can independently express its information. Finally, the decoupled features are input into a multilayer perceptron module for sentiment label prediction. This invention, through the above method, successfully improves the accuracy and generalization ability of multimodal sentiment analysis, and ensures effective fusion and decoupling of features between modalities. It provides an innovative solution applicable to various fields such as social media analysis and human-computer interaction.

[0067] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0068] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages disclosed in the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of the present invention and are not intended to limit the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments in this application without creative effort are within the scope of protection of this application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout.

[0070] It should be noted that the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products, or devices.

[0071] Similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0072] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of this invention is usually placed when in use. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0073] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," and "connect" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0074] Example

[0075] like Figure 1 As shown, the multimodal sentiment analysis method based on tensor dynamic interaction and decoupling described in this invention includes the following steps:

[0076] Step S1, Feature Extraction: Collect a multimodal sentiment intensity dataset. Input the collected raw modal data into different encoders for feature extraction to generate multimodal latent feature representations. The text modality is processed using a BERT encoder, while the audio and image modalities are processed using a recurrent neural network (RNN) and a long short-term memory network (LSTM) to capture temporal dependencies and generate feature vectors for each modality. Fully connected layers are used to map audio and visual features to a unified dimensional space, simplifying the complexity of feature processing.

[0077] Step S1 specifically includes:

[0078] ① Data preparation stage

[0079] Collect a multimodal sentiment intensity dataset A. Each data sample has a corresponding real number score y∈[-3.0,3.0], representing sentiment intensity: strong negative, negative, weak negative, neutral, weak positive, positive, and strong positive.

[0080] ② Model Design Stage

[0081] The original modal sequence X = {X1, X2, ..., X} L}={X t ,X a ,X v The data are fed into different encoders for feature extraction, generating rich latent feature representations.

[0082] Text Modality X t The BERT encoder is used for processing. BERT's multi-layered stacked structure and its pre-trained self-attention mechanism enable the model to fully learn semantic information and demonstrate superior ability in handling context. Through the self-attention mechanism, BERT can dynamically assign weights to each word, capturing complex relationships between words, thereby generating the final text feature representation vector.

[0083] Audio Modality X a and image modality X v Feature extraction employs a recurrent neural network (RNN), whose recurrent structure effectively transmits information from the previous time step to the current time step, capturing temporal dependencies in the sequence. This characteristic makes RNNs particularly suitable for processing dynamic changes in audio signals and the motion trajectories of objects in image sequences, generating audio feature representation vectors. Image feature representation vector To further enhance the understanding of temporal information in visual and audio modalities, a single-layer unidirectional Long Short-Term Memory (LSTM) network was employed. The memory units of LSTM can store and retrieve key information, dynamically adjusting the level of attention to input information through a gating mechanism, thereby effectively capturing temporal dependencies. This structure better reflects tone changes and emotional expression in audio feature sequences, while in image feature sequences it helps understand inter-frame motion and dynamic scene changes. To ensure that subsequent network models can effectively handle feature representations of different modalities, fully connected layers are used to map visual and audio features to a unified dimension. This unified feature dimension simplifies the complexity of feature processing, enabling the model to more easily process information from different modalities and leverage the complementarity between modalities to enhance overall performance. The mathematical expression and formulas for the entire process are detailed in the following equation:

[0084]

[0085]

[0086] K t =FC(X) t );

[0087] In the formula: m∈{a,v,t}, a,v,t represent audio, image, and text, respectively. and K represents the parameters of the Long Short-Term Memory (LSTM) network models related to the visual and audio modalities, respectively. m ∈R n×d This represents the projection result of the modal feature representation after LSTM encoding. The superscript n in R indicates that the size of the obtained feature dimension is n, d represents the unified feature dimension, and K represents the feature obtained after LSTM encoding. This process ensures the consistency of different modal features in terms of dimension, providing a foundation for subsequent fusion and processing.

[0088] Step S2, Feature Interaction: Based on the feature extraction, a bias term is added to each modal feature matrix. The modal features are linearly transformed using the parameter matrix obtained through training. The three modal features are combined pairwise and mapped to three dimensions through extended concatenation. The interaction relationship between the three dimensions is captured by using the Cartesian product and Einstein summation convention to generate interactive features.

[0089] Step S2 specifically includes:

[0090] A bias term is added to each modality feature matrix to enhance feature representation. After linearly transforming the modality features using the parameter matrices learned during training, the Cartesian product and Einstein summation conventions are used to capture the interaction relationships between modalities, thereby generating higher-dimensional interaction feature representations. In this process, features from different modalities are effectively fused and strengthened through interaction, promoting information sharing between modalities. Finally, these interaction features are compressed into three 128-dimensional feature vectors through flattening and average pooling operations, providing concise and effective input for subsequent feature decoupling.

[0091] To further explore the deep-seated interactions between modalities, the concatenated feature matrix is ​​expanded, and the interaction features between modalities are calculated using Cartesian product (outer product). This approach not only preserves the individual features of each modality but also learns pairwise or triadic interaction features between different modalities, thereby improving the model's expressive power.

[0092] This is achieved by concatenating features from different modalities to fuse information from each pair of modalities. This concatenation operation joins the features of two modalities together to obtain a fused feature representation.

[0093]

[0094]

[0095]

[0096] By fusing features from various modalities into a single dimensional space, each concatenated feature contains information from both modalities, enabling the model to process information from different modalities simultaneously and improving its ability to model complex tasks. When information from one modality is insufficient or affected by noise, features from other modalities can provide strong supplementation, offering more expressive dimensions for subsequent feature learning and thus enhancing the model's robustness. In each concatenated feature matrix F... x F y F z Adding a column of all 1s to the end expands the feature matrix of each dimension to 129 dimensions, allowing for better capture of the relationships between dimensions during subsequent linear transformations. The mathematical expression is:

[0097]

[0098]

[0099]

[0100] Among them 1 n A column vector consisting entirely of 1s, with a dimension of n×1.

[0101] To extract higher-dimensional features, a learnable matrix for training is defined. and These matrices will be applied to the expanded dimension matrix F. x '、F y 'and F z A linear transformation is performed on the surface to generate a new feature representation, thereby extracting higher-dimensional features.

[0102]

[0103]

[0104]

[0105] To capture the interaction relationships between different dimensions, new interaction features are generated by calculating the Cartesian product (outer product) between dimensions. First, the feature matrices F of the two dimensions are... lx F ly Expand to fit the outer shape so that they can interact element-wise.

[0106]

[0107]

[0108] The Cartesian product (outer product) is calculated using the Einstein summation convention (einsum), with particular care that duplicate indices are not summed:

[0109] fusion x,y =einsum('nxt,nty→nxy',F lx ',F ly ');

[0110]

[0111] Dimension F l ' z Extended to fit the result of Cartesian product fusion a,v The interactive shapes are fused using the Einstein summation convention (einsum) to represent the Cartesian product. a,v and dimension F l ' z The features of the three dimensions are integrated.

[0112]

[0113] fusion x,y,z =einsum('nxt,nty→nxy',fusion x,y ,F lz ');

[0114]

[0115] Step S3, Feature Decoupling: The Transformer encoder processes each modal feature to capture information of each modality and obtain unique features. The contrastive loss function is used to maximize the difference between modal features and interaction features. The reconstruction loss function is used to decouple unique features and interaction features and recover information from the original features.

[0116] Step S3 specifically includes:

[0117] For each modal feature input into the Transformer encoder, the contextual information of the features within each modality is captured through a self-attention mechanism, generating a latent feature representation K. m ∈{K a ,K v ,K t}; Use a contrastive loss function to maximize the latent feature representation K m ∈{K a ,K v,K t The distance between the fusion features and the modality allows each modality to independently express its information, as shown in the following formula:

[0118]

[0119] Among them, L spc This represents the loss function for unique features, where s is the distance threshold;

[0120] Based on Euclidean distance, we define the reconstruction information loss function L. rec During the optimization process, the alignment and learning of reconstructed information are achieved by minimizing the distance between similar samples and penalizing dissimilar samples that are too close together.

[0121]

[0122] Where: y ij The labels are generated based on a similarity metric, d ij This represents the similarity between samples, where k is a preset threshold.

[0123] Step S4, Convolution and Pooling: The calculated interactive features are processed through convolution to gradually extract higher-dimensional features, and pooling is used to reduce redundant information, flattening the feature map into a one-dimensional feature vector.

[0124] For 3D tensor fusion x,y,z To address the challenges of high-dimensional data processing and feature extraction, a 3×3×3 convolution kernel is used for multiple convolution operations to progressively extract higher-level features and reduce data dimensionality. After each convolution operation, max pooling is employed to further reduce the dimensionality of the feature map, effectively resolving issues of feature redundancy and excessive computational complexity. The output feature map H is obtained after convolution and pooling. m The vector is flattened into a one-dimensional vector V. Finally, the flattened vector V is connected to a fully connected layer for nonlinear transformation and feature integration, ultimately outputting a one-dimensional feature representation F. final Through this series of steps, the complex interactions between modal features were successfully captured. These interactions not only provide the model with richer and more diverse information but also significantly enhance its ability to handle complex tasks, enabling it to more accurately understand and distinguish information from different modalities. The size of the output feature maps of the convolutional and pooling layers can be calculated using the following formula:

[0125]

[0126]

[0127]

[0128] F final=w·V+b;

[0129] In the formula, W is the width of the 3D tensor F, K is the size of the convolution kernel, P is the padding size, S is the stride, and P0 is the depth of the convolution kernel. p P' is the pooling window size, P' is the pooling fill size, and S is the pooling fill size. p Here, F is the pooling size, H is the feature vector size, w is the weight matrix, and b is the bias term; Output Size represents the output features, Pooled Size represents the pooled features, and F is the pooled size. final This represents the final characteristic obtained.

[0130] Finally, the flattening operation is used to transform the original three-dimensional tensor of the i-th sample into a one-dimensional tensor:

[0131]

[0132] Step S5, Sentiment Prediction: The decoupled interaction features and feature-related features are concatenated and input, and sentiment label prediction is performed through a multilayer perceptron (MLP). The prediction accuracy is optimized by the mean absolute error (MAE) loss function.

[0133] Step S4 specifically includes:

[0134] The decoupled feature vector F task ∈{K m ,F con The input multimodal sentiment analysis task module uses a multilayer perceptron (MLP) architecture to predict sentiment labels. The output of the task module can be represented as:

[0135]

[0136] To optimize sentiment analysis performance, the prediction error loss term L generated by the task module... task The loss function, calculated using Mean Absolute Error (MAE), penalizes the absolute difference between the predicted and the true label, helping the model optimize its performance in sentiment analysis tasks. The specific formula is as follows:

[0137]

[0138] In the formula, For the predicted sentiment label, y i This is a real label.

[0139] Furthermore, the sum of all loss terms constitutes the global loss function L for model training. total This function not only integrates the objectives of each task but also provides the model with comprehensive optimization directions, ensuring its optimal performance in multimodal sentiment analysis. Its calculation formula is shown below:

[0140] L total =αL spc +βL rec +L task ;

[0141] Where α and β are hyperparameters used to control the weights of each loss term in the overall loss function.

[0142] The specific steps during the model training phase are as follows:

[0143] ① Divide the multimodal sentiment intensity dataset A obtained in step S1 into a training set T, a validation set V, and a test set I in a ratio of 8:1:1;

[0144] ② Modal features are extracted for each sample on the training set T. A fully connected layer maps the features of different modalities to a unified dimensional space, ensuring feature consistency and obtaining the modal feature representation K. m ∈{K a ,K v ,K t}, where K represents K respectively. a , K v , K t Features of audio, image and text modalities

[0145] ③ By concatenating and expanding features from different modalities, and using the Cartesian product and Einstein summation convention (einsum) to calculate the interaction relationships between modalities, higher-dimensional interaction features are obtained. This process not only preserves the unique information of each modality but also learns pairwise and trimodal interaction features between modalities. Further convolution and pooling processes were used to capture the complex interaction between modality-specific features and shared features.

[0146] ④ By optimizing interaction features Modal feature representation K m ∈{K a ,K v ,K t The differences between them further enhance the expressive power of each modality's features and improve the robustness of the model.

[0147] ⑤ By minimizing the global loss function using the formula, the training phase simultaneously uses the validation set V to verify the model training effect and optimize the remaining modules of the model. When the model M achieves good results on the validation set V and the accuracy does not improve with the training process, training is stopped, and finally the training is completed to obtain the model M'.

[0148] The specific steps in the model testing phase are as follows:

[0149] ① Input the multimodal sentiment intensity dataset test set I into the trained model M' to obtain the sentiment prediction results of the model for the test set samples.

[0150] ② Compare the test results with the preset performance indicators to determine whether the model has achieved the expected results. If necessary, consider adjusting the model or retraining it.

[0151] ③ After the testing phase is completed, compile the final performance report of the model and visualize it to facilitate subsequent research and application.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal sentiment analysis method based on tensor dynamic interaction and decoupling, characterized in that: Includes the following steps: Step S1, Feature Extraction: Collect a multimodal sentiment intensity dataset, and input the collected raw modal data into different encoders for feature extraction to generate multimodal latent feature representations. The text modality is processed using a BERT encoder, while the audio and image modalities are processed using a recurrent neural network (RNN) and a long short-term memory network (LSTM) to capture temporal dependencies and generate feature vectors for each modality. Fully connected layers are used to map audio and visual features to a unified dimensional space, simplifying the complexity of feature processing. Step S2, Feature Interaction: Based on the feature extraction, a bias term is added to each modal feature matrix. The modal features are linearly transformed by the parameter matrix obtained through training. The three modal features are combined pairwise and mapped to three dimensions by expansion and concatenation. The interaction relationship between the three dimensions is captured by the Cartesian product and Einstein summation convention to generate interactive features. Step S3, Feature Decoupling: The Transformer encoder processes each modal feature to capture the information of each modality and obtain unique features. The contrastive loss function is used to maximize the difference between modal features and interaction features. The reconstruction loss function is used to decouple the unique features and interaction features and recover information from the original features. Step S4, Convolution and Pooling: The calculated interaction features are processed through convolution to gradually extract higher-dimensional features, and pooling is used to reduce redundant information, flattening the feature map into a one-dimensional feature vector. Step S5, Sentiment Prediction: The decoupled interaction features and unique features are concatenated and input, and sentiment label prediction is performed through a multilayer perceptron (MLP). The prediction accuracy is optimized by the mean absolute error (MAE) loss function. In step S1, a multimodal sentiment intensity dataset is collected. Each data sample has a corresponding real score. The values ​​represent the intensity of emotion: strong negative, negative, weak negative, neutral, weak positive, positive, and strong positive. Original mode sequence The text modalities are input into different encoders for feature extraction. The text is processed using a BERT encoder, which dynamically assigns weights to each word, capturing the complex relationships between words and generating the final text feature representation vector. ; audio modality and image modality Feature extraction employs a recurrent neural network (RNN), which uses its recurrent structure to pass information from the previous time step to the current time step, capturing temporal dependencies in the sequence, processing dynamic changes in audio signals and the motion trajectories of objects in image sequences, and generating audio feature representation vectors. Image feature representation vector The system uses a single-layer unidirectional long short-term memory (LSTM) network to store and retrieve key information, and dynamically adjusts the level of attention to input information through a gating mechanism to capture time dependence. Using fully connected layers, visual and audio features are mapped to a unified dimension. The formula is as follows: ； ； ； in, , Representing audio, image, and text respectively. and These represent the parameters of the Long Short-Term Memory (LSTM) network models related to the visual and audio modalities, respectively. In step S2, features from different modalities are concatenated to fuse information from each pair of modalities, resulting in a fused feature representation. ； ； ； Features from various modalities are fused into the same dimensional space, and each concatenated feature contains information from both modalities. Each concatenated feature matrix , , Add a column of all 1s to the end to expand the feature matrix of each dimension to 129 dimensions, as shown in the following formula: ； ； ； in A column vector consisting entirely of 1s, with dimensions n×1; Define the learnable matrix for training , and Applied to the expanded dimension matrix , and A linear transformation is performed on the surface to generate a new feature representation, thereby extracting higher-dimensional features, as shown in the following formula: ； ； ； New interactive features are generated by calculating the Cartesian product between dimensions. First, the feature matrices of the two dimensions are... , Expand to fit the shape of the outer product so that they interact element-wise, as shown in the following formula: ； ； Using Einstein's summation convention einsum, the Cartesian product is calculated, excluding repeated indices: ； ； Dimension Extended to fit the result of the Cartesian product The interactive shapes are converted to Cartesian products using the Einstein summation convention einsum. and dimensions The features are fused together, combining the three dimensions, using the following formula: ； ； 。 2. The multimodal sentiment analysis method based on tensor dynamic interaction and decoupling according to claim 1, characterized in that: In step S3, for each modal feature, the input is fed into the Transformer encoder, and the contextual information of the features within each modality is captured through a self-attention mechanism to generate a latent feature representation. Use a contrastive loss function to maximize the latent feature representation. The distance between the fused features and the modality allows each modality to independently express its information, as shown in the following formula: ； in, Represents the loss function for unique features. It is a distance threshold; Based on Euclidean distance, a reconstruction information loss function is defined. By minimizing the distance between similar samples, the alignment and learning of reconstructed information can be achieved. in: The tags are generated based on similarity metrics. Indicates the similarity between samples. It is a preset threshold.

3. The multimodal sentiment analysis method based on tensor dynamic interaction and decoupling according to claim 2, characterized in that: In step S4, for the three-dimensional tensor To address the problem of high-dimensional data processing and feature extraction, a 3×3×3 convolution kernel is used for multiple convolution operations to progressively extract higher-level features and reduce data dimensionality. After each convolution operation, max pooling is used to further reduce the dimensionality of the feature map. The output feature map after convolution and pooling processing is... Flattened into a one-dimensional vector The flattened vector Connected to a fully connected layer, it performs nonlinear transformations and feature integration, outputting a one-dimensional feature representation. The size of the output feature maps of convolutional and pooling layers is calculated using the following formula: ； ； ； ； In the formula, It is a three-dimensional tensor width, It is the size of the convolution kernel. It is the size of the fill. It's stride. It is the pooling window size. It is the fill size of the pooling. It refers to the size of the pool area. It is the size of the feature vector. It is a weight matrix. It is a bias term; Represents output characteristics. Represents the characteristics after pooling. This represents the final characteristic obtained; Use the flattening operation to transform the original three-dimensional tensor of the i-th sample into a one-dimensional tensor: 。 4. The multimodal sentiment analysis method based on tensor dynamic interaction and decoupling according to claim 3, characterized in that: In step S5, the decoupled feature vectors are... The input to the multimodal sentiment analysis task module uses a multilayer perceptron (MLP) architecture to predict sentiment labels. The output of the task module is represented as follows: ； The prediction error loss term generated by the task module The loss function, calculated using the Mean Absolute Error (MAE), penalizes the absolute difference between the predicted and the true label. The formula is as follows: ； In the formula, For predicted sentiment labels, This is a real label; The sum of all loss terms constitutes the global loss function for model training. The calculation formula is as follows: ； in, , These are hyperparameters used to control the weight of each loss term in the overall loss function.

Images