Multi-modal sentiment analysis method based on tensor dynamic interaction and decoupling

Abstract

The invention discloses a multi-modal sentiment analysis method based on tensor dynamic interaction and decoupling, which belongs to the field of multi-modal information processing, and comprises the following steps: feature extraction: respectively inputting original modal data into different encoders for feature extraction to generate multi-modal potential feature representation; performing feature interaction and decoupling, expanding and splicing, and generating interaction features by using a Cartesian product and Einstein summation agreement; convolution and pooling: performing convolution operation on the interactive features to gradually extract features with higher dimensions, flattening a feature map into a one-dimensional feature vector, and processing each modal feature by using a Transform encoder; and S4, emotion prediction: inputting the decoupled feature vectors, performing emotion label prediction through a multi-layer perceptron, and optimizing prediction precision through a mean absolute error loss function. According to the method, the defects of an existing multi-modal sentiment analysis model in the aspects of processing inter-modal interaction and feature decoupling can be overcome.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal information processing, and in particular to a multimodal sentiment analysis method based on tensor dynamic interaction and decoupling. Background Art

[0002] In the process of communication between people, emotions play a vital role. Emotions are not only a reflection of an individual's inner experience, but also an important signal of social interaction. With the continuous development of artificial intelligence technology, multimodal sentiment analysis has gradually become a research hotspot, aiming to identify and understand human emotional states by analyzing multiple modal information such as human speech, facial expressions, and text. Research in this field has not only promoted the naturalization and intelligence of human-computer interaction, but also played an important role in many fields such as emotional computing, social signal processing, and mental health monitoring. However, multimodal emotion recognition faces many challenges. First, the subjectivity of emotions makes different individuals react to the same emotional stimulus in very different ways, which makes it difficult to label the data set. Secondly, the heterogeneity between different modalities leads to differences in feature distribution, which complicates the information fusion between modalities. In addition, how to design an effective fusion mechanism to make full use of the complementarity of multimodal information is also an urgent problem to be solved in this field.

[0003] In recent years, although some progress has been made in the field of multimodal emotion recognition (for example, extracting and learning multimodal features through deep learning models, a multimodal emotion analysis model construction method, analysis model and method (CN118468138A), a multimodal emotion analysis method based on a multi-perspective bipolar similarity fusion network (CN117933265A), a multimodal emotion analysis method based on similar modality completion (CN117540007A), a Mongolian multimodal emotion analysis method based on a pre-trained model and a high-resolution network (CN116738359A), a multimodal emotion analysis method and system based on an attention mechanism (CN116563751A), a multimodal emotion analysis method based on dynamic gradient and multi-view collaborative attention (CN116204850A)), the main technical means of these multimodal emotion analysis methods are: obtaining image, text and audio modal features, and obtaining multimodal features through feature fusion, and finally using deep neural networks for emotion analysis. These methods often rely on the feature representation capabilities of the feature extractor and ignore the interaction of multimodal features, resulting in the extracted feature representation being not robust enough and limiting the model performance. Summary of the invention

[0004] The purpose of the present invention is to provide a multimodal sentiment analysis method based on tensor dynamic interaction and decoupling. By performing feature extraction, tensor feature dynamic interaction and decoupling on multimodal data such as text, audio and image, the interaction features between modalities are fully explored, the accuracy and robustness of sentiment analysis are improved, and the deficiencies of existing multimodal sentiment analysis models in dealing with inter-modal interaction and feature decoupling are addressed.

[0005] To achieve the above object, the present invention provides a multimodal sentiment analysis method based on tensor dynamic interaction and decoupling, comprising the following steps:

[0006] Step S1, feature extraction: collect multimodal sentiment intensity datasets, input the collected raw modal data into different encoders for feature extraction to generate multimodal latent feature representations, use BERT encoder to process text modality, and use recurrent neural network (RNN) and long short-term memory (LSTM) to process audio and image modalities to capture temporal dependencies and generate feature vectors for each modality, and use a fully connected layer to map audio and visual features to a unified dimensional space to simplify the complexity of feature processing;

[0007] Step S2, feature interaction: After feature extraction, add a bias term to each modal feature matrix, perform linear transformation on the modal features through the parameter matrix obtained through training, map the three modal features to three dimensions by extending and splicing, and use the Cartesian product and Einstein summation convention to capture the interactive relationship of these three dimensions to generate interactive features;

[0008] Step S3, feature decoupling: each modal feature is processed by the Transformer encoder to capture the information of each modality to obtain the unique features, the contrast loss function is used to maximize the difference between the modal features and the interaction features, the reconstruction loss function is used to decouple the unique features and the interaction features, and the information is recovered from the original features;

[0009] Step S4, convolution and pooling: the calculated interaction features are subjected to convolution operations to gradually extract higher-dimensional features, and the pooling method is used to reduce redundant information and flatten the feature map into a one-dimensional feature vector;

[0010] Step S5, sentiment prediction: The decoupled interaction features are concatenated with the feature features and then input, sentiment labels are predicted through the multi-layer perceptron (MLP), and the prediction accuracy is optimized through the mean absolute error (MAE) loss function.

[0011] Preferably, in step S1, a multimodal sentiment intensity dataset A is collected, and each data sample has a corresponding real number score y∈[-3.0,3.0], representing the sentiment intensity: strongly negative, negative, weakly negative, neutral, weakly positive, positive, and strongly positive;

[0012] The original modal sequence X={X1,X2,…,X L}={X t ,X a ,X v} are input into different encoders for feature extraction, text modality X t The BERT encoder is used for processing. BERT dynamically assigns weights to each word, captures the complex relationship between words, and generates the final text feature representation vector.

[0013] Audio Mode X a and image modality X v The feature extraction uses a recurrent neural network (RNN), which transmits information from the previous moment to the current moment through its cyclic structure, captures the temporal dependencies in the sequence, processes the dynamic changes in the audio signal and the motion trajectory of objects in the image sequence, and generates an audio feature representation vector. and image feature representation vector The memory unit of a single-layer unidirectional long short-term memory network LSTM is used to store and extract key information, and the degree of attention to the input information is dynamically adjusted through a gating mechanism to capture time dependency.

[0014] Use fully connected layers to map visual and audio features to a unified dimension The formula is as follows:

[0015]

[0016]

[0017] K t =FC(X t );

[0018] Among them, m∈{a,v,t}, a,v,t represent audio, image and text respectively. and Denote the parameters of the LSTM model related to the visual modality and the audio modality, respectively, K m ∈R n×d It represents the projection result of the modal feature representation after LSTM encoding. The superscript n in R represents that the feature dimension size is n, d represents the unified feature dimension, and K represents the feature obtained after LSTM encoding.

[0019] Preferably, in step S2, features of different modalities are concatenated so as to fuse information from each pair of modalities together to obtain a fused feature representation:

[0020]

[0021]

[0022]

[0023] The features of each modality are merged into the same dimensional space. Each concatenated feature contains information from two modalities.

[0024] After each concatenation, the feature matrix F x , F y , F z Add a column of all 1s at the end of , expanding the feature matrix of each dimension to 129 dimensions. The formula is as follows:

[0025]

[0026]

[0027]

[0028] 1 of them n A column vector of all 1s, with dimension n×1;

[0029] Define the learnable matrix for training and Applied to the expanded dimension matrix F x ', F y ' and F z ' performs linear transformation on it to generate new feature representations, thereby extracting higher-dimensional features. The formula is as follows:

[0030]

[0031]

[0032]

[0033] New interaction features are generated by calculating the Cartesian product between dimensions. First, the feature matrix F of the two dimensions is transformed into lx 、F ly Expanded to a shape suitable for the outer product, so that they interact element-wise, the formula is as follows:

[0034]

[0035]

[0036] The Cartesian product is calculated using the Einstein summation convention einsum, without summing repeated indices:

[0037] fusionx,y =einsum('nxt,nty→nxy',F lx ',F ly ');

[0038]

[0039] The dimension F l ' z Expanded to fit the Cartesian product result fusion a,v To interact with the shapes, use the Einstein summation convention einsum to fuse the Cartesian product results a,v and dimension F l ' z To fuse the features of three dimensions, the formula is as follows:

[0040]

[0041] fusion x,y,z =einsum('nxt,nty→nxy',fusion x,y ,F lz ');

[0042]

[0043] Preferably, in step S3, each modal feature is input into the Transformer encoder, and the contextual information of the internal features of each modality is captured through the self-attention mechanism, and the generated latent feature representation K m ∈{K a ,K v ,K t}; Use contrastive loss function to maximize the latent feature representation K m ∈{K a ,K v ,K t} and the distance between the fusion features, prompting each modality to express its information independently. The formula is as follows:

[0044]

[0045] Among them, L spc represents the unique feature loss function, s is the distance threshold;

[0046] Based on the Euclidean distance, the reconstruction information loss function L is defined rec , by minimizing the distance between similar samples, the alignment and learning of reconstruction information are achieved:

[0047]

[0048] Where: yij is the label generated based on the similarity measure, d ij Indicates the similarity between samples, and k is the preset threshold.

[0049] Preferably, in step S4, for the three-dimensional tensor fusion x,y,z To solve the problem of high-dimensional data processing and feature extraction, a 3×3×3 convolution kernel is used to perform multiple convolution operations to gradually extract higher-level features and reduce the data dimension. After each convolution operation, the maximum pooling method is used to reduce the dimension of the feature map. After convolution and pooling, the output feature map H m Flatten it into a one-dimensional vector V, connect the flattened vector V to the fully connected layer, perform nonlinear transformation and feature integration, and output a one-dimensional feature representation F final ; The size of the output feature map of the convolution layer and the pooling layer is calculated by the following formula:

[0050]

[0051]

[0052]

[0053] F final =w·V+b;

[0054] Where W is the width of the three-dimensional tensor F, K is the size of the convolution kernel, P is the padding size, S is the stride, and P p is the pooling window size, P' is the pooling padding size, S p is the pooling size, H is the feature vector size, w is the weight matrix, and b is the bias term; Output Size represents the output feature, Pooled Size represents the pooled feature, and F final Represents the final characteristics;

[0055] Use the flatten operation to convert the original 3D tensor of the i-th sample into a 1D tensor:

[0056]

[0057] Preferably, in step S5, the decoupled feature vector F task ∈{K m ,F con} Input the multimodal sentiment analysis task module, and use the multi-layer perceptron MLP architecture to predict the sentiment label. The output of the task module is expressed as:

[0058]

[0059] The prediction error loss term L generated by the task module taskCalculated by the mean absolute error MAE, this loss function penalizes the absolute difference between the prediction and the true label. The formula is as follows:

[0060]

[0061] In the formula, is the predicted sentiment label, y i is the true label;

[0062] The accumulation of all loss terms constitutes the global loss function L for model training. total , the calculation formula is as follows:

[0063] L total =αL spc +βL rec +L task ;

[0064] Among them, α and β are hyperparameters used to control the weight of each loss term in the overall loss function.

[0065] Therefore, the present invention adopts the above-mentioned multimodal sentiment analysis method based on tensor dynamic interaction and decoupling, which has the following beneficial effects:

[0066] The present invention significantly improves the accuracy of emotion recognition and the generalization ability of the model by comprehensively analyzing multiple modal information such as text, audio and image; the present invention extracts features from the input raw modal data through different encoders, wherein the text modality uses the BERT encoder, and the audio and image modalities are processed by the recurrent neural network (RNN) and the long short-term memory network (LSTM) to capture the temporal dependency and generate the feature vectors of each modality. In order to simplify the complexity of subsequent feature fusion, the audio and image features are mapped to a unified dimensional space. Then, through the extended splicing operation, the features of the three modalities are combined and mapped to three dimensions, and the Cartesian product and Einstein summation convention (einsum) are used to capture the interactive relationship between these dimensions, and high-dimensional interactive features including modality-specific features, two-to-two modal interactive features, and three-modal interactive features are generated. These interactive features are subjected to convolution operations and pooling processing to further extract high-level features, reduce redundant information, and finally flatten into a one-dimensional feature vector. Subsequently, the Transformer encoder is used to process each modal feature with a self-attention mechanism to capture the contextual information between modalities, and the difference between modal features and shared features is maximized through the contrast loss function to ensure that each modality can express its information independently. Finally, the decoupled features are input into the multi-layer perceptron module for sentiment label prediction. Through the above method, the present invention successfully improves the accuracy of multimodal sentiment analysis and the generalization ability of the model, and ensures the effective fusion and decoupling of features between modalities, providing an innovative solution suitable for social media analysis and human-computer interaction.

[0067] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 The figure is a schematic diagram of a method flow of an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical scheme and advantages disclosed in the embodiments of the present invention clearer, the embodiments of the present invention are further described in detail in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present invention and are not intended to limit the embodiments of the present invention. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions.

[0070] It should be noted that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products or devices.

[0071] Like reference numerals and letters denote similar items in the following drawings, and thus, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0072] In the description of the present invention, it should be noted that the terms "upper", "lower", "inside", "outside", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, or are directions or positional relationships in which the product of the invention is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.

[0073] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "setting", "installation" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0074] Example

[0075] like Figure 1 As shown, the multimodal sentiment analysis method based on tensor dynamic interaction and decoupling described in the present invention includes the following steps:

[0076] Step S1, feature extraction: collect multimodal sentiment intensity datasets, input the collected original modal data into different encoders for feature extraction to generate multimodal latent feature representations, the text modality is processed by the BERT encoder, and the audio and image modalities are processed by the recurrent neural network RNN ​​and the long short-term memory network LSTM to capture temporal dependencies and generate feature vectors for each modality. The fully connected layer is used to map the audio and visual features to a unified dimensional space to simplify the complexity of feature processing.

[0077] Step S1 specifically includes:

[0078] ①Data preparation stage

[0079] A multimodal sentiment intensity dataset A is collected. Each data sample has a corresponding real number score y∈[-3.0,3.0], representing the sentiment intensity: strongly negative, negative, weakly negative, neutral, weakly positive, positive, and strongly positive.

[0080] ②Model design stage

[0081] The original modal sequence X={X1,X2,…,X L}={X t ,X a ,X v} are input into different encoders for feature extraction to generate rich potential feature representations.

[0082] Text Modal X t The BERT encoder is used for processing. BERT's multi-layer stacking structure and its pre-trained self-attention mechanism enable the model to fully learn semantic information and show superior ability in processing context. Through the self-attention mechanism, BERT can dynamically assign weights to each word, capture the complex relationship between words, and thus generate the final text feature representation vector.

[0083] Audio Mode X a and image modality X v Recurrent Neural Network (RNN) is used for feature extraction. Its cyclic structure effectively transmits information from the previous moment to the current moment, capturing the temporal dependencies in the sequence. This feature makes RNN particularly suitable for processing dynamic changes in audio signals and the motion trajectory of objects in image sequences, generating audio feature representation vectors. and image feature representation vector To further enhance the understanding of temporal information of visual and audio modalities, a single-layer unidirectional long short-term memory network (LSTM) is also used. The memory unit of LSTM can store and extract key information, and dynamically adjust the degree of attention to input information through the gating mechanism, thereby effectively capturing time dependencies. This structure can better reflect tone changes and emotional expressions in audio feature sequences, while in image feature sequences it helps to understand inter-frame motion and dynamic scene changes. In order to ensure that the subsequent network model can effectively handle feature representations of different modalities, a fully connected layer is used to map visual and audio features to a unified dimension. This unified feature dimension simplifies the complexity of feature processing, and the model can more conveniently process different modal information and use the complementarity between the modes to enhance the overall performance. The mathematical expression and formula of the whole process are as follows:

[0084]

[0085]

[0086] K t =FC(X t );

[0087] Where: m∈{a,v,t}, a,v,t represent audio, image and text respectively, and denote the parameters of the long short-term memory network (LSTM) model related to the visual modality and the audio modality, respectively, and K m ∈R n×d It represents the projection result of the modal feature representation after LSTM encoding. The superscript n in R represents the feature dimension size n, d represents the unified feature dimension, and K represents the feature obtained after LSTM encoding. This process ensures the consistency of different modal features in dimension, which provides a basis for subsequent fusion and processing.

[0088] Step S2, feature interaction: After feature extraction, add a bias term to each modal feature matrix, perform linear transformation on the modal features through the parameter matrix obtained through training, map the three modal features into three dimensions in pairs through extended concatenation, and use the Cartesian product and Einstein summation convention to capture the interactive relationship between the three dimensions and generate interactive features.

[0089] Step S2 specifically includes:

[0090] A bias term is added to each modal feature matrix to enhance the feature expression capability. After the modal features are linearly transformed by the parameter matrix learned through training, the Cartesian product and Einstein sum convention are used to capture the interaction between the modalities, thereby generating a higher-dimensional interactive feature representation. In this process, the features of different modalities are effectively fused and strengthened through interaction, which promotes information sharing between modalities. Finally, these interactive features are compressed into three 128-dimensional feature vectors through flattening and average pooling operations, providing concise and effective input for subsequent feature decoupling.

[0091] In order to further explore the deep interaction relationship between modalities, the concatenated feature matrix is ​​expanded, and the interaction features between modalities are calculated through Cartesian product (outer product). In this way, not only can the characteristics of each modality be retained, but also the pairwise or tri-way interaction features between different modalities can be learned, thereby improving the expressiveness of the model.

[0092] The features of different modalities are concatenated to fuse the information from each pair of modalities. This concatenation operation connects the features of the two modalities together to obtain a fused feature representation:

[0093]

[0094]

[0095]

[0096] The features of each modality are integrated into the same dimensional space. Each concatenated feature contains information from two modalities, which enables the model to process information from different modalities at the same time and improves the modeling ability of the model for complex tasks. When the information of a certain modality is insufficient or affected by noise, the features of other modalities can provide a powerful supplement, providing more expression dimensions for subsequent feature learning, thereby enhancing the robustness of the model. In each concatenated feature matrix F x , F y , F z Adding a column of all 1s at the end of , that is, expanding the feature matrix of each dimension to 129 dimensions, can better capture the relationship between dimensions when performing subsequent linear transformations. The mathematical expression is:

[0097]

[0098]

[0099]

[0100] 1 of them n A column vector of all ones, of dimension n×1.

[0101] In order to extract higher dimensional features, a training learnable matrix is ​​defined and These matrices will be applied to the expanded dimension matrix F x ', F y ' and F z ' to generate new feature representations and thus extract higher-dimensional features.

[0102]

[0103]

[0104]

[0105] In order to capture the interaction between different dimensions, new interaction features are generated by calculating the Cartesian product (outer product) between dimensions. First, the feature matrix F of the two dimensions is lx 、F ly Expanded to a shape suitable for the outer product so that they can interact element-wise.

[0106]

[0107]

[0108] The Cartesian product (outer product) is calculated using the Einstein summation convention (einsum). In particular, repeated indices are not summed:

[0109] fusion x,y =einsum('nxt,nty→nxy',F lx ',F ly ');

[0110]

[0111] The dimension F l ' z Expanded to fit the Cartesian product result fusion a,v Interacting shapes, using the Einstein summation convention (einsum) to fuse the Cartesian product results a,v and dimension F l ' z Fusion, integration of three-dimensional features.

[0112]

[0113] fusion x,y,z =einsum('nxt,nty→nxy',fusion x,y ,F lz ');

[0114]

[0115] Step S3, feature decoupling: each modal feature is processed through the Transformer encoder to capture the information of each modality to obtain the unique features, the contrast loss function is used to maximize the difference between the modal features and the interaction features, the reconstruction loss function is used to decouple the unique features and the interaction features, and the information is recovered from the original features.

[0116] Step S3 specifically includes:

[0117] For each modal feature input into the Transformer encoder, the contextual information of the internal features of each modality is captured through the self-attention mechanism, and the generated latent feature representation K m ∈{K a ,K v ,K t}; Use contrastive loss function to maximize the latent feature representation K m ∈{K a ,K v,K t} and the distance between the fusion features, prompting each modality to express its information independently. The formula is as follows:

[0118]

[0119] Among them, L spc represents the unique feature loss function, s is the distance threshold;

[0120] Based on the Euclidean distance, the reconstruction information loss function L is defined rec In the optimization process, the distance between similar samples is minimized and the distance between dissimilar samples is penalized to achieve alignment and learning of reconstruction information:

[0121]

[0122] Where: y ij is the label generated based on the similarity measure, d ij Indicates the similarity between samples, and k is the preset threshold.

[0123] Step S4, convolution and pooling: The calculated interaction features are subjected to convolution operations to gradually extract higher-dimensional features, and the pooling method is used to reduce redundant information and flatten the feature map into a one-dimensional feature vector.

[0124] For three-dimensional tensor fusion x,y,z To solve the problem of high-dimensional data processing and feature extraction, a 3×3×3 convolution kernel is used to perform multiple convolution operations to gradually extract higher-level features and reduce the data dimension. After each convolution operation, the maximum pooling method is used to reduce the dimension of the feature map, thereby effectively solving the problem of feature redundancy and high computational complexity. After convolution and pooling, the output feature map H m is flattened into a one-dimensional vector V. Finally, the flattened vector V is connected to the fully connected layer for nonlinear transformation and feature integration, and finally outputs a one-dimensional feature representation F final Through this series of steps, the complex interaction between modal features is successfully captured. This interaction not only provides the model with richer and more diverse information, but also significantly improves the model's ability to handle complex tasks, enabling it to more accurately understand and distinguish information from different modalities. The size of the output feature map of the convolutional layer and the pooling layer can be calculated using the following formula:

[0125]

[0126]

[0127]

[0128] F final=w·V+b;

[0129] Where W is the width of the three-dimensional tensor F, K is the size of the convolution kernel, P is the padding size, S is the stride, and P p is the pooling window size, P' is the pooling padding size, S p is the pooling size, H is the feature vector size, w is the weight matrix, and b is the bias term; Output Size represents the output features, Pooled Size represents the features after pooling, and F final Represents the final features.

[0130] Finally, the original 3D tensor of the i-th sample is converted into a 1D tensor using a flattening operation:

[0131]

[0132] Step S5, sentiment prediction: The decoupled interaction features are concatenated with the feature features and then input, sentiment labels are predicted through the multi-layer perceptron (MLP), and the prediction accuracy is optimized through the mean absolute error (MAE) loss function.

[0133] Step S4 specifically includes:

[0134] The decoupled eigenvector F task ∈{K m ,F con} Input the multimodal sentiment analysis task module and use the multi-layer perceptron (MLP) architecture to predict the sentiment label. The output of the task module can be expressed as:

[0135]

[0136] To optimize the performance of sentiment analysis, the prediction error loss term L generated by the task module is task Calculated by Mean Absolute Error (MAE), this loss function helps the model optimize its performance in sentiment analysis tasks by penalizing the absolute difference between the prediction and the true label. The specific formula is as follows:

[0137]

[0138] In the formula, is the predicted sentiment label, y i is the true label.

[0139] In addition, the accumulation of all loss terms constitutes the global loss function L for model training total , which not only integrates the goals of each task, but also provides a comprehensive optimization direction for the model to ensure its best performance in multimodal sentiment analysis. Its calculation formula is as follows:

[0140] L total =αL spc +βL rec +L task ;

[0141] Among them: α, β are hyperparameters used to control the weight of each loss term in the overall loss function.

[0142] The specific operations in the model training phase are as follows:

[0143] ① Divide the multimodal emotion intensity dataset A obtained in step S1 into a training set T, a validation set V, and a test set I according to the ratio of 8:1:1;

[0144] ② The training set T extracts modal features for each sample, maps the features of different modalities to a unified dimensional space through a fully connected layer to ensure the consistency of the features, and obtains the modal feature representation K m ∈{K a ,K v ,K t}, where they represent K a , K v , K t Features of audio, image and text modalities

[0145] ③By splicing and expanding the features of different modalities, the Cartesian product and Einstein summation convention (einsum) are used to calculate the interaction relationship between modalities to obtain higher-dimensional interaction features. This process not only retains the unique information of each modality, but also can learn the pairwise and trimodal interaction features between modalities. After further convolution and pooling processing, the complex interactive relationship between modality-specific features and shared features is captured.

[0146] ④By optimizing the interactive features and modal feature representation K m ∈{K a ,K v ,K t}, further improving the expressiveness of the features of each modality and improving the robustness of the model.

[0147] ⑤ By minimizing the global loss function with the formula, the validation set V is used during the training phase to verify the model training effect and optimize the remaining modules of the model. When the model M achieves good results on the validation set V and the accuracy does not improve with the training process, the training is stopped. Finally, the training is completed to obtain the model M'.

[0148] The specific steps of the model testing phase are as follows:

[0149] ① Input the multimodal sentiment intensity dataset test set I into the trained model M' to obtain the sentiment prediction results of the model for the test set samples.

[0150] ② Compare the test results with the preset performance indicators to determine whether the model has achieved the expected results. If necessary, consider adjusting or retraining the model.

[0151] ③After the testing phase is completed, the final performance report of the model will be compiled and visualized to facilitate subsequent research and application.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A multimodal sentiment analysis method based on tensor dynamic interaction and decoupling, characterized by: The following steps are involved: Step S1, feature extraction: collect multimodal sentiment intensity datasets, input the collected raw modal data into different encoders for feature extraction to generate multimodal latent feature representations, use BERT encoder to process text modality, and use recurrent neural network (RNN) and long short-term memory (LSTM) to process audio and image modalities to capture temporal dependencies and generate feature vectors for each modality, and use a fully connected layer to map audio and visual features to a unified dimensional space to simplify the complexity of feature processing; Step S2, feature interaction: After feature extraction, add a bias term to each modal feature matrix, perform linear transformation on the modal features through the parameter matrix obtained through training, map the three modal features to three dimensions by extending and splicing, and use the Cartesian product and Einstein summation convention to capture the interactive relationship of these three dimensions to generate interactive features; Step S3, feature decoupling: each modal feature is processed by the Transformer encoder to capture the information of each modality to obtain the unique features, the contrast loss function is used to maximize the difference between the modal features and the interaction features, the reconstruction loss function is used to decouple the unique features and the interaction features, and the information is recovered from the original features; Step S4, convolution and pooling: the calculated interaction features are subjected to convolution operations to gradually extract higher-dimensional features, and the pooling method is used to reduce redundant information and flatten the feature map into a one-dimensional feature vector; Step S5, sentiment prediction: The decoupled interaction features are concatenated with the feature features and then input, sentiment labels are predicted through the multi-layer perceptron (MLP), and the prediction accuracy is optimized through the mean absolute error (MAE) loss function.

2. The multimodal sentiment analysis method based on tensor dynamic interaction and decoupling according to claim 1 is characterized in that: In step S1, a multimodal sentiment intensity dataset A is collected, and each data sample has a corresponding real number score y∈[-3.0,3.0], representing the sentiment intensity: strongly negative, negative, weakly negative, neutral, weakly positive, positive, and strongly positive; The original modal sequence X={X1,X2,…,X L }={X t ,X a ,X v } are input into different encoders for feature extraction, text modality X t The BERT encoder is used for processing. BERT dynamically assigns weights to each word, captures the complex relationship between words, and generates the final text feature representation vector. Audio Mode X a and image modality X v The feature extraction uses a recurrent neural network (RNN), which transmits information from the previous moment to the current moment through its cyclic structure, captures the temporal dependencies in the sequence, processes the dynamic changes in the audio signal and the motion trajectory of objects in the image sequence, and generates an audio feature representation vector. and image feature representation vector The memory unit of a single-layer unidirectional long short-term memory network LSTM is used to store and extract key information, and the degree of attention to the input information is dynamically adjusted through a gating mechanism to capture time dependency. Use fully connected layers to map visual and audio features to a unified dimension The formula is as follows: K t =FC(X t ); Among them, m∈{a,v,t}, a,v,t represent audio, image and text respectively. and Denote the parameters of the LSTM model related to the visual modality and the audio modality, respectively, K m ∈R n×d It represents the projection result of the modal feature representation after LSTM encoding. The superscript n in R represents that the feature dimension size is n, d represents the unified feature dimension, and K represents the feature obtained after LSTM encoding.

3. The multimodal sentiment analysis method based on tensor dynamic interaction and decoupling according to claim 2 is characterized in that: In step S2, the features of different modalities are concatenated to fuse the information from each pair of modalities together to obtain a fused feature representation: The features of each modality are merged into the same dimensional space. Each concatenated feature contains information from two modalities. After each concatenation, the feature matrix F x , F y , F z Add a column of all 1s at the end of , expanding the feature matrix of each dimension to 129 dimensions. The formula is as follows: 1 of them n A column vector of all 1s, with dimension n×1; Define the learnable matrix for training and Applied to the expanded dimension matrix F′ x , F′ y and F′ z Perform a linear transformation on it to generate a new feature representation, thereby extracting higher-dimensional features. The formula is as follows: New interaction features are generated by calculating the Cartesian product between dimensions. First, the feature matrix F of the two dimensions is transformed into lx 、F ly Expanded to a shape suitable for the outer product, so that they interact element-wise, the formula is as follows: The Cartesian product is calculated using the Einstein summation convention einsum, without summing repeated indices: fusion x,y =einsum('nxt,nty→nxy',F′ lx ,F′ ly ); The dimension F l ' z Expanded to fit the Cartesian product result fusion a,v To interact with the shapes, use the Einstein summation convention einsum to fuse the Cartesian product results a,v and dimension F l ' z To fuse the features of three dimensions, the formula is as follows: fusion x,y,z =einsum('nxt,nty→nxy',fusion x,y ,F′ lz ); 4. The multimodal sentiment analysis method based on tensor dynamic interaction and decoupling according to claim 3 is characterized by: In step S3, each modal feature is input into the Transformer encoder, and the contextual information of the internal features of each modality is captured through the self-attention mechanism. The generated latent feature representation K m ∈{K a ,K v ,K t }; Use contrastive loss function to maximize the latent feature representation K m ∈{K a ,K v ,K t } and the distance between the fusion features, prompting each modality to express its information independently. The formula is as follows: Among them, L spc represents the unique feature loss function, s is the distance threshold; Based on the Euclidean distance, the reconstruction information loss function L is defined rec , by minimizing the distance between similar samples, the alignment and learning of reconstruction information are achieved: Where: y ij is the label generated based on the similarity measure, d ij Indicates the similarity between samples, and k is the preset threshold.

5. The multimodal sentiment analysis method based on tensor dynamic interaction and decoupling according to claim 4 is characterized in that: In step S4, for the three-dimensional tensor fusion x,y,z To solve the problem of high-dimensional data processing and feature extraction, a 3×3×3 convolution kernel is used to perform multiple convolution operations to gradually extract higher-level features and reduce the data dimension. After each convolution operation, the maximum pooling method is used to reduce the dimension of the feature map. After convolution and pooling, the output feature map H m Flatten it into a one-dimensional vector V, connect the flattened vector V to the fully connected layer, perform nonlinear transformation and feature integration, and output a one-dimensional feature representation F final ; The size of the output feature map of the convolution layer and the pooling layer is calculated by the following formula: F final =w·V+b; Where W is the width of the three-dimensional tensor F, K is the size of the convolution kernel, P is the padding size, S is the stride, and P p is the pooling window size, P' is the pooling padding size, S p is the pooling size, H is the feature vector size, w is the weight matrix, and b is the bias term; Output Size represents the output feature, Pooled Size represents the pooled feature, and F final Represents the final characteristics; Use the flatten operation to convert the original 3D tensor of the i-th sample into a 1D tensor:

6. The multimodal sentiment analysis method based on tensor dynamic interaction and decoupling according to claim 5 is characterized in that: In step S5, the decoupled feature vector F task ∈{K m ,F con } Input the multimodal sentiment analysis task module, and use the multi-layer perceptron MLP architecture to predict the sentiment label. The output of the task module is expressed as: The prediction error loss term L generated by the task module task Calculated by the mean absolute error MAE, this loss function penalizes the absolute difference between the prediction and the true label. The formula is as follows: In the formula, is the predicted sentiment label, y i is the true label; The accumulation of all loss terms constitutes the global loss function L for model training. total , the calculation formula is as follows: L total =αL spc +βL rec +L task ; Among them, α and β are hyperparameters used to control the weight of each loss term in the overall loss function.

Images