Multi-modal sentiment classification method and system based on corpus enhancement and cross-modal generation

Abstract

The invention relates to the technical field of artificial intelligence and emotion recognition, and discloses a multi-modal emotion classification method and system based on corpus enhancement and cross-modal generation, and the method comprises the following steps: 1, carrying out the preprocessing and feature extraction of multi-modal data; wherein the multi-modal data comprises a video modal, a text modal and an audio modal; 2, projecting each modal feature to a low-dimensional shared space; wherein video, text and audio modal features are respectively mapped to a unified low-dimensional space through a full connection layer so as to eliminate dimension differences among modals, and an ReLU activation function is adopted to enhance expression ability; and step 3, performing feature enhancement based on a text library on each input modal feature to output the enhanced modal features. The scheme supports single-mode, dual-mode and three-mode input, adapts to diversity of data missing in practical application, and is high in model recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and emotion recognition technology, and in particular to a multimodal emotion system based on corpus enhancement and cross-modal generation. Background Art

[0002] With the prevalence of social media and smart devices, users are increasingly demanding the ability to express their emotions through multimodal data, such as video, audio, and text. Compared to traditional single-modal sentiment analysis (e.g., relying solely on visual expressions), multimodal sentiment analysis improves the accuracy of emotion recognition by integrating information from different modalities (e.g., text, audio, and video). This has become a key technology in fields such as human-computer interaction and intelligent customer service. However, obtaining complete multimodal data in the real world often presents various challenges. For example, when the camera is obstructed, the user's image cannot be captured, and when sudden noise occurs, the user's speech cannot be obtained. The lack of multimodal data can affect the accuracy of sentiment analysis.

[0003] To address the above issues, generative learning and joint learning methods are currently commonly used to address the problem of missing modalities. Generative learning aims to handle missing modalities by generating new data that matches the observed distribution of existing modalities. Although generative learning methods can help construct complete multimodal information, the generated data may have significant characteristics different from the real modal data, which may affect the accuracy of sentiment analysis results to a certain extent. Joint learning methods aim to learn the potential features of missing modalities directly from existing modalities. This method can integrate information from different modalities, but missing modalities can easily lead to a loss of balance between modalities during feature fusion, and cannot accurately reflect the information of the remaining modalities. Summary of the Invention

[0004] In order to solve the technical problems raised in the background technology, the present invention provides a multimodal emotion system based on corpus enhancement and cross-modal generation.

[0005] The present invention adopts the following technical solution to achieve: a multimodal sentiment classification method based on corpus enhancement and cross-modal generation, comprising the following steps:

[0006] Step 1: Preprocess and extract features from multimodal data; wherein the multimodal data includes video modality, text modality, and audio modality;

[0007] Step 2: Project each modal feature into a low-dimensional shared space. The fully connected layer is used to map the video, text, and audio modal features into a unified low-dimensional space to eliminate the dimensionality differences between the modalities, and the ReLU activation function is used to enhance the expressive power.

[0008] Step 3: Perform feature enhancement on each input modal feature based on the text library to output enhanced modal features; wherein:

[0009] Step 3A: pre-build a text library containing low-dimensional text feature tensors and labels;

[0010] Step 3B: Calculate the cosine similarity matrix between the input features and the text library features, and select the top-K related features based on the learnable temperature coefficient;

[0011] Step 3C: The retrieved features are fused with the input features through a multi-head attention mechanism, and enhanced features are generated through residual connections and layer normalization.

[0012] Step 4: Generate missing modal features across modalities and output multimodal data of non-missing modalities; where:

[0013] Step 4A: Add dynamic position coding and modality type coding to the input sequence to distinguish timing information from modality attributes;

[0014] Step 4B: Restrict the direction of information flow through a masking mechanism and generate pseudo features of the missing modality in an autoregressive manner;

[0015] Step 4C: Constrain the consistency between the pseudo features and the real features generated by the missing mode using the mean square error loss;

[0016] Step 5: Multimodal dynamic fusion and classification. First, the fused multimodal features are output and sentiment classification is performed using the Transformer classifier.

[0017] Step 5A: Use temporal convolution to align the features of each modality and model the temporal dependency through the gated recurrent unit (GRU);

[0018] Step 5B: Introduce a dynamic gating mechanism to screen key features and combine it with a multi-head attention mechanism to fuse multimodal information;

[0019] Step 5C: Aggregate cross-modal features through the global information carrier MMseq to generate a joint representation;

[0020] Step 5D: Map the joint representation to the emotion category space, select L1 loss or cross entropy loss for optimization according to the task type, and then use the softmax function to convert the output into a probability distribution to represent the possibility of different emotion labels.

[0021] Furthermore, the step 1 includes the following sub-steps:

[0022] Step 1A. Use OpenFace 2.0 to extract facial motion, head posture, and timestamp features from the video modality data to generate a 709-dimensional feature vector.

[0023] Step 1B. Extract 768-dimensional semantic features from text modal data using the pre-trained BERT model;

[0024] Step 1C. Extract 33-dimensional audio features including MFCC and CQT from the audio modal data using the Librosa tool;

[0025] Furthermore, in step 3A, the construction of the text library includes the following operations:

[0026] Filter text samples from the CMU-MOSI and MELD datasets;

[0027] Compress the original high-dimensional text features into a low-dimensional tensor through a linear projection layer;

[0028] Store low-dimensional feature tensors, data labels, projection layer weights, and dimension information as structured files;

[0029] The input text features are aggregated into feature vectors according to the mean, maximum or sum mode, and L2 normalized.

[0030] Furthermore, in step 4, when generating missing modal features, the learnable multimulti tag is initialized as the generation starting point, and the inter-modal attention range is controlled by the mask matrix, limiting the generation module to focus only on the historical information of the existing modalities.

[0031] Furthermore, in step 5, the dynamic gating mechanism generates weights through a learnable linear layer and a Sigmoid function to dynamically enhance important features and suppress redundant noise.

[0032] Furthermore, the MMseq sequence achieves the following functions through the cross-modal attention mechanism and masking rules:

[0033] Actively obtain unimodal feature information as a query matrix;

[0034] Provides global context to a single modality as a key-value matrix;

[0035] Direct interaction between different modalities is prohibited, and only indirect information fusion through MMseq is allowed.

[0036] Furthermore, the method supports single-modal, dual-modal and tri-modal input combinations.

[0037] The present invention also proposes a multimodal sentiment system based on corpus enhancement and cross-modal generation to implement the above multimodal sentiment classification method, which includes:

[0038] The text database construction module is used to filter text modality samples from the multimodal dataset and project the original high-dimensional text features into low-dimensional text features through text projection to build a low-dimensional text feature library;

[0039] A text enhancement module is configured to retrieve similar samples from the text database based on the available modalities of the current input data, enhance the features of the currently available modalities using the sample features in the text database, and output an enhanced multimodal feature tensor;

[0040] The multimodal generation module, when the input data has missing modalities, uses the enhanced representation obtained by the text enhancement module and the mask-based attention mechanism to generate pseudo-modalities and output multimodal data without missing modalities;

[0041] The multimodal dynamic fusion module based on gated filtering introduces a dynamic gating mechanism after the attention calculation of the Transformer encoder layer. That is, it uses learnable linear layer weights to learn the feature mapping relationship from the input features, and converts the output of the linear layer into gating weights through the sigmoid function. These weights are used to dynamically filter key features, control the fusion strength of cross-modal information, enhance important features, achieve efficient feature screening, and output fused multimodal features.

[0042] The sentiment classification module performs sentiment prediction through the Softmax classifier based on the fused multimodal features, and dynamically selects the corresponding loss function according to the task type and label format of the multimodal dataset to adapt to different task requirements.

[0043] According to one aspect of the present invention, a computer-readable storage medium is provided, storing a computer program, which, when executed by a processor, implements the multimodal sentiment analysis method according to any one of claims 1 to 8.

[0044] According to one aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein when the processor executes the program, the multimodal sentiment analysis method according to any one of claims 1 to 8 is implemented.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. Improve the accuracy of multimodal sentiment classification

[0047] Corpus augmentation technology (retrieving similar features from a text library and fusing them with input features) strengthens key semantic information and significantly improves feature expression capabilities. In the language modality (L) task on the CMU-MOSI dataset, the accuracy reached 81.40%, outperforming competing models (such as CTFN's 80.79%).

[0048] The dynamic gating mechanism and cross-modal attention optimize the multimodal fusion process, achieving an accuracy of up to 82.77% in dual-modal combination tasks (such as L+A), surpassing existing methods (such as 82.16% of CTFN).

[0049] 2. Enhance robustness in modality-missing scenarios

[0050] Through a cross-modal generation module (based on a masking mechanism and autoregressive generation), high-quality pseudo-modal features (such as audio or video) are generated when some modalities are missing from the input. Mean squared error loss is used to constrain the consistency of the generated features with the real features. Experiments show that this method can still maintain high performance (CMU-MOSI single-modal L accuracy of 81.40%) even with single-modal input (such as text only).

[0051] 3. Dynamic Fusion and Noise Suppression

[0052] This solution introduces a global information carrier, MMseq, to indirectly fuse multimodal information through cross-modal attention, prohibiting direct interaction between modalities and preventing noise interference. A dynamic gating mechanism dynamically selects key features using learnable weights and a sigmoid function, suppressing redundant information and improving the model's adaptability to complex scenarios (for example, the bimodal L+A combination achieved an accuracy of 59.62% in the MELD dataset emotion classification task).

[0053] 4. Flexible support for multi-modal input combinations

[0054] This solution supports unimodal, bimodal, and trimodal inputs (such as L / A / V, L+A, A+V, etc.), adapting to the diverse data missingness in real-world applications. For example, in the CMU-MOSI Vision (V) unimodal task, it achieved an accuracy of 61.13%, significantly outperforming AMD (55.20%).

[0055] 5. Alleviate data imbalance problem

[0056] This solution uses the text library enhancement module to utilize the rich semantic information of external corpora (such as the CMU-MOSI and MELD datasets) to enhance the feature expression of small samples or low-quality modalities and alleviate the performance degradation caused by uneven data distribution.

[0057] 6. Efficient cross-modal alignment and generation

[0058] This solution uses temporal convolution alignment and GRU modeling of temporal dependencies to solve the problem of inconsistent multimodal sequence lengths; through dynamic position encoding and modality type encoding, it distinguishes temporal and modal attributes and improves the accuracy of the generation module (for example, the mean square error loss of generated pseudo features is significantly reduced).

[0059] 7. Multi-tasking adaptability

[0060] This scheme dynamically selects the loss function according to the task type (L1 loss for regression tasks and cross entropy for classification tasks), and performs competitively in both the sentiment and emotion dual tasks of the MELD dataset (sentiment classification accuracy of 66.82% and emotion classification accuracy of 59.62%). BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a flowchart of a multimodal sentiment analysis method based on corpus enhancement and cross-modal generation proposed by the present invention;

[0062] Figure 2 This is a principle block diagram of a multimodal sentiment analysis method based on corpus enhancement and cross-modal generation proposed in the present invention;

[0063] Figure 3 This is a bar chart comparing the multimodal sentiment analysis method proposed in this invention with various SOTA models on the CMU-MOSI dataset. DETAILED DESCRIPTION

[0064] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0065] Example 1:

[0066] Reference Figure 1-Figure 3 This embodiment takes the CMU-MOSI and MELD datasets as examples to introduce in detail a multimodal sentiment analysis method based on corpus enhancement and cross-modal generation.

[0067] CMU-MOSI: The CMU-MOSI (Carnegie Mellon University MultimodalOpinion Sentiment and Intensity) dataset is derived from 93 movie clips on YouTube and contains a total of 2,199 monologue images. In the CMU-MOSI dataset, each sample is assigned a sentiment score in the range of [-3, 3] to quantify the intensity of the sentiment.

[0068] The MELD (Multimodal EmotionLines Dataset) dataset is based on dialogue clips from the TV series "Friends." It covers over 1,400 conversations and more than 13,000 utterances. Each conversation contains multiple speakers, which is closer to real-world interaction scenarios. In the MELD dataset, each utterance is annotated with seven fine-grained emotions (anger, disgust, fear, joy, neutrality, sadness, and surprise).

[0069] Read and clean multimodal datasets, mainly using the CMU-MOSI and MELD datasets, filter out high-quality text, and store it as structured files;

[0070] Use text projection to project the original high-dimensional text features into a low-dimensional space to build a text library;

[0071] The text library contains low-dimensional text feature tensors, NumPy format backups, data labels, linear projection layer weight parameters, and dimension information.

[0072] Preprocess the input data, extract features and convert them into low-dimensional representations. The specific steps are as follows;

[0073] The OpenFace2.0 tool is used to extract features of the video modality from the face image, including timestamp, confidence, recognition success mark, eye movement, head posture, and facial movement, with a dimension of 709.

[0074] The features of the text modality are learned using the pre-trained BERT model with a dimension of 768;

[0075] The Librosa tool is used to extract audio modality features, including zero-crossing rate, Mel-frequency cepstral coefficient (MFCC) and constant Q transform (CQT) features. These features are concatenated into 33-dimensional audio features.

[0076] In order to further adapt the feature dimensions of each input modality to the text library and eliminate the dimensionality differences between modalities, the original multimodal features use a fully connected layer to project the features of each input modality and the features of the text library into a low-dimensional shared space, eliminating the dimensionality differences between modalities and simplifying the calculation of cross-modal interactions. The ReLU activation function is used to enhance the feature expression ability, and Dropout regularization is used to prevent overfitting and enhance the generalization performance of the model.

[0077] Then, a multimodal similarity calculation and feature enhancement system was implemented to strengthen key semantic information.

[0078] During the initialization phase, the system loads the pre-stored text feature library data, converts the text features and labels into PyTorch tensors, and distributes them to the computing device. The original text features are compressed into feature vectors according to the configured aggregation mode (including mean, maximum, and sum).

[0079] The implementation is as follows: Given a 3D tensor ([batch, seq_len, features]) as input, the specified aggregation operation is performed along the sequence dimension to obtain a 2D vector ([batch, features]), which is then L2-normalized. The learnable temperature coefficient parameter (initial value 0.07) and the multi-head attention mechanism module are also initialized.

[0080] When calculating similarity, the input tensor is first dynamically aggregated into feature vectors according to the modality type. The input features and the text library features are L2 normalized to eliminate the scale difference of the features. The input features and the text library vector are calculated through matrix multiplication to obtain the cosine similarity matrix. The similarity is scaled by the learnable temperature coefficient. The formula is expressed as: Similarity = (q*k^T) / τ to obtain the similarity score between the input features and each sample in the text library. Where q is the query vector, k is the text library feature vector, and τ is the temperature coefficient. Finally, the similarity matrix is sorted in descending order by row, and the similarity score and corresponding index are returned. The similarity calculation formula is as follows:

[0081]

[0082] Among them, q is the query vector from different modalities (language L, audio A, vision V), k i is the key vector, which comes from the text library feature vector and has been L2 normalized.

[0083] Feature enhancement: Retrieve the top_k relevant feature vectors based on the similarity sorting results, perform index out-of-bounds check, calculate the correlation weights between the retrieved vectors and the input features through a multi-head attention mechanism, and generate a context enhancement vector. The enhanced vector is superimposed on the original input through a residual connection, and after layer normalization, it is restored to the original dimension of the input tensor, and finally outputs the renormalized enhanced features. The feature enhancement formula is as follows:

[0084]

[0085] in is the retrieved TOP-K related features, w k It is an attention weight based on similarity, which is used to control the contribution of each retrieved feature vector Xretrieved(k) to the final enhanced feature Xenhanced.

[0086] The enhanced features are input into the multimodal generation module to generate pseudo-modal features. The specific steps are as follows:

[0087] Perform position encoding, dynamically generate position encoding according to the length of the input sequence, capture the sequence's temporal information, solve the defect of Transformer's lack of position perception, enhance the model's ability to perceive temporal relationships and modal characteristics, and improve generation accuracy.

[0088] Perform modal type encoding to distinguish different modal features and avoid cross-modal information confusion that affects model performance.

[0089] Initialize the [multi] tag. Multi is initialized through Xavier and is a learnable vector. It serves as the starting tag for generating the missing modality and aggregates the contextual information of the existing modality. During training, by simulating the autoregressive generation process, the model is forced to learn the ability to predict the next moment based on historical information. Specifically, through the masking mechanism, the model can only focus on the current and left tags during training, avoiding the leakage of future information, promoting cross-modal information flow, and enhancing model performance.

[0090] The process of integrating cross-modal information through the attention mechanism and generating the missing modality is as follows:

[0091] The splicing order is dynamically adjusted according to the input modality combination (such as L / A / V / LA, etc.) to ensure that the encoder learns the dependency between modalities, and introduces a mask matrix in the attention calculation to limit the generated modality to only focus on itself and existing modal information, and prohibit attention to future frames. Specifically, the cross-modal attention rule is to use the existing modality (such as L) as the encoder and the generated modality (such as A) as the decoder, and control the direction of information flow through the mask. At this time, the text sequence is fully interactive through the encoder self-attention, and the audio sequence is through the decoder self-attention, using the lower triangular mask to only focus on historical frames. The audio can focus on the text information through cross attention, while the text cannot focus on the audio information in reverse.

[0092] The design of the attention mask matrix adds a mask specific to the modality generation direction to the self-attention matrix. When implementing the masking mechanism, the value -inf is assigned to the part to be masked and 0 is assigned to the part to be unmasked to control the information transfer. The calculation method is as follows:

[0093]

[0094] Among them, MASK G is a global mask matrix used to control the attention weights between different positions.

[0095] Through Transformer encoding, we achieve joint modeling of local dependencies within modalities and global associations between modalities. We use a multi-head attention mechanism to capture features in parallel, with different heads focusing on different semantic relationships between modalities. For example, head 1 focuses on the emotional association between text and audio, and head 2 focuses on the temporal alignment of text and video. We use a residual structure to alleviate the vanishing gradient problem in deep networks, layer normalization stabilizes the training process, and layer-by-layer regularization is used to prevent overfitting.

[0096] After obtaining the output, the generated part is cut according to the preset modal length to generate the corresponding missing data, and then projected back to the original dimension of the target modality through the fully connected mapping. The point-by-point difference in the dimension of the generated feature sequence and the real feature sequence is compared by the mean square error to ensure that the generated pseudo-modal feature is as close as possible to the real modal feature. The loss function formula is as follows:

[0097]

[0098] Where T v is the sequence length (such as the number of video frames), x V is the original visual feature, ^x V is the reconstructed visual modality feature.

[0099] After completing the above steps, that is, after generating missing modal data, enter the prediction module of the model.

[0100] The complete information of each modality is passed through its respective temporal convolutional layer, and the features of different modalities are mapped to the same dimension, reducing redundant noise and retaining key local features.

[0101] Xavier uniform distribution is used to initialize MMseq. MMseq serves as a carrier of global information and dynamically fuses multimodal features.

[0102] Use gated recurrent units to model language feature sequences and capture contextual dependencies in language. For example, in sentiment analysis, the emotional tendency of a sentence often depends on the semantic information of the context. This allows capturing temporal features in audio, such as emotional changes in speech, and modeling dynamic features in video feature sequences, such as temporal information such as actions and expressions.

[0103] An absolute position index is generated based on the length of each modal sequence to add spatiotemporal position and modal type information to each modal feature. Position embedding and modal type encoding utilize the independence of the embedding vector in high-dimensional space and the learning ability of the neural network. They can be combined by direct addition, which simplifies calculation and is learnable, allowing the model to distinguish between the features of "different positions in the same modality" and "same position in different modalities".

[0104] At the same time, learnable positional encoding and modality encoding are added to each time step of the MMseq sequence, enabling MMseq to adaptively capture cross-modal global features, and then splice them with the embedded sequences of other modalities in the time dimension, aggregating features of different modalities such as language, audio, and video, and generating a robust joint representation for the final sentiment analysis task. It should be noted that MMseq is not a fixed predefined symbol, and the parameters of MMseq are continuously updated during the training process.

[0105] In multimodal tasks, features of different modalities usually have problems such as inconsistent sequence lengths and unbalanced information contributions. Therefore, we propose to define a multimodal dynamic fusion module based on gated screening. By introducing a dynamic gating mechanism after the attention calculation of the Transformer encoder layer, that is, using learnable linear layer weights to learn the mapping relationship of features from input features, and converting the output of the linear layer into gating weights through the sigmoid function, we use this weight to dynamically screen key features and enhance important features to achieve efficient feature screening.

[0106] The operation of aggregating multimodal information is as follows: MMseq receives information from all modalities through a masking mechanism and an attention mechanism. When the MMseq sequence extracts information from a unimodal sequence, MMseq actively obtains information as a Q matrix, and the unimodal sequence is positioned and provided as a K and V matrix; when a unimodal sequence obtains multimodal context from MMseq, MMseq serves as a key node in the attention mechanism and provides global context as a key-value. The attention mask rules at this time are as follows: MMseq is allowed to pay attention to all time steps of all modalities, and a unimodal is allowed to pay attention to all time steps of MMseq. Full self-attention is allowed within a unimodal state, and direct interaction between modalities is prohibited (for example, the language modality cannot directly pay attention to audio or video). The following is a representation of MMseq paying attention to all time steps of all modalities and a unimodal state paying attention to all time steps of MMseq and its internal full self-attention:

[0107] Y M =CA L→M (Z M , Z L )+CA A→M (Z M , Z A )+CA V→M (Z M , Z′ V )

[0108] Y L =CA M→L (Z L , Z M )+SA(Z L ), YA , Y′ V Similarly

[0109] where Z M , Z L , Z A , Z V Represents the feature embedding of the target modality M and other modalities (language L, audio A, video V), CA x→y (Y, x) represents the cross-modal attention mechanism from modality X to modality Y; SA(Z L ) represents the self-attention of the language modality.

[0110] The formula for MMseq to aggregate global information and intra-modal self-attention through cross-modal attention is as follows:

[0111]

[0112] where Y M is the output after multimodal aggregation, Q M and K are the query and key matrices, MASK U is the mask matrix, V is the value matrix, Y L is the output of the language modality, Q L and K L is the query and key matrix of the language modality, V L is the value matrix of the language mode, is a scaling factor used to prevent the attention scores from being too large.

[0113] The multi-head attention mechanism is used to split the embedding dimension into H independent heads. Each head independently calculates the attention weight based on the mask rule. The output of each head is then concatenated along the feature dimension and mapped back to the original dimension through a linear layer.

[0114] After the multi-head attention output, the input features are added to the attention results (residual connection) to retain their original features and prevent the gradient from disappearing; the results after the residual connection are normalized to accelerate convergence; then nonlinear changes are made through two linear layers and the ReLU activation function to enhance the expressive power of the model.

[0115] After multi-layer iterative processing, the unified sequence composed of MMseq and language modality, audio modality, and video modality gradually integrates the global context to generate a unified joint representation. Finally, the features corresponding to MMseq are extracted from the final layer and mapped to the emotion category space through the fully connected layer.

[0116] Full classification connection formula:

[0117]

[0118] Among them, hfinal is the final feature vector after multimodal fusion, W c is the classification weight matrix, and C is the number of categories.

[0119] The corresponding loss function is dynamically selected based on the task type and label format of the dataset. For regression tasks (such as CMU-MOSI), the L1 loss function is used; for classification tasks (such as MELD), the cross entropy loss function is used. The loss function is calculated as follows:

[0120]

[0121] Among them, L reg represents the regression loss, y i Represents the true value, ^y i represents the predicted value; L cls represents the classification loss, y i,c represents the true label, p i,c represents the predicted probability.

[0122] To better illustrate the technical effects of the present invention, a specific example is used to experimentally verify the present invention. A large number of experiments are conducted on the CMU-MOSI and MELD datasets, and the experimental results prove the effectiveness of the proposed method.

[0123] In summary, this embodiment provides a multimodal sentiment analysis method based on corpus enhancement and cross-modal generation, which obtains text data by screening data; obtains a low-dimensional representation of text library sample information by linearly projecting the text data to construct a text database; linearly projects the input data to obtain a low-dimensional representation of each modal information; the input modal data passes through a multimodal similarity calculation and input enhancement module based on a dynamic gating mechanism, and the similarity weights of different modalities are fused through the dynamic gating mechanism to improve the accuracy of the similarity calculation, select the two text features with the highest similarity, generate a context vector through the attention mechanism, fuse the input features and the context vector to generate an enhanced feature representation, and judge whether the modal information is complete based on the existing input data: if the existing modal information is complete, skip the generation module; if the existing modality is missing, input the feature representation obtained based on the enhancement into the multimodal generation module to obtain the sequence information of the generated complete modality, and then intercept the information of the missing modality;

[0124] This scheme calculates the loss through mean square error and optimizes the generated missing modal features; the final data information is temporally convolved to retain key local features, and then local modeling is performed through gated recurrent units to capture dependencies. Then, the carrier MMseq of global information aggregation is initialized, and the GRU output of each modality is concatenated with the global fusion marker MMseq to form a joint input sequence. The learnable modality type code and position code are embedded in each position in the sequence to distinguish the modal source and temporal relationship, so that the model can distinguish the features of "same modality at different positions" and "different modalities at the same position". , then interact through multiple layers of cross-modal Transformer encoders, with each layer performing cross-modal attention calculation and dynamic gating screening in turn - using the current modality feature as the query and other modality features as the key / value to generate an attention weight matrix, concatenating the original features with the attention output, and generating dynamic gating values through the learnable gating weight matrix and the Sigmoid function, performing element-by-element weighted screening on the cross-modal information, and finally outputting enhanced features through residual connections and layer normalization; after multiple layers of interaction, the global fusion state of MMseq is extracted as a sentiment representation. Finally, a loss function is used based on the task type and label format of the dataset to compare the predicted sentiment label with the real sentiment label to optimize the prediction result; a sentiment prediction model is established based on the fusion features; the embodiment of the present invention can improve the quality of the generated modal information and help improve the performance of sentiment analysis.

[0125] Benchmark datasets. In the experiments, two well-known multimodal sentiment analysis benchmark datasets were used: CMU-MOSI and MELD.

[0126] CMU-MOSI: The CMU-MOSI (Carnegie Mellon University MultimodalOpinion Sentiment and Intensity) dataset is derived from 93 movie clips on YouTube and contains a total of 2,199 monologue images. In the CMU-MOSI dataset, each sample is assigned a sentiment score in the range of [-3, 3] to quantify the intensity of the sentiment.

[0127] The MELD (Multimodal EmotionLines Dataset) dataset is based on dialogue clips from the TV series "Friends." It covers over 1,400 conversations and more than 13,000 utterances. Each conversation contains multiple speakers, which is closer to real-world interaction scenarios. In the MELD dataset, each utterance is annotated with seven fine-grained emotions (anger, disgust, fear, joy, neutrality, sadness, and surprise).

[0128] For dataset preprocessing, in the image modality, the OpenFace 2.0 toolbox was used to extract features from facial images in the image; these features include timestamps, confidence levels, successful recognition markers, eye movements, head poses, and facial movements. Each image sample ultimately forms a high-dimensional feature vector with 709 features. For the text modality, feature extraction was performed using the pre-trained BERT model. The model outputs text features with a dimension of 768, which effectively captures the semantic information in the text. For audio modality feature extraction, the Librosa library was used for processing, converting the audio signal to mono and resampling it to 16,000 Hz. The audio frame was processed based on a 512-sample partition, primarily extracting features such as zero-crossing rate, Mel-frequency cepstral coefficients (MFCCs), and constant Q transform (CQT). The resulting audio feature vector had a dimension of 33.

[0129] Experimental parameter settings: The model framework used in the experiment is based on PyTorch version 1.10.0, and the programming language is Python 3.9.21. The learning rate is set to 0.001, and the batch size is adjusted to 128. The model architecture uses an 8-head attention mechanism, the embedding dimension is set to 32, and the dropout rate of the ReLU activation function and residual block is 0.1. The Adam optimization algorithm is used during training, and gradient clipping (threshold of 0.8) is applied to prevent gradient explosion. The experimental epoch is set to 100, and the learning rate is decayed at the 20th epoch. In view of the characteristics of multimodal tasks, the input modality is selected as audio and visual (AV) alignment mode. Hyperparameter tuning uses the TPE sampler to conduct 10 trials to dynamically optimize key parameters such as the number of Transformer layers, convolution kernel size, and the depth of the multimodal fusion layer.

[0130] To demonstrate the effectiveness of the method in this experimental example, several cutting-edge methods were selected for comparison. The existing methods include:

[0131] AMD (Adversarial Multimodal Dynamics): A framework for generating adversarial multimodal dynamics that uses adversarial training strategies to enhance the robustness of models to missing or noisy modalities while optimizing the consistency and diversity of cross-modal features.

[0132] GME-LSTM (Gated Multimodal Embedding LSTM): A long short-term memory network based on gated multimodal embedding. It adaptively fuses multimodal temporal features through a dynamic gating mechanism, capturing long-term dependencies and optimizing cross-modal information interaction.

[0133] CHFusion (Cross-modal Hierarchical Fusion): A cross-modal hierarchical fusion model that gradually integrates complementary information from different modalities through multi-level feature abstraction and fusion strategies (such as low-level signal alignment and high-level semantic fusion);

[0134] MMMU-BA (Multimodal Multitask Bidirectional Attention): A multimodal multitask bidirectional attention network uses a bidirectional attention mechanism to share modal features between different tasks and dynamically adjust the weight distribution of cross-modal information. Seq2Seq2sent (Sequence-to-Sequence-to-Sentence): A cascaded sequence-to-sequence framework that first generates intermediate semantic encodings and then decodes them into natural language sentences, supporting efficient conversion of multimodal inputs to text.

[0135] MCTN (Modality-Centric Translation Network): Promotes inter-modal interactions through a modality translation strategy, learns the potential alignment relationship between modalities using a bidirectional mapping mechanism, and supports robust joint representation learning.

[0136] TransModality (Universal Transformer-based Modality Interaction): A universal framework based on cross-modal transformation that enables dynamic feature interaction between arbitrary modalities through multi-directional modal transformation modules (such as modality projection and attention alignment);

[0137] CTFN (Cross-modal Transformer Fusion Network): A cross-modal Transformer fusion network that uses a multi-head self-attention mechanism to align multimodal features and optimizes cross-modal semantic consistency through global context modeling.

[0138] The experiment validated the performance of this method by implementing multimodal sentiment analysis (covering unimodal, bimodal, and trimodal combinations) on the CMU-MOSI dataset and dual-task sentiment and emotion classification on the MELD dataset. The experiment was divided into two parts: the first part evaluated the performance of different modal combinations on CMU-MOSI, and the second part analyzed the results of the multi-task scenario on the MELD dataset.

[0139] In the CMU-MOSI dataset, the method of this embodiment is compared with other methods in single-modality (language L, audio A, vision V), dual-modality (A+V, L+V, L+A) and tri-modality (L+A+V) scenarios. The experimental results are shown in Table 1. For the single-modality case, in the language modality (L), the method outperforms all the comparison models with an accuracy of 81.40%, which is 0.61% higher than the second-place CTFN (80.79%). In the visual modality (V), the method surpasses AMD (55.20%) and CTFN (60.98%) with an accuracy of 61.13%, but the performance of the audio modality (A) is slightly lower than that of CTFN (54.42% vs. 61.43%). For the dual-modality case, in the (L+A) combination, the method significantly outperforms CTFN (82.77%) with an accuracy of 82.77%. 82.16%) and TransModality (81.25%), with an improvement of 0.61%-1.52%; in the (A+V) and (L+V) combinations, the proposed method achieved 62.04% and 81.25% respectively, both close to or exceeding the optimal baseline model; for the trimodal case, the proposed method achieved an accuracy of 81.55% in the trimodal (L+A+V) scenario, which is comparable to CTFN (82.77%), but significantly better than TransModality (82.71%) and MMMU-BA (81.25%).

[0140] In the sentiment and emotion classification tasks of the MELD dataset, the comparison between the method of this embodiment and the existing methods is shown in Table 2. In the sentiment classification task, in the unimodal (L) scenario, the method outperforms GME-LSTM (65.52%) and TransModality (67.04%) with an accuracy of 67.09%. In the bimodal (L+A) scenario, the method is close to the optimal CTFN (67.82%) with an accuracy of 66.82%, and significantly surpasses M CTN (66.27%) and Seq2Seq2Sent (63.84%). In the emotion classification task, in the unimodal (L / A) scenario, this method maintains competitiveness with mainstream models in both the language modality (58.20%) and the audio modality (49.00%). In the bimodal (L+A) combination, this method outperforms CHFusion (58.35%) and MCTN (59.96%) with an accuracy of 59.62%, but is slightly lower than TransModality (61.95%).

[0141] Table 1 Comparison of various SOTA models on the CMU-MOSI dataset;

[0142]

[0143] Table 2 Comparison of various SOTA models on the MELD dataset;

[0144]

[0145]

[0146] The above experiments show that the method of this embodiment has advantages in the following scenarios: (1) Language modality-dominated tasks: Through deep semantic encoding and cross-modal attention mechanism, the optimal performance is achieved in both the L single-modality and L+A bimodal tasks of CMU-MOSI; (2) Dynamic feature interaction: In bimodal fusion (such as the L+A combination of CMU-MOSI), the adaptive gating mechanism effectively balances the contribution between modalities and avoids redundant information interference; (3) Multi-task robustness: In the MELD dataset, the feature representation of the emotion and mood classification tasks is jointly optimized to alleviate the conflict problem between tasks.

[0147] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0148] The above embodiments are only preferred embodiments of the present invention and cannot be used to limit the scope of protection of the present invention. Any non-substantial changes and replacements made by technicians in this field on the basis of the present invention fall within the scope of protection required by the present invention.

Claims

1. A multimodal sentiment classification method based on corpus augmentation and cross-modal generation, characterized by: The steps include: Step 1: Preprocess and extract features from multimodal data; wherein the multimodal data includes video modality, text modality, and audio modality; Step 2: Project each modal feature into a low-dimensional shared space. The fully connected layer is used to map the video, text, and audio modal features into a unified low-dimensional space to eliminate the dimensionality differences between the modalities, and the ReLU activation function is used to enhance the expressive power. Step 3: Perform feature enhancement on each input modal feature based on the text library to output enhanced modal features; this includes: Step 3A: pre-build a text library containing low-dimensional text feature tensors and labels; Step 3B: Calculate the cosine similarity matrix between the input features and the text library features, and select the top-k relevant features based on the learnable temperature coefficient; Step 3C: The retrieved features are fused with the input features through a multi-head attention mechanism, and enhanced features are generated through residual connections and layer normalization. Step 4: Generate missing modal features across modalities and output multimodal data of non-missing modalities; this includes: Step 4A: Add dynamic position coding and modality type coding to the input sequence to distinguish timing information from modality attributes; Step 4B: Restrict the direction of information flow through a masking mechanism and generate pseudo features of the missing modality in an autoregressive manner; Step 4C: Constrain the consistency between the pseudo features and the real features generated by the missing mode using the mean square error loss; Step 5: Multimodal dynamic fusion and classification. First, the fused multimodal features are output and the Transformer classifier is used for sentiment classification. Step 5A: Use temporal convolution to align the features of each modality and model the temporal dependency through the gated recurrent unit (GRU); Step 5B: Introduce a dynamic gating mechanism to screen key features and combine it with a multi-head attention mechanism to fuse multimodal information; Step 5C: Aggregate cross-modal features through the global information carrier MMseq to generate a joint representation; Step 5D: Map the joint representation to the emotion category space, select L1 loss or cross entropy loss for optimization according to the task type, and then use the softmax function to convert the output into a probability distribution to represent the possibility of different emotion labels.

2. The multimodal sentiment classification method based on corpus enhancement and cross-modal generation according to claim 1, characterized in that: The step 1 includes the following sub-steps: Step 1A. Use OpenFace 2.0 to extract facial motion, head posture, and timestamp features from the video modality data to generate a 709-dimensional feature vector. Step 1B. Extract 768-dimensional semantic features from text modal data using the pre-trained BERT model; Step 1C. Extract 33-dimensional audio features including MFCC and CQT from the audio modal data using the Librosa tool; 3. The multimodal sentiment classification method based on corpus enhancement and cross-modal generation according to claim 1, characterized in that: In step 3A, the construction of the text library includes the following operations: Filter text samples from the CMU-MOSI and MELD datasets; Compress the original high-dimensional text features into a low-dimensional tensor through a linear projection layer; Store low-dimensional feature tensors, data labels, projection layer weights, and dimension information as structured files; The input text features are aggregated into feature vectors according to the mean, maximum or sum mode, and L2 normalized.

4. The multimodal sentiment classification method based on corpus enhancement and cross-modal generation according to claim 1, characterized in that: In step 4, when generating missing modal features, the learnable multimulti tag is initialized as the generation starting point, and the inter-modal attention range is controlled by the mask matrix, limiting the generation module to focus only on the historical information of the existing modalities.

5. The multimodal sentiment classification method based on corpus enhancement and cross-modal generation according to claim 1, characterized in that: In step 5, the dynamic gating mechanism generates weights through a learnable linear layer and a Sigmoid function to dynamically enhance important features and suppress redundant noise.

6. The multimodal sentiment classification method based on corpus enhancement and cross-modal generation according to claim 1, characterized in that: The MMseq sequence achieves the following functions through the cross-modal attention mechanism and mask rules: Actively obtain unimodal feature information as a query matrix; Provides global context to a single modality as a key-value matrix; Direct interaction between different modalities is prohibited, and only indirect information fusion through MMseq is allowed.

7. The multimodal sentiment classification method based on corpus enhancement and cross-modal generation according to claim 1, characterized in that: The method supports unimodal, bimodal, and trimodal input combinations.

8. The multimodal emotion system based on corpus augmentation and cross-modal generation according to claim 1, characterized in that: Implementing the method according to any one of claims 1 to 8, comprising: The text database construction module is used to filter text modality samples from the multimodal dataset and project the original high-dimensional text features into low-dimensional text features through text projection to build a low-dimensional text feature library; A text enhancement module is configured to retrieve similar samples from the text database based on the available modalities of the current input data, enhance the features of the currently available modalities using the sample features in the text database, and output an enhanced multimodal feature tensor; The multimodal generation module, when the input data has missing modalities, uses the enhanced representation obtained by the text enhancement module and the mask-based attention mechanism to generate pseudo-modalities and output multimodal data without missing modalities; The multimodal dynamic fusion module based on gated filtering introduces a dynamic gating mechanism after the attention calculation of the Transformer encoder layer. That is, it uses learnable linear layer weights to learn the feature mapping relationship from the input features, and converts the output of the linear layer into gating weights through the sigmoid function. These weights are used to dynamically filter key features, control the fusion strength of cross-modal information, enhance important features, achieve efficient feature screening, and output fused multimodal features. The sentiment classification module performs sentiment prediction through the Softmax classifier based on the fused multimodal features, and dynamically selects the corresponding loss function according to the task type and label format of the multimodal dataset to adapt to different task requirements.

9. A computer-readable storage medium, characterized in that: A computer program is stored, and when the program is executed by a processor, the multimodal sentiment analysis method according to any one of claims 1 to 8 is implemented.

10. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory, wherein when the processor executes the program, the multimodal sentiment analysis method according to any one of claims 1 to 8 is implemented.

Images