An implementation method of a multi-modal sentiment prediction system based on modal optimization and sentiment dimension alignment

Abstract

This invention discloses a method for implementing a multimodal sentiment prediction system based on modality optimization and sentiment dimension alignment. This method achieves effective fusion and semantic consistency enhancement of multimodal sentiment features from text, audio, and video through text-audio-video modal mutual optimization and multidimensional sentiment alignment mechanisms. Furthermore, it achieves accurate, stable, and interpretable joint prediction of sentiment polarity and intensity through a geometric vector prediction strategy. This invention introduces a text-audio-video bidirectional modality optimization mechanism, effectively bridging the natural differences in sentiment expression between heterogeneous modalities. Through external audio and video adapters, this invention injects non-verbal intensity features from audio and video into the text representation, enhancing the emotional intensity expression of the text. Simultaneously, it utilizes a cross-modal attention mechanism to back-calibrate audio and video features with optimized text semantics, significantly improving the information differences across different dimensions of multimodal sentiment data during the input stage.

Description

Technical Field

[0001] This invention relates to a method for implementing a multimodal sentiment prediction system based on modality optimization and sentiment dimension alignment, belonging to the field of communication technology. Background Technology

[0002] With the explosive growth of mobile internet and short video social platforms, multimodal sentiment analysis has become a research hotspot in the fields of natural language processing and multimedia computing. Traditional sentiment analysis mainly relies on the text modality, but when faced with video data containing irony, puns, or intense emotions, text alone is insufficient to capture the speaker's true intentions. Therefore, multimodal sentiment analysis techniques that integrate text content, auditory aspects such as speech tone, and visual aspects such as facial expressions and body language have emerged to provide a more comprehensive and accurate understanding of emotions. However, despite the progress made in multimodal feature fusion, existing technologies still face many challenges in processing complex emotional expressions, particularly in the refined representation of the emotional dimension, optimization of intermodal complementarity, and semantic alignment of heterogeneous features, where significant technical bottlenecks remain.

[0003] Most existing multimodal sentiment analysis methods tend to treat emotions as simple discrete categories, such as happiness, sadness, anger, or single polarity scores, such as positive or negative, ignoring the continuous and multidimensional characteristics of emotions themselves. Psychological research shows that the two-dimensional sentiment model Russell can more accurately describe human emotions, namely, it represents them through two orthogonal dimensions: emotional valence and emotional arousal. Valence reflects the degree of positive or negative pleasure in an emotion, while arousal reflects the degree of strong or weak physiological activation in an emotion. Existing deep learning models often fail to effectively decouple these two dimensions, leading to confusion between "high-arousal negative emotion" such as anger and "high-arousal positive emotion" such as surprise during prediction, thus failing to achieve accurate characterization of emotional states.

[0004] Different modalities exhibit differences in emotional expression, and existing methods lack effective modality optimization mechanisms. Text modalities are typically rich in explicit semantic information, effectively indicating emotional polarity (valence / efficacy in emotion models), but are relatively subtle in expressing emotional intensity (arousal). Conversely, audio and video modalities, while significantly advantageous in conveying emotional intensity, contain substantial environmental noise and struggle to independently determine emotional polarity without semantic guidance. Existing techniques often employ simple feature concatenation, failing to fully leverage the intensity information of audio and video to optimize text features and the text's polarity semantics to reverse-calibrate the bidirectional complementary nature of audio and video features. This results in the underutilization of the complementary potential between modalities and even the introduction of redundant noise. Furthermore, the heterogeneity of multimodal feature distributions leads to semantic misalignment. Text, audio, and video features reside in different embedding spaces, and direct fusion results in misalignment of feature distributions. Existing methods lack explicit constraints for emotional dimension alignment, meaning they cannot guarantee the consistency of features from different modalities under the same emotional state in the latent space, nor can they guarantee effective separation of features from the same modality in both valence / efficacy and arousal dimensions. This disorder in the feature space not only reduces the robustness of the model, but also limits its generalization ability in complex scenarios. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings and deficiencies of the existing technology by proposing a method for implementing a multimodal sentiment prediction system based on modal optimization and sentiment dimension alignment. This method can achieve effective fusion and semantic consistency enhancement of multimodal sentiment features of text, audio and video through text-audio-video modal mutual optimization and multidimensional sentiment alignment mechanism. Furthermore, through a geometric vector prediction strategy, it can achieve accurate, stable and interpretable joint prediction of sentiment polarity and sentiment intensity.

[0006] The technical solution adopted by this invention to solve its technical problem is: a method for implementing a multimodal sentiment prediction system based on modality optimization and sentiment dimension alignment, the method comprising the following steps: Step 1: Connect the pre-trained video adapter and audio adapter to the text feature extractor via an external connection to inject emotional intensity features from the video and audio modalities into the text feature extractor to optimize the text feature extraction process.

[0007] Step 2: Based on the optimized text features, a cross-modal attention mechanism is introduced to perform reverse optimization on the audio and video modal features, so that the emotional expression in the audio and video modal is more in line with the emotional polarity reflected in the text.

[0008] Step 3: Construct emotion polarity feature extractors and emotion intensity feature extractors for the three modalities of text, audio, and video respectively, and decouple the coupled modal features into independent emotional arousal and valence components in the corresponding two-dimensional emotion model.

[0009] Step 4: Construct a multidimensional sentiment alignment and feature optimization network based on hierarchical contrastive learning. By designing contrastive constraint mechanisms at three levels—dimension, modality, and sample—we achieve feature decoupling between the sentiment valence dimension and the arousal dimension, unified semantic space alignment between different modalities, and discriminative representation learning for samples with different sentiment scores.

[0010] Step 5: Design a cross-axis interactive fusion network to ensure the hierarchical deep fusion of text, audio and video features while maintaining dimensional independence.

[0011] Step 6: Transform the multimodal features processed by the Cross-Axis Interactive Fusion Network (C-DFN) into the final sentiment prediction vector to achieve the analysis of the emotional state of the target object in the video content.

[0012] Furthermore, the present invention optimizes the text feature extraction process by: using a pre-trained BERT model as the backbone encoder to extract general semantic features of the text; using an external audio sentiment adapter to re-encode the hidden states extracted by the backbone encoder to obtain specific sentiment intensity knowledge that is potentially associated with the text content; and fusing the general semantic features of the text extracted by the backbone encoder with the features injected by the audio / video adapter to generate the final optimized text features, calculated as follows: in, These are the pre-trained parameters of the BERT model. It is the initial feature of the text modality. It is the set of hidden states of all layers in BERT. The initial global features of the extracted text, It involves pre-training fixed adapter parameters on audio and video data. It is a specific text feature vector infused with the emotional intensity of audio and video. This represents a vector concatenation operation. and For the weights and biases of the fusion layer, To optimize text features.

[0013] Furthermore, based on the optimized text features injected with emotional intensity information, this invention introduces a cross-modal attention mechanism to perform reverse optimization on the audio and video modal features, making the emotional expression in the audio and video modal more consistent with the emotional polarity reflected in the text. The reverse optimization includes: extracting text feature vectors representing emotional polarity information from the optimized text features as the key (K) and value (V) of the cross-modal attention, and using the audio and video modal features as queries (Q), respectively. The calibrated representations of the audio and video are obtained through multi-head cross-modal attention, calculated as follows: in Represents the initial visual or acoustic modal characteristics. This represents a multi-head attention function. This represents the intermediate features of the first stage after cross-modal attention and residual connections, at which point the sentiment intensity information of the text has been incorporated. This represents the intermediate features of the second stage after the self-attention mechanism and residual connections, which further enhances the context dependency. This represents the final optimized audio or video features generated after passing through the feedforward network and residual connections. Representation layer normalization, This represents a feedforward network.

[0014] Furthermore, the present invention decouples the optimized multimodal features in two-dimensional space by constructing independent sentiment polarity feature extractors and sentiment intensity feature extractors for text, audio, and video modalities, respectively, with the following calculation formulas: in, These represent text, audio, and video modalities, respectively. The weights and biases of the affective valence feature extractor. Weights and biases for the emotional arousal feature extractor; Activation functions are used to introduce nonlinearity, thereby improving the expressive power of the model; The feature vector representing the emotional polarity obtained through mapping. This is the feature vector representing the intensity of emotion obtained through mapping.

[0015] Furthermore, at the dimensional decoupling level, to ensure the independence of sentiment polarity and sentiment intensity in the semantic space, for the same modality of the same sample, its sentiment polarity feature vector and sentiment intensity feature vector are constructed as negative sample pairs between dimensions. At the modality alignment level, to ensure the consistency of expression of the same sentiment dimension across different modalities, for the same real sample, the sentiment polarity features of text, audio, and video are constructed as positive sample pairs, and the sentiment intensity features of the three are constructed as positive sample pairs. At the supervised classification level, using real sentiment labels, for different samples within a batch, if their sentiment labels belong to the same numerical range, they are constructed as positive sample pairs; otherwise, they are constructed as negative sample pairs. The contrastive loss function adopts the InfoNCE loss function form, and the calculation formula is as follows: in, For training batches, For the sample The set of all positive sample pairs at the dimensional, modal, and sample levels. For the sample The set of all negative sample pairs, For cosine similarity calculation, The temperature coefficient is used to adjust the smoothness of the similarity distribution; the loss function aims to maximize the mutual information of positive sample pairs and minimize the mutual information of negative sample pairs, thereby driving the model to automatically complete multi-dimensional feature alignment.

[0016] Furthermore, this invention employs a cross-axis interactive fusion network (C-DFN) to maximize feature fusion efficiency through a minimalist architecture. This network abandons complex attention calculations and instead utilizes a pure multilayer perceptron (MLP) to establish a bidirectional fusion channel: information sharing and fusion are facilitated along the "modal axis," while semantic extraction is deepened along the "feature axis." This alternating mapping mechanism ensures hierarchical fusion of text, audio, and video features during transmission, as shown in the following calculation formula: in, The multimodal polarity matrix to be fused. The multimodal intensity matrix to be fused; superscript Indicates the emotional polarity dimension, superscript Indicates the dimension of emotional intensity; subscript These represent the text, audio, and video modal axis channels, respectively. This represents the dimension of the feature vector.

[0017] For multimodal feature matrices Deep interaction is performed in the Cross-Axis Interactive Fusion Network (C-DFN), and the calculation formula is as follows: in, The original multimodal feature matrix is ​​the input to the network; This is the intermediate interaction matrix after modal axis mixing; A weighted weight matrix for learning intermodal correlations; The activation function for the Gaussian error linear unit; It is the matrix transpose operator; For layer normalization processing; This is the final output matrix of cross-axis dense interaction features; This is the mapping weight matrix used for feature semantic extraction.

[0018] Furthermore, this invention transforms the multimodal features processed by the Cross-Axis Interactive Fusion Network (C-DFN) into a final sentiment prediction vector, enabling the analysis of the emotional state of target objects in video content. The calculation formula is as follows: in, and These are the polarity and intensity feature matrices of the input, respectively; This represents the average pooling operation along the modal axis dimension; This represents taking the L1 norm of the global intensity vector as a scalar gain characterizing the intensity of emotion. This indicates that L2 normalization is applied to the global polarity vector, and the resulting unit vector represents the valence-efficacy direction of sentiment. It combines intensity scalars and polarity directions to be used as input to fully connected layers for final sentiment analysis; This represents a feedforward neural network used for nonlinear transformation of features. Presentation layer normalization operation; Predict labels for the model, As a tag for genuine emotions, Let N represent the loss function for the predicted values, and N represent the number of samples. The overall loss function for the model is expressed as: , The weighting coefficients are used to balance the two loss terms.

[0019] As a preferred embodiment of the multimodal sentiment prediction method based on modality optimization and sentiment dimension alignment described in this invention, the optimization of the text feature extraction process includes: Pre-constructed audio emotion adapters: The audio emotion adapters consist of three independent cascaded adapter units, respectively mounted on layers 0, 11, and 23 of the backbone BERT model. Each adapter unit internally contains a lower projection layer, a Transformer coding layer, an upper projection layer, and a residual connection spanning the entire model. Let the input of the adapter unit be... , ( The specific layer structure definition and calculation logic for each step are as follows: in, and The weights and biases of the next projection layer, This indicates a multi-head self-attention mechanism. It is a feedforward neural network. Representation layer normalization, and The weights and biases of the upper projection layer. This indicates that the final output feature of the k-th layer attached to the backbone BERT model will be used as the input of the next-level adapter unit or as the final output of the adapter.

[0020] Training the audio emotion adapter: Extract audio and video modal features from the SIMS dataset, use the initialized adapter network as a feature encoder, connect a linear regression layer at the back end to predict the emotion intensity value corresponding to the sample, and define the mean squared error (MSE) loss function for training.

[0021] in Labels representing the true intensity of emotion. The adapter outputs the predicted values ​​after passing through the regression layer. Once the model converges, the adapter parameters are saved. In subsequent main tasks, these adapter parameters are frozen, and no further gradient updates are performed.

[0022] Using the BERT model as the backbone encoder, general semantic features of text are extracted. The BERT model includes... The Transformer coding structure of BERT. layer( The feature extraction process can be represented as follows: Indicates the first The set of hidden states output by the layer, with dimension . The final output includes the global semantic feature vector from layer 23. and the hidden states of each intermediate layer. .

[0023] Using a cascaded adapter unit auxiliary path structure, connected to layers 0, 11, and 23 of the BERT backbone respectively, the adapter feature generation adopts a hierarchical recursive mechanism. The specific calculation process is as follows: The first-level adapter unit connects to BERT layer 0, and stores the hidden states of BERT layer 0. with initial text features To splice, The result is obtained after processing by the first adapter unit. .

[0024] The output features of the first-level adapter unit mainly capture the correlation between the underlying lexical and basic sentiment signals.

[0025] The second-level adapter unit connects to BERT layer 11, hiding the intermediate layer (layer 11) of BERT. Output of the previous level adapter unit To splice, The result is obtained after processing by the second adapter unit. .

[0026] It achieves deep interaction between BERT's mid-level syntactic semantics and preceding sentiment features.

[0027] The third-level adapter unit connects to BERT layer 23, hiding the intermediate layer (layer 23) of BERT. Output of the previous level adapter unit To splice, The result is obtained after processing by the third adapter unit. That is .

[0028] Final output This refers to a specific text feature vector that has been infused with the full emotional intensity of the audio and video.

[0029] The general semantic features of the text extracted by the backbone encoder are fused with the audio and video enhancement features injected by the external adapter to generate the final optimized text features. The calculation formula is as follows: in, This represents a vector concatenation operation. and For the weights and biases of the fusion layer, These are the optimized text features ultimately used for sentiment prediction.

[0030] As a preferred embodiment of the multimodal sentiment prediction method based on modality optimization and sentiment dimension alignment described in this invention, the introduction of a cross-modal attention mechanism to perform reverse optimization of audio modality features and video modality features includes: Because different modalities have significantly different focuses in representing emotional information, the text modality usually expresses emotional tendencies directly through explicit emotional polarity words or semantic structures. Its features naturally include discrete and semantically clear emotional polarity information such as "happy", "sad", and "angry". In contrast, the audio and video modalities mainly reflect emotional intensity information implicitly through continuous signals such as tone changes, energy levels, speech rate, facial expressions, or body movements. The semantic interpretability of these modalities at the level of emotional polarity is relatively insufficient and they are easily affected by individual differences in speakers or environmental noise.

[0031] Therefore, based on the optimized text features obtained after injecting emotional intensity information, a cross-modal attention mechanism is introduced to perform reverse optimization on the audio and video modal features, making the emotional expression in the audio and video modal more consistent with the emotional polarity reflected in the text. The reverse optimization includes: extracting text feature vectors representing emotional polarity information from the optimized text features as the key (K) and value (V) of the cross-modal attention, and using the audio and video modal features as queries (Q), respectively. The calibrated representations of audio and video are obtained through multi-head cross-modal attention, and the calculation formula is as follows: in Represents the initial visual or acoustic modal characteristics. This represents a multi-head attention function. This represents the intermediate features of the first stage after cross-modal attention and residual connections, at which point the sentiment intensity information of the text has been incorporated. This represents the intermediate features of the second stage after the self-attention mechanism and residual connections, which further enhances the context dependency. This represents the final optimized audio or video features generated after passing through the feedforward network and residual connections. Representation layer normalization, This represents a feedforward network.

[0032] As a preferred embodiment of the multimodal sentiment prediction method based on modality optimization and sentiment dimension alignment described in this invention, the two-dimensional spatial dimension decoupling of the optimized multimodal features includes: constructing independent sentiment polarity feature extractors and sentiment intensity feature extractors for text, audio, and video modalities respectively, with the calculation formulas as follows: in, These represent text, audio, and video modalities, respectively. The weights and biases of the affective valence feature extractor. Weights and biases for the emotional arousal feature extractor; Activation functions are used to introduce nonlinearity, thereby improving the expressive power of the model; The feature vector representing the emotional polarity obtained through mapping. This is the feature vector representing the intensity of emotion obtained through mapping.

[0033] As a preferred embodiment of the multimodal sentiment prediction method based on modality optimization and sentiment dimension alignment described in this invention, a multidimensional sentiment alignment and feature optimization network based on hierarchical contrastive learning is constructed. At the dimensional decoupling level, to ensure the independence of sentiment polarity and sentiment intensity in the semantic space, for the same modality of the same sample, its sentiment polarity feature vector and sentiment intensity feature vector are constructed as negative sample pairs between dimensions. Regarding the modality alignment level, to ensure the consistency of expression of the same sentiment dimension across different modalities, for the same real sample, the sentiment polarity features of text, audio, and video are constructed as positive sample pairs, and the sentiment intensity features of the three are constructed as positive sample pairs. Regarding the supervised classification level, using real sentiment labels, for different samples within a batch, if their sentiment labels belong to the same numerical range, they are constructed as positive sample pairs; otherwise, they are constructed as negative sample pairs. The contrastive loss function adopts the InfoNCE loss function form, and the calculation formula is as follows: in, For training batches, For the sample The set of all positive sample pairs at the dimensional, modal, and sample levels. For the sample The set of all negative sample pairs, For cosine similarity calculation, The temperature coefficient is used to adjust the smoothness of the similarity distribution; the loss function aims to maximize the mutual information of positive sample pairs and minimize the mutual information of negative sample pairs, thereby driving the model to automatically complete multi-dimensional feature alignment.

[0034] As a preferred embodiment of the multimodal sentiment prediction method based on modality optimization and sentiment dimension alignment described in this invention, a cross-axis interactive fusion network (C-DFN) is constructed to maximize feature fusion efficiency through a minimalist architecture. This network abandons complex attention calculations and instead utilizes a pure multilayer perceptron (MLP) to establish a bidirectional fusion channel: information sharing and fusion are facilitated along the "modality axis," while semantic extraction is deepened along the "feature axis." This alternating mapping mechanism ensures hierarchical fusion of text, audio, and video features during transmission, as shown in the following calculation formula: in, The multimodal polarity matrix to be fused. The multimodal intensity matrix to be fused; superscript Indicates the emotional polarity dimension, superscript Indicates the dimension of emotional intensity; subscript These represent the text, audio, and video modal axis channels, respectively. This represents the dimension of the feature vector.

[0035] For multimodal feature matrices Deep interaction is performed in the Cross-Axis Interactive Fusion Network (C-DFN), and the calculation formula is as follows: in, The original multimodal feature matrix is ​​the input to the network; This is the intermediate interaction matrix after modal axis mixing; A weighted weight matrix for learning intermodal correlations; The activation function for the Gaussian error linear unit; It is the matrix transpose operator; For layer normalization processing; This is the final output matrix of cross-axis dense interaction features; This is the mapping weight matrix used for feature semantic extraction.

[0036] As a preferred embodiment of the multimodal sentiment prediction method based on modality optimization and sentiment dimension alignment described in this invention, the multimodal features processed by the Cross-Axis Interactive Fusion Network (C-DFN) are transformed into the final sentiment prediction vector, thereby enabling the analysis of the emotional state of the target object in the video content. The calculation formula is as follows: in, and These are the polarity and intensity feature matrices of the input, respectively; This represents the average pooling operation along the modal axis dimension; This represents taking the L1 norm of the global intensity vector as a scalar gain characterizing the intensity of emotion. This indicates that L2 normalization is applied to the global polarity vector, and the resulting unit vector represents the valence-efficacy direction of sentiment. It combines intensity scalars and polarity directions to be used as input to fully connected layers for final sentiment analysis; This represents a feedforward neural network used for nonlinear transformation of features. Presentation layer normalization operation; Predict labels for the model, As a tag for genuine emotions, Let N represent the loss function for the predicted values, and N represent the number of samples. The overall loss function for the model is expressed as: , The weighting coefficients are used to balance the two loss terms.

[0037] Beneficial effects: 1. This invention introduces a text-audio-video bidirectional modal optimization mechanism, effectively bridging the inherent differences in emotional expression between heterogeneous modalities. Through external audio and video adapters, this invention injects non-verbal intensity features from audio and video into the text representation, enhancing the emotional intensity expression of the text. Simultaneously, it utilizes a cross-modal attention mechanism to back-calibrate audio and video features with optimized text semantics, solving the problem of ambiguous emotional polarity in audio and video modalities lacking semantic guidance, and significantly improving the information differences across different dimensions of multimodal emotional data during the input stage.

[0038] 2. This invention constructs an emotion polarity feature extractor and an emotion intensity feature extractor, and utilizes a multidimensional emotion alignment network based on hierarchical contrastive learning to achieve deep decoupling and alignment of emotion valence (polarity) and arousal (intensity). By designing contrastive constraints at three levels—dimensionality, modality, and sample—this invention forces the model to distinguish between emotion polarity and intensity in the latent space, effectively avoiding confusion in prediction of high-arousal emotions (such as "anger" and "surprise"), ensuring a high degree of consistency of multimodal features under the same emotional state in the semantic space, and enhancing the model's discriminative ability in complex emotional scenarios.

[0039] 3. This invention designs a cross-axis interactive fusion network (C-DFN) and a geometric vector synthesis strategy, achieving efficient and interpretable sentiment prediction. C-DFN abandons the computationally intensive attention mechanism and utilizes a pure multilayer perceptron (MLP) to establish bidirectional interactive channels on the "modal axis" and "feature axis," significantly reducing computational complexity while achieving deep feature fusion. The final geometric prediction strategy uses sentiment polarity as the vector direction and sentiment intensity as the vector magnitude for physical quantity synthesis, providing a decision-making basis for video character sentiment analysis that combines high accuracy and strong robustness. Attached Figure Description

[0040] Figure 1 The flowchart shows the overall process of the multimodal sentiment prediction method based on modal optimization and sentiment dimension alignment provided by this invention.

[0041] Figure 2 This is a schematic diagram of the structure of the audio and video emotion adapter provided by the present invention.

[0042] Figure 3 This is a schematic diagram of the structure for cross-modal attention-based reverse optimization of audio and video features provided by the present invention.

[0043] Figure 4 This is a schematic diagram of the cross-axis interactive fusion network (C-DFN) structure provided by the present invention, which includes alternating MLP mappings of "modal axis mixing" and "feature axis mixing" to achieve hierarchical deep fusion and semantic extraction of multimodal features. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] Example 1 like Figure 1 The diagram shown is a schematic representation of the method flow in this embodiment. The specific steps include the following: S1: Inject emotional intensity features into the text feature extractor through an external audio / video adapter to optimize text feature extraction.

[0047] Pre-built audio emotion adapters, such as Figure 2As shown, the internal structure of the adapter unit is as follows: The audio emotion adapter consists of three independent cascaded adapter units, respectively mounted on layers 0, 11, and 23 of the backbone BERT model. Each adapter unit contains a lower projection layer, a Transformer coding layer, an upper projection layer, and a residual connection spanning the entire model, connected in sequence. Let the input of the adapter unit be... , ( The specific layer structure definition and calculation logic for each step are as follows: in, and The weights and biases of the next projection layer, This indicates a multi-head self-attention mechanism. It is a feedforward neural network. Representation layer normalization, and The weights and biases of the upper projection layer. This indicates that the final output feature of the k-th layer attached to the backbone BERT model will be used as the input of the next-level adapter unit or as the final output of the adapter.

[0048] Training the audio emotion adapter: Extract audio and video modal features from the SIMS dataset, use the initialized adapter network as a feature encoder, connect a linear regression layer at the back end to predict the emotion intensity value corresponding to the sample, and define the mean squared error (MSE) loss function for training.

[0049] in Labels representing the true intensity of emotion. The adapter outputs the predicted values ​​after passing through the regression layer. Once the model converges, the adapter parameters are saved. In subsequent main tasks, these adapter parameters are frozen, and no further gradient updates are performed.

[0050] Using the BERT model as the backbone encoder, general semantic features of text are extracted. The BERT model includes... The Transformer coding structure of BERT. layer( The feature extraction process can be represented as follows: Indicates the first The set of hidden states output by the layer, with dimension . The final output includes the global semantic feature vector from layer 23. and the hidden states of each intermediate layer. .

[0051] Using a cascaded adapter unit auxiliary path structure, connected to layers 0, 11, and 23 of the BERT backbone respectively, the adapter feature generation adopts a hierarchical recursive mechanism. The specific calculation process is as follows: The first-level adapter unit connects to BERT layer 0, and stores the hidden states of BERT layer 0. with initial text features To splice, The result is obtained after processing by the first adapter unit. .

[0052] The output features of the first-level adapter unit mainly capture the correlation between the underlying lexical and basic sentiment signals.

[0053] The second-level adapter unit connects to BERT layer 11, hiding the intermediate layer (layer 11) of BERT. Output of the previous level adapter unit To splice, The result is obtained after processing by the second adapter unit. .

[0054] It achieves deep interaction between BERT's mid-level syntactic semantics and preceding sentiment features.

[0055] The third-level adapter unit connects to BERT layer 23, hiding the intermediate layer (layer 23) of BERT. Output of the previous level adapter unit To splice, The result is obtained after processing by the third adapter unit. That is .

[0056] Final output This refers to a specific text feature vector that has been infused with the full emotional intensity of the audio and video.

[0057] The general semantic features of the text extracted by the backbone encoder are fused with the audio and video enhancement features injected by the external adapter to generate the final optimized text features. The calculation formula is as follows: in, This represents a vector concatenation operation. and For the weights and biases of the fusion layer, X t These are the optimized text features ultimately used for sentiment prediction.

[0058] S2: Introduce a cross-modal attention mechanism to perform reverse optimization of audio and video modal features based on optimized text features.

[0059] Because different modalities have significantly different focuses in representing emotional information, the text modality usually expresses emotional tendencies directly through explicit emotional polarity words or semantic structures. Its features naturally include discrete and semantically clear emotional polarity information such as "happy", "sad", and "angry". In contrast, the audio and video modalities mainly reflect emotional intensity information implicitly through continuous signals such as tone changes, energy levels, speech rate, facial expressions, or body movements. The semantic interpretability of these modalities at the level of emotional polarity is relatively insufficient and they are easily affected by individual differences in speakers or environmental noise.

[0060] like Figure 3 As shown, based on the optimized text features obtained after injecting emotional intensity information, a cross-modal attention mechanism is introduced to perform reverse optimization on the audio and video modal features, making the emotional expression in the audio and video modal more consistent with the emotional polarity reflected in the text. The reverse optimization includes: extracting text feature vectors representing emotional polarity information from the optimized text features as the key (K) and value (V) of the cross-modal attention, and using the audio and video modal features as queries (Q) respectively. The calibrated representations of audio and video are obtained through multi-head cross-modal attention, and the calculation formula is as follows: in Represents the initial visual or acoustic modal characteristics. This represents a multi-head attention function. This represents the intermediate features of the first stage after cross-modal attention and residual connections, at which point the sentiment intensity information of the text has been incorporated. This represents the intermediate features of the second stage after the self-attention mechanism and residual connections, which further enhances the context dependency. This represents the final optimized audio or video features generated after passing through the feedforward network and residual connections. Representation layer normalization, This represents a feedforward network.

[0061] S3: Construct an emotional polarity and intensity feature extractor to decouple the three-modal features.

[0062] Decoupling the optimized multimodal features in two dimensions includes: constructing independent sentiment polarity feature extractors and sentiment intensity feature extractors for text, audio, and video modalities, respectively, with the calculation formulas as follows: in, These represent text, audio, and video modalities, respectively. The weights and biases of the affective valence feature extractor. Weights and biases for the emotional arousal feature extractor; Activation functions are used to introduce nonlinearity, thereby improving the expressive power of the model; The feature vector representing the emotional polarity obtained through mapping. This is the feature vector representing the intensity of emotion obtained through mapping.

[0063] S4: Construct a multidimensional sentiment alignment and feature optimization network based on hierarchical contrastive learning to achieve feature decoupling and unified alignment of semantic space.

[0064] A multidimensional sentiment alignment and feature optimization network based on hierarchical contrastive learning is constructed. At the dimensional decoupling level, to ensure the independence of sentiment polarity and sentiment intensity in the semantic space, for the same modality of the same sample, its sentiment polarity feature vector and sentiment intensity feature vector are constructed as negative sample pairs between dimensions. At the modality alignment level, to ensure the consistency of expression of the same sentiment dimension across different modalities, for the same real sample, the sentiment polarity features of text, audio, and video are paired as positive sample pairs, and the sentiment intensity features of the three are paired as positive sample pairs. At the supervised classification level, using real sentiment labels, for different samples within a batch, if their sentiment labels belong to the same numerical range, they are constructed as positive sample pairs; otherwise, they are constructed as negative sample pairs. The contrastive loss function adopts the InfoNCE loss function form, and the calculation formula is as follows: in, For training batches, For the sample The set of all positive sample pairs at the dimensional, modal, and sample levels. For the sample The set of all negative sample pairs, For cosine similarity calculation, The temperature coefficient is used to adjust the smoothness of the similarity distribution; the loss function aims to maximize the mutual information of positive sample pairs and minimize the mutual information of negative sample pairs, thereby driving the model to automatically complete multi-dimensional feature alignment.

[0065] S5: Design a cross-axis interactive fusion network (C-DFN) to achieve hierarchical deep fusion of text, audio and video features.

[0066] like Figure 4 As shown, a cross-axis interactive fusion network (C-DFN) is constructed to maximize feature fusion efficiency through a minimalist architecture. This network abandons complex attention calculations and instead utilizes a pure multilayer perceptron (MLP) to establish a bidirectional fusion channel: information sharing and fusion are facilitated along the "modal axis," while semantic extraction is deepened along the "feature axis." This alternating mapping mechanism ensures hierarchical fusion of text, audio, and video features during transmission. The calculation formula is as follows: in, The multimodal polarity matrix to be fused. The multimodal intensity matrix to be fused; superscript Indicates the emotional polarity dimension, superscript Indicates the dimension of emotional intensity; subscript These represent the text, audio, and video modal axis channels, respectively. This represents the dimension of the feature vector.

[0067] For multimodal feature matrices Deep interaction is performed in the Cross-Axis Interactive Fusion Network (C-DFN), and the calculation formula is as follows: in, The original multimodal feature matrix is ​​the input to the network; This is the intermediate interaction matrix after modal axis mixing; A weighted weight matrix for learning intermodal correlations; The activation function for the Gaussian error linear unit; It is the matrix transpose operator; For layer normalization processing; This is the final output matrix of cross-axis dense interaction features; This is the mapping weight matrix used for feature semantic extraction.

[0068] S6: Convert the fused multimodal features into sentiment prediction vectors to complete the sentiment state analysis of the target object in the video.

[0069] The multimodal features processed by the Cross-Axis Interactive Fusion Network (C-DFN) are transformed into the final sentiment prediction vector, enabling the analysis of the emotional state of target objects in video content. The calculation formula is as follows: in, and These are the polarity and intensity feature matrices of the input, respectively; This represents the average pooling operation along the modal axis dimension; This represents taking the L1 norm of the global intensity vector as a scalar gain characterizing the intensity of emotion. This indicates that L2 normalization is applied to the global polarity vector, and the resulting unit vector represents the valence-efficacy direction of sentiment. It combines intensity scalars and polarity directions to be used as input to fully connected layers for final sentiment analysis; This represents a feedforward neural network used for nonlinear transformation of features. Presentation layer normalization operation; Predict labels for the model, As a tag for genuine emotions, Let N represent the loss function for the predicted values, and N represent the number of samples. The overall loss function for the model is expressed as: , The weighting coefficients are used to balance the two loss terms.

[0070] Example 2 Referring to Table 1, an embodiment of the present invention is provided, illustrating a method for implementing a multimodal sentiment prediction system based on modality optimization and sentiment dimension alignment. To verify its beneficial effects, a comparison with existing mainstream technical solutions is presented.

[0071] Table 1 shows the comparison of prediction performance of various models on the MOSI dataset. This invention utilizes the CMU-MOSI public sentiment dataset. This dataset is a leading benchmark dataset in the field of multimodal sentiment analysis, comprising video clips from 93 different speakers on YouTube. The dataset covers a rich range of head poses, facial expressions, speech intonation, and text content. The MOSI dataset contains 2199 sentiment-annotated video clips, each corresponding to a sentiment score ranging from -3 to +3.

[0072] For this dataset, we split it according to the standard split ratio, with 1284 segments used for training, 229 segments used for validation, and 686 segments used for testing.

[0073] The experimental environment was based on the PyTorch deep learning framework, and all experiments were conducted on a workstation equipped with an NVIDIA RTX 3090 GPU with 24GB of VRAM. Model training used the AdamW optimizer with an initial learning rate of 1e-5 and weight decay of 0.01. The text backbone network used the BERT-RoBERTa model (24 layers), with audio and video adapter units mounted in layers 0, 11, and 23, respectively. The training process was set to a maximum of 40 epochs, a batch size of 32, and an early stopping strategy: training stopped when the validation set loss did not decrease within 5 epochs. The model with the best performance on the validation set was then evaluated on the test set.

[0074] To evaluate the effectiveness of this invention, we used four mainstream evaluation metrics: Acc-2 (binary classification accuracy): measures the model's ability to correctly classify sentiment; F1 Score: comprehensively considers the precision and recall of the classification model; Corr (Pearson correlation coefficient): measures the linear correlation between the predicted sentiment value and the true label, which is particularly important for verifying the effectiveness of "sentiment dimension alignment" in this invention; and MAE (mean absolute error): measures the average absolute distance between the predicted value and the true value, with a lower value indicating better performance.

[0075] In our experiments, we compared our invention with eight baseline models representing different stages of development in the field of multimodal sentiment analysis, including: early fusion methods: LF-DNN late-stage fusion deep neural network, LMF low-rank multimodal fusion, TFN tensor fusion network; Transformer-based cross-modal methods: MulT multimodal Transformer; BERT-based augmentation methods: MAG-BERT multimodal adaptation gate; and the latest contrastive / hybrid learning methods: HyCon hybrid contrastive learning, Self-MM self-supervised multi-task learning, and MISA modality invariant and specific representation learning.

[0076] The experimental results are shown in the attached tables. This invention demonstrates excellent performance on the MOSI dataset, particularly showing significant advantages in metrics reflecting continuous changes in sentiment. Specifically: The present invention achieved an Acc-2 score of 85.32% and an F1 score of 85.22%, both exceeding the vast majority of baseline models, such as HyCon (85.2%) and MAG-BERT (83.54%), and performing very close to the state-of-the-art Self-MM model. This demonstrates the high accuracy of the present invention in sentiment polarity discrimination. Furthermore, the present invention achieved the highest Corr score of 0.802, outperforming all comparative models. This result strongly proves the effectiveness of the proposed "sentiment dimension alignment" and "modal optimization" mechanisms, indicating that the model can more accurately capture subtle trends in sentiment intensity, achieving a high degree of fit between predicted values ​​and true sentiment dimensions. Simultaneously, the MAE of the present invention is controlled at 0.727, maintaining a low error level.

[0077] In summary, by comparing with existing technologies, the superiority of this invention in handling complex multimodal sentiment prediction tasks is verified, especially in significantly improving the relevance and consistency of sentiment intensity prediction while ensuring high classification accuracy.

Claims

1. A method for implementing a multimodal sentiment prediction system based on modality optimization and sentiment dimension alignment, characterized in that, The method includes the following steps: Step 1: Connect the pre-trained video adapter and audio adapter to the text feature extractor via an external connection to inject emotional intensity features from the video modality and audio modality into the text feature extractor to optimize the text feature extraction process; Step 2: Based on the optimized text features, a cross-modal attention mechanism is introduced to perform reverse optimization on the audio modal features and video modal features, so that the emotional expression in the audio modal and video modal is more in line with the emotional polarity reflected in the text. Step 3: Construct an emotional polarity feature extractor and an emotional intensity feature extractor for the three modalities of text, audio, and video respectively, and decouple the coupled modal features into independent emotional arousal and valence components in the corresponding two-dimensional emotional model; Step 4: Construct a multidimensional sentiment alignment and feature optimization network based on hierarchical contrastive learning. By designing contrastive constraint mechanisms at three levels—dimension, modality, and sample—we achieve feature decoupling between the sentiment valence dimension and the arousal dimension, unified semantic space alignment between different modalities, and discriminative representation learning for samples with different sentiment scores. Step 5: Design a cross-axis interactive fusion network to ensure hierarchical deep fusion of text, audio, and video features while maintaining dimensional independence; Step 6: Transform the multimodal features processed by the Cross-Axis Interactive Fusion Network (C-DFN) into the final sentiment prediction vector to achieve the analysis of the emotional state of the target object in the video content.

2. The implementation method of a multimodal sentiment prediction system based on modality optimization and sentiment dimension alignment according to claim 1, characterized in that, The method optimizes the text feature extraction process by: using a pre-trained BERT model as the backbone encoder to extract general semantic features of the text; using an external audio sentiment adapter to re-encode the hidden states extracted by the backbone encoder to obtain specific sentiment intensity knowledge that is potentially related to the text content; and fusing the general semantic features of the text extracted by the backbone encoder with the features injected by the audio / video adapter to generate the final optimized text features, calculated as follows: in, These are the pre-trained parameters of the BERT model. It is the initial feature of the text modality. It is the set of hidden states of all layers in BERT. The initial global features of the extracted text, It involves pre-training fixed adapter parameters on audio and video data. It is a specific text feature vector infused with the emotional intensity of audio and video. This represents a vector concatenation operation. and For the weights and biases of the fusion layer, To optimize text features.

3. The implementation method of a multimodal sentiment prediction system based on modality optimization and sentiment dimension alignment according to claim 1, characterized in that, The method, based on the optimized text features injected with emotional intensity information, introduces a cross-modal attention mechanism to perform reverse optimization on the audio and video modal features, making the emotional expression in the audio and video modal more consistent with the emotional polarity reflected in the text. The reverse optimization includes: extracting text feature vectors representing emotional polarity information from the optimized text features as the key (K) and value (V) of the cross-modal attention, and using the audio and video modal features as queries (Q), respectively. The calibrated representations of the audio and video are obtained through multi-head cross-modal attention, calculated as follows: in Represents the initial visual or acoustic modal characteristics. This represents a multi-head attention function. This represents the intermediate features of the first stage after cross-modal attention and residual connections, at which point the sentiment intensity information of the text has been incorporated. This represents the intermediate features of the second stage after the self-attention mechanism and residual connections, which further enhances the context dependency. This represents the final optimized audio or video features generated after passing through the feedforward network and residual connections. Representation layer normalization, This represents a feedforward network.

4. The implementation method of a multimodal sentiment prediction system based on modality optimization and sentiment dimension alignment according to claim 1, characterized in that, The method decouples the optimized multimodal features in two-dimensional space by constructing independent sentiment polarity feature extractors and sentiment intensity feature extractors for text, audio, and video modalities, respectively, with the calculation formulas as follows: in, These represent text, audio, and video modalities, respectively. The weights and biases of the affective valence feature extractor. Weights and biases for the emotional arousal feature extractor; Activation functions are used to introduce nonlinearity, thereby improving the expressive power of the model; The feature vector representing the emotional polarity obtained through mapping. This is the feature vector representing the intensity of emotion obtained through mapping.

5. The implementation method of a multimodal sentiment prediction system based on modality optimization and sentiment dimension alignment according to claim 1, characterized in that, At the dimensional decoupling level, to ensure the independence of sentiment polarity and sentiment intensity in the semantic space, for the same modality of the same sample, its sentiment polarity feature vector and sentiment intensity feature vector are constructed as negative sample pairs between dimensions. At the modality alignment level, to ensure the consistency of expression of the same sentiment dimension across different modalities, for the same real sample, the sentiment polarity features of text, audio, and video are constructed as positive sample pairs, and the sentiment intensity features of the three are also constructed as positive sample pairs. At the supervised classification level, using real sentiment labels, for different samples within a batch, if their sentiment labels belong to the same numerical range, they are constructed as positive sample pairs; otherwise, they are constructed as negative sample pairs. The comparison loss function adopts the InfoNCE loss function form, and the calculation formula is as follows: in, For training batches, For the sample The set of all positive sample pairs at the dimensional, modal, and sample levels. For the sample The set of all negative sample pairs, For cosine similarity calculation, The temperature coefficient is used to adjust the smoothness of the similarity distribution; the loss function aims to maximize the mutual information of positive sample pairs and minimize the mutual information of negative sample pairs, thereby driving the model to automatically complete multi-dimensional feature alignment.

6. The implementation method of a multimodal sentiment prediction system based on modality optimization and sentiment dimension alignment according to claim 1, characterized in that, The proposed method employs a cross-axis interactive fusion network (C-DFN), achieving maximum feature fusion efficiency through a minimalist architecture. This network abandons complex attention calculations, instead utilizing a pure multilayer perceptron (MLP) to establish a bidirectional fusion channel: information sharing and fusion are facilitated along the "modal axis," while semantic extraction is deepened along the "feature axis." This alternating mapping mechanism ensures hierarchical fusion of text, audio, and video features during transmission. The calculation formula is as follows: in, The multimodal polarity matrix to be fused. The multimodal intensity matrix to be fused; superscript Indicates the emotional polarity dimension, superscript Indicates the dimension of emotional intensity; subscript These represent the text, audio, and video modal axis channels, respectively. The dimension of the feature vector. For multimodal feature matrices Deep interaction is performed in the Cross-Axis Interactive Fusion Network (C-DFN), and the calculation formula is as follows: in, The original multimodal feature matrix is ​​the input to the network; This is the intermediate interaction matrix after modal axis mixing; A weighted weight matrix for learning intermodal correlations; The activation function for the Gaussian error linear unit; It is the matrix transpose operator; For layer normalization processing; This is the final output matrix of cross-axis dense interaction features; This is the mapping weight matrix used for feature semantic extraction.

7. The implementation method of a multimodal sentiment prediction system based on modality optimization and sentiment dimension alignment according to claim 1, characterized in that, The method transforms the multimodal features processed by the Cross-Axis Interactive Fusion Network (C-DFN) into a final sentiment prediction vector. The calculation formula for analyzing the sentiment state of the target object in the video content is as follows: in, and These are the polarity and intensity feature matrices of the input, respectively; This represents the average pooling operation along the modal axis dimension; This represents taking the L1 norm of the global intensity vector as a scalar gain characterizing the intensity of emotion. This indicates that L2 normalization is applied to the global polarity vector, and the resulting unit vector represents the valence-efficacy direction of sentiment. It combines intensity scalars and polarity directions to be used as input to fully connected layers for final sentiment analysis; This represents a feedforward neural network used for nonlinear transformation of features. Presentation layer normalization operation; Predict labels for the model, As a tag for genuine emotions, Let N represent the loss function for the predicted values, and N represent the number of samples. The overall loss function for the model is expressed as: , The weighting coefficients are used to balance the two loss terms.

8. The implementation method of a multimodal sentiment prediction system based on modality optimization and sentiment dimension alignment according to claim 1, characterized in that, The method is a preferred embodiment of a multimodal sentiment prediction method based on modality optimization and sentiment dimension alignment, wherein optimizing the text feature extraction process includes: Pre-constructed audio emotion adapters: The audio emotion adapters consist of three independent cascaded adapter units, respectively mounted on layers 0, 11, and 23 of the backbone BERT model. Each adapter unit contains a lower projection layer, a Transformer coding layer, an upper projection layer, and a residual connection spanning the entire model, connected in sequence. Let the input of the adapter unit be... , ( The specific layer structure definition and calculation logic for each step are as follows: in, and The weights and biases of the next projection layer, This indicates a multi-head self-attention mechanism. It is a feedforward neural network. Representation layer normalization, and The weights and biases of the upper projection layer, This indicates that the final output feature of the k-th layer attached to the backbone BERT model will be used as the input of the next-level adapter unit or as the final output of the adapter. Training the audio emotion adapter: Extract audio and video modal features from the SIMS dataset, use the initialized adapter network as a feature encoder, connect a linear regression layer to the backend to predict the emotion intensity value corresponding to the sample, and define the mean squared error (MSE) loss function for training: in Labels representing the true intensity of emotion. The adapter outputs the predicted values ​​after passing through the regression layer. When the model converges, the adapter parameters are saved. In the subsequent main task, the adapter parameters are frozen and no longer updated with gradients. Using the BERT model as the backbone encoder, general semantic features of text are extracted. The BERT model includes... The layer Transformer coding structure, for the BERT's first layer... layer( The feature extraction process is represented as follows: Indicates the first The set of hidden states output by the layer, with dimension . The final output includes the global semantic feature vector from layer 23. and the hidden states of each intermediate layer. ; Using a cascaded adapter unit auxiliary path structure, connected to layers 0, 11, and 23 of the BERT backbone respectively, the adapter feature generation adopts a hierarchical recursive mechanism. The specific calculation process is as follows: The first-level adapter unit connects to BERT layer 0, and stores the hidden states of BERT layer 0. with initial text features To splice, The result is obtained after processing by the first adapter unit. : The output features of the first-level adapter unit primarily capture the correlation between the underlying lexical and basic sentiment signals. The second-level adapter unit connects to BERT layer 11, hiding the intermediate layer (layer 11) of BERT. Output of the previous level adapter unit To splice, The result is obtained after processing by the second adapter unit. : It achieves deep interaction between BERT's mid-layer syntactic semantics and preceding sentiment features; The third-level adapter unit connects to BERT layer 23, hiding the intermediate layer (layer 23) of BERT. Output of the previous level adapter unit To splice, The result is obtained after processing by the third adapter unit. That is : Final output This refers to a specific text feature vector that has been infused with the full emotional intensity of the audio and video; The general semantic features of the text extracted by the backbone encoder are fused with the audio and video enhancement features injected by the external adapter to generate the final optimized text features. The calculation formula is as follows: in, This represents a vector concatenation operation. and For the weights and biases of the fusion layer, These are the optimized text features ultimately used for sentiment prediction.

9. The implementation method of a multimodal sentiment prediction system based on modality optimization and sentiment dimension alignment according to claim 1, characterized in that, The method introduces a cross-modal attention mechanism to perform reverse optimization of audio modal features and video modal features, including: Because different modalities emphasize different aspects of emotional information representation, text modalities typically express emotional tendencies directly through explicit emotional polarity vocabulary or semantic structures, naturally including discrete and semantically clear emotional polarity information such as "happy," "sad," and "angry." In contrast, audio and video modalities implicitly reflect emotional intensity information through continuous signals such as tone changes, energy levels, speech rate, facial expressions, or body language. These modalities have relatively insufficient semantic interpretability at the emotional polarity level and are easily affected by individual speaker differences or environmental noise. Therefore, based on the optimized text features obtained after injecting emotional intensity information, a cross-modal attention mechanism is introduced to perform reverse optimization on the audio and video modal features, making the emotional expression in the audio and video modal more consistent with the emotional polarity reflected in the text. The reverse optimization includes: extracting text feature vectors representing emotional polarity information from the optimized text features as the key (K) and value (V) of the cross-modal attention, and using the audio and video modal features as queries (Q), respectively. The calibrated representations of audio and video are obtained through multi-head cross-modal attention, and the calculation formula is as follows: in Represents the initial visual or acoustic modal characteristics. This represents a multi-head attention function. This represents the intermediate features of the first stage after cross-modal attention and residual connections, at which point the sentiment intensity information of the text has been incorporated. This represents the intermediate features of the second stage after the self-attention mechanism and residual connections, which further enhances the context dependency. This represents the final optimized audio or video features generated after passing through the feedforward network and residual connections. Representation layer normalization, This represents a feedforward network.

10. The implementation method of a multimodal sentiment prediction system based on modality optimization and sentiment dimension alignment according to claim 1, characterized in that, The method described is a preferred embodiment of a multimodal sentiment prediction method based on modality optimization and sentiment dimension alignment. The two-dimensional spatial dimension decoupling of the optimized multimodal features includes: constructing independent sentiment polarity feature extractors and sentiment intensity feature extractors for text, audio, and video modalities, respectively, with the following calculation formulas: in, These represent text, audio, and video modalities, respectively. The weights and biases of the affective valence feature extractor. Weights and biases for the emotional arousal feature extractor; Activation functions are used to introduce nonlinearity, thereby improving the expressive power of the model; The feature vector representing the emotional polarity obtained through mapping. This is the feature vector representing the intensity of emotion obtained through mapping.

Images