Multi-modal sentiment analysis model based on multi-granularity features and adaptive fusion

Abstract

The invention discloses a multi-modal sentiment analysis model based on hierarchical adaptive cross-modal fusion, belongs to the field of natural language processing, and is used for solving the problems that in the prior art, multi-granularity sentiment feature extraction is insufficient, a cross-modal fusion mechanism is rigid, and the distribution difference between different-source modals is large. The method comprises the following steps: firstly, extracting features of texts, audios and visual modalities from original video data, and coding the features into advanced semantic features; secondly, multi-granularity information is fused through a hierarchical feature extractor to generate enhanced single-mode features; then, a self-adaptive cross-modal fusion network with a text as a core is adopted to realize bidirectional interaction and dynamic weighted fusion between modals; further, a dynamic contrast learning mechanism is introduced to align modal distribution in a unified potential space; and finally, inputting the optimized multi-modal features into a classifier and outputting an emotion analysis result. According to the model, through collaborative optimization of multi-granularity feature extraction, adaptive fusion and comparative learning, the accuracy of sentiment analysis and the robustness of the model are remarkably improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of natural language processing, and particularly relates to a multi-modal sentiment analysis model based on multi-granularity feature extraction and adaptive fusion mechanism. BACKGROUND

[0002] With the continuous development of natural language processing and artificial intelligence technology, sentiment analysis is gradually moving from single text analysis to comprehensive analysis of multiple information sources. Modern intelligent systems integrate text, speech, vision and other modal data, making multi-modal sentiment analysis a key technology for understanding complex human emotions. However, different modal data has significant differences in structure, granularity and semantics, and the model needs to have the ability to extract multi-level features and dynamically fuse them, which puts high demands on computing resources and model design. The existing single modal or simple fusion model has limited performance and is difficult to fully capture complementary information, so using deep learning models to realize automatic feature learning and fusion becomes a feasible solution. However, sentiment expression has subtlety, context dependence, and dynamic changes in inter-modal correlation, so how to collaboratively model different granularity features within modal and complex interactions between modal to provide accurate and robust sentiment analysis services has become a problem to be solved.

[0003] Existing researches have improved the performance of multi-modal sentiment analysis in modal interaction, feature decoupling and data enhancement through diversified fusion strategies, but most of them focus on macro fusion architecture and fail to systematically mine the inherent multi-granularity sentiment information within modal. Although multi-granularity features can help enhance single modal representation, directly using all features can increase model complexity and overfitting risk. In practical applications, the contribution of different granularity features to sentiment classification is not fixed, and the model needs to have adaptive filtering capability. However, existing methods generally lack explicit modeling of hierarchical semantic structures and adaptive aggregation mechanisms, resulting in the loss of different granularity sentiment clues. In addition, most fusion mechanisms lack dynamic adaptability and are difficult to flexibly adjust the fusion strategy according to the input content. Especially in the double-modal embedding representation after preliminary fusion, its dynamic correlation is more complex, and fixed fusion strategy often fails to effectively capture the differences between modal, causing key information to be ignored or weakened, thereby restricting the further improvement of model performance. SUMMARY

[0004] The purpose of the present application is to provide a sentiment analysis model that can fully extract multi-granularity features and realize adaptive cross-modal fusion, to solve the problems of insufficient feature extraction and rigid fusion mechanism in the prior art.

[0005] To achieve the above purpose, the present application provides a multi-modal sentiment analysis model based on multi-granularity features and adaptive fusion, the steps comprising:

[0006] Raw video data containing multiple modalities is collected, and text, audio and visual low-level modal features are extracted therefrom, and high-level semantic features in the low-level modal features are further extracted;

[0007] Hierarchical multi-granularity feature extraction and enhancement is performed on the high-level semantic features to generate enhanced single-modal features with higher discriminability;

[0008] The enhanced single-modal features are adaptively cross-modal fused to output double-modal fusion representations rich in complementary information;

[0009] Dynamic contrastive learning and distribution alignment are performed on the double-modal fusion representations to improve the robustness and discriminability of the fusion features, and then the multi-modal features are formed by splicing;

[0010] The robust multi-modal features constrained by contrastive learning are input into a classifier to output the final sentiment analysis result;

[0011] Preferably, the extracted low-level modal features include text, audio and visual modalities, wherein the high-level features of the text modality are extracted using a pre-trained language model BERT; the low-level features of the audio and visual modalities are extracted using COVAREP and Facet tools, respectively, and are respectively captured by a randomly initialized Transformer encoder to capture their temporal relationship;

[0012] Preferably, the hierarchical multi-granularity feature extraction and enhancement of the high-level semantic features specifically includes: using a parallel multi-branch convolution structure, using 1×1, 3×3 and 5×5 convolution kernels of different scales to extract feature information of different granularities from the high-level semantic features; wherein the 1×1 convolution is used for feature channel compression and adjustment, the 3×3 convolution is used to capture local fine-grained features, and the 5×5 convolution is used to model long-distance dependencies to capture global semantic information; batch normalization and ReLU activation function processing are performed on the output of each convolution branch, and the features of all branch outputs are spliced to realize preliminary fusion of multi-granularity features; a channel attention mechanism is introduced, the global statistical information of the sequence is aggregated through one-dimensional global average pooling, then the channel attention weight is generated through two fully connected layers and a Sigmoid activation function, and the weight is multiplied with the preliminary fused features by channel to adaptively enhance key features and suppress noise; finally, a 1×1 convolution layer is used to reduce the dimension of the weighted features, and the enhanced single-modal feature representation is output.

[0013] Preferably, the adaptive cross-modal fusion of the enhanced single-modal features is performed by adopting an asymmetric fusion strategy with the text modality as the core, two independent fusion paths of text-audio and text-video are constructed, and the fusion process is performed by an adaptive cross-modal fusion network, including: a cross-modal multi-head attention mechanism, the enhanced features of the text modality are projected as query vectors, the enhanced features of the non-text modality are projected as key vectors and value vectors, cross-modal attention is calculated to realize bidirectional information interaction between the modalities, a first preliminary fusion feature is generated, the query and key roles of the two modalities are exchanged, and a second fusion feature is generated;

[0014] a dynamic gating unit, a gating unit is designed, linear transformation and Sigmoid activation are performed on the two preliminary fusion features to generate dynamic weights, and element-wise multiplication is performed on the feature flow to selectively enhance discriminative features and suppress interference;

[0015] a feedforward neural network, a point-by-point feedforward network is applied to the features regulated by the gating unit, the network includes two layers of linear transformation and ReLU activation functions to further refine the feature representation;

[0016] Finally, the fusion features from different paths after processing are projected into a unified feature space through a fully connected layer to form a bimodal fusion representation.

[0017] Preferably, the dynamic contrastive learning and distribution alignment of the bimodal fusion representation are performed, specifically including: calculating the cosine similarity between the sample fusion features in a batch to construct a similarity matrix; dynamically selecting positive and negative sample pairs according to the true emotion labels of the samples; wherein the positive sample pair is a sample with the same or similar emotion label, and the negative sample pair is a sample with different emotion labels; in the training process, a difficulty perception strategy is introduced: in the early stage of training, ordinary positive and negative pairs are selected, and in the later stage of training, the difficult mode is switched to focus on the most dissimilar positive sample pairs and the most similar negative sample pairs; a supervised contrastive loss function is used to constrain the model to reduce the distance between the positive sample pairs in the feature space, while expanding the distance between the negative sample pairs, so as to improve the discriminability and robustness of the fusion features.

[0018] Preferably, the multi-modal features are input into a classifier, specifically: the text-audio fusion representation and the text-video fusion representation constrained by contrastive learning are spliced, and the spliced multi-modal features are input into a multi-layer perceptron composed of a linear layer and a ReLU activation function to predict the emotion intensity value; the total loss of model training is composed of the weighted sum of the average absolute error loss of emotion prediction and the contrastive learning loss.

[0019] Compared with the prior art, the beneficial effects of the present application are as follows:

[0020] The application fully excavates multi-granularity sentiment features from local to global through a hierarchical feature extractor, and realizes adaptive calibration of the features by using a channel attention mechanism; on this basis, an adaptive cross-modal fusion network with text as the core is used to realize dynamic weighting of inter-modal interaction through a cross-modal attention mechanism and a gating unit; and a dynamic contrast learning mechanism is further combined to align the heterogeneous modal distribution in a unified latent space. The three innovative modules work together to effectively solve the problems of insufficient feature extraction, rigid fusion mechanism and large modal distribution difference in traditional methods, and significantly improve the accuracy, robustness and generalization ability of sentiment analysis. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is the schematic diagram of the model framework of the application;

[0022] Figure 2 is the schematic diagram of the model framework of the application;

[0023] Figure 3 is the schematic diagram of the model framework of the application;

[0024] Figure 4 is the schematic diagram of the model framework of the application; DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the application will be clearly and detailedly described with reference to the drawings in the embodiments of the application. The described embodiments are only some of the embodiments of the application.

[0026] The application proposes a multi-modal sentiment analysis model based on multi-granularity features and adaptive fusion, and the main architecture of the model is as shown in Figure 2 The specific implementation process is as follows:

[0027] S1. Collect original video data containing multiple modalities, and extract low-level modal features of text, audio and vision from the original video data, and further extract high-level semantic features in the low-level modal features.

[0028] In the video collection preprocessing stage, the audio, vision and text of the same video segment are mainly processed, and each single modal sequence input is represented as m∈{t,a,v}. Where the high-level features of the text are extracted using a pre-trained language model Bert as an encoder, denoted as U t For audio and video features, this paper uses COVAREP and Facet to extract low-level features, and uses two randomly initialized Transformer encoders to extract high-level features, denoted as U β .

[0029]

[0030] Where θ Bert and θ Transformer These are parameters of a single-mode feature extractor. When extracting features, the feature extractor typically adds a special [CLS] marker. This [CLS] marker is usually placed at the beginning of the input sequence as a class label to indicate the category of the entire sequence and acts as a prediction target during training. The extracted low-level features, T... m and d m Let represent the length of the data sequence of modality m and the dimension of the eigenvector, respectively, and R indicate that the elements in the matrix belong to the set of real numbers.

[0031] S2. Perform hierarchical multi-granularity feature extraction and enhancement on the high-level semantic features to generate enhanced single-modal features with higher discriminative power.

[0032] like Figure 3 As shown, for a certain extracted modality, the low-level feature U... m In this paper, m∈{t,a,v}, a parallel multi-branch convolutional structure is used to achieve multi-granularity feature modeling. Specifically, three types of convolutional kernels, 1×1, 3×3, and 5×5, are used to extract features in parallel: 1×1 convolution is mainly used for feature channel compression and adjustment, which can reduce computational complexity while maintaining spatial resolution; 3×3 convolution utilizes its local receptive field to capture fine-grained spatial features, which helps to distinguish subtle differences in emotional expression; 5×5 convolution has a larger receptive field, which can model long-distance dependencies and is suitable for capturing global semantic information. In terms of kernel size selection, 5×5 convolution achieves a good balance between receptive field size and computational efficiency, while excessively large convolutional kernels are not only prone to introducing noise, but also significantly increase the computational burden. Batch normalization and ReLU activation functions are applied after each convolutional layer to promote training stability and enhance nonlinear representation capabilities. Finally, the fusion of multi-granularity features is achieved through concatenation, providing a robust feature foundation for subsequent cross-modal interactions. This process can be formally represented as:

[0033] F1 = ReLU(BN(Conv1D) 1×1 (U m )))

[0034] F3 = ReLU(BN(Conv1D) 3×3 (U m )))

[0035] F5 = ReLU(BN(Conv1D) 5×5 (U m )))

[0036] F = Concat(F1, F3, F5)

[0037] where BN denotes a batch normalization operation, which computes the mean and variance of all samples within the current mini-batch for each feature channel, and then normalizes the feature.

[0038] In this paper, we introduce a channel attention mechanism to dynamically adjust the weights of different feature channels, so as to enhance the hierarchical features that are crucial for sentiment discrimination. First, we use a one-dimensional global average pooling to aggregate the sequence features of each channel into a global feature, thereby preserving the global statistical information of each feature channel while eliminating the influence of variable sequence length. The calculation process is shown in the following formula, where T m represents the length of the data sequence of modality m, c = 1, 2,..., C, and C is the total number of channels.

[0039]

[0040] Next, we use two fully connected layers to process these feature dimensions. The first fully connected layer is used to reduce the dimension, and the second is used to restore the original feature dimension and generate attention weights A. Then we multiply the integrated feature map F with the attention weights A channel by channel to obtain the weighted feature map F', which enhances important features and suppresses noise through channel-wise multiplication. As shown in equations (8-9), where W fc1 ,W fc2 are the weights of the two fully connected layers, δ() is the sigmoid activation function, and ⊙ represents channel-wise multiplication.

[0041] A = δ(W fc2 · ReLU(W fc1 · F avg ))

[0042] F' = F ⊙ A

[0043] Finally, we use a 1 × 1 convolutional layer to reduce the dimension of the final feature map F' to obtain the final enhanced single-modality feature representation U m ', to meet the requirements of subsequent sentiment regression tasks. As shown in equation (10):

[0044] U m ' = Conv1D 1×1 (F')

[0045] S3. Perform adaptive cross-modality fusion on the enhanced single-modality features to output a double-modality fusion representation rich in complementary information.

[0046] As Figure 4As shown, the application adopts an asymmetric fusion strategy with text modality as the core, respectively constructs text-audio and text-visual two independent fusion paths, and does not explicitly perform direct fusion between audio-visual modalities. This design is based on the following considerations: first, the text modality usually carries the most direct and explicit emotional semantic information, which is the strongest clue in emotional analysis; second, the alignment of audio and visual signals in continuous time often has noise and uncertainty, and direct forced fusion may introduce irrelevant or even interfering correlation. By taking text as the shared hub, the model can more effectively extract complementary information in non-text modalities that is highly related to semantics. The module mainly includes three parts: cross-modal multi-head attention mechanism, gating unit and feedforward neural network, and each part embeds residual connection and layer normalization. The module introduces cross-modal multi-head attention mechanism as an extension of traditional self-attention. CM-MHA adopts cross-modal interaction, taking the features of one modality as query and the features of the other modality as key and value, to realize dynamic feature fusion between modalities. Specifically, the application designs two parallel cross-modal attention modules, which alternately assign the query and key-value roles between the two modalities, thereby realizing bidirectional interaction.

[0047] First, project the high-level features U t from the text modality to obtain the query vector Next, project the high-level features U α from the non-text modality to obtain the key vector and the value vector The calculation process is shown below, where are the mapping matrices corresponding to the query, key and value of the corresponding modality, and i represents the attention score calculation of the i-th attention head.

[0048]

[0049] Next, calculate the cross-modal attention value between the modalities as shown below. In this way, meaningful information of the text modality can be effectively integrated into the non-text modality, thereby realizing cross-modal fusion:

[0050]

[0051] Parallelly use k different mapping matrices to project the features U α and U t to obtain k groups of different query, key and value vectors where i = 1, 2,..., k. Next, perform cross-modal attention operation on each group in parallel to generate k different attention heads. Finally, concatenate these attention heads to form the first fusion feature output M tα between the two modalities.

[0052]

[0053] The essence of the cross-modal multi-head attention mechanism is to map the hidden state of one modality into the space of another modality. In this process, the two modalities exchange as query and key input into the cross-modal attention block, and the same calculation method obtains the second fusion feature M αt In this way, the information of the two modalities is fully fused to adapt to the subsequent sentiment analysis task.

[0054] At the same time, in order to selectively enhance discriminative features and suppress noise interference in cross-modal fusion, the fusion network also introduces dynamic gating units. Specifically, two gating units g tα and g αt are designed to filter out interference factors while retaining important information.

[0055] g ta =δ(Linear(M tα ))

[0056] g αt =δ(Linear(M αt ))

[0057] where δ represents the sigmoid function. These gating units linearly transform the weights of the fusion features through the sigmoid activation function, and then regulate the data flow between the multi-head attention mechanism and the feedforward network. Where ⊙ represents element-wise multiplication.

[0058]

[0059] Next, through a layer of point-by-point feedforward neural network, the feature vector at each position is independently applied to two layers of linear transformation and ReLU activation function, further refining the encoded feature representation. This step helps to convert complex cross-modal information into high-quality features suitable for sentiment analysis tasks.

[0060]

[0061] Finally, in order to project all features into the same feature space and facilitate subsequent contrastive learning, this paper processes all fusion features through a fully connected network, thereby embedding these features into a unified feature space to adapt to the classification or regression tasks of sentiment analysis.

[0062]

[0063] S4. Dynamic contrastive learning and distribution alignment are performed on the dual modality fusion representation to improve the robustness and discriminability of the fusion features, and then the multi-modal features are formed by concatenation.

[0064] The module is based on the fused sample features, calculates the cosine similarity between samples, and dynamically generates positive and negative sample pairs. Through supervised contrastive loss, the model is guided to reduce the representation distance of similar emotion samples and expand the representation difference of different emotion samples. The training process of this part is jointly optimized with the main task loss. Under the premise of keeping the overall classification structure unchanged, the model's ability to distinguish complex emotions is effectively improved.

[0065] For the incoming sample features, first calculate the cosine similarity scores between them, and construct the cosine similarity matrix S and its corresponding ranking matrix R according to the similarity scores. Then, traverse the ranking matrix and calculate the distance between the sample labels. When the label distance meets the set threshold d, it is determined as a positive pair (representing similar emotions), otherwise it is considered as a negative pair (representing different emotions), where represents the normalized feature, and τ is the temperature coefficient, which is used to scale the similarity score to adjust the sharpness of the softmax function distribution, thereby affecting the separability of positive and negative sample pairs in the contrastive learning process.

[0066]

[0067] In dynamic selection, this paper introduces a "hard mode" for the difficulty of sample pairs. In the early stage of training, the normal positive and negative pair selection strategy is adopted to cover more sample pairs, helping the model to quickly converge and adapt to the overall distribution of the sentiment analysis task. As the training progresses, to improve the generalization ability of the model, in the later stage of training, adjust to "hard mode", only select the most difficult positive pair (the most dissimilar positive sample pair, or the boundary pair between neutral samples and other emotion samples) and the most difficult negative pair (the most similar different emotion class sample pair) to participate in learning. This strategy further optimizes the model performance by dynamically focusing on difficult-to-distinguish emotion boundary samples, improving the model's ability to recognize complex emotion boundaries. The specific supervised contrastive loss function calculation is as follows.

[0068]

[0069] where the positive and negative pair set and are dynamically generated, B represents the batch size, r, j, and k are the indices representing the anchor, positive sample, and negative sample, respectively. The dynamic selection strategy significantly improves the model's ability to distinguish positive and negative samples, especially in the classification of similar emotion categories, improving task performance and enhancing the robustness of the model.

[0070] S5. Input the robust multi-modal features constrained by contrastive learning to the classifier to output the final sentiment analysis results.

[0071] The present application adopts a multi-layer perceptron (MLP) to predict the emotion intensity. The network is composed of two linear transformation layers and a ReLU nonlinear activation function, which can effectively extract the deep relationship between the input features. At the input end, this paper will fuse the two vectors U tv and U ta extracted from the multi-modal information, the fusion method is element-wise addition, denoted as symbol ⊕. In this way, the information of the two modalities can be fully combined while keeping the dimension unchanged. Then, the fused representation is input into the multi-layer perceptron to obtain the emotion intensity prediction value The calculation process is as follows. Get the sentiment analysis score

[0072]

[0073] Next, define the error between the prediction value and the true emotion label, use the mean absolute error (MAE) as the sentiment regression loss, and train and optimize the model:

[0074]

[0075] B represents the batch size, x represents the index of the sample in the batch, represents the xth emotion prediction value, y x represents the xth true value. The overall loss function during training is composed of sentiment regression loss and contrastive learning loss, mainly through minimizing the total loss to optimize the model, as shown in the following formula, where ρ is a hyperparameter that adjusts the relative weight of the two loss terms.

[0076]

[0077] The above embodiments are only specific implementations of the present application, which are used to illustrate the technical solutions of the present application but not to limit them. Those skilled in the art should understand that various modifications, changes or equivalent replacements can be made to the above embodiments without departing from the principles and spirits disclosed by the present application. The protection scope of the present application should be defined by the scope of the claims and their equivalents.

Claims

1. A multi-modal sentiment analysis model based on hierarchical adaptive cross-modal fusion, characterized in that, The method comprises the following steps: 101, collecting original video data containing multiple modalities, and extracting low-level modal features of text, audio and vision therefrom; 102, encoding the low-level modal features extracted in step 101 to extract high-level semantic features thereof; 103, inputting the high-level semantic features obtained in step 102 into a hierarchical feature extractor to perform multi-granularity feature extraction and enhancement, and generating enhanced single-modal features; 104, inputting the enhanced single-modal features obtained in step 103 into an adaptive cross-modal fusion network to perform cross-modal interaction and fusion, and outputting double-modal fusion representations rich in complementary information; 105, performing dynamic contrastive learning and distribution alignment on the double-modal fusion representations obtained in step 104 to improve the robustness and discriminability of the fusion features; 106, inputting the robust multi-modal features subjected to contrastive learning constraint in step 105 into a classifier to output a final sentiment analysis result.

2. The method of claim 1, wherein, The extraction of low-level modal features and the encoding of high-level semantic features in steps 101 and 102 are as follows: 201, the high-level features of the text modality are extracted using a pre-trained language model BERT, and the calculation process is as follows: 202, the low-level features of the audio and visual modalities are extracted using COVAREP and Facet tools respectively, and then a randomly initialized Transformer encoder is used to capture the temporal relationship to generate high-level features, and the calculation process is as follows: wherein is the original modality data, where θ Bert and θ Transformer are parameters of the corresponding feature extractor, and {t, a, v} represents the modality.

3. The method of claim 1, wherein, The construction of the hierarchical feature extractor in step 103 specifically comprises: 301, a parallel multi-branch convolution structure is adopted, and 1×1, 3×3 and 5×5 convolution kernels are used to extract feature information of different granularities from the input features in parallel, and batch normalization and ReLU activation functions are connected after each convolution layer; F1 = ReLU(BN(Conv1D 11 (U m ))) F3 = ReLU(BN(Conv1D 3×3 (U m ))) F5 = ReLU(BN(Conv1D 5×5 (U m ))) where BN represents the batch normalization operation, and for each feature channel, the mean and variance of all samples in the current mini-batch are calculated, and then normalized. 302, the features output by the three convolution branches in step 301 are spliced to realize the fusion of multi-granularity features; F=Concat(F1,F3,F5) 303, a channel attention mechanism is introduced, and the fused features after splicing in step 302 are used to generate channel attention weights through global average pooling and a fully connected layer for adaptive calibration; the calculation process of the global average pooling is as follows: where T m represents the length of the data sequence of the modality m, c = 1, 2,..., C, C is the total number of channels. The calculation method of the channel attention weight is as follows: A = δ(W fc2 • ReLU(W fc1 • F avg )) where F avg is the feature after global average pooling, W fc1 is the parameter of the fully connected layer, and δ(·) is the Sigmoid function. fc2 is the parameter of the fully connected layer, and δ(·) is the Sigmoid function. 304, multiply the attention weight A and the spliced features F channel by channel, and the calculation process is as follows: F'=F⊙A; 305, using a 1×1 convolution layer to reduce the dimension of the features F' obtained in step 204, and output the final enhanced single-modal features.

4. The method of claim 1, wherein, The construction of the adaptive cross-modal fusion network in step 104 specifically comprises: 401, an asymmetric fusion strategy with the text modality as the core is adopted, and two independent fusion paths of text-audio and text-vision are constructed; 402、Performing cross-modal bidirectional interaction and generating preliminary fusion features: In each fusion path, through the cross-modal multi-head attention mechanism, bidirectional feature interaction between the two modalities is realized, and two preliminary fusion features are generated, mainly including two steps: For text modality features U t and non-text modality features U α , a e {a, v} : 1) generate first preliminary fusion feature M tα : project text feature U t ' into query vector, non-text feature U α ' into key and value vectors, compute cross-modal attention, and concatenate after k parallel heads to form The calculation process is as follows: wherein are the mapping matrices corresponding to the modal query, key and value respectively, i denotes the i-th attention head to compute the attention score, where i = 1, 2,..., k, and there are k attention heads in total. 2) generating a second preliminary fusion feature M αt : projecting the non-text feature U α ' into the query vector, the text feature U t ' into the key and value vectors, computing the cross-modal attention, and concatenating after parallel k heads to form 403、Introducing a gating unit, designing two independent gating units, and modulating the two preliminary fusion features obtained in step 402 respectively, the calculation process is as follows, where δ() represents the sigmoid function, and represents element-wise multiplication; g tα = δ(Linear(M tα )) g αt = δ(Linear(M αt )) 404、The features modulated in step 403 are input into a feedforward neural network for nonlinear transformation, and are connected with layer normalization through residual connection to stabilize the training process, the calculation process is as follows: 405、project the features processed in step 404 to a unified feature space to form the dual-modal fusion representation, the calculation process is shown as follows, where θ FC is the parameter of the multi-layer perception.

5. The method of claim 1, wherein, The specific implementation of the dynamic contrast learning and distribution alignment in step 105 includes: 501、computing a dual-modality fused representation of the sample within the batch with a cosine similarity between the two, as follows: wherein represents the normalized features, and τ is a temperature coefficient used to scale the similarity score; 502、According to the true emotion label of the sample, dynamically select positive and negative sample pairs; wherein the positive sample pair is the sample with the same or similar emotion label, and the negative sample pair is the sample with different emotion label; 503、Use a supervised contrast loss function to optimize the feature representation, and its calculation formula is: where the positive-negative pair set and is dynamically generated, B denotes the batch size, and r, j, and k are indices representing the anchor point, positive sample, and negative sample, respectively.

6. The method of claim 5, wherein, When dynamically selecting positive and negative sample pairs in step 502, a difficulty perception strategy is implemented: in the early stage of training, a normal positive and negative pair selection strategy is used, and in the later stage of training, a difficult mode is switched to, only the most dissimilar positive sample pair and the most similar negative sample pair are selected to participate in contrast learning.

7. The method of claim 1, wherein, The construction of the classifier and the output of the sentiment analysis result in step 106 specifically includes: 701、text-visual fusion representation constrained by contrastive learning and text-audio fusion representation fusion is performed in an element-wise addition manner, and the fused features are input into a multilayer perceptron for classification prediction to obtain a sentiment prediction value where θ MLP are parameters of a multi-layer perceptron; 703、Use the mean absolute error as the sentiment regression loss function: where B represents batch size, x indicates the index of the sample in the batch, represents the xth sentiment prediction value, y x represents the xth true value; 704、Construct the overall loss function to optimize the model: where is the contrastive learning loss, and p is a hyperparameter that regulates the relative weights of the two loss terms.

Images