Multi-modal sentiment analysis method combining pre-training model and self-attention block

By combining the multimodal sentiment analysis method of pre-trained models and self-attention blocks, the problems of weak semantic correlation and data misalignment between multimodals are solved, and more efficient feature fusion is achieved, which significantly improves the accuracy of sentiment analysis. , provides more effective support for business and policy decisions.

CN118898046APending Publication Date: 2024-11-05GUANGDONG UNIV OF TECH
View PDF 0 Cites 41 Cited by

Patent Information

Application Number
CN202410921183.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-10
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

In the existing multimodal sentiment analysis technology, weak semantic correlations between multimodals and misalignment of data lead to inability to effectively fusion, resulting in insufficient accuracy of sentiment analysis.

Method used

Using a multimodal sentiment analysis method combining pre-trained models and self-attention blocks, deep features of text, images and audio are extracted through BERT, ResNet and COVAREP models, and the self-attention mechanism is used to optimize feature fusion to enhance the modes. interaction and integration capabilities.

Benefits of technology

It significantly improves the accuracy of multimodal sentiment analysis, and the recognition accuracy in text, image and audio modes has been improved by 7.48%, 6.92% and 1.24%, respectively, providing more effective multimodal sentiment analysis tools to support Business and policy decisions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118898046A_ABST
    Figure CN118898046A_ABST
Patent Text Reader

Abstract

The invention provides a multi-modal sentiment analysis method combining a pre-training model and a self-attention block, aiming at the problems of weak semantic correlation and data misalignment among existing multi-modal data, the method is based on BERT, ResNet and COVAREP models, the three models are improved respectively, and finally a multi-modal model BRCM is constructed. And key features of texts, images and audios are effectively extracted and integrated. Specifically, the multi-modal model BRCM fuses feature representations in a text processing module, an image processing module and an audio processing module, deep representations of three modals can be extracted and expressed at the same time, the fusion process of the features is optimized in combination with a self-attention mechanism, the interaction and integration ability between the modals is enhanced, and more accurate emotion judgment is achieved. The emotion recognition accuracy of the system in different modes is improved by 7.48%, 6.92% and 1.24% respectively. The method provides an effective support tool for policy decision.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention involves building a multimodal model BRCM based on the BERT, ResNet and COVAREP infrastructure, and proposes a multimodal sentiment analysis method combining a pre-trained model with a self-attention block. The self-attention mechanism is used to optimize and fuse the semantic information of the text, the visual features of the image and the audio features to solve the problem of weak semantic correlation between multimodal elements and ineffective fusion caused by data misalignment in the field of multimodal sentiment analysis. Background Art

[0002] With the rapid development of the Internet, a huge amount of text and image data has been generated on the network, which contains rich emotional information. Sentiment analysis is a discipline that applies natural language processing and text mining technology. Its goal is to deeply identify, carefully analyze and comprehensively summarize text content with emotional colors. By extracting subjective information and emotional tendencies from text data, sentiment analysis helps people understand the emotional attitudes behind the text, and plays an important role in many fields such as product reviews, social media monitoring and political public opinion analysis. Using this technology, analysts can accurately grasp consumers' purchasing tendencies, changes in public sentiment and views on specific events, and further guide marketing strategies and product design optimization.

[0003] Although sentiment analysis initially focused on text data, with the development of technology, it has now been widely used to predict movie box office, stock market trends and even election results. Wu et al. established detailed text sentiment annotation criteria based on the OCC sentiment classification model, and used convolutional neural network (CNN) to perform sentiment classification, significantly improving the accuracy and efficiency of analysis. Zhang et al. constructed an extended sentiment dictionary and combined it with a machine learning model to achieve complex sentiment classification of text, providing diversified analysis tools and methods.

[0004] However, relying solely on text data often fails to capture the complexity of human emotions. For example, the word "excellent" is usually interpreted as a positive evaluation without context, but if it is combined with exaggerated or sarcastic expressions, its emotional color may be completely opposite. Therefore, multimodal sentiment analysis technology came into being, integrating different data types such as text, images, and voice to interpret emotions more comprehensively and accurately. According to a 2015 research report, multimodal systems are always better than single-modal systems in terms of the accuracy of sentiment analysis, verifying the advantages of multimodal analysis in processing complex emotional expressions. Majumder et al. proposed a contextual high-dimensional fusion model, which obtained rich emotional features by multi-level fusion of text, sound and image data. However, the application of attention mechanism shows that the key information that determines emotional expression is usually concentrated in specific words of the text or specific areas of the image and audio, and the fusion model may introduce noise, resulting in information excess. To address this problem, Huang et al. used the attention mechanism to deeply model the text and image data respectively, screened out key features for fusion, and performed sentiment prediction through the fully connected layer, effectively reducing information redundancy.

[0005] In view of this, this paper introduces a new multimodal sentiment classification method based on the improved BERT model proposed by Yu et al. on the basis of natural language processing and computer vision technology. This method not only improves the accuracy of sentiment analysis, but also opens up a new direction for the application of sentiment analysis technology, and is expected to show its wide application potential in multiple fields. Summary of the invention

[0006] In view of the limitations and challenges of existing solutions, the present invention proposes a multimodal sentiment analysis method combining a pre-trained model with a self-attention block. In view of the problem that the existing semantic correlation between multimodal elements is weak and the multimodal data is not aligned, the method is based on the BERT, ResNet, and COVAREP models, and the three models are improved respectively. Finally, a multimodal model BRCM is constructed to effectively extract and integrate the key features of text, image, and audio. Specifically, the multimodal model BRCM integrates the feature representations in the text processing module, the image processing module, and the audio processing module, and can simultaneously extract and express the deep representations of the three modalities. The self-attention mechanism is combined to optimize the feature fusion process, enhance the interaction and integration capabilities between modalities, and achieve more accurate sentiment judgment. The system has improved the sentiment recognition accuracy of different modalities by 7.48%, 6.92%, and 1.24%, respectively. This method not only provides new ideas and methods for the field of multimodal sentiment analysis, but also provides more effective support tools for business and policy decisions.

[0007] A multimodal sentiment analysis method combining a pre-trained model and a self-attention block of the present invention comprises the following steps:

[0008] S1. The present invention constructs a multimodal model BRCM, which improves the feature extraction capability of multimodal data by fusing the features of three modalities: text, image and audio. The model combines text processing, image processing and audio processing modules, and generates the final multimodal feature representation through preprocessing layer, feature extraction layer, attention mechanism and fully connected layer. The text processing module is based on BERT, combined with bidirectional GRU and multi-head attention mechanism to extract deep semantic features. The image processing module uses convolutional neural network, global average pooling and multi-head attention mechanism on the basis of ResNet to extract high-level image features. The audio processing module is based on COVAREP, combined with convolutional neural network, LSTM and multi-head attention mechanism to extract deep audio features. Finally, the feature representations of text, image and audio are fused to generate multimodal feature representation. Through this fusion, the BRCM model can simultaneously extract and express the deep features of the three modalities, and realize more comprehensive multimodal data analysis and processing.

[0009] Text processing module: In order to enhance the ability to extract text features, improvements are made based on BERT, using a method that combines bidirectional GRU and multi-head attention mechanisms. The new architecture combines BERT's contextual understanding ability, BiGRU's sequence modeling ability, and the multi-head attention mechanism's global information capture ability, which can better extract the deep semantic features of text;

[0010] Input embedding layer: embeds the input text and converts each word into a corresponding vector representation. These vectors can capture the semantic information of the words. Assume that the input text is T = [t1, t2, ..., t n ], and the representation after the embedding layer is:

[0011] E t =[e1,e2,...,e n ]

[0012] Among them, e i Representation word t i The embedding vector of

[0013] Pre-trained BERT layer: The embedding representation is fed into a pre-trained BERT model to obtain context-sensitive word representations. These vectors can capture the meaning of words in different contexts:

[0014] H BERT =BERT(E t )

[0015] Among them, the output of BERT is H BERT, H BERT =[h1,h2,...,h n ], each h i It is the context-sensitive word vector output by the BERT model;

[0016] Bidirectional GRU layer: The context-sensitive word vector output by BERT is input into the bidirectional GRU layer to capture the bidirectional dependencies in the sequence. The forward and backward dependencies of the word vectors are processed separately. The forward GRU layer processes the forward dependency in the sequence, and the backward GRU layer processes the backward dependency in the sequence.

[0017] Assume that the output of the forward GRU is The output of the backward GRU is

[0018] The final GRU output is expressed as:

[0019]

[0020] in, is the output of the bidirectional GRU;

[0021] Multi-head attention layer: The output of the bidirectional GRU is input into the multi-head attention mechanism to capture global contextual information. Multiple attention heads process information in parallel, and then combine this information to generate the output of the attention mechanism:

[0022] H Attn =MultiHeadAttention(H GRU ,H GRU ,H GRU )

[0023] in, is the output of the multi-head attention mechanism;

[0024] Fully connected layer: The output of the multi-head attention mechanism is averaged and pooled to reduce the dimension of the feature. Assume that the feature after pooling is represented as H pool :

[0025]

[0026] The pooled features are input into the fully connected layer, and the final text feature representation is generated through the ReLU activation function:

[0027] H final =ReLU(W f ·H pool +b f )

[0028] Among them, W f and b fare the weights and bias parameters of the fully connected layer, H final is the final text feature representation;

[0029] Image processing module: In order to enhance the image feature extraction capability, the new architecture is improved on the basis of ResNet, adopting a deeper convolutional neural network structure and combining global average pooling and attention mechanism to improve the accuracy of feature expression; the new architecture will include the following main parts: preprocessing layer, ResNet layer, global average pooling layer, multi-head attention layer and fully connected layer;

[0030] Preprocessing layer: standardize the input image, including adjusting the image size, normalization, etc.; assuming that the input image is I, the preprocessed image is represented as I', and the formula is:

[0031] I′=Normalize(Resize(I))

[0032] Among them, Resize means adjusting the image size, and Normalize means normalization processing;

[0033] ResNet layer: The preprocessed image is input into the pre-trained ResNet model to extract high-level image features; based on the ResNet-50 model, the ResNet formula is expressed as:

[0034] H ResNet =ResNet(I′)

[0035] Among them, H ResNet Represents the feature map output by ResNet;

[0036] Global average pooling layer: Perform global average pooling on the feature map output by ResNet to compress the high-dimensional feature map into a feature vector of fixed size. The formula is:

[0037] H GAP =GlobalAveragePooling(H ResNet )

[0038] Among them, H GAP is the feature vector after global average pooling;

[0039] Multi-head attention layer: The feature vector after global average pooling is input into the multi-head attention mechanism to capture global contextual information:

[0040] H Attn =MultiHeadAttention(H GAP ,H GAP ,H GAP )

[0041] Among them, H Attn It is the output feature representation of the multi-head attention mechanism;

[0042] Fully connected layer: The output of the multi-head attention layer is reduced in dimension through the fully connected layer to obtain the final image feature representation:

[0043] H final =ReLU(W f ·H Attn +b f )

[0044] Among them, W f and b f are the weights and bias parameters of the fully connected layer, H final is the final image feature representation;

[0045] Audio processing module: In order to enhance the audio feature extraction capability, improvements are made on the basis of COVAREP, using a more complex convolutional neural network (CNN) structure combined with a long short-term memory (LSTM) network, and using a multi-head attention mechanism to improve the accuracy of feature expression; the new architecture will include the following main parts: preprocessing layer, COVAREP feature extraction layer, convolution layer, LSTM layer, multi-head attention layer, and fully connected layer;

[0046] Preprocessing layer: standardize the input audio signal, including noise reduction and normalization. Assuming the input audio is A, the preprocessed audio is represented as A':

[0047] A′=Normalize(Denoise(A))

[0048] Among them, Denoise means noise reduction processing, Normalize means normalization processing;

[0049] COVAREP feature extraction layer: The preprocessed audio signal is input into the COVAREP model to extract the basic features of the audio:

[0050] H COVAREP =COVAREP(A′)

[0051] Among them, H COVAREP Represents the audio features output by COVAREP;

[0052] Convolutional layer: The features output by COVAREP are input into the convolutional layer to extract higher-level feature representations:

[0053] H Conv =Conv1D(H COVAREP )

[0054] Among them, H Conv It is the feature representation of the convolutional layer output;

[0055] LSTM layer: The output of the convolutional layer is input into the LSTM layer to capture the long-term and short-term dependencies in the time series:

[0056] H LSTM =LSTM(H Conv )

[0057] Among them, H LSTM Represents the feature representation of LSTM output;

[0058] Multi-head attention layer: The output of LSTM is input into the multi-head attention mechanism to capture global contextual information:

[0059] H Attn =MultiHeadAttention(H LSTM ,H LSTM ,H LSTM )

[0060] Among them, H Attn It is the output feature representation of the multi-head attention mechanism;

[0061] Fully connected layer: The output of the multi-head attention layer is reduced in dimension through the fully connected layer to obtain the final audio feature representation:

[0062] H final =ReLU(W f ·H Attn +b f )

[0063] Among them, W f and b f are the weights and bias parameters of the fully connected layer, H final is the final audio feature representation;

[0064] Constructing the BRCM model: The BRCM model improves the feature extraction capability of multimodal data by integrating the features of three different modalities: text, image, and audio. The model combines text processing, image processing, and audio processing modules, and integrates the feature representations of the above three different modalities through the preprocessing layer, feature extraction layer, attention mechanism, and fully connected layer to generate the final multimodal feature representation.

[0065] Assume that the feature representations of text, image and audio are The final multimodal feature is represented as H BRCM , expressed as:

[0066]

[0067] Among them, Concat represents the concatenation operation of features;

[0068] Through this fusion, the BRCM model can simultaneously extract and express the deep features of text, images, and audio, achieving more comprehensive multimodal data analysis and processing;

[0069] Finally, a multimodal model BRCM was constructed;

[0070] S2: The present invention proposes a method that combines attention mechanism and gating mechanism to effectively fuse multimodal information. Multimodal sentiment analysis uses text, visual and acoustic information from the same video clip, and generates text-driven multimodal representation by integrating this information to accurately predict sentiment. Text features are extracted through the BERT model, visual features are extracted through the ResNet model, and acoustic features are extracted through the COVAREP model. These features are processed through a one-dimensional convolutional network to capture temporal characteristics and project them into a fixed dimension. In order to enhance feature processing capabilities, a text-oriented cross-attention module is introduced, which includes a cross-attention block and a self-attention block, and information fusion and redundant filtering are performed between the two through a gating mechanism. The self-attention mechanism allows the model to dynamically assign attention weights according to the input data. When processing different text, image and audio inputs, the model can automatically adjust the attention to each modality to adapt to sentiment analysis in different contexts. analysis needs; through the self-attention mechanism, the model can more deeply explore the emotional information in text and images; the self-attention mechanism also helps the model capture contextual information in multimodal data; by calculating the attention weights at different positions, the model can understand the relationship between elements in text and images, so as to better understand the overall emotional expression; the features are sent to the feedforward network for processing after layer normalization, forming a residual connection to retain important original information; the output of the module is formed by connecting the outputs of each layer to form the final feature representation, which is further input into the multi-layer perceptron to predict the emotional label; in order to deeply understand the emotional semantics of each modality, a single-modal joint learning module is introduced, using an encoder with shared weights to process single-modal features and predict through a multi-layer perceptron; the optimization target is based on L1 loss, combining the loss of multimodal prediction with the loss of single-modal prediction, and balancing the two losses by adjusting the hyperparameter λ to improve the prediction accuracy and generalization ability of the model;

[0071] S3: The present invention proposes a multimodal method for sentiment analysis, which is suitable for processing video content containing visual, textual and audio modalities, and can work effectively even when the audio modality is missing; the method converts visual and textual modalities into modality-specific features through a Transformer-based encoder; in the absence of audio features, the audio features are reconstructed using visual and textual signals through a knowledge transfer network, which includes multiple Transformer modules to ensure that the reconstructed audio features can accurately represent the original audio information; in order to verify the quality of the reconstructed audio, a consistency loss function is used, which optimizes the model by minimizing the Euclidean distance between the reconstructed audio features and the real audio features; the audio features reconstructed by vision and text are combined and further encoded through another set of Transformer modules to enhance the expressive power of the audio information; The interaction between features of different modalities is handled through the cross-modal attention mechanism, which maps and calculates the relationship between different modalities through a weight matrix, and effectively integrates information through queries, keys and values ​​in the latent space; through residual connections, the features obtained through cross-modal attention calculation are combined with the original target features to form the final feature representation, which not only retains the key original information but also enhances the expressiveness of the features; these features are then used for sentiment analysis prediction tasks, where the decoding process is completed by a group of self-attention Transformers to ensure full decoding and integration of information from each modality; the model loss function combines the standard cross-entropy loss with the consistency loss of visual and textual modalities to optimize the overall model performance and ensure a high degree of consistency in the reconstructed audio; this design not only improves the accuracy of the model, but also enhances its robustness in the absence of data from a certain modality.

[0072] According to a specific implementation of the embodiment of the present invention, the specific steps of S2 are:

[0073] S2. Multimodal sentiment analysis: Using text t, visual v, and acoustic a information from the same video clip as input, represented as each modality m∈{t,v,a} Among them, T m and d m Respectively represent the sequence length and feature dimension of mode m; the present invention integrates the information in these emotion-related unaligned multimodal sequences; generates a text-driven multimodal representation to accurately predict the results of sentiment analysis;

[0074] According to claim 1, text features, visual features and audio features are extracted by a multimodal model BRCM, a one-dimensional convolutional network is applied to process these features to capture temporal characteristics, and these features are projected into a fixed dimension to be represented as where m∈{t,v,a};

[0075]

[0076] Among them, F a It is the processed acoustic features used for subsequent modal fusion and sentiment analysis tasks; are the original acoustic features extracted by the multimodal model BRCM; 1Da is a one-dimensional convolutional network applied to the acoustic features, which is used to process these features to capture temporal dependencies and normalize the feature dimensions; parameters Represents the parameters of the one-dimensional convolutional network;

[0077]

[0078] Among them, F v It is the visual feature after one-dimensional convolution processing; is the image feature extracted by the multimodal model BRCM; 1D v It is a one-dimensional convolutional network specifically designed to process visual features, with the goal of extracting time series information and performing dimensionality standardization; the parameters Represents the parameters of this network;

[0079]

[0080] in, It is the text feature extracted by the multimodal model BRCM, including sentence and document level embedding, which aims to process text data to capture possible sequence dependencies and perform appropriate dimensionality transformation; parameter Represents the parameters of the one-dimensional convolutional network, and the processed result F t Represents text features optimized for sentiment analysis tasks;

[0081] Text-oriented cross-attention module: This module contains a cross-attention block for text-oriented feature fusion and a self-attention block for refining text features. A gating mechanism is added to the module to effectively integrate the useful information between the two attention blocks and filter out redundancy. The cross-attention block receives and As input, where m∈{a,v}, the superscript i indicates the i-th layer;

[0082] Perform layer normalization on features: and

[0083] The normalized features are fed into the cross-attention module:

[0084]

[0085] in, W * represents a learnable weight matrix by which corresponding queries, keys, and values ​​can be obtained;

[0086] The self-attention block will As input, it produces

[0087]

[0088] Among them, W * Represents the learnable weight matrix associated with the query, key, and value of the text modality;

[0089] In order to mitigate the noise and redundant interference caused by feature fusion, two gating mechanisms are introduced; these gates adaptively adjust and The transfer ratio of , thereby optimizing the feature representation of the final prediction;

[0090] Memory Gate m is responsible for determining the proportion of components to be transferred in the target modality, while the fusion gate g f Determines what proportion of the fusion component to inject into the target modality; these gating signals are generated from the two modality sequence representations in the same layer and are specifically expressed as:

[0091]

[0092] in, is a dedicated projection matrix. The subscripts m and f are used to distinguish the two gates. σ represents the Sigmoid nonlinear function. Represents the connection of vectors;

[0093] Through these gating mechanisms, the output feature expression is:

[0094]

[0095] Where ⊙ represents element-by-element multiplication;

[0096] This operation can effectively filter out the noise information and redundant features generated by cross-modal interaction, while coordinating the fusion ratio of the two modalities;

[0097] In the processing mechanism, following the feedforward structure of Transformer, and Perform position-level feedforward network processing, including residual connections; for features from modality m to text t First, after layer normalization, it is input into the feedforward network for processing, and the original features before the input are added to form a residual connection:

[0098]

[0099] Among them, FFN(·) represents feed-forward network, LN(·) represents layer normalization;

[0100]

[0101] Among them, FFN(·) represents feed-forward network, LN(·) represents layer normalization;

[0102] This processing method can enhance the model's nonlinear expression ability of features while retaining important original information;

[0103] By connecting the outputs of the last layer of each module, the final feature representation can be calculated;

[0104] Each module consists of two heads, and a total of four heads are used for feature fusion;

[0105] Combining the outputs of the four heads of the two text-oriented cross-modal attention modules, the final feature representation F final It can be expressed as:

[0106]

[0107] in, Represents the connection of vectors;

[0108] The fused feature F final Input into the multi-layer perceptron to predict the final sentiment label y pred :

[0109]

[0110] in, represents the parameters of the multilayer perceptron;

[0111] In order to deeply understand the potential emotional semantics of each modality, the present invention introduces a unimodal joint learning module; this module uses a shared weight encoder ε to process the unimodal features F m , thus achieving unimodal prediction of all three modalities, namely text t, vision v, and audio a:

[0112] F′ m =∈(F m θ ∈ )

[0113] Among them, m∈{t,v,a},θ ∈ refers to the parameters of the shared weight encoder; the encoder is a one-dimensional convolutional network;

[0114] After the modal features are processed, they are input into a multi-layer perceptron to generate a single modal prediction y m :

[0115]

[0116] in, represents the parameters of the multilayer perceptron;

[0117] The optimization target is based on L1 loss, and the loss function of multimodal prediction is expressed as:

[0118]

[0119] Where K is the number of multimodal samples; and Represent the prediction and true label of the kth sample respectively;

[0120] In the unimodal joint learning part, the loss can be expressed as:

[0121]

[0122] in, is the predicted unimodal label of modality m in the kth sample;

[0123] The total loss is calculated by combining the multimodal and unimodal losses:

[0124]

[0125] Among them, λ is an adjustable hyperparameter used to balance the impact of the two losses.

[0126] According to a specific implementation of the embodiment of the present invention, the specific steps of S3 are:

[0127] S3. A multimodal sentiment analysis method based on a self-attention mechanism according to any one of claims 1 to 2, which can effectively perform sentiment analysis on videos containing visual, textual and audio modalities even when the audio modality is missing; specifically, the present invention proposes a knowledge transfer network, which reconstructs missing audio features using available visual and textual signals, and models the relationship between visual, textual and reconstructed audio features through a cross-modal attention network, wherein the network structure includes a set of Transformer modules;

[0128] The visual, textual, and audio modalities are converted into modality-specific features f V 、f T and f A, which is implemented by a Transformer-based encoder; in order to ensure the quality of the reconstructed audio features, the present invention adopts a consistency loss function to minimize the Euclidean distance between the reconstructed audio features and the real audio features; the loss function is defined for the visual-to-audio conversion as follows:

[0129]

[0130] Among them, θ represents the knowledge transfer network of the visual modality, which consists of a set of Transformer modules; θ(f V ) represents the audio features reconstructed from the visual modality;

[0131] The loss function for text-to-audio conversion is defined as follows:

[0132]

[0133] in, The knowledge transfer network representing the text modality consists of a set of Transformer modules; φ(f T ) represents the audio features reconstructed from the text modality;

[0134] In order to maximize the utilization of multimodal data, the audio features f′ reconstructed from visual and textual modalities are combined A :

[0135] f′ A =δ([θ(f V )||φ(f T )])

[0136] Among them, the parallel operation || represents the connection of features, which combines the features from vision θ(f V ) and text φ(f T )’s reconstructed audio features f′ A ;

[0137] These fused features are further encoded through a set of Transformer modules δ to effectively express the reconstructed audio information;

[0138] The present invention handles the mutual influence between modalities through the cross-modal attention mechanism. Specifically, the interaction between the source modality feature m and the target modality feature m′ is mapped and calculated through the weight matrix of query Q, key K and value V. The target modality feature m′ is mapped to the latent space as the query The keys and values ​​are derived from the source modal features and Get, where the weight matrix and For this mapping process; the latent adaptation of the source modality m to the target query feature m′ is achieved through cross-modal attention Z′ m :=CM m→m′ (m′,m), expressed as follows;

[0139]

[0140] in, is a scaling parameter used to control the normalization process; d k Represents the dimension length of the target feature;

[0141] Using residual connection, the original target feature m′ is combined with the feature Z′ calculated by cross-modal attention m Combined to form the final feature representation:

[0142] m′=m′+Z′ m

[0143] This allows the model to retain key original information and enhance the expressiveness of features;

[0144] These comprehensive features with inter-modal correlation information are then used for sentiment analysis prediction tasks, where the prediction results are denoted as y′; the decoding process consists of a set of self-attention Transformers to ensure that the information of each modality is fully decoded;

[0145] The final model loss function L combines the standard cross entropy loss function l ce , which is used to calculate the difference between the predicted value y′ and the true value y, and introduces the consistency loss of visual and textual modalities:

[0146]

[0147] The overall loss function L is defined as follows:

[0148]

[0149] Among them, λ1 and λ2 are the adjustment parameters of the consistency loss of visual and textual modalities respectively;

[0150] This design helps ensure that the reconstructed audio features remain highly consistent with the original data while optimizing the overall performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0151] Figure 1 A flowchart of the method is shown in FIG.

[0152] Figure 2 It is a schematic diagram of the overall architecture of the multimodal model BRCM;

[0153] Figure 3This is a schematic diagram of the overall architecture of the multimodal fusion sentiment analysis model based on the BRCM self-attention mechanism;

[0154] Figure 4 Schematic diagram of combining self-attention mechanism and gating mechanism;

[0155] Figure 5 Schematic diagram of the knowledge transfer network for reconstructing the audio modality. DETAILED DESCRIPTION

[0156] In order to facilitate those skilled in the art to understand and implement the present invention, the present invention is further described in detail below with reference to examples and drawings.

[0157] As attached Figure 1 and attached Figure 2 As shown in the figure, a multimodal sentiment analysis method combining a pre-trained model and a self-attention block includes the following steps:

[0158] Step 1: The present invention constructs a multimodal model BRCM, which improves the feature extraction capability of multimodal data by fusing the features of three modalities: text, image and audio. The model combines text processing, image processing and audio processing modules, and generates the final multimodal feature representation through a preprocessing layer, a feature extraction layer, an attention mechanism and a fully connected layer. The text processing module is based on BERT, combined with a bidirectional GRU and a multi-head attention mechanism to extract deep semantic features. The image processing module, based on ResNet, uses a convolutional neural network, global average pooling and a multi-head attention mechanism to extract high-level image features. The audio processing module is based on COVAREP, combined with a convolutional neural network, LSTM and a multi-head attention mechanism to extract deep audio features. Finally, the feature representations of text, image and audio are fused to generate a multimodal feature representation. Through this fusion, the BRCM model can simultaneously extract and express the deep features of the three modalities, and achieve more comprehensive multimodal data analysis and processing.

[0159] Text processing module: In order to enhance the ability to extract text features, improvements are made based on BERT, using a method that combines bidirectional GRU and multi-head attention mechanisms. The new architecture combines BERT's contextual understanding ability, BiGRU's sequence modeling ability, and the multi-head attention mechanism's global information capture ability, which can better extract the deep semantic features of text;

[0160] Input embedding layer: embeds the input text and converts each word into a corresponding vector representation. These vectors can capture the semantic information of the words. Assume that the input text is T = [t1, t2, ..., t n ], and the representation after the embedding layer is:

[0161] Et =[e1,e2,...,e n ]

[0162] Among them, e i Representation word t i The embedding vector of

[0163] Pre-trained BERT layer: The embedding representation is fed into a pre-trained BERT model to obtain context-sensitive word representations. These vectors can capture the meaning of words in different contexts:

[0164] H BERT =BERT(E t )

[0165] Among them, the output of BERT is H BERT , H BERT =[h1,h2,...,h n ], each h i It is the context-sensitive word vector output by the BERT model;

[0166] Bidirectional GRU layer: The context-sensitive word vector output by BERT is input into the bidirectional GRU layer to capture the bidirectional dependencies in the sequence. The forward and backward dependencies of the word vectors are processed separately. The forward GRU layer processes the forward dependency in the sequence, and the backward GRU layer processes the backward dependency in the sequence.

[0167] Assume that the output of the forward GRU is The output of the backward GRU is

[0168] The final GRU output is expressed as:

[0169]

[0170] in, is the output of the bidirectional GRU;

[0171] Multi-head attention layer: The output of the bidirectional GRU is input into the multi-head attention mechanism to capture global contextual information. Multiple attention heads process information in parallel, and then combine this information to generate the output of the attention mechanism:

[0172] H Attn =MultiHeadAttention(H GRU ,H GRU ,H GRU )

[0173] in, is the output of the multi-head attention mechanism;

[0174] Fully connected layer: The output of the multi-head attention mechanism is averaged and pooled to reduce the dimension of the feature. Assume that the feature after pooling is represented as H pool :

[0175]

[0176] The pooled features are input into the fully connected layer, and the final text feature representation is generated through the ReLU activation function:

[0177] H final =ReLU(W f ·H pool +b f )

[0178] Among them, W f and b f are the weights and bias parameters of the fully connected layer, H final is the final text feature representation;

[0179] Image processing module: In order to enhance the image feature extraction capability, the new architecture is improved on the basis of ResNet, adopting a deeper convolutional neural network structure and combining global average pooling and attention mechanism to improve the accuracy of feature expression; the new architecture will include the following main parts: preprocessing layer, ResNet layer, global average pooling layer, multi-head attention layer and fully connected layer;

[0180] Preprocessing layer: standardize the input image, including adjusting the image size, normalization, etc.; assuming that the input image is I, the preprocessed image is represented as I', and the formula is:

[0181] I′=Normalize(Resize(I))

[0182] Among them, Resize means adjusting the image size, and Normalize means normalization processing;

[0183] ResNet layer: The preprocessed image is input into the pre-trained ResNet model to extract high-level image features; based on the ResNet-50 model, the ResNet formula is expressed as:

[0184] H ResNet =ResNet(I′)

[0185] Among them, H ResNet Represents the feature map output by ResNet;

[0186] Global average pooling layer: Perform global average pooling on the feature map output by ResNet to compress the high-dimensional feature map into a feature vector of fixed size. The formula is:

[0187] H GAP =GlobalAveragePooling(H ResNet )

[0188] Among them, H GAP is the feature vector after global average pooling;

[0189] Multi-head attention layer: The feature vector after global average pooling is input into the multi-head attention mechanism to capture global contextual information:

[0190] H Attn =MultiHeadAttention(H GAP ,H GAP ,H GAP )

[0191] Among them, H Attn It is the output feature representation of the multi-head attention mechanism;

[0192] Fully connected layer: The output of the multi-head attention layer is reduced in dimension through the fully connected layer to obtain the final image feature representation:

[0193] H final =ReLU(W f ·H Attn +b f )

[0194] Among them, W f and b f are the weights and bias parameters of the fully connected layer, H final is the final image feature representation;

[0195] Audio processing module: In order to enhance the audio feature extraction capability, improvements are made on the basis of COVAREP, using a more complex convolutional neural network (CNN) structure combined with a long short-term memory (LSTM) network, and using a multi-head attention mechanism to improve the accuracy of feature expression; the new architecture will include the following main parts: preprocessing layer, COVAREP feature extraction layer, convolution layer, LSTM layer, multi-head attention layer, and fully connected layer;

[0196] Preprocessing layer: standardize the input audio signal, including noise reduction and normalization. Assuming the input audio is A, the preprocessed audio is represented as A':

[0197] A′=Normalize(Denoise(A))

[0198] Among them, Denoise means noise reduction processing, Normalize means normalization processing;

[0199] COVAREP feature extraction layer: The preprocessed audio signal is input into the COVAREP model to extract the basic features of the audio:

[0200] H COVAREP =COVAREP(A′)

[0201] Among them, H COVAREP Represents the audio features output by COVAREP;

[0202] Convolutional layer: The features output by COVAREP are input into the convolutional layer to extract higher-level feature representations:

[0203] H Conv =Conv1D(H COVAREP )

[0204] Among them, H Conv It is the feature representation of the convolutional layer output;

[0205] LSTM layer: The output of the convolutional layer is input into the LSTM layer to capture the long-term and short-term dependencies in the time series:

[0206] H LSTM =LSTM(H Conv )

[0207] Among them, H LSTM Represents the feature representation of LSTM output;

[0208] Multi-head attention layer: The output of LSTM is input into the multi-head attention mechanism to capture global contextual information:

[0209] H Attn =MultiHeadAttention(H LSTM ,H LSTM ,H LSTM )

[0210] Among them, H Attn It is the output feature representation of the multi-head attention mechanism;

[0211] Fully connected layer: The output of the multi-head attention layer is reduced in dimension through the fully connected layer to obtain the final audio feature representation:

[0212] H final =ReLU(W f ·H Attn +b f )

[0213] Among them, W f and b f are the weights and bias parameters of the fully connected layer, H finalis the final audio feature representation;

[0214] Constructing the BRCM model: The BRCM model improves the feature extraction capability of multimodal data by integrating the features of three different modalities: text, image, and audio. The model combines text processing, image processing, and audio processing modules, and integrates the feature representations of the above three different modalities through the preprocessing layer, feature extraction layer, attention mechanism, and fully connected layer to generate the final multimodal feature representation.

[0215] Assume that the feature representations of text, image and audio are The final multimodal feature is represented as H BRCM , expressed as:

[0216]

[0217] Among them, Concat represents the concatenation operation of features;

[0218] Through this fusion, the BRCM model can simultaneously extract and express the deep features of text, images, and audio, achieving more comprehensive multimodal data analysis and processing;

[0219] Finally, a multimodal model BRCM was constructed;

[0220] As attached Figure 3 and attached Figure 4 As shown in the figure, a multimodal sentiment analysis method combining a pre-trained model and a self-attention block includes the following steps:

[0221] Step 2: Multimodal sentiment analysis: Using the text t, visual v, and acoustic a information from the same video clip as input, they are represented as Among them, T m and d m Respectively represent the sequence length and feature dimension of mode m; the present invention integrates the information in these emotion-related unaligned multimodal sequences; generates a text-driven multimodal representation to accurately predict the results of sentiment analysis;

[0222] According to claim 1, text features, visual features and audio features are extracted by a multimodal model BRCM, a one-dimensional convolutional network is applied to process these features to capture temporal characteristics, and these features are projected into a fixed dimension to be represented as where m∈{t,v,a};

[0223]

[0224] Among them, F a It is the processed acoustic features used for subsequent modal fusion and sentiment analysis tasks; are the original acoustic features extracted by the multimodal model BRCM; 1Da is a one-dimensional convolutional network applied to the acoustic features, which is used to process these features to capture temporal dependencies and normalize the feature dimensions; parameters Represents the parameters of the one-dimensional convolutional network;

[0225]

[0226] Among them, F v It is the visual feature after one-dimensional convolution processing; is the image feature extracted by the multimodal model BRCM; 1D v It is a one-dimensional convolutional network specifically designed to process visual features, with the goal of extracting time series information and performing dimensionality standardization; the parameters Represents the parameters of this network;

[0227]

[0228] in, It is the text feature extracted by the multimodal model BRCM, including sentence and document level embedding, which aims to process text data to capture possible sequence dependencies and perform appropriate dimensionality transformation; parameter Represents the parameters of the one-dimensional convolutional network, and the processed result F t Represents text features optimized for sentiment analysis tasks;

[0229] Text-oriented cross-attention module: This module contains a cross-attention block for text-oriented feature fusion and a self-attention block for refining text features. A gating mechanism is added to the module to effectively integrate the useful information between the two attention blocks and filter out redundancy. The cross-attention block receives and As input, where m∈{a,v}, the superscript i indicates the i-th layer;

[0230] Perform layer normalization on features: and

[0231] The normalized features are fed into the cross-attention module:

[0232]

[0233] in, W * represents a learnable weight matrix by which corresponding queries, keys, and values ​​can be obtained;

[0234] The self-attention block will As input, it produces

[0235]

[0236] Among them, W * Represents the learnable weight matrix associated with the query, key, and value of the text modality;

[0237] In order to mitigate the noise and redundant interference caused by feature fusion, two gating mechanisms are introduced; these gates adaptively adjust and The transfer ratio of , thereby optimizing the feature representation of the final prediction;

[0238] Memory Gate m is responsible for determining the proportion of components to be transferred in the target modality, while the fusion gate g f Determines what proportion of the fusion component to inject into the target modality; these gating signals are generated from the two modality sequence representations in the same layer and are specifically expressed as:

[0239]

[0240] in, is a dedicated projection matrix. The subscripts m and f are used to distinguish the two gates. σ represents the Sigmoid nonlinear function. Represents the connection of vectors;

[0241] Through these gating mechanisms, the output feature expression is:

[0242]

[0243] Where ⊙ represents element-by-element multiplication;

[0244] This operation can effectively filter out the noise information and redundant features generated by cross-modal interaction, while coordinating the fusion ratio of the two modalities;

[0245] In the processing mechanism, following the feedforward structure of Transformer, and Perform position-level feedforward network processing, including residual connections; for features from modality m to text t First, after layer normalization, it is input into the feedforward network for processing, and the original features before the input are added to form a residual connection:

[0246]

[0247] Among them, FFN(·) represents feed-forward network, LN(·) represents layer normalization;

[0248]

[0249] Among them, FFN(·) represents feed-forward network, LN(·) represents layer normalization;

[0250] This processing method can enhance the model's nonlinear expression ability of features while retaining important original information;

[0251] By connecting the outputs of the last layer of each module, the final feature representation can be calculated;

[0252] Each module consists of two heads, and a total of four heads are used for feature fusion;

[0253] Combining the outputs of the four heads of the two text-oriented cross-modal attention modules, the final feature representation F final It can be expressed as:

[0254]

[0255] in, Represents the connection of vectors;

[0256] The fused feature F final Input into the multi-layer perceptron to predict the final sentiment label y pred :

[0257]

[0258] in, represents the parameters of the multilayer perceptron;

[0259] In order to deeply understand the potential emotional semantics of each modality, the present invention introduces a unimodal joint learning module; this module uses a shared weight encoder ε to process the unimodal features F m , thus achieving unimodal prediction of all three modalities, namely text t, vision v, and audio a:

[0260] F′ m =∈(F m θ ∈ )

[0261] Among them, m∈{t,v,a},θ ∈ refers to the parameters of the shared weight encoder; the encoder is a one-dimensional convolutional network;

[0262] After the modal features are processed, they are input into a multi-layer perceptron to generate a single modal prediction y m :

[0263]

[0264] in, represents the parameters of the multilayer perceptron;

[0265] The optimization target is based on L1 loss, and the loss function of multimodal prediction is expressed as:

[0266]

[0267] Where K is the number of multimodal samples; and Represent the prediction and true label of the kth sample respectively;

[0268] In the unimodal joint learning part, the loss can be expressed as:

[0269]

[0270] in, is the predicted unimodal label of modality m in the kth sample;

[0271] The total loss is calculated by combining the multimodal and unimodal losses:

[0272]

[0273] Among them, λ is an adjustable hyperparameter used to balance the impact of the two losses.

[0274] As attached Figure 5 As shown in the figure, a multimodal sentiment analysis method combining a pre-trained model and a self-attention block includes the following steps:

[0275] Step 3: A multimodal sentiment analysis method based on a self-attention mechanism according to any one of the methods described in claims 1-2 can effectively perform sentiment analysis on videos containing visual, textual and audio modalities even when the audio modality is missing; specifically, the present invention proposes a knowledge transfer network that uses available visual and textual signals to reconstruct missing audio features, and models the relationship between visual, textual and reconstructed audio features through a cross-modal attention network, wherein the network structure includes a group of Transformer modules;

[0276] The visual, textual, and audio modalities are converted into modality-specific features f V 、f T and f A , which is implemented by a Transformer-based encoder; in order to ensure the quality of the reconstructed audio features, the present invention adopts a consistency loss function to minimize the Euclidean distance between the reconstructed audio features and the real audio features; the loss function is defined for the visual-to-audio conversion as follows:

[0277]

[0278] Among them, θ represents the knowledge transfer network of the visual modality, which consists of a set of Transformer modules; θ(f V ) represents the audio features reconstructed from the visual modality;

[0279] The loss function for text-to-audio conversion is defined as follows:

[0280]

[0281] in, The knowledge transfer network representing the text modality consists of a set of Transformer modules; φ(f T ) represents the audio features reconstructed from the text modality;

[0282] In order to maximize the utilization of multimodal data, the audio features f′ reconstructed from visual and textual modalities are combined A :

[0283] f′ A =δ([θ(f V )||φ(f T )])

[0284] Among them, the parallel operation || represents the connection of features, which combines the features from vision θ(f V ) and text φ(f T )’s reconstructed audio features f′ A ;

[0285] These fused features are further encoded through a set of Transformer modules δ to effectively express the reconstructed audio information;

[0286] The present invention handles the mutual influence between modalities through the cross-modal attention mechanism. Specifically, the interaction between the source modality feature m and the target modality feature m′ is mapped and calculated through the weight matrix of query Q, key K and value V. The target modality feature m′ is mapped to the latent space as the query The keys and values ​​are derived from the source modal features and Get, where the weight matrix and For this mapping process; the latent adaptation of the source modality m to the target query feature m′ is achieved through cross-modal attention Z′ m :=CM m→m′ (m′,m), expressed as follows;

[0287]

[0288] in, is a scaling parameter used to control the normalization process; dk Represents the dimension length of the target feature;

[0289] Using residual connection, the original target feature m′ is combined with the feature Z′ calculated by cross-modal attention m Combined to form the final feature representation:

[0290] m′=m′+Z′ m

[0291] This allows the model to retain key original information and enhance the expressiveness of features;

[0292] These comprehensive features with inter-modal correlation information are then used for sentiment analysis prediction tasks, where the prediction results are denoted as y′; the decoding process consists of a set of self-attention Transformers to ensure that the information of each modality is fully decoded;

[0293] The final model loss function L combines the standard cross entropy loss function l ce , which is used to calculate the difference between the predicted value y′ and the true value y, and introduces the consistency loss of visual and textual modalities:

[0294]

[0295] The overall loss function L is defined as follows:

[0296]

[0297] Among them, λ1 and λ2 are the adjustment parameters of the consistency loss of visual and textual modalities respectively;

[0298] This design helps ensure that the reconstructed audio features remain highly consistent with the original data while optimizing the overall performance of the model.

Claims

1. A multimodal sentiment analysis method combining a pre-trained model and a self-attention block, characterized in that The following steps are involved: S1: The present invention constructs a multimodal model BRCM, which improves the feature extraction capability of multimodal data by fusing the features of three modalities: text, image and audio. The model combines text processing, image processing and audio processing modules, and generates the final multimodal feature representation through preprocessing layer, feature extraction layer, attention mechanism and fully connected layer. The text processing module is based on BERT, combined with bidirectional GRU and multi-head attention mechanism to extract deep semantic features. The image processing module uses convolutional neural network, global average pooling and multi-head attention mechanism on the basis of ResNet to extract high-level image features. The audio processing module is based on COVAREP, combined with convolutional neural network, LSTM and multi-head attention mechanism to extract deep audio features. Finally, the feature representations of text, image and audio are fused to generate multimodal feature representation. Through this fusion, the BRCM model can simultaneously extract and express the deep features of the three modalities, and realize more comprehensive multimodal data analysis and processing. Text processing module: In order to enhance the ability to extract text features, improvements are made based on BERT, using a method that combines bidirectional GRU and multi-head attention mechanisms. The new architecture combines BERT's contextual understanding ability, BiGRU's sequence modeling ability, and the multi-head attention mechanism's global information capture ability, which can better extract the deep semantic features of text; Input embedding layer: embeds the input text and converts each word into a corresponding vector representation. These vectors can capture the semantic information of the words. Assume that the input text is T = [t1, t2, …, t n ], and the representation after the embedding layer is: THE t =[e1,e2,…,e n ] Among them, e i Representation word t i The embedding vector of Pre-trained BERT layer: The embedding representation is fed into a pre-trained BERT model to obtain context-sensitive word representations. These vectors can capture the meaning of words in different contexts: H BERT =BERT(E t ) Among them, the output of BERT is H BERT , H BERT =[h1,h2,…,h n ], each h i It is the context-sensitive word vector output by the BERT model; Bidirectional GRU layer: The context-sensitive word vector output by BERT is input into the bidirectional GRU layer to capture the bidirectional dependencies in the sequence. The forward and backward dependencies of the word vectors are processed separately. The forward GRU layer processes the forward dependency in the sequence, and the backward GRU layer processes the backward dependency in the sequence. Assume that the output of the forward GRU is The output of the backward GRU is The final GRU output is expressed as: in, is the output of the bidirectional GRU; Multi-head attention layer: The output of the bidirectional GRU is input into the multi-head attention mechanism to capture global contextual information. Multiple attention heads process information in parallel, and then combine this information to generate the output of the attention mechanism: H Attn =MultiHeadAttention(H GRU ,H GRU ,H GRU ) in, is the output of the multi-head attention mechanism; Fully connected layer: The output of the multi-head attention mechanism is averaged and pooled to reduce the dimension of the feature. Assume that the feature after pooling is represented as H pool : The pooled features are input into the fully connected layer, and the final text feature representation is generated through the ReLU activation function: A final =ReLU(W f H pool +b f ) Among them, W f and b f are the weights and bias parameters of the fully connected layer, H final is the final text feature representation; Image processing module: In order to enhance the image feature extraction capability, the new architecture is improved on the basis of ResNet, adopting a deeper convolutional neural network structure and combining global average pooling and attention mechanism to improve the accuracy of feature expression; the new architecture will include the following main parts: preprocessing layer, ResNet layer, global average pooling layer, multi-head attention layer and fully connected layer; Preprocessing layer: standardize the input image, including adjusting the image size, normalization, etc.; assuming that the input image is I, the preprocessed image is represented as I', and the formula is: I′=Normalize(Resize(I)) Among them, Resize means adjusting the image size, and Normalize means normalization processing; ResNet layer: The preprocessed image is input into the pre-trained ResNet model to extract high-level image features; based on the ResNet-50 model, the ResNet formula is expressed as: H ResNet =ResNet(I′) Among them, H ResNet Represents the feature map output by ResNet; Global average pooling layer: Perform global average pooling on the feature map output by ResNet to compress the high-dimensional feature map into a feature vector of fixed size. The formula is: H GAP =GlobalAveragePooling(H ResNet ) Among them, H GAP is the feature vector after global average pooling; Multi-head attention layer: The feature vector after global average pooling is input into the multi-head attention mechanism to capture global contextual information: H Attn =MultiHeadAttention(H GAP ,H GAP ,H GAP ) Among them, H Attn It is the output feature representation of the multi-head attention mechanism; Fully connected layer: The output of the multi-head attention layer is reduced in dimension through the fully connected layer to obtain the final image feature representation: A final =ReLU(W f H Attn +b f ) Among them, W f and b f are the weights and bias parameters of the fully connected layer, H final is the final image feature representation; Audio processing module: In order to enhance the audio feature extraction capability, improvements are made on the basis of COVAREP, using a more complex convolutional neural network (CNN) structure combined with a long short-term memory (LSTM) network, and using a multi-head attention mechanism to improve the accuracy of feature expression; the new architecture will include the following main parts: preprocessing layer, COVAREP feature extraction layer, convolution layer, LSTM layer, multi-head attention layer, and fully connected layer; Preprocessing layer: standardize the input audio signal, including noise reduction and normalization. Assuming the input audio is A, the preprocessed audio is represented as A': A′=Normalize(Denoise(A)) Among them, Denoise means noise reduction processing, Normalize means normalization processing; COVAREP feature extraction layer: The preprocessed audio signal is input into the COVAREP model to extract the basic features of the audio: H COVAREP =COVAREP(A′) Among them, H COVAREP Represents the audio features output by COVAREP; Convolutional layer: The features output by COVAREP are input into the convolutional layer to extract higher-level feature representations: H Conv =Conv1D(H COVAREP ) Among them, H Conv It is the feature representation of the convolutional layer output; LSTM layer: The output of the convolutional layer is input into the LSTM layer to capture the long-term and short-term dependencies in the time series: H LSTM =LSTM(H Conv ) Among them, H LSTM Represents the feature representation of LSTM output; Multi-head attention layer: The output of LSTM is input into the multi-head attention mechanism to capture global contextual information: H Attn =MultiHeadAttention(H LSTM ,H LSTM ,H LSTM ) Among them, H Attn It is the output feature representation of the multi-head attention mechanism; Fully connected layer: The output of the multi-head attention layer is reduced in dimension through the fully connected layer to obtain the final audio feature representation: A final =ReLU(W f H Attn +b f ) Among them, W f and b f are the weights and bias parameters of the fully connected layer, H final is the final audio feature representation; Constructing the BRCM model: The BRCM model improves the feature extraction capability of multimodal data by integrating the features of three different modalities: text, image, and audio. The model combines text processing, image processing, and audio processing modules, and integrates the feature representations of the above three different modalities through the preprocessing layer, feature extraction layer, attention mechanism, and fully connected layer to generate the final multimodal feature representation. Assume that the feature representations of text, image and audio are The final multimodal feature is represented as H BRCM , expressed as: Among them, Concat represents the concatenation operation of features; Through this fusion, the BRCM model can simultaneously extract and express the deep features of text, images, and audio, achieving more comprehensive multimodal data analysis and processing; Finally, a multimodal model BRCM was constructed; S2: The present invention proposes a method that combines attention mechanism and gating mechanism to effectively fuse multimodal information. Multimodal sentiment analysis uses text, visual and acoustic information from the same video clip, and generates text-driven multimodal representation by integrating this information to accurately predict sentiment. Text features are extracted through the BERT model, visual features are extracted through the ResNet model, and acoustic features are extracted through the COVAREP model. These features are processed through a one-dimensional convolutional network to capture temporal characteristics and project them into a fixed dimension. In order to enhance feature processing capabilities, a text-oriented cross-attention module is introduced, which includes a cross-attention block and a self-attention block, and information fusion and redundant filtering are performed between the two through a gating mechanism. The self-attention mechanism allows the model to dynamically assign attention weights according to the input data. When processing different text, image and audio inputs, the model can automatically adjust the attention to each modality to adapt to sentiment analysis in different contexts. analysis needs; through the self-attention mechanism, the model can more deeply explore the emotional information in text and images; the self-attention mechanism also helps the model capture contextual information in multimodal data; by calculating the attention weights at different positions, the model can understand the relationship between elements in text and images, so as to better understand the overall emotional expression; the features are sent to the feedforward network for processing after layer normalization, forming a residual connection to retain important original information; the output of the module is formed by connecting the outputs of each layer to form the final feature representation, which is further input into the multi-layer perceptron to predict the emotional label; in order to deeply understand the emotional semantics of each modality, a single-modal joint learning module is introduced, using an encoder with shared weights to process single-modal features and predict through a multi-layer perceptron; the optimization target is based on L1 loss, combining the loss of multimodal prediction with the loss of single-modal prediction, and balancing the two losses by adjusting the hyperparameter λ to improve the prediction accuracy and generalization ability of the model; S3: The present invention proposes a multimodal method for sentiment analysis, which is suitable for processing video content containing visual, textual and audio modalities, and can work effectively even when the audio modality is missing; the method converts visual and textual modalities into modality-specific features through a Transformer-based encoder; in the absence of audio features, the audio features are reconstructed using visual and textual signals through a knowledge transfer network, which includes multiple Transformer modules to ensure that the reconstructed audio features can accurately represent the original audio information; in order to verify the quality of the reconstructed audio, a consistency loss function is used, which optimizes the model by minimizing the Euclidean distance between the reconstructed audio features and the real audio features; the audio features reconstructed by vision and text are combined and further encoded through another set of Transformer modules to enhance the expressive power of the audio information; The interaction between features of different modalities is handled through the cross-modal attention mechanism, which maps and calculates the relationship between different modalities through a weight matrix, and effectively integrates information through queries, keys and values ​​in the latent space; through residual connections, the features obtained through cross-modal attention calculation are combined with the original target features to form the final feature representation, which not only retains the key original information but also enhances the expressiveness of the features; these features are then used for sentiment analysis prediction tasks, where the decoding process is completed by a group of self-attention Transformers to ensure full decoding and integration of information from each modality; the model loss function combines the standard cross-entropy loss with the consistency loss of visual and textual modalities to optimize the overall model performance and ensure a high degree of consistency in the reconstructed audio; this design not only improves the accuracy of the model, but also enhances its robustness in the absence of data from a certain modality.

2. A multimodal sentiment analysis method combining a pre-trained model and a self-attention block according to claim 1, characterized in that The specific method of step S2 is: Multimodal sentiment analysis: Using text t, visual v, and acoustic a information from the same video clip as input, represented as m∈{t,v,a} for each modality Among them, T m and d m Respectively represent the sequence length and feature dimension of mode m; the present invention integrates the information in these emotion-related unaligned multimodal sequences; generates a text-driven multimodal representation to accurately predict the results of sentiment analysis; According to claim 1, text features, visual features and audio features are extracted by a multimodal model BRCM, a one-dimensional convolutional network is applied to process these features to capture temporal characteristics, and these features are projected into a fixed dimension to be represented as where m∈{t,v,a}; Among them, F a It is the processed acoustic features used for subsequent modal fusion and sentiment analysis tasks; are the original acoustic features extracted by the multimodal model BRCM; 1Da is a one-dimensional convolutional network applied to the acoustic features, which is used to process these features to capture temporal dependencies and normalize the feature dimensions; parameters Represents the parameters of the one-dimensional convolutional network; Among them, F v It is the visual feature after one-dimensional convolution processing; is the image feature extracted by the multimodal model BRCM; 1D v It is a one-dimensional convolutional network specifically designed to process visual features, with the goal of extracting time series information and performing dimensionality standardization; the parameters Represents the parameters of this network; Among them, F t BRCM It is the text feature extracted by the multimodal model BRCM, including sentence and document level embedding, which aims to process text data to capture possible sequence dependencies and perform appropriate dimensionality transformation; parameter Represents the parameters of the one-dimensional convolutional network, and the processed result F t Represents text features optimized for sentiment analysis tasks; Text-oriented cross-attention module: This module contains a cross-attention block for text-oriented feature fusion and a self-attention block for refining text features. A gating mechanism is added to the module to effectively integrate the useful information between the two attention blocks and filter out redundancy. The cross-attention block receives F t [i] and As input, where m∈{a,v}, the superscript i indicates the i-th layer; Perform layer normalization on features: and F t [i] =LN(F t [i] ); The normalized features are fed into the cross-attention module: in, F t [0] =F t , W * represents a learnable weight matrix by which corresponding queries, keys, and values ​​can be obtained; The self-attention block converts F t [i] As input, generate F t [i+1] ; Among them, W * Represents the learnable weight matrix associated with the query, key, and value of the text modality; In order to mitigate the noise and redundant interference caused by feature fusion, two gating mechanisms are introduced; these gates adaptively adjust and The transfer ratio of , thereby optimizing the feature representation of the final prediction; Memory Gate m is responsible for determining the proportion of components to be transferred in the target modality, while the fusion gate g f Determines what proportion of the fusion component to inject into the target modality; these gating signals are generated from the two modality sequence representations in the same layer and are specifically expressed as: in, is a dedicated projection matrix. The subscripts m and f are used to distinguish the two gates. σ represents the Sigmoid nonlinear function. Represents the connection of vectors; Through these gating mechanisms, the output feature expression is: Where ⊙ represents element-by-element multiplication; This operation can effectively filter out the noise information and redundant features generated by cross-modal interaction, while coordinating the fusion ratio of the two modalities; In the processing mechanism, following the feedforward structure of Transformer, and F t [i+1] Perform position-level feedforward network processing, including residual connections; for features from modality m to text t First, after layer normalization, it is input into the feedforward network for processing, and the original features before the input are added to form a residual connection: Among them, FFN(·) represents feed-forward network, LN(·) represents layer normalization; F t [i+1] =FFN(LN(F t [i+1] ))+F t [i+1] Among them, FFN(·) represents feed-forward network, LN(·) represents layer normalization; This processing method can enhance the model's nonlinear expression ability of features while retaining important original information; By connecting the outputs of the last layer of each module, the final feature representation can be calculated; Each module consists of two heads, and a total of four heads are used for feature fusion; Combining the outputs of the four heads of the two text-oriented cross-modal attention modules, the final feature representation F final It can be expressed as: in, Represents the connection of vectors; The fused feature F final Input into the multi-layer perceptron to predict the final sentiment label y pred : in, represents the parameters of the multilayer perceptron; In order to deeply understand the potential emotional semantics of each modality, the present invention introduces a unimodal joint learning module; this module uses a shared weight encoder ε to process the unimodal features F m , thus achieving unimodal prediction of all three modalities, namely text t, vision v, and audio a: F′ m =∈(F m ;θ ∈ ) Among them, m∈{t,v,a},θ ∈ refers to the parameters of the shared weight encoder; the encoder is a one-dimensional convolutional network; After the modal features are processed, they are input into a multi-layer perceptron to generate a single modal prediction y m : in, represents the parameters of the multilayer perceptron; The optimization target is based on L1 loss, and the loss function of multimodal prediction is expressed as: Where K is the number of multimodal samples; and Represent the prediction and true label of the kth sample respectively; In the unimodal joint learning part, the loss can be expressed as: in, is the predicted unimodal label of modality m in the kth sample; The total loss is calculated by combining the multimodal and unimodal losses: Among them, λ is an adjustable hyperparameter used to balance the impact of the two losses.

3. A multimodal sentiment analysis method combining a pre-trained model and a self-attention block according to claim 1, characterized in that The specific steps in step S3 are: A multimodal sentiment analysis method based on a self-attention mechanism based on any one of the methods described in claims 1-2 can perform effective sentiment analysis on videos containing visual, textual and audio modalities even when the audio modality is missing; specifically, the present invention proposes a knowledge transfer network, which uses available visual and textual signals to reconstruct missing audio features, and models the relationship between visual, textual and reconstructed audio features through a cross-modal attention network, the network structure of which includes a group of Transformer modules; The visual, textual, and audio modalities are converted into modality-specific features f V 、f T and f A , which is implemented by a Transformer-based encoder; in order to ensure the quality of the reconstructed audio features, the present invention adopts a consistency loss function to minimize the Euclidean distance between the reconstructed audio features and the real audio features; the loss function is defined for the visual-to-audio conversion as follows: in, θ represents the knowledge transfer network of the visual modality, which consists of a set of Transformer modules; θ(f V ) represents the audio features reconstructed from the visual modality; The loss function for text-to-audio conversion is defined as follows: in, The knowledge transfer network representing the text modality consists of a set of Transformer modules; φ(f T ) represents the audio features reconstructed from the text modality; In order to maximize the utilization of multimodal data, the audio features f′ reconstructed from visual and textual modalities are combined A : f′ A =δ([θ(f V )||φ(f T )]) Among them, the parallel operation || represents the connection of features, which combines the features from vision θ(f V ) and text φ(f T )’s reconstructed audio features f′ A ; These fused features are further encoded through a set of Transformer modules δ to effectively express the reconstructed audio information; The present invention handles the mutual influence between modalities through the cross-modal attention mechanism. Specifically, the interaction between the source modality feature m and the target modality feature m′ is mapped and calculated through the weight matrix of query Q, key K and value V. The target modality feature m′ is mapped to the latent space as the query The keys and values ​​are derived from the source modal features and Get, where the weight matrix and For this mapping process; the latent adaptation of the source modality m to the target query feature m′ is achieved through cross-modal attention Z′ m :=CM m→m′ (m′,m), expressed as follows; in, is a scaling parameter used to control the normalization process; d k Represents the dimension length of the target feature; Using residual connection, the original target feature m′ is combined with the feature Z′ calculated by cross-modal attention m Combined to form the final feature representation: ″′ m=m+Z m This allows the model to retain key original information and enhance the expressiveness of features; These comprehensive features with inter-modal correlation information are then used for sentiment analysis prediction tasks, where the prediction results are denoted as y′; the decoding process consists of a set of self-attention Transformers to ensure that the information of each modality is fully decoded; The final model loss function L combines the standard cross entropy loss function l ce , which is used to calculate the difference between the predicted value y′ and the true value y, and introduces the consistency loss of visual and textual modalities: The overall loss function L is defined as follows: Among them, λ1 and λ2 are the adjustment parameters of the consistency loss of visual and textual modalities respectively; This design helps ensure that the reconstructed audio features remain highly consistent with the original data while optimizing the overall performance of the model.

Citation Information

Cited By

  • Comparative micro-expression recognition method based on text position attention

    CN119028001A

  • Controllable video generation method and system based on multi-modal fusion

    CN119091362A

  • Voice and sight line multi-modal fusion-based intention recognition method and system

    CN119206424A

  • Multi-modal sentiment analysis and interactive adjustment method and system based on large model

    CN119475099A

  • A Multimodal Sentiment Analysis and Interaction Adaptation Method and System Based on Large Models

    CN119475099B