A multi-modal sentiment analysis method and system based on decoupling language focus and multi-scale cross-modal semantic enhancement

Abstract

The application discloses a kind of based on decoupling language focus and multi-scale cross-modal semantic enhancement multi-modal sentiment analysis method and system, solve the technical problem that current multi-modal sentiment analysis method cannot capture short-time emotional fluctuation and long-time emotional trend simultaneously, leading to the accuracy of emotional recognition is low.Method includes obtaining multi-modal sentiment sample, and the feature coding is carried out to multi-modal sentiment sample, and outputs multi-modal sentiment feature;Modal feature decoupling learning is carried out to multi-modal sentiment feature, and split obtains multi-modal private feature and multi-modal shared feature;Shared sentiment semantic enhancement is carried out to multi-modal shared feature, and enhanced shared sentiment semantic representation is output;According to multi-modal private feature and enhanced shared sentiment semantic representation, generate multi-modal calibration representation;Multi-modal calibration representation and enhanced shared sentiment semantic representation are spliced and fused and sentiment prediction, and output final sentiment analysis prediction result.

Description

Technical Field

[0001] This invention relates to the field of multimodal sentiment analysis technology, and in particular to a multimodal sentiment analysis method and system based on decoupled language focusing and multi-scale cross-modal semantic enhancement. Background Technology

[0002] Emotions are an important component of human psychological activity, and they have a significant impact on an individual's behavioral expression, social interaction, and cognitive decision-making.

[0003] With the development of natural language processing, speech processing, and computer vision technologies, researchers have begun to combine multimodal data such as text, speech, and video to more comprehensively and accurately identify users' emotional states. Text modality typically contains explicit semantic information, speech modality reflects changes in tone, rhythm, and intensity, while video modality provides facial expressions and motor cues. This type of multimodal sentiment analysis method has broad application value in fields such as human-computer interaction, public opinion monitoring, smart education, mental health, and intelligent customer service.

[0004] Existing multimodal sentiment analysis methods typically extract features from different modalities separately and then fuse them through feature concatenation, cross-modal attention, or shared representation learning. However, in practical applications, these methods only employ a single-scale temporal modeling approach, failing to simultaneously capture short-term emotional fluctuations and long-term sentiment trends, resulting in low accuracy in sentiment recognition. Summary of the Invention

[0005] This invention provides a multimodal sentiment analysis method and system based on decoupled language focusing and multi-scale cross-modal semantic enhancement, which solves the technical problem that existing multimodal sentiment analysis methods cannot simultaneously capture short-term emotional fluctuations and long-term emotional trends, resulting in low accuracy of sentiment recognition.

[0006] The first aspect of this invention provides a multimodal sentiment analysis method based on decoupled language focusing and multi-scale cross-modal semantic enhancement, comprising:

[0007] Acquire multimodal sentiment samples, encode features of the multimodal sentiment samples, and output multimodal sentiment features;

[0008] Modal feature decoupling learning is performed on the multimodal emotion features to separate them into multimodal private features and multimodal shared features;

[0009] The multimodal shared features are subjected to shared sentiment semantic enhancement to output an enhanced shared sentiment semantic representation;

[0010] Based on the multimodal private features and the enhanced shared sentiment semantic representation, a multimodal calibration representation is generated;

[0011] The multimodal calibration representation and the enhanced shared sentiment semantic representation are spliced ​​and fused together for sentiment prediction, and the final sentiment analysis prediction result is output.

[0012] Optionally, the multimodal emotion sample includes a text sequence, an audio feature sequence, and a visual feature sequence; the multimodal emotion feature includes initial text modality features, initial audio modality features, and initial visual modality features; the feature encoding of the multimodal emotion sample to output multimodal emotion features includes:

[0013] The text sequence is semantically encoded in context, and then dimensional alignment is achieved by sequentially passing it through the Dropout operator and one-dimensional convolution to obtain the initial features of the text modality.

[0014] The audio feature sequence and the visual feature sequence are dimensionally projected respectively to obtain the initial features of the audio modality and the initial features of the visual modality.

[0015] Optionally, the multimodal private features include text-private features, audio-private features, and visual-private features; the multimodal shared features include text-shared features, audio-shared features, and visual-shared features; the step of performing modal feature decoupling learning on the multimodal sentiment features to decompose them into multimodal private features and multimodal shared features includes:

[0016] A shared encoder with shared parameters is used to perform dedicated feature learning on the initial features of the text modality, the initial features of the audio modality, and the initial features of the visual modality, respectively, to obtain text-private features, audio-private features, and visual-private features;

[0017] The initial features of the text modality, the initial features of the audio modality, and the initial features of the visual modality are encoded respectively, and combined with the shared similarity constraints in the training phase, so as to learn cross-modal common sentiment semantics and obtain text shared features, audio shared features, and visual shared features.

[0018] Optionally, the step of performing shared sentiment semantic enhancement on the multimodal shared features and outputting an enhanced shared sentiment semantic representation includes:

[0019] Self-attention enhancement and learnable attention pooling operations are sequentially performed on the text-shared features, the audio-shared features, and the visual-shared features to obtain text-shared pooling representations, audio-shared pooling representations, and visual-shared pooling representations, respectively.

[0020] The text shared pooling representation, the audio shared pooling representation, and the visual shared pooling representation are concatenated by feature dimensions, and then input into a residual multilayer perceptron to complete feature mapping and enhancement, thereby obtaining a preliminary fused shared sentiment semantic representation;

[0021] The initially fused shared sentiment semantic representation is sequentially subjected to linear projection and layer normalization processing to output an enhanced shared sentiment semantic representation.

[0022] Optionally, the multimodal calibration representation includes a text calibration representation, an audio-related calibration representation, and a visual-related calibration representation; the step of generating the multimodal calibration representation based on the multimodal private features and the enhanced shared sentiment semantic representation includes:

[0023] The text-private features are modeled using a high-level temporal modeling method with a text memory Transformer. The audio-private features and the visual-private features are then modeled using a global descriptor cross-modal dual attention method and a corresponding memory Transformer model, respectively. The output consists of a high-level sequence of the text itself, a high-level sequence of language-related features enhanced by audio, and a high-level sequence of language-related features enhanced by visual.

[0024] After performing temporal multi-scale semantic relabeling on the high-level sequence of the text itself, the high-level sequence of language related to the audio enhancement, and the high-level sequence of language related to the visual enhancement, and then processing them through temporal pooling, residual multilayer perceptron and projection layer, we obtain the high-level private representation of the text, the high-level private representation of the audio related, and the high-level private representation of the visual related.

[0025] After performing high-level residual MLP and projection layer processing on the high-level private representation of text, the high-level private representation of audio, and the high-level private representation of vision respectively, and then combining the enhanced shared sentiment semantic representation, shared semantic gating feedback injection processing is carried out to output text calibration representation, audio-related calibration representation, and vision-related calibration representation.

[0026] Optionally, the text-private features are subjected to high-level temporal modeling using a text memory Transformer, and the audio-private features and visual-private features are subjected to global descriptor cross-modal dual attention processing and corresponding memory Transformer modeling with the text-private features, respectively, outputting a high-level sequence of the text itself, a high-level sequence of language-related features enhanced by audio, and a high-level sequence of language-related features enhanced by visual, including:

[0027] Content projection processing is performed on the audio private features and the visual private features respectively to obtain the audio source modal content representation and the visual source modal content representation;

[0028] Assignment mapping and attention weights are performed on the audio source modal content representation and the visual source modal content representation respectively to obtain the audio descriptor attention weight and the visual descriptor attention weight;

[0029] Feature aggregation is performed on the audio source modality content representation and the audio descriptor attention weight, and on the visual source modality content representation and the visual descriptor attention weight, respectively, to obtain the audio global semantic descriptor and the visual global semantic descriptor;

[0030] The text private features and the audio global semantic descriptor are subjected to reception weights at multiple time steps, and semantic distribution is completed to obtain the audio cross-modal context representation.

[0031] The text-private features and the visual global semantic descriptor are subjected to reception weights at multiple time steps, and semantic distribution is completed to obtain a visual cross-modal context representation.

[0032] After controlling the injection strength of auxiliary semantics into the text private features and the audio cross-modal context representation, residual scaling, Dropout, and layer normalization are performed sequentially to obtain the audio-to-text cross-modal enhancement sequence.

[0033] After controlling the injection intensity of auxiliary semantics into the text private features and the visual cross-modal context representation, residual scaling, Dropout, and layer normalization are performed sequentially to obtain the visual-to-text cross-modal enhancement sequence.

[0034] The text-private features are input into a text memory Transformer for high-level temporal modeling to obtain the text's own high-level sequence.

[0035] The audio-to-text cross-modal enhancement sequence is input into the audio memory Transformer to complete high-level temporal modeling, resulting in a high-level audio-enhanced language-related sequence.

[0036] The vision-to-text cross-modal enhancement sequence is input into the visual memory Transformer to complete high-level temporal modeling, resulting in a high-level sequence of language-related visual enhancements.

[0037] Optionally, after performing temporal multi-scale semantic relabeling on the high-level sequence of the text itself, the high-level sequence of language relevance enhanced by audio, and the high-level sequence of language relevance enhanced by vision, and then processing them through temporal pooling, residual multilayer perceptron, and projection layer, high-level private representations of text, audio relevance, and vision relevance are obtained, including:

[0038] The feature channels of the high-level sequence of the text itself, the high-level sequence of language related to the audio enhancement, and the high-level sequence of language related to the visual enhancement are divided into four groups respectively. The emotional change features at multiple time scales are extracted by one-dimensional deep convolution branches with different kernel sizes and dilation rates. The four groups of convolution results are then concatenated to obtain the multi-scale temporal features of the text, the multi-scale temporal features of the audio, and the multi-scale temporal features of the visual.

[0039] Group normalization and gating function calculation are performed sequentially on the text multi-scale temporal features, the audio-related multi-scale temporal features, and the visual-related multi-scale temporal features to generate temporal attention weights for the text multi-scale temporal features, the audio-related multi-scale temporal features, and the visual-related multi-scale temporal features, respectively.

[0040] Based on the temporal attention weights of the text multi-scale temporal features, the audio-related multi-scale temporal features, and the visual-related multi-scale temporal features, adaptive recalibration is performed on the text's own high-level sequence, the audio-enhanced language-related high-level sequence, and the visual-enhanced language-related high-level sequence, outputting the text recalibration high-level sequence, the audio-related recalibration high-level sequence, and the visual-related recalibration high-level sequence.

[0041] Temporal pooling, residual multilayer perceptron, and projection layer processing are performed on the text recalibration high-level sequence, the audio-related recalibration high-level sequence, and the visual-related recalibration high-level sequence, respectively, to obtain the text high-level private representation, the audio-related high-level private representation, and the visual-related high-level private representation.

[0042] Optionally, after performing high-level residual MLP and projection layer processing on the high-level private representation of text, the high-level private representation of audio, and the high-level private representation of vision respectively, and then combining the enhanced shared sentiment semantic representation, shared semantic gating feedback injection processing is carried out to output text calibration representation, audio-related calibration representation, and visual-related calibration representation, including:

[0043] High-level residual MLP and projection layer processing are performed on the text high-level private representation, the audio-related high-level private representation, and the visual-related high-level private representation respectively to obtain the processed text high-level private representation, the processed audio-related high-level private representation, and the processed visual-related high-level private representation.

[0044] The enhanced shared sentiment semantic representation is projected onto the text, audio, and visual-specific feature spaces respectively to obtain the text domain shared semantic representation, the audio domain shared semantic representation, and the visual domain shared semantic representation.

[0045] The processed high-level private representation of text, the processed high-level private representation of audio, and the processed high-level private representation of vision are concatenated with the corresponding shared semantic representation of text domain, shared semantic representation of audio domain, and shared semantic representation of vision domain, respectively, and then gating mapping is performed to generate text gating weights, audio gating weights, and vision gating weights.

[0046] Based on the text gating weight, the audio gating weight, and the visual gating weight, shared semantic adaptive injection is performed on the processed high-level private representation of text, the processed high-level private representation of audio, and the processed high-level private representation of vision, and the text calibration representation, audio-related calibration representation, and vision-related calibration representation are output.

[0047] Optionally, the step of concatenating and fusing the multimodal calibration representation and the enhanced shared sentiment semantic representation for sentiment prediction, and outputting the final sentiment analysis prediction result, includes:

[0048] The text calibration representation, the audio-related calibration representation, the visual-related calibration representation, and the enhanced shared sentiment semantic representation are concatenated along their feature dimensions to generate a global concatenated and fused feature.

[0049] The global splicing and fusion features are input into the residual fully connected fusion layer to complete deep semantic optimization, resulting in the optimized fusion representation;

[0050] The optimized fusion representation is input into the linear output layer to obtain the final sentiment analysis prediction result.

[0051] The second aspect of this invention provides a multimodal sentiment analysis system based on decoupled language focusing and multi-scale cross-modal semantic enhancement, comprising:

[0052] The acquisition module is used to acquire multimodal sentiment samples, perform feature encoding on the multimodal sentiment samples, and output multimodal sentiment features;

[0053] The learning module is used to perform modal feature decoupling learning on the multimodal emotion features, and to separate them into multimodal private features and multimodal shared features;

[0054] The enhancement module is used to perform shared sentiment semantic enhancement on the multimodal shared features and output an enhanced shared sentiment semantic representation;

[0055] A generation module is used to generate a multimodal calibration representation based on the multimodal private features and the enhanced shared sentiment semantic representation;

[0056] The prediction module is used to concatenate and fuse the multimodal calibration representation and the enhanced shared sentiment semantic representation and perform sentiment prediction, outputting the final sentiment analysis prediction result.

[0057] As can be seen from the above technical solutions, the present invention has the following advantages:

[0058] The present invention provides a multimodal sentiment analysis method based on decoupled language focusing and multi-scale cross-modal semantic enhancement. The method acquires multimodal sentiment samples, encodes their features, and outputs multimodal sentiment features. It then performs modal feature decoupling learning on these features, separating them into multimodal private features and multimodal shared features. Finally, it enhances the shared sentiment semantics of the multimodal shared features, outputting an enhanced shared sentiment semantic representation. Based on the multimodal private features and the enhanced shared sentiment semantic representation, it generates a multimodal calibration representation. Finally, it concatenates and fuses the multimodal calibration representation and the enhanced shared sentiment semantic representation, performing sentiment prediction to output the final result. Final sentiment analysis prediction results: Based on the above scheme, this invention decouples private features carrying modality-specific sentiment information from shared features carrying cross-modal common sentiment information through modal feature decoupling learning, providing mutually independent feature spaces for modeling temporal sentiment features at different scales. At the same time, it uses enhanced shared sentiment semantic representation to adaptively calibrate the private features of each modality, further strengthening the sentiment discrimination ability and cross-modal consistency of temporal features at different scales. Finally, through the complete capture of multi-scale temporal features and the synergistic enhancement of cross-modal semantics, it effectively makes up for the temporal modeling defects of existing methods and significantly improves the overall accuracy of sentiment recognition. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a flowchart illustrating the steps of a multimodal sentiment analysis method based on decoupled language focusing and multi-scale cross-modal semantic enhancement, as provided in Embodiment 1 of the present invention.

[0061] Figure 2 This is an overall framework diagram of a multimodal sentiment analysis method based on decoupled language focusing and multi-scale cross-modal semantic enhancement provided in Embodiment 1 of the present invention;

[0062] Figure 3 This is a structural block diagram of a multimodal sentiment analysis system based on decoupled language focusing and multi-scale cross-modal semantic enhancement, provided in Embodiment 2 of the present invention. Detailed Implementation

[0063] This invention provides a multimodal sentiment analysis method and system based on decoupled language focusing and multi-scale cross-modal semantic enhancement. It is applicable to sentiment tendency identification and sentiment intensity prediction of multimodal data containing text, audio and video information. The research object is the emotional state expressed by the speaker in multi-source information such as language content, phonetic prosody and facial expressions. It solves the technical problem that existing multimodal sentiment analysis methods cannot simultaneously capture short-term emotional fluctuations and long-term emotional trends, resulting in low accuracy of sentiment recognition.

[0064] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be noted that in the optional embodiments of the present invention, the object information and other related data involved require the permission or consent of the object when the embodiments of the present invention are applied to specific products or technologies, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards. That is to say, if the embodiments of the present invention involve data related to the object, it needs to be obtained with the authorization and consent of the object, the authorization and consent of relevant departments, and in compliance with relevant laws, regulations, and standards. If personal information is involved in the embodiments, the acquisition of all personal information requires the consent of the individual. If sensitive information is involved, the separate consent of the information subject is required, and the embodiments also need to be implemented with the authorization and consent of the object.

[0065] Please see Figure 1 , Figure 1 The flowchart illustrates the steps of a multimodal sentiment analysis method based on decoupled language focusing and multi-scale cross-modal semantic enhancement, as provided in Embodiment 1 of the present invention.

[0066] This invention provides a multimodal sentiment analysis method based on decoupled language focusing and multi-scale cross-modal semantic enhancement, comprising:

[0067] Step 101: Obtain multimodal sentiment samples, perform feature encoding on the multimodal sentiment samples, and output multimodal sentiment features.

[0068] Multimodal sentiment samples include text sequences, audio feature sequences, and visual feature sequences.

[0069] Multimodal emotion features include text modality initial features, audio modality initial features, and visual modality initial features.

[0070] It should be noted that the text sequence extracts contextual semantic features through a pre-trained language model, while the audio and visual feature sequences are adapted for dimensionality through convolutional projection. The three types of modal features are finally aligned to a unified hidden dimension, thus completing the output of multimodal sentiment features.

[0071] Further, step 101 may include the following sub-steps:

[0072] S11. Perform contextual semantic encoding on the text sequence, and then perform dimensional alignment by passing it through the Dropout operator and one-dimensional convolution to obtain the initial features of the text modality.

[0073] S12. Perform dimensional projection on the audio feature sequence and the visual feature sequence respectively to obtain the initial features of the audio modality and the initial features of the visual modality.

[0074] It should be noted that basic features are extracted from different modal information (text, audio, video) to build a preliminary understanding of each modality. The feature extraction method is as follows:

[0075] a) Text Feature Extraction: For primary text features, this invention employs the RoBERTa (Robustly Optimized BERT Pretraining Approach) pre-trained model to extract contextual semantic features, capturing sentiment polarity, negation, transition, and contextual dependencies within the text. Subsequently, the RoBERTa output is processed through dropout and one-dimensional convolutional projection to map it to a unified hidden dimension, yielding the text modality features, i.e., the initial text modality features.

[0076] ;

[0077] in, This represents the input text sequence. RoBERTa() indicates that the pre-trained language model performs contextual semantic encoding on the text sequence, and Conv1D() represents a one-dimensional convolutional projection operation used to map the text semantic representation to a unified hidden dimension. This represents the initial features of the final text modality.

[0078] Further, b) Audio and video feature extraction: For audio and video modalities, acoustic and visual feature sequences are first obtained, and then mapped to a hidden dimension consistent with the text modal through a one-dimensional convolutional projection layer to obtain audio embedding representation and visual embedding representation, i.e., initial features of the audio modal and initial features of the visual modal.

[0079] ;

[0080] in, This represents the input audio feature sequence. This represents the input video or visual feature sequence. This represents the initial features of the audio modalities obtained after projection. This represents the initial features of the video modalities obtained after projection. The three modalities are uniformly represented as... form, For sequence length, The batch size, To unify the dimensions of hidden layers.

[0081] Step 102: Perform modal feature decoupling learning on the multimodal sentiment features to separate them into multimodal private features and multimodal shared features.

[0082] Multimodal private features include text private features, audio private features, and visual private features.

[0083] Multimodal shared features include text-shared features, audio-shared features, and visual-shared features.

[0084] It should be noted that the unique emotional expression information of each modality is extracted by an independent private encoder, and the common emotional semantic information across modalities is extracted by a shared encoder with shared parameters. The two types of features are decoupled and split by parallel encoding with dual encoders, and finally multimodal private features and multimodal shared features are obtained.

[0085] Furthermore, step 102 may include the following sub-steps:

[0086] S21. Perform dedicated feature learning on the initial features of the text modality, the initial features of the audio modality, and the initial features of the visual modality respectively to obtain text-private features, audio-private features, and visual-private features;

[0087] S22. A shared encoder with shared parameters is used to encode the initial features of the text modality, the initial features of the audio modality, and the initial features of the visual modality, respectively. Combined with the shared similarity constraints in the training phase, it learns the common emotional semantics across modalities, and obtains the shared features of the text, audio, and visual modalities.

[0088] It should be noted that, in order to study both cross-modal commonalities and individual modal characteristics simultaneously, the model divides the temporal features of each modality into two relatively independent information domains: the modality-private information domain and the modality-shared information domain.

[0089] For the private information domain, to more precisely capture the uniqueness of different modalities, this invention introduces three independent private encoders, which perform private representation learning for text, audio, and video respectively. The private encoders retain modality-specific information such as emotion words in text, intonation variations in audio, and facial expressions and gestures in video, providing differentiated semantic representations for subsequent language-focused cross-modal fusion.

[0090] ;

[0091] in, This represents the private encoder corresponding to the m-th mode. Representing text, audio, and visual respectively. This represents modal private features, including text private features, audio private features, and visual private features.

[0092] For the shared information domain, in order to extract common information reflecting emotional states across different modalities, this invention employs a shared encoder to uniformly model text, audio, and video features, obtaining shared semantic representations for the three modalities. This includes text-sharing features, audio-sharing features, and visual-sharing features:

[0093] ;

[0094] in, This is a modality-shared encoder. To ensure a clear division of labor between shared and private features, during the training phase, this invention concatenates the private and shared features of the same modality and inputs them into the decoder to reconstruct the original projected features of the corresponding modality.

[0095] ;

[0096] Meanwhile, through specific reconstruction constraints, orthogonal constraints, and shared similarity constraints, the private features retain modality-specific information as much as possible, and the shared features maintain consistency in the cross-modal sentiment semantic space. That is, this invention introduces specific reconstruction constraints, orthogonal constraints, and shared similarity constraints to form a joint loss function. Through backpropagation, the network parameters of each encoder and decoder are iteratively optimized, driving the private features to retain modality-specific information as much as possible and reducing semantic redundancy with shared features. At the same time, it ensures that the shared features of different modalities maintain a consistent distribution in the cross-modal sentiment semantic space, ensuring that the private features and shared features have clear division of labor and clear semantic boundaries.

[0097] Among them, the reconstruction constraint is used to reconstruct the original modal projection features based on private and shared features, ensuring the information integrity of the features after decoupling; the specific reconstruction constraint is used to re-input the reconstructed features into the corresponding private encoder, so that the reconstructed private representation is consistent with the original private representation, thereby stabilizing the modal specific information.

[0098] Orthogonal constraints are loss terms used to distinguish the semantic boundaries between private and shared features. By constraining the vector orthogonality of private and shared features under the same modality, semantic redundancy between the two types of features is reduced, ensuring a clear division of functions between them.

[0099] Shared similarity constraint is a loss term used to constrain the cross-modal consistency of shared features. By narrowing the distance between shared features from different modalities in the semantic space, it enables shared features to represent common sentiment semantics across modalities, thereby improving the cross-modal generality of shared features.

[0100] Step 103: Perform shared sentiment semantic enhancement on the multimodal shared features and output the enhanced shared sentiment semantic representation.

[0101] It should be noted that the shared features of the three modalities are first subjected to self-attention enhancement and attention pooling respectively to obtain the shared pooling representation of each modality. Then, the three pooling representations are concatenated and subjected to fusion mapping and standardization processing by residual multilayer perceptron to finally output the enhanced shared sentiment semantic representation.

[0102] Furthermore, step 103 may include the following sub-steps:

[0103] S31. Perform self-attention enhancement and learnable attention pooling operations on the text shared features, audio shared features, and visual shared features respectively to obtain the text shared pooling representation, audio shared pooling representation, and visual shared pooling representation;

[0104] S32. The text shared pooling representation, audio shared pooling representation, and visual shared pooling representation are concatenated in terms of feature dimensions, and then input into the residual multilayer perceptron to complete feature mapping and enhancement, so as to obtain a preliminary fused shared sentiment semantic representation.

[0105] S33. Perform linear projection and layer normalization processing on the initially fused shared sentiment semantic representation in sequence, and output the enhanced shared sentiment semantic representation.

[0106] It should be noted that after obtaining the shared features of the three modalities, this invention does not directly use the last time step as the shared summary. Instead, it performs self-attention enhancement on the text shared features, audio shared features, and video shared features respectively, and uses learnable attention pooling to generate modality-level shared representations.

[0107] ;

[0108] ;

[0109] ;

[0110] in, This represents the hidden state of a shared feature sequence of a certain modality at time step t. and Represents the learnable attention pooling parameters; This represents the attention score at time step t; This represents the attention weights at time step t after softmax normalization. This represents the modal-level shared pooling representation obtained by weighted summation of the modal shared feature sequences.

[0111] Furthermore, the three shared pooling representations are then concatenated and input into a residual multilayer perceptron to obtain an enhanced shared sentiment semantic representation:

[0112] ;

[0113] ;

[0114] in, These represent the shared pooled representations of the text modality, audio modality, and video modality after self-attention enhancement and attention pooling, respectively. The MLP() method represents the shared fusion representation obtained by concatenating the shared pooling representations of the three modalities; MLP() represents the residual multilayer perceptron mapping. This represents the shared sentiment semantic representation obtained after reinforcement by a residual multilayer perceptron, which is used as a semantic hub for subsequent sharing to private gated feedback.

[0115] This shared representation is further processed through linear projection and layer normalization to form a semantic hub for subsequent sharing to private gated feedback.

[0116] Step 104: Generate a multimodal calibration representation based on multimodal private features and enhanced shared sentiment semantic representation.

[0117] Multimodal calibration representations include text calibration representations, audio-related calibration representations, and visual-related calibration representations.

[0118] It should be noted that this invention first uses text-private features as the primary target sequence and audio and visual private features as auxiliary modalities. Content projection is performed sequentially on the private features of the two auxiliary modalities to obtain the source modal content representation. Attention weights are calculated by assigning mappings and weighted aggregation is performed on the source modal content to obtain the global semantic descriptor for the corresponding modality. Then, the reception weight of the global semantic descriptor at each time step of the text sequence is calculated, and the sentiment semantics of the auxiliary modalities are distributed to the corresponding positions in the text sequence to obtain the cross-modal context representations corresponding to audio and visual information. Information injection gating is combined to control the semantic injection intensity. After residual scaling, Dropout, and layer normalization, the high-level sequence of the text itself and the language-related high-level sequence of the audio enhancement are obtained. The process involves three high-level sequences related to language, including sequential and visual enhancements. These sequences are then divided into multiple groups based on their feature channels. One-dimensional deep convolutions with different kernel sizes and dilation rates are used to extract emotional change features at different time scales. After concatenating these multi-scale features, group normalization and gating functions are applied to generate temporal attention weights. The original high-level sequences are then adaptively recalibrated and processed sequentially through temporal pooling, a residual multilayer perceptron, and a projection layer to obtain three high-level private representations. Finally, the enhanced shared emotional semantic representation is projected onto the feature spaces corresponding to text, audio, and vision, respectively. This is then concatenated with the high-level private representations of the same modality, and gating weights are calculated. Based on these weights, the shared semantics are adaptively injected into each modality's private representation, ultimately generating a multimodal calibrated representation. This process simultaneously captures short-term local emotional fluctuations and long-term overall emotional trends, mitigating redundant interference between shared and private features, effectively improving the accuracy of emotional feature discrimination, and addressing the problems of incomplete emotional information capture and low recognition accuracy caused by single-scale temporal modeling.

[0119] Furthermore, step 104 may include the following sub-steps:

[0120] S41. Perform high-level temporal modeling of text private features using text memory Transformer, and perform global descriptor cross-modal dual attention processing and corresponding memory Transformer modeling on audio private features and visual private features respectively with text private features, outputting the high-level sequence of the text itself, the high-level sequence of language related to audio enhancement, and the high-level sequence of language related to visual enhancement.

[0121] It should be noted that, firstly, audio and visual private features are used as auxiliary modalities, and then global semantic descriptors are generated through content projection and attention weighted aggregation. Then, the emotional information carried by the global semantic descriptors is distributed to the text sequence to obtain two cross-modal enhanced sequences. Subsequently, the text private features and the two cross-modal enhanced sequences are respectively fed into the corresponding memory Transformer to complete high-level temporal modeling, and finally three corresponding high-level sequences are output.

[0122] Further, step S41 may include the following sub-steps:

[0123] S411. Perform content projection processing on the audio private features and visual private features respectively to obtain the audio source modal content representation and the visual source modal content representation;

[0124] S412. Perform assignment mapping and calculate attention weights for the audio source modal content representation and the visual source modal content representation respectively to obtain the audio descriptor attention weight and the visual descriptor attention weight;

[0125] S413. Perform feature aggregation on the audio source modality content representation and audio descriptor attention weights, and on the visual source modality content representation and visual descriptor attention weights, respectively, to obtain the audio global semantic descriptor and the visual global semantic descriptor;

[0126] S414. Calculate the reception weights of the text private features and the audio global semantic descriptor at multiple time steps, and complete the semantic distribution to obtain the audio cross-modal context representation.

[0127] S415. Calculate the reception weights of the text private features and the visual global semantic descriptor at multiple time steps, and complete the semantic distribution to obtain the visual cross-modal context representation.

[0128] S416. After controlling the injection strength of auxiliary semantics into the text private features and audio cross-modal context representation, residual scaling, Dropout, and layer normalization are performed in sequence to obtain the audio-to-text cross-modal enhancement sequence.

[0129] S417. After controlling the injection intensity of auxiliary semantics into the text private features and the visual cross-modal context representation, residual scaling, Dropout, and layer normalization are performed in sequence to obtain the visual-to-text cross-modal enhancement sequence.

[0130] S418. Input the private features of the text into the text memory Transformer to perform high-level temporal modeling and obtain the high-level sequence of the text itself.

[0131] S419. Input the audio-to-text cross-modal enhancement sequence into the audio memory Transformer to complete high-level temporal modeling and obtain the language-related high-level sequence of audio enhancement;

[0132] S4110. Input the visual-to-text cross-modal enhancement sequence into the visual memory Transformer to complete high-level temporal modeling and obtain the visually enhanced language-related high-level sequence.

[0133] It should be noted that in the cross-modal fusion stage, this invention is primarily text-based, with audio and video serving as auxiliary modalities to supplement language semantics. Unlike traditional cross-modal Transformers that directly perform dense token-frame alignment, this invention employs a global descriptor-based cross-modal dual attention mechanism. This mechanism includes an aggregation stage and a distribution stage.

[0134] a) Aggregation Phase: Using private audio or video sequences as the source modality, the source modality content representation is first obtained through content projection, and then generated through assignment mapping. The attention weights of each global semantic descriptor are used to aggregate the complete auxiliary modality sequence into several global sentiment descriptors:

[0135] ;

[0136] ;

[0137] ;

[0138] in, Represents a global semantic descriptor obtained by aggregating from audio or video sequences; This represents the weights assigned to the auxiliary modal sequences and the K global semantic descriptors; This represents the modality-specific sequence representation obtained after the m-th auxiliary mode is processed by the private encoder; Indicates the use of auxiliary modality-specific sequences The source modal content representation obtained after content projection; Represents the content projection weight matrix; This indicates the mapping weight matrix for assigning descriptors.

[0139] b) Distribution phase: Using the private text sequence as the target sequence, calculate the allocation weight of different global descriptors received at each time step in the text, and distribute the global semantic descriptors of the auxiliary modality back to the text sequence:

[0140] ;

[0141] ;

[0142] in, This represents the reception weight of each time step in the text sequence for different global semantic descriptors; This represents the cross-modal context representation distributed to the text sequence by the m-th auxiliary modality, i.e., the supplementary sentiment information received by the text sequence from the global semantic descriptor of audio or video, including audio cross-modal context representation and visual cross-modal context representation; The descriptor receives the mapping weight matrix.

[0143] Subsequently, the cross-modal augmentation sequence is obtained through output projection, gating, dropout, residual scaling, and layer normalization:

[0144] ;

[0145] ;

[0146] in, This indicates a cross-modal information injection gating system, used to control the strength of auxiliary modal context information injection into the text sequence; This represents the text-related cross-modal sequence representation after the m-th auxiliary modality enhancement, including audio-to-text cross-modal enhancement sequences and visual-to-text cross-modal enhancement sequences; LN() represents the layer normalization operation; represents the learnable residual scaling factor; Dropout() represents the random deactivation operation.

[0147] Therefore, audio and video information is first compressed into a global semantic descriptor, and then selectively received by the language sequence, which can reduce the impact of weak time alignment and redundant noise on cross-modal fusion.

[0148] Furthermore, all three memory Transformers adopt a multi-layer Transformer encoder structure, model the dependencies between different time steps within the sequence through a multi-head self-attention mechanism, and complete the high-level temporal feature modeling through feedforward networks, residual connections and layer normalization. After multi-layer encoding, the high-level temporal sequence of the corresponding path is output.

[0149] S42. After performing temporal multi-scale semantic relabeling on the high-level sequence of the text itself, the high-level sequence of language related to audio enhancement, and the high-level sequence of language related to visual enhancement, the high-level private representation of the text, the high-level private representation of audio related, and the high-level private representation of visual related are obtained through temporal pooling, residual multilayer perceptron and projection layer processing.

[0150] It should be noted that the three high-level sequences are first split by channel and then multi-time period sentiment features are extracted using deep convolutions of different scales. Temporal attention weights are generated to adaptively recalibrate the original sequences. Then, temporal pooling is used to compress the temporal features into fixed-dimensional vector representations. Subsequently, feature fusion and enhancement are completed through residual multilayer perceptrons. Finally, the projection layer is used to map to the target feature space to obtain the corresponding high-level private representations of the three channels.

[0151] Further, step S42 may include the following sub-steps:

[0152] S421. The feature channels of the high-level sequence of the text itself, the high-level sequence of language related to audio enhancement, and the high-level sequence of language related to visual enhancement are divided into four groups respectively. The emotional change features at multiple time scales are extracted by one-dimensional deep convolution branches with different kernel sizes and dilation rates. The four groups of convolution results are then concatenated to obtain the multi-scale temporal features of the text, the multi-scale temporal features of audio, and the multi-scale temporal features of visual.

[0153] S422. Perform group normalization and gating function calculation on the multi-scale temporal features of text, audio-related multi-scale temporal features, and visual-related multi-scale temporal features respectively to generate the temporal attention weights of the multi-scale temporal features of text, audio-related multi-scale temporal features, and visual-related multi-scale temporal features.

[0154] S423. Based on the temporal attention weights of text multi-scale temporal features, audio-related multi-scale temporal features, and visual-related multi-scale temporal features, adaptive recalibrate the high-level sequences of the text itself, the audio-enhanced language-related high-level sequences, and the visual-enhanced language-related high-level sequences, and output the text recalibrated high-level sequences, audio-related recalibrated high-level sequences, and visual-related recalibrated high-level sequences.

[0155] S424. Perform temporal pooling, residual multilayer perceptron, and projection layer processing on the text recalibration high-level sequence, audio-related recalibration high-level sequence, and visual-related recalibration high-level sequence respectively to obtain the text high-level private representation, audio-related high-level private representation, and visual-related high-level private representation.

[0156] It should be noted that after obtaining the high-level sequence of the language itself, the audio-to-language enhancement sequence, and the video-to-language enhancement sequence, this invention introduces a temporal multi-scale semantic relabeling module. This module transforms the multi-scale attention concept originally used for two-dimensional spatial features into a structure suitable for one-dimensional temporal sequences.

[0157] Specifically, the input sequence is first transformed from Convert to The channels are then divided into four groups, each using one-dimensional depthwise convolution with different kernel sizes and dilation rates to capture sentiment changes across multiple time scales.

[0158] ;

[0159] ;

[0160] in, This represents the i-th channel group obtained after dividing the input sequence features by channel; DWConv1D i() represents the i-th one-dimensional depthwise convolutional branch. Different branches use different kernel sizes and dilation rates to capture emotional changes at different time scales. This represents the temporal feature obtained after the i-th channel group undergoes a one-dimensional depthwise convolution; This represents the multi-scale temporal features obtained by concatenating the four scale branches, including text multi-scale temporal features, audio-related multi-scale temporal features, and visual-related multi-scale temporal features.

[0161] Subsequently, temporal attention weights are generated through group normalization and gating functions, namely, the temporal attention weights for text multi-scale temporal features, audio-related multi-scale temporal features, and visual-related multi-scale temporal features:

[0162] ;

[0163] Finally, residual scaling is used to recalibrate the original sequence:

[0164] ;

[0165] in, The original sequence features representing the input temporal multi-scale semantic relabeling module include the high-level sequence of the text itself, the high-level sequence of language-related features enhanced by audio, and the high-level sequence of language-related features enhanced by visual. This represents the output sequence features enhanced by the temporal multi-scale semantic relabeling module, including high-level text relabeling sequences, high-level audio-related relabeling sequences, and high-level visual-related relabeling sequences. These are learnable residual coefficients used to control the strength of the impact of recalibration results on the original sequence. This design allows the model to automatically highlight emotion triggers in text, intonation variations in audio, and key facial expression frames in video while maintaining model stability.

[0166] After recalibration, temporal pooling is performed on the three recalibrated high-level sequences to compress the temporal features into a fixed-dimensional global feature vector. Temporal pooling can employ global attention pooling, global mean pooling, or piecewise mean pooling. Subsequently, the pooled feature vectors are input into a residual multilayer perceptron, where nonlinear feature mapping and semantic enhancement are performed through multiple fully connected layers and residual transition paths. Finally, the enhanced feature vectors are input into a linear projection layer, mapped to a preset target feature dimension, thereby obtaining high-level private representations for text, audio-related high-level private representations, and visual-related high-level private representations.

[0167] S43. After performing high-level residual MLP and projection layer processing on the high-level private representation of text, audio-related high-level private representation, and visual-related high-level private representation respectively, and then combining the enhanced shared sentiment semantic representation, the shared semantic gating feedback injection processing is carried out to output the text calibration representation, audio-related calibration representation, and visual-related calibration representation.

[0168] It should be noted that the three high-level private representations are first fed into the corresponding high-level residual MLP (Multi-Layer Perceptron) to complete high-order nonlinear semantic enhancement, and then aligned to the feature space that matches the shared semantics through the projection layer. The enhanced shared sentiment semantic representations are then projected onto the feature spaces of the three modalities to obtain the corresponding domain shared semantic vectors. Subsequently, the private feature vectors of the same modality are concatenated with the domain shared semantic vectors and the gating weights are calculated. The degree of injection of shared semantics is dynamically adjusted according to the weights to complete adaptive semantic calibration, and finally the three corresponding calibration representations are output.

[0169] Furthermore, step S43 may include the following sub-steps:

[0170] S431. Perform high-level residual MLP and projection layer processing on the high-level private representation of text, audio-related high-level private representation, and visual-related high-level private representation respectively to obtain the processed high-level private representation of text, the processed high-level private representation of audio, and the processed high-level private representation of visual.

[0171] S432. Project the enhanced shared sentiment semantic representation onto the text, audio, and visual-specific feature spaces respectively to obtain the text domain shared semantic representation, the audio domain shared semantic representation, and the visual domain shared semantic representation.

[0172] S433. The processed high-level private representation of text, the processed high-level private representation of audio, and the processed high-level private representation of vision are concatenated with the corresponding text domain shared semantic representation, audio domain shared semantic representation, and visual domain shared semantic representation, respectively, and then gating mapping is performed to generate text gating weights, audio gating weights, and visual gating weights.

[0173] S434. Based on text gating weights, audio gating weights, and visual gating weights, perform shared semantic adaptive injection on the processed high-level private representations of text, audio-related high-level private representations, and visual-related high-level private representations, and output text calibration representations, audio-related calibration representations, and visual-related calibration representations.

[0174] It should be noted that after obtaining the three high-level private representations of text, audio, and video, this invention further utilizes shared sentiment semantics to calibrate the private representations of each modality. First, the enhanced shared semantic representations are mapped to the feature spaces of text, audio, and video, respectively:

[0175] ;

[0176] in, , , These represent the learnable linear mapping parameters that map the shared semantic representation to the text, audio, and video feature spaces, respectively. , , These represent the shared semantic representations mapped to the text, audio, and video feature spaces, respectively.

[0177] Then, the private representations of each modality are concatenated with the corresponding shared mapping representations, and injection weights are generated using a sigmoid gating mechanism.

[0178] ;

[0179] ;

[0180] ;

[0181] in, , , These represent the three high-level private representations of text, audio, and video respectively, namely the processed high-level private representation of text, the processed high-level private representation of audio, and the processed high-level private representation of video. , , These represent the learnable parameters for text, audio, and video gating mappings, respectively. , , These represent the shared semantic injection gating weights for text, audio, and video modalities, respectively, namely text gating weight, audio gating weight, and visual gating weight.

[0182] Finally, shared semantics are selectively injected into the private representations of each modality through learnable residual coefficients:

[0183] ;

[0184] ;

[0185] ;

[0186] in, , , These represent the text, audio, and video calibration representations obtained after shared semantic gating feedback injection, namely, text calibration representation, audio-related calibration representation, and visual-related calibration representation; , , These represent the learnable residual coefficients for text, audio, and video modalities, respectively, and are used to control the intensity of shared semantic injection.

[0187] This mechanism can adaptively control the injection intensity of shared emotional information based on the reliability of different samples and modalities, thereby mitigating redundant interference between shared features and private features.

[0188] Step 105: Concatenate and fuse the multimodal calibration representation and the enhanced shared sentiment semantic representation for sentiment prediction, and output the final sentiment analysis prediction result.

[0189] It should be noted that the three types of multimodal calibration representations and enhanced shared sentiment semantic representations are first spliced ​​and aggregated along the feature channel dimension to obtain a global fusion feature containing private calibration information and global shared semantics. Then, the fusion feature is subjected to deep semantic optimization through a fully connected structure. Finally, it is connected to the linear output layer to complete the mapping calculation of sentiment attributes and output the final sentiment analysis prediction result.

[0190] Furthermore, step 105 may include the following sub-steps:

[0191] S51. The text calibration representation, audio-related calibration representation, visual-related calibration representation and enhanced shared sentiment semantic representation are concatenated along the feature dimensions to generate a global concatenated and fused feature.

[0192] S52. Input the global splicing and fusion features into the residual fully connected fusion layer to complete the deep semantic optimization and obtain the optimized fusion representation;

[0193] S53. Input the optimized fusion representation into the linear output layer to obtain the final sentiment analysis prediction result.

[0194] It should be noted that after global descriptor cross-modal dual attention, temporal multi-scale semantic recalibration, and shared semantic gating feedback, this invention obtains calibrated text representation, audio representation, video representation, and enhanced shared semantic representation. Subsequently, these four representations are concatenated along the feature dimension and input into a residual fully connected fusion layer.

[0195] ;

[0196] ;

[0197] Finally, the sentiment prediction result is obtained through the output layer:

[0198] ;

[0199] in, This represents the fused representation after enhancement by the residual fully connected fusion layer; and This represents the learnable weights and biases of the output layer; This represents the sentiment prediction result output by the model. The output result can be either a continuous sentiment intensity regression value or converted into a binary, quinary, or septial sentiment result depending on the task requirements.

[0200] It is worth mentioning that during the training phase, this invention retains a multi-head supervision structure, including text-private branch prediction, audio-private branch prediction, video-private branch prediction, shared branch prediction, and final fusion branch prediction. The total loss is composed of a weighted average of task loss, reconstruction loss, specific reconstruction loss, orthogonality loss, and shared similarity loss, thereby simultaneously constraining the effectiveness of shared features, private features, and final fusion features.

[0201] Furthermore, this invention underwent experimental verification, and the experimental results are shown in Table 1, where the superscript "a" indicates that the data comes from the literature, and "b" indicates that the data comes from the original paper. Through experimental verification, this invention has achieved certain results. The experimental parameters are set as follows:

[0202] This invention was experimentally validated on two publicly available multimodal sentiment analysis datasets: CMU-MOSI (Carnegie Mellon University Multimodal Opinion Sentiment and Interaction Dataset) and CMU-MOSEI (Carnegie Mellon University Multimodal Opinion Sentiment and Emotion Intensity Dataset). The input modalities included text, audio, and video. Aligned multimodal feature data was used in the experiments. The CMU-MOSI dataset contained 1284 training samples, 229 validation samples, and 686 test samples; the CMU-MOSEI dataset contained 16326 training samples, 1871 validation samples, and 4659 test samples.

[0203] In terms of models, the text modality uses a RoBERTa pre-trained language model for contextual semantic encoding, which participates in fine-tuning during training; the audio and video modalities are projected onto a unified hidden dimension through one-dimensional convolution. The unified hidden dimension is set to 50 in both experiments, the number of attention heads is set to 10, and the number of Transformer encoding layers is set to 2. In the CMU-MOSI experiment, the temporal multi-scale semantic relabeling module uses a convolution kernel size of [3,5,7,9] and a dilation rate of [1,1,1,1] in the text branch, and a convolution kernel size of [3,3,5,5] and a dilation rate of [1,1,2,2] in the audio and video branches; in the CMU-MOSEI experiment, the text branch also uses a convolution kernel size of [3,5,7,9] and a dilation rate of [1,1,1,1], while the audio and video branches use a convolution kernel size of [3,5,5,7] and a dilation rate of [1,1,1,1].

[0204] For training, the Adam optimizer was used on both datasets with a learning rate of 0.0001 and a batch size of 16. The weight decay was set to 0.005 for the CMU-MOSI experiment and 0.001 for the CMU-MOSEI experiment. The training loss consisted of the final task prediction loss, the prediction loss for each modality branch, the reconstruction loss, the specific reconstruction loss, the orthogonal constraint loss, and the shared similarity loss.

[0205] Analysis of experimental results:

[0206] Based on the Acc2 (Binary Classification Accuracy) and F1 scores, this invention achieved 87.35 and 87.35 respectively on CMU-MOSI, representing improvements of 2.29 and 2.31 compared to the baseline; and 86.02 and 85.97 respectively on CMU-MOSEI, representing improvements of 0.60 and 0.70 compared to the baseline. These results indicate that this invention can improve the ability to determine positive and negative sentiment polarity and maintain good overall classification performance even under class imbalance conditions.

[0207] From the perspective of the Corr (Correlation Coefficient, Pearson Correlation Coefficient) index, this invention improved from 0.7814 to 0.8426 on CMU-MOSI and from 0.7636 to 0.7788 on CMU-MOSEI. Corr reflects the correlation between predicted sentiment intensity and actual sentiment intensity. The improvement in this index indicates that global descriptor cross-modal dual attention and temporal multi-scale semantic recalibration can more accurately model sentiment change trends.

[0208] From the perspective of the MAE (Mean Absolute Error) index, this invention reduced the MAE from 0.7308 to 0.6279 on CMU-MOSI and from 0.5361 to 0.5268 on CMU-MOSEI. A lower MAE indicates a smaller average error between the predicted value and the actual sentiment intensity, suggesting that this invention not only improves classification accuracy but also enhances the precision of sentiment intensity regression prediction.

[0209] In summary, the improvement on the MOSI dataset is more significant, indicating that the present invention has a strong modeling ability for key semantic segments and cross-modal complementary information in small-scale multimodal sentiment data. The MOSEI dataset is larger in scale and has more diverse sample expressions, yet the present invention still achieves stable improvements on Acc2, F1, Corr and MAE, indicating that the proposed method has a certain generalization ability.

[0210] Table 1. Experimental results of the baseline model

[0211]

[0212] For comparison of technical effectiveness, existing technologies can be referenced. Current multimodal sentiment analysis methods typically extract features from different modalities separately and then fuse them through feature concatenation, cross-modal attention, or shared representation learning. However, such methods have the following shortcomings in practical applications:

[0213] Insufficient cross-modal semantic interaction: Existing cross-modal attention methods mostly employ time-step intensive attention computation between text sequences and audio / video frames, i.e., directly calculating the correlation between text tokens and each time segment in the audio / video sequence. This approach relies on good temporal alignment. When the audio / video sequence is long or contains silent segments, expression-irrelevant frames, or background noise frames, attention is easily diverted to invalid segments. It lacks a mechanism for first performing global semantic compression and key sentiment cues aggregation on the auxiliary modality, leading to misalignment and semantic dilution in cross-modal interaction.

[0214] Temporal feature modeling suffers from a single scale: Existing methods typically rely on Transformer global attention, fixed convolutional kernels, or pooling at the last time step to obtain sequence representations, lacking the ability to explicitly model different time windows. Pitch changes, pauses, and accents in audio are usually short-term local variations, while facial expressions and movement trends in video can persist for longer periods. Using a single-scale temporal modeling approach makes it difficult to simultaneously capture short-term emotional fluctuations, moderate-duration semantic changes, and emotional trends over longer timeframes.

[0215] Redundancy exists in the fusion of shared and specific features: Existing shared-specific decoupling methods typically learn modality shared features and modality specific features separately through reconstruction loss, orthogonal loss, etc., but in the final fusion stage, they often use simple concatenation, direct addition, or fixed weight fusion. This approach lacks a gating mechanism to adaptively feed shared sentiment semantics back to each modality specific space, and cannot dynamically control the intensity of shared information injection based on the reliability of different samples and modalities. This easily leads to duplicate expression or mutual interference between shared and specific features, weakening the discriminativeness and robustness of the final fused representation.

[0216] To address the aforementioned issues, this invention provides a multimodal sentiment analysis method based on decoupled language focusing and multi-scale cross-modal semantic enhancement. This method enables shared-specific decoupled modeling of text, audio, and video modalities under the guidance of language semantics. It aggregates and distributes key sentiment cues from auxiliary modalities through global descriptor cross-modal dual attention, enhances the representation of sentiment segments at different time scales through temporal multi-scale semantic recalibration, and reduces redundant interference between shared features and modality-specific features through a shared feature adaptive gating back-injection mechanism, thereby improving the accuracy and reliability of sentiment recognition.

[0217] Specifically, such as Figure 2 As shown, this invention provides a multimodal sentiment analysis method based on decoupled language focusing and multi-scale cross-modal semantic enhancement, aiming to solve the problems of insufficient cross-modal semantic interaction, single scale of temporal sentiment cue capture, and redundant interference between shared features and modality-specific features in existing multimodal sentiment analysis, thereby improving the accuracy and robustness of sentiment recognition.

[0218] The method of this invention includes: firstly, acquiring multimodal sentiment data and extracting features from it to obtain text-based single-modal features, audio-based single-modal features, and visual-based single-modal features. Specifically, semantic features of the text modality can be extracted using a pre-trained language model, while speech acoustic features and visual expression features are extracted for the audio and visual modalities, respectively. Subsequently, these single-modal features are mapped to a unified feature dimension through one-dimensional convolution and input into a modality-specific encoder and a modality-shared encoder, respectively, to obtain the private features and shared semantic features of each modality. The private encoder is used to learn discriminative specific representations in each modality through a decoupling learning mechanism, enabling the text, audio, and visual modalities to form modality-specific features that can represent their own emotional expression characteristics; the shared encoder is used to learn consistent sentiment semantic representations among different modalities, enabling the three modalities to form aligned common features in a shared semantic space.

[0219] In the cross-modal fusion stage, a language modality-led cross-modal dual-attention mechanism is adopted. This mechanism uses audio and visual modalities as auxiliary information sources, interacting with language-specific features respectively. First, a semantic aggregation process generates multiple global semantic descriptors from the auxiliary modal sequence to compress and preserve key emotional cues in the audio or visual sequences. Subsequently, a semantic distribution process adaptively receives global semantic descriptions from the auxiliary modalities at each temporal position in the language sequence, thereby obtaining language-guided audio-enhanced and visual-enhanced representations. This cross-modal dual-attention mechanism avoids the problems of noise and sequence length differences that affect traditional time-step cross-attention, and can more stably complete cross-modal semantic alignment and information supplementation.

[0220] After obtaining the cross-modal augmented sequences, a multi-scale temporal semantic relabeling mechanism is introduced. Specifically, the text, audio augmented sequences, and visual augmented sequences are divided into multiple channel groups, and one-dimensional deep convolutions with different kernel scales are used to extract sentiment change features over short, medium, and long time periods. Subsequently, temporal attention weights are generated through normalization and gating functions to adaptively relabel the original sequence features, and residual scaling is used to maintain training stability. This mechanism can simultaneously capture local emotional fluctuations and semantic trends over longer time periods, enhancing the model's ability to model complex temporal sentiment expressions.

[0221] In the shared semantic fusion stage, self-attention modeling and attention pooling are performed on the shared features of text, audio, and vision respectively to obtain unified shared semantic fusion features. Subsequently, the shared semantic fusion features are projected onto the specific spaces of each modality and adaptively injected into the high-level private representations of text, audio, and vision through a gating mechanism. This shared-to-private gating fusion mechanism can dynamically control the injection intensity of shared information according to the current representation state of each modality, reducing redundant interference between shared and private features while improving the discriminativeness of the fused features.

[0222] Finally, the enhanced text-specific representations, audio-specific representations, visual-specific representations, and shared semantic fusion features are concatenated and input into the residual fully connected fusion layer and the prediction layer, outputting the sentiment analysis results. The overall model achieves full fusion of textual, audio, and visual sentiment information through the synergistic effects of modal decoupling, cross-modal dual attention, multi-scale temporal recalibration, and shared semantic gating backinjection, thereby improving the accuracy and reliability of multimodal sentiment recognition.

[0223] In this embodiment of the invention, a multimodal sentiment analysis method based on decoupled language focusing and multi-scale cross-modal semantic enhancement is provided. The method acquires multimodal sentiment samples, encodes their features, and outputs multimodal sentiment features. Modal feature decoupling learning is performed on the multimodal sentiment features to separate them into multimodal private features and multimodal shared features. Shared sentiment semantic enhancement is applied to the shared features to output an enhanced shared sentiment semantic representation. A multimodal calibration representation is generated based on the multimodal private features and the enhanced shared sentiment semantic representation. Finally, the multimodal calibration representation and the enhanced shared sentiment semantic representation are concatenated and fused for sentiment prediction. The final sentiment analysis prediction result is output. Based on the above scheme, this invention decouples private features carrying modality-specific sentiment information from shared features carrying cross-modal common sentiment information through modal feature decoupling learning, providing mutually independent feature spaces for modeling temporal sentiment features at different scales. At the same time, it uses enhanced shared sentiment semantic representation to adaptively calibrate the private features of each modality, further strengthening the sentiment discrimination ability and cross-modal consistency of temporal features at different scales. Finally, through the complete capture of multi-scale temporal features and the synergistic enhancement of cross-modal semantics, it effectively makes up for the temporal modeling defects of existing methods and significantly improves the overall accuracy of sentiment recognition.

[0224] Please see Figure 3 , Figure 3 This is a structural block diagram of a multimodal sentiment analysis system based on decoupled language focusing and multi-scale cross-modal semantic enhancement, provided in Embodiment 2 of the present invention.

[0225] This invention provides a multimodal sentiment analysis system based on decoupled language focusing and multi-scale cross-modal semantic enhancement, comprising:

[0226] The acquisition module 301 is used to acquire multimodal sentiment samples, perform feature encoding on the multimodal sentiment samples, and output multimodal sentiment features;

[0227] Learning module 302 is used to perform modal feature decoupling learning on multimodal sentiment features, and to separate them into multimodal private features and multimodal shared features;

[0228] Enhancement module 303 is used to enhance the shared sentiment semantics of multimodal shared features and output an enhanced shared sentiment semantic representation;

[0229] The generation module 304 is used to generate a multimodal calibration representation based on multimodal private features and enhanced shared sentiment semantic representation;

[0230] The prediction module 305 is used to splice and fuse the multimodal calibration representation and the enhanced shared sentiment semantic representation for sentiment prediction, and output the final sentiment analysis prediction result.

[0231] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system and modules described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0232] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0233] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0234] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0235] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0236] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal sentiment analysis method based on decoupled language focusing and multi-scale cross-modal semantic enhancement, characterized in that, include: Acquire multimodal sentiment samples, encode features of the multimodal sentiment samples, and output multimodal sentiment features; The multimodal emotion samples include text sequences, audio feature sequences, and visual feature sequences; the multimodal emotion features include initial text modal features, initial audio modal features, and initial visual modal features; Modal feature decoupling learning is performed on the multimodal emotion features to separate them into multimodal private features and multimodal shared features; the multimodal private features include text private features, audio private features, and visual private features; the multimodal shared features include text shared features, audio shared features, and visual shared features; The multimodal shared features are subjected to shared sentiment semantic enhancement to output an enhanced shared sentiment semantic representation; Based on the multimodal private features and the enhanced shared sentiment semantic representation, a multimodal calibration representation is generated; The multimodal calibration representation and the enhanced shared sentiment semantic representation are concatenated and fused together for sentiment prediction, and the final sentiment analysis prediction result is output. The multimodal calibration representation includes text calibration representation, audio-related calibration representation, and visual-related calibration representation; the step of generating a multimodal calibration representation based on the multimodal private features and the enhanced shared sentiment semantic representation includes: The text-private features are modeled using a high-level temporal modeling method with a text memory Transformer. The audio-private features and the visual-private features are then modeled using a global descriptor cross-modal dual attention method and a corresponding memory Transformer model, respectively. The output consists of a high-level sequence of the text itself, a high-level sequence of language-related features enhanced by audio, and a high-level sequence of language-related features enhanced by visual. After performing temporal multi-scale semantic relabeling on the high-level sequence of the text itself, the high-level sequence of language related to the audio enhancement, and the high-level sequence of language related to the visual enhancement, and then processing them through temporal pooling, residual multilayer perceptron and projection layer, we obtain the high-level private representation of the text, the high-level private representation of the audio related, and the high-level private representation of the visual related. After performing high-level residual MLP and projection layer processing on the high-level private representation of text, the high-level private representation of audio, and the high-level private representation of vision respectively, and then combining the enhanced shared sentiment semantic representation, shared semantic gating feedback injection processing is carried out to output text calibration representation, audio-related calibration representation, and vision-related calibration representation. After performing high-level residual MLP and projection layer processing on the high-level private representation of text, the high-level private representation of audio, and the high-level private representation of vision respectively, and then combining the enhanced shared sentiment semantic representation, shared semantic gating feedback injection processing is carried out to output text calibration representation, audio-related calibration representation, and visual-related calibration representation, including: High-level residual MLP and projection layer processing are performed on the text high-level private representation, the audio-related high-level private representation, and the visual-related high-level private representation respectively to obtain the processed text high-level private representation, the processed audio-related high-level private representation, and the processed visual-related high-level private representation. The enhanced shared sentiment semantic representation is projected onto the text, audio, and visual-specific feature spaces respectively to obtain the text domain shared semantic representation, the audio domain shared semantic representation, and the visual domain shared semantic representation. The processed high-level private representation of text, the processed high-level private representation of audio, and the processed high-level private representation of vision are concatenated with the corresponding shared semantic representation of text domain, shared semantic representation of audio domain, and shared semantic representation of vision domain, respectively, and then gating mapping is performed to generate text gating weights, audio gating weights, and vision gating weights. Based on the text gating weight, the audio gating weight, and the visual gating weight, shared semantic adaptive injection is performed on the processed high-level private representation of text, the processed high-level private representation of audio, and the processed high-level private representation of vision, and the text calibration representation, audio-related calibration representation, and vision-related calibration representation are output.

2. The multimodal sentiment analysis method based on decoupled language focusing and multi-scale cross-modal semantic enhancement according to claim 1, characterized in that, The process of feature encoding of multimodal emotion samples and outputting multimodal emotion features includes: The text sequence is semantically encoded in context, and then dimensional alignment is achieved by sequentially passing it through the Dropout operator and one-dimensional convolution to obtain the initial features of the text modality. The audio feature sequence and the visual feature sequence are dimensionally projected respectively to obtain the initial features of the audio modality and the initial features of the visual modality.

3. The multimodal sentiment analysis method based on decoupled language focusing and multi-scale cross-modal semantic enhancement according to claim 2, characterized in that, The process of performing modal feature decoupling learning on the multimodal sentiment features to separate them into multimodal private features and multimodal shared features includes: Dedicated feature learning is performed on the initial features of the text modality, the initial features of the audio modality, and the initial features of the visual modality to obtain text-private features, audio-private features, and visual-private features respectively; A shared encoder with shared parameters is used to encode the initial features of the text modality, the initial features of the audio modality, and the initial features of the visual modality, respectively. Combined with the shared similarity constraint in the training phase, it learns cross-modal common sentiment semantics to obtain shared text features, shared audio features, and shared visual features.

4. The multimodal sentiment analysis method based on decoupled language focusing and multi-scale cross-modal semantic enhancement according to claim 3, characterized in that, The step of enhancing the shared sentiment semantics of the multimodal shared features and outputting an enhanced shared sentiment semantic representation includes: Self-attention enhancement and learnable attention pooling operations are sequentially performed on the text-shared features, the audio-shared features, and the visual-shared features to obtain text-shared pooling representations, audio-shared pooling representations, and visual-shared pooling representations, respectively. The text shared pooling representation, the audio shared pooling representation, and the visual shared pooling representation are concatenated by feature dimensions, and then input into a residual multilayer perceptron to complete feature mapping and enhancement, thereby obtaining a preliminary fused shared sentiment semantic representation; The initially fused shared sentiment semantic representation is sequentially subjected to linear projection and layer normalization processing to output an enhanced shared sentiment semantic representation.

5. The multimodal sentiment analysis method based on decoupled language focusing and multi-scale cross-modal semantic enhancement according to claim 1, characterized in that, The text-private features are subjected to high-level temporal modeling using a text memory Transformer. Global descriptor cross-modal dual attention processing and corresponding memory Transformer modeling are then performed on the audio-private features and the visual-private features, respectively, in conjunction with the text-private features. The output includes a high-level sequence of the text itself, a high-level sequence of language-related features enhanced by audio, and a high-level sequence of language-related features enhanced by visual, including: Content projection processing is performed on the audio private features and the visual private features respectively to obtain the audio source modal content representation and the visual source modal content representation; Assignment mapping and attention weights are performed on the audio source modal content representation and the visual source modal content representation respectively to obtain the audio descriptor attention weight and the visual descriptor attention weight; Feature aggregation is performed on the audio source modality content representation and the audio descriptor attention weight, and on the visual source modality content representation and the visual descriptor attention weight, respectively, to obtain the audio global semantic descriptor and the visual global semantic descriptor; The text private features and the audio global semantic descriptor are subjected to reception weights at multiple time steps, and semantic distribution is completed to obtain the audio cross-modal context representation. The text-private features and the visual global semantic descriptor are subjected to reception weights at multiple time steps, and semantic distribution is completed to obtain a visual cross-modal context representation. After controlling the injection strength of auxiliary semantics into the text private features and the audio cross-modal context representation, residual scaling, Dropout, and layer normalization are performed sequentially to obtain the audio-to-text cross-modal enhancement sequence. After controlling the injection intensity of auxiliary semantics into the text private features and the visual cross-modal context representation, residual scaling, Dropout, and layer normalization are performed sequentially to obtain the visual-to-text cross-modal enhancement sequence. The text-private features are input into a text memory Transformer for high-level temporal modeling to obtain the text's own high-level sequence. The audio-to-text cross-modal enhancement sequence is input into the audio memory Transformer to complete high-level temporal modeling, resulting in a high-level audio-enhanced language-related sequence. The vision-to-text cross-modal enhancement sequence is input into the visual memory Transformer to complete high-level temporal modeling, resulting in a high-level sequence of language-related visual enhancements.

6. The multimodal sentiment analysis method based on decoupled language focusing and multi-scale cross-modal semantic enhancement according to claim 1, characterized in that, After performing temporal multi-scale semantic relabeling on the high-level sequence of the text itself, the high-level sequence of language relevance enhanced by audio, and the high-level sequence of language relevance enhanced by vision, and then processing through temporal pooling, residual multilayer perceptron, and projection layer, high-level private representations of text, audio relevance, and vision relevance are obtained, including: The feature channels of the high-level sequence of the text itself, the high-level sequence of language related to the audio enhancement, and the high-level sequence of language related to the visual enhancement are divided into four groups respectively. The emotional change features at multiple time scales are extracted by one-dimensional deep convolution branches with different kernel sizes and dilation rates. The four groups of convolution results are then concatenated to obtain the multi-scale temporal features of the text, the multi-scale temporal features of the audio, and the multi-scale temporal features of the visual. Group normalization and gating function calculation are performed sequentially on the text multi-scale temporal features, the audio-related multi-scale temporal features, and the visual-related multi-scale temporal features to generate temporal attention weights for the text multi-scale temporal features, the audio-related multi-scale temporal features, and the visual-related multi-scale temporal features, respectively. Based on the temporal attention weights of the text multi-scale temporal features, the audio-related multi-scale temporal features, and the visual-related multi-scale temporal features, adaptive recalibration is performed on the text's own high-level sequence, the audio-enhanced language-related high-level sequence, and the visual-enhanced language-related high-level sequence, outputting the text recalibration high-level sequence, the audio-related recalibration high-level sequence, and the visual-related recalibration high-level sequence. Temporal pooling, residual multilayer perceptron, and projection layer processing are performed on the text recalibration high-level sequence, the audio-related recalibration high-level sequence, and the visual-related recalibration high-level sequence, respectively, to obtain the text high-level private representation, the audio-related high-level private representation, and the visual-related high-level private representation.

7. The multimodal sentiment analysis method based on decoupled language focusing and multi-scale cross-modal semantic enhancement according to claim 1, characterized in that, The process of concatenating and fusing the multimodal calibration representation and the enhanced shared sentiment semantic representation for sentiment prediction, and outputting the final sentiment analysis prediction result, includes: The text calibration representation, the audio-related calibration representation, the visual-related calibration representation, and the enhanced shared sentiment semantic representation are concatenated along their feature dimensions to generate a global concatenated and fused feature. The global splicing and fusion features are input into the residual fully connected fusion layer to complete deep semantic optimization, resulting in the optimized fusion representation; The optimized fusion representation is input into the linear output layer to obtain the final sentiment analysis prediction result.

8. A multimodal sentiment analysis system based on decoupled language focusing and multi-scale cross-modal semantic enhancement, applied to the multimodal sentiment analysis method based on decoupled language focusing and multi-scale cross-modal semantic enhancement as described in claim 1, characterized in that, include: The acquisition module is used to acquire multimodal sentiment samples, perform feature encoding on the multimodal sentiment samples, and output multimodal sentiment features; The learning module is used to perform modal feature decoupling learning on the multimodal emotion features, and to separate them into multimodal private features and multimodal shared features; The enhancement module is used to perform shared sentiment semantic enhancement on the multimodal shared features and output an enhanced shared sentiment semantic representation; A generation module is used to generate a multimodal calibration representation based on the multimodal private features and the enhanced shared sentiment semantic representation; The prediction module is used to concatenate and fuse the multimodal calibration representation and the enhanced shared sentiment semantic representation and perform sentiment prediction, outputting the final sentiment analysis prediction result.

Images