A multi-modal sentiment analysis method based on multi-layer feature perception and emotional feature state transition

Abstract

The application relates to a multi-modal sentiment analysis method based on multi-layer feature perception and emotion feature state transition, and belongs to the field of natural language processing. The method comprises the following steps: 1. text semantic features are extracted by using a pre-trained language model, visual and audio features are sequentially coded and linearly mapped to form a unified multi-modal input representation; 2. through a cross-modal feature translation module, non-text emotion features enhanced by text are obtained; 3. a cross-modal interaction mechanism is introduced to update the global multi-modal context and the local single-modal feature; 4. a public emotion feature state transition module is introduced to construct stable cross-modal emotion correlation through a state transition process; 5. a double-layer feature recovery mechanism is introduced to enhance the robustness of the missing mode through low-layer feature reconstruction and high-layer feature attraction. Through feature translation, emotion feature state transition and double-layer features, the model has strong emotion recognition capability.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and relates to a multimodal sentiment analysis method based on the fusion of text enhancement and modal completion perception. Background Technology

[0002] Multimodal sentiment analysis (MMA) aims to predict the emotional polarity or intensity during verbal communication. Compared to sentiment analysis methods that rely solely on a single modality, MMA can simultaneously utilize textual semantic information, prosodic features such as pitch, energy, and speech rate, as well as visual behavioral features such as facial expressions, eye movements, and head movements, thus forming a more complete representation of emotional states. In real-world communication scenarios, human emotional expression is typically implicit, complementary, and cross-modal. However, some non-textual modalities may also be affected by environmental noise, occlusion, acquisition errors, or modality gaps, leading to insufficient or unstable emotional cues. Therefore, relying solely on simple feature concatenation or single-modality judgment is insufficient to fully uncover the deep emotional connections between different modalities.

[0003] Multimodal sentiment analysis has significant application value in fields such as natural language processing, intelligent human-computer interaction, affective computing, and social intelligence analysis. By jointly modeling multi-source information such as text, acoustics, and vision, the system can more accurately understand users' emotional tendencies, attitude changes, and underlying intentions during communication, thereby improving the ability of intelligent systems to perceive and understand complex human behavior. In educational scenarios, multimodal sentiment analysis can combine learners' language feedback, speech patterns, and facial expression changes to provide a basis for personalized teaching interventions and learning feedback. In healthcare scenarios, this technology can be used to assist in analyzing individuals' semantic expressions, speech prosody, and facial expressions during interviews, consultations, or daily interactions, providing objective references for assessing psychological states such as depression, anxiety, and stress. In the fields of social media and public safety, multimodal sentiment analysis can comprehensively understand information such as video, audio, and text, and is used for public opinion situation awareness, group emotion analysis, modeling of emotion propagation patterns, and early warning of potential risks.

[0004] However, multimodal sentiment analysis still faces many challenges. On the one hand, textual, acoustic, and visual information may be affected by noise, occlusion, recognition errors, or device malfunctions during acquisition, transmission, or preprocessing, leading to missing, degraded, or incomplete features of a particular modality, thus weakening the model's ability to comprehensively perceive emotional cues. On the other hand, different modalities exhibit inherent heterogeneity, with inconsistent feature dimensions, time scales, and semantic expressions, making direct fusion prone to cross-modal alignment difficulties and information redundancy. Therefore, how to effectively mine auxiliary emotional information in visual and acoustic modalities while maintaining the dominant role of textual semantics, and how to achieve stable cross-modal interaction and robust fusion under conditions of modality loss, modal noise, and modal inconsistency, is one of the key research focuses of current multimodal sentiment analysis methods.

[0005] Existing multimodal sentiment analysis methods typically use the text modality as the dominant information source. While this fully utilizes the stable expressive power of textual semantics, it tends to overlook fine-grained emotional cues contained in non-textual modalities such as visual expressions and acoustic prosody. Some cross-modal enhancement methods improve modal interaction capabilities by injecting visual or acoustic information into text representations, but model performance tends to degrade when non-textual modalities are missing or interfered with by noise. Another type of modal completion or feature reconstruction method enhances model robustness through strategies such as random modal discarding and missing feature recovery, but it mainly focuses on recovering missing information and often fails to fully characterize the deep emotional correlations and complementary relationships between modalities.

[0006] Based on the above discussion, we propose a multimodal sentiment analysis method based on multi-layer feature perception and sentiment feature state transition. This method first extracts text semantic representations using a pre-trained language model and performs temporal encoding and linear mapping on acoustic and visual features to form a trimodal input representation under a unified common feature space. Then, using the text modality as the semantic core, a cross-modal feature translation module is constructed, generating text-enhanced non-textual interaction features through cross-modal attention. Subsequently, a cross-modal interaction mechanism is introduced, utilizing global multimodal context and local modal features for bidirectional updates. Building upon this, a common sentiment feature state transition module is designed to generate stable visual and acoustic common sentiment features. Finally, a missing view is constructed using random modal masks, and multi-layer feature constraints are introduced for joint optimization. Through this mechanism, we conducted experiments on commonly used datasets and achieved state-of-the-art and stable performance. Summary of the Invention

[0007] The present invention aims to solve the problems existing in the prior art and proposes a method to improve the accuracy of multimodal sentiment analysis.

[0008] To address the problems encountered in existing research, the present invention provides the following solution:

[0009] A multimodal sentiment analysis method based on multi-layer feature perception and emotional feature state transition includes the following steps:

[0010] Step 1: Extract semantic representation of text using a pre-trained language model, and perform temporal encoding and linear mapping on acoustic and visual features to form a unified multimodal input representation;

[0011] Step 2: Construct a cross-modal feature translation module, using text features as the semantic core, generating text-enhanced non-text sentiment features through cross-modal interaction, and reducing modal heterogeneity through distribution constraints and reconstruction constraints;

[0012] Step 3: Introduce a cross-modal interaction module, and use discourse-level global multimodal context and local features of each modality for bidirectional updates;

[0013] Step 4: Design a public sentiment feature state transition module to generate stable visual and acoustic public sentiment features through controlled state transitions;

[0014] Step 5: Utilize the two-layer feature recovery and sentiment prediction module to enhance the robustness of the model through multi-layer feature constraints, and achieve joint optimization of sentiment classification and sentiment intensity regression on the fusion representation.

[0015] Furthermore, the specific process of step one is as follows:

[0016] For the input utterance sample, suppose it contains a text sequence Visual frame-level features Harmony and acoustic frame-level features First, a pre-trained language model is used to perform contextual semantic encoding on the text sequence to obtain text features:

[0017]

[0018] in, Indicates the parameters of the pre-trained language model. and The parameters are learnable mappings; for the visual and acoustic modalities, a temporal encoder is used to extract their dynamic emotional features, which are then mapped to a common feature space through a linear transformation:

[0019]

[0020]

[0021] in, Indicates visual or acoustic input features. This represents the timing encoder corresponding to the mode. These are the intermediate features after encoding. These are the mapped visual or acoustic features; ultimately, a unified-dimensional multimodal representation is obtained.

[0022]

[0023] in, These represent the text, visual, and acoustic modalities in the common feature space, respectively.

[0024] Furthermore, the specific process of step two is as follows:

[0025] Text features Using semantics as the core, cross-modal feature translation relationships are constructed between text and visual features, and between text and acoustic features, respectively, to alleviate semantic heterogeneity between different modalities and obtain text-enhanced non-text sentiment representations. First, a cross-modal attention mechanism is adopted, using text features as query terms and visual or acoustic features as keys and values, to obtain text-guided non-text interaction features:

[0026]

[0027] in, Visual interaction features derived from text features or auditory characteristics , This represents the learnable parameters of the corresponding cross-modal attention module; subsequently, a back-translation branch is introduced to constrain the cross-modal translation process, using non-textual features as query terms and textual features as keys and values ​​to generate corresponding pseudo-non-textual features:

[0028]

[0029] in Indicates pseudo-visual features, These are the learnable parameters for the reverse cross-modal attention module. To reduce the bias caused by the inconsistency in the distribution of different modalities during cross-modal translation, a distribution-constrained loss is used to ensure that the non-textual interaction features after text guidance are consistent with the textual features in the common space.

[0030]

[0031] in, Represents the Wasserstein distance function. To constrain the distribution differences between cross-modal interaction features and text features, a reconstruction constraint loss is introduced to ensure that the pseudo-non-text features generated during reverse cross-modal translation retain the effective information of the original non-text modality.

[0032]

[0033] in, This is used to measure the reconstruction error between pseudo-non-textual features and original non-textual features; here, the cross-modal feature translation constraint loss is expressed as:

[0034]

[0035] Furthermore, step three specifically includes the following process:

[0036] Since textual, visual, and acoustic modalities typically have different sampling frequencies and sequence lengths, directly employing pairwise interactions between local features can easily introduce a large amount of redundant computation and may cause the model to focus on irrelevant local noise. Therefore, this method does not directly perform dense interactions on the local sequences of all modalities. Instead, it first constructs a discourse-level global multimodal context and then enables bidirectional interactions between it and the local features of each modality. First, discourse-level aggregation is performed on the obtained features of the three modalities, with the text modality using a special label. The corresponding hidden state is used as a discourse-level representation. Non-textual modalities are represented by temporal average pooling to obtain discourse-level representations. :

[0037]

[0038]

[0039] in, Representing non-textual modality The time step length; then, the utterance-level representations of each modality are concatenated to obtain the initial global multimodal context:

[0040]

[0041] in Used to represent the global multimodal sentiment semantics of the current discourse;

[0042] In the During layer interaction, for any modality Cross-modal attention enables bidirectional information transfer between global context and local modal features: First, local modal features are used... As a query term, the global multimodal context Using keys and values, local features are guided by global sentiment semantics:

[0043]

[0044] At the same time, use a global multimodal context. Local modal features as a query term As keys and values, the global context is transformed from modal. Extracting effective sentiment information from local sequences:

[0045]

[0046] in, This represents cross-modal attention. Representation layer normalization, and These are learnable parameters; then, the features after bidirectional interaction are input into the attention layer and the feedforward network to obtain updated local modality features and candidate global context:

[0047]

[0048]

[0049] in This indicates a self-attention operation. This represents a feedforward neural network;

[0050] Since different modalities generate corresponding candidate global contexts, attention pooling is used to aggregate them to obtain the next layer of global multimodal contexts:

[0051]

[0052] in, For modality The context contribution weight is calculated as follows:

[0053]

[0054] in, , and These are learnable parameters;

[0055] go through After layer interaction, we obtain the enhanced local feature set and the final global multimodal context. Finally, the final global context is concatenated with the enhanced utterance-level representations of each modality to obtain the fused representation after global and local interactions:

[0056]

[0057] Furthermore, the specific process of step four includes:

[0058] First, the global semantic features of the text As a source of emotion transmission, judging non-textual initial state features The contact matrix is ​​obtained by identifying locations in the text that are most susceptible to the influence of textual sentiment and semantics.

[0059]

[0060] in, Indicates an indicator function, The contact rate is used to control the conditions for state transitions between text sentiment semantics and non-text modal features; then, a random matrix is ​​introduced during the state transition process. And based on the infection rate Generate an infection status record matrix:

[0061]

[0062] in, This represents element-wise multiplication. Used to record the location where the initial state transitions to the infection state characteristics; the locations of information still retained in the initial state are represented as follows:

[0063]

[0064] During the recovery state transition, the location in the infection state characteristics that can transition to the recovery state is determined; the recovery contact matrix is ​​defined as:

[0065]

[0066] Next, a random matrix is ​​introduced. According to the recovery rate Obtain the recovery status record matrix:

[0067]

[0068] in, Used to record the location where the infection state features transition to the recovery state features; since some infection state information is transferred to the recovery state, the infection state record matrix needs to be updated:

[0069]

[0070] Finally, based on the location of valid information in the initial state, infected state, and recovered state, non-textual modal common sentiment features are generated:

[0071]

[0072] in, This represents pseudo-non-text recovery state features, i.e., pseudo-non-text features generated by the cross-modal feature translation module. The representation obtained after discourse-level aggregation yields visual public sentiment features. Harmony and public sentiment characteristics To ensure that public sentiment features retain effective information relevant to sentiment judgments, a public sentiment feature supervision loss is introduced:

[0073]

[0074]

[0075] in, and These represent the fully connected mappings corresponding to visual and auditory public sentiment features, respectively. A label indicating the true intensity of emotion. Used to guide public emotional characteristics toward the emotional discrimination space. This represents the mean absolute error loss function.

[0076] Furthermore, step five specifically includes the following process:

[0077] First, during the training phase, a missing view is constructed from the complete multimodal input; the random mask corresponding to any modality representation is... ,in This indicates that the location is concealed. This indicates that the position is retained; a missing view is represented as:

[0078]

[0079] in, Represents any mode The missing view features obtained after masking share the parameters of the aforementioned feature encoding, cross-modal translation, global and local interaction, and common sentiment feature state transition modules with the complete view, thus yielding the complete view fusion representation. Merging with missing views To recover low-level feature information from missing modalities, low-level feature reconstruction constraints are introduced; these constraints are applied to the interactive modal features in the missing view. Decoding is performed to obtain the reconstructed features:

[0080]

[0081] in, Representing modes The corresponding reconstruction decoder; the low-level feature reconstruction loss is:

[0082]

[0083] Here we use Indicates smoothness The loss function measures the difference between the original and reconstructed features at the masked location; to enhance the consistency between the complete and missing views in the high-level sentiment semantic space, a high-level feature attraction constraint is introduced.

[0084]

[0085] in, For negative cosine similarity, For projection function, For the prediction function, This indicates that the gradient stopping operation is in progress. Used to constrain missing view representation To full view Proximity enhances the stability of the model's emotional representation in modality-deficient scenarios;

[0086] The final sentiment representation is obtained by concatenating the fused representation from the missing view, the global text representation, the visual common sentiment features, and the auditory common sentiment features.

[0087]

[0088] Ultimately, the overall optimization objective of this method is:

[0089]

[0090] in, , , , and The loss weight coefficients are used to balance the contributions of each training objective. Through the above joint optimization, missing modal information is recovered in the low-level feature space, the consistency between the complete view and the missing view is maintained in the high-level semantic space, and sentiment classification and regression prediction are jointly completed by using common sentiment features and global fusion representation.

[0091] The beneficial effects of this invention are as follows: This invention proposes a multimodal sentiment analysis method based on multi-layer feature perception and emotional feature state transfer, including: (1) a text-driven cross-modal feature translation and common emotional feature state transfer mechanism is proposed: deep interaction between text semantic information and visual and acoustic emotional cues is carried out, which can more fully explore the complementary relationship and emotional relevance between different modalities and alleviate the problems of modal heterogeneity and modal semantic inconsistency. (2) A cross-modal interaction mechanism is introduced, which guides the local feature update of each modality with discourse-level global context, while retaining key emotional information and reducing invalid feature interference, and obtaining a stable and effective fusion representation. (3) A two-layer feature recovery mechanism is designed, which enables the model to maintain a strong sentiment recognition ability under conditions of random modal loss, non-text feature degradation or noise interference, and effectively improves the discrimination ability and anti-interference ability in the fusion stage.

[0092] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0093] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0094] Figure 1 This is a multimodal sentiment analysis process based on multi-layer feature perception and emotional feature state transition;

[0095] Figure 2 This is a diagram of a multimodal sentiment analysis model based on multi-layer feature perception and emotional feature state transition. Detailed Implementation

[0096] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0097] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0098] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0099] like Figure 1 As shown, this invention provides a multimodal sentiment analysis method based on multi-layer feature perception and sentiment feature state transition. First, a pre-trained language model is used to extract semantic representations of the text, and acoustic and visual features are temporally encoded and linearly mapped to form a unified multimodal input representation. Then, with the text modality as the semantic core, a cross-modal feature translation module is constructed. Cross-modal attention is used to generate text-enhanced non-textual interaction features, and distribution constraints and reconstruction constraints are combined to reduce modal heterogeneity. Simultaneously, a cross-modal interaction mechanism is introduced, utilizing global and local modal features for bidirectional updates to improve the fusion efficiency of misaligned multimodal sequences. Next, a common sentiment feature state transition module is used to generate stable visual and acoustic common sentiment features. Finally, a missing view is constructed using random modal masks, multi-layer feature constraints are introduced, and joint optimization is performed using sentiment classification and sentiment intensity regression tasks.

[0100] Figure 2 This is a system model diagram of the present invention. The following description, in conjunction with the accompanying drawings, includes the following modules:

[0101] Module 1: Text modalities obtain semantic representations through pre-trained language models, while visual and acoustic modalities obtain temporal features through encoding networks and are mapped to a unified common feature space;

[0102] Module 2: Using text modality as semantic guide, feature interaction is achieved through cross-modal attention mechanism to generate text-enhanced non-text features, and distribution constraints and reconstruction constraints are combined to improve the stability of feature translation;

[0103] Module 3: Achieve interaction between global and local information through a bidirectional attention mechanism, and generate visual and acoustic public sentiment features using a public sentiment state transfer mechanism;

[0104] Module 4: Construct complete and missing views during the training phase, and enhance the robustness of the model under modality missing conditions by utilizing multi-layer feature constraints;

[0105] Module 5: The enhanced multimodal representation, common sentiment features, and restored robust features are fused together, and sentiment classification and sentiment intensity regression are completed through the prediction layer.

[0106] Optionally, module one specifically includes:

[0107] For the input utterance sample, suppose it contains a text sequence Visual frame-level features Harmony and acoustic frame-level features First, a pre-trained language model is used to perform contextual semantic encoding on the text sequence to obtain text features:

[0108]

[0109] in, This represents the parameters of the pre-trained language model. and These are learnable mapping parameters. For the visual and acoustic modalities, a temporal encoder is used to extract their dynamic emotional features, which are then mapped to a common feature space through a linear transformation.

[0110]

[0111]

[0112] in, Indicates visual or acoustic input features. This represents the timing encoder corresponding to the mode. These are the intermediate features after encoding. These are the mapped visual or acoustic features. Ultimately, a unified-dimensional multimodal representation is obtained:

[0113]

[0114] in, Represent the text, visual, and acoustic modalities in the common feature space, respectively.

[0115] Optionally, Module Two specifically includes:

[0116] Text features Using semantics as the core, cross-modal feature translation relationships are constructed between text and visual features, and between text and acoustic features, respectively, to alleviate semantic heterogeneity between different modalities and obtain text-enhanced non-text sentiment representations. First, a cross-modal attention mechanism is employed, using text features as queries and visual or acoustic features as keys and values, to obtain text-guided non-text interaction features:

[0117]

[0118] in, Visual interaction features derived from text features or auditory characteristics , This represents the learnable parameters of the corresponding cross-modal attention module. Subsequently, a back-translation branch is introduced to constrain the cross-modal translation process, using non-textual features as query terms and textual features as keys and values ​​to generate corresponding pseudo-non-textual features:

[0119]

[0120] in Indicates pseudo-visual features, These are the learnable parameters for the inverse cross-modal attention module. To reduce the bias caused by the inconsistency in the distribution of different modalities during cross-modal translation, this step uses distribution-constrained loss to ensure that the non-textual interaction features after text guidance are consistent with the textual features in the common space.

[0121]

[0122] in, Represents the Wasserstein distance function. This is used to constrain the distribution differences between cross-modal interaction features and text features. To ensure that the pseudo-non-text features generated during reverse cross-modal translation retain the effective information of the original non-textual modality, a reconstruction constraint loss is introduced:

[0123]

[0124] in, This is used to measure the reconstruction error between pseudo-non-textual features and original non-textual features. Here, the cross-modal feature translation constraint loss is expressed as:

[0125]

[0126] Optionally, Module 3 specifically includes:

[0127] Since textual, visual, and acoustic modalities typically have different sampling frequencies and sequence lengths, directly using pairwise interactions between local features can easily introduce a large amount of redundant computation and may cause the model to focus on irrelevant local noise. Therefore, this method does not directly perform dense interactions on the local sequences of all modalities. Instead, it first constructs a discourse-level global multimodal context and then allows it to interact bidirectionally with the local features of each modality. First, discourse-level aggregation is performed on the obtained features of the three modalities. For the text modality, the hidden state corresponding to a special label is used as the discourse-level representation, and for the non-text modality, time-dimensional average pooling is used to obtain the discourse-level representation.

[0128]

[0129]

[0130] in, Representing non-textual modality The time step length. Then, the utterance-level representations of each modality are concatenated to obtain the initial global multimodal context:

[0131]

[0132] in Used to represent the global multimodal sentiment semantics of the current discourse.

[0133] In the During layer interaction, for any modality Cross-modal attention is used to achieve bidirectional information transfer between global context and local modal features. First, local modal features are used... As a query term, the global multimodal context Using keys and values, local features are guided by global sentiment semantics:

[0134]

[0135] At the same time, use a global multimodal context. Local modal features as a query term As keys and values, the global context is transformed from modal. Extracting effective sentiment information from local sequences:

[0136]

[0137] in, This represents cross-modal attention. Representation layer normalization, and These are learnable parameters. The features resulting from the bidirectional interaction are then input into the attention layer and the feedforward network to obtain updated local modality features and candidate global context:

[0138]

[0139]

[0140] in This indicates a self-attention operation. This represents a feedforward neural network.

[0141] Since different modalities generate corresponding candidate global contexts, attention pooling is used to aggregate them to obtain the next layer of global multimodal contexts:

[0142]

[0143] in, For modality The context contribution weight is calculated as follows:

[0144]

[0145] in, , and These are learnable parameters.

[0146] go through After layer interaction, an enhanced set of local features is obtained. With the final global multimodal context Finally, the final global context is concatenated with the enhanced modal discourse-level representations to obtain the fused representation after global and local interactions:

[0147]

[0148] Optionally, Module 4 specifically includes:

[0149] First, the global semantic features of the text As a source of emotion transmission, judging non-textual initial state features The contact matrix is ​​obtained by identifying locations in the text that are most susceptible to the influence of textual sentiment and semantics.

[0150]

[0151] in, Indicates an indicator function, The contact rate is used to control the state transition conditions between text sentiment semantics and non-text modal features. Then, a random matrix is ​​introduced during the state transition process. And based on the infection rate Generate an infection status record matrix:

[0152]

[0153] in, This represents element-wise multiplication. Used to record the location where the initial state transitions to the infected state characteristics. The locations where information is still retained in the initial state are represented as:

[0154]

[0155] During the recovery state transition, the location from the infection state characteristics that can transition to the recovery state is determined. The recovery contact matrix is ​​defined as follows:

[0156]

[0157] Next, a random matrix is ​​introduced. According to the recovery rate Obtain the recovery status record matrix:

[0158]

[0159] in, This is used to record the locations where infection state features transition to recovery state features. Since some infection state information is transferred to the recovery state, the infection state record matrix needs to be updated.

[0160]

[0161] Finally, based on the location of valid information in the initial state, infected state, and recovered state, non-textual modal common sentiment features are generated:

[0162]

[0163] This leads to the visual public sentiment characteristics. Harmony and public sentiment characteristics To ensure that public sentiment features retain effective information relevant to sentiment judgments, a public sentiment feature supervision loss is introduced:

[0164]

[0165]

[0166] in, and These represent the fully connected mappings corresponding to visual and auditory public sentiment features, respectively. A label indicating the true intensity of emotion. It is used to guide public emotional characteristics to approach the emotional discrimination space.

[0167] Optionally, Module 5 specifically includes:

[0168] First, during the training phase, a missing view is constructed from the complete multimodal input. The random mask corresponding to any modality representation is: ,in This indicates that the location is concealed. This indicates that the position is reserved. A missing view is represented as:

[0169]

[0170] The missing view and the complete view share the parameters of the aforementioned feature encoding, cross-modal translation, global and local interaction, and common sentiment feature state transition modules, respectively, to obtain the complete view fusion representation. Merging with missing views To recover low-level feature information from missing modalities, low-level feature reconstruction constraints are introduced. These constraints are applied to the interactive modal features in the missing view. Decoding is performed to obtain the reconstructed features:

[0171]

[0172] in, Representing modes The corresponding reconstruction decoder. The low-level feature reconstruction loss is:

[0173]

[0174] This loss only constrains the reconstruction error at the occluded locations, guiding the model to recover missing modal features based on visible modal information. To enhance the consistency between the complete and missing views in the high-level sentiment semantic space, a high-level feature attraction constraint is introduced:

[0175]

[0176] in, For negative cosine similarity, Used to constrain missing view representation To full view Proximity enhances the stability of the model's emotional representation in modality-deficient scenarios.

[0177] The final sentiment representation is obtained by concatenating the fused representation from the missing view, the global text representation, the visual common sentiment features, and the auditory common sentiment features.

[0178]

[0179] Ultimately, the overall optimization objective of this method is:

[0180]

[0181] in, , , , and The loss weight coefficients are used to balance the contributions of each training objective. Through the above joint optimization, missing modal information is recovered in the low-level feature space, the consistency between the complete view and the missing view is maintained in the high-level semantic space, and sentiment classification and regression prediction are jointly completed by using common sentiment features and global fusion representation.

[0182] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multimodal sentiment analysis method based on multi-layer feature perception and emotional feature state transition, characterized in that, First, a pre-trained language model is used to semantically encode the input text. Simultaneously, temporal modeling and linear mapping are performed on acoustic and visual modal features, projecting text, acoustic, and visual features onto a common feature space of a unified dimension to obtain a unified multimodal input representation. Second, a cross-modal feature translation module is introduced. With the text modality as the semantic core, the interaction relationship between the text modality and the non-text modality is established through a cross-modal attention mechanism, generating text-enhanced visual interaction features, text-enhanced acoustic interaction features, and corresponding pseudo-visual and pseudo-acoustic features. Secondly, a cross-modal interaction mechanism is introduced, utilizing the discourse-level representations of each modality to construct a global multimodal context, and enabling it to interact and update with local unimodal temporal features. Then, a common sentiment feature state transition module is used, employing global semantic features of the text as a guiding signal for sentiment propagation, generating stable common sentiment features through the state transition process. Finally, combined with a two-layer feature recovery module, a missing modality view is generated through random modality masks. Low-level feature reconstruction and high-level feature attraction constraints enhance the model's robustness to modality loss and noise interference. Furthermore, joint optimization of sentiment polarity classification and sentiment intensity regression is completed based on the fused representation. The steps of the model in identifying sentiment are as follows: Step 1: Use a pre-trained language model to perform contextual semantic encoding on the input text sequence to obtain local and global semantic features of the text. Perform temporal modeling and linear mapping on the frame-level features of the acoustic and visual modalities, and project the text, acoustic and visual features onto a common feature space to form a unified multimodal input representation. Step 2: Introduce a cross-modal feature translation module. With text features as the semantic core, establish feature translation relationships between text and vision, and text and acoustics through a cross-modal attention mechanism to generate text-enhanced visual interaction features and acoustic interaction features, as well as corresponding pseudo-visual features and pseudo-acoustic features. Step 3: Introduce a cross-modal interaction module to construct a global multimodal context using the discourse-level representations of each modality. This allows for bidirectional interactive updates with local temporal features of text, acoustics, and vision, enhancing the ability to effectively fuse emotional information in unaligned multimodal sequences. Step 4: Using the public sentiment feature state transition module, the global semantic features of the text are used as the guidance signal for sentiment propagation. The visual and acoustic features are modeled as initial state, infection state and recovery state features, respectively. Visual public sentiment features and acoustic public sentiment features are generated through a controlled state transition process. Step 5: Design a two-layer feature recovery and sentiment prediction module. Generate a missing modality view through random modality masking, and enhance the model's robustness to modality loss by using low-level feature reconstruction and high-level feature attraction constraints. Based on the fused robust multimodal representation, achieve multimodal sentiment analysis through joint optimization.

2. The multimodal sentiment analysis method based on multi-layer feature perception and sentiment feature state transition according to claim 1, characterized in that: The specific process described in step one includes: For the input utterance sample, suppose it contains a text sequence Visual frame-level features Harmony and acoustic frame-level features First, a pre-trained language model is used to perform contextual semantic encoding on the text sequence to obtain text features: in, Indicates the parameters of the pre-trained language model. and The parameters are learnable mappings; for the visual and acoustic modalities, a temporal encoder is used to extract their dynamic emotional features, which are then mapped to a common feature space through a linear transformation: in, Indicates visual or acoustic input features. This represents the timing encoder corresponding to the mode. These are the intermediate features after encoding. These are the mapped visual or acoustic features; ultimately, a unified-dimensional multimodal representation is obtained. in, These represent the text, visual, and acoustic modalities in the common feature space, respectively.

3. The multimodal sentiment analysis method based on multi-layer feature perception and sentiment feature state transition according to claim 1, characterized in that: The specific process described in step two includes: Text features Using semantics as the core, cross-modal feature translation relationships are constructed between text and visual features, and between text and acoustic features, respectively, to alleviate semantic heterogeneity between different modalities and obtain text-enhanced non-text sentiment representations. First, a cross-modal attention mechanism is adopted, using text features as query terms and visual or acoustic features as keys and values, to obtain text-guided non-text interaction features: in, Visual interaction features derived from text features or auditory characteristics , This represents the learnable parameters of the corresponding cross-modal attention module; subsequently, a back-translation branch is introduced to constrain the cross-modal translation process, using non-textual features as query terms and textual features as keys and values ​​to generate corresponding pseudo-non-textual features: in Indicates pseudo-visual features, These are the learnable parameters for the reverse cross-modal attention module. To reduce the bias caused by the inconsistency in the distribution of different modalities during cross-modal translation, a distribution-constrained loss is used to ensure that the non-textual interaction features after text guidance are consistent with the textual features in the common space. in, Represents the Wasserstein distance function. To constrain the distribution differences between cross-modal interaction features and text features, a reconstruction constraint loss is introduced to ensure that the pseudo-non-text features generated during reverse cross-modal translation retain the effective information of the original non-text modality. in, This is used to measure the reconstruction error between pseudo-non-textual features and original non-textual features; here, the cross-modal feature translation constraint loss is expressed as:

4. The multimodal sentiment analysis method based on multi-layer feature perception and sentiment feature state transition according to claim 1, characterized in that: The specific process described in step three includes: First, the obtained three types of modal features are aggregated at the discourse level, with the text modality using special tags. The corresponding hidden state is used as a discourse-level representation. Non-textual modalities are represented by temporal average pooling to obtain discourse-level representations. : in, Representing non-textual modality The time step length; then, the utterance-level representations of each modality are concatenated to obtain the initial global multimodal context: in Used to represent the global multimodal sentiment semantics of the current discourse; In the During layer interaction, for any modality Cross-modal attention enables bidirectional information transfer between global context and local modal features: First, local modal features are used... As a query term, the global multimodal context Using keys and values, local features are guided by global sentiment semantics: At the same time, use a global multimodal context. Local modal features as a query term As keys and values, the global context is transformed from modal. Extracting effective sentiment information from local sequences: in, This represents cross-modal attention. Representation layer normalization, and These are learnable parameters; then, the features after bidirectional interaction are input into the attention layer and the feedforward network to obtain updated local modality features and candidate global context: in This indicates a self-attention operation. This represents a feedforward neural network; Since different modalities generate corresponding candidate global contexts, attention pooling is used to aggregate them to obtain the next layer of global multimodal contexts: in, For modality The context contribution weight is calculated as follows: in, , and These are learnable parameters; go through After layer interaction, we obtain the enhanced local feature set and the final global multimodal context. Finally, the final global context is concatenated with the enhanced utterance-level representations of each modality to obtain the fused representation after global and local interactions:

5. The multimodal sentiment analysis method based on multi-layer feature perception and sentiment feature state transition according to claim 1, characterized in that: The specific process described in step four includes: First, the global semantic features of the text As a source of emotion transmission, determine the features of the non-textual initial state. The contact matrix is ​​obtained by identifying locations in the text that are most susceptible to the influence of textual sentiment semantics. in, Indicates an indicator function, The contact rate is used to control the conditions for state transitions between text sentiment semantics and non-text modal features; then, a random matrix is ​​introduced during the state transition process. And based on the infection rate Generate an infection status record matrix: in, This represents element-wise multiplication. Used to record the location where the initial state transitions to the infection state characteristics; the locations of information still retained in the initial state are represented as follows: During the recovery state transition, the location from the infection state characteristics that can be transitioned to the recovery state is determined; the recovery contact matrix is ​​defined as: Next, a random matrix is ​​introduced. According to the recovery rate Obtain the recovery status record matrix: in, Used to record the location where the infection state features transition to the recovery state features; since some infection state information is transferred to the recovery state, the infection state record matrix needs to be updated: Finally, based on the location of valid information in the initial state, infected state, and recovered state, non-textual modal common sentiment features are generated: in, This represents pseudo-non-text recovery state features, i.e., pseudo-non-text features generated by the cross-modal feature translation module. The representation obtained after discourse-level aggregation yields visual public sentiment features. Harmony and public emotional characteristics To ensure that public sentiment features retain effective information relevant to sentiment judgments, a public sentiment feature supervision loss is introduced: in, and These represent the fully connected mappings corresponding to visual and auditory public sentiment features, respectively. A label indicating the true intensity of emotion. Used to guide public emotional characteristics toward the emotional discrimination space. This represents the mean absolute error loss function.

6. The multimodal sentiment analysis method based on multi-layer feature perception and sentiment feature state transition according to claim 1, characterized in that: Step five describes the specific process, which includes: First, during the training phase, a missing view is constructed from the complete multimodal input; the random mask corresponding to any modality representation is... ,in This indicates that the location is concealed. This indicates that the position is retained; a missing view is represented as: in, Represents any mode The missing view features obtained after masking share the parameters of the aforementioned feature encoding, cross-modal translation, global and local interaction, and common sentiment feature state transition modules with the complete view, thus yielding the complete view fusion representation. Merging with missing views To recover low-level feature information from missing modalities, low-level feature reconstruction constraints are introduced; these constraints are applied to the interactive modal features in the missing view. Decoding is performed to obtain the reconstructed features: in, Representing modes The corresponding reconstruction decoder; the low-level feature reconstruction loss is: Here we use Indicates smoothness The loss function measures the difference between the original and reconstructed features at the masked location; to enhance the consistency between the complete and missing views in the high-level sentiment semantic space, a high-level feature attraction constraint is introduced. in, For negative cosine similarity, For projection function, For the prediction function, This indicates that the gradient stopping operation is in progress. Used to constrain missing view representation To full view Proximity enhances the stability of the model's emotional representation in modality-deficient scenarios; The final sentiment representation is obtained by concatenating the fused representation from the missing view, the global text representation, the visual common sentiment features, and the auditory common sentiment features. Ultimately, the overall optimization objective of this method is: in, , , , and The loss weight coefficients are used to balance the contributions of each training objective. Through the above joint optimization, missing modal information is recovered in the low-level feature space, the consistency between the complete view and the missing view is maintained in the high-level semantic space, and sentiment classification and regression prediction are jointly completed by using common sentiment features and global fusion representation.

Images