Multi-modal sentiment analysis method based on text enhancement and modal completion perception fusion

Abstract

The invention relates to a multi-modal sentiment analysis method based on text enhancement and modal completion perception fusion, and belongs to the field of natural language processing. The method comprises the following steps: 1, extracting text semantic features by utilizing a pre-training language model, and performing linear transformation on non-text features to form unified multi-modal input representation; 2, injecting a cross-modal enhancement module into the pre-training language model, fusing non-text information by taking a text as a core, and performing multi-modal input representation; 3, introducing a modal completion module, generating a missing modal completion representation through reconstruction loss and random modal discarding, and suppressing noise features in combination with a convolution gating structure; and guiding full-connection network learning modal weight to realize joint optimization of sentiment classification and regression. According to the method provided by the invention, on the premise of keeping the dominance of the text, through cross-modal interaction and dynamic weighting and in combination with a modal completion mechanism, the model has relatively strong emotion recognition capability.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and relates to a multimodal sentiment analysis method based on the fusion of text enhancement and modal completion perception. Background Technology

[0002] Multimodal sentiment analysis refers to the process of identifying and modeling human emotional states in communication, interaction, and expression by comprehensively utilizing information from multiple modalities such as text, speech, and vision, to complete tasks such as sentiment recognition or sentiment intensity prediction. Unlike single-modal sentiment analysis, multimodal sentiment analysis can capture emotional cues from semantic content, speech prosody features, and non-verbal signals such as facial expressions and postures, thus forming a more comprehensive and accurate portrayal of emotional states. Due to the diversity and complexity of human emotional expression, a single modality is often insufficient to support reliable inferences about true emotions. The combination of multiple modalities can effectively alleviate the challenges caused by incomplete information and modal inconsistencies, enabling sentiment recognition systems to maintain stability and robustness in real and complex scenarios.

[0003] Multimodal sentiment analysis is of great significance in fields such as natural language processing, intelligent interaction, and social computing. It not only helps systems more accurately understand underlying human emotional intentions but also facilitates effective human-computer interaction and communication. In education, multimodal sentiment analysis can be used to monitor learners' attention and emotional states, supporting personalized instructional feedback. In healthcare, it can assist in identifying psychological states such as depression and anxiety, enhancing the objectivity and intelligence of clinical diagnosis. In social media and public safety, multimodal sentiment analysis is also widely used for tasks such as public opinion monitoring, emotion propagation modeling, and risk warning.

[0004] However, multimodal sentiment analysis still faces many challenges. On the one hand, different modalities are prone to being missing or incomplete during collection and processing, preventing the model from fully capturing all cues of emotional expression. On the other hand, the emotional information carried by different modalities may be inconsistent or even conflicting; for example, positive textual expression may be accompanied by negative speech or facial expressions. Such modal contradictions increase the difficulty of model judgment. Therefore, effectively addressing the problems of modality loss and inconsistency in multimodal sentiment analysis, and reasonably modeling the correlation and differences between modalities, has become the core of research in this field.

[0005] In multimodal sentiment analysis, models often over-rely on the text modality, leading to the underutilization of fine-grained emotional information from visual and acoustic perspectives. Furthermore, modality loss or degradation is common, affecting the stability of cross-modal alignment and interaction. To address this, cross-modal enhancement methods inject non-linguistic modal features into language models to improve discriminative ability, while modality completion methods enhance robustness under missing modality conditions through target reconstruction and random modality discarding strategies. However, the former's performance degrades in modality loss scenarios, and while the latter is robust, it struggles to capture fine-grained complementary information across modalities, limiting the overall improvement in sentiment recognition accuracy.

[0006] Based on the above discussion, we propose a multimodal sentiment analysis method based on the fusion of text enhancement and modal completion awareness. This method first extracts semantic representations of the text using a pre-trained language model and combines them with feature-transformed acoustic and visual features to form a unified trimodal input representation. Then, a cross-modal enhancement module is injected into the intermediate layer of the language model to fuse non-textual information with text as the core, and controlled reverse interaction is used to strengthen the alignment consistency between modalities. Subsequently, a modal completion module is designed to generate completed representations through reconstruction loss and random modality dropout training, combined with convolutional gating to suppress noise. On this basis, a fully connected network supervised by the reconstructed signal is used to dynamically allocate the weights of each modality, obtaining a stable fused representation. Finally, sentiment classification and regression are performed simultaneously on this representation, and the overall performance is improved by jointly optimizing multiple loss functions. Using this mechanism, we conducted experiments on commonly used datasets and achieved state-of-the-art and stable performance. Summary of the Invention

[0007] The present invention aims to solve the problems existing in the prior art and proposes a method to improve the accuracy of multimodal sentiment analysis.

[0008] To address the problems encountered in existing research, the present invention provides the following solution:

[0009] A multimodal sentiment analysis method based on the fusion of text enhancement and modal completion perception includes the following steps:

[0010] Step 1: Extract semantic representation of text using a pre-trained language model, while acquiring acoustic and visual features, and form a unified input representation through feature transformation;

[0011] Step 2: Inject a cross-modal enhancement module into the language model, which integrates acoustic and visual features with text as the core, and introduces controlled reverse interaction;

[0012] Step 3: Design a modality completion module to generate completed representations by using reconstruction loss and random modality dropout, and combine it with a convolutional gating mechanism to filter features;

[0013] Step 4: Using modal completion signal supervision, guide the fully connected network to dynamically learn modal weights and obtain a weighted fusion representation;

[0014] Step 5: Employ multi-task prediction in the fusion representation to achieve joint optimization of sentiment classification and regression.

[0015] Furthermore, the specific process of step one is as follows:

[0016] Let the input samples be text modal sequences T = {t1, t2, ..., t3}. n}, the acoustic modal sequence A={a1,a2,...,a m} and the visual modality sequence V={v1,v2,...,v k For text modalities, the text modal sequence is encoded using a pre-trained language model to obtain a context-dependent semantic representation of the text.

[0017]

[0018] Where d is the dimension of the shared representation. For visual and acoustic modal sequences, frame-level features are first obtained using feature extraction tools. and Then, a learnable linear mapping is used to project it into the same representation space as the text modality sequence:

[0019]

[0020] in, The mapping matrix is ​​used to obtain dimensions consistent with the text representation. Normalization is applied to the projected acoustic and visual features, and a mask matrix is ​​used to mark valid and padded frames. Finally, the text features H... t The mapped acoustic features and visual features They are unified to form a multimodal input representation:

[0021]

[0022] Furthermore, the specific process of step two is as follows:

[0023] Using textual features as queries and acoustic and visual features as keys and values, a multi-head attention mechanism is employed to extract supplementary information about the text from non-textual modalities.

[0024]

[0025] Where Attention(Q, K, V) is the standard multi-head attention calculation:

[0026]

[0027] For learnable parameters, the obtained The acoustic and visual modal enhancements to the text modality are represented, respectively. Then, the two enhancement results are fused with the original text representation to obtain the cross-modal enhanced text representation:

[0028] H′ t =H t +α a Z a +α v Z v ,

[0029] in Learnable fusion weights are used to balance the contributions of different modalities. A controlled reverse interaction is introduced, using text features as queries and feeding them back to the acoustic and visual modalities as alignment and consistency constraints during the training phase.

[0030]

[0031] Obtained through reverse interaction As a supervisory signal during training, it is used to calculate modality alignment and consistency loss, thereby enhancing the collaborative modeling capability between different modalities. Finally, the output of the cross-modal enhancement module is:

[0032]

[0033] As input for subsequent modal completion and fusion stages.

[0034] Furthermore, step three specifically includes the following process:

[0035] During the training phase, a modality discarding operation is randomly applied to non-textual modalities:

[0036]

[0037] in Let H' be a randomly sampled mask vector, where ⊙ denotes element-wise multiplication. Next, the missing parts are reconstructed using cross-modal information: using the text-enhanced feature H′. t As a condition, complete representations of the acoustic and visual modalities are generated through a linear projection network:

[0038]

[0039] in Let be the learnable completion mapping matrix. To constrain the effectiveness of the completion process, a modality reconstruction loss function is defined to measure the difference between the reconstructed representation and the original features:

[0040]

[0041] Simultaneously, convolutional gating units are used to filter features from non-textual modalities. For the completed features... This gating mechanism selectively preserves effective features while suppressing noise information.

[0042]

[0043] Finally, the output of the modal completion module is With text features H′ t Both are used as input.

[0044] Furthermore, the specific process of step four includes:

[0045] Because different modalities differ, some modalities are also affected by noise, missing information, or completion errors, impacting the model's emotion recognition performance. Therefore, we utilize modal completion signals as supervision, dynamically allocating weights for each modality during the fusion process. This allows the model to fully leverage the core advantages of the text modality while incorporating supplementary information from the acoustic and visual modalities. For each modality's completion features, a fully connected network is used to extract its global quality representation and map it to weight scores.

[0046]

[0047] At the same time, the same processing is applied to the text modality to obtain the text modality quality score:

[0048] q t =MLP(H′) t )

[0049] Obtain the mass fractions {q} of all modes t q a q v After that, the mass weights of each mode are calculated through the Softmax normalization operation:

[0050]

[0051] Where α t α a α v ∈(0,1) and satisfy constraint α t +α a +α v =1.

[0052] Based on these weights, the three modes are weighted and fused to obtain the final multimodal representation:

[0053]

[0054] Furthermore, step five specifically includes the following process:

[0055] After obtaining the fused representation, a multi-task learning framework is used to simultaneously perform sentiment polarity classification and sentiment intensity regression, fully utilizing the complementary information contained in the multimodal features. The fused representation is input into a fully connected layer and mapped to the predicted distribution of sentiment categories.

[0056]

[0057] in C represents the number of classification categories. The loss function for the classification task is cross-entropy loss:

[0058]

[0059] Where y c This is a one-hot encoding for the real label.

[0060] Secondly, continuous values ​​of sentiment intensity are predicted using independent regression heads:

[0061]

[0062] in The loss function uses the mean absolute error:

[0063]

[0064] Where y i This represents the true emotional intensity value of the sample.

[0065] The beneficial effects of this invention are as follows: This invention proposes a multimodal sentiment analysis method based on the fusion of text enhancement and modal completion perception, including: (1) a text-dominated bidirectional cross-modal enhancement mechanism is proposed: on the basis of maintaining text as the core discrimination path, a controlled reverse cross-modal interaction is introduced to strengthen the alignment and consistency between modalities during the training phase, thereby effectively improving the alignment quality and robustness between multimodalities without destroying the text-dominance premise. (2) A modal completion perception mechanism is designed, which effectively improves the robustness of the model in modal missing scenarios by using reconstruction loss and random modal dropout training. (3) Using the reconstructed signal as supervision, a fully connected network is used to dynamically learn the weights of each modality and perform weighted fusion, thereby replacing the traditional simple splicing method and effectively improving the discrimination ability in the fusion stage.

[0066] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0067] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0068] Figure 1 This is a multimodal sentiment analysis process based on the fusion of text enhancement and modal completion perception;

[0069] Figure 2 A diagram of a multimodal sentiment analysis model based on the fusion of text enhancement and modal completion perception; Detailed Implementation

[0070] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0071] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0072] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0073] like Figure 1As shown, this invention provides a multimodal sentiment analysis method based on the fusion of text enhancement and modal completion perception. First, a pre-trained language model is used to semantically encode the input text sequence, simultaneously extracting acoustic and visual frame-level features to form the model's standardized input. Then, a cross-modal enhancement module is injected into the intermediate layer of the language model, using text features as the core and employing multi-head attention to select and write the sentiment cues most relevant to the current semantics from acoustic and visual data. Simultaneously, a controlled reverse interaction path is introduced during the training phase, imposing alignment and consistency constraints on non-text modalities using text as the query. Next, a modal completion module is used to generate completion features based on the enhanced text representation under the condition of random modal discarding, and gated convolutional units are used to filter the features of the reconstructed results. Subsequently, the modal completion signal is used as supervision to dynamically learn modal weights and perform weighted fusion. Finally, sentiment classification is performed on the fused representation.

[0074] Figure 2 This is a system model diagram of the present invention. The following description, in conjunction with the accompanying drawings, includes the following modules:

[0075] Module 1: Extracting basic semantic and feature information from multimodal inputs. The text modality obtains context-dependent semantic representations through a pre-trained language model, while the acoustic and visual modalities undergo feature normalization and dimensionality alignment through linear mapping networks, thereby forming multimodal input vectors in a unified representation space.

[0076] Module 2: Centered on the text modality, it achieves feature interaction and complementation with the acoustic and visual modalities through a multi-head attention mechanism, introduces a controlled reverse interaction path, and uses text features to constrain the consistency of acoustic and visual features.

[0077] Module 3: Non-textual modalities are randomly discarded, and the enhanced text features are used to generate complete representations of the missing modalities through a reconstruction network; at the same time, convolutional gating units are used to perform feature filtering and noise suppression on the reconstruction results;

[0078] Module 4: The model uses modal completion signals as supervision to guide the fully connected network to calculate the importance weights of each modality, thereby achieving weighted feature fusion;

[0079] Module 5: By jointly optimizing classification loss, regression loss, and reconstruction constraints, a comprehensive modeling and alignment correction of emotional states is achieved.

[0080] Optionally, module one specifically includes:

[0081] Let the input samples be text modal sequences T = {t1, t2, ..., t3}. n}, the acoustic modal sequence A={a1,a2,...,a m} and the visual modality sequence V={v1,v2,...,v k For text modalities, the text modal sequence is encoded using a pre-trained language model to obtain a context-dependent semantic representation of the text.

[0082]

[0083] Where d is the dimension of the shared representation. For visual and acoustic modal sequences, frame-level features are first obtained using feature extraction tools. and Then, a learnable linear mapping is used to project it into the same representation space as the text modality sequence:

[0084]

[0085] in, The mapping matrix is ​​used to obtain dimensions consistent with the text representation. Normalization is applied to the projected acoustic and visual features, and a mask matrix is ​​used to mark valid and padded frames. Finally, the text features H... t The mapped acoustic features and visual features They are unified to form a multimodal input representation:

[0086]

[0087] Optionally, Module Two specifically includes:

[0088] Using textual features as queries and acoustic and visual features as keys and values, a multi-head attention mechanism is employed to extract supplementary information about the text from non-textual modalities.

[0089]

[0090] Where Attention(Q,K,X) is the standard multi-head attention calculation:

[0091]

[0092] For learnable parameters, the obtained The acoustic and visual modal enhancements to the text modality are represented, respectively. Then, the two enhancement results are fused with the original text representation to obtain the cross-modal enhanced text representation:

[0093] H′ t =H t +α a Z a +α v Z v ,

[0094] in Learnable fusion weights are used to balance the contributions of different modalities. A controlled reverse interaction is introduced, using text features as queries and feeding them back to the acoustic and visual modalities as alignment and consistency constraints during the training phase.

[0095]

[0096] Obtained through reverse interaction As a supervisory signal during training, it is used to calculate modality alignment and consistency loss, thereby enhancing the collaborative modeling capability between different modalities. Finally, the output of the cross-modal enhancement module is:

[0097]

[0098] As input for subsequent modal completion and fusion stages.

[0099] Optionally, Module 3 specifically includes:

[0100] During the training phase, a modality discarding operation is randomly applied to non-textual modalities:

[0101]

[0102] in Let H' be a randomly sampled mask vector, where ⊙ denotes element-wise multiplication. Next, the missing parts are reconstructed using cross-modal information: using the text-enhanced feature H′. t As a condition, complete representations of the acoustic and visual modalities are generated through a linear projection network:

[0103]

[0104] in Let be the learnable completion mapping matrix. To constrain the effectiveness of the completion process, a modality reconstruction loss function is defined to measure the difference between the reconstructed representation and the original features:

[0105]

[0106] Simultaneously, convolutional gating units are used to filter features from non-textual modalities. For the completed features... This gating mechanism selectively preserves effective features while suppressing noise information.

[0107]

[0108] Finally, the output of the modal completion module is With text features H′ t Both are used as input.

[0109] Optionally, Module 4 specifically includes:

[0110] Because different modalities differ, some modalities are also affected by noise, missing information, or completion errors, impacting the model's emotion recognition performance. Therefore, we utilize modal completion signals as supervision, dynamically allocating weights for each modality during the fusion process. This allows the model to fully leverage the core advantages of the text modality while incorporating supplementary information from the acoustic and visual modalities. For each modality's completion features, a fully connected network is used to extract its global quality representation and map it to weight scores.

[0111]

[0112] At the same time, the same processing is applied to the text modality to obtain the text modality quality score:

[0113] q t =MLP(H)

[0114] Obtain the mass fractions {q} of all modes t q a q v After that, the mass weights of each mode are calculated through the Softmax normalization operation:

[0115]

[0116] Where α t α a α v ∈(0,1) and satisfy constraint α t +α a +α v =1.

[0117] Based on these weights, the three modes are weighted and fused to obtain the final multimodal representation:

[0118]

[0119] Optionally, Module 5 specifically includes:

[0120] After obtaining the fused representation, a multi-task learning framework is used to simultaneously perform sentiment polarity classification and sentiment intensity regression, fully utilizing the complementary information contained in the multimodal features. The fused representation is input into a fully connected layer and mapped to the predicted distribution of sentiment categories.

[0121]

[0122] in C represents the number of classification categories. The loss function for the classification task is cross-entropy loss:

[0123]

[0124] Where y c This is a one-hot encoding for the real label.

[0125] Secondly, continuous values ​​of sentiment intensity are predicted using independent regression heads:

[0126]

[0127] in The loss function uses the mean absolute error:

[0128]

[0129] Where y i This represents the true emotional intensity value of the sample.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multi-modal sentiment analysis method based on text enhancement and modal completion perception fusion, characterized in that, First, the pre-trained language model is used to encode the semantic of the input text, and the features of acoustic and visual modalities are extracted at the same time. The non-text modalities are discretely represented by feature transformation method to obtain a unified multi-modal input representation. Second, a cross-modal enhancement module is injected into the middle layer of the language model to fuse acoustic and visual information with text as the core. A controlled reverse interaction mechanism is introduced to strengthen the alignment and consistency between modalities during the training phase, so that the text representation can effectively perceive the emotional cues of non-text modalities. Third, a modal completion module is designed to guide the model to generate complete modal representations under the condition of missing modalities by reconstruction loss and random modal dropout strategy during the training process. The convolution gating mechanism is used to suppress noise features. Then, the modal completion signal is used as supervision to guide the fully connected network to dynamically learn the weights of each modality, and a weighted fusion representation is obtained. Finally, the sentiment polarity classification and sentiment intensity regression are performed based on the fusion representation, and the multi-task joint optimization strategy is used to improve the overall performance of the model in multi-modal sentiment recognition. The steps of the model to recognize the sentiment of the utterance in the dialogue are as follows: Step one: Use the pre-trained language model to encode the semantic of the input text sequence, and extract the frame-level features of acoustic and visual modalities. The non-text modalities are discretely processed by feature transformation method to reduce the distribution difference between modalities, and a unified multi-modal input representation is formed. Step two: Inject a cross-modal enhancement module into the middle layer of the language model to fuse acoustic and visual modalities with text features as the core. A controlled reverse interaction is introduced to strengthen the alignment and consistency between modalities during the training phase, so as to strengthen the collaborative modeling ability between different modalities. Step three: Design a modal completion module to guide the model to generate complete modal representations under the condition of missing modalities by combining reconstruction loss and random modal dropout mechanism during the training process. The convolution gating mechanism is used to suppress noise features to improve the robustness of the model in the presence of missing modalities and noise. Step four: Use the modal completion signal as supervision information to guide the lightweight fully connected network to dynamically learn the weights of each modality for weighted fusion, and obtain a more stable and rich multi-modal representation. Step five: On the basis of the fusion representation, use multi-task prediction head to complete sentiment polarity classification and sentiment intensity regression, and perform sentiment analysis through joint optimization.

2. The multi-modal sentiment analysis method based on the fusion of text enhancement and modality completion perception according to claim 1, characterized in that: The specific process of step one includes Let the input samples be text modality sequence T = {t1, t2,..., t n}, acoustic modality sequence A = {a1, a2,..., a m}, and visual modality sequence V = {v1, v2,..., v k}. For the text modality, the text modality sequence is encoded by a pre-trained language model to obtain a text semantic representation related to the context: where d is the dimension of the shared representation. For visual and auditory modalities sequences, frame-level features are first obtained by feature extraction tools and The learned linear mapping is then used to project them into the same representation space as the textual modality sequences: where, is the mapping matrix, resulting in a consistent dimensionality with the text representation. Standardization operations are applied to the projected acoustic and visual features, and a mask matrix is utilized to mark valid frames and padding frames. Finally, the text feature H t , the mapped acoustic feature and visual feature are unified to form the multi-modal input representation:

3. The method of claim 2, wherein the text-based enhancement and modality completion perception fusion-based multi-modal sentiment analysis method is characterized by: The specific process of step two includes Text features are used as queries, acoustic and visual features as keys and values, and multi-head attention mechanism is used to obtain supplementary information of non-text modalities to text: Where Attention(Q, K, V) is the standard multi-head attention calculation: To learnable parameters, get respectively represent the enhanced features of the acoustic and visual modalities for the text modality. Then, the two enhanced results are fused with the original text representation to obtain the cross-modal enhanced text representation: H' t = H t + a a Z a + a v Z v , where are learnable fusion weights to balance the contributions of different modalities. Controlled back-interaction is introduced, with text features as queries, feeding back to acoustic and visual modalities in the training phase as alignment and consistency constraints: obtained by reverse interaction As a supervisory signal during training, it is used to calculate the modality alignment and consistency loss, which strengthens the collaborative modeling ability between different modalities. Finally, the output of the cross-modal enhancement module is: As the input of the subsequent modal completion and fusion stage.

4. The multi-modal sentiment analysis method based on the fusion of text enhancement and modality completion perception according to claim 3, characterized in that: The specific process of step three includes Random modal dropout operation is applied to non-text modalities during the training phase: where is the randomly sampled mask vector, and denotes element-wise multiplication. Next, the missing parts are reconstructed using cross-modal information: the text-enhanced feature H′ t As a condition, the completion representations for both acoustic and visual modalities are generated by linear projection networks: where is a learnable completion mapping matrix. To constrain the effectiveness of the completion process, a modality reconstruction loss function is defined to measure the difference between the reconstructed representation and the original feature: Meanwhile, the convolutional gating unit is combined to filter the features of non-text modalities. For the completed features Through this gating mechanism, effective features are selectively retained and noise information is suppressed: Finally, the output of the modal completion module is m e {a, v} along with the text features H' t as input.

5. The method of claim 4, wherein the text-based enhancement and modality completion perception fusion-based multi-modal sentiment analysis method is characterized by: The specific process of step four includes: Due to the differences between different modalities, some modalities are also affected by noise, missing or error of completion, which affects the performance of the model in emotion recognition. Therefore, we use the modality completion signal as supervision to dynamically allocate the weight of each modality in the fusion process, so that the model can fully utilize the core advantages of the text modality while combining the supplementary information of acoustic and visual modalities. For the completion features of each modality, its global quality representation is extracted through a fully connected network and mapped to the weight score: At the same time, the same processing is also done for the text modality to obtain the quality score of the text modality: q t = MLP(H' (x) ) = MLP(H(x) ) = MLP(x) = y t ) The quality score {q t , q a , q v} of all modalities is obtained t , q a , q v} of all modalities is obtained t , q a , q v} of all modalities is obtained t , q <000001 Where α t α a α v ∈(0,1) and satisfy constraint α t +α a +α v =1. Based on these weights, the three modalities are weighted and fused to obtain the final multi-modal representation:

6. The method of claim 5, wherein the text-based enhancement and modality completion perception fusion-based multi-modal sentiment analysis method is characterized by: The specific process described in step five includes: After obtaining the fused representation, the sentiment polarity classification and sentiment intensity regression are completed simultaneously through a multi-task learning framework to fully utilize the complementary information contained in the multi-modal features. The fused representation is input into a fully connected layer to map it to the prediction distribution of the sentiment category: wherein C denotes the number of classification categories. The loss function for the classification task employs a cross-entropy loss: where y c is a one-hot encoding of the true label. Secondly, an independent regression head is used to predict the continuous value of the sentiment intensity: wherein The loss function employs mean absolute error: where y i is the true sentiment intensity value of the sample. In addition, the modality completion loss introduced in step three continues to be used as a regularization term in this step to ensure the stability and consistency of the modality completion process during prediction. Finally, the overall optimization objective is the weighted sum of the three loss terms: where λ cls , λ reg , λ rec are loss balancing coefficients. Through the above multi-task prediction and joint optimization mechanism, the model not only can accurately complete the sentiment polarity classification, but also can capture the fine-grained changes of the sentiment intensity at the same time, and use the reconstruction constraint to enhance the overall robustness and generalization ability, thereby realizing more comprehensive multi-modal sentiment analysis.

Images