Prompt-driven mode-missing-oriented multi-mode sentiment analysis method and system

Abstract

The invention discloses a prompt-driven mode missing-oriented multi-mode sentiment analysis method and system, and relates to the technical field of sentiment analysis, and the method comprises the steps: inputting multi-mode data containing a missing mode into a multi-mode prompt learning model for analysis, and obtaining a sentiment prediction result; the multi-modal prompt learning model comprises a missing modal generator, a dynamic prompt weight module and a modal prior guide fusion module; the dynamic prompt weight module adaptively generates a prompt weight by extracting global context information from an available mode, dynamically adjusts prompt intensity and an action range, and realizes accurate local compensation of missing mode features; the modal prior guidance fusion module carries out explicit modeling on a modal missing type, actively fuses prior correlation knowledge between modals, accurately guides a fusion network to realize deep complementation and global optimization of cross-modal features, and significantly improves the robustness and generalization ability of the model in various modal missing scenes.

Description

Technical Field

[0001] The present invention relates to the field of sentiment analysis technology, and in particular to a prompt-driven modality-missing multimodal sentiment analysis method and system. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] In recent years, multimodal sentiment analysis (MSA), a typical multimodal task, has been widely used in fields such as social media monitoring, human-computer interaction, and intelligent customer service by comprehensively leveraging text, audio, and visual information to accurately identify user emotions. However, most MSA methods assume that all modal data is complete during training and testing. In real-world scenarios, modal data is often missing due to reasons such as device failure, privacy constraints, or data collection difficulties, severely impacting model performance and generalization.

[0004] Early research on the problem of missing modalities mainly focused on joint learning methods and cross-modal generation methods. Specifically, joint learning methods construct robust representations through modal alignment or shared space to alleviate the negative impact of missing modalities. However, this type of method relies heavily on the selection of similarity metrics and mask filling operations, which can easily lose key information and introduce noise. Cross-modal generation methods use existing modal information to reconstruct missing modalities. However, they are limited by the heterogeneity between modalities and it is difficult to obtain high-quality reconstruction results. In addition, the above two types of methods often focus on specific modality missing scenarios, that is, using complete modality data for training, and modality missing only occurs in the testing phase. This is usually impractical in practical applications and significantly limits the generalization ability of the model and its scalability in actual deployment.

[0005] With the rise of MMTs (Multimodal Transformers), prompt-based methods have garnered widespread attention. Leveraging prompt fine-tuning techniques, these methods effectively transfer the capabilities of pre-trained MMTs on full multimodal datasets to tasks with modality loss, gradually becoming a mainstream approach to addressing modality loss. Among them, MAPs (Mapped Image Processing) (MAPs) first applied prompt learning to the problem of modality loss, considering a more general and realistic setting where modality loss may occur during both training and testing, more closely aligned with practical application requirements. This approach significantly reduces computational cost by freezing the pre-trained backbone and fine-tuning only the parameters of the newly introduced prompts. However, this strategy results in the number of prompts growing exponentially with the number of modalities, severely limiting its scalability to MSA tasks. To alleviate this issue, MPLMM (Multiple Parallel Multi-Level Multi-Modal Modeling) introduces generation prompts, missing signal prompts, and missing type prompts to more efficiently represent missing modality information, effectively reducing the number of prompts to a linear relationship with the number of modalities. However, the prompts in these methods remain static and difficult to dynamically adapt to varying contextual information and modality loss combinations, resulting in inaccurate compensation for missing modal features. In addition, these methods ignore the inherent differences between modalities and the prior association knowledge of missing types during the cross-modal fusion process, resulting in insufficient overall complementarity of the fused representation. Summary of the Invention

[0006] To overcome the shortcomings of the above-mentioned existing technologies, the present invention provides a prompt-driven multimodal sentiment analysis method and system for modal loss, which aims to handle more general modal loss settings. It includes two key modules: a dynamic prompt weight module and a modal prior fusion module, which significantly improve the robustness and generalization ability of the MMT model in various modal loss scenarios from the two levels of local single-modal dynamic compensation and global cross-modal fusion optimization.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: In a first aspect, the present invention provides a prompt-driven modality-deficient multimodal sentiment analysis method, comprising: Obtain multimodal data including missing modalities; Inputting the multimodal data into a multimodal cue learning model for analysis to obtain an emotion prediction result; the multimodal cue learning model includes a missing modality generator, a dynamic cue weight module, a modality prior guided fusion module and a classifier; The missing mode generator receives the multimodal data and obtains preliminary features of the missing mode in combination with the reconstruction hint of the missing mode; The dynamic prompt weight module dynamically generates a prompt weight vector corresponding to the missing modality through context perception and a prompt weight network, fuses the prompt weight vector with the missing prompt to obtain a dynamic prompt of the missing modality, and fuses the dynamic prompt with the preliminary features of the missing modality to obtain the missing modality completion feature; at the same time, based on the available modality and its corresponding existence prompt, it obtains the enhanced modality feature of the available modality; The modality prior guided fusion module receives enhanced modality features and missing modality completion features, combines missing type prompts with modality prior fusion prompts to obtain overall cross-modal fusion features, and inputs them into the classifier to obtain the final emotion prediction result.

[0008] In a further technical solution, the missing modality generator performs cross-modal feature conversion on the missing modality in the multimodal data to generate preliminary compensation features, and obtains preliminary features of the missing modality based on the reconstruction hints of the missing modality and the preliminary compensation features.

[0009] According to a further technical solution, the dynamic prompt weight module receives the features of the available modalities, projects the features of the available modalities, extracts the global context features through the enhanced attention pooling module, and dynamically generates the prompt weight vector of the missing modality based on the global context features.

[0010] A further technical solution is that the modal prior guided fusion module splices and fuses the enhanced modal features and the missing modal completion features in the cross-modal interaction network to obtain initial cross-modal fusion features, and the initial cross-modal fusion features are fused with the corresponding missing type prompts to obtain modal fusion features. The modal fusion features are spliced ​​and combined with the modal prior fusion prompts to obtain the overall cross-modal fusion features.

[0011] In a further technical solution, the modality prior guided fusion module introduces a modality prior projection matrix, and generates missing type prompts based on the modality prior projection matrix and a predefined missing type prompt set.

[0012] A further technical solution is that the modality prior guide fusion module constructs a modality prior fusion network, first globally pools the existence prompts of each modality to obtain the global existence prompts of each modality, and then obtains the fusion prompt vector through the modality prior fusion network after splicing.

[0013] A further technical solution is to map the fusion prompt vector to the same dimensional space as the modal fusion feature to obtain a modal prior fusion prompt.

[0014] In a second aspect, the present invention provides a prompt-driven multimodal sentiment analysis system for modality loss, comprising: A data acquisition module is configured to: acquire multimodal data including missing modalities; a model analysis module configured to: input the multimodal data into a multimodal cue learning model for analysis to obtain a sentiment prediction result; the multimodal cue learning model includes a missing modality generator, a dynamic cue weight module, a modality prior guided fusion module, and a classifier; The missing mode generator receives the multimodal data and obtains preliminary features of the missing mode in combination with the reconstruction hint of the missing mode; The dynamic prompt weight module dynamically generates a prompt weight vector corresponding to the missing modality through context perception and a prompt weight network, fuses the prompt weight vector with the missing prompt to obtain a dynamic prompt of the missing modality, and fuses the dynamic prompt with the preliminary features of the missing modality to obtain the missing modality completion feature; at the same time, based on the available modality and its corresponding existence prompt, it obtains the enhanced modality feature of the available modality; The modality prior guided fusion module receives enhanced modality features and missing modality completion features, combines missing type prompts with modality prior fusion prompts to obtain overall cross-modal fusion features, and inputs them into the classifier to obtain the final emotion prediction result.

[0015] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the prompt-driven modality-missing multimodal sentiment analysis method as described in the first aspect.

[0016] In a fourth aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the prompt-driven modality-missing multimodal sentiment analysis method as described in the first aspect are implemented.

[0017] One or more of the above technical solutions have the following beneficial effects: This paper uses an innovative dynamic prompt weighting mechanism (DPW) and a modality prior guided fusion mechanism (MPGF) in combination with a missing modality generator (MMG) to significantly improve the robustness, accuracy and adaptability of the multimodal sentiment analysis model when facing the missing of any modality during the training or testing phase, thereby effectively overcoming the data uncertainty problem caused by modality missing and ultimately achieving more accurate sentiment classification.

[0018] To address the lack of dynamic adjustment capabilities of static prompts to adapt to changes in input data features, the present invention designs a dynamic prompt weight module (DPW). DPW extracts global contextual information from available modalities, adaptively generates prompt weights, and dynamically adjusts prompt strength and scope to achieve precise local compensation for missing modal features.

[0019] To overcome the shortcomings of existing fusion strategies that ignore inherent differences between modalities and prior knowledge of associations, this paper further designs a Modal Prior Fusion Module (MPGF). This module explicitly models the type of modal loss and actively incorporates prior knowledge of intermodal associations. This module precisely guides the fusion network to achieve deep cross-modal feature complementarity and global optimization. This collaborative design of local dynamic compensation and global fusion optimization effectively improves the model's robustness and generalization capabilities in different modal loss scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0021] Figure 1 This is an overall framework diagram of a multimodal sentiment analysis method based on dynamic prompts for modality loss and modality prior guidance according to an embodiment of the present invention; Figure 2 This is a flowchart of missing modality completion according to an embodiment of the present invention; Figure 3 This is a flow chart of the missing prior-guided fusion module in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0023] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0024] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0025] Existing technologies use cue learning to enhance the robustness of pre-trained MMT in the missing modality scenario, but they still have the following limitations: (1) Most of the cues in existing methods are static and fixed, and cannot be dynamically adjusted according to the feature differences of different input samples, resulting in insufficient compensation accuracy for missing modal features. (2) Current fusion strategies ignore the inherent differences between modalities and the prior knowledge of missing combinations, resulting in insufficient feature complementarity in the cross-modal fusion stage, which restricts the overall quality of the fused representation. To address the shortcomings of existing technologies, a cue-driven multimodal sentiment analysis method and system for modality missing is proposed, aiming to handle a more general modality missing setting.

[0026] Example 1 This embodiment discloses a prompt-driven multimodal sentiment analysis method for modality loss, which includes the following steps: S1: Obtain multimodal data including missing modalities; In this embodiment, the present invention studies the multimodal sentiment analysis task in the missing modality scenario, where the modality missing may occur during the training or testing phase of the data. Specifically, the present invention collects multimodal data to construct a multimodal dataset. , including audio ( )、Visual( ) and text ( ) three modes. For any data sample in the data set, it is recorded as ,in 、 、 Represent the features of audio, visual and text modalities respectively. In order to more clearly describe the problem definition and cover various possible modality missing situations, this paper uses To represent the missing state of a certain modality. For example, when both audio and visual modalities are missing, the data sample is represented as The core goal of this invention is to accurately predict the overall sentiment category (positive, neutral, or negative) in the context of data uncertainty caused by missing modalities.

[0027] S2: inputting the multimodal data into a multimodal cue learning model for analysis to obtain an emotion prediction result; the multimodal cue learning model includes a missing modality generator, a dynamic cue weight module, a modality prior guided fusion module and a classifier; The missing mode generator receives the multimodal data and obtains preliminary features of the missing mode in combination with the reconstruction hint of the missing mode; The dynamic prompt weight module dynamically generates a prompt weight vector corresponding to the missing modality through context perception and a prompt weight network, fuses the prompt weight vector with the missing prompt to obtain a dynamic prompt of the missing modality, and fuses the dynamic prompt with the preliminary features of the missing modality to obtain the missing modality completion feature; at the same time, based on the available modality and its corresponding existence prompt, it obtains the enhanced modality feature of the available modality; The modality prior guided fusion module receives enhanced modality features and missing modality completion features, combines missing type prompts with modality prior fusion prompts to obtain overall cross-modal fusion features, and inputs them into the classifier to obtain the final emotion prediction result.

[0028] In this embodiment, the present invention proposes a prompt-driven multimodal sentiment analysis method for modality loss, called DPMP (Dynamic Prompt Weighting and Modality-Prior Guided Fusion). This method is built on the cross-modal Transformer architecture MulT to obtain a multimodal prompt learning model. Figure 1 As shown in the figure, the multimodal prompt learning model is an end-to-end deep neural network model composed of multiple functional modules. The model is aimed at multimodal sentiment analysis tasks under modality-missing scenarios and is mainly composed of the following four sub-network modules: Missing Modality Generator (MMG), Dynamic Prompt Weight (DPW), Modality-Prior Guided Fusion (MPGF) and classifier (fully connected classifier). Each module is composed of several neural network layers, which comprehensively improve the robustness and generalization ability of the MMTs model under modality-missing conditions from three levels: modality completion, prompt adjustment and fusion optimization.

[0029] 1. Missing Modality Generator (MMG) The Missing Modality Generator (MMG) aims to generate preliminary completion features for the missing modality. Its core consists of reconstruction hints and cross-modal mapping. The Missing Modality Generator consists of a cross-modal feature mapping layer and a cross-modal fusion layer connected in sequence.

[0030] The missing modality generator receives multimodal data and designs independent reconstruction prompts for the three modalities of text, audio, and vision. ,in and Represent the cue feature dimension and cue sequence length, respectively. Reconstructed cues, as part of the model parameters, are learned end-to-end via gradient backpropagation during training to adaptively model the information compensation requirements for different missing modalities. During the initialization phase, the Xavier uniform initialization strategy is used to ensure training stability.

[0031] The cross-modal feature mapping layer consists of a one-dimensional convolutional layer (Conv1D) and a nonlinear activation function (such as GELU), which is used to map the existing modal features to the feature space of the missing modality to achieve cross-modal feature conversion. Mapping to modality The cross-modal feature conversion layer is defined as:

[0032] in, Indicates modality The original feature sequence of , Indicates modality The characteristic dimension of Indicates modality The sequence length of Indicates that the modal To Modal The mapping function of It is an activation function (Gaussian Error Linear Unit), which is used to introduce nonlinear transformation; Indicates that the input channel dimension is , the output channel dimension is One-dimensional convolution operation is used to achieve cross-modal feature mapping.

[0033] The data containing the missing modality is input into the cross-modal feature mapping layer to generate preliminary compensatory features of the missing modality. The reconstruction hint of the missing modality and the preliminary compensatory features of the missing modality are spliced ​​along the sequence dimension to obtain the spliced ​​features and input them into the cross-modal fusion layer. The cross-modal fusion layer is used to adjust the length dimension of the spliced ​​features and map them to the sequence length of the target missing modality to obtain the preliminary features of the missing modality.

[0034] Taking text as an example with missing mode, the data with missing mode is represented as To complete the text modality, input the available audio and visual modalities and use the corresponding cross-modal feature mapping layer 、 Generate preliminary compensatory features for text modality: ,

[0035] Reconstruction hints corresponding to text Splicing it with the preliminary compensation features generated above along the sequence dimension to obtain the splicing features, expressed as:

[0036] in, Represents the splicing feature, 、 、 Represent the sequence lengths of audio, visual, and text modalities respectively.

[0037] The cross-modal fusion layer is a one-dimensional convolutional layer , use the cross-modal fusion layer to adjust the length dimension of the splicing features and map them to the sequence length of the target modality text , get the preliminary features of the text:

[0038] in, Represents the preliminary features of the text.

[0039] The process for handling missing audio or visual modalities is similar. Through these operations, MMG effectively utilizes information conversion and reconstruction cues between modalities, significantly improving the model's initial ability to complete missing modal information and laying the foundation for more accurate subsequent modal compensation and fusion.

[0040] (2) Dynamic Prompt Weight Module (DPW) like Figure 2 As shown in Figure 2, the Dynamic Hint Weight (DPW) module is designed to flexibly adjust the strength of feature representations based on the absence or presence of a modality. Each modality contains two types of hints: missing hints, which are used to compensate for the missing modality and enhance the reconstruction of the missing modality; and present hints, which are used to provide additional contextual information when the modality is fully present, further stabilizing the representation of the existing modality.

[0041] (1) Missing and existing prompts Specifically, each modality Both the absence and presence cues of are represented as learnable parameter matrices: 、 .

[0042] in, Indicates modality Missing prompt, Indicates modality The existence of prompts, represents the dimension of the prompt vector, Indicates modality The length of the characteristic sequence.

[0043] If the text modality is missing (i.e. Not available), the missing hint for the text modal will be enabled to assist in compensation; if the audio modality exists, its original features and presence cues are used simultaneously Perform joint modeling.

[0044] Both missing and present cues are initialized with all-zero matrices to reduce initial parameter fluctuations and are updated end-to-end via back-propagation during training.

[0045] (2) Global context extraction In order to achieve dynamic adjustment of the cue strength, the original features of the available modalities (non-missing modalities) are first projected to extract global context information. , using a one-dimensional convolutional layer ( ) to project the feature dimension:

[0046] in, Represents the modal characteristics after projection.

[0047] The global context features of the modality are then extracted through the Enhanced Attention Pooling (EAP) module:

[0048] in, Indicates modality Global context features.

[0049] Specifically, the EAP module provides Defines a learnable query vector , Indicates modality The query vector is a length The query vector is a real number vector. This query vector is used as an abstract representation of the global context and participates in the feature sequence attention score calculation at each time step. It is used to guide the model to focus on the key positions in the sequence and perform weighted summation of the features based on the score to ultimately obtain the global context features of each modality. The detailed calculation process is as follows: 1) Modal features after projection At each time step Calculate the attention score:

[0050] in, express The attention score at the time step, Indicates modality The characteristic dimension of Indicates modality The length of the characteristic sequence.

[0051] 2) Use the softmax function to normalize the attention score to obtain the attention weight of each time step:

[0052] in, represents the attention weight.

[0053] 3) Use the obtained attention weights to perform weighted summation on the projected modality features to obtain the initial global context features of the modality:

[0054] in, Represents the initial global context features.

[0055] 4) To further improve stability, global mean information is added to the initial global context features. The final global context features are:

[0056] in, represents the global context feature, Indicates the number of time steps involved in the mean calculation.

[0057] The enhanced attention pooling (EAP) module can capture richer and more detailed semantic information within the modality, ensuring that the subsequently generated prompt weights have higher accuracy and effectiveness.

[0058] (3) Prompt Weight Network When there is a missing modality, the fused context features are obtained by averaging the global context features of the non-missing modalities to reduce the parameter scale. The fused context features are input into the lightweight prompt weight network to dynamically generate a prompt weight vector, and the dynamically generated prompt weight vector is applied to the missing prompt of the missing modality. .

[0059] Taking text as an example, when the text modality is missing, assuming that the global context features of audio and visual modalities are and , by averaging the audio and visual context information to reduce the parameter size:

[0060] in, Represents the fusion of context features.

[0061] The fused context features are then fed into a lightweight cue weight network, which consists of two linear mapping layers and a nonlinear activation function to dynamically generate a cue weight vector:

[0062] in, represents the prompt weight vector for text modality, Represents the sigmoid function, which limits the prompt weight value to the interval [0,1] to ensure the stability and controllability of the prompt range; represents the second linear mapping layer, represents a nonlinear activation function, represents the first linear mapping layer, Represents a dimension (sequence length of text modality). That is, Contains the same number of elements as the text modal time steps, and is used to control the prompt intensity on a time-step basis.

[0063] Finally, the dynamically generated prompt weight vector Missing hints applied to the corresponding text superior:

[0064] in, A dynamic prompt that represents text. represents the element-wise multiplication of the missing hint and the hint weight vector, Represents a dimension The two-dimensional real matrix space of prompt vectors, each of which has a dimension of .

[0065] (4) Dynamic prompt fusion and modal completion The dynamic prompts of the missing modality are fused with the preliminary features of the missing modality generated by MMG to obtain the missing modality completion features, which are used to finally complete the feature representation of the missing modality.

[0066] Take the text missing mode as an example, and change the dynamic prompt of the text Preliminary features of the text By fusion, we can obtain the text completion features, which can be expressed as:

[0067] in, Represents a text completion feature.

[0068] For other modalities that are fully present (such as audio and vision), the projected features of the modality are directly compared with the corresponding presence cues. 、 Add together to obtain enhanced modal features:

[0069] in, represents enhanced audio features, Indicates enhanced visual features.

[0070] DPW achieves flexible compensation for missing features and enhancement of existing modality representations through global context-driven cue modulation, effectively improving the quality of fusion representation.

[0071] (3) Modal Prior Guided Fusion Module (MPGF) like Figure 3 As shown in the figure, Modality Prior Guided Fusion (MPGF) explicitly introduces prior knowledge of modal missing types. It models the deep complementary relationship between modalities for different missing combination scenarios, enhancing the targetedness of missing compensation and the effectiveness of fusion representation. MPGF consists of two key components: missing type hints and modality prior fusion.

[0072] (1) Missing type hints Different combinations of modal loss correspond to different context feature distributions and information redundancy levels, and the compensation information required in the fusion process also varies greatly. In order to display information representing different modal loss combinations, a loss type prompt is introduced. Considering the three modalities of text (t), audio (a), and vision (v), there are at most possible combinations of missing modes (not considering the case where all modes are missing).

[0073] In order to efficiently handle these missing combination scenarios, the present invention introduces a modal prior projection matrix. Each mode corresponds to a projection parameter , which is used to map the modality cue (presence or absence of cue) from the cue space to the fused representation space. are modality-specific learnable projection matrix parameters that are optimized end-to-end during model training.

[0074] Specifically, the present invention designs a set of missing prior projection matrices for each modality missing combination, explicitly encoding the prior knowledge of modality missing. These hint matrices are parameterized using modality-specific projection matrices to capture the global prior information between modalities and thus guide the fusion process. In actual calculations, according to the specific modality missing combination, the presence hints corresponding to the modalities are used to and missing prompts , through the projection parameters Get the modal prior projection matrix under the corresponding missing combination.

[0075] If the text modality is missing, the modality prior projection matrix is ​​calculated as follows:

[0076] in, represents the modal prior projection matrix, Indicates the presence of audio. Indicates the presence of visual cues, Indicates missing text hint, Represents the projection parameters of the audio, Represents the projection parameters of vision, Represents the projection parameters of the text.

[0077] Then hint the collection via the predefined missing type , get the specific missing type hint required for modality fusion:

[0078] in, Refers to a set of initial prompt vectors set for each of the three modalities (text, audio, and vision), which are used to indicate how the current modality should be guided to participate in fusion in a specific missing combination scenario. This set is trained end-to-end as part of the network structure and optimized through the back-propagation mechanism during training to form a modality missing type prompt with semantic expression capabilities; Indicates missing type hints after projection, Represents a set of prompts corresponding to three modes (text, audio, and visual), each of which is represented by prompt vectors, each of which has a dimension of 2 .

[0079] In this way, the present invention achieves efficient active modeling of missing modal combinations using only linearly growing prompt parameters, avoiding the problem of the number of parameters growing exponentially with the number of modalities.

[0080] Combine the projected missing type hints with the feature sequences of each modality after interaction through the cross-modal interaction network Transformer. First, the cross-modal interaction network Transformer captures the interaction relationship between the text modality and other modalities:

[0081]

[0082] in, Represents the interactive features between text and audio, Represents the interactive features between text and vision.

[0083] The interactive feature sequences are then concatenated along the feature dimension to form a preliminary cross-modal fusion representation of the text modality:

[0084] in, Represents the initial cross-modal fusion features of the text.

[0085] Then prompt the missing type corresponding to the text modal Spliced ​​to the initial cross-modal fusion feature, and fusion modeling is performed through an additional self-attention layer, and finally the end feature of the sequence is taken as the fusion feature of the modality:

[0086] in, Represents text modality fusion features.

[0087] The fusion process of audio and visual modalities is similar, and the final audio modality fusion features are obtained respectively. and visual modality fusion features .

[0088] (2) Modal Prior Fusion (MPF) In order to unify the integration of each modality fusion representation and missing type guidance, the present invention further constructs a modality prior fusion network. First, the existence of each modality is prompted Perform global pooling to obtain global presence hints for each modality :

[0089] Then the global existence prompts of the three modes Perform splicing and capture the deeper interaction between modalities through the modality prior fusion network (MPFN) to obtain the final fusion prompt vector :

[0090] in, represents the second layer linear transformation weight matrix, represents the first layer linear transformation weight matrix, Indicates the global existence prompt of the text, Indicates the global presence of audio. Represents the global presence cue of vision, Represents the first layer bias term.

[0091] To ensure that the fused hint vector can be integrated with the cross-modal deep features of the model (i.e. 、 、 ) indicates effective fusion, and a linear projection layer is further used to map the fused hint vector to the same dimensional space as the modality fusion feature:

[0092] in, represents the modality prior fusion hint, represents the weight matrix of the linear transformation, represents the bias term.

[0093] Finally, the final fusion features of the three modalities are concatenated and combined with the modality prior fusion prompts Combine to form the overall cross-modal fusion features:

[0094]

[0095] in, represents multimodal splicing features, Represents the overall cross-modal fusion features.

[0096] This approach significantly improves the model's ability to actively model modal missing type information, and enhances the fusion robustness and expression accuracy under multiple missing combinations.

[0097] The following is an explanation of the experimental method: (1) Dataset and evaluation metrics To evaluate the effectiveness of the DPMP method, we conduct comprehensive experiments on three widely used MSA benchmark datasets.

[0098] 1) CMU-MOSI is a popular multimodal (audio, text, and video) sentiment analysis dataset consisting of 2,199 speech video clips taken from 93 YouTube videos, in which 89 different narrators share their opinions on interesting topics. Each clip is manually annotated with a sentiment score ranging from -3 (strongly negative) to +3 (strongly positive).

[0099] 2) CMU-MOSEI is an extension of CMU-MOSI. It consists of 23,454 movie review video clips from YouTube, featuring over 1,000 speakers and 250 topics. Compared to CMU-MOSI, it covers a wider range of topics and uses the same labeling style as CMU-MOSI.

[0100] 3) CH-SIMS is a Chinese MSA dataset with fine-grained unimodal annotations. It contains 2,281 refined video clips from various movies, TV series, and variety shows, annotated with sentiment scores ranging from -1 (strongly negative) to 1 (strongly positive).

[0101] In terms of evaluation metrics, for CMU-MOSI and CMU-MOSEI, this paper uses seven-category accuracy (Acc-7), two-category accuracy (ACC), F1 score (F1), mean absolute error (MAE), and Pearson correlation (Corr) as evaluation indicators. For CH-SIMS, two-category accuracy (ACC), F1 score (F1), mean absolute error (MAE), and Pearson correlation (Corr) are used. Acc-7 indicates correct predictions within the [-3, 3] sentiment score range. MAE calculates the mean absolute difference between the predicted value and the true value. Corr measures the degree of prediction bias.

[0102] (2) Experimental details To simulate real-world scenarios, we selected CMU-MOSEI as a high-resource dataset and CMU-MOSI and CH-SIMS as low-resource datasets. We first pre-trained the backbone network MulT on CMU-MOSEI, and then evaluated our method on the three datasets.

[0103] During multimodal feature extraction, the methods described in MulT were followed for CMU-MOSEI and CMU-MOSI. Different feature extraction methods were used for different datasets, demonstrating the generalization capability of our method. The following details the feature extraction methods for each modality in the two datasets.

[0104] 1) Textual Modality. For MOSI and MOSEI, the video text is converted into pre-trained Glove word embeddings (Glove.840b.300d), which are 300-dimensional vectors. For SIMS, the pre-trained BERT is used as a feature extractor to encode the transcribed word sequences into textual modality features, where the feature dimension is 768.

[0105] 2) Visual Modality. MOSI and MOSEI use Facet to represent 35 facial action units, recording facial muscle movements to express basic and advanced emotions in each frame. SIMS uses the MTCNN face detection algorithm to extract aligned faces and then uses the MultiComp OpenFace 2.0 toolkit to extract facial features.

[0106] 3) Audio modality. MOSI and MOSEI use COVAREP to extract low-level acoustic features, which include 12 Mel-frequency cepstral coefficients (MFCCs), pitch tracking and voiced / voiced segmentation features, glottal source parameters, peak slope parameters, and maximum dispersion quotient. SIMS uses LibROSA.

[0107] This paper focuses on the more general missingness setting, where modality missingness can occur during both training and testing, where each modality of each data sample may be missing. We define the missingness rate η% as the proportion of modality-incomplete data in the entire dataset. In this experiment, the missingness rate η is set to 70% by default.

[0108] For model training, we first pretrained the backbone network on the CMU-MOSEI dataset. We then froze all parameters of the backbone network and trained only the learnable prompts, Conv layers, and output layers. Our experiments were all conducted on an NVIDIA RTX 3090 GPU. We performed a grid search on the training set to find the optimal hyperparameter set. During training, we randomly discarded models with a missing data rate of η = 70%. Detailed parameters are summarized in Table 1.

[0109] Table 1 Detailed hyperparameter settings for experiments on the CMU-MOSEI, CMU-MOSI, and CH-SIMS datasets

[0110] (3) Baseline model To evaluate the effectiveness and robustness of the proposed DPMP method, it is compared with the following baselines: Lower Bound (LB) is trained using different modal combinations.

[0111] Lower Bound (MS) replaces missing modals with default values ​​or placeholders.

[0112] Modality Dropout (MD) is a model that randomly drops modalities during training.

[0113] MCTN learns a robust joint representation by translating between modalities to handle missing information.

[0114] MMIN learns a robust joint multimodal representation that can predict the representation of any missing modality given the available modalities.

[0115] MAPs use missing-aware cues to guide the model to solve the missing modality problem.

[0116] MPLMM introduces three types of prompts: generation prompts, missing signal prompts, and missing type prompts to represent missing modal information in a more efficient way. (4) Results and analysis To comprehensively evaluate the effectiveness of the Dynamic Prompt Weight (DPW) module and the Modality Prior Guided Fusion (MPGF) module in DPMP, detailed experimental comparisons were conducted on three datasets. The baselines LB, MS, MD, MAPs, and MPLMM share the same backbone, demonstrating the effectiveness of DPMP in handling missing modalities. Detailed experimental results are shown in Table 2. The following section provides an in-depth analysis of these experimental results from multiple perspectives.

[0117] Overall Performance Trend Analysis. Looking at the overall results on the three datasets in Table 2, our method (Ours) achieves the best or second-best results among all existing methods in various modality-missing scenarios, demonstrating its superior generalization and robustness in addressing modality-missing scenarios. Specifically, the joint learning (MMIN) and cross-modal generation method (MCTN) show some improvement over the baseline method (LB), but perform poorly in scenarios with extreme modality-missing or severe information deficiency, suggesting that these methods still have limitations in dynamically compensating for missing modal features. Hint learning methods (MAPs and MPLMM) significantly outperform traditional methods in addressing modality-missing scenarios. MPLMM significantly outperforms MAPs due to its use of multiple cues (generation cues, missing signal cues, and missing type cues). Our method further improves performance compared to MPLMM, demonstrating that the dynamic nature of the DPW and MPGF modules, combined with their integration of prior knowledge, significantly outperforms existing static cue designs. Furthermore, our method demonstrates consistent leading performance on three different language datasets: MOSI, MOSEI, and SIMS, particularly in complex missing scenarios. This consistent and stable performance across languages ​​and datasets demonstrates the good generalization ability of the proposed method and further verifies the robustness of the DPW and MPGF modules.

[0118] Analysis of Single-Modality Scenarios. When only the text modality {t} is present, the three datasets generally achieve high baseline performance in this scenario, demonstrating the importance of the text modality in emotion recognition tasks. Our method performs outstandingly in MOSI (80.79), MOSEI (80.21), and SIMS (79.70), outperforming all other methods, such as MPLMM (80.12, 79.12, 79.75) and MAPs (79.81, 78.21, 78.56). This demonstrates that the DPW module effectively enhances the depth and accuracy of information utilization from a single text modality. Model performance is poor when only the visual modality {v} and only the audio modality {a} are present, reflecting the inability of a single non-verbal modality to fully convey emotional information in MSA tasks. Despite this, our method significantly outperforms all baselines in both the {v} and {a} scenarios across all three datasets. For example, in the vision-only scenario {v} of the SIMS dataset, the proposed method achieved an ACC of 66.39, significantly exceeding MPLMM (66.02) and MAPs (65.40). This demonstrates that the MPGF module effectively enhances the emotional expression capability of non-verbal modal information by introducing modal prior knowledge.

[0119] Table 2 Quantitative results for six possible missing modalities on the CMU-MOSI, CMU-MOSEI, and CH-SIMS datasets, where the modality missing rate η is 70% for both the training and test sets. {} denotes the available modalities; for example, "{a}" indicates that the audio modality is available, but video and text are missing. "Avg." is the average performance of the six possible cases. Bold: best result Underline: second best result

[0120] Analysis in scenarios with dual modality. While text-missing scenarios {a, v} typically pose significant challenges, our method performs reliably in this scenario, outperforming other methods in accuracy on the MOSI (65.55), MOSEI (69.08), and SIMS (66.15) datasets. In particular, on MOSI, it achieves an improvement of over 15 percentage points over the LB baseline (50.07). This demonstrates that the DPW module successfully enhances the deep complementarity between non-linguistic modalities through real-time dynamic prompt compensation. In scenarios with visual loss {a, t} and audio loss {v, t}, the presence of text modality significantly mitigates the negative impact of unimodal loss, further demonstrating the critical role of text modality in sentiment analysis. Our method achieves the best accuracy and F1 score in both the {a, t} and {v, t} scenarios across all three datasets. For example, in the {a, t} scenario on SIMS, the accuracy reaches 79.4, significantly exceeding the LB baseline's 76.84, further demonstrating the MPGF module's effective utilization of modality prior information.

[0121] (5) Effects of each component To further validate the effectiveness of each key component in DPMP, we conducted ablation experiments on each component. In this experimental setup, the modality loss rate was 70% during both the training and testing phases. The results are shown in Table 3.

[0122] Table 3 Ablation study of the contribution of each component in the DPMP model. The data in the table are the results when the missing rate of training and test modalities is 70%. The best results are shown in bold.

[0123] In the DPW-removed setting, all cues are fixed statically, no longer dynamically adjusted based on context. The results in Table 3 show that when DPW is removed, metrics such as ACC and F1 on the CMU-MOSI and CH-SIMS datasets experience varying degrees of decline. This demonstrates that static cues are unable to effectively compensate for the missing modality, affecting the model's discrimination boundary and final prediction performance. This fully demonstrates the necessity of context-based dynamic weight adjustment for addressing local modality loss.

[0124] Remove In the setup, the present invention disassembles the two types of prompts in DPW respectively to observe the effect of using only one type of prompt on the model performance. When , F1 and Corr both drop significantly. The model cannot accurately distinguish between "information missing" and "information zero", which hinders the learning of semantic compensation for missing positions and causes local compensation failure. When , the fluctuation of MAE and Corr intensifies, the existing modes lack additional regularization, the stability of modal representation decreases, and the model performance deteriorates. It is worth noting that the results in Table 3 show that removing The performance degradation is significantly greater than removing , this is due to the missing hint plays a dominant role in dynamic compensation, validating the necessity of dynamic modulation of missing cues in DPW.

[0125] When MPGF is removed, the model no longer uses any prior information for modal fusion, relying solely on DPW and a standard cross-modal interaction layer. Table 3 shows that ACC and F1 decrease, MAE increases, and the Corr metric drops significantly. The model loses its ability to provide guidance for missing priors during the fusion phase, making it difficult to capture prior associations between modalities and, consequently, unable to effectively compensate for semantic conflicts caused by missing information. This demonstrates that MPGF effectively enhances cross-modal collaboration and improves fusion stability.

[0126] Remove missing type hints In this setting, only the missing type hint is removed, retaining the fusion network structure. In this case, the model cannot identify the missing pattern of the current input modality combination, making it impossible for the fusion network to perform targeted global information fusion and compensation, resulting in a significant decline in model performance. In the MGF setting, the modal prior fusion network is removed. The modal prior fusion network is used to deeply interact the presence prompts with the current fusion representation. After removal, the prior information cannot deeply participate in the decision-making process in the fusion stage, affecting the transmission and compensation effect of global information, resulting in a certain degree of decline in the overall performance.

[0127] Overall, DPW and MPGF work best together. DPW focuses on precise compensation for locally missing modalities, while MPGF emphasizes deep collaboration across global multimodal networks. Together, they form a unified framework that adapts to missing modalities while also enabling stable fusion. The ablation results show consistent trends on CMU-MOSI and CH-SIMS datasets, two datasets with widely varying distributions of style, language, and modality features, demonstrating the robustness and universality of our approach.

[0128] Example 2 This embodiment discloses a prompt-driven multimodal sentiment analysis system for modality loss, including: A data acquisition module is configured to: acquire multimodal data including missing modalities; a model analysis module configured to: input the multimodal data into a multimodal cue learning model for analysis to obtain a sentiment prediction result; the multimodal cue learning model includes a missing modality generator, a dynamic cue weight module, a modality prior guided fusion module, and a classifier; The missing mode generator receives the multimodal data and obtains preliminary features of the missing mode in combination with the reconstruction hint of the missing mode; The dynamic prompt weight module dynamically generates a prompt weight vector corresponding to the missing modality through context perception and a prompt weight network, fuses the prompt weight vector with the missing prompt to obtain a dynamic prompt of the missing modality, and fuses the dynamic prompt with the preliminary features of the missing modality to obtain the missing modality completion feature; at the same time, based on the available modality and its corresponding existence prompt, it obtains the enhanced modality feature of the available modality; The modality prior guided fusion module receives enhanced modality features and missing modality completion features, combines missing type prompts with modality prior fusion prompts to obtain overall cross-modal fusion features, and inputs them into the classifier to obtain the final emotion prediction result.

[0129] Example 3 The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of embodiment 1 when executing the program.

[0130] Example 4 The purpose of this embodiment is to provide a computer-readable storage medium, a computer-readable storage medium having a computer program stored thereon, which performs the steps of the method of embodiment 1 when executed by a processor.

[0131] The steps involved in the apparatuses of Examples 3 and 4 above correspond to those of Method Example 1. For detailed implementation, please refer to the relevant description of Example 1. The term "computer-readable storage medium" should be understood to mean a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and causing the processor to perform any of the methods of the present invention.

[0132] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0133] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

[0134] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A prompt-driven multimodal sentiment analysis method for modality loss, characterized by: include: Obtain multimodal data including missing modalities; Inputting the multimodal data into a multimodal cue learning model for analysis to obtain an emotion prediction result; The multimodal cue learning model includes a missing modality generator, a dynamic cue weight module, a modality prior guided fusion module and a classifier; The missing mode generator receives the multimodal data and obtains preliminary features of the missing mode in combination with the reconstruction hint of the missing mode; The dynamic prompt weight module dynamically generates a prompt weight vector corresponding to the missing modality through context perception and a prompt weight network, fuses the prompt weight vector with the missing prompt to obtain a dynamic prompt of the missing modality, and fuses the dynamic prompt with the preliminary features of the missing modality to obtain the missing modality completion feature; at the same time, based on the available modality and its corresponding existence prompt, it obtains the enhanced modality feature of the available modality; The modality prior guided fusion module receives enhanced modality features and missing modality completion features, combines missing type prompts with modality prior fusion prompts to obtain overall cross-modal fusion features, and inputs them into the classifier to obtain the final emotion prediction result.

2. The prompt-driven modality-missing multimodal sentiment analysis method according to claim 1, wherein: The missing modality generator performs cross-modal feature conversion on the missing modality in the multimodal data to generate preliminary compensation features, and obtains preliminary features of the missing modality based on the reconstruction hint of the missing modality and the preliminary compensation features.

3. The prompt-driven modality-missing multimodal sentiment analysis method according to claim 1, wherein: The dynamic prompt weight module receives the features of the available modalities, projects the features of the available modalities, extracts global context features through the enhanced attention pooling module, and dynamically generates a prompt weight vector for the missing modality based on the global context features.

4. The prompt-driven modality-missing multimodal sentiment analysis method according to claim 1, wherein: The modality prior guided fusion module splices and fuses the enhanced modality features and the missing modality completion features in the cross-modal interaction network to obtain the initial cross-modality fusion features. The initial cross-modality fusion features are fused with the corresponding missing type prompts to obtain the modality fusion features. The modality fusion features are spliced ​​and combined with the modality prior fusion prompts to obtain the overall cross-modality fusion features.

5. The prompt-driven modality-missing multimodal sentiment analysis method according to claim 4, characterized in that: The modality prior guided fusion module introduces a modality prior projection matrix, and generates missing type prompts based on the modality prior projection matrix and a predefined missing type prompt set.

6. The prompt-driven modality-missing multimodal sentiment analysis method according to claim 4, characterized in that: The modality prior guided fusion module constructs a modality prior fusion network, first globally pools the presence prompts of each modality to obtain the global presence prompts of each modality, and then obtains a fusion prompt vector through the modality prior fusion network after splicing.

7. The prompt-driven modality-missing multimodal sentiment analysis method according to claim 6, characterized in that: The fusion prompt vector is mapped to the same dimensional space as the modality fusion feature to obtain a modality prior fusion prompt.

8. A prompt-driven multimodal sentiment analysis system for modality loss, characterized by: include: A data acquisition module is configured to: acquire multimodal data including missing modalities; a model analysis module configured to: input the multimodal data into a multimodal cue learning model for analysis to obtain a sentiment prediction result; the multimodal cue learning model includes a missing modality generator, a dynamic cue weight module, a modality prior guided fusion module, and a classifier; The missing mode generator receives the multimodal data and obtains preliminary features of the missing mode in combination with the reconstruction hint of the missing mode; The dynamic prompt weight module dynamically generates a prompt weight vector corresponding to the missing modality through context perception and a prompt weight network, fuses the prompt weight vector with the missing prompt to obtain a dynamic prompt of the missing modality, and fuses the dynamic prompt with the preliminary features of the missing modality to obtain the missing modality completion feature; at the same time, based on the available modality and its corresponding existence prompt, it obtains the enhanced modality feature of the available modality; The modality prior guided fusion module receives enhanced modality features and missing modality completion features, combines missing type prompts with modality prior fusion prompts to obtain overall cross-modal fusion features, and inputs them into the classifier to obtain the final emotion prediction result.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the prompt-driven modality-missing multimodal sentiment analysis method as described in any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps in the prompt-driven modality-missing multimodal sentiment analysis method according to any one of claims 1 to 7 are implemented.

Images