Multimodal sentiment analysis method and system based on semantic reconstruction and prompt refinement
By employing semantic reconstruction and cue refinement methods, missing multimodal structural features are reconstructed using text modal information and progressively refined, thus addressing the feature instability problem caused by modality loss and improving the robustness and accuracy of multimodal sentiment analysis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANDONG UNIV
- Filing Date
- 2026-04-30
- Publication Date
- 2026-07-21
Smart Images
Figure CN122433743A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of multimodal sentiment analysis technology, and particularly relates to a multimodal sentiment analysis method and system based on semantic reconstruction and prompt refinement. Background Technology
[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.
[0003] Multimodal sentiment analysis (MSA), a key task in the field of artificial intelligence, aims to infer human emotional states by jointly modeling multi-source signals such as language, audio, and vision. Leveraging the complementary advantages of multi-source signals, MSA can obtain more comprehensive and detailed emotional semantic information compared to single-modal analysis. However, in practical applications, limitations such as environmental noise, sensor failures, and network transmission errors often lead to the loss or corruption of some modal data. This incompleteness of modality not only disrupts the structural continuity of the single-modal flow at low levels, resulting in unstable feature representations or semantic dilution, but also triggers semantic discrepancies and conflicts between modalities at high levels, thus severely affecting the model's generalization ability and recognition accuracy in real-world scenarios.
[0004] Current mainstream technologies address the modality loss problem primarily employ three types of methods: feature-based reconstruction, language-driven fusion, and surrogate modality fusion. The first type attempts to recover the lost modalities using generative models before fusion; the second type utilizes the stability of textual modalities as an anchor point for integration; and the third type introduces latent variables to simulate cross-modal uncertainty. While these methods alleviate the impact of modality loss to some extent, they still face technical bottlenecks. (1) Reconstruction-based methods usually rely on low-level feature recovery at a single level, which not only makes it difficult to capture fine structural patterns, but also often introduces reconstruction noise due to imperfect recovery and propagates downstream, resulting in distortion of subsequent fusion features; language-dominated methods often lack mechanisms to explicitly repair or enhance non-textual modalities (such as audio and vision), resulting in cross-modal structural gaps that cannot be effectively filled and key complementary information is lost.
[0005] (2) Existing proxy modal or fusion methods fail to effectively enforce fine-grained semantic alignment, making it difficult to ensure the consistency of high-order cross-modal semantics while recovering missing content, resulting in a lack of discriminative power in features. At the same time, they cannot coordinate the processing of low-level structural semantic repair and high-level cross-modal semantic refinement, making the model prone to relying on false correlations or overconfident single-modal signals when faced with severe modal missingness, thus limiting its robustness in complex real-world environments. Summary of the Invention
[0006] To overcome the shortcomings of the prior art, this invention provides a multimodal sentiment analysis method and system based on semantic reconstruction and prompt refinement. By reconstructing first and then refining, it can avoid the problems of low-order structural damage and high-order semantic discrepancies caused by modality loss, thereby improving the robustness and accuracy of sentiment analysis under incomplete multimodal input.
[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a multimodal sentiment analysis method based on semantic reconstruction and prompt refinement.
[0008] Multimodal sentiment analysis methods based on semantic reconstruction and prompt refinement include: Obtain the original feature sequences of text modality, audio modality, and image modality, and project the features of each modality into a unified latent space; The original feature sequences of each modality are processed into incomplete modal features through dynamic random masking; Semantic guidance prompts are generated using the effective information of the unmasked text modalities and injected into the mask features of each modality. The missing low-order structural feature units are predicted and reconstructed by the modality decoder to obtain the reconstructed modal features. The self-guided prompts are used to progressively refine each modality, resulting in refined high-order features. The refined high-order features of each modality are then fused with the corresponding prompt features to generate a multimodal joint feature representation based on a shared fusion layer, and sentiment prediction is performed.
[0009] Furthermore, the original feature sequences of the text modality, audio modality, and image modality are obtained, including: obtaining token-level features of the text modality, audio modality, and image modality respectively through a pre-trained language model, an acoustic feature extraction tool, and a visual feature extraction network; projecting the extracted modal features to a shared embedding space through a linear mapping layer, and performing layer normalization processing to make the modal features have the same feature dimension.
[0010] Furthermore, semantic guidance prompts are generated using effective information from the unmasked text modality, including: initializing learnable prompt feature units for the audio modality and the image modality respectively; using the prompt feature units as query vectors, and using the unmasked text modality hidden layer features as key vectors and value vectors, calculating attention weights through a cross-modal attention mechanism; and aggregating semantic information from the text modality based on the attention weights to generate semantic guidance prompts for the current modality.
[0011] Furthermore, the semantic guidance prompts are injected into the mask features of each modality, including: introducing learnable adjustment parameters to control the fusion ratio of modality-specific information and external semantic guidance information; based on the adjustment parameters, performing a weighted summation of the masked modal features and the linearly transformed semantic guidance prompts to obtain semantically enhanced modal features; re-inserting mask markers at the masked positions in the semantically enhanced modal features and inputting them into the transformer decoder to recover the missing feature units.
[0012] Furthermore, the self-guided prompts are progressively refined for each modality, including two stages: self-guided refinement and semantic alignment refinement. In the first stage, the self-guided prompts for each modality are initialized, and the correlation between the self-guided prompts and the corresponding modality's reconstructed features is calculated using a self-attention mechanism to capture fine-grained structural features within the modality. In the second stage, for audio and image modalities, the self-guided prompts processed in the first stage are jointly attention-based with the text modality features. The self-guided features and text alignment features are fused using learnable gating parameters, so that the high-order features of the non-text modality move closer to the text semantic space.
[0013] Furthermore, the training process for sentiment prediction employs a multi-task joint loss function comprising sentiment prediction loss, global reconstruction loss, and mask reconstruction loss. The sentiment prediction loss is used to calculate the mean squared error between the predicted sentiment score and the true label. The global reconstruction loss constrains sequence consistency by calculating the mean squared error between the reconstructed complete feature sequence and the original feature sequence. The mask reconstruction loss only calculates the reconstruction error of the feature units at the masked locations, thereby improving the accuracy of local recovery.
[0014] Furthermore, the sentiment prediction includes: concatenating the refined modal features with the final cue features, and obtaining the global embedding representation of each modality through average pooling; adding the global embedding representations of text, audio, and image element-wise, and outputting the final sentiment analysis result through a fully connected layer.
[0015] A second aspect of this invention provides a multimodal sentiment analysis system based on semantic reconstruction and prompt refinement.
[0016] A multimodal sentiment analysis system based on semantic reconstruction and prompt refinement includes: The feature extraction and masking module is configured to: acquire the original feature sequences of text modality, audio modality and image modality, project the features of each modality onto a unified latent space; and process the original feature sequences of each modality into incomplete modality features through dynamic random masking; The semantic awareness reconstruction module is configured to: generate semantic guidance prompts using effective information of the unmasked text modal and inject them into the mask features of each modality; and predict and reconstruct the missing low-order structural feature units through the modality decoder to obtain the reconstructed modal features. The progressive prompting refinement module is configured to: progressively refine each modality through the self-guided prompts to obtain refined higher-order features; The fusion prediction module is configured to fuse the refined high-order features of each modality with the corresponding cue features, generate a multimodal joint feature representation based on the shared fusion layer, and perform sentiment prediction.
[0017] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps of the multimodal sentiment analysis method based on semantic reconstruction and prompt refinement as described in the first aspect of the present invention.
[0018] A fourth aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the multimodal sentiment analysis method based on semantic reconstruction and prompt refinement as described in the first aspect of the present invention.
[0019] The above one or more technical solutions have the following beneficial effects: This invention employs dynamic masking and reconstruction design for text, audio, and image modalities. It utilizes semantic cues to guide the recovery of low-order features of missing modalities, making it suitable not only for single-modal missing features but also for complex scenarios involving simultaneous multimodal missing features, ensuring the semantic accuracy and consistency of the recovered features. Furthermore, this invention designs a progressive refinement mechanism from single-modal to dual-modal features. By introducing learnable self-guided cues, it enhances the expressive power of fine-grained structures within each modality, solving the problem of semantic ambiguity or over-smoothing of modal features before fusion.
[0020] This invention constructs a high-order semantic alignment strategy based on hierarchical prompts. In the refining stage, semantic guidance prompts are used again to calibrate the features of each modality, ensuring that all modalities maintain a high degree of consistency in the high-level feature space, thereby effectively solving the cross-modal semantic divergence problem. At the same time, based on the unified framework of "reconstruction before refining" of this invention, it can collaboratively handle the structural deficiencies at the lower level and the semantic conflicts at the higher level, ensuring that the system still has strong robustness and recognition accuracy under the condition of missing combinations of any modality.
[0021] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0023] Figure 1 This is a flowchart of the multimodal sentiment analysis method based on semantic reconstruction and prompt refinement in Embodiment 1 of the present invention.
[0024] Figure 2 This is a schematic diagram of the operating mechanism of the three modes simultaneously performing internal random missing conditions in Embodiment 1 of the present invention.
[0025] Figure 3 This is a schematic diagram of the attention weight distribution at the feature unit level under semantic guidance in Embodiment 1 of the present invention.
[0026] Figure 4 This is a comparative schematic diagram of the framework in Embodiment 1 of the present invention; wherein, Figure 4 (a) in the diagram represents the existing single-stage "post-reconstruction fusion" framework. Figure 4 (b) in the diagram represents the existing "language-driven integration" framework. Figure 4 (c) in the diagram represents the existing "proxy modality fusion" framework. Figure 4 (d) in the diagram represents the "reconstruction first, refinement later" framework of the present invention.
[0027] Figure 5 This is a visualization diagram of the cross-modal attention heatmap under semantic guidance in Embodiment 1 of the present invention; wherein, Figure 5 In the diagram, (a) represents the word-level attention heatmap from text to vision. Figure 5 In the diagram, (b) represents the word-level attention heatmap from text to audio. Figure 5 (c) in the figure represents the distribution of attention differences between visual and audio. Detailed Implementation
[0028] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0029] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.
[0030] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.
[0031] Example 1 This embodiment discloses a multimodal sentiment analysis method based on semantic reconstruction and prompt refinement.
[0032] like Figure 1 As shown, the multimodal sentiment analysis method based on semantic reconstruction and prompt refinement includes: Step S1: Obtain the original feature sequences of text modality, audio modality and image modality, and project the features of each modality into a unified latent space; process the original feature sequences of each modality into incomplete modality features through dynamic random masking; Step S2: Generate semantic guidance prompts using the effective information of the unmasked text modal and inject them into the mask features of each modality. Then, predict and reconstruct the missing low-order structural feature units through the modality decoder to obtain the reconstructed modal features. Step S3: The self-guided prompts are used to progressively refine each modality to obtain the refined higher-order features; Step S4: Fuse the refined high-order features of each modality with the corresponding cue features, generate a multimodal joint feature representation based on the shared fusion layer, and perform sentiment prediction.
[0033] Based on the above process, this invention, through a reconstruction-then-refinement approach, avoids the problems of low-order structural damage and high-order semantic discrepancies caused by modality loss, thereby improving the robustness and accuracy of sentiment analysis under incomplete multimodal input. To facilitate understanding of the technical solution of this invention, the specific implementation methods are further explained and described below.
[0034] In step S1, the original feature sequences of text, audio and image are obtained, and each modal feature is projected onto a latent space of a unified dimension; then, dynamic random masking operation is performed on the feature sequences of the three modalities of text, audio and image respectively to generate masked incomplete modal features.
[0035] Token-level features for text, audio, and image modalities are obtained using a pre-trained language model, an acoustic feature extraction tool, and a visual feature extraction network, respectively. The extracted modal features are then projected onto a shared embedding space through a linear mapping layer and normalized to ensure that all modal features have the same feature dimensionality. This can be achieved through the following method: 1) Text modality: The input text is processed using a BERT pre-trained encoder to extract token-level feature sequences that can represent the semantic context.
[0036] 2) Audio modalities: The audio signal was processed using the Librosa acoustic signal analysis toolkit to extract the acoustic feature sequences at each time step.
[0037] 3) Visual modality: The video frames were processed using the OpenFace facial behavior analysis toolkit to extract visual feature sequences containing facial motion units and gaze estimation.
[0038] Furthermore, recording modes The characteristic sequence is ;in, Represents text, Indicates audio. Indicates visual; Indicates the sequence length. This represents the original feature dimension. To unify the feature processing dimension and facilitate subsequent interaction, the extracted modal feature sequences are... The inputs are respectively fed into a linear mapping layer and projected onto a dimension of... The shared latent space is normalized to obtain the standardized input features.
[0039] In step S2, semantic guidance prompts are generated using the effective modal information of the unmasked text; the semantic guidance prompts are injected into the mask features of each modality, and the missing low-order structural feature units are predicted and reconstructed by the modality-specific decoder, so as to achieve preliminary low-order semantic alignment while restoring the integrity of the feature sequence.
[0040] To address the issues of introducing noise and discarding lost information when directly using damaged modalities, this invention constructs a semantic-aware reconstruction module. This module extends masking autoencoder technology to multimodal input, utilizing text modalities as stable semantic anchors to guide the recovery of damaged modalities.
[0041] First, a random masking strategy is applied to the text modality, audio modality, and image modality respectively. A binary indicator vector is defined. Feature sequence after masking The calculation is as follows: ; in, Represents the primitive mode The first in the sequence Each feature unit Indicates the first The feature units are not masked (i.e., they are visible); Indicates the first Each feature unit is masked.
[0042] Only satisfy The unmasked visible feature units are input into a modality-specific Transformer encoder to capture context-aware hidden layer representations. Let... The operation that removes the masked position results in the context-aware hidden state of the remaining subsequence. The calculation is as follows: ; in, This indicates a mode-specific Transformer encoder. Representing modes The feature sequence after masking, This represents a binary indicator vector used to indicate the mask location.
[0043] To provide explicit semantic guidance for non-textual modalities, specific guidelines are needed for audio and image modalities (i.e., ...). Introducing learnable cue feature units Through cross-modal attention operations and the hidden state of the text modality To interact. Specifically, by Generate query vectors and from text features through linear projection Obtaining the key vector Sum value vector The calculation is as follows: ; in, This indicates semantic guidance or prompting. This represents the matrix transpose operation; This represents the dimension of the feature vector, used to scale the dot product to prevent gradient vanishing.
[0044] This operation extracts semantic guidance prompts rich in emotional information from the text. Since textual modalities typically contain clear emotional indications, this cue acts as a key semantic anchor during the reconstruction process.
[0045] To incorporate the aforementioned semantic cues into the representation of the damaged modality, they are combined with context-aware hidden states. The fusion process involves first reshaping the cue feature units into shapes that match the hidden layer dimensions, and then fusing them through a learnable interpolation weight. Injection: ; in, This represents the semantically enhanced representation, which can be learned parameters. It is used to control the balance between preserving modality-specific information and introducing text-derived semantic guidance.
[0046] Finally, the semantically enhanced representation The data is fed into the reconstruction decoding network. The recovery operation is then performed. Placeholders are re-inserted at the masked positions to ensure the original sequence order is restored, and decoding is performed using a lightweight Transformer decoder: ; in, This indicates a lightweight Transformer decoder. This represents the reconstructed feature sequence.
[0047] This process enables the model to predict missing feature units, restoring sequence continuity and structural fidelity while aligning with sentiment-related textual cues, thus transforming the reconstruction task into a semantically grounded reconstruction.
[0048] The reconstruction process is supervised by two complementary loss functions: global reconstruction loss and mask reconstruction loss. The global reconstruction loss enforces consistency across the entire sequence, while the mask reconstruction loss emphasizes the accuracy of recovering damaged regions. The global reconstruction loss and mask reconstruction loss are expressed as follows: ; ; in, Indicates the global reconstruction loss. Represents the mean square error function; This represents the loss from mask reconstruction.
[0049] The combination of these two approaches ensures both local fidelity and global coherence, enabling the model to maintain the integrity of the sequence while recovering emotion-related details.
[0050] In the inference phase, to improve efficiency, the decoder is omitted. The encoder output, which has already been enhanced through semantic guidance during training, is directly passed to the progressive cue refinement module in the next stage. This design not only repairs incomplete modalities but also provides lightweight and semantically consistent feature representations for downstream fusion.
[0051] like Figure 2 As shown, the overall network architecture involved in this invention mainly includes two core modules: a semantic perception reconstruction module and a progressive prompt refinement module. The semantic perception reconstruction module uses semantic guidance prompts from the text modality as anchors to guide the reconstruction of damaged (masked) audio and visual feature units with structural fidelity and semantic grounding. The progressive prompt refinement module introduces learnable self-guided prompts to interact with single-modal features to extract fine-grained information, which is then aligned and fused with text semantic prompts. Finally, multimodal sentiment prediction is achieved through pooling and linear layers.
[0052] like Figure 3The diagram shows the distribution of attention weights at the feature unit level under semantic guidance. It reveals the distribution of text-guided cross-modal attention weights at the token level during the semantic perception reconstruction stage. The blue and orange bars represent the absolute attention weights of the text to the visual and audio modalities, respectively. The analysis shows that, guided by textual semantics, both visual and audio modalities exhibit consistent cross-modal activation patterns while retaining their own independent activation peaks. This verifies that text not only provides basic semantic anchors for cross-modal communication but also offers differentiated activation guidance based on the structural characteristics of different modalities.
[0053] In step S3, a progressive prompt refinement operation is performed, namely: a learnable self-guided prompt is introduced for each modality. First, fine-grained discriminative information within each modality is extracted through a self-attention mechanism. Then, cross-modal interaction is performed in conjunction with the semantic guidance prompts to calibrate the features of each modality to a unified semantic space, thereby obtaining refined features with high-order semantic consistency.
[0054] To enhance cross-modal semantic coherence while preserving modality-specific information, this invention constructs a progressive cue refinement module. This module gradually transforms local unimodal cues into robust multimodal representations through two stages: self-guided refinement and semantically guided refinement.
[0055] In the specific implementation process, for each mode (representing text, audio, and image respectively), initialize a learnable cue vector. The cue vector and the reconstructed feature sequence of the modality itself. The interaction was performed, and the calculations are as follows: ; in, This represents the self-guided prompt feature vector obtained after the first stage of self-attention mechanism interaction. This represents the standard attention mechanism computation function. , and This represents the linear mapping function in the self-attention mechanism. This step enhances intramodal consistency by having each cue vector focus on the most informative region within its modality (e.g., audio cue vectors might focus on pitch and intensity patterns, while visual cue vectors might focus on facial expressions). By preserving these discriminative cues, it ensures that single-modal features are not masked or diluted before cross-modal alignment.
[0056] To align non-textual modalities with textual semantics, for audio and image modalities, a joint alignment calculation is performed between their self-guided cue vectors and their own features, as well as the feature representations of the textual modalities: ; in, Indicates the concatenation of features. For text modal features, The weights are learnable and adjustable (initialized to 0.5 at the start of training and optimized during training). Representing modes The final semantic alignment cue feature vector. This step "landed" the acoustic and visual flow into the text semantics, ensuring that the cross-modal fusion process focuses on emotion-related patterns and achieves higher-order semantic calibration.
[0057] like Figure 4 The diagram shows a comparison of the frames shown; in which, Figure 4 (a) in the diagram represents the existing single-stage "post-reconstruction fusion" framework. Figure 4 (b) in the diagram represents the existing "language-driven integration" framework. Figure 4 (c) in the diagram represents the existing "proxy modality fusion" framework. Figure 4 (d) in the figure represents a schematic diagram of the "reconstruction-then-refinement" framework of the present invention. This figure intuitively illustrates that the present invention not only restores the damaged structural information at the bottom layer through a masking mechanism combined with semantic guidance prompts, but also enhances the consistency of cross-modal semantics at the top layer through a self-guiding prompt mechanism.
[0058] In step S4, the refined modal features and corresponding cue features are concatenated and aggregated to generate a multimodal joint feature representation based on the shared fusion layer, and sentiment classification or regression prediction is performed.
[0059] After feature refinement, the final cue vector is concatenated with the corresponding reconstructed features, and then aggregated into a modality-level embedding representation through average pooling. : ; in, This indicates the average pooling operation.
[0060] Subsequently, the embedding representations of the three modalities are combined through element-wise summation to form the final multimodal joint representation. It generates sentiment prediction results through a fully connected linear layer. : ; ; in, Representing the embedding representation at the text modality level. An embedding representation representing the audio modality level. Embedded representations at the visual modality level; This represents a multimodal joint representation. This indicates the final sentiment prediction result. This represents a fully connected linear layer mapping operation.
[0061] Through the above process, this embodiment preserves the fine unimodal structure while aligning it with the text guidance, thereby generating a representation that is both discriminative and semantically coherent for final sentiment analysis.
[0062] Furthermore, the method proposed in this invention performs overall optimization by jointly training the reconstruction target and the prediction target, resulting in a total loss function. Defined as: ; in, It is the sentiment prediction loss, which is calculated by the mean squared error (MSE) between the predicted value and the true sentiment label to ensure the model's ability to discriminate for the target task; It is the mask reconstruction loss, used to improve the accuracy of feature recovery in damaged areas; It is a global reconstruction loss, used to maintain the overall coherence at the sequence level; , and These are the weighting coefficients that balance the various loss terms. In this embodiment, they are set to 1, 0.05, and 0.05 respectively based on experience.
[0063] By integrating these objectives, this invention achieves a complete technical paradigm of "reconstruction first, then refinement": using a semantic-aware reconstruction module to physically recover low-order feature units, and using a progressive cueing refinement module to logically refine high-order semantics. This combination significantly enhances the robustness of the model under noise interference or modal incompleteness conditions, while ensuring high efficiency in the inference phase.
[0064] like Figure 5 The diagram shows a visualization of a semantically guided cross-modal attention heatmap; where, Figure 5 In the diagram, (a) represents the word-level attention heatmap from text to vision. Figure 5 In the diagram, (b) represents the word-level attention heatmap from text to audio. Figure 5 (c) in the figure represents the attention difference distribution map between visual and audio modes. The analysis results intuitively show that the attention signal generated by the text can accurately and selectively focus on semantically salient regions in each modality (such as spatially coherent segments in the visual modality and temporally discriminative frames in the audio modality), further proving that the semantic guidance mechanism in this invention can achieve accurate and complementary cross-modal feature alignment.
[0065] To verify the proposed multimodal sentiment analysis method based on semantic reconstruction and prompt refinement (hereinafter referred to as the method), this invention aims to... To assess the effectiveness of the model, this embodiment was extensively evaluated on three standard multimodal sentiment analysis benchmark datasets, including CMU-MOSI, CMU-MOSEI, and SIMS.
[0066] The CMU-MOSI dataset contains 2199 video clips with sentiment scores (1284 for training, 229 for validation, and 686 for testing), with sentiment scores ranging from -3 to +3.
[0067] The CMU-MOSEI dataset is an extended version of MOSI, containing 22,856 video clips (16,326 for training, 1,871 for validation, and 4,659 for testing), with the sentiment rating range also being [-3, +3].
[0068] SIMS dataset: A Chinese multimodal dataset containing 2281 video clips (1368 in the training set, 456 in the validation set, and 457 in the test set), with sentiment labels ranging from -1 to +1.
[0069] These datasets cover English and Chinese, scripted and spontaneous speech, providing diverse scenarios for validating the generalization ability of the models.
[0070] For evaluation metrics, accuracy (ACC-2) and F1 score were used for binary classification tasks (positive / negative sentiment); multi-class accuracy (ACC-3, ACC-5, ACC-7) was used for fine-grained sentiment classification; and mean absolute error (MAE) and Pearson correlation coefficient (Corr) were used for regression tasks. All experimental results are the average of three randomized seed runs.
[0071] To evaluate the model's robustness in modality-missing scenarios, this embodiment employs a random masking strategy to simulate incomplete input. During the testing phase, feature units in each modality are proportionally... Random cover is performed, among which, The specific comparison results are shown in Tables 1-3, and the comparisons in Tables 1-3 are average index values with a missing rate of 0.0-0.9. Furthermore, all models were trained on an NVIDIA RTX 6000 Ada GPU (48GB VRAM). The number of training epochs was set to 200, and repeated experiments were performed using random seeds 1111, 1112, and 1113 to ensure the reliability of the results.
[0072] Table 1. Comparison of the method of this invention with other algorithms on the MOSI dataset.
[0073] As shown in Table 1, using the MOSI dataset, the proposed method achieves superior performance across all evaluation metrics compared to existing baseline algorithms. In particular, it demonstrates significant improvements in binary classification accuracy (ACC-2 accuracy of 72.22 / 73.93) and F1 score, while reducing the mean absolute error (MAE) to 1.033. The analysis results indicate that by combining low-level mask feature reconstruction with high-level semantic cue refinement mechanisms, this invention can effectively correct reconstruction noise and provide more reliable sentiment prediction results even with incomplete modality data.
[0074] Table 2 compares the method of this invention with other algorithms on the MOSEI dataset.
[0075] As shown in Table 2, using the MOSEI dataset for comparison, the method of this invention also exhibits consistent and stable performance improvements, especially in fine-grained sentiment classification (ACC-5 reaches 48.25, ACC-7 reaches 46.79), demonstrating a significant advantage. Furthermore, the MAE is further reduced to 0.652, and the Pearson correlation coefficient (Corr) is improved to 0.597. This reflects the high efficiency and robustness of the unified "reconstruction-refinement" strategy of this invention when processing large-scale and complex multimodal interaction data.
[0076] Table 3. Comparison of the proposed method with other algorithms on the SIMS dataset.
[0077] As shown in Table 3, using the SIMS Chinese dataset for comparison, this invention still achieves the best balanced robustness. Its ACC-2 score is improved to 74.67, and its MAE is reduced to 0.496. This analysis highlights the mechanism of combining semantic-aware reconstruction with progressive refinement, which can still extract stable and discriminative joint feature representations even in real-world scenarios with severe modality loss and high noise.
[0078] Example 2 This embodiment discloses a multimodal sentiment analysis system based on semantic reconstruction and prompt refinement.
[0079] A multimodal sentiment analysis system based on semantic reconstruction and prompt refinement includes: The feature extraction and masking module is configured to: acquire the original feature sequences of text modality, audio modality and image modality, project the features of each modality onto a unified latent space; and process the original feature sequences of each modality into incomplete modality features through dynamic random masking; The semantic awareness reconstruction module is configured to: generate semantic guidance prompts using effective information of the unmasked text modal and inject them into the mask features of each modality; and predict and reconstruct the missing low-order structural feature units through the modality decoder to obtain the reconstructed modal features. The progressive prompting refinement module is configured to: progressively refine each modality through the self-guided prompts to obtain refined higher-order features; The fusion prediction module is configured to fuse the refined high-order features of each modality with the corresponding cue features, generate a multimodal joint feature representation based on the shared fusion layer, and perform sentiment prediction.
[0080] Example 3 The purpose of this embodiment is to provide a computer-readable storage medium.
[0081] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the multimodal sentiment analysis method based on semantic reconstruction and prompt refinement as described in Embodiment 1 of this disclosure.
[0082] Example 4 The purpose of this embodiment is to provide an electronic device.
[0083] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the multimodal sentiment analysis method based on semantic reconstruction and prompt refinement as described in Embodiment 1 of this disclosure.
[0084] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.
[0085] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.
[0086] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.
Claims
1. A multimodal sentiment analysis method based on semantic reconstruction and prompt refinement, characterized in that, include: Obtain the original feature sequences of text modality, audio modality, and image modality, and project the features of each modality into a unified latent space; The original feature sequences of each modality are processed into incomplete modal features through dynamic random masking; Semantic guidance prompts are generated using the effective information of the unmasked text modalities and injected into the mask features of each modality. The missing low-order structural feature units are predicted and reconstructed by the modality decoder to obtain the reconstructed modal features. The self-guided prompts are used to progressively refine each modality, resulting in refined high-order features. The refined high-order features of each modality are then fused with the corresponding prompt features to generate a multimodal joint feature representation based on a shared fusion layer, and sentiment prediction is performed.
2. The multimodal sentiment analysis method based on semantic reconstruction and prompt refinement as described in claim 1, characterized in that, Obtain the original feature sequences of text modality, audio modality, and image modality, including: obtaining token-level features of text modality, audio modality, and image modality respectively through a pre-trained language model, an acoustic feature extraction tool, and a visual feature extraction network; projecting the extracted modal features to a shared embedding space through a linear mapping layer and performing layer normalization to make each modal feature have the same feature dimension.
3. The multimodal sentiment analysis method based on semantic reconstruction and prompt refinement as described in claim 1, characterized in that, Generating semantic guidance prompts using effective information from unmasked text modalities includes: initializing learnable prompt feature units for audio and image modalities respectively; using the prompt feature units as query vectors, and using the unmasked text modal hidden layer features as key and value vectors, calculating attention weights through a cross-modal attention mechanism; and aggregating semantic information from the text modal based on the attention weights to generate semantic guidance prompts for the current modality.
4. The multimodal sentiment analysis method based on semantic reconstruction and prompt refinement as described in claim 1, characterized in that, Injecting the semantic guidance prompts into the mask features of each modality includes: introducing learnable adjustment parameters to control the fusion ratio of modality-specific information and external semantic guidance information; based on the adjustment parameters, performing a weighted summation of the masked modal features and the linearly transformed semantic guidance prompts to obtain semantically enhanced modal features; re-inserting mask markers at the masked positions in the semantically enhanced modal features and inputting them into the transformer decoder to recover the missing feature units.
5. The multimodal sentiment analysis method based on semantic reconstruction and prompt refinement as described in claim 1, characterized in that, The self-guided prompts are used to progressively refine each modality, including two stages: self-guided refinement and semantic alignment refinement. In the first stage, the self-guided prompts of each modality are initialized, and the correlation between the self-guided prompts and the corresponding modality’s own reconstruction features is calculated using a self-attention mechanism to capture the fine-grained structural features within the modality. In the second stage, for audio and image modalities, the self-guided prompts processed in the first stage are combined with text modal features for joint attention calculation. The self-guided features and text alignment features are fused through learnable gating parameters, so that the high-order features of non-text modalities move closer to the text semantic space.
6. The multimodal sentiment analysis method based on semantic reconstruction and prompt refinement as described in claim 1, characterized in that, The training process for sentiment prediction employs a multi-task joint loss function comprising sentiment prediction loss, global reconstruction loss, and mask reconstruction loss. The sentiment prediction loss calculates the mean squared error between the predicted sentiment score and the true label. The global reconstruction loss constrains sequence consistency by calculating the mean squared error between the reconstructed complete feature sequence and the original feature sequence. The mask reconstruction loss calculates only the reconstruction error of the feature units at the masked locations, improving the accuracy of local recovery.
7. The multimodal sentiment analysis method based on semantic reconstruction and prompt refinement as described in claim 1, characterized in that, The sentiment prediction includes: concatenating the refined modal features with the final cue features, and obtaining the global embedding representation of each modality through average pooling; adding the global embedding representations of text, audio and image element by element, and outputting the final sentiment analysis result through a fully connected layer.
8. A multimodal sentiment analysis system based on semantic reconstruction and prompt refinement, characterized in that: include: The feature extraction and masking module is configured to: acquire the original feature sequences of text modality, audio modality and image modality, project the features of each modality onto a unified latent space; and process the original feature sequences of each modality into incomplete modality features through dynamic random masking; The semantic awareness reconstruction module is configured to: generate semantic guidance prompts using effective information of the unmasked text modal and inject them into the mask features of each modality; and predict and reconstruct the missing low-order structural feature units through the modality decoder to obtain the reconstructed modal features. The progressive prompting refinement module is configured to: progressively refine each modality through the self-guided prompts to obtain refined higher-order features; The fusion prediction module is configured to fuse the refined high-order features of each modality with the corresponding cue features, generate a multimodal joint feature representation based on the shared fusion layer, and perform sentiment prediction.
9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the multimodal sentiment analysis method based on semantic reconstruction and prompt refinement as described in any one of claims 1-7.
10. An electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the multimodal sentiment analysis method based on semantic reconstruction and prompt refinement as described in any one of claims 1-7.