A text-guided multi-modal sentiment analysis method under an incomplete data scenario

Abstract

The present application belongs to the technical field of multi-modal sentiment analysis, and discloses a text-guided multi-modal sentiment analysis method under an incomplete data scene: step one: original multi-modal samples are subjected to feature extraction to obtain original complete modal representations; random missing processing is performed to obtain incomplete multi-modal inputs, and a training set is obtained; step two: a multi-modal sentiment analysis model is constructed; step three: the multi-modal sentiment analysis model is trained to obtain a trained multi-modal sentiment analysis model; step four: the multi-modal samples to be predicted are subjected to feature extraction, and information missing positions are subjected to filling processing to obtain incomplete multi-modal inputs to be predicted; the incomplete multi-modal inputs to be predicted are sent to the trained multi-modal sentiment analysis model, and a sentiment prediction result is output. The method of the present application can solve the problem of serious decline in sentiment analysis performance of the prior art under a multi-modal data random missing scene.

Description

Technical Field

[0001] This invention belongs to the field of multimodal sentiment analysis technology, and relates to a text-guided multimodal sentiment analysis method in the context of incomplete data. Background Technology

[0002] Multimodal sentiment analysis integrates information from multiple sources, including text, speech, and vision, leveraging the correlation and complementarity between different modalities to simulate the comprehensive processing of multisensory information by humans during emotional expression. In recent years, Transformer-based multimodal fusion methods have made significant progress. Their core idea is to use cross-modal attention mechanisms to align and fuse semantic information from different modalities. However, the practical deployment and application of existing methods are severely limited by their dependence on complete modal input. Specifically, in real-world applications, multimodal data often suffers from varying degrees of loss or incompleteness due to factors such as data acquisition conditions, environmental noise interference, sensor malfunctions, and privacy restrictions. For example, camera occlusion can lead to the loss of visual information, microphone malfunctions can cause speech signal interruptions, and errors in automatic speech recognition can result in information loss in the text modality. When modal data is incomplete, the cross-modal alignment process of traditional attention mechanisms suffers severe distortion due to missing or noisy tokens, leading to a sharp decline in model performance.

[0003] To address the problem of missing modalities, existing technologies attempt to complete missing modalities using generative methods or directly model available modalities through joint learning frameworks. Generative methods attempt to complete missing modal information using generative adversarial networks or variational autoencoders, while joint learning methods attempt to directly learn robust multimodal representations under missing conditions. However, these methods generally suffer from two limitations: First, they lack proactive protection and correction of the dominant modality. In multimodal sentiment analysis, the text modality usually contains the richest and most direct emotional cues. When the text modality itself is impaired due to noise or recognition errors, existing methods fail to proactively evaluate and enhance the reliability of the dominant modality in missing scenarios, resulting in subsequent fusion being based on unreliable representations. Second, the cross-modal fusion mechanism lacks sufficient robustness against interference. The fusion structure based on the Transformer attention mechanism is highly sensitive to the integrity of the input. In scenarios with missing modalities or noise interference, its alignment process and weight allocation are prone to distortion: missing modalities make the interactive objects unreliable, while noisy modalities may introduce erroneous signals and be further amplified, ultimately leading to large fluctuations in the fusion representation and unstable generalization.

[0004] Therefore, how to enable models to actively perceive modal integrity, autonomously correct dominant modal representations, and achieve stable cross-modal fusion under incomplete data conditions is a technical challenge that urgently needs to be solved in this field. Summary of the Invention

[0005] The purpose of this invention is to provide a text-guided multimodal sentiment analysis method for incomplete data scenarios, in order to solve the problem that the sentiment analysis performance of existing technologies is severely reduced in scenarios with random missing multimodal data due to the lack of a dominant modality protection mechanism and insufficient anti-interference ability of cross-modal fusion.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A text-guided multimodal sentiment analysis method for scenarios with incomplete data includes the following steps: Step 1: Extract features from the original multimodal samples to obtain the original complete modality representation, which includes three modalities: text modality, speech modality, and visual modality; perform random missing operations on the original complete modality representation to obtain incomplete multimodal input, which includes incomplete text modality input, incomplete speech modality input, and incomplete visual modality input; and use the incomplete multimodal input and its corresponding real sentiment values ​​to form a training set. Step 2: Construct a multimodal sentiment analysis model to implement the following process: S1: For each modality of the incomplete multimodal input, feature extraction and dimensionality unification are performed to obtain incomplete modal representations, including incomplete text modal representation, incomplete speech modal representation, and incomplete visual modal representation; S2: Using the text modality as the dominant modality, perform dominant modality correction on the incomplete modal representations obtained in S1, including: performing integrity prediction on the incomplete text modal representation to obtain the integrity prediction value of the text modality, and constructing an integrity check loss based on the integrity prediction value of the text modality; generating a proxy dominant representation based on the incomplete speech modal representation and the incomplete visual modal representation, using the proxy dominant representation and the incomplete text modal representation as the two types of inputs to the discriminator, respectively, and constructing an adversarial learning loss; and using the integrity prediction value of the text modality to perform adaptive weighted fusion on the proxy dominant representation and the incomplete text modal representation to obtain the corrected dominant representation. S3: Construct an adaptive supermodal learning unit guided by the dominant modality. The adaptive supermodal learning unit is composed of multiple adaptive supermodal learning layers stacked together. The corrected dominant representation obtained in S2 is encoded layer by layer to obtain corrected dominant representations at different levels. The corrected dominant representations at different levels are used to guide the interaction between the incomplete speech modality representation and the incomplete visual modality representation obtained in S1, and the supermodal representation is updated layer by layer. Subsequently, cross-modal sequence modeling is performed by combining the corrected dominant representation of the final layer and the supermodal representation to obtain a cross-modal fusion representation. S4: Input the cross-modal fusion representation obtained in S3 into the sentiment predictor, obtain the sentiment prediction value, and construct the sentiment prediction loss based on the sentiment prediction value and the corresponding real sentiment value; S5: Construct a reconstructor for each modality. Input the incomplete multimodal input obtained in step one into the reconstructor for each modality to obtain the reconstructed representation corresponding to each original complete modality representation. Construct the reconstructed loss based on the difference between the reconstructed representation and the corresponding original complete modality representation. S6: The total loss function is obtained by weighted summing of the integrity check loss obtained in S2, the adversarial learning loss obtained in S4, and the reconstruction loss obtained in S5. Step 3: Use the training set obtained in Step 1 to train the multimodal sentiment analysis model constructed in Step 2, and obtain the trained multimodal sentiment analysis model; Step 4: Extract features from the multimodal samples to be predicted, fill in the missing information in the feature extraction results to obtain the incomplete multimodal input to be predicted; send the incomplete multimodal input to be predicted to the trained multimodal sentiment analysis model obtained in Step 3, and output the sentiment prediction result.

[0008] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention performs integrity checks on text modalities under incomplete data conditions and generates proxy dominant representations using speech and visual modalities to achieve adaptive correction and enhancement of dominant modalities, thereby effectively alleviating the problem of weakening of dominant sentiment cues caused by modal missingness or noise interference, and achieving highly robust sentiment prediction under incomplete data conditions.

[0009] (2) The present invention designs a cross-modal Mamba fusion mechanism, which replaces the traditional Transformer attention mechanism with a bidirectional Mamba network for cross-modal fusion. By modeling the sequence dependency relationship through a state space model, a more stable cross-modal fusion representation can be obtained under incomplete data conditions, effectively alleviating the alignment distortion and weight distribution divergence problems that easily occur in attention mechanism under modality missing or noise interference scenarios.

[0010] (3) This invention coordinates the design of the dominant modality correction and dominant modality-guided adaptive supermodal learning unit and the reconstructor, and achieves the organic unity of missing information recovery and dominant modality enhancement through multi-task loss joint optimization, further improving the sentiment analysis performance and generalization ability of the model in incomplete data scenarios.

[0011] (4) This invention utilizes different levels of correction-dominant representations to guide incomplete speech modal representations and incomplete visual modal representations to interact in parallel, and accumulates the speech modal information and visual modal information obtained from each layer of interaction to the supermodal representation layer by layer, so that the complementary emotional information in the speech modal and visual modal gradually converges under the guidance of the correction-dominant representation, reducing the interference of missing information and noise information on the subsequent fusion process, thereby improving the discriminability and stability of the cross-modal fusion representation. Detailed Implementation

[0012] The technical approach of this invention is as follows: First, features are extracted from the input text, speech, and visual modal data to obtain the original complete modal representation. Then, random missing data processing is applied to the original complete modal representation to construct an incomplete multimodal input, and further feature extraction and dimensionality unification are performed to obtain incomplete modal representations. Next, the integrity of the text modality is evaluated through dominant modality correction, and surrogate dominant representations are generated using the speech and visual modalities. The incomplete text modal representation is adaptively corrected and enhanced using the surrogate dominant representation to obtain the corrected dominant modal representation, i.e., the corrected dominant representation. Based on this, an adaptive supermodal learning unit guided by the dominant modality is used, guided by the corrected dominant modal representation, to generate a supermodal representation through multi-head attention interaction between the adaptive supermodal learning layer and the speech and visual modalities. The corrected dominant modal representation and the supermodal representation are then input into a bidirectional Mamba network. Cross-modal sequence dependencies are captured through bidirectional Mamba interaction to obtain a cross-modal enhanced representation, which is then enhanced by residual connections and a feedforward network to output a cross-modal fusion representation. Furthermore, this invention also includes a reconstructor for reconstructing missing information from each modality. Finally, the cross-modal fusion representation is input into the sentiment predictor to obtain the sentiment prediction result, thus achieving multimodal sentiment prediction output.

[0013] In practical applications of multimodal sentiment analysis, text, speech, and visual modalities often suffer from varying degrees of loss or incompleteness due to factors such as data acquisition conditions, environmental noise interference, and sensor malfunctions. For example, speech signals may be submerged by background noise or partially lost due to microphone failure; facial expressions may be missed due to changes in posture or camera capture failure; and textual modalities may suffer from information loss due to errors in automatic speech recognition or data truncation. To address these issues, simply fusing multimodal features or relying on attention mechanisms for interaction can lead to distortions in cross-modal alignment when modal data is incomplete. Missing modalities can render interactive objects unreliable, thus affecting the accuracy and stability of sentiment analysis results.

[0014] This invention introduces a dominant modality correction mechanism, employing integrity checks to dynamically assess the usability of text modalities and generating surrogate dominant representations using speech and visual modalities. Based on the integrity prediction values ​​of the text modalities, it adaptively weights and fuses incomplete text modal representations with surrogate dominant representations, enabling the model to obtain reliable semantic supplementation when the dominant modality is impaired, thus fundamentally avoiding the weakening of dominant sentiment cues. Simultaneously, this invention utilizes a bidirectional Mamba network to replace the traditional Transformer attention mechanism for cross-modal fusion, modeling sequence dependencies through a state-space model. Compared to the attention mechanism, this approach is more robust to missing tokens and noise interference in the input, effectively suppressing the spread of erroneous information. Furthermore, through multiple auxiliary optimizations using integrity check loss, adversarial learning loss, and reconstruction loss, the model is guided to learn more robust modal representations and the ability to recover missing information. Therefore, the proposed method can achieve stable and effective multimodal sentiment feature modeling in modality-deficient scenarios, improving the overall performance and robustness of multimodal sentiment analysis tasks, demonstrating good feasibility and beneficial effects.

[0015] To verify the feasibility and effectiveness of this invention, comparative experiments were conducted on three publicly available multimodal sentiment analysis datasets: MOSI, MOSEI, and SIMS, comparing the proposed method with existing mainstream methods. Experimental results show that, under the same experimental conditions, the proposed method achieves optimal or near-optimal results on multiple evaluation metrics, including Acc-7, Acc-2, and MAE. Further analysis indicates that by introducing a dominant modality correction mechanism, the model can effectively compensate for text modal impairments through surrogate dominant representations. Furthermore, as the modality missing rate increases from 0 to 0.9, the performance degradation of the proposed method on most evaluation metrics is relatively smaller than that of existing methods. The use of the cross-modal Mamba fusion mechanism enables the model to maintain the discriminative power of the fused representation more stably during multimodal interactions under incomplete data conditions, demonstrating good robustness under different levels of missing data.

[0016] Example 1 This embodiment presents a text-guided multimodal sentiment analysis method for scenarios with incomplete data, including the following steps: Step 1: Extract features from the original multimodal samples to obtain the original complete modality representation, which includes three modalities: text modality, speech modality, and visual modality. Randomly omit the original complete modality representation to obtain incomplete multimodal input, which includes incomplete text modality input, incomplete speech modality input, and incomplete visual modality input. The incomplete multimodal input and its corresponding real sentiment values ​​are used to form a training set.

[0017] Step one specifically includes the following operations: Obtain the original multimodal samples and extract features from them to obtain the original complete modality representation. ,in Representing text modality, speech modality, and visual modality respectively; representing the original complete modality. Random missing data processing is performed to obtain incomplete multimodal input, including incomplete text modal input. Incomplete speech modal input and incomplete visual modal input Specifically, random missing data handling includes: ① For the first... For each modality of a sample, according to a preset missing proportion Randomly erase some information, and use the complement of the missing proportion as the integrity label for the text modality of that sample. ; ② For feature sequences in incomplete speech modal input and incomplete visual modal input, fill the erased positions with zero vectors of the corresponding dimensions; for text word sequences in incomplete text modal input, fill the erased text words with unknown word markers [UNK] from the BERT vocabulary.

[0018] Step 2: Construct a multimodal sentiment analysis model to implement the following process: S1: Perform feature extraction and dimensionality unification on each modality of the incomplete multimodal input to obtain incomplete modal representations, including incomplete text modal representation, incomplete speech modal representation, and incomplete visual modal representation.

[0019] S1 specifically includes the following operations: For each modality, a low-dimensional token will be randomly initialized. With mode Corresponding incomplete modal input Perform vector concatenation (where Indicates the sequence length. This represents the feature dimension of the corresponding modality. (These represent text modality, speech modality, and visual modality, respectively), and are then input into a modality embedding encoder for feature extraction and dimensionality unification, resulting in incomplete representations of each modality. :

[0020] in, Incomplete modal representations, among which These represent text modality, speech modality, and visual modality, respectively. This indicates that each modality is embedded with an encoder; in this invention, a two-layer Transformer encoder is used. This represents a vector concatenation operation; This represents a low-dimensional token that is randomly initialized. Representing modes The corresponding incomplete input.

[0021] S2: Using the text modality as the dominant modality, perform dominant modality correction on the incomplete modal representations obtained in S1, including: performing integrity prediction on the incomplete text modal representation to obtain the integrity prediction value of the text modality, and constructing an integrity check loss based on the integrity prediction value of the text modality; generating surrogate dominant representations based on the incomplete speech modal representations and incomplete visual modal representations, using the surrogate dominant representations and incomplete text modal representations as two types of inputs to the discriminator, determining whether the input representations originate from the text modality, and constructing an adversarial learning loss based on the discrimination result; and using the integrity prediction value of the text modality to adaptively weight and fuse the surrogate dominant representations and incomplete text modal representations to obtain the corrected dominant representation.

[0022] S2 introduces a proxy dominant representation when text modal information is damaged, supplementing and correcting incomplete text modal representations to improve the integrity and reliability of the dominant modal representation.

[0023] S2 includes the following sub-steps: S2-1: Constructing an integrity check encoder This is used to evaluate the information completeness of the incomplete text modal representation obtained by S1. Integrity check encoder. It consists of a feature extraction part and an integrity prediction part. The feature extraction part includes two stacked Transformer encoder layers, which are used to perform contextual semantic modeling on the incomplete text modality representation obtained by S1 to obtain the output features corresponding to the integrity token. The integrity prediction part includes a first fully connected layer, a LeakyReLU activation function, a second fully connected layer and a Sigmoid activation function, which are used to map the output features corresponding to the integrity token to a scalar value in the range [0,1]. This scalar value is the integrity prediction value of the text modality.

[0024] Integrity token initialized randomly Modal representation of incomplete text Encoder for input integrity check after splicing Output the text modality integrity prediction value :

[0025] in, The predicted value represents the integrity of the text modality. Indicates an integrity check encoder. This represents a vector concatenation operation. Indicates a randomly initialized integrity token. This represents an incomplete text modal representation.

[0026] To optimize the integrity prediction process, an integrity check loss is constructed based on the predicted integrity values ​​and then subjected to supervised learning. Specifically, the integrity check loss employs... loss:

[0027] in, This indicates the number of samples in the training set. Indicates the first The completeness prediction value of the text modality for each sample. Indicates the first Integrity labels for the text modality of each sample. express The square of the norm, This indicates a loss during integrity checks.

[0028] S2-2: Randomly initialized representation vectors Incomplete speech modal representation and incomplete visual modality representation After concatenation, the data is input into the agent-dominated representation generator. (Including two layers of Transformer encoders), output agent-dominated representation :

[0029] in, The proxy dominates the generator. The parameters of the generator are indicated by the proxy. This represents a vector concatenation operation. This represents a representation vector that is randomly initialized. This represents an incomplete speech modality representation. This represents an incomplete visual modality representation. This indicates that the agent is in control.

[0030] To ensure the agent's dominant representation It can provide a perspective different from speech and visual modalities, introducing a discriminator. This is used to identify whether the input representation originates from a text modality. Discriminator It includes a binary classifier and a gradient inversion layer. The gradient inversion layer preserves the input features during forward propagation and multiplies the gradient by a negative number (i.e., inverts it) during backward propagation, thus enabling the subsequent binary classifier to generate an adversarial gradient towards the generator during training. The binary classifier, connected after the gradient inversion layer, maps the features processed by the gradient inversion layer to a scalar probability value, which represents the probability that the input representation originates from the text modality. Specifically:

[0031] in, Represents the discriminator. Discriminator The parameters, This represents the discriminator's prediction of whether the input representation originates from a text modality. This indicates that the agent is in control. This represents an incomplete text modal representation.

[0032] Agent-dominated representation generator and discriminator To construct an adversarial learning structure, during training, for each training sample, the discriminator... It accepts two types of input representations: one is an incomplete text modal representation. Another type is proxy-led representation. For the former, its source tag is set to... This indicates that the input originates from a text modality; for the latter, the source label is set to... 0 indicates that the input comes from the generator rather than the text modality. The discriminator aims to identify whether the representation originates from the text modality, while the agent-dominated representation generator aims to challenge the discriminator's ability to make accurate predictions. The adversarial learning loss is:

[0033] in, This indicates the number of samples in the training set. Indicates the first The source label of each input representation, where the input representation originates from a text modality. When the input representation originates from a proxy-dominated representation generator, 0; Indicates the discriminator on the first Each input represents a prediction result derived from a text modality. Represents minimizing loss. Represents maximizing loss. Represents the logarithmic function. This indicates an attempt to counteract learning loss.

[0034] S2-3: Utilizing Integrity Predictions Representative of the agent and incomplete text modal representation Adaptive weighted fusion is performed to obtain the corrected dominant representation. :

[0035] in, This indicates that the correction is dominant. The predicted value represents the integrity of the text modality. This indicates that the agent is in control. This represents an incomplete text modal representation.

[0036] S3: Construct an adaptive supermodal learning unit guided by the dominant modality. The adaptive supermodal learning unit is composed of multiple adaptive supermodal learning layers stacked together. The corrected dominant representation obtained in S2 is encoded layer by layer to obtain corrected dominant representations at different levels. The corrected dominant representations at different levels are used to guide the interaction between the incomplete speech modality representation and the incomplete visual modality representation obtained in S1, and the supermodal representation is updated layer by layer. Subsequently, cross-modal sequence modeling is performed by combining the corrected dominant representation of the final layer and the supermodal representation to obtain the cross-modal fusion representation.

[0037] S3 enhances the discriminativeness and robustness of cross-modal fusion representations by leveraging the semantic guidance of the text-dominant modality and selectively utilizing complementary information from the speech and visual modalities.

[0038] S3 includes the following sub-steps: S3-1: Construct an adaptive supermodal learning unit, which is composed of multiple stacked adaptive supermodal learning layers; in this embodiment, the number of adaptive supermodal learning layers is set to 3. The first adaptive supermodal learning layer includes two parallel multi-head attention modules, and the second and third adaptive supermodal learning layers each include a Transformer encoder and two parallel multi-head attention modules. In the first adaptive supermodal learning layer, the corrected dominant representation obtained in S2 is simultaneously used as the query for both multi-head attention modules. In the second and third adaptive supermodal learning layers, the outputs of the Transformer encoders contained in each layer are simultaneously used as the queries for the two multi-head attention modules in the corresponding adaptive supermodal learning layer.

[0039] The corrected dominant representation obtained from S2 As the primary representation of the first-level correction, for The Transformer encoder is used to update the data layer by layer to obtain the first... Layer correction dominant representation The update process is as follows:

[0040] in, This indicates the layer number of the adaptive supermodal learning layer. Indicates the first Transformer encoder in layer adaptive hypermodal learning layer Indicates the first Layer correction is the dominant representation.

[0041] for , with the first i Layer correction dominant representation As the query, it is represented by the incomplete speech modalities obtained from S1. and incomplete visual modality representation The key and value are used for two attention interactions, and the residual connection is used to update the first... Supermodal representation of the layer:

[0042] in, and They represent the first Layer and first The supermodal representation of the layer, when Time (i.e., the first adaptive supermodal learning layer), , It is a random initialization vector, used as the initial value for the supermode representation, when hour, This represents the supermodal representation output by the previous adaptive supermodal learning layer. This indicates a multi-head attention module; Indicates the first Layer correction is the dominant representation; This represents an incomplete speech modal representation; This represents an incomplete visual modality representation.

[0043] S3-2: Obtaining the corrected dominant representation at layer 3 and supermode representation Then, the dominant representation of the correction at layer 3 is first determined. Normalization is performed, and the corrected dominant representations for the third layer after normalization are respectively. and supermode representation Add learnable source identifier vectors and This is used to distinguish between two representation sequences:

[0044]

[0045] in, Presentation layer normalization operation, This represents a learnable source identifier vector. This represents the supermode representation of the third layer. This represents the corrected dominant representation of layer 3 with the source identifier vector added. This represents the hypermodal representation of the third layer with the source identifier vector added.

[0046] Then and By concatenating the sequences, we obtain the fused input sequence:

[0047] in, This represents the hypermodal representation of layer 3 with the source identifier vector added. This represents the corrected dominant representation of layer 3 with the source identifier vector added. This represents a vector concatenation operation. This represents the fused input sequence for the third layer.

[0048] The fusion input sequence of layer 3 The input is fed into a bidirectional Mamba network. The forward branch directly performs Mamba state space modeling on the fused input sequence of layer 3. The backward branch first reverses the fused input sequence of layer 3 along the sequence dimension, performs Mamba state space modeling, and then reverses the sequence dimension again to output the result.

[0049]

[0050] in, and These represent the output sequence representations for the forward and reverse directions, respectively. This represents the fused input sequence of layer 3. This represents a reverse operation along the sequence dimension. This represents the Mamba state-space modeling module.

[0051] The output sequences of the two branches are then concatenated to obtain the cross-modal enhancement representation. :

[0052] in, and These represent the output sequence representations for the forward and reverse directions, respectively. This represents a vector concatenation operation. This represents cross-modal enhancement representation.

[0053] The correction of the third layer is dominant. With cross-modal enhancement representation By concatenating the vectors, we obtain the joint representation. The joint representation is input into the feedforward network and further enhanced through residual connections to obtain the cross-modal fused representation:

[0054]

[0055] in, Indicates joint expression, This indicates the dominant correction representation of layer 3. This represents cross-modal enhancement representation. Represents a feedforward network. This represents a vector concatenation operation. This represents cross-modal fusion representation.

[0056] S4: Input the cross-modal fusion representation obtained in S3 into the sentiment predictor to obtain the sentiment prediction value, and construct the sentiment prediction loss based on the sentiment prediction value and the corresponding real sentiment value. The specific steps are as follows: Cross-modal fusion representation Input a sentiment predictor composed of multilayer perceptrons to obtain sentiment prediction values:

[0057] in, This represents a sentiment predictor composed of multiple perceptrons; This represents cross-modal fusion representation; This represents the sentiment prediction value.

[0058] Sentiment prediction loss is adopted Loss-based supervised learning:

[0059] in, Indicates the number of samples in the training set; Indicates the first The true sentiment value of each sample; Indicates the first Sentiment prediction values ​​for each sample; Indicates loss in sentiment prediction; express The square of the norm.

[0060] S5: Construct a reconstructor for each modality. Input the incomplete multimodal input obtained in step one into the reconstructor for each modality to obtain the reconstructed representation corresponding to each original complete modality representation. Construct the reconstructed loss based on the difference between the reconstructed representation and the corresponding original complete modality representation.

[0061] Specifically, corresponding reconstructors are constructed for the text modality, speech modality, and visual modality, respectively. (Including two layers of Transformer encoders), for each mode, the mode Corresponding incomplete input The inputs are fed into the corresponding reconstructors to obtain the original complete modal representations. Corresponding reconstruction representation :

[0062] in, Representative mode The corresponding refactorer; Representation of the original complete modality representation The corresponding reconstructed representation; Representing modes The corresponding incomplete input.

[0063] Specifically, the reconstruction loss adopts Differences between loss-constrained reconstructed representations and original complete modal representations:

[0064] in, Indicates the number of samples in the training set; Indicates the first The original complete modal representation of each sample; Indicates the first The reconstructed representation corresponding to each sample is output by the reconstructor; Indicates the reconstruction loss; express The square of the norm, These represent text modality, speech modality, and visual modality, respectively.

[0065] S6: Integrity check loss obtained from S2 Combating learning loss Sentiment prediction loss obtained from S4 The reconstruction loss obtained from S5 By performing a weighted summation, we obtain the total loss function:

[0066] in, Represents the total loss function. Indicates loss due to integrity check. Indicating resistance to learning loss, Indicates the reconstruction loss. Indicates loss in sentiment prediction. These are all hyperparameters used to balance the contributions of each loss term during training.

[0067] Step 3: Use the training set obtained in Step 1 to train the multimodal sentiment analysis model constructed in Step 2, and obtain the trained multimodal sentiment analysis model.

[0068] Specifically, the trainable parameters of each modality embedding encoder, integrity check encoder, surrogate-dominated representation generator, discriminator, adaptive hypermodal learning unit, bidirectional Mamba network, sentiment predictor, and reconstructor are jointly optimized.

[0069] Step 4: Extract features from the multimodal samples to be predicted. Since the multimodal samples to be predicted may have missing information, there is no need to perform random missing information processing as in Step 1. Instead, fill in the missing information in the feature extraction results to obtain the incomplete multimodal input to be predicted. Send the incomplete multimodal input to be predicted to the trained multimodal sentiment analysis model obtained in Step 3 to output the sentiment prediction result.

[0070] Specifically, filling in the missing information in the feature extraction results means: for the speech and visual modalities in the feature extraction results, filling the missing information in the corresponding dimension with zero vectors; for the text modalities, filling the missing information in the BERT vocabulary with the unknown word marker [UNK].

[0071] Specifically, in model testing or robustness evaluation scenarios, the original complete modal representation after feature extraction can be randomly missing according to a preset missing ratio to construct an incomplete multimodal input to be predicted.

[0072] Step four is the inference stage, where only the trained model parameters are used for forward computation, without involving the calculation of the reconstructor, discriminator, and various loss functions, in order to reduce computational complexity and improve inference efficiency. The sentiment prediction result output in step four is the final output of the method of this invention and can be directly used in practical application scenarios.

[0073] Example 2 This embodiment provides an electronic device, which includes: Memory, used to store executable instructions; When a processor executes executable instructions or computer programs stored in the memory, it implements the method of Embodiment 1.

[0074] Example 3 This embodiment provides a computer-readable storage medium storing executable instructions or a computer program, characterized in that the executable instructions, when executed by a processor, implement the method of Embodiment 1.

[0075] The above descriptions are merely various embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0076] To verify the effectiveness of the method of this invention in scenarios with incomplete data, incomplete multimodal inputs were constructed according to the aforementioned random missing data handling method, and experiments were conducted on three publicly available multimodal sentiment analysis datasets: MOSI, MOSEI, and SIMS. The method of this invention was compared with several existing methods. Acc-7, Acc-5, Acc-3, Acc-2, F1, MAE, and Corr were used as evaluation metrics. Higher values ​​for Acc-7, Acc-5, Acc-3, Acc-2, F1, and Corr indicate better performance, while lower values ​​for MAE indicate better performance. For the MOSI and MOSEI datasets, the values ​​before and after the slashes in Acc-2 and F1 represent the results under two binary classification settings: negative / non-negative and negative / positive, respectively. The negative / non-negative setting retains neutral samples with a sentiment value of 0, while the negative / positive setting removes neutral samples with a sentiment value of 0. The experimental results are shown in Tables 1 to 3.

[0077] (a) Experimental results on the MOSI dataset As shown in Table 1, on the MOSI dataset, the method of this invention achieves optimal results in Acc-7, Acc-5, Acc-2, F1, and MAE metrics, and only achieves suboptimal results in Corr. Specifically, the method of this invention improves the Acc-7 and Acc-5 scores by 1.79 and 1.08 percentage points, respectively, compared to the corresponding suboptimal methods. In binary classification evaluation, the method of this invention improves the Acc-2 score by 0.64 and 0.16 percentage points, respectively, and the F1 score by 0.25 and 0.14 percentage points, respectively, compared to TF-Mamba. In terms of regression metrics, the MAE score is further reduced by 0.008 compared to the suboptimal TF-Mamba, and the Corr score is only 0.006 lower than the optimal method. These results indicate that the method of this invention can perform well in multi-class, binary, and regression tasks on the MOSI dataset, especially showing good advantages in overall error control and sentiment category discrimination.

[0078] Table 1. Comparative experimental results on the MOSI dataset.

[0079] (II) Experimental results on the MOSEI dataset As shown in Table 2, on the MOSEI dataset, the method of this invention achieves the best results in Acc-7, Acc-5, Acc-2, MAE, and Corr metrics. Specifically, Acc-7 and Acc-5 improve upon the second-best CENET by 0.15 and 0.48 percentage points, respectively, while Acc-2 improves upon the second-best P-RMF by 0.47 and 0.64 percentage points, respectively. In the regression task, the MAE of the method of this invention is reduced by 0.002 compared to the second-best P-RMF, and the Corr is improved by 0.010. Although the method of this invention does not achieve the best result in the F1 score, it is still higher than most of the comparison methods. The above results indicate that the method of this invention significantly improves the overall classification accuracy, prediction error, and sentiment intensity correlation on the MOSEI dataset.

[0080] Table 2 Comparative experimental results on the MOSEI dataset.

[0081] (III) Comparative Experiment Results on the SIMS Dataset As shown in Table 3, on the SIMS dataset, the method of this invention achieves the best results in all three classification accuracy metrics (Acc-5, Accc-3, and Accc-2), and the second-best results in MAE and Corr. Acc-5 improves by 0.14 percentage points compared to the second-best P-RMF, Acc-3 improves by 2.09 percentage points compared to the second-best LNLN, and Acc-2 improves by 0.09 percentage points compared to the second-best TF-Mamba. In terms of regression metrics, the MAE and Corr of the method of this invention are only slightly lower than P-RMF, with differences of 0.002 and 0.005, respectively. These results demonstrate that the method of this invention is applicable to Chinese multimodal sentiment analysis scenarios and maintains strong overall performance in both multi-classification and regression tasks.

[0082] Table 3 shows the comparative experimental results on the SIMS dataset.

[0083] As can be seen from Tables 1, 2, and 3, the method of this invention achieves optimal results on most multi-class accuracy metrics on the MOSI, MOSEI, and SIMS datasets, and achieves optimal or near-optimal results on regression metrics such as MAE and Corr. This indicates that the method of this invention can balance sentiment category discrimination and sentiment intensity prediction under incomplete data conditions. Experimental results show that the collaborative design of techniques such as dominant modality correction, dominant modality-guided adaptive supermodal learning units, cross-modal sequence modeling based on bidirectional Mamba networks, and reconstruction of missing information by the reconstructor is effective, enabling the model to maintain good overall performance and applicability on different datasets.

Claims

1. A text-guided multimodal sentiment analysis method for scenarios with incomplete data, characterized in that, Includes the following steps: Step 1: Extract features from the original multimodal samples to obtain the original complete modality representation, which includes three modalities: text modality, speech modality, and visual modality; perform random missing operations on the original complete modality representation to obtain incomplete multimodal input, which includes incomplete text modality input, incomplete speech modality input, and incomplete visual modality input; and use the incomplete multimodal input and its corresponding real sentiment values ​​to form a training set. Step 2: Construct a multimodal sentiment analysis model to implement the following process: S1: For each modality of the incomplete multimodal input, feature extraction and dimensionality unification are performed to obtain incomplete modal representations, including incomplete text modal representation, incomplete speech modal representation, and incomplete visual modal representation; S2: Using the text modality as the dominant modality, perform dominant modality correction on the incomplete modal representations obtained in S1, including: performing integrity prediction on the incomplete text modal representation to obtain the integrity prediction value of the text modality, and constructing an integrity check loss based on the integrity prediction value of the text modality; generating a proxy dominant representation based on the incomplete speech modal representation and the incomplete visual modal representation, using the proxy dominant representation and the incomplete text modal representation as the two types of inputs to the discriminator, respectively, and constructing an adversarial learning loss; and using the integrity prediction value of the text modality to perform adaptive weighted fusion on the proxy dominant representation and the incomplete text modal representation to obtain the corrected dominant representation. S3: Construct an adaptive supermodal learning unit guided by the dominant modality. The adaptive supermodal learning unit is composed of multiple adaptive supermodal learning layers stacked together. The corrected dominant representation obtained in S2 is encoded layer by layer to obtain corrected dominant representations at different levels. The corrected dominant representations at different levels are used to guide the interaction between the incomplete speech modality representation and the incomplete visual modality representation obtained in S1, and the supermodal representation is updated layer by layer. Subsequently, cross-modal sequence modeling is performed by combining the corrected dominant representation of the final layer and the supermodal representation to obtain a cross-modal fusion representation. S4: Input the cross-modal fusion representation obtained in S3 into the sentiment predictor to obtain the sentiment prediction value, and construct the sentiment prediction loss based on the sentiment prediction value and the corresponding real sentiment value; S5: Construct a reconstructor for each modality. Input the incomplete multimodal input obtained in step one into the reconstructor for each modality to obtain the reconstructed representation corresponding to each original complete modality representation. Construct the reconstructed loss based on the difference between the reconstructed representation and the corresponding original complete modality representation. S6: The total loss function is obtained by weighted summing of the integrity check loss obtained in S2, the adversarial learning loss obtained in S4, and the reconstruction loss obtained in S5. Step 3: Use the training set obtained in Step 1 to train the multimodal sentiment analysis model constructed in Step 2, and obtain the trained multimodal sentiment analysis model; Step 4: Extract features from the multimodal samples to be predicted, fill in the missing information in the feature extraction results to obtain the incomplete multimodal input to be predicted; send the incomplete multimodal input to be predicted to the trained multimodal sentiment analysis model obtained in Step 3, and output the sentiment prediction result. S2 includes the following sub-steps: S2-1: Constructing an integrity check encoder It consists of a feature extraction part and an integrity prediction part. The feature extraction part includes two stacked Transformer encoder layers. The integrity prediction part includes a first fully connected layer, a LeakyReLU activation function, a second fully connected layer, and a Sigmoid activation function connected in sequence. The output of the integrity check encoder is the integrity prediction value of the text modality. Integrity token initialized randomly R cc Modal representation of incomplete text Encoder for input integrity check after splicing Output the text modality integrity prediction value : in, The predicted value represents the integrity of the text modality. Indicates an integrity check encoder. This represents a vector concatenation operation. This represents a randomly initialized integrity token. Indicates an incomplete text modal representation; Construct the integrity check loss based on the integrity prediction values: in, This indicates the number of samples in the training set. Indicates the first The completeness prediction value of the text modality for each sample. Indicates the first Integrity labels for the text modality of each sample. express The square of the norm, Indicates loss due to integrity check; S2-2: Randomly initialized representation vectors Incomplete speech modal representation and incomplete visual modality representation After concatenation, the data is input into the agent-dominated representation generator. Output agent dominant representation : in, The proxy dominates the generator. The parameters of the generator are indicated by the proxy. This represents a vector concatenation operation. This represents a representation vector that is randomly initialized. This represents an incomplete speech modality representation. This represents an incomplete visual modality representation. This indicates that the agent is in control. Introducing a discriminator The discriminator The input is an incomplete text modal representation. and agent-led expression The output is a prediction of whether the input representation originates from a text modality: in, This represents the discriminator's prediction of whether the input representation originates from a text modality. Represents the discriminator. This indicates that the agent is in control. This represents an incomplete text modal representation. Discriminator Parameters; Construct an adversarial learning loss based on the discriminator's prediction of whether the input representation originates from a text modality: in, This indicates the number of samples in the training set. Indicates the first The source label of each input representation, where the input representation originates from a text modality. When the input representation originates from a proxy-dominated representation generator, 0; Indicates the discriminator on the first Each input represents a prediction result derived from a text modality. Represents minimizing loss. Represents maximizing loss. Represents the logarithmic function. This indicates an attempt to counteract learning loss; S2-3: Utilizing Integrity Predictions Representative of the agent and incomplete text modal representation Adaptive weighted fusion is performed to obtain the corrected dominant representation. : in, This indicates that the correction is dominant. The predicted value represents the integrity of the text modality. This indicates that the agent is in control. Indicates an incomplete text modal representation; S3 includes the following sub-steps: S3-1: Construct an adaptive supermodal learning unit, which is composed of multiple stacked adaptive supermodal learning layers; the number of adaptive supermodal learning layers is set to 3, wherein the first adaptive supermodal learning layer includes two parallel multi-head attention modules, and the second and third adaptive supermodal learning layers each include a Transformer encoder and two parallel multi-head attention modules; in the first adaptive supermodal learning layer, the corrected dominant representation obtained in S2 is used as the query of the two multi-head attention modules simultaneously; in the second and third adaptive supermodal learning layers, the output of the Transformer encoder contained in each layer is used as the query of the two multi-head attention modules in the corresponding adaptive supermodal learning layer; The corrected dominant representation obtained from S2 As the primary representation of the first-level correction, for The Transformer encoder is used to update the data layer by layer to obtain the first... Layer correction dominant representation The update process is as follows: in, This indicates the layer number of the adaptive supermodal learning layer. Indicates the first Transformer encoder in layer adaptive hypermodal learning layer Indicates the first Layer correction is the dominant representation; for , with the first i Layer correction dominant representation As the query, it is represented by the incomplete speech modalities obtained from S1. and incomplete visual modality representation The key and value are used for two attention interactions, and the residual connection is used to update the first... Supermodal representation of the layer: in, and They represent the first Layer and first The supermodal representation of the layer, when hour, , It is a random initialization vector, used as the initial value for the supermode representation, when hour, This represents the supermodal representation output by the previous adaptive supermodal learning layer. This indicates a multi-head attention module; Indicates the first Layer correction is the dominant representation; This represents an incomplete speech modality representation; This represents an incomplete visual modality representation. S3-2: Obtaining the corrected dominant representation at layer 3 and supermode representation Then, the dominant representation of the correction at layer 3 is first determined. Normalization is performed, and the corrected dominant representations for the third layer after normalization are respectively. and supermode representation Add learnable source identifier vectors and This is used to distinguish between two representation sequences: in, Presentation layer normalization operation, This represents a learnable source identifier vector. This represents the supermode representation of the third layer. This represents the corrected dominant representation of layer 3 with the source identifier vector added. This represents the hypermodal representation of layer 3 with the source identifier vector added. Then and By concatenating the sequences, we obtain the fused input sequence: in, This represents the hypermodal representation of layer 3 with the source identifier vector added. This represents the corrected dominant representation of layer 3 with the source identifier vector added. This represents a vector concatenation operation. This represents the fused input sequence of layer 3; The fusion input sequence of layer 3 The input is fed into a bidirectional Mamba network. The forward branch directly performs Mamba state space modeling on the fused input sequence of layer 3. The backward branch first reverses the fused input sequence of layer 3 along the sequence dimension, performs Mamba state space modeling, and then reverses the sequence dimension again to output the result. in, and These represent the output sequence representations for the forward and reverse directions, respectively. This represents the fused input sequence of layer 3. This represents a reverse operation along the sequence dimension. Represents the Mamba state-space modeling module; The output sequences of the two branches are then concatenated to obtain the cross-modal enhancement representation. : in, and These represent the output sequence representations for the forward and reverse directions, respectively. This represents a vector concatenation operation. Indicates cross-modal augmentation representation; The correction of the third layer is dominant. With cross-modal enhancement representation By concatenating the vectors, we obtain the joint representation. The joint representation is input into the feedforward network and passed through residual connections to obtain the cross-modal fused representation: in, Indicates joint expression, This indicates the dominant correction representation of layer 3. This represents cross-modal enhancement representation. Represents a feedforward network. This represents a vector concatenation operation. This represents cross-modal fusion representation.

Images