Multimodal sentiment analysis method based on diffusion model and self-paced learning

Abstract

The invention provides a multi-modal sentiment analysis method based on a diffusion model and self-paced learning. The method comprises the following steps: firstly, dividing a data set into a missing image modal data set and a complete modal data set according to image modal integrity; thirdly, constructing a feature alignment diffusion model, and performing image generation; training the diffusion model by adopting a self-paced learning strategy and a missing image data set; and based on the trained diffusion model, guiding a reverse process through text features to generate feature representation of the missing image. And carrying out weighted fusion on the generated image features and text features by using an attention mechanism, and dynamically adjusting contribution weights of all modalities to generate a complete multi-modal feature representation. And finally, integrating a missing modal completion result and the complete modal features to form a unified multi-modal representation, inputting the unified multi-modal representation into a multi-modal sentiment classification module, and outputting a sentiment classification result. According to the method, the problem of multi-modal sentiment analysis under random missing of image modals is effectively solved, and the generation quality and semantic consistency are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence, and particularly relates to a multi-modal sentiment analysis method based on a diffusion model and self-paced learning. BACKGROUND

[0002] With the rapid development of artificial intelligence and deep learning technology, sentiment analysis, as an important research direction in natural language processing (NLP), has been widely applied. Sentiment analysis aims to automatically identify and classify the sentiment orientation in text, including positive, negative, happy, angry, and other emotional types. This technology is widely used in social media monitoring, online review analysis, brand reputation management, and other fields, and has important commercial value and social impact. Especially in the processing and analysis of large-scale text data, sentiment analysis can help enterprises or governments timely capture the changes in public sentiment, so as to take effective measures.

[0003] Nowadays, the research of sentiment analysis mostly focuses on text data, and traditional analysis methods, including rule-based strategies and single deep learning models, face many challenges in dealing with complex sentiment analysis tasks. These traditional methods mainly rely on text content and fail to fully utilize the potential of sentiment expression containing multiple forms of information such as images, videos, and audio. In fact, multi-modal data can provide more rich and detailed sentiment clues, thus more accurately capturing the emotional state of users. However, most current sentiment analysis techniques are still limited to processing single type of data, limiting the depth and breadth of sentiment understanding. In order to improve the effect of sentiment analysis, researchers need to develop new methods that can integrate and analyze multi-modal information.

[0004] In recent years, with the rapid development of multi-modal machine learning and deep learning technology, the application of multi-modal information in sentiment analysis has gradually attracted the attention of researchers, especially in the integration of images and text. Diffusion models, as an important progress in generative models, have shown excellent performance in image generation and restoration tasks, and can effectively reconstruct the missing areas in images based on known patterns, thus providing a new technical path for information supplement of image modal in sentiment analysis. However, in practical application scenarios, multi-modal data generally have the problem of modal missing, especially when image information is unavailable or of limited quality, how to construct a robust sentiment analysis model to cope with the challenges brought by modal missing is still a key scientific problem that needs to be solved. SUMMARY

[0005] To address this issue, this invention proposes a multimodal sentiment analysis method based on a diffusion model and auto-step learning, which can effectively perform sentiment analysis in scenarios where random image modalities are missing. Guided by the diffusion model, missing image information can be repaired. Simultaneously, by combining a cross-modal attention mechanism and an auto-step learning training strategy, image and text information are fused, further improving the accuracy and robustness of sentiment analysis. This method not only represents a breakthrough in the accuracy of sentiment analysis but also provides a new approach and solution for the field of multimodal sentiment analysis.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A multimodal sentiment analysis method based on diffusion models and autosynchronous learning includes the following steps:

[0008] Step S1: Obtain a multimodal sentiment analysis dataset containing both text and image data. Randomly discard image samples according to a predefined missing proportion to obtain an image modality randomly missing multimodal dataset.

[0009] Step S2: Construct a feature alignment diffusion model. Its forward process adds Gaussian noise to the original image data of the random missing multimodal dataset of image modalities step by step, transforming it from the original image data distribution to noise data with a near-standard Gaussian distribution. Its reverse process integrates a noise suppression module and a feature alignment module to achieve accurate noise removal and enhance the cross-modal semantic alignment capability of the generated images during the image generation process.

[0010] Step S3: Based on the image modality random missing multimodal dataset from Step S1, a three-stage self-learning strategy is designed to train the feature alignment diffusion model described in Step S2. The three stages are as follows: the primary stage focuses on removing high-intensity noise; the intermediate stage reduces noise intensity and optimizes cross-modal feature alignment between images and text; the advanced stage further reduces noise and comprehensively optimizes denoising, alignment, and perceptual quality to ensure high quality and semantic consistency of the generated images.

[0011] Step S4: Based on the feature alignment diffusion model trained in step S3, and guided by the text features corresponding to the missing image, generate the feature representation of the missing image through a back diffusion process.

[0012] Step S5: Based on the attention mechanism, the image features generated in step S4 and the corresponding text features are adaptively weighted and fused. By calculating cross-modal correlation, the contribution weights of each modality are dynamically adjusted to generate a semantically rich and complete multimodal feature representation.

[0013] Step S6: For data with complete image modalities, extract features using a text encoder and an image encoder respectively, and then concatenate the two to obtain a fused multimodal feature representation.

[0014] Step S7: Integrate the multimodal features generated in Step S5 and Step S6 to form a complete multimodal representation, input it into the multimodal sentiment classification module, and output the sentiment classification result.

[0015] The predefined missing ratio dr∈{0.2, 0.4, 0.6, 0.8} in step S1 is used to simulate different degrees of image modality missing in real-world situations. Preprocessing includes the following operations: (1) segmenting, removing stop words, and vectorizing the text modality data; (2) normalizing and resizing the image modality data; and (3) aligning the preprocessed text and image data according to sentiment tags.

[0016] The forward process of the diffusion model in step 2 is defined as follows:

[0017]

[0018] In the above formula (1), This indicates that it follows a Gaussian distribution. It is the variance of the normal distribution. For time step Noise estimation, The noise addition intensity at each time step is controlled by preset noise scheduling parameters. These parameters determine the noise addition pattern and distribution characteristics at each time step. By employing cosine scheduling or linear scheduling strategies, the continuity, controllability, and mathematical reversibility of the noise addition process are ensured. The value range is strictly limited to the interval [0.0001, 0.02] to ensure that the noise intensity increases smoothly and predictably throughout the diffusion process, avoiding problems such as image information loss, model training instability, or degraded generation quality caused by noise jumps or discontinuous changes. The reverse process is modeled as a highly parameterized conditional Gaussian distribution:

[0019]

[0020] In the above formula (2), For time step Noise estimation, These are optimizable components for neural network model parameters, encompassing the weight matrix, bias vector, batch normalization parameters, query-key-value parameters for the attention mechanism, and residual connection parameters of multi-layer convolutional neural networks. and The mean and variance of the predictions are used to model and describe the uncertainty of the prediction results and the statistical characteristics of the noise distribution. The training objective is then to minimize the noise prediction error.

[0021]

[0022] In the above formula (3), Gaussian noise was added to the image to represent reality. The noise predicted by the model. Noise prediction loss function. By calculating the true noise vector randomly sampled from the standard Gaussian distribution and combining it with the neural network model based on the input noise image and corresponding time step, The L2 norm squared error between the predicted noise vectors is used to comprehensively measure the noise prediction accuracy and reconstruction capability of the model. By continuously optimizing this loss function on large-scale datasets, the diffusion model can gradually learn accurate and robust noise recognition and removal capabilities, thereby achieving high-fidelity image generation, reconstruction and semantic preservation.

[0023] The noise suppression module in step S2 employs a stepwise denoising approach to recover the semantic information and structure of the image. Noise in the image modality is added progressively and removed gradually through training. During this process, text features provide guidance in the denoising process to ensure that the generated image remains consistent with the text information. Specifically, text features... The pre-trained text encoder performs deep semantic encoding, transforming discrete sequences of text symbols into continuous, high-dimensional, dense text embedding vectors. :

[0024]

[0025] In the above formula (4), It is the embedding vector at the time step, in the text encoder. Employing a BERT language model based on a multi-layer Transformer architecture, this approach captures long-range dependencies, semantic hierarchy, and sentiment information in text through multi-head self-attention, positional encoding, residual connections, and layer normalization. The encoding process begins with text preprocessing, including word segmentation, tokenization, and special symbol addition. Then, a lexical embedding layer maps words to initial vector representations. Next, a multi-layer encoder progressively extracts and abstracts semantic features, resulting in a 768-dimensional text embedding vector that comprehensively expresses the deep semantic information, sentiment, and contextual features of the text. Following this, image feature extraction is performed. A specially designed U-Net network encoder performs multi-scale, multi-level feature extraction and downsampling on input images containing varying levels of noise. Finally, image features are extracted and transformed into feature vectors. :

[0026]

[0027] In formula (5) above, the image encoder adopts a symmetrical deep convolutional neural network architecture, which contains multiple fully functional encoding blocks. Each encoding block consists of convolutional layers, batch normalization layers, ReLU activation functions, and max pooling or stride convolution. By gradually reducing the spatial resolution of the feature maps while increasing the feature channel dimensions, it systematically extracts complete multi-scale image representations from low-level edge, texture, and color features to high-level shape, object, and semantic features. These hierarchical feature representations can comprehensively capture the visual information, structural patterns, and semantic content in the image. After extracting the features of the text and image, the next step is noise prediction. This step utilizes a specially constructed multimodal noise prediction network to comprehensively leverage multi-scale visual features extracted from the image, deeply encoded textual semantic features, and the positional encoding information t of the current diffusion time step. This allows for accurate prediction and localization of the noise component distribution and intensity pattern within the current image features. The noise prediction network employs an advanced multimodal fusion architecture. First, it achieves deep semantic interaction and information alignment between image and textual features through a cross-attention mechanism. Then, it converts the time step information into a temporal embedding vector using sinusoidal positional encoding and fuses it with the multimodal features. Finally, it outputs a noise prediction result that perfectly matches the input image size through a multilayer perceptron, convolutional neural network, and upsampling operations, thus achieving denoising.

[0028]

[0029] In the above formula (6), It is a time step Predicted image features, This is a noise prediction network. The noise distribution is modeled as a Gaussian distribution, specifically expressed as:

[0030]

[0031] In the above formula (7), The mean and standard deviation of the noise are used for prediction by the network. To further improve the quality and semantic consistency of the generated images, a cross-modal contrastive loss is added to optimize the alignment between the image and text. This loss function maximizes the similarity between image and text features while minimizing the similarity between mismatched image and text features, as shown in the formula:

[0032]

[0033] In the above formula (8), and Image and text features extracted for expert models. The marginal hyperparameters of the contrast loss are defined. Finally, the final loss function of the noise suppression module considers a weighted combination of denoising error and cross-modal contrast loss to ensure a balance between denoising performance and semantic consistency in the generated image. The final loss function is as follows:

[0034]

[0035] In the above formula (9), For the loss of noise removal, For cross-modal contrast loss, The weighted hyperparameters are used to balance the denoising and semantic alignment losses. Through adaptive noise modeling and cross-modal alignment mechanisms, the noise suppression module can not only effectively remove noise but also maintain the high quality and semantic consistency of the generated images, thereby improving the performance of multimodal sentiment analysis.

[0036] In step S2, the feature alignment module follows the noise suppression module and aims to further optimize the alignment of image and text features, improving the quality and semantic consistency of the generated image. During the reverse process of the diffusion model, the feature alignment module optimizes the noise-suppressed image features through a self-attention mechanism. (Image features) and text features The interaction is achieved through a self-attention mechanism, which calculates the correlation between the features to generate aligned image features. The formula is as follows:

[0037]

[0038] In the above formula (10), It is the image feature after noise suppression; It is a text feature. , and It is a learnable linear transformation matrix; , and It is a bias term; This refers to the feature dimension. First, the image features generated by the noise suppression module and the corresponding text features are used as input to interact with the self-attention mechanism. The self-attention mechanism dynamically adjusts the image features by calculating the similarity between them, better aligning them with the text features in the latent space. Specifically, the image features and text features are mapped through linear transformations to obtain new queries, keys, and values. The inner product of the image features and text features is then calculated using... The weights are normalized using a function to determine the degree of influence of each text feature on the image features. These weights are then applied to the text features. Generate aligned image features Furthermore, to ensure semantic consistency between image and text features, the feature alignment module introduces an image-text consistency loss. This loss ensures alignment between image and text features in the latent space by minimizing the projection error of image and text features in the low-dimensional space. The loss function is defined as follows:

[0039]

[0040] In the above formula (11), and These are low-dimensional representations of image and text features after linear projection, respectively. By minimizing the projection error, the model can effectively align the semantic information of the image and text. The final feature alignment module loss function incorporates a noise removal loss. Image-text consistency loss The body is as follows:

[0041]

[0042] In the above formula (12), It is the noise removal loss. It is image-text consistency loss. This is a hyperparameter that adjusts the importance of both. Through this design, the feature alignment module ensures cross-modal alignment between images and text, not only removing noise from the image but also maintaining a high degree of semantic consistency between the image and text in the latent space. This module significantly improves the quality and semantic consistency of the generated images, thereby enhancing the performance of multimodal sentiment analysis.

[0043] In step S3, a three-stage self-learning strategy is used to train the feature alignment diffusion model. In each stage, the model progressively optimizes noise removal, image-text feature alignment, and the perceptual quality of the generated image to ensure that the final generated image is both high-quality and semantically consistent with the text description. The initial training stage focuses on removing high-intensity noise and restoring the basic structure of the image. In this stage, the noise intensity is set to a high value, and the model primarily learns how to recover clear image contours from noisy images. The noise suppression module generates denoised image features... This removes noise from the image. Loss function The loss function is used to measure the effectiveness of noise removal. Its goal is to remove noise from an image and restore its basic information.

[0044]

[0045] In the above formula (13), Original image features The image features are defined after noise removal. The initial stage aims to remove high-intensity noise and provide a clear image foundation for subsequent feature alignment and optimization. In the intermediate stage, as noise intensity decreases, the model aims to optimize the feature alignment between the image and text. To enhance the image-text feature alignment, an adaptive feature alignment loss is designed. This loss not only relies on a global contrastive loss but also adaptively adjusts the similarity metric between the image and text, thus more accurately capturing the semantic relationships between them. A weighted self-attention alignment loss is introduced, combining the self-attention mechanism with feature alignment. This weighted alignment of image and text features enhances the model's learning of feature alignment. The formula is as follows:

[0046]

[0047] In the above formula (14), It is a weighted interaction of image and text features calculated through a self-attention mechanism. It is an adaptive weight that dynamically adjusts the impact of alignment loss based on key regions of image features. This is a low-dimensional projection representation of text features. This stage adaptively adjusts the alignment weights between the image and text, making the alignment process more refined while avoiding redundant alignment. Finally, the training focus and core objective of the advanced stage is to establish deep semantic associations and correspondences between image visual features and text semantic features. This enables the model to not only have efficient noise removal capabilities but also accurately understand and maintain the consistency and matching between the semantic content of the image and the corresponding text description. During training, the weight ratio of contrastive loss is gradually increased through a progressive strategy. The advanced stage loss function is learned automatically. Designed as:

[0048]

[0049] In the above formula (15), These are the weight parameters of the loss function in the intermediate stage. The ultimate training goal of this stage is to achieve high accuracy in semantic alignment, high perceptual quality of generated images, and comprehensive optimization of overall model performance, while ensuring basic noise removal and cross-modal alignment capabilities. Through refined multi-objective joint optimization and balanced learning, the model can generate high-quality image representations that are both semantically accurate and visually realistic.

[0050] In step S5, an attention mechanism is used to adaptively weight and fuse text features and generated image features. Advanced self-attention and cross-attention mechanisms are implemented to accurately calculate the semantic relevance and importance weight distribution between features of different modalities. The calculation of attention weights is implemented using the standard Transformer attention formula, specifically expressed as follows:

[0051]

[0052] In the above formula (16), It is the fused multimodal feature representation; , , These represent the query matrix, key matrix, and value matrix, respectively. The dimension of the key vector is used as a scaling factor. The function ensures the probability normalization of the weight distribution. Through this bidirectional attention mechanism, the model can adaptively identify and emphasize important correlations and complementary features between different modalities. The resulting complete multimodal feature representation has stronger expressive power, better generalization performance, and higher feature quality.

[0053] In step S6, for data with complete image modalities, features are extracted using both a text encoder and an image encoder, and then the two are concatenated. The text encoder uses BERT to extract features, while the image encoder uses ViT. Concatenation occurs after feature extraction.

[0054] The performance evaluation metrics in step S7 include: (1) Precision: (2) Recall: (3) F1 value: ; where P represents precision, R represents recall, F1 represents F1 value, TP represents the number of samples that are actually positive and predicted as positive by the model; FN represents the number of samples that are predicted as negative by the model but are actually positive; FP represents the number of samples that are predicted as positive by the model but are actually negative; TN represents the number of samples that are actually negative and predicted as negative by the model. The higher the scores of precision, recall and F1 value, the better the performance of the sentiment analysis model. (4) FID: evaluates the distribution similarity between the generated image and the real image. The lower the FID value, the closer the distribution of the generated image and the real image in the feature space is, that is, the higher the quality of the generated image. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating a multimodal sentiment analysis method based on diffusion models and autosynchronous learning.

[0056] Figure 2 This is the network structure diagram of the diffusion model. Detailed Implementation

[0057] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and examples.

[0058] like Figure 1 As shown, this invention presents a multimodal sentiment analysis method based on a diffusion model and autosynchronous learning, the process of which is as follows:

[0059] Step 1: Acquire multimodal sentiment analysis corpus containing both text and image modalities. Randomly discard image samples according to a predefined missing proportion. To evaluate the model's robustness to missing image modalities, this invention randomly discards image samples. The missing model's handling strategy uses a discard ratio of {0.2, 0.4, 0.6, 0.8} to simulate the scale of modal missingness. Comprehensive preprocessing is performed on the acquired multimodal data. This includes preprocessing of text and image modalities, multimodal tagging and alignment, and final data format conversion. The specific steps are as follows:

[0060] (1) The preprocessing of text data first requires word segmentation, breaking the text down into individual words or subwords. The text involved in this invention is English text, segmented by spaces or punctuation marks. Next, stop words are removed. Stop words do not contribute substantially to sentiment analysis or information extraction, and therefore need to be removed from the text. Removing stop words helps reduce noise and improve model efficiency. Subsequently, vectorization methods are used to convert the text into a numerical form that machine learning can understand. This invention uses pre-trained BERT for word embedding. Through the above steps, each text is converted into a vector for subsequent model use.

[0061] (2) The first step in preprocessing the image data is normalization, which aims to scale the range of image pixel values ​​to a standard range (e.g., between 0 and 1) to reduce the impact of differences in brightness and contrast between different images. This helps to speed up the training process and improve the stability of the model. Next, resizing is performed, scaling the width and height of all images to a fixed size of 224x224 pixels, because different images have different sizes and resolutions, while the model requires input images to have a uniform size. Finally, feature extraction is performed, that is, extracting key information from the images that is helpful for task completion. This invention accelerates the training process and improves performance by utilizing a pre-trained ViT model to extract features.

[0062] (3) Align the preprocessed text and image data according to sentiment tags (positive, neutral, negative).

[0063] Step 2: Construct a feature-aligned diffusion model that includes a noise suppression module and a feature alignment module. The model structure is as follows: Figure 2 As shown, the self-noise suppression module's main task is to reduce early noise during the generation process. It dynamically predicts and suppresses noise, rapidly adjusting the image content in the initial stages of generation to effectively reduce noise. This module combines cross-modal features (such as text and image information) for noise prediction, ensuring the denoising process adapts adaptively to the currently generated content. The feature alignment module is primarily responsible for reducing mid-stage noise during generation, especially as the generation process nears completion and detailed information begins to emerge. The feature alignment module introduces an expert model to align features across multiple generation tasks, ensuring that each generation step achieves the most suitable noise removal within the multimodal space.

[0064] In the diffusion model framework, image generation is typically a progressive denoising process. Starting with random noise, the model guides image generation based on textual descriptions through multiple iterations, gradually approximating the target image. However, noise and image information generated during the generation process are not always perfectly removed. Especially in multimodal sentiment analysis tasks, due to the influence of noise during generation, the generated image data often contradicts the semantic features of the text or sentiment itself, potentially deviating from the textual description and resulting in poor image quality that fails to accurately represent the textual content. Therefore, an adaptive noise suppression module is introduced into the backward propagation process of the diffusion model. This module dynamically adjusts noise estimation during the backward generation process to achieve cross-modal aligned denoising. For the noise suppression module, the goal at each time step t is to estimate the noise in the current image features and subtract the noise to obtain clearer image features. The specific formula is as follows:

[0065]

[0066]

[0067] In the above formulas (1) and (2), It is a time step Predicted image features, This represents a noise suppression network. These are the features of the input image. It is text feature embedding. It is a time step. and Here, represents the mean and standard deviation of the noise. The noise suppression module progressively removes noise from the input image, restoring its original semantic content. During noise removal, the model estimates noise based on the image and text features at the current time step and removes the estimated noise from the image. To further improve the quality and semantic consistency of the generated image, a cross-modal contrastive loss is added to optimize the alignment between the image and text. This loss function maximizes the similarity between image and text features while minimizing the similarity between mismatched image and text features, as shown in the formula:

[0068]

[0069] In the above formula (3), and Image and text features extracted for expert models. The marginal hyperparameters of the contrast loss are defined. Finally, the final loss function of the noise suppression module considers a weighted combination of denoising error and cross-modal contrast loss to ensure a balance between denoising performance and semantic consistency in the generated image. The final loss function is as follows:

[0070]

[0071] In the above formula (4), For the loss of noise removal, For cross-modal contrast loss, The weighted hyperparameters are used to balance the denoising and semantic alignment losses. Through adaptive noise modeling and cross-modal alignment mechanisms, the noise suppression module can not only effectively remove noise but also maintain the high quality and semantic consistency of the generated images, thereby improving the performance of multimodal sentiment analysis.

[0072] In the process of cross-modal data generation and understanding, simply relying on feature alignment of text-image pairs may still result in semantic gaps or biases. Therefore, this invention introduces a feature alignment module after the noise suppression module to further optimize the alignment of image and text features, improving the quality and semantic consistency of the generated images. During the reverse process of the diffusion model, the feature alignment module optimizes the noise-suppressed image features through a self-attention mechanism. Image features and text features The interaction is achieved through a self-attention mechanism, which calculates the correlation between the features to generate aligned image features. The formula is as follows:

[0073]

[0074] In the above formula (5), It is based on the image features after noise suppression. It is a text feature. , and These are linear transformation matrices corresponding to image and text features, respectively. , and It is a bias term. This refers to the feature dimension. First, the image features generated by the noise suppression module and the corresponding text features are used as input to interact with the self-attention mechanism. The self-attention mechanism dynamically adjusts the image features by calculating the similarity between them, better aligning them with the text features in the latent space. Specifically, the image features and text features are mapped through linear transformations to obtain new queries, keys, and values. The inner product of the image features and text features is then calculated using... The weights are normalized using a function to determine the degree of influence of each text feature on the image features. These weights are then applied to the text features. Generate aligned image features Furthermore, to ensure semantic consistency between image and text features, the feature alignment module introduces an image-text consistency loss. This loss ensures alignment between image and text features in the latent space by minimizing the projection error of image and text features in the low-dimensional space. The loss function is defined as follows:

[0075]

[0076] In the above formula (6), and These are low-dimensional representations of image and text features after linear projection, respectively. By minimizing the projection error, the model can effectively align the semantic information of the image and text. The final feature alignment module loss function incorporates a noise removal loss. Image-text consistency loss The body is as follows:

[0077]

[0078] In the above formula (7) It is the noise removal loss. It is image-text consistency loss. This is a hyperparameter that adjusts the importance of both. Through this design, the feature alignment module ensures cross-modal alignment between images and text, not only removing noise from the image but also maintaining a high degree of semantic consistency between the image and text in the latent space. This module significantly improves the quality and semantic consistency of the generated images, thereby enhancing the performance of multimodal sentiment analysis.

[0079] Step 3: Train the feature diffusion model using a self-stepping learning strategy, enabling the model to dynamically generate missing image modality data based on text modality information. In multimodal generation tasks, due to the complexity of cross-modal data and the randomness of noise introduction, direct model training may face optimization difficulties, such as fluctuations in generation quality and instability in cross-modal alignment. To train the model more efficiently and improve generation performance, a self-stepping learning strategy is introduced, gradually unfolding the training process from simple to complex. At each stage, the model progressively optimizes noise removal, image-text feature alignment, and the perceptual quality of the generated image, ensuring that the final generated image is both high-quality and semantically consistent with the text description. The initial training stage focuses on removing high-intensity noise and restoring the basic structure of the image. In this stage, the noise intensity is set to a high value, and the model mainly learns how to recover clear image contours from noisy images. The noise suppression module generates denoised image features. This removes noise from the image. Loss function The loss function is used to measure the effectiveness of noise removal. Its goal is to remove noise from an image and restore its basic information.

[0080]

[0081] In the above formula (8), Original image features The image features are defined after noise removal. The initial stage aims to remove high-intensity noise and provide a clear image foundation for subsequent feature alignment and optimization. In the intermediate stage, as noise intensity decreases, the model aims to optimize the feature alignment between the image and text. To enhance the image-text feature alignment, an adaptive feature alignment loss is designed. This loss not only relies on a global contrastive loss but also adaptively adjusts the similarity metric between the image and text, thus more accurately capturing the semantic relationships between them. A weighted self-attention alignment loss is introduced, combining the self-attention mechanism with feature alignment. This weighted alignment of image and text features enhances the model's learning of feature alignment. The formula is as follows:

[0082]

[0083] In the above formula (9), It is a weighted interaction of image and text features calculated through a self-attention mechanism; It is an adaptive weight that dynamically adjusts the impact of alignment loss based on key regions of image features. This is a low-dimensional projection representation of text features. This stage adaptively adjusts the alignment weights between the image and text, making the alignment process more refined while avoiding redundant alignment. Finally, the training focus and core objective of the advanced stage is to establish deep semantic associations and correspondences between image visual features and text semantic features. This enables the model to not only have efficient noise removal capabilities but also accurately understand and maintain the consistency and matching between the semantic content of the image and the corresponding text description. During training, a progressive strategy is used to gradually increase the weight ratio of the contrastive loss, and the advanced stage loss function is learned automatically. Designed as:

[0084]

[0085] In the above formula (10), These are the weight parameters of the loss function in the intermediate stage. The ultimate training goal of this stage is to achieve high accuracy in semantic alignment, high perceptual quality of generated images, and comprehensive optimization of overall model performance, while ensuring basic noise removal and cross-modal alignment capabilities. Through refined multi-objective joint optimization and balanced learning, the model can generate high-quality image representations that are both semantically accurate and visually realistic.

[0086] Step 4: Multimodal fusion is performed between the image modal data generated by the alignment diffusion model and the existing text modal data to obtain a complete multimodal feature representation. This complete multimodal feature representation is then concatenated with the remaining multimodal features. The fusion employs an attention mechanism, weighting the text features and the generated image features. The specific formula is as follows:

[0087]

[0088] In the above formula (11), These are the fused multimodal features; This represents the text features after encoding. The model generates image features based on text semantics; The non-missing multimodal features are composed of concatenated text and image features. The text features are encoded using BERT, and the image features are encoded using ViT. and It is a learnable weight matrix. Image features and text features are fused through an attention mechanism. Finally, it is fed into a classifier for classification; the specific formula is:

[0089]

[0090] In the above formula (12), It is a weight matrix responsible for applying linear transformations to the multimodal features. This is a bias term. The function is used to convert the result of the linear transformation into probabilities, representing the predicted value for each sentiment category. Finally, by maximizing... The output probabilities yield the final sentiment category prediction result:

[0091]

[0092] In the above formula (13), It is the final category (negative, neutral, positive) in the sentiment analysis task, representing the sentiment label predicted by the model.

[0093] Step 5: Perform sentiment analysis based on the obtained multimodal feature representations and output the sentiment classification results. Compare and evaluate the results with previous methods using precision, recall, and F1 score to achieve robust multimodal sentiment analysis even when random image modalities are missing.

[0094] Table 1 shows the confusion matrix required for the calculation:

[0095] Table 1 Confusion Matrix

[0096] Predicted positive Predicted negative Actual positive TP FN Predicted negative FP TN

[0097] The formula for calculating accuracy is: The formula for calculating recall rate is: (3) The formula for calculating the F1 value is: Where P represents precision, R represents recall, F1 represents F1 score, TP represents the number of samples that were actually positive and were predicted as positive by the model; FN represents the number of samples that were predicted as negative by the model but were actually positive; FP represents the number of samples that were predicted as positive by the model but were actually negative; and TN represents the number of samples that were actually negative and were predicted as negative by the model. Higher scores for precision, recall, and F1 score indicate better performance of the sentiment analysis model.

[0098] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should be included in the scope of the present invention.

Claims

1. A multimodal sentiment analysis method based on diffusion model and autosynchronous learning, characterized in that, Includes the following steps: Step S1: Obtain a multimodal sentiment analysis dataset containing both text and image data. Randomly discard image samples according to a predefined missing proportion to obtain an image modality randomly missing multimodal dataset. Step S2: Construct a feature alignment diffusion model. Its forward process adds Gaussian noise to the original image data of the random missing multimodal dataset of image modalities step by step, transforming it from the original image data distribution to noise data with a near-standard Gaussian distribution. Its reverse process integrates a noise suppression module and a feature alignment module to achieve accurate noise removal and enhance the cross-modal semantic alignment capability of the generated images during the image generation process. Step S3: Based on the image modality random missing multimodal dataset from Step S1, a three-stage self-learning strategy is designed to train the feature alignment diffusion model described in Step S2. The three stages are as follows: the primary stage focuses on removing high-intensity noise; the intermediate stage reduces noise intensity and optimizes cross-modal feature alignment between images and text; the advanced stage further reduces noise and comprehensively optimizes denoising, alignment, and perceptual quality to ensure high quality and semantic consistency of the generated images. Step S4: Based on the feature alignment diffusion model trained in step S3, and guided by the text features corresponding to the missing image, generate the feature representation of the missing image through a back diffusion process. Step S5: Based on the attention mechanism, the image features generated in step S4 and the corresponding text features are adaptively weighted and fused. By calculating cross-modal correlation, the contribution weights of each modality are dynamically adjusted to generate a semantically rich and complete multimodal feature representation. Step S6: For data with complete image modalities, extract features using a text encoder and an image encoder respectively, and then concatenate the two to obtain a fused multimodal feature representation.

2. Step S7: Integrate the multimodal features generated in steps S5 and S6 to form a complete multimodal representation, input it into the multimodal sentiment classification module, and output the sentiment classification result.

3. The multimodal sentiment analysis method based on diffusion model and autosynchronous learning according to claim 1, characterized in that, The noise suppression module described in step S2 is designed to recover the semantic information and structure of the image through a stepwise denoising approach. Noise in the image modality is gradually added and then gradually removed through training: input image features and text features The noise suppression module processes the data to obtain the denoised image features at each time step. The formula is as follows: ； In the above formula (1), This represents the noise removal function. These are the features of the input image. It is text feature embedding. It is a time step; The noise suppression module gradually removes noise from the input image, restoring the original semantic content of the image; During noise removal, the model estimates noise based on the image and text features at the current time step and removes the estimated noise from the image. The noise estimation function dynamically adjusts the noise prediction result by combining text and image features, and the formula is as follows: ； In the above formula (2), It is a time step Predicted image features, It is a noise prediction network; the noise distribution is modeled as a Gaussian distribution, specifically represented as: ； In the above formula (3), and The mean and standard deviation of the noise are used for prediction via a noise prediction network. A cross-modal contrastive loss function is incorporated to optimize the alignment between the image and text. This loss function maximizes the similarity between image and text features while minimizing the similarity between mismatched image and text features. The formula is as follows: ； In the above formula (4), and Image and text features extracted for expert models. To compare the marginal hyperparameters of the loss, the final loss function of the noise suppression module considers a weighted combination of denoising error and cross-modal contrastive loss, ensuring a balance between denoising performance and semantic consistency of the generated image; the final loss function is as follows: ； In the above formula (5), Loss due to noise removal For cross-modal contrast loss, This is a weighted hyperparameter used to balance the loss between denoising and semantic alignment.

4. The multimodal sentiment analysis method based on diffusion model and autosynchronous learning according to claim 1, characterized in that, The feature alignment module described in step S2 follows immediately after the noise suppression module. During the reverse process of the diffusion model, the feature alignment module optimizes the noise-suppressed image features through a self-attention mechanism; image features and text features The interaction is achieved through a self-attention mechanism, which calculates the correlation between the features to generate aligned image features. The formula is as follows: ； In the above formula (6), It is based on the image features after noise suppression. It is a text feature. , and It is a learnable linear transformation matrix. , and It is a bias term. It is the feature dimension; First, the image features and corresponding text features generated by the noise suppression module are used as input to interact with the self-attention mechanism. The self-attention mechanism dynamically adjusts the image features by calculating the similarity between them, better aligning them with the text features in the latent space. The image features and text features are then mapped using linear transformations to obtain new queries, keys, and values. Finally, the inner product of the image features and text features is calculated using... The weights are normalized using a function to obtain the degree of influence of each text feature on the image feature. Then, these weights are applied to the text features. Generate aligned image features Furthermore, to ensure semantic consistency between image and text features, the feature alignment module introduces an image-text consistency loss. This loss ensures alignment in the latent space by minimizing the projection error between image and text features in the low-dimensional space. The loss function is defined as follows: ； In the above formula (7), and These are low-dimensional representations of image and text features after linear projection. By minimizing the projection error, the model can effectively align the semantic information of the image and text. The final feature alignment module loss function incorporates noise removal loss. Image-text consistency loss : ； In the above formula (8), It is the noise removal loss. It is image-text consistency loss. It is a hyperparameter that adjusts the importance of both.

5. The multimodal sentiment analysis method based on diffusion model and autosynchronous learning according to claim 1, characterized in that, The three-stage self-learning strategy described in step S3 trains the feature alignment diffusion model: the initial training stage focuses on removing high-intensity noise and restoring the basic structure of the image; in this stage, the noise intensity is set to a high value, and the model mainly learns how to recover clear image contours from noisy images; the noise suppression module generates denoised image features. This removes noise from the image; loss function The loss function is used to measure the effectiveness of noise removal. Its goal is to remove noise from an image and restore its basic information. ； In the above formula (9), Original image features Image features after noise removal; The goal of the initial stage is to remove high-intensity noise and provide a clear image foundation for subsequent feature alignment and optimization; in the intermediate stage, the noise intensity is reduced and the model aims to optimize the feature alignment between the image and the text. By introducing a weighted self-attention alignment loss, the self-attention mechanism is combined with feature alignment. This enhances the model's learning of feature alignment by weighting the alignment of image and text features; the formula is as follows: ； In the above formula (10), It is a weighted interaction of image and text features calculated through a self-attention mechanism. It is an adaptive weight that dynamically adjusts the impact of alignment loss based on key regions of image features. It is a low-dimensional projection representation of text features; this stage makes the alignment process more refined by adaptively adjusting the alignment weights between the image and the text, while avoiding redundant alignment between the image and the text; finally, the training focus and core goal of the advanced stage is to establish a deep semantic association and correspondence between the visual features of the image and the semantic features of the text, so that the model not only has efficient noise removal capabilities, but also can accurately understand and maintain the consistency and matching between the semantic content of the image and the corresponding text description. During the training process, the weight ratio of contrast loss is gradually increased through a progressive strategy. Advanced Self-Learning Stage The loss function is designed as follows: ； In the above formula (11), The weight parameters are those of the loss function in the intermediate stage. The ultimate training goal of this stage is to achieve high accuracy in semantic alignment, high perceptual quality of generated images, and comprehensive optimization of overall model performance while ensuring basic noise removal and cross-modal alignment capabilities. Through refined multi-objective joint optimization and balanced learning, the model can generate high-quality image representations that are both semantically accurate and visually realistic.

6. The multimodal sentiment analysis method based on diffusion model and autosynchronous learning according to claim 1, characterized in that, Step S5 describes the use of an attention mechanism to adaptively weight and fuse text features and generated image features. Advanced self-attention and cross-attention mechanisms are implemented to accurately calculate the semantic relevance and importance weight distribution among different modal features. The calculation of attention weights is achieved using the standard Transformer attention formula, specifically expressed as follows: ； In the above formula (12), It is a fused multimodal feature representation; For text features, For the generated image features; , , These represent the query matrix, key matrix, and value matrix, respectively. The dimension of the key vector, used as a scaling factor; The function ensures the probability normalization of the weight distribution. Through this bidirectional attention mechanism, the model can adaptively identify and emphasize important correlation information and complementary features between different modalities. The resulting complete multimodal feature representation has stronger expressive power, better generalization performance, and higher feature quality.

Images