Multi-modal sentiment analysis method, system, device and medium based on diffusion model

Abstract

The application discloses a multi-modal sentiment analysis method, system, device and medium based on a diffusion model, belongs to the technical field of artificial intelligence and multi-modal information processing, and aims to solve the technical problem of how to overcome the situation of missing modes or incomplete data in the multi-modal sentiment analysis process in the prior art, realize high-quality recovery of a dominant mode and effective fusion of cross-modal information, improve sentiment recognition accuracy and robustness of a model, and the technical scheme is as follows: constructing a multi-modal sentiment analysis training data set; extracting multi-modal data features; constructing a multi-modal sentiment analysis model, specifically including: missing perception; mode reliability evaluation; reliability-guided diffusion dominant mode repair; graph neural network enhanced multi-modal fusion; missing data reconstruction; and sentiment prediction.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and multimodal information processing technology, specifically a multimodal sentiment analysis method, system, device, and medium based on a diffusion model. Background Technology

[0002] Existing multimodal sentiment analysis methods typically assume that data for each modality is complete and available during both training and inference phases. However, in real-world applications, factors such as speech noise, video occlusion, sensor limitations, or privacy concerns often lead to missing modalities or incomplete data, resulting in a significant decline in model performance. Furthermore, existing technologies generally suffer from poor modality recovery quality, insufficient semantic consistency, and weak generalization ability when dealing with missing modalities, making it difficult to meet the needs of practical applications.

[0003] With the continuous growth of internet multimedia content and the expanding application scope of intelligent interactive systems, the problem of missing multimodal data in complex open environments is becoming increasingly prominent. Especially in real-world business scenarios, random modal missingness often exhibits significant uncertainty. In recent years, research on the modal missingness problem has mainly focused on two types of methods: one is feature reconstruction-based methods, which use generative models or mapping mechanisms to complete missing modalities; the other is joint representation learning-based methods, which learn shared representations to enhance the model's adaptability to missing data. However, these methods typically have the following shortcomings: firstly, most methods only model fixed missing patterns, making it difficult to adapt to the complex situation of random missing data in reality; secondly, existing methods often reconstruct features on a single sample basis, ignoring the structured semantic relationships between modalities, leading to deviations between reconstructed features and true semantics; furthermore, while existing research has shown that language modalities play a dominant role in sentiment analysis, existing methods have failed to effectively guarantee the integrity and quality of language modalities, thus limiting the overall performance improvement.

[0004] Therefore, how to overcome the situation of modality missing or data incompleteness in the multimodal sentiment analysis process in the existing technology, achieve high-quality recovery of the dominant modality and effective fusion of cross-modal information, and improve the accuracy of sentiment recognition and the robustness of the model are the technical problems that urgently need to be solved. Summary of the Invention

[0005] The technical objective of this invention is to provide a multimodal sentiment analysis method, system, device, and medium based on a diffusion model to address the issues of modality loss or incomplete data in existing multimodal sentiment analysis processes, achieve high-quality recovery of the dominant modality and effective fusion of cross-modal information, and improve the accuracy of sentiment recognition and the robustness of the model.

[0006] The technical objective of this invention is achieved as follows: a multimodal sentiment analysis method based on a diffusion model, the specific method of which is as follows:

[0007] Constructing a multimodal sentiment analysis training dataset: Collect multimodal data including language data, audio data, and video data, and perform word segmentation and encoding on the language data, feature standardization and time alignment on the audio data, and frame alignment and key feature extraction on the video data to obtain the corresponding language sequences, audio signals, and video frame information, and attach corresponding sentiment tags to construct a multimodal sentiment analysis training dataset.

[0008] Extracting multimodal data features: Feature extraction is performed on language sequences, audio signals and video frame information respectively to obtain corresponding language features, audio features and video features. Then, random missing features are constructed on the language features, audio features and video features, with the discard ratio ranging from 0 to 100% to obtain the corresponding missing language features, missing audio features and missing video features.

[0009] A multimodal sentiment analysis model is constructed, specifically as follows: Missing Features Perception: Modeling the missing features of corresponding modalities based on missing language features, missing audio features, and missing video features, generating corresponding language missing masks, audio missing masks, and video missing masks; Modal Reliability Assessment: Evaluating the effectiveness of missing audio features, missing video features, audio missing masks, and video missing masks, generating weighted audio features and weighted video features; Reliability-Guided Diffusion Dominant Modal Repair: Gradually repairing missing language features through forward, reverse, and mask-preserving recovery processes to obtain repaired language features and ensure the integrity of the dominant modality's information; Graph Neural Network Enhanced Multimodal Fusion: Enhancing the repaired language features, missing audio features, and missing video features through node initialization, graph construction, and graph neural networks, and interactively modeling multimodal information to achieve interaction and fusion of multimodal features, obtaining enhanced language features, enhanced audio features, and enhanced video features; Missing Data Reconstruction: Using Transformer... The layer reconstructs multimodal data including missing language features, missing audio features, and missing video features, obtains language features, reconstructs audio features, and reconstructs video features, and further calculates the missing data reconstruction loss for subsequent optimization; sentiment prediction: the enhanced language features, enhanced audio features, and enhanced video features are pooled and then concatenated, and then the sentiment prediction value is obtained through a classifier, and the sentiment prediction loss is further obtained.

[0010] As a preferred method, the specific features extracted from multimodal data are as follows:

[0011] Language feature extraction: Language features are extracted by performing contextual semantic encoding on language sequences using a pre-trained BERT-base-uncased model. ;in, Indicates the length of the language sequence; Represents the dimensions of language features;

[0012] Audio feature extraction: The audio signal is extracted using the Librosa tool to obtain audio features. ;in, Indicates the length of the audio frame; Indicates the dimension of audio features;

[0013] Video feature extraction: Facial expression analysis was performed on video frame information using the OpenFace tool to extract video features. ;in, Indicates the length of the video frame. Indicates the dimension of video features;

[0014] Constructing multimodal missing features: Missing language features are represented using masking, missing audio features using zero-padding, and missing video features using zero-padding to obtain missing language features. Missing audio features and missing video features .

[0015] More specifically, the missing information perception is based on missing language features. Missing audio features and missing video features Generate language missing masks respectively Audio missing mask and video missing mask The expression is as follows:

[0016] ;

[0017] in, This represents the characteristics of mode M at time step t; This represents the missing mask value at the corresponding position. A mask value of 1 indicates that the corresponding position is missing; a mask value of 0 indicates that the corresponding position is complete.

[0018] Modal reliability assessment specifically involves: calculating modal reliability weights based on the degree of missing auxiliary modes and feature quality; specifically, it involves applying audio missing masks. The missing percentage is calculated to obtain the audio modality integrity score, using the following formula: ;in, Indicates the length of the audio sequence; This indicates the missing state of the audio modality at time step t; This represents the integrity score of the audio modality; simultaneously, it assesses the missing audio features through feature quality evaluation. To perform a quality assessment, obtain the quality score for the missing audio features. The formula is: ;in, Represents the audio features at time step t; Represents the L2 norm; This represents the audio feature quality score; then the audio feature quality score is... and audio modality integrity score To perform reliability fusion calculations and obtain the audio modal reliability weights, the formula is as follows: ;in, , and These are learnable parameters; Represents the Sigmoid function; This represents the audio modality reliability weight; the audio features are weighted according to the audio modality reliability weight to obtain the weighted audio features, as shown in the formula: ; Calculate the missing proportion of the video missing mask to obtain the video modality integrity score, using the following formula: ;in, Indicates the length of the video sequence; This indicates the missing state of the video modality at time step t; This represents the integrity score of the video modality; simultaneously, feature quality assessment is performed on missing video features to obtain the missing video feature quality score, using the following formula: ;in, This represents the video feature at time step t; Represents the L2 norm; The video feature quality score is represented by the video feature quality score. Then, the missing video feature quality score and the video modality integrity score are reliably fused to obtain the video modality reliability weight, as shown in the formula: ;in, , and These are learnable parameters; σ(⋅) represents the Sigmoid function; This represents the video modality reliability weight; the video features are weighted according to the video modality reliability weight to obtain the weighted video features, as shown in the formula: .

[0019] More specifically, reliability-guided diffusion-dominant mode repair is as follows:

[0020] Forward diffusion process: defining the original complete linguistic features as The forward diffusion process using the diffusion model is carried out iteratively. Gaussian noise is added to generate a series of samples with gradually increasing noise. (The diffusion process of language features evolving from a clear state to a noisy state), the positive diffusion process is formally defined as: ;in, , Indicates preset variance scheduling. ; Let it be the identity matrix; set any time step of Direct sampling: ;in, Structured missing features are introduced into the language modality, and Gaussian noise is added to obtain the missing language features. , will lack language features Treat it as a noisy linguistic feature and treat the missing linguistic feature as... Used for noise reduction and recovery in the reverse diffusion process;

[0021] Diffusion Reverse Process: From Noisy Language Features China Resumption The forward diffusion process is reversed; specifically, the missing language features are used as the starting point for recovery, and weighted audio features are processed through a multi-head cross-attention mechanism. Weighted video features and language missing mask The input is processed by a denoising network to progressively denoise and recover lost or damaged semantic information in the language modality. After multiple iterative denoising steps, diffused recovery of language features is gradually obtained. The formula is as follows: ;

[0022] Mask Preservation Recovery: Recovering Linguistic Features Based on Diffusion Missing language features and language missing mask Only missing regions are updated, while the original linguistic features of non-missing regions remain unchanged. The formula is: ;in, This represents element-wise multiplication. This indicates the language features after restoration.

[0023] More specifically, graph neural networks enhance multimodal fusion as follows:

[0024] Node initialization: Initialize a node containing A graph of nodes, where each node corresponds to an input mode. One lexical unit; for time step At that time, the mode is The nodes, initial features Directly set it as the corresponding word feature vector, the formula is: , is used to use the feature vectors of each mode at the corresponding time step as the initial representation of the corresponding node in the graph; where, Representing nodes in a graph neural network Initial features at layer 0; This represents the node corresponding to mode M at time step t; These represent the language modality, video modality, and audio modality, respectively. The characteristic matrix representing mode M The eigenvector corresponding to the t-th time step; when hour, Indicates the restored language features ;when hour, Indicates missing video features ;when hour, Indicates missing audio features ;

[0025] Graph Construction: First, establish intramodal and intermodal connections based on language modal nodes, audio modal nodes, and video modal nodes to form a cross-modal graph structure composed of intramodal edges and cross-modal edges; then, calculate the edges E and their weights in the cross-modal graph structure using an attention mechanism. Where u represents a node and v represents an adjacent node; for modal inner edges, the weight formula for language modality-language modality (LL) edges is: ;in, and Represents the learnable projection matrix; The dimensions represent the key and the query; the weight formula for video modality-video modality (VV) edges is: ;in, and The projection matrix represents the learnable video modality; the weight formula for the audio modality-audio modality (AA) edge is: ;in, and This represents the learnable projection matrix corresponding to the audio modality; for cross-modal edges, language nodes are computed using language features as queries. and video nodes Between and and audio nodes The edge weights between them are as follows:

[0026] ;

[0027] ;

[0028] in, and The learnable projection matrix is ​​represented; language-guided edge weights enable the graph structure to dynamically enhance relevant cross-modal connections while pruning irrelevant ones; the edge weight formula between video and audio modalities (V–A) is as follows:

[0029] ;

[0030] in, This represents the learnable query projection matrix corresponding to the video modality; This represents the learnable key projection matrix corresponding to the audio modality;

[0031] Update the graph neural network: Use a three-layer graph attention network (GAT) for feature propagation and refinement; specifically: for the first... Layered GNN, each node Receive from its neighboring nodes The message, and calculate the attention coefficient. Then update the corresponding features using the following formula: ;in, Indicates a shared linear transformation matrix; This represents the activation function; after passing through a three-layer graph neural network (GNN), the updated node features are obtained, and the updated node features are reorganized into sequence representations for each modality: ; ; ;in, This indicates the enhanced linguistic features; Indicates the enhanced audio features; This indicates the enhanced video features.

[0032] More specifically, the missing data reconstruction involves: reconstructing the missing information of each modality, including missing language features, missing audio features, and missing video features, through two Transformer layers, and learning from the missing modalities contaminated by noise. Input restored to original clean modal features The reconstruction formula is: ;in, Indicates modality The reconstructor; This indicates the original clean feature. Reconstruction characteristics;

[0033] The formula for sentiment prediction is: ;in, This indicates average pooling or max pooling operations; This indicates the sentiment prediction result.

[0034] Preferably, the method also includes training optimization of the multimodal sentiment analysis model, as follows:

[0035] Constructing the loss function: The multimodal sentiment analysis model is jointly optimized by diffusion model loss, missing data reconstruction loss, and sentiment prediction loss. Specifically, the diffusion loss function is first introduced. The diffusion loss function is used to constrain the multimodal sentiment analysis model's ability to predict noise during backdiffusion, improve the accuracy of language modality recovery, and ensure the effective recovery of missing language features. The formula is: ;in, Represents real noise; This represents the noise predicted by the multimodal sentiment analysis model; Indicates the first step in the diffusion process The noisy language features of the steps; These represent missing audio features and missing video features, respectively. The diffusion time step is represented; secondly, a reconstruction loss function is introduced. The reconstruction loss function measures the difference between the reconstructed features and the original features, constraining the multimodal sentiment analysis model and improving the completeness and consistency of feature representations across different modalities. The formula is: ;in, Indicates the number of samples in the training set; Representing modes The The original input features of each sample; Indicates the first The modal features are reconstructed from each sample; the loss function minimizes the difference between the original features and the reconstructed features, thereby improving the performance of other components; finally, a sentiment prediction loss is constructed. The sentiment prediction loss is used to measure the difference between the prediction results of a multimodal sentiment analysis model and the true labels, thereby improving the prediction performance of sentiment analysis. The formula is: ;in, Indicates the number of training samples; Indicates the first The true sentiment labels of each sample; This represents the predicted value from the multimodal sentiment analysis model;

[0036] Model optimization: The diffusion model loss, missing data reconstruction loss, and sentiment prediction loss are weighted and summed to obtain the total loss function. Then, the gradient descent method is used to optimize and train the multimodal sentiment analysis model parameters. The formula is as follows: ;in, The hyperparameters representing the loss are used to balance the influence of different loss terms. During the training of the multimodal sentiment analysis model, the model parameters are continuously updated through the backpropagation algorithm, so that the total loss function gradually converges. When the multimodal sentiment analysis model is not yet fully trained, iterative optimization is performed on the multimodal sentiment analysis training dataset. After the multimodal sentiment analysis model is trained, it can perform sentiment classification or sentiment intensity prediction on the input multimodal data.

[0037] A diffusion-based multimodal sentiment analysis system is provided, which implements the diffusion-based multimodal sentiment analysis system method described above; the system includes:

[0038] The multimodal sentiment analysis training dataset construction unit is used to collect multimodal data including language data, audio data and video data, and to perform word segmentation and encoding processing on language data, feature standardization and time alignment processing on audio data, and frame alignment and key feature extraction processing on video data to obtain corresponding language sequences, audio signals and video frame information, and attach corresponding sentiment tags to construct a multimodal sentiment analysis training dataset.

[0039] The multimodal data feature extraction unit is used to extract features from language sequences, audio signals and video frame information respectively, to obtain corresponding language features, audio features and video features, and to perform random missing construction processing on language features, audio features and video features, with the discard ratio ranging from 0 to 100%, to obtain corresponding missing language features, missing audio features and missing video features.

[0040] A multimodal sentiment analysis model building unit is used to construct a multimodal sentiment analysis model; the multimodal sentiment analysis model building unit includes:

[0041] The missing feature awareness module is used to model the missing status of the corresponding modalities based on missing language features, missing audio features, and missing video features, and generate corresponding language missing feature masks, audio missing feature masks, and video missing feature masks.

[0042] The modal reliability assessment module is used to evaluate the effectiveness of missing audio features, missing video features, audio missing masks, and video missing masks, and to generate weighted audio features and weighted video features.

[0043] The reliability-guided diffusion dominant modality repair module is used to gradually repair missing language features through forward process, reverse process and mask preservation recovery, obtain the repaired language features, and ensure the information integrity of the dominant modality.

[0044] The graph neural network-enhanced multimodal fusion module is used to enhance the repaired language features, missing audio features, and missing video features through node initialization, graph construction, and graph neural networks. It also performs interactive modeling of multimodal information to achieve the interaction and fusion of multimodal features and obtain enhanced language features, enhanced audio features, and enhanced video features.

[0045] The missing data reconstruction module is used to reconstruct multimodal data, including missing language features, missing audio features, and missing video features, through the Transformer layer. It obtains language features, reconstructs audio features, and reconstructs video features, and further calculates the missing data reconstruction loss for subsequent optimization.

[0046] The sentiment prediction module performs pooling operations on the enhanced language features, enhanced audio features, and enhanced video features, then concatenates them, obtains the sentiment prediction value through a classifier, and further obtains the sentiment prediction loss.

[0047] The multimodal sentiment analysis model training unit is used to train the multimodal sentiment analysis model based on the multimodal sentiment analysis dataset. By constructing sentiment prediction loss function, diffusion loss function and reconstruction loss function, the parameters of the multimodal sentiment analysis model are optimized, so that the multimodal sentiment analysis model has stable sentiment analysis capabilities.

[0048] An electronic device includes: a memory and at least one processor;

[0049] The memory contains computer programs;

[0050] The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the multimodal sentiment analysis system method based on the diffusion model as described above.

[0051] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the diffusion-based multimodal sentiment analysis system method described above.

[0052] The multimodal sentiment analysis method, system, device, and medium based on the diffusion model of the present invention have the following advantages:

[0053] (i) This invention introduces a diffusion model to perform high-quality generative recovery of the dominant language modality and combines graph neural networks to perform structured modeling and fusion of multimodal information, which effectively improves the accuracy and robustness of emotion recognition in modality-deficient scenarios. Compared with existing methods, this invention can adapt to random modality-deficient situations, solves the problem that traditional methods are only applicable to fixed-deficient patterns, and significantly improves the overall semantic expression ability by strengthening the dominant role of language modality, thereby overcoming the defects of existing technologies that ignore cross-modal structural relationships and the insufficient quality of the dominant modality.

[0054] (II) This invention enables reliability-guided diffusion-dominant modality repair, and can perform high-quality generative reconstruction of missing or damaged language modalities, thereby improving the processing capability of multimodal sentiment analysis systems in modality-deficient scenarios, enhancing the model's adaptability and analysis efficiency to incomplete input data, and solving the problems of low reconstruction quality and insufficient semantic consistency in the missing modality recovery process in existing technologies. It achieves accurate recovery and enhancement of semantic information of language modalities. Based on this, it enhances multimodal fusion by graph neural networks, introduces language-guided cross-modal graph structures, and explicitly models cross-modal relationships, that is, models and propagates the structural relationships between different modalities, thereby achieving efficient collaboration of multimodal information and enhanced emotional expression capabilities. It also reconstructs the missing modal features of the input through missing data reconstruction, assists in the reconstruction of multimodal features, further improves the stability and reliability of model prediction, and improves the consistency of multimodal feature representation by constraining the differences between reconstructed features and original features. It also provides a new modeling idea and technical path for cross-modal interaction in multimodal sentiment analysis.

[0055] (iii) This invention can effectively alleviate the performance degradation problem caused by modality loss or incomplete data in multimodal emotion analysis tasks, and can still maintain high emotion recognition accuracy and stability in complex real-world environments;

[0056] (iv) The present invention can effectively perform multimodal information collaborative modeling through the processing flow of "first recovering the dominant mode and then performing multimodal fusion", thereby improving the robustness and generalization ability of the model in complex missing scenarios.

[0057] (v) This invention leverages the synergistic effect of reliability-guided diffusion-led modal repair and graph neural network-enhanced multimodal fusion to enable multimodal sentiment analysis models to possess stronger cross-modal semantic modeling and structured representation capabilities, thereby significantly improving the overall performance of multimodal sentiment analysis models and demonstrating superior results in multiple datasets and under different missing conditions. Attached Figure Description

[0058] The invention will be further described below with reference to the accompanying drawings.

[0059] Appendix Figure 1 A flowchart for building a multimodal sentiment analysis model;

[0060] Appendix Figure 2 A flowchart for reliability-guided diffusion-dominant mode repair;

[0061] Appendix Figure 3 This is a flowchart illustrating the process of enhancing multimodal fusion using graph neural networks. Detailed Implementation

[0062] The following detailed description of the diffusion-based multimodal sentiment analysis method, system, device, and medium of the present invention is provided with reference to the accompanying drawings and specific embodiments.

[0063] Example 1:

[0064] This embodiment provides a multimodal sentiment analysis method based on a diffusion model, as detailed below:

[0065] S1. Constructing a multimodal sentiment analysis training dataset: Collect multimodal data including language data, audio data, and video data. Perform word segmentation and encoding on the language data, feature standardization and time alignment on the audio data, and frame alignment and key feature extraction on the video data to obtain the corresponding language sequences, audio signals, and video frame information, and attach corresponding sentiment tags to construct a multimodal sentiment analysis training dataset. The sentiment tags can be discrete sentiment categories or continuous sentiment intensity values.

[0066] S2. Extract multimodal data features: Extract features from language sequences, audio signals and video frame information respectively to obtain corresponding language features, audio features and video features. Then, randomly construct missing language features, audio features and video features by randomly discarding them, with the discard ratio ranging from 0 to 100%, to obtain corresponding missing language features, missing audio features and missing video features.

[0067] S3. Construct a multimodal sentiment analysis model;

[0068] S4. Train and optimize the multimodal sentiment analysis model.

[0069] The specific steps for extracting multimodal data features in step S2 of this embodiment are as follows:

[0070] S201. Extracting Language Features: Language features are extracted by performing contextual semantic encoding on the language sequence using a pre-trained BERT-base-uncased model. ;in, Indicates the length of the language sequence; Represents the dimensions of language features;

[0071] S202. Extracting Audio Features: Using the Librosa tool, extract features from the audio signal to obtain audio features. ;in, Indicates the length of the audio frame; Indicates the dimension of audio features;

[0072] S203. Extracting Video Features: Using the OpenFace tool, perform facial expression analysis on video frame information to extract video features. ;in, Indicates the length of the video frame. Indicates the dimension of video features;

[0073] S204. Constructing multimodal missing features: Missing language features are represented using masking, missing audio features using zero-padding, and missing video features using zero-padding to obtain missing language features. Missing audio features and missing video features .

[0074] As attached Figure 1 As shown, the construction of the multimodal sentiment analysis model in step S3 of this embodiment is as follows:

[0075] S301, Missing Features Detection: Modeling the missing features of corresponding modalities based on missing language features, missing audio features, and missing video features, generating corresponding language missing masks, audio missing masks, and video missing masks; specifically: based on missing language features... Missing audio features and missing video features Generate language missing masks respectively Audio missing mask and video missing mask The expression is as follows:

[0076] ;

[0077] in, This represents the characteristics of mode M at time step t; This represents the missing mask value at the corresponding position. A mask value of 1 indicates that the corresponding position is missing; a mask value of 0 indicates that the corresponding position is complete.

[0078] S302. Modal Reliability Assessment: The effectiveness of missing audio features, missing video features, audio missing masks, and video missing masks is assessed to generate weighted audio features and weighted video features. Specifically, modal reliability weights are calculated based on the degree of missing auxiliary modalities and feature quality.

[0079] S30201, Audio Missing Mask The missing proportion is calculated to obtain the audio modality integrity score, using the following formula:

[0080] ;

[0081] in, Indicates the length of the audio sequence; This indicates the missing state of the audio modality at time step t; Indicates the integrity score of the audio modality;

[0082] S30202, Simultaneously, missing audio features are evaluated through feature quality assessment. To perform a quality assessment, obtain the quality score for the missing audio features, using the following formula:

[0083] ;

[0084] in, Represents the audio features at time step t; Represents the L2 norm; This indicates the audio feature quality score;

[0085] S30203, Audio Feature Quality Score and audio modality integrity score To perform reliability fusion calculations and obtain audio modality reliability weights, the formula is as follows:

[0086] ;

[0087] in, , and These are learnable parameters; Represents the Sigmoid function; Indicates the audio modality reliability weight;

[0088] S30204. Weight the audio features according to the audio modality reliability weights to obtain the weighted audio features. The formula is as follows: ;

[0089] S30205. Calculate the missing proportion of the video missing mask to obtain the video modality integrity score, using the following formula:

[0090] ;

[0091] in, Indicates the length of the video sequence; This indicates the missing state of the video modality at time step t; Indicates the integrity score of the video modality;

[0092] S30206. Simultaneously, perform feature quality assessment on the missing video features to obtain the missing video feature quality score, using the following formula:

[0093] ;

[0094] in, This represents the video feature at time step t; Represents the L2 norm; Indicates the video feature quality score;

[0095] S30207. Perform reliability fusion calculation on the missing video feature quality score and the video modality integrity score to obtain the video modality reliability weight, as shown in the following formula:

[0096] ;

[0097] in, , and These are learnable parameters; σ(⋅) represents the Sigmoid function; Indicates the video modality reliability weight;

[0098] S30208. Weight the video features according to the video modality reliability weight to obtain the weighted video features. The formula is as follows: ;

[0099] S303, Reliability-Guided Diffusion Dominant Mode Repair: See attached Figure 2 As shown, missing linguistic features are progressively repaired through forward processing, reverse processing, and mask-preserving recovery to obtain the repaired linguistic features and ensure the integrity of the dominant modality's information; specifically as follows:

[0100] S30301, Forward Diffusion Process: Defining the original complete linguistic features as... The forward diffusion process using the diffusion model is carried out iteratively. Gaussian noise is added to generate a series of samples with gradually increasing noise. (The diffusion process of language features evolving from a clear state to a noisy state), the positive diffusion process is formally defined as: ;in, , Indicates preset variance scheduling. ; Let it be the identity matrix; set any time step of Direct sampling: ;in, Introduce structured missing features (e.g., unknown lexical units) into the language modality and add Gaussian noise to obtain the missing language features. , will lack language features Treat it as a noisy linguistic feature and treat the missing linguistic feature as... Used for noise reduction and recovery in the reverse diffusion process;

[0101] S30302, Diffusion Reverse Process: From Noisy Language Features China Resumption The forward diffusion process is reversed; specifically, the missing language features are used as the starting point for recovery, and weighted audio features are processed through a multi-head cross-attention mechanism. Weighted video features and language missing mask The input is processed through a denoising network to progressively denoise and recover lost or damaged semantic information from the language modality. This information guides the language feature recovery process, allowing the model to prioritize the use of more reliable auxiliary modality information and focus on recovering missing regions within the language modality. After multiple iterative denoising steps, the language features are progressively recovered through diffusion, as shown in the formula: ;

[0102] The backtransfer is modeled as a Gaussian distribution and processed by a denoising network. The mean and variance of the reverse diffusion process are parameterized in the denoising network. The prediction of injecting noise at time step t is as follows:

[0103] ;

[0104] ;

[0105] The calculation process for the mean and variance of the diffusion reversal process is as follows: First, the denoising direction predicted by the network is separated, and then the mean is obtained by removing the corresponding predicted noise components from the current state; the formula is as follows:

[0106] ;

[0107] ;

[0108] in, This represents the noise predicted by the denoising network;

[0109] S30303, Mask Preservation Recovery: Recovering Linguistic Features Based on Diffusion Missing language features and language missing mask Only missing regions are updated, while the original linguistic features of non-missing regions remain unchanged. The formula is as follows:

[0110] ;

[0111] in, This represents element-wise multiplication; Indicates the restored language features;

[0112] S304, Graph Neural Network Enhanced Multimodal Fusion: As attached Figure 3 As shown, the system enhances the repaired language features, missing audio features, and missing video features through node initialization, graph construction, and graph neural networks. It also performs interactive modeling of multimodal information to achieve the interaction and fusion of multimodal features, obtaining enhanced language features, enhanced audio features, and enhanced video features. Specifically:

[0113] S30401, Node Initialization: Initialize a node containing... A graph of nodes, where each node corresponds to an input mode. One lexical unit; for time step At that time, the mode is The nodes, initial features Directly set it as the corresponding word feature vector, the formula is: , is used to use the feature vectors of each mode at the corresponding time step as the initial representation of the corresponding node in the graph; where, Representing nodes in a graph neural network Initial features at layer 0; This represents the node corresponding to mode M at time step t; These represent the language modality, video modality, and audio modality, respectively. The characteristic matrix representing mode M The eigenvector corresponding to the t-th time step; when hour, Indicates the restored language features ;when hour, Indicates missing video features ;when hour, Indicates missing audio features ;

[0114] S30402. Graph Construction: First, establish intramodal and intermodal connections based on language modal nodes, audio modal nodes, and video modal nodes to form a cross-modal graph structure composed of intramodal edges and cross-modal edges; then, calculate the edges E and their weights in the cross-modal graph structure using an attention mechanism. Where u represents a node and v represents an adjacent node; for modal inner edges, the weight formula for language modality-language modality (LL) edges is as follows:

[0115] ;

[0116] in, and Represents the learnable projection matrix; Represents the key and the dimension of the query;

[0117] The weight formula for video modality-video modality (VV) edges is: ;in, and The projection matrix representing the learnable video modalities; The dimensions represent the key and the query; the weight formula for the audio modality-audio modality (AA) edge is: ;in, and This represents the learnable projection matrix corresponding to the audio modality; Represents the key and the dimension of the query;

[0118] For cross-modal edges, language nodes are computed using language features as queries. and video nodes Between and and audio nodes The edge weights between them are as follows:

[0119] ;

[0120] ;

[0121] in, and The projection matrix is ​​a learnable representation; language-guided edge weights enable the graph structure to dynamically enhance relevant cross-modal connections while pruning irrelevant ones.

[0122] The formula for the edge weights between the video and audio modalities (V–A) is as follows:

[0123] ;

[0124] in, This represents the learnable query projection matrix corresponding to the video modality; This represents the learnable key projection matrix corresponding to the audio modality;

[0125] 30403. Update the graph neural network: Use a three-layer graph attention network (GAT) for feature propagation and refinement; specifically: for the first... Layered GNN, each node Receive from its neighboring nodes The message, and calculate the attention coefficient. Then update the corresponding features, using the following formula:

[0126] ;

[0127] in, Indicates a shared linear transformation matrix; This represents the activation function; after passing through a three-layer graph neural network (GNN), the updated node features are obtained, and the updated node features are reorganized into sequence representations for each modality:

[0128] ;

[0129] ;

[0130] ;

[0131] in, This indicates the enhanced linguistic features; Indicates the enhanced audio features; This represents the enhanced video features;

[0132] S305, Missing Data Reconstruction: Multimodal data including missing language features, missing audio features, and missing video features are reconstructed using Transformer layers. This process retrieves language features, reconstructs audio features, and reconstructs video features, and further calculates the missing data reconstruction loss for subsequent optimization. Specifically, two Transformer layers are used to reconstruct the missing information for each modality, including missing language features, missing audio features, and missing video features, learning from noise-contaminated missing modalities. Input restored to original clean modal features The reconstruction formula is as follows:

[0133] ;

[0134] in, Indicates modality The reconstructor; This indicates the original clean feature. Reconstruction characteristics;

[0135] S306. Sentiment Prediction: The enhanced language features, enhanced audio features, and enhanced video features are pooled separately, then concatenated, and a classifier is used to obtain the sentiment prediction value. The sentiment prediction loss is then calculated. The formula is as follows:

[0136] ;

[0137] in, This indicates average pooling or max pooling operations; This indicates the sentiment prediction result.

[0138] The specific training and optimization of the multimodal sentiment analysis model in step S4 of this embodiment is as follows:

[0139] S401. Constructing the loss function: The multimodal sentiment analysis model is jointly optimized using diffusion model loss, missing data reconstruction loss, and sentiment prediction loss, as detailed below:

[0140] First, introduce the diffusion loss function. The diffusion loss function is used to constrain the multimodal sentiment analysis model's ability to predict noise during backdiffusion, improve the accuracy of language modality recovery, and ensure the effective recovery of missing language features. The formula is as follows:

[0141] ;

[0142] in, Represents real noise; This represents the noise predicted by the multimodal sentiment analysis model; Indicates the first step in the diffusion process The noisy language features of the steps; These represent missing audio features and missing video features, respectively. Indicates the diffusion time step;

[0143] Secondly, a reconstruction loss function is introduced. The reconstruction loss function measures the difference between the reconstructed features and the original features, constraining the multimodal sentiment analysis model and improving the completeness and consistency of feature representations across different modalities. The formula is as follows:

[0144] ;

[0145] in, Indicates the number of samples in the training set; Representing modes The The original input features of each sample; Indicates the first The modal features reconstructed from each sample; the loss function minimizes the difference between the original features and the reconstructed features, thereby improving the performance of other components;

[0146] Finally, construct the sentiment prediction loss. The sentiment prediction loss is used to measure the difference between the prediction results of a multimodal sentiment analysis model and the true labels, thereby improving the prediction performance of sentiment analysis. The formula is as follows:

[0147] ;

[0148] in, Indicates the number of training samples; Indicates the first The true sentiment labels of each sample; This represents the predicted value from the multimodal sentiment analysis model;

[0149] S402. Model Optimization: The diffusion model loss, missing data reconstruction loss, and sentiment prediction loss are weighted and summed to obtain the total loss function. The gradient descent method is then used to optimize and train the parameters of the multimodal sentiment analysis model, as shown in the following formula:

[0150] ;

[0151] in, The hyperparameters representing the loss are used to balance the influence of different loss terms. During the training of the multimodal sentiment analysis model, the model parameters are continuously updated through the backpropagation algorithm, so that the total loss function gradually converges. When the multimodal sentiment analysis model is not yet fully trained, iterative optimization is performed on the multimodal sentiment analysis training dataset. After the multimodal sentiment analysis model is trained, it can perform sentiment classification or sentiment intensity prediction on the input multimodal data.

[0152] The multimodal sentiment analysis model in this embodiment achieved better results than other methods on the MOSI dataset. The comparison of experimental results is shown in Table 1.

[0153] Table 1 Comparison of experimental results on the MOSI dataset

[0154]

[0155] As shown in Table 1, the experimental results on the MOSI dataset demonstrate that the proposed multimodal sentiment analysis method achieves good results across multiple evaluation metrics, exhibiting stability in accuracy, F1 score, and relevance metrics. This result indicates that the proposed method effectively completes sentiment classification and sentiment polarity discrimination tasks, and demonstrates good performance in fine-grained sentiment recognition. Furthermore, in terms of relevance and error metrics, the proposed method maintains high consistency and low prediction error, indicating a good match between the model output and the true labels, demonstrating good stability. The reasons for these technical effects are as follows: the proposed method introduces reliability-guided diffusion to lead modality repair, restoring missing language modalities and ensuring the integrity of semantic information; simultaneously, through missing data perception and modality reliability assessment, the model can adaptively adjust the utilization of auxiliary information according to the degree of missing modalities; furthermore, by enhancing multimodal fusion through graph neural networks, the proposed method performs structured modeling of multimodal features, achieving effective interaction and fusion of cross-modal information. Therefore, the proposed method can maintain relatively stable sentiment recognition capabilities even when multimodal data is missing.

[0156] The multimodal sentiment analysis model in this embodiment achieved better results than other methods on the MOSEI dataset. The comparison of experimental results is shown in Table 2.

[0157] Table 2 Comparison of experimental results on the MOSEI dataset

[0158]

[0159] As shown in Table 2, the experimental results on the MOSEI dataset demonstrate that the multimodal sentiment analysis method proposed in this embodiment achieves stable results across all evaluation metrics, exhibiting good performance in accuracy, F1 score, and relevance. Even under large-scale data conditions, this embodiment maintains good sentiment recognition capabilities, while demonstrating high consistency and low error in relevance and error metrics, indicating good stability in complex data environments. The reasons for these technical effects are as follows: This embodiment uses reliability-guided diffusion to repair the dominant modality, generatively recovering missing language modalities and improving the integrity of dominant modality information; simultaneously, it combines missing modality perception and modality reliability assessment, enabling the model to adaptively utilize effective information even in cases of random modality loss; furthermore, it enhances multimodal fusion through graph neural networks, improving cross-modal semantic modeling capabilities and making the fusion results more comprehensive. Therefore, this embodiment can achieve stable sentiment recognition results even in complex data environments.

[0160] In summary, this embodiment can achieve stable experimental results on datasets of different sizes and complexities, indicating that this embodiment has good adaptability and stability under multimodal missing conditions.

[0161] The performance comparison of the multimodal sentiment analysis model in this embodiment on the MOSI dataset under different modal data missing conditions is shown in Table 3, where the value in each cell represents the F1 score.

[0162] Table 3. Comparison of experimental results under different modal data missing conditions in this embodiment on the MOSI dataset.

[0163]

[0164] Table 3 presents the emotion recognition results of this embodiment under different modality missing conditions on the MOSI dataset. It can be seen that this embodiment can obtain stable recognition results under both single-modal and multimodal joint missing conditions. These results demonstrate that by introducing a missing condition perception mechanism and a modality reliability assessment mechanism, this embodiment enables the diffusion model to adaptively adjust based on the effectiveness of auxiliary modalities when recovering the dominant language modality, thus maintaining high emotion recognition capability even with incomplete modality information. Furthermore, under severe multimodal missing conditions, this method can still output effective results, indicating that the proposed reliability-guided diffusion-dominant modality repair and graph neural network-enhanced multimodal fusion possesses good robustness.

[0165] The performance comparison of the multimodal sentiment analysis model in this embodiment on the MOSEI dataset under different modal data missing conditions is shown in Table 4, where the value in each cell represents the F1 score.

[0166] Table 4. Comparison of experimental results under different modal data missing conditions in this embodiment on the MOSEI dataset.

[0167]

[0168] Table 4 presents the experimental results of this embodiment under different modality missing conditions on the MOSEI dataset. As shown in Table 4, this embodiment achieves stable emotion recognition results under different missing modes. This result further illustrates that this embodiment effectively reduces the impact of modality missing on model performance by introducing modality reliability weights to dynamically weight auxiliary modality information and combining reliability-guided diffusion-based dominant modality repair to recover the dominant modality. Therefore, this embodiment still exhibits good adaptability and stability in complex modality missing scenarios.

[0169] In summary, the experimental results demonstrate that the missing perception and reliability-guided diffusion-dominant modality repair proposed in this embodiment can effectively improve the stability and reliability of emotion recognition when multimodal data is missing or incomplete.

[0170] In multimodal sentiment analysis tasks, this embodiment overcomes the problem that sentiment recognition performance is significantly reduced due to the lack or incompleteness of modal data such as language, audio and video in practical applications. It achieves accurate recognition of sentiment information in the case of missing modalities and improves the robustness and generalization ability of the model.

[0171] Example 2:

[0172] This embodiment provides a multimodal sentiment analysis system based on a diffusion model, which is used to implement the multimodal sentiment analysis system method based on a diffusion model as described above; the system includes:

[0173] The multimodal sentiment analysis training dataset construction unit is used to collect multimodal data including language data, audio data, and video data, preprocess and label the multimodal data to construct a multimodal sentiment analysis dataset, and randomly omit some modalities in the multimodal data to generate training and test data containing the missing modalities. Specifically, the language data is segmented and encoded, the audio data is normalized and time-aligned, and the video data is frame-aligned and key feature extracted to obtain the corresponding language sequences, audio signals, and video frame information, and attach corresponding sentiment labels to construct the multimodal sentiment analysis training dataset.

[0174] The multimodal data feature extraction unit is used to extract features from language sequences, audio signals, and video frame information respectively, obtaining corresponding language features, audio features, and video features. It then performs random missing feature extraction on these language, audio, and video features, with the discard ratio ranging from 0% to 100%, resulting in missing language features, missing audio features, and missing video features. Specifically, firstly, language data is used as input and encoded using the BERT model to obtain language features representing semantic information. Secondly, audio data is used as input and Librosa is used to extract audio modal features. Finally, video data is used as input and OpenFace is used to extract video modal features. After obtaining each modal feature, data missing feature extraction is performed on each modal data to form missing language features, missing audio features, and missing video features. Therefore, the input of the modal input and feature extraction module is language data, audio data and video data, and the output is language features, audio features, video features and missing language features, missing audio features and missing video features; among them, missing language features, missing audio features and missing video features are sent to the missing feature perception module.

[0175] Multimodal sentiment analysis model building unit, used to build multimodal sentiment analysis models;

[0176] The multimodal sentiment analysis model training unit is used to train the multimodal sentiment analysis model based on the multimodal sentiment analysis dataset. By constructing sentiment prediction loss function, diffusion loss function and reconstruction loss function, the parameters of the multimodal sentiment analysis model are optimized, so that the multimodal sentiment analysis model has stable sentiment analysis capabilities.

[0177] The multimodal sentiment analysis model construction unit in this embodiment includes:

[0178] The missing feature awareness module models the missing status of corresponding modalities based on missing language features, missing audio features, and missing video features, generating corresponding language missing feature masks, audio missing feature masks, and video missing feature masks. Specifically, first, a language missing feature mask is constructed for language features; second, an audio missing feature mask is constructed for audio features; and finally, a video missing feature mask is constructed for video features. A mask value of 1 indicates that the position is missing; a mask value of 0 indicates that the position is complete. The final outputs are the language missing feature mask, audio missing feature mask, and video missing feature mask, and this mask information is then sent to the modal reliability assessment module and the reliability-guided diffusion-dominant modal repair module.

[0179] The modal reliability assessment module is used to evaluate the effectiveness of missing audio features, missing video features, audio missing masks, and video missing masks, generating weighted audio features and weighted video features. Specifically, firstly, the missing audio features are assessed using feature quality assessment to obtain a feature quality score; the audio missing mask is statistically analyzed using a missing proportion calculation to obtain an integrity score; then, the feature quality score and integrity score are fused using reliability fusion calculation to obtain an audio modal reliability weight; the audio features are then weighted according to the audio modal reliability weight to obtain weighted audio features. Secondly, the missing video features are assessed using feature quality assessment to obtain a feature quality score; the video missing mask is statistically analyzed using a missing proportion calculation to obtain an integrity score; then, the feature quality score and integrity score are fused using reliability fusion calculation to obtain a video modal reliability weight; the video features are then weighted according to the video modal reliability weight to obtain weighted video features; finally, the weighted audio features and weighted video features are sent to the reliability-guided diffusion-dominant modal repair module.

[0180] The reliability-guided diffusion dominant modality repair module is used to progressively repair missing language features through forward, backward, and mask-preserving recovery processes to obtain repaired language features and ensure the integrity of the dominant modality's information. Specifically, the reliability-guided diffusion dominant modality repair includes forward, backward, and mask-preserving recovery processes. First, the forward process uses the language features as the starting state and constructs a diffusion process of language features evolving from a clear state to a noisy state by progressively adding Gaussian noise. Second, the backward process uses the missing language features as the recovery starting point and progressively denoises the language features using weighted audio features, weighted video features, and a language missing mask as conditional inputs to recover lost or damaged semantic information in the language modality, gradually obtaining the diffused and recovered language features. Finally, the mask-preserving recovery takes the diffused and recovered language features, the missing language features, and the language missing mask as inputs, updating only the missing regions to finally obtain the repaired language features. The repaired language features are then fed into the graph neural network-enhanced multimodal fusion module as the dominant modality representation.

[0181] The graph neural network-enhanced multimodal fusion module is used to enhance repaired language features, missing audio features, and missing video features through node initialization, graph construction, and graph neural networks. It also performs interactive modeling of multimodal information to achieve interaction and fusion of multimodal features, obtaining enhanced language features, enhanced audio features, and enhanced video features. Specifically, firstly, the repaired language features, missing audio features, and missing video features are used as inputs to construct corresponding language nodes, audio nodes, and video nodes, completing node initialization. Secondly, using the language nodes, audio nodes, and video nodes as inputs, intramodal and intermodal connections are established to form a cross-modal graph structure. Finally, the constructed graph structure is input into a three-layer graph neural network for message passing and feature updating, yielding enhanced language features, enhanced audio features, and enhanced video features. Therefore, the inputs of the graph neural network-enhanced multimodal fusion module are repaired language features, missing audio features, and missing video features, and the outputs are enhanced language features, enhanced audio features, and enhanced video features. These enhanced language features, enhanced audio features, and enhanced video features are then fed into the sentiment prediction module.

[0182] The missing data reconstruction module reconstructs multimodal data, including missing language features, missing audio features, and missing video features, using Transformer layers. It obtains language features, reconstructs audio features, and reconstructs video features, and further calculates the missing data reconstruction loss for subsequent optimization. Specifically, first, missing language features are input and processed by a Transformer layer to obtain reconstructed language features; second, missing audio features are input and processed by a Transformer layer to obtain reconstructed audio features; finally, missing video features are input and processed by a Transformer layer to obtain reconstructed video features. Subsequently, the reconstructed language features, reconstructed audio features, and reconstructed video features are compared with their corresponding original features to obtain the reconstruction loss, which is used for subsequent model training optimization. Therefore, the inputs to the missing data reconstruction module are missing language features, missing audio features, and missing video features, and the outputs are reconstructed language features, reconstructed audio features, reconstructed video features, and the reconstruction loss; the reconstruction loss is used for subsequent model training optimization.

[0183] The sentiment prediction module performs pooling operations on enhanced language features, enhanced audio features, and enhanced video features separately, then concatenates them, and finally obtains the sentiment prediction value through a classifier, and further calculates the sentiment prediction loss. Specifically, firstly, the enhanced language features, enhanced audio features, and enhanced video features are used as input, and the multimodal features are compressed and aggregated through a pooling layer to obtain a feature representation suitable for classification. Then, the pooled multimodal features are concatenated to obtain a fused feature representation. Next, the fused feature representation is fed into a classifier for nonlinear mapping and category discrimination, outputting a sentiment prediction value. Finally, the sentiment prediction loss is calculated based on the difference between the sentiment prediction value and the true label. Therefore, the inputs to the sentiment prediction module are enhanced language features, enhanced audio features, and enhanced video features, and the outputs are the sentiment prediction value and the sentiment prediction loss.

[0184] Example 3:

[0185] This embodiment also provides an electronic device, including: a memory and a processor;

[0186] The memory stores the instructions executed by the computer.

[0187] The processor executes computer execution instructions stored in the memory, causing the processor to perform the diffusion-based multimodal sentiment analysis method according to any embodiment of the present invention.

[0188] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0189] Memory can be used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0190] Example 4:

[0191] This embodiment also provides a computer-readable storage medium storing multiple instructions, which are loaded by a processor to cause the processor to execute the diffusion-based multimodal sentiment analysis method according to any embodiment of the present invention. Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0192] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0193] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0194] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0195] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0196] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal sentiment analysis method based on a diffusion model, characterized in that, The method is as follows: Constructing a multimodal sentiment analysis training dataset: Collect multimodal data including language data, audio data, and video data, and perform word segmentation and encoding on the language data, feature standardization and time alignment on the audio data, and frame alignment and key feature extraction on the video data to obtain the corresponding language sequences, audio signals, and video frame information, and attach corresponding sentiment tags to construct a multimodal sentiment analysis training dataset. Extracting multimodal data features: Feature extraction is performed on language sequences, audio signals and video frame information respectively to obtain corresponding language features, audio features and video features. Then, random missing features are constructed on the language features, audio features and video features, with the discard ratio ranging from 0 to 100% to obtain the corresponding missing language features, missing audio features and missing video features. A multimodal sentiment analysis model is constructed, specifically as follows: Missing features perception: Modeling the missing features of the corresponding modal based on missing language features, missing audio features, and missing video features, generating corresponding language missing masks, audio missing masks, and video missing masks; Modal reliability assessment: Evaluating the effectiveness of missing audio features, missing video features, audio missing masks, and video missing masks, generating weighted audio features and weighted video features. Reliability-guided diffusion dominant modality repair: Missing language features are progressively repaired through forward, reverse, and mask-preserving recovery processes to obtain repaired language features and ensure the integrity of the dominant modality's information. Graph neural network-enhanced multimodal fusion: Repaired language features, missing audio features, and missing video features are enhanced through node initialization, graph construction, and graph neural networks. Interactive modeling of multimodal information is also performed to achieve interaction and fusion of multimodal features, obtaining enhanced language features, enhanced audio features, and enhanced video features. Missing data reconstruction: Multimodal data including missing language features, missing audio features, and missing video features is reconstructed using Transformer layers to obtain language features, reconstructed audio features, and reconstructed video features. Sentiment prediction: The enhanced language features, enhanced audio features, and enhanced video features are pooled separately and then concatenated. The sentiment prediction value is then obtained through a classifier. Specifically, the reliability-guided diffusion-dominant mode repair is as follows: Forward diffusion process: defining the original complete linguistic features as The forward diffusion process using the diffusion model is carried out iteratively. Add Gaussian noise to generate a series of samples with gradually increasing noise. The forward diffusion process is formally defined as: ;in, , Indicates preset variance scheduling. ; Let it be the identity matrix; set any time step of Direct sampling: ;in, Structured missing features are introduced into the language modality, and Gaussian noise is added to obtain the missing language features. , will lack language features Treat it as a noisy linguistic feature and treat the missing linguistic feature as... Used for noise reduction and recovery in the reverse diffusion process; Diffusion Reverse Process: From Noisy Language Features China Resumption The forward diffusion process is reversed; specifically, the missing language features are used as the starting point for recovery, and weighted audio features are processed through a multi-head cross-attention mechanism. Weighted video features and language missing mask The input is processed by a denoising network to progressively denoise and recover lost or damaged semantic information in the language modality. After multiple iterative denoising steps, diffused recovery of language features is gradually obtained. The formula is as follows: ; Mask Preservation Recovery: Recovering Linguistic Features Based on Diffusion Missing language features and language missing mask Only missing regions are updated, while the original linguistic features of non-missing regions remain unchanged. The formula is: ;in, This represents element-wise multiplication. This indicates the language features after restoration.

2. The multimodal sentiment analysis method based on a diffusion model according to claim 1, characterized in that, The specific steps for extracting features from multimodal data are as follows: Language feature extraction: Language features are extracted by performing contextual semantic encoding on language sequences using a pre-trained BERT-base-uncased model. ;in, Indicates the length of the language sequence; Represents the dimensions of language features; Audio feature extraction: The audio signal is extracted using the Librosa tool to obtain audio features. ;in, Indicates the length of the audio frame; Indicates the dimension of audio features; Video feature extraction: Facial expression analysis was performed on video frame information using the OpenFace tool to extract video features. ;in, Indicates the length of the video frame. Indicates the dimension of video features; Constructing multimodal missing features: Missing language features are represented using masking, missing audio features using zero-padding, and missing video features using zero-padding to obtain missing language features. Missing audio features and missing video features .

3. The multimodal sentiment analysis method based on a diffusion model according to claim 1 or 2, characterized in that, The specific meaning of missing language perception is: based on missing language features Missing audio features and missing video features Generate language missing masks respectively Audio missing mask and video missing mask The expression is as follows: ； in, This represents the characteristics of mode M at time step t; This represents the missing mask value at the corresponding position. A mask value of 1 indicates that the corresponding position is missing; a mask value of 0 indicates that the corresponding position is complete. Modal reliability assessment specifically involves: calculating modal reliability weights based on the degree of missing auxiliary modes and feature quality; specifically, it involves applying audio missing masks. The missing percentage is calculated to obtain the audio modality integrity score, using the following formula: ;in, Indicates the length of the audio sequence; This indicates the missing state of the audio modality at time step t; This represents the integrity score of the audio modality; simultaneously, it assesses the missing audio features through feature quality evaluation. To perform a quality assessment, obtain the quality score for the missing audio features. The formula is: ;in, Represents the audio features at time step t; Represents the L2 norm; This represents the audio feature quality score; then the audio feature quality score is... and audio modality integrity score To perform reliability fusion calculations and obtain the audio modal reliability weights, the formula is as follows: ;in, , and These are learnable parameters; Represents the Sigmoid function; This represents the audio modality reliability weight; the audio features are weighted according to the audio modality reliability weight to obtain the weighted audio features, as shown in the formula: ; Calculate the missing proportion of the video missing mask to obtain the video modality integrity score, using the following formula: ;in, Indicates the length of the video sequence; This indicates the missing state of the video modality at time step t; This represents the integrity score of the video modality; simultaneously, feature quality assessment is performed on missing video features to obtain the missing video feature quality score, using the following formula: ;in, This represents the video feature at time step t; Represents the L2 norm; The video feature quality score is represented by the video feature quality score. Then, the missing video feature quality score and the video modality integrity score are reliably fused to obtain the video modality reliability weight, as shown in the formula: ;in, , and These are learnable parameters; σ(⋅) represents the Sigmoid function; This represents the video modality reliability weight; the video features are weighted according to the video modality reliability weight to obtain the weighted video features, as shown in the formula: .

4. The multimodal sentiment analysis method based on a diffusion model according to claim 3, characterized in that, The specific details of graph neural network-enhanced multimodal fusion are as follows: Node initialization: Initialize a node containing A graph of nodes, where each node corresponds to an input mode ( A lexical element in ); for time steps At that time, the mode is The nodes, initial features Directly set it as the corresponding word feature vector, the formula is: , is used to use the feature vectors of each mode at the corresponding time step as the initial representation of the corresponding node in the graph; where, Representing nodes in a graph neural network Initial features at layer 0; This represents the node corresponding to mode M at time step t; These represent the language modality, video modality, and audio modality, respectively. The characteristic matrix representing mode M The eigenvector corresponding to the t-th time step; when hour, Indicates the restored language features ;when hour, Indicates missing video features ;when hour, Indicates missing audio features ; Graph Construction: First, establish intramodal and intermodal connections based on language modal nodes, audio modal nodes, and video modal nodes to form a cross-modal graph structure composed of intramodal edges and cross-modal edges; then, calculate the edges E and their weights in the cross-modal graph structure using an attention mechanism. Where u represents a node and v represents an adjacent node; for modal inner edges, the weight formula for language modality-language modality edges is: ;in, and Represents the learnable projection matrix; The dimensions represent the key and the query; the weight formula for video modality-video modality edges is: ;in, and The projection matrix represents the learnable video modality; the weight formula for audio modality-audio modality edges is: ;in, and This represents the learnable projection matrix corresponding to the audio modality; for cross-modal edges, language nodes are computed using language features as queries. and video nodes Between and and audio nodes The edge weights between them are as follows: ； ； in, and The projection matrix is ​​a learnable representation; language-guided edge weights enable the graph structure to dynamically enhance relevant cross-modal connections while pruning irrelevant ones. The formula for the edge weights between the video and audio modalities is as follows: ； in, This represents the learnable query projection matrix corresponding to the video modality; This represents the learnable key projection matrix corresponding to the audio modality; Updated Graph Neural Network: A three-layer graph attention network is used for feature propagation and refinement; specifically: for the first... Layered GNN, each node Receive from its neighboring nodes The message, and calculate the attention coefficient. Then update the corresponding features using the following formula: ;in, Indicates a shared linear transformation matrix; This represents the activation function; after passing through a three-layer graph neural network, the updated node features are obtained, and the updated node features are reorganized into sequence representations for each modality: ; ; ;in, This indicates the enhanced linguistic features; Indicates the enhanced audio features; This indicates the enhanced video features.

5. The multimodal sentiment analysis method based on a diffusion model according to claim 4, characterized in that, Missing data reconstruction specifically involves reconstructing missing information for each modality, including missing language features, missing audio features, and missing video features, through two Transformer layers, and learning from the missing modalities contaminated by noise. Input restored to original clean modal features The reconstruction formula is: ;in, Indicates modality The reconstructor; This indicates the original clean feature. Reconstruction characteristics; The formula for sentiment prediction is: ;in, This indicates average pooling or max pooling operations; This indicates the sentiment prediction result.

6. The multimodal sentiment analysis method based on a diffusion model according to claim 1, characterized in that, This method also includes training optimization of the multimodal sentiment analysis model, as detailed below: Constructing the loss function: The multimodal sentiment analysis model is jointly optimized by diffusion model loss, missing data reconstruction loss, and sentiment prediction loss. Specifically, the diffusion loss function is first introduced. The diffusion loss function is used to constrain the multimodal sentiment analysis model's ability to predict noise during backdiffusion, improve the accuracy of language modality recovery, and ensure the effective recovery of missing language features. The formula is: ;in, Represents real noise; This represents the noise predicted by the multimodal sentiment analysis model; Indicates the first step in the diffusion process The noisy language features of the steps; These represent missing audio features and missing video features, respectively. The diffusion time step is represented; secondly, a reconstruction loss function is introduced. The reconstruction loss function measures the difference between the reconstructed features and the original features, constraining the multimodal sentiment analysis model and improving the completeness and consistency of feature representations across different modalities. The formula is: ;in, Indicates the number of samples in the training set; Representing modes The The original input features of each sample; Indicates the first Modal features reconstructed from individual samples; finally, a sentiment prediction loss is constructed. The sentiment prediction loss is used to measure the difference between the prediction results of a multimodal sentiment analysis model and the true labels, thereby improving the prediction performance of sentiment analysis. The formula is as follows: ;in, Indicates the number of training samples; Indicates the first The true sentiment labels of each sample; This represents the predicted value from the multimodal sentiment analysis model; Model optimization: The diffusion model loss, missing data reconstruction loss, and sentiment prediction loss are weighted and summed to obtain the total loss function. Then, the gradient descent method is used to optimize and train the multimodal sentiment analysis model parameters. The formula is as follows: ;in, The hyperparameters representing the loss are used to balance the influence of different loss terms. During the training of the multimodal sentiment analysis model, the model parameters are continuously updated through the backpropagation algorithm, so that the total loss function gradually converges. When the multimodal sentiment analysis model is not yet fully trained, iterative optimization is performed on the multimodal sentiment analysis training dataset. After the multimodal sentiment analysis model is trained, it can perform sentiment classification or sentiment intensity prediction on the input multimodal data.

7. A multimodal sentiment analysis system based on a diffusion model, characterized in that, This system is used to implement the multimodal sentiment analysis method based on the diffusion model as described in any one of claims 1 to 6; the system includes: The multimodal sentiment analysis training dataset construction unit is used to collect multimodal data including language data, audio data and video data, and to perform word segmentation and encoding processing on language data, feature standardization and time alignment processing on audio data, and frame alignment and key feature extraction processing on video data to obtain corresponding language sequences, audio signals and video frame information, and attach corresponding sentiment tags to construct a multimodal sentiment analysis training dataset. The multimodal data feature extraction unit is used to extract features from language sequences, audio signals and video frame information respectively, to obtain corresponding language features, audio features and video features, and to perform random missing construction processing on language features, audio features and video features, with the discard ratio ranging from 0 to 100%, to obtain corresponding missing language features, missing audio features and missing video features. A multimodal sentiment analysis model building unit is used to construct a multimodal sentiment analysis model; the multimodal sentiment analysis model building unit includes: The missing feature awareness module is used to model the missing status of the corresponding modalities based on missing language features, missing audio features, and missing video features, and generate corresponding language missing feature masks, audio missing feature masks, and video missing feature masks. The modal reliability assessment module is used to evaluate the effectiveness of missing audio features, missing video features, audio missing masks, and video missing masks, and to generate weighted audio features and weighted video features. The reliability-guided diffusion dominant modality repair module is used to gradually repair missing language features through forward process, reverse process and mask preservation recovery, obtain the repaired language features, and ensure the information integrity of the dominant modality. The graph neural network-enhanced multimodal fusion module is used to enhance the repaired language features, missing audio features, and missing video features through node initialization, graph construction, and graph neural networks. It also performs interactive modeling of multimodal information to achieve the interaction and fusion of multimodal features and obtain enhanced language features, enhanced audio features, and enhanced video features. The missing data reconstruction module is used to reconstruct multimodal data, including missing language features, missing audio features, and missing video features, through the Transformer layer, to obtain language features, reconstruct audio features, and reconstruct video features, and further calculate the missing data reconstruction loss. The sentiment prediction module performs pooling operations on the enhanced language features, enhanced audio features, and enhanced video features, then concatenates them, obtains the sentiment prediction value through a classifier, and further obtains the sentiment prediction loss. The multimodal sentiment analysis model training unit is used to train the multimodal sentiment analysis model based on the multimodal sentiment analysis dataset. By constructing sentiment prediction loss function, diffusion loss function and reconstruction loss function, the parameters of the multimodal sentiment analysis model are optimized, so that the multimodal sentiment analysis model has stable sentiment analysis capabilities.

8. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the diffusion-based multimodal sentiment analysis method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the diffusion-based multimodal sentiment analysis method as described in any one of claims 1 to 6.

Images