Multi-modal sentiment analysis model and method, electronic equipment and medium

Abstract

The invention provides a multi-modal sentiment analysis model and method, electronic equipment and a medium, and the model comprises a feature enhancement module which is used for extracting original multi-modal data features through an exclusive tool, constructing a graph structure, and enhancing the graph structure through a graph convolutional network to obtain multi-modal enhanced features; the modal multi-stage balance module is used for processing enhanced features by using different multi-stage network structures and outputting multi-modal consistency representation; the modal noise reduction decoupling and specificity recombination module is used for obtaining low-noise representation through a modal noise reduction decomposer based on the global information and recombining the low-noise representation in a modal bank to generate low-noise multi-modal specificity representation; and the hierarchical fusion prediction module is used for fusing consistency and specificity representation according to single-peak, double-peak and three-peak modes, and outputting an emotion prediction result through a multi-layer perceptron.

Description

Technical Field

[0001] This invention relates to the field of sentiment analysis technology, specifically to a multimodal sentiment analysis method, apparatus, electronic device, and medium. Background Technology

[0002] Emotion is an indispensable component of human cognition and social interaction, influencing an individual's judgment, decision-making, and behavioral expression. As one of the core dimensions of human intelligence, emotion is often naturally expressed through various modalities such as language, facial expressions, tone of voice, and body movements. With the gradual integration of artificial intelligence into daily life, human-computer interaction systems with emotional intelligence have become a key direction for technological development. Whether it's virtual assistants, social robots, or wearable devices, the ability to understand and respond to user emotions has become a core element of personalized interactive experiences. Therefore, accurately perceiving and understanding emotional states has significant application value for intelligent perception and decision-making, understanding behavioral intentions, and optimizing human-computer interaction.

[0003] Early research focused on unimodal sentiment analysis, primarily text-based. While this approach is applicable in some scenarios, as interactive scenarios become more complex, emotional expression exhibits greater complexity and modal diversity, making it difficult for traditional unimodal sentiment recognition methods to comprehensively capture complex emotions. Therefore, unimodal sentiment analysis has gradually evolved into multimodal sentiment analysis. The intuitive advantage of MSA (Multimodal Sentiment Analysis) lies in its ability to effectively complement the emotional tendency of the text modality when it is unclear. For example, text may be semantically neutral, but by combining pleasant facial expressions in visual information and positive tone in audio information, its positive emotion can be inferred more accurately. Although multimodal data compensates for the shortcomings of unimodal data in sentiment analysis through intermodal information complementarity, existing multimodal data fusion methods generally rely on feature alignment between modalities, neglecting the modeling of deep coupling mechanisms between modalities. Summary of the Invention

[0004] In view of this, the present disclosure provides a multimodal sentiment analysis model, method, electronic device, and medium, which at least partially solves the problems existing in the prior art.

[0005] In a first aspect, embodiments of this disclosure provide a multimodal sentiment analysis model, which includes: The feature enhancement module is used to obtain multimodal feature representations of the original multimodal data through different feature extraction tools, construct graphs of the multimodal feature representations, and then enhance the multimodal data using a graph convolutional network to obtain multimodal enhanced features. A modal multi-level balancing module is used to process the multimodal enhancement features using different network structures at each stage to obtain a multimodal consistent representation; The modal denoising decoupling and specific recombination module is used to decompose the multimodal enhancement features according to the multimodal global information through the modal denoising decomposer to obtain a low-noise multimodal representation, and to reconstruct the low-noise multimodal representation in the modal bank to obtain a low-noise multimodal specific representation. The hierarchical fusion prediction module is used to process and fuse the input multimodal consistency representation and the input low-noise multimodal specific representation in the manner of modal unimodal, modal bimodal, and modal trimodal, and to predict the result of the processing and fusion through a multilayer perceptron to obtain the sentiment prediction result.

[0006] According to a specific implementation of this disclosure, the feature enhancement module obtains feature representations of the original multimodal data using different feature extraction tools, including: The original text data is processed using a pre-trained model specifically designed for sentiment analysis to obtain text modal feature representations; The feature enhancement module uses a speech emotion feature extraction tool to extract audio modal information from the original video data to obtain an audio modal feature representation; The feature enhancement module uses facial expression analysis tools to extract visual modal information from the original video data to obtain visual modal feature representations.

[0007] According to a specific implementation of this disclosure, the step of constructing a graph from the multimodal representation and then using a graph convolutional network to enhance the multimodal data to obtain multimodal enhanced features includes: The feature enhancement module constructs a graph from all samples in a subset according to each modality, obtaining text modality graphs, audio modality graphs, and video modality graphs respectively, where each sample in each modality graph is a node; Construct an adjacency matrix for the graph, dividing all nodes into two categories. Nodes belonging to the same category have a value of 1 in the adjacency matrix, and otherwise, their value is set to 0. By using graph convolutional networks to aggregate features of nodes in a graph structure, the internal features of the multimodal data are enhanced to obtain multimodal enhanced features.

[0008] According to a specific implementation of an embodiment of this disclosure, the feature enhancement module randomly removes some edges from each modal graph and randomly adds a preset number of cross-class edges to each modal graph.

[0009] According to a specific implementation of an embodiment of this disclosure, the modal multi-level balancing module processes the multimodal enhancement features using different network structures at each stage to obtain a multimodal consistent representation, including: Generative adversarial networks are constructed using generators and discriminators; The generative adversarial network is used to adjust the feature distribution of the non-text modality, thereby achieving preliminary alignment between the non-text modality and the text modality; Using audio or visual modalities as keys and values ​​in the Transformer architecture, and text modalities as queries, attention scores are calculated. Then, the attention scores are normalized using an activation function to obtain an attention weight matrix of the degree of response of non-text modal features to text semantics. Multiplying the attention weight matrix by the value yields a non-textual modal feature representation that incorporates textual semantic information; By employing residual connections, the pre-aligned non-textual modal features are added to the non-textual modal features that have been fused with textual semantic information to obtain attention features; The attention features are input into the feedforward neural network, and the output of the feedforward neural network is again residually connected with the attention features to obtain a non-text modality augmented representation. The lower bound of mutual information between the text modal features and the non-text modal features is estimated, and the lower bound of mutual information is maximized to obtain the multimodal consistency representation.

[0010] According to a specific implementation of this disclosure, the modal denoising decoupling and specific recombination module decomposes the enhanced features based on global information using a modal denoising decomposer to obtain a low-noise multimodal representation, and reassembles the low-noise multimodal representation in a modal bank to obtain a low-noise multimodal specific representation, including: The multimodal enhancement features are concatenated to obtain a guidance vector for multimodal global information; Independent modal denoising decomposers are used to suppress noise in each modal representation of the multimodal enhancement representation, and the denoised feature vectors of each modality are obtained. The guiding vector and the feature vectors of each modality are concatenated and a dynamic weighting factor is generated through nonlinear transformation. The dynamic weighting factor is multiplied by the denoised feature vectors of each modality to obtain the specific representation of each modality. The process of obtaining the specific representation is repeated until the preset conditions are met; The modality-specific representations obtained each time in the repeated execution are recombined in the modality bank to obtain low-noise multimodality-specific representations.

[0011] According to a specific implementation of this disclosure, the modal denoising decoupling and specific recombination module uses orthogonal constraint loss to constrain the modal consistency representation and the modal specificity representation.

[0012] According to a specific implementation of an embodiment of this disclosure, the hierarchical fusion prediction module processes and fuses the input multimodal consistency representation and the input low-noise multimodal specific representation in a modal unimodal, modal bimodal, and modal trimodal manner, and uses a multilayer perceptron to predict the processing and fusion result to obtain a sentiment prediction result, including: The consistency representation and specific representation of a single modality are concatenated to obtain the initial single modality sentiment representation; A gated weighting factor is generated by linear transformation, and the gated weighting factor is multiplied by the corresponding single-modal sentiment representation to obtain the fusion vectors of text modality, visual modality and audio modality respectively; The fusion vectors of the text modality, the visual modality, and the audio modality are averaged to obtain a single-peak fusion representation; Any pairwise modalities in the initial unimodal sentiment representation are merged to obtain the corresponding bimodal feature vector, and the similarity between the two modalities in the bimodal feature vector is calculated. The fusion weight is obtained based on the gating weighting factor and the similarity between the two modes in the bimodal feature vector; Multiplying the fusion weights by the corresponding bimodal feature vectors yields a bimodal fusion representation; The consistency representation and specific representation of the three modalities are concatenated, and the three-modal fusion weights are extracted through an attention mechanism. The three-modal fusion weights are then scaled and normalized to obtain the attention scores for the corresponding three modalities. The consistency representation and specific representation of the three modalities are stacked and multiplied by the attention score to obtain the three-peak fusion representation; The single-peak fusion representation, the double-peak fusion representation, and the triple-peak fusion representation are concatenated, and the result is predicted using a multilayer perceptron to obtain the sentiment prediction result.

[0013] In a second aspect, embodiments of this disclosure provide a multimodal sentiment analysis method, the method using the multimodal sentiment analysis model provided in the first aspect for sentiment analysis, the method comprising: The original multimodal data is subjected to feature extraction using different feature extraction tools to obtain a multimodal representation of the original multimodal data. A graph is constructed on the multimodal representation, and then a graph convolutional network is used to enhance the multimodal data to obtain multimodal enhanced features. The multimodal enhancement features are processed using different network structures at each stage to obtain a multimodal consistent representation; Based on multimodal global information, the multimodal enhancement features are decomposed by a modal denoising decomposer to obtain a low-noise multimodal representation, and the low-noise multimodal representation is reorganized in a modal bank to obtain a low-noise multimodal specific representation. The multimodal consistency representation and the low-noise multimodal specific representation are processed and fused in the manner of modal unimodal, modal bimodal, and modal trimodal, and the processing and fusion results are predicted by a multilayer perceptron to obtain the sentiment prediction result.

[0014] Thirdly, embodiments of this disclosure provide an electronic device, the electronic device comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the multimodal sentiment analysis method described above.

[0015] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the aforementioned multimodal sentiment analysis method.

[0016] In summary, compared with the prior art, this embodiment has the following advantages: 1. Through a multi-level modal balancing module, the audio and visual modalities are brought closer to the text modal in terms of distribution and semantics through the progressive action of adversarial alignment, text-guided Transformer, and mutual information constraints. This design avoids the limitation of traditional methods that only focus on surface feature alignment while neglecting deep semantic consistency, and effectively reduces the difference between structured and unstructured information between modalities.

[0017] 2. Utilizing a modal denoising decoupling and specific recombination module, modal noise is weakened through multiple rounds of denoising decomposition, and deep specific information is mined by combining global guidance vectors. Simultaneously, orthogonal constraints are used to separate modal consistency representations from specific representations, thereby avoiding information redundancy. This module solves the problems of excessive modal feature noise and the submergence of important semantics by redundant information in traditional methods.

[0018] 3. A hierarchical fusion prediction module is constructed, which processes data according to a progressive path of single-modal fusion, bimodal fusion, and trimodal fusion. Single-modal fusion focuses on integrating the consistency and specificity information of a single modality; bimodal fusion splices two modalities together and dynamically evaluates the contribution of modality pairs based on similarity; trimodal fusion adaptively fuses the consistency and specificity information of the three modalities through an attention mechanism. This design avoids the limitations of traditional fusion methods that tend to ignore local modal cues or over-rely on alignment, thereby fully exploring the sentiment associations between modalities. Attached Figure Description

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0020] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0021] Figure 1 A schematic diagram of the multimodal sentiment analysis model provided in the first embodiment of the present invention; Figure 2 A schematic diagram of a generative adversarial network provided in an embodiment of this application; Figure 3 A schematic diagram of a text-guided Transformer architecture provided for an embodiment of this application; Figure 4 A schematic diagram of a hierarchical fusion prediction module provided in an embodiment of this application; Figure 5 A flowchart illustrating the multimodal sentiment analysis method provided in the second embodiment of the present invention; Figure 6 An exemplary structural diagram of a device capable of implementing the method according to an embodiment of the present invention is shown. Detailed Implementation

[0022] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0023] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0024] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0025] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0026] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0027] Please see Figure 1 This application provides a multimodal sentiment analysis model, such as... Figure 1 As shown, the model includes: a feature enhancement module 101, a modal multi-level balancing module 102, a modal denoising decoupling and specific recombination module 103, and a hierarchical fusion prediction module 104.

[0028] First, we will introduce the feature enhancement module 101 in detail.

[0029] Optionally, the feature enhancement module obtains multimodal feature representations of the original multimodal data through different feature extraction tools, processes the original text data using a pre-trained model dedicated to sentiment analysis to obtain text modal feature representations, extracts audio modal information from the original video data using a speech sentiment feature extraction tool to obtain audio modal feature representations, and extracts visual modal information from the original video data using a facial expression analysis tool to obtain visual modal feature representations.

[0030] Specifically, for text modalities, with the rise of Transformer technology, numerous Transformer-based pre-trained models have played an important role in improving the performance of various tasks. As shown in formula (1), in this embodiment, the feature enhancement module uses a pre-trained model to process the original text data to obtain the text modal. For English text data, the SentiLARE pre-trained model, which is dedicated to sentiment analysis, is used to obtain the text representation, resulting in a text modal feature vector with a dimension of 768. SentiLARE combines part-of-speech tagging and dependency parsing to extract structured language features, and obtains sentiment tendencies from the SentiWordNet sentiment dictionary resource based on the sentiment attention mechanism. It dynamically fuses word-level sentiment polarity with sentence-level semantic representation, thereby effectively injecting display language knowledge into the pre-training process. For Chinese text data, the BERT model is used to obtain contextual dependency information and generate text modal feature representations.

[0031] For the original audio data, the feature enhancement module uses speech emotion feature extraction tools (e.g., the open-source tool COVAREP) to extract features such as Mel-Frequency Cepstral Coefficients, line spectral frequencies, and zero-crossing rates from the original video frame data to capture subtle tone and intonation information, thereby reflecting emotional changes in speech. For the visual modality, the feature enhancement module uses facial expression analysis tools (e.g., Pyfeat) to extract features such as facial expressions, action units, and head posture, and to mine visual cues related to emotional expression. In order to obtain the temporal features of the audio and visual modalities, as shown in Equation (2), the feature enhancement module inputs the processed modal features into an LSTM (Long Short-Term Memory) network to finally obtain complete audio modal feature representations and visual modal feature representations.

[0032] It should be noted that LSTM is a component in the feature enhancement module used to extract temporal features of audio and visual modalities.

[0033] (1)

[0034] (2) in , , These are the parameters of each network. For text modal feature vectors, The original text data, For audio modal feature vectors, The original audio data, For visual modality feature vectors, This is the raw visual data.

[0035] Optionally, the feature enhancement module uses graph convolutional networks to perform internal feature enhancement on each modality data. It constructs a graph for each modality from all samples in a subset, obtaining text modality graph, audio modality graph and video modality graph respectively. Each sample in each modality graph is a node. Construct an adjacency matrix for the graph, dividing all nodes into two categories. Nodes belonging to the same category have a value of 1 in the adjacency matrix, and otherwise, their value is set to 0. By using graph convolutional networks to aggregate features of nodes in a graph structure, the internal features of multimodal data are enhanced to obtain multimodal enhanced features.

[0036] Specifically, the feature enhancement module constructs a graph from all samples in a batch (subset), where each sample is a node, and the three modalities constitute three graphs respectively. As shown in formula (3), the adjacency matrix of the graph can be constructed based on the K-means clustering method: all nodes are divided into two categories, and nodes belonging to the same category are set to 1 in the adjacency matrix, and vice versa.

[0037] Optionally, the feature enhancement module randomly removes some edges from each modality graph and randomly adds a preset number of cross-class edges to each modality graph.

[0038] Specifically, considering the classification error that K-means clustering may introduce and its impact on the accuracy of the adjacency matrix, as shown in Equation (4), the feature enhancement module optimizes the adjacency matrix. On the one hand, to reduce unreliable connections caused by mis-clustering of noisy samples, the feature enhancement module randomly removes some edges to reduce their interference; on the other hand, K-means may ignore some potential cross-class connections. To enhance the model's ability to model complex structures, the feature enhancement module randomly adds a small number (preset number) of cross-class edges to expand the structural information of the graph. Introducing random perturbation during graph construction helps to alleviate structural bias caused by inaccurate initial clustering assumptions. As shown in Equation (5), after the adjacency matrix is ​​constructed, the feature enhancement module uses a graph convolutional network to perform feature aggregation on the nodes in the graph structure to achieve feature enhancement within the modality.

[0039] (3) (4) (5) in , t Represents the text modality. a Represents audio modality, v Representing the visual modality, U is the optimizer. It is a degree matrix. It is an adjacency matrix. The initial eigenvector matrix of a certain mode. m For modal identification, This is the optimized adjacency matrix.

[0040] The modal multilevel balancing module is described below.

[0041] Optionally, the modal multilevel balancing module utilizes generators and discriminators to construct a generative adversarial network; The generative adversarial network is used to adjust the feature distribution of the non-text modality, thereby achieving preliminary alignment between the non-text modality and the text modality; Using audio or visual modalities as keys and values ​​in the Transformer architecture, and text modalities as queries, attention scores are calculated. Then, the attention scores are normalized using an activation function to obtain an attention weight matrix of the degree of response of non-text modal features to text semantics. Multiplying the attention weight matrix by the value yields a non-textual modal feature representation that incorporates textual semantic information; By employing residual connections, the pre-aligned non-textual modal features are added to the non-textual modal features that have been fused with textual semantic information to obtain attention features; The attention features are input into a feedforward neural network, and the output of the feedforward neural network is again residually connected with the attention features to obtain a non-text modality augmented representation. The lower bound of mutual information between text modal features and non-text modal features is estimated, and the lower bound of mutual information is maximized to obtain the multimodal consistency representation.

[0042] Multimodal data exhibits significant differences in the distribution of emotional information, with text modalities typically carrying richer emotional cues. To mitigate representational biases between non-textual modalities (audio and visual modalities) and textual modalities, a multi-level modality balancing module employs a multi-stage modality balancing mechanism. Different stages play different roles, aiming to gradually narrow the semantic and heterogeneous gaps between modalities and extract cross-modal consistency features. First, an adversarial alignment layer introduces a generative adversarial network to achieve initial alignment of audio and visual modalities towards the textual modality in the representation space. Second, a guidance enhancement layer, using a cross-modal Transformer structure and guided by the textual modality, enhances the contextual modeling and semantic expression capabilities of both audio and visual modalities. Finally, an information constraint layer introduces a mutual information constraint mechanism to further optimize and adjust cross-modal features, enhancing intermodal consistency at the semantic level.

[0043] The adversarial alignment layer achieves initial alignment between non-textual and textual modalities. To this end, the module introduces a generator and a discriminator to construct a generative adversarial network (GAN) to gradually adjust the feature distribution of the non-textual modalities, making them closer to the representation space of the textual modalities. See details... Figure 2 , Figure 2 A schematic diagram of a generative adversarial network provided in an embodiment of this application, such as... Figure 2 As shown, the generator is responsible for mapping non-textual modalities such as audio and visual modalities to representations similar in style to textual modalities; the discriminator is used to determine whether the input features are genuine text representations or forged representations generated by the generator. During training, the generator continuously optimizes its generation strategy to deceive the discriminator, while the discriminator continuously improves its discrimination ability to more accurately distinguish between real text and forged features. The two constitute a dynamic mini-maximum game process; as training iterates, the representational distance between modalities gradually decreases, laying the foundation for subsequent semantic guidance and feature fusion.

[0044] Audio enhancement features and visual enhancement features are input into the text generator to generate pseudo-text features that resemble the style of the original text modality. Then, the generated pseudo-text features are input together with real text features (real samples) into the discriminator, which performs binary classification on each input sample to distinguish its source. In the early stages of training, the discriminator is prone to misclassification due to its limited capabilities. Backpropagation is used to optimize the discriminator and continuously improve its accuracy. Simultaneously, as the discriminator's capabilities increase, the generator faces greater difficulty in deceiving users, requiring further optimization of the generation strategy to enhance the realism of the forged features. This process is repeated between the generator and the discriminator until a Nash equilibrium is reached, where their capabilities are relatively balanced. As shown in equation (6-8), the loss function of the entire adversarial training process consists of two parts: the generator loss and the discriminator loss. and discriminator loss The generator loss measures the ability of generated features to deceive the discriminator, while the discriminator loss measures its accuracy in distinguishing between real and fake features. These two loss components work together to drive continuous model optimization, achieving initial alignment of the modal representation space.

[0045] (6) in This indicates a pseudo-text generation operation. It is an enhanced audio representation. It is an enhanced visual representation. It is a discriminator.

[0046] (7) in It is an enhanced text representation. It is an enhanced audio representation. It is an enhanced visual representation. This indicates a pseudo-text generation operation. It is a discriminator.

[0047] (8) in It is the generator loss. It is discriminator loss.

[0048] It can be seen that the modal multilevel balancing module achieves preliminary spatial alignment between non-textual and textual modalities through generative adversarial networks. Based on this, the module needs to further focus on cross-modal interaction at the semantic level. Considering that textual modalities usually carry more explicit and richer emotional cues, such as... Figure 3 As shown, Figure 3 This application provides a schematic diagram of a text-guided Transformer architecture. Figure 3The architecture shown aims to leverage its global context modeling capabilities to mitigate semantic differences between different modalities through a semantic guidance mechanism, thereby enhancing the expressive power of non-textual modalities at the emotional level.

[0049] Specifically, as shown in formulas (9-10), the modal multilevel balancing module uses the audio or visual modality as the key (K) and value (V), and the text modality as the query (Q). After calculating the attention score, it is normalized using the softmax activation function to obtain an attention weight matrix reflecting the degree of response of non-textual modal features to textual semantics. This matrix is ​​then multiplied by V to obtain a non-textual feature representation that incorporates textual semantic information. To prevent the loss of semantic information during transmission, the modal multilevel balancing module can use residual connections to retain the key information of the original modality, connecting the features before and after the attention mechanism. Next, the connected features are input into a feedforward neural network, which introduces a non-linear activation function to prevent the model from degenerating into a linear mapping, thereby improving the model's ability to model complex semantic structures. The output of the feedforward network is again residually connected to the input to form the enhanced non-textual modality representation of the current layer. Due to the stackable nature of the Transformer structure, the model can gradually capture deeper levels of non-textual modal semantic representation during multiple rounds of semantically guided interaction.

[0050] (9) in , , , These are learnable parameters. It is an activation function. These are enhancement features for non-textual modalities. It is an enhancement feature of the text modality.

[0051] (10) in , Represents a multi-head attention mechanism. It's the number of heads that attract attention. Representative level normalization. It is a feedforward network.

[0052] Furthermore, the consistency and discriminative ability of features are enhanced by maximizing the mutual information between modalities. Although non-textual modalities have gradually converged with textual modalities in the first two stages, there may still be a lack of high-level semantic information or potential non-cooperation between modalities. To address this issue, this stage employs a mutual information neural estimation (MINE) mechanism to model and optimize the mutual information between text and audio, and text and visual modal pairs.

[0053] The core idea of ​​MINE is to estimate and maximize the lower bound of mutual information between two variables by training a discriminative network. Specifically, as shown in equations (11-12), positive samples (true pairs (x,y)) and negative samples (independent pairs (x,y') obtained by shuffling y) are first constructed, where x represents the text modality feature. y represents audio or visual modal features Subsequently, the concatenated sample pairs are input into the discriminative network T(x,y), and its output is used as part of the Donsker–Varadhan expression to estimate the lower bound of mutual information. During training, by maximizing this lower bound, the model tends to assign higher scores to true paired samples while suppressing responses to randomly combined samples, thereby improving the consistency of cross-modal representations and the coupling of mutual information. The mutual information loss is shown in Equation (13).

[0054] (11) (12) (13) in It is a probability distribution. It is a parameterized discriminant network.

[0055] The following section provides a detailed introduction to the modal noise reduction decoupling and specific reconstruction module.

[0056] Optionally, the multimodal augmentation representations can be concatenated to obtain a guidance vector for multimodal global information; Independent modal denoising decomposers are used to suppress noise in each modal representation of the multimodal enhancement representation, and the denoised feature vectors of each modality are obtained. The guiding vector and each modal feature vector are concatenated and a dynamic weighting factor is generated through nonlinear transformation. The dynamic weighting factor is multiplied by each denoised modal feature vector to obtain a specific representation of each modality. The process of obtaining the specific representation is repeated until the preset conditions are met; The specific representation obtained each time in the repeated execution is recombined in the modality bank to obtain a low-noise multimodal specific representation.

[0057] Specifically, multimodal data exhibits significant differences in its inherent characteristics and sentiment expression methods. Modality specificity reflects the unique information of each modality, revealing subtle changes within the modality and possessing unique value for sentiment analysis. However, during the forward and backward propagation of neural networks, noise in the modal information may be gradually amplified and propagated into the deeper structure of the network, thus affecting the accuracy of sentiment analysis tasks. To address this, the modality denoising decoupling and specific reconstruction module can employ a cyclic denoising decoupling framework, aiming to enhance the specific semantic information of single-modal data while suppressing the interference of potential noise on the model.

[0058] As shown in equations (14-16), to avoid bias in representation learning during modality decomposition, the three modality representations are first concatenated to construct a global guidance vector to guide subsequent modality-specific decomposition operations. Then, each modality representation is input into an independent Modality Denoising Decomposer (MDD), specifically an audio modality denoising decomposer, a text modality denoising decomposer, and a visual modality denoising decomposer. Noise is suppressed to remove interfering information. Next, the global guidance vector is concatenated with the denoised modality vector, and a dynamic weighting factor is generated through a nonlinear transformation. This factor is used to adjust the extraction intensity of modality-specific information by the current decomposition layer. The weighting factor is multiplied by the denoised vector to obtain the current modality-specific representation, which is then stored in the representation bank. Considering that a single denoising decomposition cannot completely remove the coupling features between modes, the modal denoising decoupling and specific recombination module adopts a multi-round iterative denoising decomposition method to achieve more accurate feature separation and extract more specific low-noise modal representations from shallow to deep. Finally, as shown in formula (17), the modal features obtained from all decomposition rounds in the representation bank are accumulated and integrated into the final modality-specific representation. This process not only effectively reduces noise interference in modal representation, but also fully explores the unique properties of each modality.

[0059] (14) in , , It is an enhanced text, audio, and visual modality. It's a splicing operation.

[0060] (15) in , It is the global guidance vector. It is a modal noise reduction decomposer.

[0061] (16) After Formula 16 is executed, it returns to Formula 14 and continues to execute until q=3 (preset condition).

[0062] (17) in , It is a modal noise reduction decomposer. It is a modal bank.

[0063] Modality consistency focuses on the commonalities among different modalities, reflecting the unity of text, audio, and visual modalities at the emotional semantic level; while modality specificity focuses on the unique information of each modality, emphasizing the unique expression within a modality. Ideally, these two types of features should be independent of each other and exhibit an orthogonal relationship. To this end, as shown in Equation (18), the modality denoising decoupling and specificity recombination module introduces an orthogonality constraint loss (OCL) to constrain the modality consistency representation and the modality specificity representation, so as to ensure the non-redundancy between modality features.

[0064] (18) in , It is a consistent representation of all modes. It is a specific representation of each modality.

[0065] The following section provides a detailed introduction to the hierarchical fusion prediction module.

[0066] Optionally, the hierarchical fusion prediction module concatenates the consistency representation and the specific representation of the single modality to obtain an initial single modality sentiment representation; A gated weighting factor is generated by linear transformation, and the gated weighting factor is multiplied by the corresponding single-modal sentiment representation to obtain the fusion vectors of text modality, visual modality and audio modality respectively; The fusion vectors of the text modality, the visual modality, and the audio modality are averaged to obtain a single-peak fusion representation; Any pairwise modalities in the initial unimodal sentiment representation are merged to obtain the corresponding bimodal feature vector, and the similarity between the two modalities in the bimodal feature vector is calculated. The fusion weight is obtained based on the gating weighting factor and the similarity between the two modes in the bimodal feature vector; Multiplying the fusion weights by the corresponding bimodal feature vectors yields a bimodal fusion representation; The consistency representation and specific representation of the three modalities are concatenated, and the three-modal fusion weights are extracted through an attention mechanism. The three-modal fusion weights are then scaled and normalized to obtain the attention scores for the corresponding three modalities. The consistency representation and specific representation of the three modalities are stacked and multiplied by the attention score to obtain the three-peak fusion representation; The single-peak fusion representation, the double-peak fusion representation, and the triple-peak fusion representation are concatenated, and the result is predicted using a multilayer perceptron to obtain the sentiment prediction result.

[0067] See details Figure 4 , Figure 4 This is a schematic diagram of a hierarchical fusion prediction module provided in an embodiment of this application, as shown below. Figure 4 As shown, the hierarchical fusion prediction module fuses modality consistency representation and modality specific representation in the manner of modality unimodal, modality bimodal, and modality trimodal. Finally, the fusion result is input into the multilayer perceptron to complete sentiment prediction.

[0068] First, we introduce the single-modal fusion process. The hierarchical fusion prediction module concatenates the consistency representation and the specific representation of a single modality to construct an initial single-modal sentiment representation. Then, a series of linear transformations generate gated weighting factors, which are used to adjust the degree of injection of this sentiment representation during the fusion process. Multiplying the gated weighting factors by the corresponding single-modal sentiment representation yields fusion vectors for the text modality, audio modality, and visual modality, respectively. Finally, averaging the three fusion vectors yields the single-modal fusion representation.

[0069] (19) in , It is a consistent representation of all modes. It is a specific representation of each modality. It is a splicing operation. (20) in , It is a linear layer. It is a single-modal emotion representation. It is an activation function.

[0070] (twenty one) in , , These represent the gating weighting factors for text, audio, and visual modalities, respectively.

[0071] Next, we will introduce the bimodal fusion, as shown in formulas (22-25). The hierarchical fusion prediction module first merges the sentiment representations of any two modalities to obtain the corresponding bimodal feature vectors. Next, to measure the redundancy between modes, the hierarchical fusion prediction module calculates the similarity between each pair of modes. To ensure that the similarity results are in the [0,1] interval, the hierarchical fusion prediction module... Softmax normalization is performed. Based on this, a single-modal representation is further constructed. and With bimodal representation The fusion weight between them. The numerator of this weight represents the mode. m 1 and m The sum of importance of 2, with the similarity between the two in the denominator. Since higher similarity indicates greater information overlap between modalities and less new information generated by interactions, we introduce an adjustment factor θ to mitigate this phenomenon and prevent excessively low weights due to excessively high similarity. This mechanism allows the fusion weights to adjust the redundancy between modalities while considering the importance of the modalities themselves, achieving more reasonable feature interaction control. Finally, the hierarchical fusion prediction module multiplies the fusion weights by the corresponding feature vectors to obtain a bimodal fusion representation.

[0072] (twenty two) in , This indicates a splicing operation.

[0073] (twenty three) in This indicates the transpose operation.

[0074] (twenty four) in These are parameters to avoid excessively high similarity. This indicates the similarity between two modes.

[0075] (25) Finally, the three-modal fusion is introduced. In the three-modal fusion stage, the hierarchical fusion prediction module integrates the consistency information and the specific information of the three modalities. To this end, the hierarchical fusion prediction module adopts a modal fusion method based on an attention mechanism to adaptively integrate multimodal features. Specifically, as shown in formulas (26-27), the input features of the three modalities (i.e., modal consistency and modal specificity representations) are first concatenated, and potential fusion weights are extracted through a multilayer perceptron. Then, the fusion weights are scaled and normalized to transform them into attention scores for the corresponding three modalities, which are used to measure the importance of each modal feature. Next, the three input features are stacked and multiplied by the attention scores to achieve weighted fusion. Finally, the three-modal fused representation is obtained.

[0076] (26) in This represents the attention fusion mechanism.

[0077] (27) in This represents the consistency of the three modes after splicing. This represents the spliced ​​three-modal specificity representation.

[0078] As shown in formulas (28-29), the hierarchical fusion prediction module concatenates the fusion vectors obtained from each layer and completes the prediction through nonlinear transformation. The loss of the prediction task is shown in formula (30).

[0079] (28) in This is a single-peak fusion representation. This is a representation of bimodal fusion. This is a representation of the integration of three peaks. This is a vector concatenation operation.

[0080] (29) in This is the weight matrix. For bias terms, For activation function, This is the final fused feature vector.

[0081] (30) in For sentiment prediction results, To predict losses, For the sample size, For the true sentiment labels of i samples, Let i be the predicted sentiment value for the i-th sample. Its function is to quantify the difference between the model's prediction results and the true labels, providing gradient update directions for model training.

[0082] Secondly, embodiments of this application provide a multimodal sentiment analysis method, which can be executed by an electronic device, specifically by one or more processors within the electronic device, and implements the following steps: S501 extracts features from the original multimodal data using different feature extraction tools to obtain the feature representation of the original multimodal data, constructs a graph of the multimodal representation, and then uses a graph convolutional network to enhance the multimodal data to obtain multimodal enhanced features.

[0083] Optionally, the electronic device obtains multimodal feature representations of the original multimodal data through different feature extraction tools, processes the text modality using a pre-trained model specifically for sentiment analysis to obtain text modal feature representations, extracts audio modal information from the original video data using a speech sentiment feature extraction tool to obtain audio modal feature representations, and extracts visual modal information from the original video data using a facial expression analysis tool to obtain visual modal feature representations.

[0084] Next, the electronic device uses a graph convolutional network to enhance the internal features of each modality data. All samples in a subset are used to construct a graph according to each modality, resulting in text modality graphs, audio modality graphs, and video modality graphs. Each sample in each modality graph is a node. An adjacency matrix of the graph is constructed, and all nodes are divided into two categories. Nodes belonging to the same category have a value of 1 in the adjacency matrix, and otherwise, the value is set to 0. The graph convolutional network is used to aggregate features of the nodes in the graph structure to enhance the internal features of the multimodal data and obtain multimodal enhanced features.

[0085] S502 utilizes different network structures at each stage to process multimodal enhancement features, thereby obtaining a multimodal consistent representation.

[0086] The electronic device utilizes the adversarial network to adjust the feature distribution of the non-text modality, achieving initial alignment between the non-text modality and the text modality. Using the audio or visual modality as keys and values ​​in the Transformer architecture, and the text modality as the query, an attention score is calculated. This attention score is then normalized using an activation function to obtain an attention weight matrix representing the degree of non-text modality feature response to text semantics. The attention weight matrix is ​​multiplied by the value to obtain a non-text modality feature representation incorporating text semantic information. A residual connection is used to add the initially aligned non-text modality features to the non-text modality features incorporating text semantic information, resulting in attention features. These attention features are then input into a feedforward neural network, and the output of the feedforward neural network is again residually connected to the attention features to obtain an enhanced non-text modality representation. Finally, the lower bound of the mutual information between the text modality features and the non-text modality features is estimated, and this lower bound is maximized to obtain a multimodal consistency representation.

[0087] S503, based on multimodal global information, decomposes multimodal enhancement features through a modal denoising decomposer to obtain low-noise multimodal representations, and reassembles the low-noise multimodal representations in a modal bank to obtain low-noise multimodal specific representations.

[0088] The electronic device concatenates the multimodal augmentation representations to obtain a guidance vector of multimodal global information. The electronic device then uses an independent modal denoising decomposer to suppress noise in each modal representation of the multimodal augmentation representation, obtaining denoised modal feature vectors. The guidance vector and each modal feature vector are concatenated, and a dynamic weighting factor is generated through a nonlinear transformation. This dynamic weighting factor is multiplied by each denoised modal feature vector to obtain a specific representation for each modality. The process of obtaining specific representations is repeated until a preset condition is met. The specific representations obtained in each iteration are recombined in a modality bank to obtain a low-noise multimodal specific representation.

[0089] S504 processes and fuses multimodal consistency representation and low-noise multimodal specific representation in the manner of modal unimodal, modal bimodal, and modal trimodal methods, and uses a multilayer perceptron to predict the processing and fusion results to obtain sentiment prediction results.

[0090] In this embodiment, the electronic device can concatenate the consistent representation and the specific representation of a single modality to obtain an initial single-modal sentiment representation; generate a gated weighting factor through linear transformation, and multiply the gated weighting factor with the corresponding single-modal sentiment representation to obtain fusion vectors for the text modality, visual modality, and audio modality, respectively; perform an averaging operation on the fusion vectors of the text modality, visual modality, and audio modality to obtain a single-peak fusion representation; merge any pairwise modalities in the initial single-modal sentiment representation to obtain the corresponding bimodal feature vector, and calculate the similarity between the two modalities in the bimodal feature vector; based on the gated factor and the bimodal feature vector... The similarity between two modalities in the data is used to obtain the fusion weight. The fusion weight is multiplied by the corresponding bimodal feature vector to obtain the bimodal fusion representation. The consistency representation and the specific representation of the three modalities are concatenated, and the trimodal fusion weight is extracted through an attention mechanism. The trimodal fusion weight is scaled and normalized to obtain the attention scores for the corresponding three modalities. The consistency representation and the specific representation of the three modalities are stacked and multiplied by the attention scores to obtain the trimodal fusion representation. The unimodal fusion representation, bimodal fusion representation and trimodal fusion representation are concatenated, and the result is predicted through a multilayer perceptron to obtain the sentiment prediction result.

[0091] To verify the performance of the embodiments of this application in multimodal sentiment analysis, simulation verification was also performed on the multimodal sentiment analysis model provided in this embodiment. The details are as follows: Experimental validation was performed on three benchmark datasets, including two English datasets, CMU-MOSI

[41] and CMU-MOSEI

[42] , and a Chinese dataset, CH-SIMS

[43] . Table 1 summarizes the partitioning of the three datasets.

[0092] The CMU-MOSI dataset consists of 2199 video clips from YouTube movie reviews, covering text, visual, and audio modalities. Each clip is labeled with a sentiment value in the interval [-3,3] and provides rigorous annotation information, including subjective labels, sentiment intensity labels, frame-by-frame and point-by-point visual features, and millisecond-by-millisecond audio features.

[0093] The CMU-MOSEI dataset is an expanded version of CMU-MOSI, containing 23,453 annotated video clips from 1,000 different speakers, covering 250 topics. Each clip is labeled with emotion on a Likert scale of [-3,3], and 7 emotion categories and 6 basic emotion labels are provided.

[0094] The CH-SIMS dataset is designed specifically for Chinese contexts, collecting 60 original videos from movies, TV series, and variety shows, and organizing them into 2281 sentence-level video clips. The sentiment intensity of each clip is labeled in the range [-1,1]. This dataset provides independent annotations for both multimodal and unimodal modes.

[0095] Table 1

[0096] The multimodal sentiment analysis method provided in this embodiment is compared with the following baseline model.

[0097] TFN achieves high-order interaction and fusion between modes by calculating the tensor product of single-mode, dual-mode, and tri-mode modes.

[0098] MulT utilizes a bidirectional cross-modal attention mechanism to align heterogeneous features between modalities and models the dependencies between modalities.

[0099] MISA maps modal representations to modality-invariant and modality-specific spaces, learning common and individual modal information respectively.

[0100] Self-MM utilizes a label generation strategy to guide modal representation learning, combining multimodal and unimodal task training to capture consistency and difference information.

[0101] MTSA alleviates the problem of lack of emotional information in non-textual modalities by translating non-textual modalities into textual modal features.

[0102] TETFN uses a text-driven attention mechanism to fuse textual and non-textual information to obtain a unified multimodal representation.

[0103] EMT enables cross-modal interaction of unaligned data under the guidance of global multimodal information and improves robustness in scenarios with missing modalities.

[0104] DTN utilizes generative adversarial networks to achieve two-stage modality transformation and distribution matching, thereby capturing core information in the data.

[0105] TIEMFF combines attention mechanisms and an expanded sentiment lexicon to enhance sentiment associations between modalities and refine feature representations.

[0106] SIMSUF supplements other modes with the dominant mode and designs a bi-branch mode enhancement strategy to obtain an effective multimodal representation.

[0107] MCL-MCF reduces the differences between modalities through contrastive learning and strengthens high-level semantic features by combining convolutional fusion, thus alleviating the problem of data imbalance.

[0108] MIG-HCL effectively captures complex intermodal interactions and suppresses redundancy through bidirectional feature enhancement and mutual information selection mechanisms, while achieving robust representation learning even in the case of modality loss.

[0109] The comparison results are shown in Table 2, which presents the experimental results on the CMU-MOSI dataset. The model provided in this embodiment of the application is compared with the baseline mentioned above. Table 2

[0110] Table 3 presents the experimental results on the CMU-MOSEI dataset. It can be seen that the model in this embodiment exhibits certain advantages in all metrics. Specifically, compared to the suboptimal model, MAE decreased by 0.003, and Corr increased by 0.004; under the "negative / non-negative" evaluation system, Acc-2 and F1 scores increased by 0.38% and 0.57%, respectively; under the "negative / positive" evaluation system, Acc-2 and F1 scores increased by 0.14% and 0.31%, respectively.

[0111] Table 3

[0112] Table 4 presents the experimental results on the CH-SIMS dataset. Compared with the suboptimal model EMT, the model provided in this embodiment improves the Acc-2 metric by 0.86%, the F1-Score by 1.49%, and the Corr by 0.001, while also demonstrating competitiveness in other metrics.

[0113] Table 4

[0114] In summary, compared with the prior art, this embodiment has the following advantages: 1. Through a multi-level modal balancing module, the audio and visual modalities are brought closer to the text modal in terms of distribution and semantics through the progressive action of adversarial alignment, text-guided Transformer, and mutual information constraints. This design avoids the limitation of traditional methods that only focus on surface feature alignment while neglecting deep semantic consistency, and effectively reduces the difference between structured and unstructured information between modalities.

[0115] 2. Utilizing a modal denoising decoupling and specific recombination module, modal noise is weakened through multiple rounds of denoising decomposition, and deep specific information is mined by combining global guidance vectors. Simultaneously, orthogonal constraints are used to separate modal consistency representations from specific representations, thereby avoiding information redundancy. This module solves the problems of excessive modal feature noise and the submergence of important semantics by redundant information in traditional methods.

[0116] 3. A hierarchical fusion prediction module is constructed, which processes data according to a progressive path of single-modal fusion, bimodal fusion, and trimodal fusion. Single-modal fusion focuses on integrating the consistency and specificity information of a single modality; bimodal fusion splices two modalities together and dynamically evaluates the contribution of modality pairs based on similarity; trimodal fusion adaptively fuses the consistency and specificity information of the three modalities through an attention mechanism. This design avoids the limitations of traditional fusion methods that tend to ignore local modal cues or over-rely on alignment, thereby fully exploring the sentiment associations between modalities.

[0117] The third embodiment of the present invention also provides an electronic device, the electronic device comprising: At least one processor; and, The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the multimodal sentiment analysis method of any of the foregoing embodiments.

[0118] The fourth embodiment of the present invention also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the multimodal sentiment analysis method described in any of the foregoing embodiments.

[0119] The fifth embodiment of the present invention also provides a computer program product, which includes a computing program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions that, when executed by a computer, cause the computer to perform the multimodal sentiment analysis method of any of the foregoing embodiments.

[0120] The sixth embodiment of the present invention also provides a computer program, which includes program instructions that, when executed by a computer, cause the computer to perform the multimodal sentiment analysis method of any of the foregoing embodiments.

[0121] Figure 6 The diagram illustrates a method or device 1000 that can implement embodiments of the present invention. In some embodiments, it may include more or fewer devices than illustrated. In some embodiments, it may be implemented using a single or multiple devices. In some embodiments, it may be implemented using cloud-based or distributed devices.

[0122] like Figure 6As shown, device 1000 includes a processor 1001, which can perform various appropriate operations and processes based on programs and / or data stored in read-only memory (ROM) 1002 or programs and / or data loaded from storage portion 1008 into random access memory (RAM) 1003. Processor 1001 may be a multi-core processor or may contain multiple processors. In some embodiments, processor 1001 may include a general-purpose main processor and one or more special coprocessors, such as a central processing unit (CPU), graphics processing unit (GPU), neural network processor (NPU), digital signal processor (DSP), etc. Various programs and data required for the operation of device 1000 are also stored in RAM 1003. Processor 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0123] The processor and memory described above are used together to execute programs stored in the memory. When the program is executed by a computer, it can implement the methods, steps, or functions described in the above embodiments.

[0124] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, touchscreen, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 1010 as needed so that computer programs read from it can be installed into storage section 1008 as needed. Figure 6 The diagram only shows a portion of the components and does not imply that the device 1000 only includes... Figure 6 The components shown.

[0125] The systems, devices, modules, or units described in the above embodiments can be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, smartphone, personal computer, laptop computer, in-vehicle human-machine interface device, personal digital assistant, media player, navigation device, game console, tablet computer, wearable device, smart TV, Internet of Things system, smart home, industrial computer, server, or a combination thereof.

[0126] Although not shown, in this embodiment of the invention, a computer-readable storage medium is provided having a computer program / instructions stored thereon, which, when executed by a processor, implements the multimodal sentiment analysis method described in Embodiment 1.

[0127] Storage media in embodiments of the present invention include articles that are permanent and non-permanent, removable and non-removable, capable of storing information by any method or technology. Examples of storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0128] Although not shown, embodiments of the present invention also provide a computer program product, including: a computer program / instructions that, when executed by a processor, implement the multimodal sentiment analysis method described in Embodiment 1.

[0129] The methods, programs, systems, apparatuses, etc., in embodiments of the present invention can be executed or implemented in one or more networked computers, or practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks can be performed by remote processing devices connected via a communication network.

[0130] Those skilled in the art will understand that the embodiments described in this specification can be provided as methods, systems, or computer program products. Therefore, those skilled in the art will realize that the functional modules / units or controllers and related method steps described in the above embodiments can be implemented in software, hardware, or a combination of both.

[0131] Unless explicitly stated otherwise, the actions or steps of the methods and procedures described in the embodiments of the present invention do not necessarily have to be performed in a specific order and can still achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0132] This document describes several embodiments of the present invention; however, for the sake of brevity, the descriptions of the embodiments are not exhaustive, and identical or similar features or parts between the embodiments may be omitted. In this document, "one embodiment," "some embodiments," "example," "specific example," or "some examples" refers to embodiments applicable to at least one, but not all, of the present invention. The above terms do not necessarily refer to the same embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of the different embodiments or examples.

[0133] The exemplary systems and methods of the present invention have been specifically shown and described with reference to the above embodiments, which are merely examples of the best mode for implementing the systems and methods. Those skilled in the art will understand that various changes can be made to the embodiments of the systems and methods described herein without departing from the spirit and scope of the invention as defined in the appended claims when implementing the systems and / or methods.

Claims

1. A multimodal sentiment analysis model, characterized in that, include: The feature enhancement module is used to obtain multimodal feature representations of the original multimodal data through different feature extraction tools, construct graphs of the multimodal feature representations, and then enhance the multimodal data using a graph convolutional network to obtain multimodal enhanced features. A modal multi-level balancing module is used to process the multimodal enhancement features using different network structures at each stage to obtain a multimodal consistent representation; The modal denoising decoupling and specific recombination module is used to decompose the multimodal enhancement features according to the multimodal global information through the modal denoising decomposer to obtain a low-noise multimodal representation, and to reconstruct the low-noise multimodal representation in the modal bank to obtain a low-noise multimodal specific representation. The hierarchical fusion prediction module is used to process and fuse the input multimodal consistency representation and the input low-noise multimodal specific representation in the manner of modal unimodal, modal bimodal, and modal trimodal, and to predict the result of the processing and fusion through a multilayer perceptron to obtain the sentiment prediction result.

2. The multimodal sentiment analysis model according to claim 1, characterized in that, The feature enhancement module obtains multimodal feature representations of the original multimodal data through different feature extraction tools, including: The original text data is processed using a dedicated pre-trained model to obtain text modal feature representations; The feature enhancement module uses a speech emotion feature extraction tool to extract audio modal information from the original video data to obtain an audio modal feature representation; The feature enhancement module uses facial expression analysis tools to extract visual modal information from the original video data to obtain visual modal feature representations.

3. The multimodal sentiment analysis model according to claim 1, characterized in that, The process of constructing a graph from the multimodal representation and then using a graph convolutional network to enhance the multimodal data to obtain multimodal enhanced features includes: All samples in a subset are plotted into a graph according to each modality, resulting in text modality graph, audio modality graph, and video modality graph, where each sample in each modality graph is a node; Construct an adjacency matrix for the graph, dividing all nodes into two categories. Nodes belonging to the same category have a value of 1 in the adjacency matrix, and otherwise, their value is set to 0. By using graph convolutional networks to aggregate features of nodes in a graph structure, the internal features of the multimodal data are enhanced to obtain multimodal enhanced features.

4. The multimodal sentiment analysis model according to claim 3, characterized in that, The feature enhancement module randomly removes some edges from each modal graph and randomly adds a preset number of cross-class edges to each modal graph.

5. The multimodal sentiment analysis model according to claim 1, characterized in that, The modal multi-level balancing module processes the multimodal enhancement features using different network structures at each stage to obtain a multimodal consistent representation, including: Generative adversarial networks are constructed using generators and discriminators; The generative adversarial network is used to adjust the feature distribution of the non-text modality, thereby achieving preliminary alignment between the non-text modality and the text modality; Using audio or visual modalities as keys and values ​​in the Transformer architecture, and text modalities as queries, attention scores are calculated. Then, the attention scores are normalized using an activation function to obtain an attention weight matrix of the degree of response of non-text modal features to text semantics. Multiplying the attention weight matrix by the value yields a non-textual modal feature representation that incorporates textual semantic information; By employing residual connections, the pre-aligned non-textual modal features are added to the non-textual modal features that have been fused with textual semantic information to obtain attention features; The attention features are input into the feedforward neural network, and the output of the feedforward neural network is again residually connected with the attention features to obtain a non-text modality augmented representation. The lower bound of mutual information between the text modal features and the non-text modal features is estimated, and the lower bound of mutual information is maximized to obtain the multimodal consistency representation.

6. The multimodal sentiment analysis model according to claim 1, characterized in that, The modal denoising decoupling and specific recombination module, based on global information, decomposes the enhanced features using a modal denoising decomposer to obtain a low-noise multimodal representation, and reassembles the low-noise multimodal representation in a modal bank to obtain a low-noise multimodal specific representation, including: The multimodal enhancement features are concatenated to obtain a guidance vector for multimodal global information; Independent modal denoising decomposers are used to suppress noise in each modal representation of the multimodal enhancement features to obtain the denoised modal feature vectors. The guiding vector and each modal feature vector are concatenated and a dynamic weighting factor is generated through nonlinear transformation. The dynamic weighting factor is multiplied by each denoised modal feature vector to obtain a modality-specific representation. Repeat the process of obtaining the modality-specific representations until the preset conditions are met; The modality-specific representations obtained each time in the repeated execution are recombined in the modality bank to obtain low-noise multimodality-specific representations.

7. The multimodal sentiment analysis model according to claim 6, characterized in that, The modal denoising decoupling and specific recombination module uses orthogonal constraint loss to constrain modal consistency representation and modal specificity representation.

8. The multimodal sentiment analysis model according to claim 1, characterized in that, The hierarchical fusion prediction module processes and fuses the input multimodal consistency representation and the input low-noise multimodal specific representation in a modal unimodal, modal bimodal, and modal trimodal manner, and uses a multilayer perceptron to predict the processing and fusion results to obtain sentiment prediction results, including: The initial unimodal sentiment representation is obtained by concatenating the consistency representation of the unimodality with the specific representation of the unimodality. A gated weighting factor is generated by linear transformation, and the gated weighting factor is multiplied by the corresponding single-modal sentiment representation to obtain the fusion vectors of text modality, visual modality and audio modality respectively; The fusion vectors of the text modality, the visual modality, and the audio modality are averaged to obtain a single-peak fusion representation; Any pairwise modalities in the initial unimodal sentiment representation are merged to obtain the corresponding bimodal feature vector, and the similarity between the two modalities in the bimodal feature vector is calculated. The fusion weight is obtained based on the gating weighting factor and the similarity between the two modes in the bimodal feature vector; Multiplying the fusion weights by the corresponding bimodal feature vectors yields a bimodal fusion representation; The consistency representation and specific representation of the three modalities are concatenated, and the three-modal fusion weights are extracted through an attention mechanism. The three-modal fusion weights are then scaled and normalized to obtain the attention scores for the corresponding three modalities. The consistency representation and specific representation of the three modalities are stacked and multiplied by the attention score to obtain the three-peak fusion representation; The single-peak fusion representation, the double-peak fusion representation, and the triple-peak fusion representation are concatenated, and the result is predicted using a multilayer perceptron to obtain the sentiment prediction result.

9. The multimodal sentiment analysis method according to claim 1, characterized in that, The multimodal sentiment analysis method uses the multimodal sentiment analysis model described in any one of claims 1-8 to perform sentiment analysis, wherein the multimodal sentiment analysis includes: The original multimodal data is subjected to feature extraction using different feature extraction tools to obtain a multimodal representation of the original multimodal data. A graph is constructed on the multimodal representation, and then a graph convolutional network is used to enhance the multimodal data to obtain multimodal enhanced features. The multimodal enhancement features are processed using different network structures at each stage to obtain a multimodal consistent representation; Based on multimodal global information, the multimodal enhancement features are decomposed by a modal denoising decomposer to obtain a low-noise multimodal representation, and the low-noise multimodal representation is reorganized in a modal bank to obtain a low-noise multimodal specific representation. The multimodal consistency representation and the low-noise multimodal specific representation are processed and fused in the manner of modal unimodal, modal bimodal, and modal trimodal, and the processing and fusion results are predicted by a multilayer perceptron to obtain the sentiment prediction result.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the multimodal sentiment analysis method of claim 9.

Images