Multi-modal sentiment recognition method based on uncertainty probability modeling and expert product

Abstract

This invention discloses a multimodal emotion recognition method based on uncertainty probability modeling and expert product, comprising: acquiring original signal sequences of multiple modalities and extracting features from the original signal sequences of each modality to obtain initial features for each modality; encoding the initial features for each modality to obtain features for each modality; performing uncertainty-aware probability modeling on the features for each modality to obtain a latent distribution for each modality, wherein the latent distribution for each modality is a Gaussian distribution, wherein the mean of the Gaussian distribution represents the semantic content of the modality, and the variance characterizes the uncertainty of the modality; employing an expert product fusion method to fuse the latent distributions of multiple modalities to obtain a fused latent distribution, wherein the fused latent distribution has a fused mean and a fused variance; and mapping the fused mean to obtain the emotion prediction result. This invention can achieve more accurate and reliable multimodal emotion recognition.

Description

Technical Field

[0001] This invention belongs to the field of multimodal emotion recognition technology, specifically involving a multimodal emotion recognition method based on uncertainty probability modeling and expert product. Background Technology

[0002] Multimodal Emotion Recognition (MER) technology aims to identify and judge human emotional states by jointly analyzing information from multiple modalities. In this technology, emotional information typically originates from multiple modalities, including language content, speech signals, and visual information, each reflecting an individual's emotional expression characteristics from different perspectives. In the basic setup of MER, the system simultaneously receives data input from multiple modalities and models and fuses the information from different modalities to form a comprehensive judgment of the target's emotional state. Compared to emotion recognition methods that rely solely on a single modality, MER can utilize complementary information between different modalities, improving the comprehensiveness of emotion analysis. This technology has been widely applied in human-computer interaction, emotion analysis, and intelligent perception, serving as an important technical means for automatically understanding and analyzing human emotional states.

[0003] Current research primarily focuses on multimodal fusion and cross-modal representation learning. Early studies integrated features from different modalities to form a unified multimodal representation. Based on the fusion stage, existing schemes are generally categorized into early fusion and late fusion. Early fusion methods directly concatenate or jointly model multimodal features at the input or feature layer, but these methods have limitations when handling complex intra-modal temporal dynamics. Late fusion methods first model each individual modality separately and output prediction results, then fuse these prediction results. These approaches reduce modeling difficulty to some extent but limit deep information interaction between modalities. To alleviate the imbalance in information content and reliability among different modalities, some studies have proposed techniques based on dynamic weighting or expert models, such as EMOE, which adaptively adjusts the contribution of each modality to the emotion recognition result through a dynamic expert mechanism. Cross-modal representation learning aims to obtain richer multimodal embeddings by exchanging information between different modalities, such as progressive alignment, MulT, MFN, RAVEN, EMT, and decoupling methods DMD, DLF, and SPMN.

[0004] Multimodal emotion recognition technology under modality missing conditions aims to analyze and judge the emotional state of a target object when some modalities are unavailable or unusable in multimodal data. In such applications, multimodal emotion recognition systems typically use multiple modalities such as language, speech, and vision as input. However, during actual data acquisition and transmission, some modalities may be missing or degraded in quality due to equipment limitations, environmental interference, or incomplete data collection. In multimodal emotion recognition settings under modality missing conditions, the emotion recognition model can only utilize the available modal information to complete emotion modeling and prediction during the training or inference phases. This technology focuses on achieving effective inference of emotional states without relying on complete modal input, while maintaining the stable operation of the multimodal system under different modality availability conditions.

[0005] To address the problem of missing modalities, existing technologies have proposed various solutions. Some methods supplement missing modal information through cross-modal generation or mapping mechanisms. For example, MCTN uses a recurrent translation mechanism to convert information between different modalities, thereby reconstructing the missing modality. Another type of method directly generates feature representations of the missing modality through cross-modal imagination or inference mechanisms. For instance, MMIN infers missing modal information through a cross-modal imagination network, while IF-MMIN further improves the robustness of the model in modality-missing scenarios by extracting cross-modal invariant features. In addition, some technical solutions attempt to alleviate the problem of missing modalities through a unified representation space or structured modeling. For example, MRAN maps multimodal features to a text space for unified modeling, and GCNet uses a graph-based approach to characterize the relationships between multimodalities.

[0006] Probabilistic representation techniques aim to model data in the form of random variables to describe the distribution characteristics and uncertainties of the data in the feature space. In this type of technique, individual data samples are no longer represented as fixed, deterministic vectors, but rather characterized as probability distributions, such as using a Gaussian distribution to model the central location and dispersion of the samples. By introducing probability distributions, probabilistic representation techniques can simultaneously express the main characteristics of a sample and its possible range of variation, thus providing a richer information foundation for data similarity measurement, matching, and inference. This technique is often used in conjunction with probabilistic matching or distance metrics to compare and analyze different samples at the distribution level.

[0007] Existing probabilistic representation techniques typically achieve this by learning probabilistic embeddings, representing samples using distributional forms rather than deterministic vectors. This approach can characterize the variability of data at the representational level and can be used to evaluate the reliability of input data. Related theoretical research shows that, under the condition of introducing an appropriate probability matching loss function, probabilistic representation learning can effectively recover random uncertainty, theoretically verifying the rationality of probability mapping in characterizing input ambiguity. Currently, probabilistic embedding techniques have been widely applied in various tasks with ambiguity or high variability, including word embedding, image representation, face understanding, 2D-to-3D pose estimation, and video understanding.

[0008] In summary, existing multimodal emotion recognition technologies have the following drawbacks: (1) Difficulty in effectively expressing emotional ambiguity, limiting the accuracy of recognition results: Existing technologies usually compress emotional information into a single deterministic representation, ignoring the inherent ambiguity and contextual dependence in emotional expression, which makes it impossible for the model to retain multiple reasonable emotional interpretations at the same time. In complex or ambiguous situations, it is easy to make misjudgments, affecting the accuracy of emotion recognition.

[0009] (2) Multimodal fusion is sensitive to noise and conflict information, and lacks stability and robustness: Existing multimodal emotion recognition methods fail to distinguish the reliability of different modal emotion cues during the fusion process. When a certain modality has noise, conflict or unclear expression, it may still cause a large interference to the fusion result, resulting in large fluctuations in the model prediction results and difficulty in working stably in real scenarios.

[0010] (3) Performance degrades significantly and applicability is limited when modality is missing or information is incomplete: In practical applications, multimodal data often suffers from modality loss due to equipment failure, occlusion or privacy restrictions. However, existing technologies mostly rely on heuristic processing methods such as zero padding or reconstruction, lacking a unified and effective processing mechanism, which leads to a significant decline in emotion recognition performance and limits the applicability of the system. Summary of the Invention To address the aforementioned problems in existing technologies, this invention provides a multimodal emotion recognition method based on uncertainty probability modeling and expert product.

[0011] The technical problem to be solved by this invention is achieved through the following technical solution: This invention provides a multimodal emotion recognition method based on uncertainty probability modeling and expert product, comprising: S1. Obtain the original signal sequences of multiple modes, and extract the features of the original signal sequences of each mode to obtain the initial features of each mode; S2. Encode the initial features of each modality to obtain the features of each modality; S3. Perform uncertainty-aware probability modeling on the features of each modality to obtain the latent distribution of each modality. The latent distribution of each modality is a Gaussian distribution, where the mean of the Gaussian distribution represents the semantic content of the modality, and the variance characterizes the uncertainty of the modality. S4. Using an expert product fusion method, the potential distributions of the multiple modalities are fused to obtain a fused potential distribution, wherein the fused potential distribution has a fused mean and a fused variance; S5. The fused mean is mapped to obtain the sentiment prediction result.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) Improved the expressive power of emotion representation. This invention uses Uncertainty-aware Probabilistic Modeling (UPM) to represent each modality as a Gaussian distribution, where the mean encodes semantic content and the variance characterizes modality uncertainty, thereby naturally capturing a variety of reasonable emotional interpretations.

[0013] (2) Enhance the stability and robustness of multimodal fusion. This invention adopts the Uncertainty-aware Probabilistic Fusion (UPF) and Product-of-Experts (PoE) strategies to fuse the potential distributions of each modality according to uncertainty weights. High-confidence modalities contribute more, while noisy modalities automatically have reduced weights, and uncertain but informational modal signals are retained. The fused multimodal representation is more stable and robust, and can effectively resist the interference of noise and conflicting modalities, thereby improving the reliability of multimodal emotion recognition.

[0014] (3) Preserve the integrity of multimodal information and prevent low-confidence modalities from being ignored: This invention introduces Modal Consistency Loss (MCL), which minimizes the KL divergence between the fused latent distribution and the latent distribution of each single modality, so that all modalities can make effective contributions during the learning process; the fused multimodal representation can retain useful information from uncertain modalities, and improve the model's ability to perceive fine-grained emotional cues.

[0015] (4) Improved applicability under conditions of missing or incomplete modalities. The PoE fusion strategy of the present invention can perform joint distribution calculation using only available modalities, and missing modalities are naturally excluded without the need for additional completion or retraining, so that reasonable prediction performance can still be maintained under missing conditions, thereby enhancing the applicability and stability of the system in real-world scenarios.

[0016] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the multimodal emotion recognition method based on uncertainty probability modeling and expert product provided in this embodiment of the invention. Figure 2 This is a schematic diagram of the overall framework of the multimodal emotion recognition model provided in this embodiment of the invention; Figure 3 This is a schematic diagram of unimodal emotional ambiguity and different representation paradigms provided in the embodiments of the present invention. Detailed Implementation

[0018] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0019] This invention proposes a unified uncertainty-aware probabilistic modeling and product-of-experts fusion method, UPOE (Uncertainty-aware Probabilistic Modeling and Product-of-Experts Fusion), for multimodal emotion recognition (MER). This method first encodes each modality as a Gaussian distribution using uncertainty-aware probabilistic modeling, thereby capturing semantic content and modality-specific uncertainty. Then, through an expert product strategy, the resulting distribution is aggregated using uncertainty-aware probabilistic fusion, emphasizing consistency between modalities while reducing the weight of noise or unreliable signals. Finally, a modality consistency loss aligns the fused posterior distribution with the single-modal distribution, preserving information cues from lower-confidence modalities and ensuring balanced contributions from each modality. Due to the use of the PoE fusion strategy, it can effectively aggregate information across modalities and naturally support reasoning in scenarios with missing modalities. The task studied in this invention is multimodal emotion recognition, and for this task, the following three core issues need to be addressed: (1) The lack of modeling for the uncertainty and inherent ambiguity of emotional expression: In the process of multimodal emotion recognition, emotional signals of single modalities such as text, vision and speech often have obvious ambiguity and context dependence. The same modal input may correspond to multiple reasonable emotional states in different contexts. However, existing methods usually use deterministic feature representation, which makes it difficult to characterize this one-to-many or many-to-many emotional mapping relationship, resulting in insufficient expression of emotional information. Therefore, the first technical problem to be solved by this invention is how to effectively model the inherent ambiguity in single-modal emotional expression in order to improve the expressive power of emotional representation.

[0020] (2) The problem of insufficient stability of fusion results due to the failure to characterize the uncertainty differences of multimodal emotions: There are significant differences in the reliability and clarity of different modalities in emotion expression. However, existing multimodal emotion recognition methods generally ignore the uncertainty differences of each modality's emotion cues during feature fusion, making them susceptible to interference from noisy modalities or conflicting information, thus affecting the stability of the fused representation and the accuracy of emotion recognition. Therefore, the second technical problem to be solved by this invention is how to characterize and utilize the uncertainty differences between different modal emotion cues to improve the stability and robustness of multimodal emotion fusion.

[0021] (3) The problem of insufficient sentiment inference capability in scenarios with missing modalities or incomplete information: In practical applications, multimodal sentiment data often suffers from missing modalities or incomplete information due to sensor failure, occlusion, or privacy restrictions. Existing methods usually rely on heuristic completion strategies and lack a unified modeling mechanism, resulting in a significant decrease in the sentiment prediction performance of the model under conditions of missing modalities. Therefore, the third technical problem to be solved by this invention is how to achieve stable and reasonable multimodal sentiment inference under conditions of missing modalities or incomplete information.

[0022] In some embodiments, Figure 1 This is a flowchart illustrating a multimodal emotion recognition method based on uncertainty probability modeling and expert product provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes: S1. Obtain the original signal sequences of multiple modes, and extract the features of the original signal sequences of each mode to obtain the initial features of each mode.

[0023] Multiple modalities include: language modality, visual modality, and acoustic modality, denoted as follows: Multimodal emotion recognition aims to identify emotions using these multimodal signals. The original signal sequence of multiple modalities refers to the signal sequence composed of the original signals of the linguistic, visual, and acoustic modalities of the same object within the same time period. In this invention, for the three original signal sequences of the linguistic, visual, and acoustic modalities of the same object, corresponding existing feature extraction methods can be used to extract the initial features of each modality. For example, for the linguistic modality, a pre-trained BERT model can be used to extract linguistic features and obtain hidden states as word embeddings; for the visual modality, the Facet tool can be used to provide visual features of facial action units; and for the acoustic modality, the COVAREP tool can be used to provide acoustic features. This invention does not limit the specific feature extraction method.

[0024] S2. Encode the initial features of each modality to obtain the features of each modality.

[0025] Here, for each mode This invention employs an encoder corresponding to this mode to perform feature encoding on the initial features of each mode. Each mode The features are represented as , , Representing each mode initial characteristics, Representing each mode Features Representing each mode The corresponding neural network encoder, It is the dimension for extracting features.

[0026] S3. Perform uncertainty-aware probability modeling on the features of each modality to obtain the latent distribution of each modality. The latent distribution of each modality is a Gaussian distribution, where the mean of the Gaussian distribution represents the semantic content of the modality, and the variance characterizes the uncertainty of the modality.

[0027] Existing multimodal emotion recognition methods typically employ a deterministic modeling paradigm, representing each unimodal emotion embedding as a point in a latent space. This approach implicitly assumes a one-to-one correspondence between modalities and the emotions they convey, oversimplifying the inherent complexity and subtlety of emotional expression in real-world scenarios. In reality, the relationship between modalities and emotions is inherently uncertain and probabilistic: a single modality may convey multiple plausible emotional interpretations, while different modalities may express the same emotion in different ways. This leads to one-to-many and many-to-many mappings between modality features and emotion labels. Therefore, deterministic point embeddings are insufficient to capture this inherent ambiguity, limiting the model's ability to represent intermodal variability and subtle interactions. To address these limitations, this invention proposes a probabilistic modeling framework where each modality is represented as a probability distribution rather than a fixed vector. In this way, the model can naturally encode the semantic content and uncertainty of each modality, thus flexibly capturing multiple plausible emotional expressions. This probabilistic formulation provides richer and more expressive multimodal embeddings that better reflect the inherent variability and complexity of emotional signals in real-world scenarios.

[0028] Here, for each mode By using uncertainty-aware probability modeling, Parameters mapped to a Gaussian distribution in the shared latent space. Specifically, for each mode... First, a lightweight MLP is used to learn a simple mapping from modal features to parameters; then, a simple linear layer is applied to predict the mean and element-wise variance of the latent distribution. , , , , and They represent 3D real vector space and A non-negative real vector space and It is each mode Neural networks, i.e., each mode Uncertainty-aware probability modeling module Representing each mode The semantic content, and Representing each mode The uncertainty. and Each generated mode The latent distribution is defined as: ,in, Indicates a normal distribution. Represents a diagonal matrix. ,and, This represents the latent variable. This probabilistic formulation allows the model to separate semantic information from reliability, where the mean encodes the mode. The variance reflects the confidence level.

[0029] S4. Using the expert product fusion method, the potential distributions of multiple modalities are fused to obtain the fused potential distribution, which has a fused mean and a fused variance.

[0030] This invention employs an uncertainty-aware probabilistic fusion strategy to integrate the latent distributions of specific modalities into a unified multimodal representation. Within this framework, each modality contributes a Gaussian expert distribution, which encodes semantic content and its associated uncertainty. The expert product (PoE) multiplicatively aggregates the distributions of all available modalities, effectively weighting them according to the confidence level of each modality: modalities with lower uncertainty contribute more, while noisier or less reliable modalities are naturally weighted less. This ensures that the fused representation highlights reliable signals while mitigating the impact of ambiguous or inconsistent inputs.

[0031] A key advantage of the PoE formula lies in its flexibility in handling incomplete or variable modal inputs. For any given sample, the fused representation can be computed using only the distribution of available modalities, with missing modalities automatically excluded from the aggregation. This structural characteristic enables the model to perform robust inference in real-world scenarios where sensor failures, occlusion, or privacy constraints may prevent access to all modalities, without requiring retraining or heuristic imputation.

[0032] Since each modality distribution is a Gaussian distribution with a diagonal covariance, the fused distribution is also Gaussian. This closed-form property allows for direct computation of the mean and variance of the joint multimodal embeddings, providing a principled estimate of the combined representation and its uncertainty. In summary, the embeddings generated by the UPF policy are both expressive and robust, capturing the most informative cues from all available modalities while preserving uncertainty information. This makes predictions for downstream tasks (such as sentiment recognition) more reliable and interpretable.

[0033] When each mode The latent distribution is defined as: In this invention, the PoE fusion method is used to obtain a unified multimodal representation: .

[0034] Since the latent distributions of all modes follow a Gaussian distribution with diagonal covariance, the fused latent distribution is still a Gaussian distribution. ; The parameters of the fused latent distribution can be derived in closed-form. Specifically, the accuracy (reciprocal of the variance) of the fused latent distribution is: The inverse operation is performed element-wise.

[0035] Therefore, the variance of the fused potential distribution is: The mean of the fused potential distribution is obtained by precision-weighted combination of the means of each mode: ,in," " is the symbol for inverse, This indicates element-wise multiplication.

[0036] S5. The fused mean is mapped to obtain the sentiment prediction result.

[0037] Here, the present invention uses the mean of the fused potential distribution. As a multimodal representation for sentiment prediction, the regression head maps the mean of the fused latent distribution to the final prediction result, thus obtaining the sentiment prediction result. , It indicates a return to the beginning.

[0038] In this invention, steps S2 to S5 are implemented by a trained multimodal emotion recognition model. The multimodal emotion recognition model includes: multiple encoders, multiple uncertainty-aware probability modeling modules, an uncertainty-aware probability fusion module, multiple unimodal prediction / regression heads, and a multimodal prediction / regression head. Specifically, the multiple encoders correspond one-to-one with multiple modalities, and each encoder is used to encode the initial features of a corresponding modality. The output of each encoder is connected to the input of an uncertainty-aware probability modeling module, and each uncertainty-aware probability modeling module is used to perform uncertainty-aware probability modeling on the features of a modality. The output of each uncertainty-aware probability modeling module is connected to both the corresponding unimodal prediction / regression head and the uncertainty-aware probability fusion module, and the output of the uncertainty-aware probability fusion module is connected to the input of the multimodal prediction / regression head. Specifically, each uncertainty-aware probability modeling module includes: two sets of networks with identical structures, each set consisting of a multilayer perceptron (MLP) and a linear layer. These two sets of networks are used to predict the mean and variance, respectively, and the parameters of the multilayer perceptrons in these two sets of networks are determined by the final training results. The uncertainty-aware probability fusion module uses the PoE method to fuse multiple probability distributions (i.e., latent distributions). For example, Figure 2 This is a schematic diagram of the overall framework of a multimodal emotion recognition model. For example... Figure 2 As shown, the overall framework of the multimodal emotion recognition model includes three main modules: feature extraction, probabilistic modeling, and PoE fusion and constraint. The feature extraction module contains a language encoder, a visual encoder, and an audio encoder; the probabilistic modeling module contains three UPM modules; and the PoE fusion and constraint module contains one UPF module and four regression heads. Figure 2 As shown, each encoder Initial features for the corresponding mode ( Further feature extraction is performed to obtain the features. Then the features The data is fed into a corresponding UPM module, which then processes the features belonging to the deterministic representation. Uncertainty-aware probabilistic modeling is performed to obtain the latent distribution under the corresponding modality, which is the uncertainty representation under the corresponding modality. Then, the mean of the latent distribution under the corresponding modality is input into a corresponding regression head to obtain the single-modality prediction result of that modality. At the same time, the latent distribution under the corresponding modality is input into the UPF module. The UPF module uses the PoE strategy to fuse the latent distributions of the three modalities to obtain the fused latent distribution. The mean of the fused latent distribution is input into a regression head to obtain the multimodal prediction result.

[0039] The latent distribution obtained through PoE fusion may be dominated by modes with lower uncertainty, potentially leading to insufficient representation of information-rich but more ambiguous modes. To address this issue and ensure that all available modes, including those with higher uncertainty, make meaningful contributions, this invention introduces a "modality consistency loss." This loss encourages the fused latent distribution to align with the latent distribution of each individual mode, thereby enhancing the robustness and expressiveness of the learned multimodal embeddings. For example, the modality consistency loss is defined as: ; in, This represents the modal consistency loss. These represent the language modality, visual modality, and acoustic modality, respectively. This indicates KL's loss. Indicates the potential distribution after fusion. Representing modes Potential distribution, Representing latent variables, , Representing modes The initial characteristics.

[0040] In this invention, each sample initial features Includes language ( ), visual ( ) and acoustics The initial characteristics of these three modes are represented as follows: . These are three different feature sequences, and... , and ,in, , Indicates sample modality Length, Representing modes The feature dimension. Therefore, the training set of the model can be defined as... ,in, It is the sample size. It is the first The sentiment labels for each sample. The prediction results can be a predefined set. One of the categories ( It can also be a continuous intensity variable. ).

[0041] In this invention, the loss function used to train the multimodal emotion recognition model includes: a multimodal prediction loss for supervising the fused representation, a single-modal prediction loss for supervising the single modality, and a modality consistency loss for encouraging the fused latent distribution to remain consistent with the latent distribution of each modality. For example, the expressions for the multimodal prediction loss and the single-modal prediction loss are as follows: ; ; in, and These represent the multimodal prediction loss and the single-modal prediction loss, respectively. Indicates the first The sentiment labels of each sample Indicates the first Sentiment prediction results for each sample Indicates according to the first Modalities of a sample The sentiment prediction result is obtained from the mean of the latent distribution. These represent the language modality, visual modality, and acoustic modality, respectively. This indicates taking the absolute value.

[0042] In summary, the loss function of the multimodal emotion recognition model can be expressed as follows: ,in, This represents the loss function used to train the multimodal emotion recognition model. and These are used for control and The hyperparameters of the contributions of these two items.

[0043] In combination with the above Figure 2 As can be seen, the multimodal emotion recognition model first uses a modality-specific encoder to independently encode each modality to extract rich modality-specific features. These features are then projected into a probabilistic latent embedding space (UPF) through a UPF module, capturing semantic content and modality-specific uncertainty. By representing the input as a probability distribution rather than a deterministic vector, this probabilistic embedding explicitly captures the inherent uncertainty and ambiguity. The UPF module then integrates these embeddings into a joint multimodal representation, adaptively weighting each modality based on the estimated uncertainty. The fusion strategy employed by the UPF module effectively aggregates information across modalities and naturally supports inference in scenarios with missing modalities. Finally, the modality consistency loss forces the multimodal posterior distribution to align with the individual unimodal distributions, preventing overconfident modalities from dominating while preserving informational cues from uncertain but valuable sources, ensuring semantic consistency between the aggregated representation and individual representations, and mitigating modality bias.

[0044] To illustrate the advantages of this probabilistic modeling in the present invention, Figure 3 A schematic diagram of unimodal affective ambiguity and different representation paradigms is given. For example... Figure 3 As shown, the same text, "It's okay, I can do it myself.", can convey a variety of emotions depending on tone, facial expression, or context, such as "confident," "reluctant," "disappointed," and so on. Deterministic methods, however, only map the input to a single emotion, ignoring the variability of emotions. But the probabilistic modeling method proposed in this invention models possible emotion distributions, capturing uncertainty and diversity.

[0045] This invention also provides a multimodal emotion recognition device based on uncertainty probability modeling and expert product, including a processor, a communication interface, a memory, and a communication bus. The processor, communication interface, and memory communicate with each other via the communication bus. The memory stores computer programs. The processor, when executing the program stored in the memory, implements the steps of the aforementioned multimodal emotion recognition method based on uncertainty probability modeling and expert product.

[0046] The present invention has the following advantages: (1) Improve the expressive power of emotion representation: Existing technologies usually adopt deterministic single-modal feature representation, assuming a one-to-one correspondence between modality and emotion state, which makes it difficult to capture the inherent emotional ambiguity and diversity in single modality. This invention uses uncertainty perception probability modeling to represent each modality as a Gaussian distribution, where the mean encodes semantic content and the variance characterizes modality uncertainty, thereby naturally capturing multiple reasonable emotional interpretations.

[0047] (2) Enhancing the stability and robustness of multimodal fusion: Existing technologies do not consider the confidence differences of each modality during multimodal fusion, making them susceptible to being dominated by noisy or inconsistent modalities. This invention employs UPF and PoE strategies to fuse the potential distributions of each modality according to uncertainty weights. High-confidence modalities contribute more, while noisy modalities automatically have reduced weights, and uncertain but informative modal signals are retained. The fused multimodal representation is more stable and robust, effectively resisting interference from noise and conflicting modalities, and improving the reliability of multimodal emotion recognition.

[0048] (3) Preserving the integrity of multimodal information and preventing low-confidence modalities from being ignored: Existing multimodal methods tend to have the fusion result dominated by modalities with low uncertainty, thus ignoring potentially valuable fuzzy modal information. This invention introduces modality consistency loss, which minimizes the KL divergence between the fused latent distribution and the latent distributions of each individual modality, ensuring that all modalities contribute effectively during the learning process. The fused multimodal representation can retain useful information from uncertain modalities, improving the model's ability to perceive fine-grained emotional cues.

[0049] (4) Improved applicability under conditions of missing or incomplete modalities: In practical applications, some modalities may be missing due to sensor failure, occlusion, or privacy restrictions. Existing methods usually rely on zero-padding or heuristic completion, which leads to a decrease in prediction performance. The PoE fusion strategy of this invention can perform joint distribution calculation using only available modalities, naturally excluding missing modalities without the need for additional completion or retraining. This allows the system to maintain reasonable prediction performance even under missing modalities, enhancing its applicability and stability in real-world scenarios.

[0050] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0051] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0052] In this specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. While different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce a good effect.

[0053] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A multimodal emotion recognition method based on uncertainty probability modeling and expert product, characterized in that, include: S1. Obtain the original signal sequences of multiple modes, and extract the features of the original signal sequences of each mode to obtain the initial features of each mode; S2. Encode the initial features of each modality to obtain the features of each modality; S3. Perform uncertainty-aware probability modeling on the features of each modality to obtain the latent distribution of each modality. The latent distribution of each modality is a Gaussian distribution, where the mean of the Gaussian distribution represents the semantic content of the modality, and the variance characterizes the uncertainty of the modality. S4. Using an expert product fusion method, the potential distributions of the multiple modalities are fused to obtain a fused potential distribution, wherein the fused potential distribution has a fused mean and a fused variance; S5. The fused mean is mapped to obtain the sentiment prediction result.

2. The multimodal emotion recognition method based on uncertainty probability modeling and expert product according to claim 1, characterized in that, Steps S2 to S5 are implemented by the trained multimodal emotion recognition model, which includes: multiple encoders, multiple uncertainty-aware probability modeling modules, an uncertainty-aware probability fusion module, multiple single-modal prediction and regression heads, and a multimodal prediction and regression head; The plurality of encoders correspond one-to-one with the plurality of modalities, and each encoder is used to encode the initial features of a corresponding modality. The output of each encoder is connected to the input of an uncertainty-aware probability modeling module, and each uncertainty-aware probability modeling module is used to perform uncertainty-aware probability modeling on the features of a modality. The output of each uncertainty-aware probability modeling module is connected to the corresponding single-modality prediction and regression head on one hand, and to the uncertainty-aware probability fusion module on the other hand. The output of the uncertainty-aware probability fusion module is connected to the input of the multimodal prediction and regression head.

3. The multimodal emotion recognition method based on uncertainty probability modeling and expert product according to claim 2, characterized in that, Each uncertainty-aware probability modeling module includes: two sets of networks with identical structures, each set consisting of a multilayer perceptron and a linear layer; the two sets of networks with identical structures are used to predict the mean and variance, respectively. The uncertainty-aware probability fusion module uses the PoE method to fuse multiple potential distributions.

4. The multimodal emotion recognition method based on uncertainty probability modeling and expert product according to claim 1, characterized in that, The various modalities include: language modality, visual modality, and acoustic modality.

5. The multimodal emotion recognition method based on uncertainty probability modeling and expert product according to claim 1, characterized in that, The latent distribution for each mode is a normal distribution.

6. The multimodal emotion recognition method based on uncertainty probability modeling and expert product according to claim 4, characterized in that, The expressions for the fused variance and the fused mean are as follows: ； in, and Let V be the variance and Mean after fusion, respectively. These represent the language modality, visual modality, and acoustic modality, respectively. and Representing modes The uncertainty and semantic content, " is the symbol for inverse, This indicates element-wise multiplication.

7. The multimodal emotion recognition method based on uncertainty probability modeling and expert product according to claim 2, characterized in that, The loss function used to train the multimodal emotion recognition model includes: a multimodal prediction loss for supervising the fused representation, a single-modal prediction loss for supervising the single modality, and a modality consistency loss for encouraging the fused latent distribution to remain consistent with the latent distribution of each modality.

8. The multimodal emotion recognition method based on uncertainty probability modeling and expert product according to claim 7, characterized in that, The expression for the modal consistency loss is as follows: ； in, This represents the modal consistency loss. These represent the language modality, visual modality, and acoustic modality, respectively. This indicates KL's loss. This represents the potential distribution after fusion. Representing modes Potential distribution, Representing latent variables, , Representing modes The initial characteristics.

9. The multimodal emotion recognition method based on uncertainty probability modeling and expert product according to claim 7, characterized in that, The expressions for the multimodal prediction loss and the single-modal prediction loss are as follows: ； ； in, and These represent the multimodal prediction loss and the single-modal prediction loss, respectively. Indicates the first The sentiment labels of each sample Indicates the first Sentiment prediction results for each sample Indicates according to the first Modalities of a sample The sentiment prediction result is obtained from the mean of the latent distribution. These respectively represent the language modality, visual modality, and acoustic modality among the multiple modalities. This indicates taking the absolute value.

10. The multimodal emotion recognition method based on uncertainty probability modeling and expert product according to claim 7, characterized in that, The expression for the loss function used to train the multimodal emotion recognition model is as follows: ； in, This represents the loss function used to train the multimodal emotion recognition model. , and These represent the modal consistency loss, the multimodal prediction loss, and the single-modal prediction loss, respectively. and These are used for control and The hyperparameters of the contributions of these two items.

Images