Adaptive dual-path evidence fusion method and system for multi-modal emotion recognition, medium

Abstract

The application discloses a multi-modal emotion recognition adaptive dual-path evidence fusion method and system, medium, and belongs to the field of artificial intelligence and emotion computing technology.The method comprises the following steps: acquiring and encoding multi-modal data, obtaining modal invariant features and modal specific features through representation decoupling; estimating single-modal evidence and cognitive uncertainty based on the modal specific features, and detecting a modal conflict vector; generating the reliability weight of each mode by using the uncertainty and the conflict vector; in the alignment path, performing conflict modulation cross-modal fusion on the weighted features; in the conflict path, performing hierarchical evidence fusion on the modulated evidence; finally, generating class-by-class path weights, and adaptively fusing the dual-path prediction results.The application significantly improves the emotion recognition accuracy and robustness in the modal divergence scene by explicitly detecting modal conflicts and adaptively switching fusion paths.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and affective computing, and in particular to an adaptive dual-path evidence fusion method, system, and medium for multimodal emotion recognition. Background Technology

[0002] Multimodal emotion recognition is an important research direction in the fields of artificial intelligence and human-computer interaction. It aims to accurately perceive and judge a user's emotional state by integrating heterogeneous information from multiple sources, such as text content, speech prosody, facial expressions, and gestures. Existing technologies typically employ strategies such as feature-level fusion, attention-level fusion, or decision-level fusion when handling multimodal emotion recognition tasks.

[0003] For example, tensor-based fusion methods capture higher-order interactions by performing outer product operations on features from different modalities; Transformer-based cross-modal attention mechanisms (such as MulT) achieve feature alignment and fusion by learning latent associations between modalities layer by layer; and evidence-based fusion methods (such as TMC) utilize Dempster-Shafer evidence theory to model and combine the classification uncertainties of each modality. These approaches have achieved significant progress on controlled scenarios or datasets with high modality consistency.

[0004] However, most of the existing solutions implicitly follow a modality consistency assumption, which assumes that the emotional tendencies reflected by different modalities are essentially synergistic or complementary. Under this assumption, fusion models usually adopt a fixed, single-path processing approach, failing to fully consider the modality divergence phenomenon that is common in real-world interaction scenarios. In real-world applications, the following situations often occur: (1) the text content expresses a positive meaning, while the tone of voice presents a negative or sarcastic meaning; (2) facial expressions are subjectively restrained or concealed, while acoustic signals expose the true emotional state; (3) there is a discrepancy between the surface semantics of the text and the speaker's true intention, such as irony, metaphor, or sarcasm. In the above modality conflict scenarios, different modalities are not simply complementary, but rather there are obvious information ambiguities or even contradictions.

[0005] If the traditional fixed single-path fusion method is still used, the following technical problems will occur: 1. Excessive interaction between conflicting modes leads to feature interference. When there are discrepancies between modes, forcing deep fusion at the feature layer can cause effective information from reliable modes to be overwhelmed by interference from noisy or contradictory modes, thereby reducing the discriminative power of the feature representation.

[0006] Second, the model lacks the ability to explicitly estimate the dynamic reliability of modes. Existing methods usually treat each mode in a static or global manner, and cannot dynamically quantify the credibility of each mode at the current moment based on the context of the input samples. Therefore, they are easily dominated by low-quality or misleading modal information.

[0007] Third, there is a lack of an adaptive fusion path switching mechanism for consistent and conflicting samples. Feature-level fusion and decision-level fusion each have their advantages in different scenarios, but existing methods cannot adaptively select the most suitable fusion strategy based on the degree of consistency or conflict between modalities.

[0008] Fourth, the handling of uncertainty in evidence fusion is relatively rigid. Existing evidence theory-based methods usually treat cognitive uncertainty as a static attribute, lacking a mechanism to adjust the strength of effective evidence based on dynamic conflicts between modalities, thus limiting their robustness in modal conflict scenarios.

[0009] Therefore, how to explicitly perceive modal conflicts in multimodal emotion input, dynamically evaluate the reliability of each modality, and adaptively select the optimal fusion path for modal consistency and modal conflict scenarios are the technical challenges that urgently need to be solved in the field of multimodal emotion recognition technology. Summary of the Invention

[0010] To achieve the above-mentioned objectives and other advantages of the present invention, a first objective of the present invention is to provide an adaptive dual-path evidence fusion method for multimodal emotion recognition, comprising the following steps: Acquire multimodal input data of the object to be identified, wherein the multimodal input data includes at least two of the following: text data, voice data, and video data; Feature extraction and unified mapping are performed on the multimodal input data to obtain unified features for each modality; The unified features of each mode are characterized and decoupled to obtain the mode-invariant features and mode-specific features of each mode respectively; Based on the modality-specific features of each modality, the single-modal evidence of each modality and its corresponding cognitive uncertainty are estimated; The modal conflict vector is calculated based on the difference in modal-specific features between different modalities. Based on the cognitive uncertainty and the modal conflict vector, a reliability weight for each modality is generated; The modality-invariant features and the modality-specific features weighted by the reliability weights are input into the alignment path, and cross-modal fusion of conflict modulation is performed to obtain the alignment path prediction result. The single-mode evidence, after being inversely modulated by the reliability weight, is input into the conflict path, and hierarchical evidence fusion is performed to obtain the conflict path prediction result. Based on the modal conflict vector, the cognitive uncertainty, and the reliability weight, class-wise path weights are generated, and the class-wise path weights are used to perform class-wise adaptive fusion of the alignment path prediction results and the conflict path prediction results to obtain the final emotion recognition result.

[0011] Furthermore, the step of calculating the modal conflict vector based on the difference in modal-specific features between different modalities also includes: Calculate the normalized cosine distance between modal-specific features of any two modes to obtain the feature-level conflict score; Based on the cognitive uncertainty of each of the two modalities, the feature-level conflict score is amplified to obtain the conflict score with enhanced uncertainty. The information of the conflict scores after the uncertainty enhancement is combined to form the modal conflict vector.

[0012] Furthermore, the step of calculating the modal conflict vector also includes: A large language model is introduced to perform offline reasoning on text data, and the difference between the original text features and the semantic features obtained by reasoning is calculated as a semantic-level conflict signal. The semantic-level conflict signal and the conflict score with enhanced uncertainty are used together to form the modal conflict vector.

[0013] Furthermore, the step of generating reliability weights for each modality based on the cognitive uncertainty and the modal conflict vector includes: The vector of cognitive uncertainty of each modality is concatenated with the modal conflict vector; The concatenated vector is input into a multilayer perceptron network with a Softmax function that includes a temperature parameter, and the reliability weights of each mode are output. In the alignment path, the reliability weight is used to modulate the modal specific features corresponding to forward modulation; in the conflict path, the reliability weight is used to modulate the effective uncertainty of the modality corresponding to backward modulation.

[0014] Furthermore, the cross-modal fusion step of performing conflict modulation includes: The modality-invariant features of each modality and the modality-specific features weighted by reliability weights are concatenated to construct a multimodal feature sequence; Construct an attention bias matrix based on the feature-level conflict scores between pairwise modalities; During the multi-head self-attention mechanism calculation, the attention bias matrix is ​​injected into the attention score to suppress the attention weights between high-conflict mode pairs.

[0015] Furthermore, the hierarchical evidence fusion step includes: Based on the single-mode evidence of each mode and its effective uncertainty after inverse modulation, a belief vector for each mode is generated, wherein the sum of the belief vector of each mode and its corresponding effective uncertainty is 1. Following a pre-defined hierarchical order, the belief vectors of at least two modalities are iteratively fused using the Dempster-Shafer evidence combination rule; In each fusion process, an upper limit constraint is set on the conflict coefficient in the combination rules to avoid the normalization factor being too large.

[0016] Further, the step of generating class-specific path weights and using the class-specific path weights to perform class-specific adaptive fusion of the aligned path prediction results and the conflict path prediction results includes: The modal conflict vector, the cognitive uncertainty vector of each modality, and the reliability weight vector of each modality are concatenated. The concatenated vector is input into a multilayer perceptron network with a sigmoid function as the output layer, and a path weight between 0 and 1 is generated independently for each sentiment category. For each emotion category k, the final predicted probability is calculated by combining the conflict path prediction result and the alignment path prediction result using the formula... We obtain the result by weighting k, where... This refers to the path weight output by the multilayer perceptron network for the k-th emotion category, with a value ranging from 0 to 1. K represents the total number of emotion categories. This represents the final predicted probability that the object to be identified belongs to the k-th sentiment category; This represents the predicted probability of the k-th sentiment category in the conflict path prediction results; This represents the predicted probability of the k-th sentiment category in the alignment path prediction result.

[0017] Furthermore, the step of estimating the unimodal evidence for each modality and its corresponding cognitive uncertainty includes: The modality-specific features of each modality are input into the evidence estimation network, which outputs a non-negative vector as the evidence parameter of the Dirichlet distribution. Adding 1 to the evidence parameters yields the concentration parameters of the Dirichlet distribution; Based on the concentration parameter, the total amount of evidence is calculated, and based on the total number of sentiment categories K, it is determined using the formula... Cognitive uncertainty in calculating mode m ,in This represents the total amount of evidence for a single modality.

[0018] A second objective of this invention is to provide an adaptive dual-path evidence fusion system for multimodal emotion recognition, comprising: The multimodal data acquisition module is used to acquire multimodal input data of the object to be identified; The feature encoding and unified mapping module is used to extract features and perform unified mapping on the multimodal input data to obtain unified features for each modality. The characterization and decoupling module is used to characterize and decouple the unified features of each mode, and obtain the mode-invariant features and mode-specific features of each mode respectively. The evidence and uncertainty estimation module is used to estimate the single-modal evidence of each modality and its corresponding cognitive uncertainty based on the modality-specific features of each modality. The modal conflict detection module is used to calculate the modal conflict vector based on the difference in modal-specific features between different modalities; The conflict-aware modal reweighting module is used to generate reliability weights for each modality based on the cognitive uncertainty and the modal conflict vector. The alignment path fusion module is used to perform cross-modal fusion of the modality-invariant features and the modality-specific features weighted by the reliability weights to obtain the alignment path prediction result; The conflict path evidence fusion module is used to perform hierarchical evidence fusion on the single-modal evidence after the reliability weight is back-modulated to obtain the conflict path prediction result. The class-wise adaptive routing module is used to generate class-wise path weights based on the modality conflict vector, the cognitive uncertainty, and the reliability weights, and to use the class-wise path weights to perform class-wise adaptive fusion of the alignment path prediction results and the conflict path prediction results to obtain the final emotion recognition result.

[0019] Furthermore, the alignment path fusion module is further configured as follows: An attention bias matrix is ​​constructed based on the feature-level conflict scores between pairs of modalities, and this bias matrix is ​​injected into the attention score of the multi-head self-attention mechanism to achieve cross-modal fusion of conflict modulation.

[0020] Furthermore, the conflict path evidence fusion module is further configured as follows: Belief vectors for each modality are generated based on the single-modal evidence for each modality and its effective uncertainty after inverse modulation. Following a pre-defined hierarchical order, the belief vectors of each modality are iteratively fused using the Dempster-Shafer evidence combination rule with upper limit constraints on the conflict coefficient.

[0021] Furthermore, the class-by-class adaptive routing module is further configured as follows: Generate an independent path weight for each emotion category; For each emotion category, the aligned path prediction result and the conflict path prediction result are weighted and fused according to their corresponding path weights.

[0022] A third objective of the present invention is to provide a computer-readable storage medium having program instructions stored thereon, which, when executed, implement the method described above.

[0023] Compared with the prior art, the beneficial effects of the present invention are: This invention provides an adaptive dual-path evidence fusion method, system, and medium for multimodal emotion recognition. It can explicitly detect multimodal emotion conflicts, rather than assuming all modalities are always consistent. It can dynamically evaluate the reliability of each modality in the current sample, reducing the adverse effects of noisy modalities. Through dual-path design, consistent and conflicting samples are fused using more appropriate methods. Conflict-modulated attention suppresses unreasonable feature alignment between high-conflict modalities. Reliability-modulated evidence fusion better preserves the independent judgment of credible modalities in conflict scenarios. Class-wise adaptive routing achieves finer-grained path allocation than global single-gating. Optionally, it combines offline inference results from large language models to enhance the recognition of complex text emotions such as irony, satire, and metaphor. Experimental metrics on high-conflict-proportion datasets demonstrate that this invention significantly improves both overall recognition performance and conflict scenario recognition performance. Even in complex scenarios where 71.4% of samples exhibit modal disagreement, it still achieves optimal results, improving W-F1 accuracy by 2.17 percentage points and conflict subset accuracy by 4.01 percentage points, indicating that this invention is particularly suitable for high-conflict multimodal emotion recognition tasks.

[0024] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it according to the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Specific embodiments of the present invention are given in detail below with reference to the accompanying drawings. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a schematic diagram illustrating modal consistency and modal conflict scenarios. Figure 2 This is a diagram of the overall architecture for adaptive dual-path evidence fusion in multimodal emotion recognition. Figure 3 A flowchart for an adaptive dual-path evidence fusion method for multimodal emotion recognition. Figure 1 ; Figure 4A flowchart for an adaptive dual-path evidence fusion method for multimodal emotion recognition. Figure 2 ; Figure 5 A flowchart for estimating single-modal evidence for each modality and its corresponding cognitive uncertainty based on modality-specific features of each modality; Figure 6 The process for calculating modal conflict vectors based on the difference in modal-specific features between different modes Figure 1 ; Figure 7 The process for calculating modal conflict vectors based on the difference in modal-specific features between different modes Figure 2 ; Figure 8 A flowchart for generating reliability weights for each mode based on cognitive uncertainty and the modal conflict vector; Figure 9 To input modality-invariant features and modality-specific features weighted by reliability weights into the alignment path, cross-modal fusion with conflict modulation is performed to obtain the alignment path prediction result flowchart; Figure 10 Flowchart of conflict paths; Figure 11 The flowchart shows how to input single-modal evidence, after reverse modulation of reliability weights, into the conflict path, perform hierarchical evidence fusion, and obtain the conflict path prediction results. Figure 12 The flowchart shows how to generate class-specific path weights based on modal conflict vectors, cognitive uncertainty, and reliability weights, and then use these class-specific path weights to adaptively fuse the aligned path prediction results and conflict path prediction results to obtain the final emotion recognition result. Figure 13 Schematic diagram of CAMR and PCAR structure; Figure 14 A schematic diagram of the Transformer structure for conflict modulation cross-modal fusion; Figure 15 Visualize the experimental results. Figure 16 This is a schematic diagram illustrating a typical conflict sample case analysis; Figure 17 A schematic diagram of computer equipment; Figure 18 This is a schematic diagram of a computer-readable storage medium. Detailed Implementation

[0026] The present invention will now be further described with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0027] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention.

[0028] The drawing numbers in this application are only used to distinguish the steps in the scheme and are not used to limit the execution order of the steps. The specific execution order is as described in the specification.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0030] Figure 1 A comparative diagram of modal-consistent and modal-conflicting scenarios is shown. For example... Figure 1 As shown, in modality-consistent scenarios, the emotional tendencies reflected by text, speech, and video modalities are consistent, and traditional fusion methods can usually predict them correctly. However, in modality-conflicting scenarios, there are obvious emotional discrepancies between different modalities. For example, text content expresses positive emotions while speech tone presents negative emotions. Static fusion methods are easily affected by feature interference, leading to a decline in recognition performance. In the EmotionTalk dataset used in this invention, statistics show that approximately 71.4% of the samples exhibit intermodal emotional conflict. The adaptive dual-path evidence fusion framework proposed in this invention can distinguish between the two scenarios through a conflict-aware routing mechanism, thereby improving the recognition robustness in modality-conflicting scenarios. The specific scheme is as follows: Example 1

[0031] An adaptive dual-path evidence fusion method for multimodal emotion recognition, such as Figures 2-4 As shown, it includes the following steps: S100. Obtain multimodal input data of the object to be identified, wherein the multimodal input data includes at least two of the following: text data, voice data, and video data; In this embodiment, the multimodal input data includes three modalities: text data, voice data, and video data.

[0032] Specifically, text data can be dialogue text, transcribed text, or subtitle text; speech data can be speech segments corresponding to the text; and video data can be a sequence of video frames synchronized with the speech. These three modalities should maintain a synchronous correspondence in time.

[0033] S200. Perform feature extraction and unified mapping on the multimodal input data to obtain unified features for each modality; This embodiment utilizes pre-trained encoders to extract features from three modalities. In a preferred embodiment, text features are extracted by a text encoder, such as BERT or RoBERTa pre-trained language models, to obtain text feature vectors; speech features are extracted by a speech encoder, such as wav2vec 2.0 or HuBERT, to obtain speech feature vectors; and video features are extracted by a visual encoder at the frame level, and then processed by temporal attention pooling to obtain video-level representations, such as 3DCNN or Video Swin Transformer.

[0034] Let the features of text, speech, and video after mapping through the unified projection layer be as follows: ,in To unify feature dimensions, a unified mapping is used to project features from different modalities into the same semantic space, facilitating subsequent modal interactions and fusion.

[0035] S300. The unified features of each mode are characterized and decoupled to obtain the mode-invariant features and mode-specific features of each mode respectively. In this embodiment, the uniform features of each mode are input into the shared encoder and the mode-specific encoder respectively, and decomposed into mode-invariant features and mode-specific features.

[0036] Specifically, for modal ,have: in, These represent modality-invariant features, used to represent sentiment semantic information shared across modalities; It represents modality-specific features, used to represent emotional cues unique to the corresponding modality. For a shared mode-invariant feature encoder, A modality-specific feature encoder, A projection layer specific to the modality.

[0037] In a preferred embodiment of this example, distribution alignment constraints and adversarial constraints can also be introduced to further weaken the modality source information remaining in the modality invariant features, so that the modality invariant features more purely reflect the sentiment semantics shared across modalities.

[0038] S400. Based on the modality-specific features of each modality, estimate the single-modal evidence of each modality and its corresponding cognitive uncertainty; such as Figure 5 As shown, specifically including: S410. Input the modality-specific features of each modality into the evidence estimation network, and the network outputs a non-negative vector as the evidence parameter of the Dirichlet distribution. S420. Add 1 to the evidence parameters to obtain the concentration parameters of the Dirichlet distribution; Specifically, for modal : in, Let K be the concentration parameter of the Dirichlet distribution, K be the total number of sentiment categories, and 1 be a vector of all 1s. The Softplus function ensures that the output is non-negative, and the addition operation ensures that the concentration parameter is at least 1, corresponding to the prior of a uniform distribution.

[0039] S430. Based on the concentration parameter, calculate the total amount of evidence, and based on the total number of sentiment categories K, use the formula... Cognitive uncertainty in calculating mode m ,in This represents the total amount of evidence for a single modality.

[0040] Specifically, let the total amount of evidence for mode m be: This leads to the cognitive uncertainty of mode m: in, The larger the value, the less evidence is available for that modality, and the less reliable the prediction. When there is sufficient evidence, the uncertainty approaches 0; when there is very little evidence, the uncertainty approaches 1.

[0041] S500: Calculate the modal conflict vector based on the difference in modal-specific features between different modalities; In some embodiments, such as Figure 6 As shown, the step of calculating the modal conflict vector based on the difference in modal-specific features between different modalities further includes: S510. Calculate the normalized cosine distance between modal-specific features of any two modes to obtain the feature-level conflict score. For modes i and j: in, The value of this score ranges from [0,1]. The larger the value, the greater the directional difference between the specific features of the two modalities in the semantic space, that is, the higher the degree of conflict.

[0042] S520. Based on the cognitive uncertainty of each of the two modalities, the feature-level conflict score is amplified to obtain the conflict score with enhanced uncertainty. Considering that conflicts arising from high-uncertainty modes are often more noteworthy, the conflict score is further amplified: In a preferred embodiment of this invention, the amplification factor is controlled within a preset upper limit, for example, limiting the amplification factor to no more than 1.5 times, to avoid distortion of the conflict signal caused by local outliers. This amplification mechanism further enhances the conflict signal when two modes simultaneously have high cognitive uncertainty, thereby attracting more attention from subsequent modules.

[0043] S530. Combine the information from the uncertainty-enhanced conflict scores to form the modal conflict vector: The conflict vector It serves as an important input for subsequent modules, guiding modal reliability assessment and path selection.

[0044] In the modal conflict detection step, semantic-level conflict signals are also introduced. Furthermore, such as... Figure 7 As shown, the step of calculating the modal conflict vector further includes: S540. Introduce a large language model to perform offline reasoning on text data, calculate the difference between the original text features and the semantic features obtained by reasoning, and use it as a semantic-level conflict signal. S550. The semantic-level conflict signal and the conflict score after uncertainty enhancement are used together to form the modal conflict vector.

[0045] Specifically, a Large Language Model (LLM) is introduced to perform offline inference on text data to obtain inferred text features. And calculate its features compared with the original text. Differences: when When the value is large, it indicates a significant discrepancy between the surface semantics and the deeper intention of the text, such as complex semantic phenomena like irony, sarcasm, and metaphor.

[0046] In this embodiment, the final conflict vector is expanded as follows: By introducing semantic-level conflict signals, this embodiment can more effectively handle complex semantic phenomena at the text level, and further improve the accuracy of emotion recognition in scenarios such as irony and sarcasm.

[0047] S600. Based on the cognitive uncertainty and the modal conflict vector, generate reliability weights for each modality; such as... Figure 8 As shown, it specifically includes: S610. Concatenate the vector of cognitive uncertainty of each modality with the modal conflict vector; S620. Input the concatenated vector into a multilayer perceptron network with a Softmax function that has a temperature parameter, and output the reliability weights of each mode. Specifically, the vector formed by the cognitive uncertainty of each modality The modal conflict vector c is concatenated with the modal conflict vector c and input into a multilayer perceptron network. The reliability weights of each modality are then output through a Softmax function with a temperature parameter. in, , This is a temperature parameter used to control the smoothness of the output distribution. When When the weights are smaller, the output weight distribution is steeper, meaning it tends to selectively utilize reliable modes; when... When the weights are larger, the weight distribution becomes smoother.

[0048] In the alignment path, the reliability weight is used to positively modulate the corresponding modal-specific features; The physical meaning of this modulation method lies in: when the reliability weight of a certain mode... When the value is below the average level, the specific features of this mode will be suppressed; when When the value is above average, the contribution of specific features of this mode is enhanced. In this way, the alignment path can focus more on information from reliable modes.

[0049] In the conflict path, the reliability weight is used to inversely modulate the effective uncertainty of the corresponding mode: in, As the stability cutoff constant, it is taken in this embodiment. When modal reliability is low ( When it is smaller, This is amplified, thus weakening the modality's influence in evidence fusion; when the modality's reliability is high ( When it is relatively large, The modality is reduced in size, thereby enhancing its influence in evidence fusion.

[0050] Figure 13 This is a schematic diagram of the structure of the Conflict-Aware Modal Reweighting (CAMR) module and the Class-by-Class Adaptive Routing (PCAR) module in this invention. The CAMR estimates modal reliability by combining uncertainty and conflict signals, and applies them to the alignment path and conflict path respectively; the PCAR generates routing weights independently for each sentiment category.

[0051] S700. Input the modality-invariant features and the modality-specific features weighted by the reliability weights into the alignment path, perform cross-modal fusion of conflict modulation, and obtain the alignment path prediction result. Alignment paths are used to handle scenarios where the overall modality is consistent or can be enhanced through complementary interactions. For example... Figure 9 As shown, it specifically includes: S710. Concatenate the modality-invariant features of each modality and the modality-specific features weighted by reliability weights to construct a multimodal feature sequence; S720: Construct an attention bias matrix based on the feature-level conflict scores between pairs of modalities; First, the pairwise feature-level conflict scores are mapped to attention bias matrices. In this embodiment, the original three feature-level conflict scores are... Given a mapping network, generate a 3×3 attention bias matrix B: in, Each element of the bias matrix B This represents the attention bias from mode i to mode j; highly conflicting mode pairs will be given a larger negative bias.

[0052] S730. During the multi-head self-attention mechanism calculation process, the attention bias matrix is ​​injected into the attention score to suppress the attention weights between high-conflict mode pairs.

[0053] In multi-head self-attention computation, the bias matrix is ​​injected into the attention logits: Where Q, K, and V are the query, key, and value matrices, respectively. It is a single-head dimension. In this way, the attention weights between highly conflicting modalities are actively suppressed, avoiding excessive coupling of conflicting information at the feature level. Figure 14 This is a schematic diagram of the attention mechanism of Conflict Modulation Cross-Modal Fusion Transformer (CMFT) used in the alignment path of this invention. The conflict scores are mapped to form an attention bias matrix and participate in the attention calculation.

[0054] The fused modality-invariant features are concatenated with the reliability-weighted modality-specific features and then input into the classification head to obtain the alignment path prediction result. .

[0055] S800: Input the single-mode evidence after the reliability weight inverse modulation into the conflict path, perform hierarchical evidence fusion, and obtain the conflict path prediction result. Conflict paths are used to handle scenarios with significant modal disagreements, employing evidence-level fusion instead of deep feature interactions. For example... Figure 10 , Figure 11 As shown, it specifically includes: S810. Based on the single-mode evidence of each mode and its effective uncertainty after inverse modulation, generate the belief vector of each mode, wherein the sum of the belief vector of each mode and its corresponding effective uncertainty is 1. Specifically, the modality-specific features of each modality are input into the auxiliary classifier to obtain the logits of each modality. And, combined with effective uncertainty, generate a single-modal belief vector: in, This is a temperature parameter used to control the smoothness of the classification probabilities. This generation method ensures that: That is, the belief vector of each mode. With effective uncertainty The sum is 1, which conforms to the basic axiom of evidence theory.

[0056] S820. Following a preset hierarchical order, the belief vectors of at least two modalities are iteratively fused using the Dempster-Shafer evidence combination rule. For two sources of evidence and The combination rules can be expressed as: in, The conflict coefficient represents the degree of inconsistency between two sources of evidence.

[0057] To avoid numerical instability caused by an excessively large normalization factor in high-conflict scenarios, in a preferred embodiment of this invention, the normalization factor is... Set upper limit constraints, for example When the calculated conflict coefficient exceeds this upper limit, it is truncated to 0.95.

[0058] In this embodiment, a hierarchical fusion strategy is adopted: first, text and voice evidence are fused, then the fusion result is fused with video evidence, and finally, the conflict path prediction result is output. It should be noted that the fusion order can be dynamically adjusted according to modal priority, and is not limited to the order described in this embodiment.

[0059] S900: Based on the modal conflict vector, the cognitive uncertainty, and the reliability weight, generate class-specific path weights, and use these class-specific path weights to perform class-specific adaptive fusion of the aligned path prediction result and the conflict path prediction result to obtain the final emotion recognition result. Figure 12 As shown, it specifically includes: S910. The modal conflict vector, the cognitive uncertainty vector of each modality, and the reliability weight vector of each modality are concatenated. S920. Input the concatenated vector into a multilayer perceptron network with a sigmoid function as the output layer, and generate a path weight between 0 and 1 for each sentiment category independently. Specifically, the conflict vector, cognitive uncertainty vector, and modality reliability weights are input into the class-by-class adaptive routing network to generate path weights for each emotion category: in, , The sigmoid function is used to ensure that each path weight is between 0 and 1. Unlike traditional single-scalar gating, this embodiment generates an independent path weight for each sentiment category, making it more suitable for application scenarios where different sentiment categories have varying sensitivities to modal conflict.

[0060] S930. For each sentiment category k, the final predicted probability is calculated by combining the conflict path prediction result and the alignment path prediction result using the formula... We get the weighted average, where, This represents the final predicted probability that the object to be identified belongs to the k-th sentiment category; This represents the predicted probability of the k-th sentiment category in the conflict path prediction results; This represents the predicted probability of the k-th sentiment category in the alignment path prediction result; The path weight output by the multilayer perceptron network for the k-th emotion category has a value range between 0 and 1. K represents the total number of emotion categories. When When the value is close to 1, the system tends to use predictions of conflicting paths; when... When the value is close to 0, the system tends to use the prediction of the aligned path.

[0061] In summary, the complete reasoning process of this embodiment can be summarized as follows: Input text features, speech features, and video features of a uniform dimension; By performing representation decoupling, we obtain the mode-invariant features and mode-specific features of each mode; Estimating Dirichlet evidence and cognitive uncertainty based on modal-specific features; Calculate the eigen-level conflict scores for each pairwise mode and perform uncertainty enhancement amplification; Based on the conflict vector and uncertainty vector, modal reliability weights are generated by conflict-aware modal reweighting. In the alignment path, reliability weights are used to modulate modal-specific features, and a conflict modulation cross-modal fusion Transformer is performed to generate the alignment path result. In the conflict path, single-mode beliefs are generated by utilizing the effective uncertainty after inverse modulation, and hierarchical Dempster-Shafer evidence fusion is performed to generate conflict path results; By using a class-specific adaptive routing network, path weights are generated for each emotion category. The results of the two paths are then fused to output the final emotion recognition result.

[0062] To verify the effectiveness of the technical solution of this invention, the inventors conducted experimental verification based on the Chinese multimodal psychological dialogue dataset EmotionTalk. This dataset contains 19,250 discourse samples, covering 7 emotion categories, with 15,413 samples in the training set, 1,908 in the validation set, and 1,929 in the test set. Statistical analysis showed that 71.4% of the samples exhibited at least one pair of emotional discrepancies between modalities, making it suitable for verifying conflict perception fusion schemes.

[0063] The experiment used accuracy (Acc), weighted F1 score (W-F1), macro-average F1 score (M-F1), and conflict subset accuracy (Conf-Acc) as evaluation metrics. Among them, the conflict subset was a subset of test samples where the text annotation and the comprehensive sentiment annotation were inconsistent, totaling 1096 samples, accounting for 56.8% of the test set.

[0064] To fully illustrate the technical effects of the present invention, the results of the present invention and representative multimodal emotion recognition baseline methods on the EmotionTalk test set are presented below.

[0065] Table 1. Performance comparison of the present invention and baseline methods on the EmotionTalk test set.

[0066] As can be seen from the table above, the complete solution of the present invention achieves the best results in all four indicators: Acc, W-F1, M-F1, and Conf-Acc. This shows that the present invention not only outperforms existing representative fusion methods in overall recognition performance, but also shows stronger advantages in conflict sample recognition.

[0067] On this high-conflict dataset, this invention achieves state-of-the-art results on the high-conflict dataset EmotionTalk. Compared to the best baseline for weighted F1 (MMIM), the W-F1 score of this invention improves from 62.25% to 64.42%, an improvement of 2.17 percentage points, reaching a statistically significant level. Compared to the best baseline for conflict subset accuracy (TFN), the Conf-Acc score of this invention improves from 56.63% to 60.64%, an improvement of 4.01 percentage points. Compared to the best baseline for Acc (TFN), the Acc score of this invention improves from 62.92% to 64.28%, an improvement of 1.36 percentage points. Compared to the best baseline for M-F1 (MMIM), the M-F1 score of this invention improves from 54.55% to 56.38%, an improvement of 1.83 percentage points. The significant improvement achieved even in a 71.4% modality divergence scenario indicates that this invention has stronger adaptability to conflict samples. The standard deviation of the random seed in W-F1 is only 0.43%, indicating good stability of the scheme.

[0068] Therefore, the key technical effect of this invention is not to simply pursue a single absolute accuracy value, but to still achieve recognition performance superior to existing methods in data environments with frequent modal conflicts and higher fusion difficulty through conflict detection, reliability estimation and dual-path adaptive fusion mechanism.

[0069] Even after removing offline semantic reasoning features and retaining only the core dual-path conflict awareness architecture, the following metrics can still be achieved.

[0070] Table 2 Performance comparison after removing offline semantic reasoning features

[0071] This indicates that even without introducing additional semantic reasoning features, the core conflict-aware architecture itself still provides significant benefits to conflict samples; offline semantic reasoning features are optional enhancement modules and not necessary conditions for the main scheme to be valid; after introducing offline semantic reasoning features, W-F1 is further improved by 2.94 percentage points.

[0072] The present invention further performs ablation verification on key modules, and the results of a single seed are as follows.

[0073] Table 3 Ablation Experiment Results of Key Modules

[0074] The results show that removing dual-path routing reduces Conf-Acc by 4.01 percentage points, indicating that dual-path switching is the key mechanism of this invention; removing CAMR reduces Conf-Acc by 2.00 percentage points, indicating that modality reliability estimation has a significant effect on conflict sample identification; removing CMFT reduces Conf-Acc by 1.27 percentage points, indicating that conflict modulation attention is effective in suppressing interference between high-conflict modalities; and removing uncertainty modeling reduces W-F1 by 1.34 percentage points, indicating that cognitive uncertainty has a direct impact on the quality of evidence fusion.

[0075] like Figure 15 As shown, the experimental results are visualized to show the comparison of key indicators in the ablation experiment, the distribution of CAMR modal weights, the relationship between PCAR routing weights and conflict levels, and the comparison of F1 scores for each category.

[0076] The present invention further observed the following phenomena: the average distribution of modal reliability weights was 0.348 for text, 0.376 for audio, and 0.276 for visual, indicating that the system can automatically favor more reliable modalities in actual samples; on all test samples, the average routing weight of conflicting paths was 0.637; it increased to 0.72 in conflict subsets and decreased to 0.51 in consistent subsets, indicating that the present invention does indeed achieve adaptive path switching according to the degree of conflict; in a typical ironic sample, the visual modal weight was reduced to 0.15, while the routing weight of conflicting paths increased to about 0.78, ultimately resulting in a correct prediction, indicating that the present invention has good interpretability.

[0077] Figure 16 This paper presents a case study of the reasoning process of the present invention (CIA-Net) on typical conflict samples. The case study of typical conflict samples demonstrates the conflict detection, CAMR weighting, PCAR routing, and final prediction process.

[0078] The preferred embodiment of this invention has approximately 5.3 million trainable parameters, an inference throughput of approximately 850 samples / second, and a single training session duration of approximately 30 minutes. Compared to the reference scheme, the number of parameters has increased, but the weighted F1 score has increased by 2.17 percentage points, indicating that this invention has a better performance-to-benefit ratio.

[0079] The above description is merely a preferred embodiment of the present invention. Those skilled in the art will understand that various modifications and improvements can be made without departing from the principles of the present invention. Several optional modifications are described below, all of which fall within the protection scope of the present invention.

[0080] The input modalities of this invention are not limited to text, voice, and video; they can also be extended to physiological signals (such as electrocardiogram and skin conductance), sensor signals, images, or other emotion-related modalities. When more modalities are used, the modal conflict detection module is correspondingly expanded to perform calculations on any two modalities, while other processing flows remain unchanged.

[0081] The modal encoder of the present invention is not limited to the specific pre-trained model listed in the embodiments, and can also be replaced by any encoding network capable of outputting semantic representation. For example, the text encoder can use alternative models such as XLNet and ELECTRA, and the speech encoder can use alternative models such as Whisper and WavLM.

[0082] The fusion structure in the alignment path of the present invention is not limited to the number of layers and dimensions described in the embodiments. Other attention networks with conflict bias injection mechanisms, such as linear attention, sparse attention and other variant structures, can also be used.

[0083] The order of evidence fusion in the conflict path of this invention can be dynamically adjusted according to modal priority, and is not limited to the order of fusion of text and speech first, and then video. For example, in scenarios with high video quality, video and speech evidence can be fused first.

[0084] The routing mechanism of this invention can output category-level, task-level, or sample-level weights according to the specific task requirements. In this embodiment, category-level routing is the preferred solution, but the use of sample-level or task-level routing in specific tasks is not excluded.

[0085] The semantic-level conflict signals of this invention are not limited to those generated by large language models, but can also be obtained through knowledge graph reasoning, rule engines, or dedicated semantic analyzers, as long as they can quantify the difference between the surface semantics and the deep intent of the text.

[0086] This invention can be widely applied in scenarios such as intelligent customer service and companion robots, psychological health assessment assistance, intelligent in-vehicle cockpits, online education, interview analysis, public service and safety monitoring, etc. Specifically, intelligent customer service and companion robots: by recognizing users' true emotional states, they provide a more humanized interactive experience and emotional support; psychological health assessment assistance: used for early screening and emotional monitoring of psychological problems such as depression and anxiety, assisting in clinical diagnosis; intelligent in-vehicle cockpits: recognizing drivers' fatigue, anger, and other emotional states, issuing timely warnings or adjusting the cockpit environment; online education: analyzing learners' emotional feedback, evaluating teaching effectiveness, and enabling personalized teaching adjustments; interview analysis and public service and safety monitoring: through multimodal behavioral understanding, assisting in talent assessment and public safety early warning. This invention combines strong algorithmic performance, good interpretability, and high engineering deployability, possessing clear industrial application value.

[0087] This invention provides an adaptive dual-path evidence fusion scheme for multimodal emotion recognition, addressing the problems of existing technologies that cannot distinguish between modality-consistent and modality-conflicting samples, lack dynamic quantification of modality reliability, force feature-level deep fusion in conflict scenarios leading to decreased recognition performance, and fail to allocate fine-grained fusion paths based on the sensitivity of different emotion categories to conflict. The scheme provided by this invention can fully utilize cross-modal complementary information in modality-consistent scenarios and prioritize the retention of independent discriminative evidence of reliable modalities in modality-conflicting scenarios, thereby improving the accuracy, robustness, and interpretability of multimodal emotion recognition.

[0088] Example 2 An adaptive dual-path evidence fusion system for multimodal emotion recognition is disclosed. This system corresponds to the method described in Embodiment 1. For a detailed description of the method, please refer to the corresponding description in the above method embodiments, which will not be repeated here. Figure 2 As shown, the system includes: The multimodal data acquisition module is used to acquire multimodal input data of the object to be identified; The feature encoding and unified mapping module is used to extract features and perform unified mapping on the multimodal input data to obtain unified features for each modality. The characterization and decoupling module is used to characterize and decouple the unified features of each mode, and obtain the mode-invariant features and mode-specific features of each mode respectively. The evidence and uncertainty estimation module is used to estimate the single-modal evidence of each modality and its corresponding cognitive uncertainty based on the modality-specific features of each modality. The modal conflict detection module is used to calculate the modal conflict vector based on the difference in modal-specific features between different modalities; The conflict-aware modal reweighting module is used to generate reliability weights for each modality based on the cognitive uncertainty and the modal conflict vector. The alignment path fusion module is used to perform cross-modal fusion of the modality-invariant features and the modality-specific features weighted by the reliability weights to obtain the alignment path prediction result; The conflict path evidence fusion module is used to perform hierarchical evidence fusion on the single-modal evidence after the reliability weight is back-modulated to obtain the conflict path prediction result. The class-wise adaptive routing module is used to generate class-wise path weights based on the modality conflict vector, the cognitive uncertainty, and the reliability weights, and to use the class-wise path weights to perform class-wise adaptive fusion of the alignment path prediction results and the conflict path prediction results to obtain the final emotion recognition result.

[0089] The alignment path fusion module is further configured to: An attention bias matrix is ​​constructed based on the feature-level conflict scores between pairs of modalities, and this bias matrix is ​​injected into the attention score of the multi-head self-attention mechanism to achieve cross-modal fusion of conflict modulation.

[0090] The conflict path evidence fusion module is further configured as follows: Belief vectors for each modality are generated based on the single-modal evidence for each modality and its effective uncertainty after inverse modulation. Following a pre-defined hierarchical order, the belief vectors of each modality are iteratively fused using the Dempster-Shafer evidence combination rule with upper limit constraints on the conflict coefficient.

[0091] The class-by-class adaptive routing module is further configured as follows: Generate an independent path weight for each emotion category; For each emotion category, the aligned path prediction result and the conflict path prediction result are weighted and fused according to their corresponding path weights.

[0092] This invention provides an adaptive dual-path evidence fusion framework. This framework constructs an unsupervised modal conflict detection mechanism by leveraging the distance relationships and cognitive uncertainty between modality-specific features, eliminating the need for additional conflict labels. Through a conflict-aware modality reweighting mechanism, it combines conflict signals and uncertainty signals to dynamically output reliability weights for each modality. Two parallel fusion paths are set: an alignment path and a conflict path. In the alignment path, a conflict-modulated cross-modal fusion Transformer is employed to suppress attention between high-conflict modalities. In the conflict path, hierarchical Dempster-Shafer evidence fusion is used, prioritizing the fusion of classification beliefs from reliable modalities. A class-wise adaptive routing mechanism is implemented, assigning fusion weights to the two paths for different sentiment categories, rather than using a single global gating. Optionally, offline inference features from a large language model can be introduced to construct semantic-level conflict signals within text modalities, enhancing the recognition of complex semantic phenomena such as irony, sarcasm, and metaphor.

[0093] Example 3 A computer device 800, such as Figure 17 As shown, the system includes a memory 810, a processor 820, and a computer program 830 stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of an adaptive dual-path evidence fusion method for multimodal emotion recognition. For a detailed description of the method, please refer to the corresponding description in the above method embodiments; it will not be repeated here.

[0094] Example 4 A computer-readable storage medium, such as Figure 18As shown, a computer program is stored thereon. When executed by a processor, the computer program implements the steps of an adaptive dual-path evidence fusion method for multimodal emotion recognition. For a detailed description of the method, please refer to the corresponding description in the above method embodiments, which will not be repeated here.

[0095] The number of devices and processing scale described herein are for the purpose of simplifying the description of the invention. Applications, modifications, and variations of the invention will be readily apparent to those skilled in the art.

[0096] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

[0097] The apparatus, computer device, and non-volatile computer storage medium and method provided in the embodiments of this specification are corresponding. Therefore, the apparatus, computer device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, computer device, and non-volatile computer storage medium will not be repeated here.

[0098] Those skilled in the art will also know that, besides implementing the controller in the form of purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the controller take the form of logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices included within it for implementing various functions can also be considered structures within that hardware component. Alternatively, the devices for implementing various functions can be considered as both software units implementing the method and structures within a hardware component.

[0099] The systems, apparatuses, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. For ease of description, the above apparatuses are described separately as various units based on their functions. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0100] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0102] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0103] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0104] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0105] This specification may be described in the general context of computer-executable instructions, such as program units, that are executed by a computer. Generally, program units include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification may also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program units may reside in local and remote computer storage media, including storage devices.

[0106] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0107] The above description is merely an embodiment of this specification and is not intended to limit the scope of one or more embodiments of this specification. Various modifications and variations can be made to one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of one or more embodiments of this specification.

Claims

1. An adaptive dual-path evidence fusion method for multimodal emotion recognition, characterized in that, Includes the following steps: Acquire multimodal input data of the object to be identified, wherein the multimodal input data includes at least two of the following: text data, voice data, and video data; Feature extraction and unified mapping are performed on the multimodal input data to obtain unified features for each modality; The unified features of each mode are characterized and decoupled to obtain the mode-invariant features and mode-specific features of each mode respectively; Based on the modality-specific features of each modality, the single-modal evidence of each modality and its corresponding cognitive uncertainty are estimated; The modal conflict vector is calculated based on the difference in modal-specific features between different modalities. Based on the cognitive uncertainty and the modal conflict vector, a reliability weight for each modality is generated; The modality-invariant features and the modality-specific features weighted by the reliability weights are input into the alignment path, and cross-modal fusion of conflict modulation is performed to obtain the alignment path prediction result. The single-modal evidence, after being inversely modulated by the reliability weight, is input into the conflict path, and hierarchical evidence fusion is performed to obtain the conflict path prediction result. Based on the modal conflict vector, the cognitive uncertainty, and the reliability weight, class-wise path weights are generated, and the class-wise path weights are used to perform class-wise adaptive fusion of the alignment path prediction results and the conflict path prediction results to obtain the final emotion recognition result. The step of generating class-specific path weights and using the class-specific path weights to perform class-specific adaptive fusion of the aligned path prediction results and the conflict path prediction results includes: The modal conflict vector, the cognitive uncertainty vector of each modality, and the reliability weight vector of each modality are concatenated. The concatenated vector is input into a multilayer perceptron network with a sigmoid function as the output layer, and a path weight between 0 and 1 is generated independently for each sentiment category. For each emotion category k, the final predicted probability is calculated by combining the conflict path prediction result and the alignment path prediction result using the formula... We get the weighted average, where, This refers to the path weight output by the multilayer perceptron network for the k-th emotion category, with a value ranging from 0 to 1. K represents the total number of emotion categories. This represents the final predicted probability that the object to be identified belongs to the k-th sentiment category; This represents the predicted probability of the k-th sentiment category in the conflict path prediction results; This represents the predicted probability of the k-th sentiment category in the alignment path prediction result; The steps for estimating the single-modal evidence for each modality and its corresponding cognitive uncertainty include: The modality-specific features of each modality are input into the evidence estimation network, which outputs a non-negative vector as the evidence parameter of the Dirichlet distribution. Adding 1 to the evidence parameters yields the concentration parameters of the Dirichlet distribution; Based on the concentration parameter, the total amount of evidence is calculated, and based on the total number of sentiment categories K, it is determined using the formula... Calculate the mode Cognitive uncertainty ,in This represents the total amount of evidence for a single modality.

2. The adaptive dual-path evidence fusion method for multimodal emotion recognition as described in claim 1, characterized in that, The step of calculating the modal conflict vector based on the difference degree of modal-specific features between different modalities further includes: Calculate the normalized cosine distance between modal-specific features of any two modes to obtain the feature-level conflict score; Based on the cognitive uncertainty of each of the two modalities, the feature-level conflict score is amplified to obtain the conflict score with enhanced uncertainty. The information of the conflict scores after the uncertainty enhancement is combined to form the modal conflict vector.

3. The adaptive dual-path evidence fusion method for multimodal emotion recognition as described in claim 2, characterized in that, The step of calculating the modal conflict vector also includes: A large language model is introduced to perform offline reasoning on text data, and the difference between the original text features and the semantic features obtained by reasoning is calculated as a semantic-level conflict signal. The semantic-level conflict signal and the conflict score with enhanced uncertainty are used together to form the modal conflict vector.

4. The adaptive dual-path evidence fusion method for multimodal emotion recognition as described in claim 1, characterized in that, The step of generating reliability weights for each modality based on the cognitive uncertainty and the modality conflict vector includes: The vector of cognitive uncertainty of each modality is concatenated with the modal conflict vector; The concatenated vector is input into a multilayer perceptron network with a Softmax function that includes a temperature parameter, and the reliability weights of each mode are output. In the alignment path, the reliability weight is used to modulate the modal specific features corresponding to forward modulation; in the conflict path, the reliability weight is used to modulate the effective uncertainty of the modality corresponding to backward modulation.

5. The adaptive dual-path evidence fusion method for multimodal emotion recognition as described in claim 1, characterized in that, The cross-modal fusion step of performing conflict modulation includes: The modality-invariant features of each modality and the modality-specific features weighted by reliability weights are concatenated to construct a multimodal feature sequence; Construct an attention bias matrix based on the feature-level conflict scores between pairwise modalities; During the multi-head self-attention mechanism calculation, the attention bias matrix is ​​injected into the attention score to suppress the attention weights between high-conflict mode pairs.

6. The adaptive dual-path evidence fusion method for multimodal emotion recognition as described in claim 1, characterized in that, The hierarchical evidence fusion step includes: Based on the single-mode evidence of each mode and its effective uncertainty after inverse modulation, a belief vector for each mode is generated, wherein the sum of the belief vector of each mode and its corresponding effective uncertainty is 1. Following a pre-defined hierarchical order, the belief vectors of at least two modalities are iteratively fused using the Dempster-Shafer evidence combination rule; In each fusion process, an upper limit constraint is set on the conflict coefficient in the combination rules to avoid the normalization factor being too large.

7. An adaptive dual-path evidence fusion system for multimodal emotion recognition, used to implement the method as described in claim 1, characterized in that, include: The multimodal data acquisition module is used to acquire multimodal input data of the object to be identified; The feature encoding and unified mapping module is used to extract features and perform unified mapping on the multimodal input data to obtain unified features for each modality. The characterization and decoupling module is used to characterize and decouple the unified features of each mode, and obtain the mode-invariant features and mode-specific features of each mode respectively. The evidence and uncertainty estimation module is used to estimate the single-modal evidence of each modality and its corresponding cognitive uncertainty based on the modality-specific features of each modality. The modal conflict detection module is used to calculate the modal conflict vector based on the difference in modal-specific features between different modalities; The conflict-aware modal reweighting module is used to generate reliability weights for each modality based on the cognitive uncertainty and the modal conflict vector. The alignment path fusion module is used to perform cross-modal fusion of the modality-invariant features and the modality-specific features weighted by the reliability weights to obtain the alignment path prediction result; The conflict path evidence fusion module is used to perform hierarchical evidence fusion on the single-modal evidence after the reliability weight is back-modulated to obtain the conflict path prediction result. The class-wise adaptive routing module is used to generate class-wise path weights based on the modality conflict vector, the cognitive uncertainty, and the reliability weights, and to use the class-wise path weights to perform class-wise adaptive fusion of the alignment path prediction results and the conflict path prediction results to obtain the final emotion recognition result.

8. The adaptive dual-path evidence fusion system for multimodal emotion recognition as described in claim 7, characterized in that, The alignment path fusion module is further configured to: An attention bias matrix is ​​constructed based on the feature-level conflict scores between pairs of modalities, and this bias matrix is ​​injected into the attention score of the multi-head self-attention mechanism to achieve cross-modal fusion of conflict modulation.

9. The adaptive dual-path evidence fusion system for multimodal emotion recognition as described in claim 7, characterized in that, The conflict path evidence fusion module is further configured as follows: Belief vectors for each modality are generated based on the single-modal evidence for each modality and its effective uncertainty after inverse modulation. Following a pre-defined hierarchical order, the belief vectors of each modality are iteratively fused using the Dempster-Shafer evidence combination rule with upper limit constraints on the conflict coefficient.

10. The adaptive dual-path evidence fusion system for multimodal emotion recognition as described in claim 7, characterized in that, The class-by-class adaptive routing module is further configured as follows: Generate an independent path weight for each emotion category; For each emotion category, the aligned path prediction result and the conflict path prediction result are weighted and fused according to their corresponding path weights.

11. A computer-readable storage medium, characterized in that, It stores program instructions that, when executed, implement the method as described in any one of claims 1 to 6.

Images