A multi-modal sentiment state recognition method based on quality-guided mixed experts

Abstract

The application discloses a kind of multi-modal affective state recognition methods based on quality guide hybrid expert, audio signal and visual signal are collected, timing coding is carried out, obtain audio timing representation and visual timing representation, cross-modal interaction representation is obtained by interactive modeling;According to audio timing representation and visual timing representation, obtain audio reliability score, visual reliability score;Four kinds of expert networks are constructed, and the corresponding expert quality prior is respectively;After audio timing representation, visual timing representation and cross-modal interaction representation are spliced, inject into gate network, generate initial routing score, quality corrected routing weight, the output of four kinds of expert networks is dynamically weighted and fused, and quality-guided multi-modal fusion representation is obtained;Timing pooling and classification processing are carried out, and the emotional state category of target object is output.The application can be widely applied in multi-modal interaction system, intelligent terminal, digital human system, emotional computing platform and emotional state analysis system and the like scene.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, multimodal signal processing, emotion computing, and behavioral state recognition, specifically to a multimodal emotion state recognition method based on quality-guided hybrid experts. Background Technology

[0002] With the development of intelligent interaction, online interviews, virtual customer service, digital human interaction, distance education, and human-computer collaborative systems, the automatic identification of the state of target objects based on multimodal behavioral signals such as audio, images, or video has significant application value. Compared with single-modal signals, multimodal signals can reflect the state changes of target objects from multiple perspectives, such as speech, facial expressions, head posture, eye behavior, and temporal actions, and are therefore widely used in tasks such as affective computing, behavior analysis, and state perception.

[0003] Existing multimodal state recognition methods typically employ feature concatenation, decision-level fusion, cross-modal attention, or deep fusion networks to jointly model audio and visual information. However, in real-world acquisition environments, audio modalities may be affected by environmental noise, prolonged silence, reduced speech energy, or unclear pronunciation, while visual modalities may be affected by facial occlusion, head movements, lighting variations, or unstable face tracking. These factors can cause continuous fluctuations in the reliability of different modalities at different time steps. Most existing methods assume that each modality has relatively stable availability throughout the sequence, or perform static fusion based solely on content features, lacking modeling of "which modality is reliable at the current time step, whether cross-modal fusion is suitable, and whether the influence of low-quality modalities should be reduced." Therefore, when a modality locally degrades, low-quality information can easily propagate into the fusion representation space, thereby reducing the stability and robustness of the state classification results.

[0004] Therefore, there is an urgent need in this field for a multimodal state classification technology that can simultaneously address dynamic fluctuations in audio and visual modal quality, achieve time-step-level adaptive reliability assessment, and coordinate robust fusion of single-modal experts, cross-modal fusion experts, and global baseline experts. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal emotion state recognition method based on quality-guided hybrid experts. This method enables the adaptive adjustment of the participation level of different experts according to the modal reliability state at the current time step, thereby preserving complementary audio and visual information while reducing the interference of low-quality modal information such as noise, occlusion, silence, and tracking instability on the fused representation, thus improving the robustness and stability of multimodal emotion state classification.

[0006] To achieve the above functions, this invention designs a multimodal emotion state recognition method based on quality-guided hybrid experts, which executes the following steps S1-S6 to complete the emotion state category judgment of the target object:

[0007] Step S1: Acquire the audio signal and visual signal of the target object, and extract features from the audio signal and visual signal respectively to obtain audio features and visual features. After preprocessing, obtain audio feature sequence and visual feature sequence.

[0008] Step S2: Perform temporal encoding on the audio feature sequence and the visual feature sequence respectively to obtain audio temporal representation and visual temporal representation. Then, perform interactive modeling on the audio temporal representation and visual temporal representation based on the bidirectional cross-modal attention mechanism to obtain cross-modal interactive representation.

[0009] Step S3: Based on the audio temporal representation and the visual quality index obtained from the visual signal processing, estimate the audio reliability score and visual reliability score at each time step to obtain the audio reliability score sequence and visual reliability score sequence for all time steps.

[0010] Step S4: Based on the audio reliability score and visual reliability score at each time step, construct four types of expert networks, including audio expert network, visual expert network, cross-modal fusion expert network and global safety net expert network, as well as corresponding expert quality priors;

[0011] Step S5: After concatenating the audio temporal representation, visual temporal representation, and cross-modal interaction representation, inject them into the gating network to generate an initial routing score. Combine this with the expert quality prior to obtain the quality-corrected routing weights. Based on the quality-corrected routing weights, dynamically weight and fuse the outputs of the audio expert network, visual expert network, cross-modal fusion expert network, and global safety net expert network to obtain a quality-guided multimodal fusion representation.

[0012] Step S6: Perform temporal pooling and classification processing on the quality-guided multimodal fusion representation, output the emotional state category of the target object, and complete the emotional state category judgment of the target object.

[0013] As a preferred technical solution of the present invention: In step S1, the audio features include at least one of acoustic features, deep audio representation, speech energy features, spectral features, fundamental frequency features, and prosodic features; the visual features include at least one of facial behavior features, facial key point features, head posture features, eye behavior features, facial expression dynamic features, and deep visual representation; after obtaining the audio features and visual features, they are preprocessed, including timestamp alignment, length unification, and serialization, to obtain an audio feature sequence. and visual feature sequences .

[0014] As a preferred technical solution of the present invention, the specific steps in step S2 are as follows:

[0015] Step S2.1: Input the audio feature sequence and visual feature sequence into the corresponding temporal encoders to obtain the audio temporal representation. and visual temporal representation ;

[0016] Step S2.2: Based on audio temporal representation and visual temporal representation, construct a bidirectional cross-modal attention mechanism to obtain cross-modal interaction representation: ,in, Representing cross-modal interaction representation, This indicates that audio information provides a supplementary representation of the visual state. This represents the supplementary representation of audio state by visual information. This represents the feature mapping function.

[0017] As a preferred embodiment of the present invention, the specific steps in step S3 are as follows:

[0018] Step S3.1: Based on the visual quality index obtained from the visual signal processing, calculate the visual reliability score at time step t as follows:

[0019] ;

[0020] in, This represents the visual reliability score at time step t. This refers to the visual quality index obtained during the visual signal processing process. This represents the lower limit of the visual reliability score;

[0021] Step S3.2: Input the audio temporal representation into the lightweight multilayer perceptron and calculate the audio reliability score at time step t:

[0022] ;

[0023] in, This represents the audio reliability score at time step t. This represents the Sigmoid activation function. and Let represent the learnable weight matrix and bias term of the lightweight multilayer perceptron, respectively. This represents the audio timing representation at time step t.

[0024] Finally, the audio reliability score sequence for all time steps is obtained. and visual reliability score sequence ; This represents the audio reliability score across all time steps. This represents the visual reliability score for all time steps, where T is the total number of time steps.

[0025] As a preferred technical solution of the present invention, the specific steps in step S4 are as follows:

[0026] Step S4.1: Based on the audio reliability score and visual reliability score at time step t, construct the expert quality prior:

[0027] ;

[0028] in, This represents the expert quality prior at time step t. This represents the audio expert quality prior at time step t. This represents the visual expert quality prior at time step t. This represents the cross-modal fusion expert quality prior at time step t. This represents the global guaranteed expert quality prior at time step t; Indicates a splicing operation;

[0029] The quality priors of each expert are as follows:

[0030] ;

[0031] in, This represents the audio reliability score at time step t. This represents the visual reliability score at time step t.

[0032] Step S4.2: Set up four types of expert networks, including an audio expert network, a visual expert network, a cross-modal fusion expert network, and a global safety net expert network. The inputs of the four types of expert networks are represented as follows:

[0033] ;

[0034] ;

[0035] in, , , , Let represent the inputs of the audio expert network, the visual expert network, the cross-modal fusion expert network, and the global safety net expert network at time step t, respectively. Represents the global representation mapping function; This represents the audio timing representation at time step t. Represents the visual temporal representation at time step t; Represents the cross-modal interaction representation at time step t; Indicates a splicing operation;

[0036] Step S4.3: The expert representations output by the four types of expert networks are respectively expressed as follows:

[0037] ;

[0038] in, The t-th time step Input to expert-like networks Indicates the first Expert-like networks Indicates the first The expert representation output by the expert network at time step t; Values The time indicates the audio expert network, Values The time represents a visual expert network, Values Time represents a cross-modal fusion expert network. Values The time indicates a global safety net expert network;

[0039] The four types of expert networks are implemented using at least one of the following: multilayer perceptron, convolutional neural network, recurrent neural network, or Transformer encoder.

[0040] As a preferred technical solution of the present invention, the specific steps in step S5 are as follows: Step S5.1: Concatenate the audio temporal representation, visual temporal representation, and cross-modal interaction representation to obtain the gating network input: ; in, This represents the gating network input at time step t; Step S5.2: Input the gating network Inputting the data into the gated network yields the initial routing scores for four types of expert networks: ; in, This represents the initial routing score at time step t. , , , The initial activation tendencies of the audio expert network, visual expert network, cross-modal fusion expert network, and global safety net expert network at time step t are respectively. and These represent the learnable weight matrix and bias term of the gated network, respectively. Step S5.3: Inject expert quality priors into the initial route scores using a logarithmic bias method to obtain the quality-corrected route scores: ; in, This represents the route score after quality correction at time step t. This represents the expert quality prior at time step t. This indicates the prior influence of mass on the strength coefficient. This represents a constant used to prevent numerical overflow; Step S5.4: Normalize the quality-corrected route scores to obtain the expert route weights: ; in, This represents the expert routing weight at time step t; Represents the Softmax function; Step S5.5: Dynamically weight and fuse the expert representations output by the four types of expert networks according to the expert routing weights to obtain the quality-guided multimodal fusion representation; The quality-guided multimodal fusion representation at time step t is expressed as: ; in, Let represent the quality-guided multimodal fusion representation at time step t. Indicates the first The expert routing weights of the expert network at time step t. Indicates the first The expert representation output by the expert network at time step t. , Values The time indicates the audio expert network, Values The time represents a visual expert network, Values Time represents a cross-modal fusion expert network. Values The time indicates the global safety net expert network.

[0041] As a preferred embodiment of the present invention, the specific steps of step S6 are as follows:

[0042] Temporal global average pooling is performed on the quality-guided multimodal fusion representation to obtain sample-level state representations; these sample-level state representations are then input into a classification head to obtain the probability distribution of the target object's emotional state category.

[0043] ;

[0044] in, This represents the sample-level state representation. Represents the probability distribution of emotional state categories. and Let these represent the learnable weight matrix and bias term of the classification head, respectively; based on the sentiment state category probability distribution, output the sentiment state category of the target object:

[0045] ;

[0046] in, This indicates the emotional state category of the target object in the output. This represents the probability corresponding to the i-th emotional state; the emotional state categories include at least two pre-defined emotional state categories based on the application scenario. This represents the i-th emotional state that causes the function to reach its maximum value.

[0047] The present invention also designs an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the program to implement the aforementioned multimodal emotion state recognition method based on quality-guided hybrid experts.

[0048] The present invention also designs a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multimodal emotion state recognition method based on quality-guided hybrid experts.

[0049] Beneficial effects: Compared with the prior art, the advantages of the present invention include:

[0050] This invention, under real-world noise and occlusion conditions, dynamically evaluates the reliability of audio and visual modalities at the same time step through time-step modal reliability estimation. This avoids the problem of existing multimodal fusion methods treating modal quality as a static attribute, thus improving the adaptability to modal quality fluctuations in real-world interaction scenarios. Furthermore, through a quality-guided hybrid expert dynamic fusion mechanism, the model can adaptively adjust the participation level of each expert based on the reliability status of different modalities. While preserving complementary information between audio and visual modes, it improves the stability of emotional state classification results. This invention can be applied to multimodal interaction systems, smart terminals, digital human systems, emotion computing platforms, and emotion state analysis systems. Attached Figure Description

[0051] Figure 1 This is a flowchart of a multimodal emotion state recognition method based on quality-guided hybrid experts according to an embodiment of the present invention;

[0052] Figure 2 This is a schematic diagram of the overall structure of the model provided according to an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of the modal quality sensing process provided according to an embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of a quality-guided hybrid expert fusion process provided according to an embodiment of the present invention. Detailed Implementation

[0055] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0056] This invention provides a multimodal emotion state recognition method based on quality-guided hybrid experts, referring to... Figure 1 Perform the following steps S1-S6 to complete the judgment of the target object's emotional state category:

[0057] Step S1: Acquire the audio signal and visual signal of the target object, and extract features from the audio signal and visual signal respectively to obtain audio features and visual features. After preprocessing, obtain audio feature sequence and visual feature sequence.

[0058] First, audio and visual signals of the target object during the interaction are acquired using microphones, cameras, smart terminals, interactive devices, or other multimodal acquisition devices. Audio signals may include information such as speech, pauses, tone changes, and energy changes, while visual signals may include facial expressions, head posture, eye movements, facial key point movements, and other observable behavioral information.

[0059] The audio features include at least one of acoustic features, deep audio representation, speech energy features, spectral features, fundamental frequency features, and prosodic features; the visual features include at least one of facial behavior features, facial key point features, head posture features, eye behavior features, facial expression dynamics features, and deep visual representation; after obtaining the audio and visual features, they are preprocessed, including timestamp alignment, length unification, and serialization, to obtain an audio feature sequence. and visual feature sequences .

[0060] Step S2: Perform temporal encoding on the audio feature sequence and the visual feature sequence respectively to obtain audio temporal representation and visual temporal representation. Then, perform interactive modeling on the audio temporal representation and visual temporal representation based on the bidirectional cross-modal attention mechanism to obtain cross-modal interactive representation.

[0061] Reference Figure 2 The specific steps in step S2 are as follows:

[0062] Step S2.1: Input the audio feature sequence and visual feature sequence into the corresponding temporal encoders to obtain the audio temporal representation. and visual temporal representation ;

[0063] The time encoder employs a one-dimensional temporal convolutional network and a bidirectional long short-term memory network; the one-dimensional temporal convolutional network is used to extract dynamic change patterns within a local time window, and the bidirectional long short-term memory network is used to model long-term dependencies between consecutive time steps.

[0064] Step S2.2: Based on audio temporal representation and visual temporal representation, construct a bidirectional cross-modal attention mechanism to obtain cross-modal interaction representation: ,in, Representing cross-modal interaction representation, This indicates that audio information provides a supplementary representation of the visual state. This represents the supplementary representation of audio state by visual information. This represents the feature mapping function.

[0065] Through the processing in step S2, the model can not only obtain the temporal features of the audio modality and the visual modality, but also obtain the complementary relationship between the two modalities, providing a basic representation for subsequent modality reliability estimation and quality-guided expert fusion.

[0066] Step S3: Based on the audio temporal representation and the visual quality index obtained from the visual signal processing, estimate the audio reliability score and visual reliability score at each time step to obtain the audio reliability score sequence and visual reliability score sequence for all time steps.

[0067] Reference Figure 2 , Figure 3 The specific steps in step S3 are as follows:

[0068] Step S3.1: Based on the visual quality index obtained from the visual signal processing, calculate the visual reliability score at time step t as follows:

[0069] ;

[0070] in, This represents the visual reliability score at time step t. The visual quality index obtained during the visual signal processing process includes, in the embodiments, at least one of detection confidence, face tracking stability, key point extraction confidence, visual feature integrity, or image quality index. This represents the lower limit of the visual reliability score;

[0071] Step S3.2: Input the audio temporal representation into the lightweight multilayer perceptron and calculate the audio reliability score at time step t:

[0072] ;

[0073] in, This represents the audio reliability score at time step t. This represents the Sigmoid activation function. and Let represent the learnable weight matrix and bias term of the lightweight multilayer perceptron, respectively. This represents the audio timing representation at time step t.

[0074] Finally, the audio reliability score sequence for all time steps is obtained. and visual reliability score sequence ; This represents the audio reliability score across all time steps. This represents the visual reliability score for all time steps, where T is the total number of time steps.

[0075] Step S4: Based on the audio reliability score and visual reliability score at each time step, construct four types of expert networks, including audio expert network, visual expert network, cross-modal fusion expert network and global safety net expert network, as well as corresponding expert quality priors;

[0076] Reference Figure 2 The specific steps in step S4 are as follows:

[0077] Step S4.1: Based on the audio reliability score and visual reliability score at time step t, construct the expert quality prior:

[0078] ;

[0079] in, This represents the expert quality prior at time step t. This represents the audio expert quality prior at time step t. This represents the visual expert quality prior at time step t. This represents the cross-modal fusion expert quality prior at time step t. This represents the global guaranteed expert quality prior at time step t; Indicates a splicing operation;

[0080] in, Used to represent the suitability of an audio expert network at time step t. Used to represent the applicability of a visual expert network at time step t. Used to represent the applicability of cross-modal fusion expert networks under the condition that both modes are reliable. Used to represent the basic participation level of a global baseline expert network under any modal quality state;

[0081] The quality priors of each expert are as follows:

[0082] ;

[0083] in, This represents the audio reliability score at time step t. This represents the visual reliability score at time step t.

[0084] Step S4.2: Set up four types of expert networks, including an audio expert network, a visual expert network, a cross-modal fusion expert network, and a global safety net expert network. The inputs of the four types of expert networks are represented as follows:

[0085] ;

[0086] ;

[0087] in, , , , Let represent the inputs of the audio expert network, the visual expert network, the cross-modal fusion expert network, and the global safety net expert network at time step t, respectively. Represents the global representation mapping function; This represents the audio timing representation at time step t. Represents the visual temporal representation at time step t; Represents the cross-modal interaction representation at time step t; Indicates a splicing operation;

[0088] Step S4.3: The expert representations output by the four types of expert networks are respectively expressed as follows:

[0089] ;

[0090] in, The t-th time step Input to expert-like networks Indicates the first Expert-like networks Indicates the first The expert representation output by the expert network at time step t; Values The time indicates the audio expert network, Values The time represents a visual expert network, Values Time represents a cross-modal fusion expert network. Values The time indicates a global safety net expert network;

[0091] The four types of expert networks are implemented using at least one of the following: multilayer perceptron, convolutional neural network, recurrent neural network, or Transformer encoder.

[0092] Step S5: After concatenating the audio temporal representation, visual temporal representation, and cross-modal interaction representation, inject them into the gating network to generate an initial routing score. Combine this with the expert quality prior to obtain the quality-corrected routing weights. Based on the quality-corrected routing weights, dynamically weight and fuse the outputs of the audio expert network, visual expert network, cross-modal fusion expert network, and global safety net expert network to obtain a quality-guided multimodal fusion representation.

[0093] Reference Figure 2 , Figure 4 The specific steps in step S5 are as follows:

[0094] Step S5.1: Concatenate the audio temporal representation, visual temporal representation, and cross-modal interaction representation to obtain the gating network input:

[0095] ;

[0096] in, This represents the gating network input at time step t;

[0097] Step S5.2: Input the gating network Inputting the data into the gated network yields the initial routing scores for four types of expert networks:

[0098] ;

[0099] in, This represents the initial routing score at time step t. , , , The initial activation tendencies of the audio expert network, visual expert network, cross-modal fusion expert network, and global safety net expert network at time step t are respectively. and These represent the learnable weight matrix and bias term of the gated network, respectively.

[0100] Step S5.3: Inject expert quality priors into the initial route scores using a logarithmic bias method to obtain the quality-corrected route scores:

[0101] ;

[0102] in, This represents the route score after quality correction at time step t. This represents the expert quality prior at time step t. This indicates the prior influence of mass on the strength coefficient. This represents a constant used to prevent numerical overflow;

[0103] Step S5.4: Normalize the quality-corrected route scores to obtain the expert route weights:

[0104] ;

[0105] in, This represents the expert routing weight at time step t; Represents the Softmax function;

[0106] Step S5.5: Dynamically weight and fuse the expert representations output by the four types of expert networks according to the expert routing weights to obtain the quality-guided multimodal fusion representation;

[0107] The quality-guided multimodal fusion representation at time step t is expressed as:

[0108] ;

[0109] in, Let represent the quality-guided multimodal fusion representation at time step t. Indicates the first The expert routing weights of the expert network at time step t. Indicates the first The expert representation output by the expert network at time step t. , Values The time indicates the audio expert network, Values The time represents a visual expert network, Values Time represents a cross-modal fusion expert network. Values The time indicates the global safety net expert network.

[0110] Step S6: Perform temporal pooling and classification processing on the quality-guided multimodal fusion representation, output the emotional state category of the target object, and complete the emotional state category judgment of the target object.

[0111] The specific steps of step S6 are as follows:

[0112] Temporal global average pooling is performed on the quality-guided multimodal fusion representation to obtain sample-level state representations; these sample-level state representations are then input into a classification head to obtain the probability distribution of the target object's emotional state category.

[0113] ;

[0114] in, This represents the sample-level state representation. Represents the probability distribution of emotional state categories. and These represent the learnable weight matrix and bias term of the classification head, respectively;

[0115] In the embodiments, temporal global average pooling includes at least one of global average pooling, global max pooling, attention pooling, or weighted pooling;

[0116] Based on the probability distribution of sentiment state categories, output the sentiment state category of the target object:

[0117] ;

[0118] in, This indicates the emotional state category of the target object in the output. This represents the probability corresponding to the i-th emotional state; the emotional state categories include at least two pre-defined emotional state categories based on the application scenario. This represents the i-th emotional state that causes the function to reach its maximum value.

[0119] During model training, the emotional state classification loss, quality-guided consistency loss, and anti-deactivation regularization term are jointly optimized. The emotional state classification loss is used to constrain the emotional state classification results, the quality-guided consistency loss is used to constrain the routing weight distribution of non-bottom-line experts to maintain a consistent trend with the corresponding expert quality prior distribution, and the anti-deactivation regularization term is used to prevent audio experts, visual experts, or cross-modal fusion experts from being in a low-activation state for a long time during training.

[0120] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the program, it implements the aforementioned multimodal emotion state recognition method based on quality-guided hybrid experts.

[0121] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned multimodal emotion state recognition method based on quality-guided hybrid experts.

[0122] To verify the effectiveness of the invention and the validity of the model framework, a quantitative analysis method was used. Experiments were conducted on the multimodal sentiment state classification model using the PyTorch framework on a computing platform with an NVIDIA GeForce RTX 4090 graphics card.

[0123] The experiment used precision, recall, and F1 score as evaluation metrics to quantify the model's classification performance across different sentiment states. The formulas for each metric are as follows:

[0124] ;

[0125] ;

[0126] ;

[0127] Wherein, TP represents the number of correctly classified positive samples, FP represents the number of negative samples misclassified as positive samples, and FN represents the number of positive samples misclassified as negative samples. A higher Precision value indicates a higher proportion of correctly classified sentiment state categories in the output of the method of this invention; a higher Recall value indicates a higher proportion of target sentiment state category samples that the method of this invention can identify; the F1 value is the harmonic mean of Precision and Recall, which comprehensively reflects the balance of classification performance.

[0128] Table 1 presents the Precision, Recall, and F1 scores of the method of this invention compared to other methods on multimodal datasets. This comparative experiment uses the English multimodal sentiment state dataset DAIC-WOZ and the Chinese multimodal sentiment state dataset SEARCH. In the experiment, each dataset contains audio and visual modal signals, covering features such as speech, facial expressions, head posture, and eye behavior of the target object in different emotional states, simulating noise, occlusion, silence, and signal quality fluctuations that may occur in real-world interaction scenarios. For each dataset, this embodiment first extracts features and temporally encodes the audio and visual signals, then calculates the expert quality prior at each time step, and dynamically weights and fuses them using a quality-guided hybrid expert network. The final output sentiment state category or state tendency score is used for comparison with the ground truth annotations to calculate the Precision, Recall, and F1 scores.

[0129] Table 1. Comparative Experimental Results

[0130]

[0131] The experimental results are shown in Table 1. To comprehensively verify the effectiveness and advancement of the quality-guided hybrid expert-based multimodal emotion state recognition method described in this invention, this embodiment introduces several mainstream multimodal fusion benchmark methods for comparative experiments. The multimodal fusion benchmark methods specifically include: Linear Matrix Factorization (LMF), Multimodal Transformer (MulT), Multimodal Invariant and Specific Representation Learning (MISA), Spatiotemporal Synchronous Transformer (STST), and Multimodal Bidirectional Latent Variable Method (TMBL).

[0132] Analysis of the experimental data in Table 1 shows that traditional multimodal fusion methods (such as LMF and MulT) cannot effectively cope with the local quality degradation of multimodal signals over time in real-world scenarios due to their static fusion strategies or globally uniform attention weight allocation. When faced with complex samples containing noise, silence, or visual occlusion, they easily introduce invalid interference information, leading to a significant decrease in classification precision and recall. In contrast, the method of this invention significantly outperforms all the aforementioned comparative methods in terms of precision, recall, and F1 scores on multimodal sentiment datasets such as DAIC-WOZ and SEARCH.

[0133] In particular, compared with the benchmark method TMBL, which has the closest performance, the method of this invention achieves substantial improvements of 2.6% and 1.1% in various core evaluation metrics, respectively. This result fully demonstrates that by introducing a modality quality perception module to dynamically estimate the reliability score at the time step level and injecting it as an explicit quality prior into the architecture design of the hybrid expert gating network, the present invention can extremely effectively suppress the negative interference of low-quality modalities when single-modal signals fluctuate or degrade, and significantly enhance the classification stability and accuracy of the multimodal emotion state recognition system in real-world robust environments.

[0134] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A multimodal emotion state recognition method based on quality-guided hybrid experts, characterized in that, Perform the following steps S1-S6 to complete the judgment of the target object's emotional state category: Step S1: Acquire the audio signal and visual signal of the target object, and extract features from the audio signal and visual signal respectively to obtain audio features and visual features. After preprocessing, obtain audio feature sequence and visual feature sequence. Step S2: Perform temporal encoding on the audio feature sequence and the visual feature sequence respectively to obtain audio temporal representation and visual temporal representation. Then, perform interactive modeling on the audio temporal representation and visual temporal representation based on the bidirectional cross-modal attention mechanism to obtain cross-modal interactive representation. Step S3: Based on the audio temporal representation and the visual quality index obtained from the visual signal processing, estimate the audio reliability score and visual reliability score at each time step to obtain the audio reliability score sequence and visual reliability score sequence for all time steps. The specific steps in step S3 are as follows: Step S3.1: Based on the visual quality index obtained from the visual signal processing, calculate the visual reliability score at time step t as follows: ； in, This represents the visual reliability score at time step t. This refers to the visual quality index obtained during the visual signal processing process. This represents the lower limit of the visual reliability score; Step S3.2: Input the audio temporal representation into the lightweight multilayer perceptron and calculate the audio reliability score at time step t: ； in, This represents the audio reliability score at time step t. This represents the Sigmoid activation function. and Let represent the learnable weight matrix and bias term of the lightweight multilayer perceptron, respectively. This represents the audio timing representation at time step t. Finally, the audio reliability score sequence for all time steps is obtained. and visual reliability score sequence ; This represents the audio reliability score across all time steps. This represents the visual reliability score for all time steps, where T is the total number of time steps. Step S4: Based on the audio reliability score and visual reliability score at each time step, construct four types of expert networks, including audio expert network, visual expert network, cross-modal fusion expert network and global safety net expert network, as well as corresponding expert quality priors; The specific steps in step S4 are as follows: Step S4.1: Based on the audio reliability score and visual reliability score at time step t, construct the expert quality prior: ； in, This represents the expert quality prior at time step t. This represents the audio expert quality prior at time step t. This represents the visual expert quality prior at time step t. This represents the cross-modal fusion expert quality prior at time step t. This represents the global guaranteed expert quality prior at time step t; Indicates a splicing operation; The quality priors of each expert are as follows: ； in, This represents the audio reliability score at time step t. This represents the visual reliability score at time step t. Step S4.2: Set up four types of expert networks, including an audio expert network, a visual expert network, a cross-modal fusion expert network, and a global safety net expert network. The inputs of the four types of expert networks are represented as follows: ； ； in, , , , Let represent the inputs of the audio expert network, the visual expert network, the cross-modal fusion expert network, and the global safety net expert network at time step t, respectively. This represents the global representation mapping function; This represents the audio timing representation at time step t. Represents the visual temporal representation at time step t; Represents the cross-modal interaction representation at time step t; Indicates a splicing operation; Step S4.3: The expert representations output by the four types of expert networks are respectively expressed as follows: ； in, The t-th time step Input to expert-like networks Indicates the first Expert-like networks Indicates the first The expert representation output by the expert network at time step t; Values The time indicates the audio expert network, Values The time represents a visual expert network, Values Time represents a cross-modal fusion expert network. Values The time indicates a global safety net expert network; The four types of expert networks are implemented using at least one of the following: multilayer perceptron, convolutional neural network, recurrent neural network, or Transformer encoder. Step S5: After concatenating the audio temporal representation, visual temporal representation, and cross-modal interaction representation, inject them into the gating network to generate an initial routing score. Combine this with the expert quality prior to obtain the quality-corrected routing weights. Based on the quality-corrected routing weights, dynamically weight and fuse the outputs of the audio expert network, visual expert network, cross-modal fusion expert network, and global safety net expert network to obtain a quality-guided multimodal fusion representation. Step S6: Perform temporal pooling and classification processing on the quality-guided multimodal fusion representation, output the emotional state category of the target object, and complete the emotional state category judgment of the target object.

2. The multimodal emotion state recognition method based on quality-guided hybrid experts according to claim 1, characterized in that, In step S1, the audio features include at least one of acoustic features, deep audio representation, speech energy features, spectral features, fundamental frequency features, and prosodic features; the visual features include at least one of facial behavior features, facial key point features, head posture features, eye behavior features, facial expression dynamics features, and deep visual representation; after obtaining the audio features and visual features, they are preprocessed, including timestamp alignment, length unification, and serialization, to obtain an audio feature sequence. and visual feature sequences .

3. The multimodal emotion state recognition method based on quality-guided hybrid experts according to claim 1, characterized in that, The specific steps in step S2 are as follows: Step S2.1: Input the audio feature sequence and visual feature sequence into the corresponding temporal encoders to obtain the audio temporal representation. and visual temporal representation ; Step S2.2: Based on audio temporal representation and visual temporal representation, construct a bidirectional cross-modal attention mechanism to obtain cross-modal interaction representation: ,in, Representing cross-modal interaction representation, This indicates that audio information provides a supplementary representation of the visual state. This represents the supplementary representation of audio state by visual information. This represents the feature mapping function.

4. The multimodal emotion state recognition method based on quality-guided hybrid experts according to claim 3, characterized in that, The specific steps in step S5 are as follows: Step S5.1: Concatenate the audio temporal representation, visual temporal representation, and cross-modal interaction representation to obtain the gating network input: ； in, This represents the gating network input at time step t; Step S5.2: Input the gating network Inputting the data into the gated network yields the initial routing scores for four types of expert networks: ； in, This represents the initial routing score at time step t. , , , The initial activation tendencies of the audio expert network, visual expert network, cross-modal fusion expert network, and global safety net expert network at time step t are respectively. and These represent the learnable weight matrix and bias term of the gated network, respectively. Step S5.3: Inject expert quality priors into the initial route scores using a logarithmic bias method to obtain the quality-corrected route scores: ； in, This represents the route score after quality correction at time step t. This represents the expert quality prior at time step t. This indicates the prior influence of mass on the strength coefficient. This represents a constant used to prevent numerical overflow; Step S5.4: Normalize the quality-corrected route scores to obtain the expert route weights: ； in, This represents the expert routing weight at time step t; This represents the Softmax function; Step S5.5: Dynamically weight and fuse the expert representations output by the four types of expert networks according to the expert routing weights to obtain the quality-guided multimodal fusion representation; The quality-guided multimodal fusion representation at time step t is expressed as: ； in, Let represent the quality-guided multimodal fusion representation at time step t. Indicates the first The expert routing weights of the expert network at time step t. Indicates the first The expert representation output by the expert network at time step t. , Values The time indicates the audio expert network, Values The time represents a visual expert network, Values Time represents a cross-modal fusion expert network. Values The time indicates the global safety net expert network.

5. The multimodal emotion state recognition method based on quality-guided hybrid experts according to claim 4, characterized in that, The specific steps of step S6 are as follows: Temporal global average pooling is performed on the quality-guided multimodal fusion representation to obtain sample-level state representations; these sample-level state representations are then input into a classification head to obtain the probability distribution of the target object's emotional state category. ； in, This represents the sample-level state representation. Represents the probability distribution of emotional state categories. and Let these represent the learnable weight matrix and bias term of the classification head, respectively; based on the sentiment state category probability distribution, output the sentiment state category of the target object: ； in, This indicates the emotional state category of the target object in the output. This represents the probability corresponding to the i-th emotional state; the emotional state categories include at least two pre-defined emotional state categories based on the application scenario. This represents the i-th emotional state that causes the function to reach its maximum value.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the program, it implements the multimodal emotion state recognition method based on quality-guided hybrid experts as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the multimodal emotion state recognition method based on quality-guided hybrid experts as described in any one of claims 1 to 5.

Images