Multi-modal sentiment recognition method and system based on gating feature selection and expert modeling

Abstract

The application discloses a multimodal emotion recognition method and system based on gating feature selection and expert modeling, and the method comprises the following steps: after obtaining original speech signals and original texts, different levels of speech features and text features are extracted, the most suitable speech features and text features are dynamically selected from the speech features and text features of different levels through a corresponding gating feature selection network, and the selected speech features and text features are weighted and fused to obtain fused speech features and fused text features, and then the multimodal fusion features are obtained by splicing the fused speech features and the fused text features; information of different modes in the multimodal fusion features is further captured through a modal mixed expert Transformer module, and then the information is fused again to obtain new multimodal fusion features, and then the new multimodal fusion features are processed through an expert model in the mixed expert Transformer module which is most suitable for processing the new multimodal fusion features, so that a final prediction result is obtained, and the emotion state is predicted based on the prediction result. The application improves the prediction accuracy of emotions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer information, in particular to a multi-modal emotion recognition method and system based on gating feature selection and expert modeling. BACKGROUND

[0002] Multi-modal emotion recognition systems automatically perceive human expressed emotions through various signals such as speech, text, and visual signals, and play a crucial role in human-computer interaction systems. Due to its wide and valuable applications, multi-modal emotion recognition systems have been an active research topic for the past few decades, especially through the use of Transformers. Typical works include early summation and early concatenation. Both methods are simple and easy to implement, but they largely ignore the information alignment between different heterogeneous signals.

[0003] To address this issue, more and more research is devoted to more complex and efficient multi-modal emotion recognition designs to explore the inherent interactions and synergies between different signals. These methods usually involve cross-attention and hierarchical attention (also known as model / intermediate fusion strategies). Specifically, for cross-attention, it exchanges query embeddings between two independent Transformer streams (each stream corresponds to one modality). By doing so, each modality conditions on the other modality, so it can implicitly learn cross-modal contextual information. However, this approach cannot extract intra-modal information because there is no self-attention in each modality. For hierarchical attention, multi-modal inputs are encoded by independent Transformer streams, and their outputs are then concatenated and fused by a subsequent Transformer to form a merged stream. This approach preserves the independence of single-modal representations and captures cross-modal interactions.

[0004] However, the above methods face challenges in dealing with cross-modal generality and flexibility. Moreover, most methods also overlook the importance of feature selection. For example, extracting features from fixed layers of pre-trained models for subsequent multi-modal fusion limits the adaptability and performance of the model on different datasets. The selection of fixed layer features may not fully capture the key information in the task and dataset, thus affecting the overall effectiveness of the model. SUMMARY

[0005] The technical problem to be solved by the present application: In view of the above problems of the prior art, a multi-modal emotion recognition method and system based on gating feature selection and expert modeling is provided, which dynamically selects and fuses layer features of pre-trained models through a gating mechanism, and combines expert models for intra-modal and specific task modeling and fusion, thereby improving the prediction accuracy of emotions.

[0006] To solve the above technical problems, the technical scheme adopted by the present application is:

[0007] A multi-modal sentiment recognition method based on gating feature selection and expert modeling, comprising the following steps:

[0008] Obtain the original speech signal and the original text;

[0009] Extract different levels of speech features and text features from the original speech signal and the original text respectively, dynamically select the most suitable speech features and text features from different levels of speech features and text features through the corresponding gating feature selection network of speech and text respectively, and weight and fuse the selected speech features and text features respectively to obtain fused speech features and fused text features, and splice the fused speech features and the fused text features to obtain multi-modal fusion features;

[0010] Further capture the information of different modalities in the multi-modal fusion features through the modal expert model of the modal mixed expert Transformer module, and then fuse them again to obtain new multi-modal fusion features, and then process the new multi-modal fusion features through the expert model in the mixed expert Transformer module that is most suitable for processing the new multi-modal fusion features to obtain the final prediction result;

[0011] Predict the sentiment state according to the prediction result.

[0012] Further, when extracting different levels of speech features and text features from the original speech signal and the original text respectively, it specifically includes:

[0013] Input the original speech signal into a pre-trained speech feature extraction model to obtain speech features of all levels;

[0014] Input the original text into a pre-trained text feature extraction model to obtain text features of all levels.

[0015] Further, when dynamically selecting the most suitable speech features and text features from different levels of speech features and text features through the corresponding gating feature selection network of speech and text respectively, and weighting and fusing the selected speech features and text features, both include:

[0016] Input all levels of features into a linear layer, and then input the output of the linear layer into a softmax layer to obtain the weight probability of each level of feature;

[0017] According to the order from large to small of the weight probability, a specified number of features are selected in turn, the weight probability value of the selected features is normalized to obtain the weighted probability of the selected features, and the selected features are input into a linear layer after being weighted according to the weighted probability to adjust the feature dimension, thereby obtaining the fused features.

[0018] Further, the modal mixed expert Transformer module includes 3 layers of modal mixed expert Transformer networks, each layer of modal mixed expert Transformer network includes 2 layers of modal mixed expert Transformer, and when the information of different modalities in the multi-modal fusion features is further captured and then fused again by the modal expert model of the modal mixed expert Transformer module, specifically including:

[0019] The output of the first layer of modal mixed expert Transformer is obtained by further capturing the information of different modalities in the multi-modal fusion features by the modal expert model of the first layer of modal mixed expert Transformer and then fusing again.

[0020] The output of the first layer of modal mixed expert Transformer is input into the second layer of modal mixed expert Transformer, and the output of the second layer of modal mixed expert Transformer is obtained by further capturing the information of different modalities in the multi-modal fusion features by the modal expert model of the second layer of modal mixed expert Transformer and then fusing again.

[0021] The output of the second layer of modal mixed expert Transformer is re-divided into text features and speech features.

[0022] After the speech features are input into a merging module, the output of the merging module is spliced with the text features to form the input of the next layer of modal mixed expert Transformer network, and the merging module is composed of an adaptive average pooling layer and a linear layer.

[0023] Further, when the information of different modalities in the multi-modal fusion features is further captured and then fused again by the modal expert model of the first layer of modal mixed expert Transformer and the modal expert model of the second layer of modal mixed expert Transformer, both include:

[0024] The multi-modal fusion features are input into a multi-head attention mechanism.

[0025] The output of the multi-head attention mechanism is divided into text features and speech features according to the order of splicing the multi-modal fusion features.

[0026] input the text features into the text modality expert model and input the speech features into the speech modality expert model;

[0027] concatenate the outputs of the text modality expert model and the speech modality expert model in order to form the input of the next layer of the modality mixed expert Transformer.

[0028] Further, when the expert model most suitable for processing the new multi-modal fusion feature in the mixed expert Transformer module processes the new multi-modal fusion feature, the method comprises:

[0029] inputting the multi-modal fusion feature into the multi-head attention mechanism;

[0030] inputting the output of the multi-head attention mechanism into a gating network to obtain the weight distribution of each expert model;

[0031] selecting the most suitable expert model from all expert models according to the weight assigned by the gating network, taking the multi-modal fusion feature as the input data of the selected expert model, obtaining the output result of the selected expert model, and performing weighted summation according to the weight assigned by the gating network to obtain the final prediction result.

[0032] Further, when predicting the emotional state according to the prediction result, the method specifically comprises inputting the prediction result into an emotion classification module to obtain the probability of different emotional categories, and the emotion classification module is composed of two linear layers and a ReLU activation function.

[0033] Further, after predicting the emotional state according to the prediction result, the method further comprises designing a loss function, and the expression is as follows:

[0034]

[0035] wherein, L represents the loss function, is a cross-entropy loss function, is an expert model load balancing loss.

[0036] Further, the cross-entropy loss function and the expert model load balancing loss have the following expressions:

[0037]

[0038]

[0039] C represents the number of emotional categories, represents a real label, probabilities of different emotion classes representing predicted emotion states, N represents the number of expert models in the mixed expert Transformer module, probabilities of the i-th expert model being selected.

[0040] The application further provides a multi-modal emotion recognition system based on gated feature selection and expert modeling, comprising a microprocessor and a computer readable storage medium connected to each other, the microprocessor being programmed or configured to perform any one of the multi-modal emotion recognition methods based on gated feature selection and expert modeling.

[0041] Compared with the prior art, the application has the following advantages:

[0042] The application dynamically selects and fuses the layer features of the pre-training model through the gating mechanism, and combines the modal mixed expert Transformer module and the mixed expert Transformer module to perform more detailed modeling and fusion, thereby improving the accuracy of emotion recognition. In addition, the application can unify the architecture of single modal and multi-modal into one model by converting different modal experts, thereby improving the universality of the model. In addition, the application further designs a hierarchical merging layer for reducing the information redundancy and computational load of the speech modal. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 The method of the embodiment of the application is shown in the flowchart.

[0044] Figure 2 The detailed flowchart of the gated feature selection network in the method of the embodiment of the application is shown.

[0045] Figure 3 The detailed flowchart of the modal mixed expert Transformer module in the method of the embodiment of the application is shown.

[0046] Figure 4 The flowchart of the mixed expert Transformer module in the method of the embodiment of the application is shown. DETAILED DESCRIPTION

[0047] The application will be further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the application is not limited in this way.

[0048] Before introducing the specific embodiments of the application, the related concepts or terms are described.

[0049] Features: Features refer to the key information in data that describes certain attributes or patterns. In machine learning and deep learning, features are the basis for models to recognize and classify input data. Features can be manually designed or automatically extracted through data-driven methods, such as edges and color distribution in images, or pitch and spectrum in speech signals. Features are composed of multiple vectors.

[0050] Feature extraction model: A feature extraction model is an algorithm or neural network specifically designed to extract useful features from raw data. It automatically learns patterns in raw data and transforms them into feature representations suitable for downstream tasks such as classification, regression, etc. Common feature extraction models include self-supervised learning-based models such as wav2vec2.0, WavLM for speech processing, and BERT, Roberta for natural language processing.

[0051] All layer features extracted from the feature extraction model: Modern deep learning models are usually composed of multiple layers of neural networks such as Transformers, each layer learning different levels of data representation. All layer features of the feature extraction model refer to the feature representations generated by each layer in the model, which gradually enrich from low-level shallow features such as local patterns or edges to high-level deep features such as global semantic information. Combining these hierarchical features helps enhance the generalization ability of the model.

[0052] Modality: Modality refers to different forms of data representation, such as images, speech, text, etc., which represent different sources of information and ways of perception. In multi-modal learning, models can better understand and analyze complex tasks by processing data from multiple modalities.

[0053] Expert model: Refers to a set of specialized sub-models or networks, each composed of two linear layers, which perform better when processing specific types of input or specific tasks.

[0054] Weighted probabilities: In feature fusion, weighted probabilities are a way to calculate the relative importance of each feature in the fused feature by normalizing the probability values of the top three features with the highest probability. These weighted probabilities reflect the contribution of each feature, allowing the selected features to be weighted and summed according to these weights during feature fusion, resulting in a more comprehensive and effective feature representation. This method can highlight important features and improve the performance of the model in downstream tasks.

[0055] Example one

[0056] The embodiment proposes a multimodal emotion recognition method based on gating feature selection and expert modeling. The method fully utilizes features at different levels and uses expert models to model and fuse multimodal features more carefully, so that the information of different modalities is more fully utilized, and the prediction accuracy of emotions is improved.

[0057] As shown in Figure 1 The method of the embodiment includes the following three stages:

[0058] S1) Feature extraction stage, which uses a pre-trained feature extraction model to extract features at all levels, then uses a gating feature selection network to select the most suitable layer features, and uses weighted probability weighting to make them into fused features. This stage specifically includes:

[0059] S11) Obtain the original speech signal and the original text, and extract speech features and text features at different levels from the original speech signal and the original text respectively;

[0060] S12) Dynamically select the most suitable speech features and text features from different levels of speech features and text features through the corresponding gating feature selection network of speech and text respectively, and fuse the selected speech features and text features respectively to obtain fused speech features and fused text features. The fused speech features and the fused text features are spliced to obtain the multimodal fusion features;

[0061] S2) Expert modeling stage, which replaces the feed-forward neural network (Feed-Forward Neural Network) in the original Transformer network with different expert models, which can model and fuse the information of different modalities more carefully, and can also assign a special expert to each feature vector in the multimodal input, so that each expert can focus on the emotion task and provide more effective information for the emotion classification task. This stage specifically includes:

[0062] S21) Further capture the information of different modalities in the multimodal fusion features through the modal expert model of the modal mixed expert Transformer module and then fuse them again, so as to model and fuse the information of different modalities (such as text and speech) more carefully to obtain new multimodal fusion features; In the modal mixed expert Transformer module in the embodiment, the modal expert model is trained to focus on a specific modality (such as vision, hearing, text, etc.). In this way, the model can fully utilize the information from different modalities and improve the overall performance;

[0063] S22) The new multi-modal fusion features are processed by mixing the expert models in the mixed expert Transformer module that are most suitable for processing the new multi-modal fusion features, to obtain the final prediction result of the model; in the mixed expert Transformer module in this embodiment, the role of the expert model is to assign a dedicated expert to each feature vector in the multi-modal input, so as to make full use of the information of different modalities. Each expert can focus on the processing of a specific modality or feature, and the most suitable expert is dynamically selected for response through a gating mechanism. This method not only improves the performance of the model on each modality, but also enhances the comprehensive understanding ability of the model for complex tasks. In addition, the cooperation of multiple experts can improve the robustness of the model, reduce the bias caused by a single model, and optimize the computational efficiency.

[0064] S3) Classification and loss calculation stage, which classifies the output generated in the expert modeling stage through the emotion classification module, and uses the corresponding loss function to constrain the gradient update of the model, specifically including:

[0065] S31) Predicting the emotional state according to the prediction result, specifically inputting the output generated in the expert modeling stage, i.e., the final prediction result of the model, into the emotion classification module, and the emotion classification module outputs the probability of different emotional categories;

[0066] S32) Establishing a classification loss and an expert load balancing loss to design a loss function to constrain the gradient update of the model during model training.

[0067] The steps will be described in detail below.

[0068] Step S11 of this embodiment aims to extract the features of all levels of the pre-trained model and input them into the gating feature selection network. When extracting different levels of speech features and text features from the original speech signal and the original text, respectively, the pre-trained speech feature extraction model and the text feature extraction model are used for feature extraction, specifically including the following steps:

[0069] S111) Input the original speech signal A into the pre-trained speech feature extraction model to obtain speech features of all levels where n represents the number of speech features contained;

[0070] S112) Input the original text T into the pre-trained text feature extraction model to obtain text features of all levels where m represents the number of text features contained.

[0071] In this embodiment, the speech feature extraction model uses a pre-trained speech encoder WavLM-large, and how to use the speech encoder for speech feature extraction is known to those skilled in the art, and the present embodiment will not be repeated. Through the speech feature extraction process, a total of 24 levels of speech features are extracted, i.e. n = 24. The text feature extraction model uses a pre-trained text encoder Roberta-large, and how to use the text encoder for text feature extraction is known to those skilled in the art, and the present embodiment will not be repeated. Through the text feature extraction process, a total of 24 levels of text features are extracted, i.e. n = 24.

[0072] Step S12 of the present embodiment aims to dynamically select the most suitable speech and text features through the gating feature selection network, and obtain the fused speech and text features through weighted fusion. The most suitable speech and text features are dynamically selected from different levels of speech features and text features by the speech and text corresponding gating feature selection network respectively, and the selected speech and text features are weighted and fused respectively, as shown in Figure 2 , which includes the following steps:

[0073] S121) input all levels of features into a linear layer, then input the output of the linear layer into a softmax layer to obtain the weight probability of each level of feature; S122) select a specified number of features (in the present embodiment, the specified number is 3) in order according to the weight probability from large to small, normalize the weight probability value of the selected features to obtain the weighting probability of the selected features, and input the weighted features into a linear layer to adjust the feature dimension to obtain the fused features.

[0074] Specifically, the obtained all levels of speech features are input into a linear layer, and then the output of the linear layer is input into a softmax layer to obtain the weight probability of each level of speech feature; according to the weight probability, the top 3 features with the highest weight probability are selected, and the weight probability of the three features is normalized to obtain the weighting probability of the features, and the fused speech features are obtained by weighting the three layer features through a linear layer to adjust the feature dimension .

[0075] Similarly, the obtained all levels of text features are input into a linear layer, and then the output of the linear layer is input into a softmax layer to obtain the weight probability of each level of text feature; according to the weight probability, the top 3 features with the highest weight probability are selected, and the weight probability of the three features is normalized to obtain the weighting probability of the features, and the fused text features are obtained by weighting the three layer features through a linear layer to adjust the feature dimension .

[0076] The above process expression is as follows:

[0077]

[0078]

[0079]

[0080]

[0081]

[0082]

[0083] wherein and are weight and bias matrices of linear layers, used to obtain the weight probability of each level feature. The Select() function represents selecting the top 3 probabilities and their corresponding layer features, represent all levels of speech features or text features. and are also weight and bias matrices of linear layers, used to adjust the dimension of the feature.

[0084] In step S12 of the embodiment, when the fused speech features and the fused text features are spliced to obtain the multi-modal fusion features, the obtained speech fusion features and the text fusion features are spliced to obtain the multi-modal fusion features , wherein the text features are in the front and the speech features are in the back.

[0085] In the embodiment, the modal mixing expert Transformer module includes 3 layers of modal mixing expert Transformer networks, each layer of modal mixing expert Transformer network includes 2 layers of modal mixing expert Transformer, and a merging layer is further added after each layer of modal mixing expert Transformer network to gradually reduce the length of the speech features, reduce the calculation amount, and gradually condense the semantic information of the speech. Step S21 aims to input the fusion features into the modal mixing expert Transformer module to model different modalities (such as text and speech) more carefully, and further capture the information of different modalities in the multi-modal fusion features through the modal expert model of the modal mixing expert Transformer module before re-fusion, which specifically includes:

[0086] S211) The information of different modalities in the multi-modal fusion feature is further captured by the modal expert model of the first layer modal mixed expert Transformer, and then fused again to obtain the output of the first layer modal mixed expert Transformer;

[0087] S212) The output of the first layer modal mixed expert Transformer is input into the second layer modal mixed expert Transformer, the information of different modalities in the multi-modal fusion feature is further captured by the modal expert model of the second layer modal mixed expert Transformer, and then fused again to obtain the output of the second layer modal mixed expert Transformer. Through the stacking result, the potential speech information in the multi-modal fusion feature can be further captured;

[0088] S213) The output of the second layer modal mixed expert Transformer is re-divided into text features and speech features;

[0089] S214) The speech features are input into a merging module, and the output of the merging module is spliced with the text features to form the input of the next layer modal mixed expert Transformer network, wherein the splicing order is text features first and speech features last. The merging module is composed of an adaptive average pooling layer and a linear layer, which is used to reduce the length of the speech features, reduce the overall calculation amount of the model, and condense the speech features to reduce information redundancy.

[0090] The above merging process is expressed as follows:

[0091]

[0092]

[0093] wherein, represents 2D uniform pooling for reducing the length of the speech features, and the linear layer added subsequently is used to map the pooled features to enhance the expression ability of the model, and the weight matrix and the bias matrix thereof are and , and finally represents layer normalization, which is used to normalize the features to accelerate the training, stabilize the gradient and improve the generalization ability of the model.

[0094] In this embodiment, when the information of different modalities in the multi-modal fusion feature is further captured by the modal expert model of the first layer modal mixed expert Transformer and then fused again, and when the information of different modalities in the multi-modal fusion feature is further captured by the modal expert model of the second layer modal mixed expert Transformer and then fused again, both are as follows: Figure 3As shown, comprising the following steps:

[0095] S211) input the multi-modal fusion feature into the multi-head attention mechanism;

[0096] S212) divide the output of the multi-head attention mechanism according to the order in which the multi-modal fusion feature is spliced, and redivide it into text features and speech features;

[0097] S213) input the text features into the text modal expert model and the speech features into the speech modal expert model;

[0098] S214) splice the outputs of the text modal expert model and the speech modal expert model in order to form the input of the next layer of modal mixed expert Transformer, and splice the text features in front and the speech features in back.

[0099] The above process expression is as follows:

[0100] (1)

[0101] (2)

[0102] (3)

[0103] (4)

[0104] Where MSA represents the multi-head attention mechanism, represents the modal mixed expert Transformer of the current layer, represents the modal mixed expert Transformer of the previous layer, so represents the output of the modal mixed expert Transformer of the previous layer, and represents the output of the MSA module in the modal mixed expert Transformer of the current layer, represents the length of the text feature, also represents the connection position of the text feature and the speech feature in the multi-modal fusion feature, so from position split can represent the text feature, and can represent the speech feature. and represent the outputs of the text modal expert model and the speech modal expert model, respectively. represents the output of the modal mixed expert Transformer of the current layer, which is also the input of the modal mixed expert Transformer of the next layer. and represent the text and speech modal expert models, respectively.

[0105] Step S22 of the embodiment is intended to input the multi-modal feature output by the modal mixed expert Transformer module into the mixed expert Transformer module, dynamically adjust the weight of different expert models according to the task requirement, flexibly adapt to different data requirements, and process the new multi-modal fusion feature through the expert model in the mixed expert Transformer module that is most suitable for processing the new multi-modal fusion feature, as shown in the following formula (5): Figure 4 The formula (5) includes the following steps:

[0106] S221) input the multi-modal fusion feature into the multi-head attention mechanism;

[0107] S222) input the output of the multi-head attention mechanism into a gating network, the gating network dynamically determines which expert model should be allocated according to the characteristics of the input sample, so as to obtain the weight distribution of each expert model, the gating network is usually a linear layer, which maps the input to the weight distribution of the expert model through the softmax function, each expert model will obtain a weight indicating its applicability to process the current input;

[0108] S223) select the most suitable expert model from all expert models according to the weight allocated by the gating network, and input the multi-modal fusion feature as the input data of the selected expert model, in the embodiment, the first two expert models are selected from the six expert models according to the weight allocated by the gating network, they will calculate the output independently according to the same input data, and each expert model focuses on different data subspaces or specific tasks;

[0109] S224) obtain the output result of the selected expert model, and perform weighted summation according to the weight allocated by the gating network to obtain the final prediction result, and this fusion mechanism ensures that the contribution of each expert matches its applicability to the task.

[0110] The above process is expressed as follows:

[0111] (5)

[0112] (6)

[0113] (7)

[0114] wherein, wherein represents the weight matrix of the gating network, represents the output of the multi-head attention mechanism of the mixed expert Transformer module, represents the probability of the expert model corresponding to the multi-modal feature vector, and Top2 represents the probability of selecting the two expert models with the highest probability , is a two-dimensional vector, , are elements in , denotes the output multi-modal feature vector after combining multiple expert models.

[0115] Step S31 of the embodiment is intended to predict the emotional state of the speaker according to the output of the hybrid expert Transformer module, specifically, inputting the output of the hybrid expert Transformer module into the emotion classification module, and the emotion classification module outputs the probability of different emotional categories. The emotion classification module is composed of two linear layers and a ReLU activation function.

[0116] Step S32 of the embodiment is intended to design a corresponding loss function for model training. In the emotion classification task, in order to better optimize the performance of the model, not only a single cross-entropy loss function is used, but also a multi-task loss mechanism is introduced to jointly act on the gradient update of the model, thereby improving the generalization ability and robustness of the model. This mechanism introduces an additional expert model load balancing loss to ensure that the load of each expert model in the hybrid expert Transformer module is as balanced as possible, so as to avoid overuse of a small number of experts and improve the overall efficiency of the model. Therefore, the designed loss function expression is as follows:

[0117] (8)

[0118] (9)

[0119] (10)

[0120] wherein, L denotes the loss function, is a cross-entropy loss function, is an expert model load balancing loss, C represents the number of emotional categories, denotes the true label, denotes the probability of different emotional categories of the predicted emotional state, N represents the number of expert models in the hybrid expert Transformer module, denotes the probability of the i-th expert model being selected.

[0121] After the loss function is calculated, the gradient of the loss function with respect to the network parameters can be calculated, and then the network parameters can be updated using an optimization algorithm according to the gradient. It should be noted that how to calculate the gradient of the loss function with respect to the network parameters, and then how to use these gradients to update the parameters of the network is well known to those skilled in the art, and the method of the present embodiment does not involve improvements in the related process, so the specific calculation process will not be described here.

[0122] Embodiment two

[0123] The embodiment also provides a multi-modal sentiment recognition system based on gated feature selection and expert modeling, which comprises a microprocessor and a computer readable storage medium connected to each other, and the microprocessor is programmed or configured to execute the multi-modal sentiment recognition method based on gated feature selection and expert modeling according to the embodiment one.

[0124] In summary, the present application provides a multi-modal sentiment recognition method and system based on gated feature selection and expert modeling. The method makes full use of features at different levels, and uses modal expert models to model and fuse multi-modal features more carefully, so that the information of different modalities is fully utilized, and each feature vector in the multi-modal input is assigned a special expert, so that each expert can focus on the sentiment task. Compared with the prior art, the present application makes full use of features at different levels, strengthens the modeling of multi-modal information, and further improves the prediction accuracy of emotions, and has the following advantages:

[0125] 1. The gated feature network can utilize the information of features at different levels;

[0126] 2. The modal expert model can further capture information specific to the modal and model more carefully;

[0127] 3. The expert model focuses on emotion modeling, which further improves the accuracy of emotion prediction;

[0128] 4. The experimental results are better than other latest methods in qualitative and quantitative evaluation.

[0129] The above only describes the preferred embodiments of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments. Any technical solutions falling within the concept of the present application shall be considered as falling within the protection scope of the present application. It should be noted that for ordinary skilled persons in the art, some improvements and refinements without departing from the principles of the present application shall also be considered as falling within the protection scope of the present application.

Claims

1. A multimodal emotion recognition method based on gated feature selection and expert modeling, characterized in that, Includes the following steps: Acquire the raw speech signal and raw text; Speech features and text features at different levels are extracted from the original speech signal and the original text respectively. The most suitable speech features and text features are dynamically selected from the speech features and text features at different levels through the gating feature selection network corresponding to speech and text respectively. The selected speech features and text features are then weighted and fused to obtain fused speech features and fused text features. The fused speech features and fused text features are then concatenated to obtain multimodal fused features. The modal expert model in the modal fusion expert module further captures and fuses the information of different modalities in the multimodal fusion features to obtain new multimodal fusion features. Then, the expert model in the fusion expert module that is most suitable for processing the new multimodal fusion features is used to process the new multimodal fusion features to obtain the final prediction result. Predict the emotional state based on the prediction results; The modality fusion expert Transformer module includes a 3-layer modality fusion expert Transformer network, with each layer comprising 2 layers of modality fusion expert Transformers. When the modality expert model of the modality fusion expert Transformer module further captures and fuses information from different modalities in the multimodal fusion features, the specific steps include: The modal expert model of the first-layer modal fusion expert Transformer further captures and fuses the information of different modes in the multimodal fusion features to obtain the output of the first-layer modal fusion expert Transformer. The output of the first-layer modal fusion expert Transformer is input into the second-layer modal fusion expert Transformer. The modal expert model of the second-layer modal fusion expert Transformer further captures the information of different modes in the multimodal fusion features and then fuses them again to obtain the output of the second-layer modal fusion expert Transformer. The output of the second-layer modality mixing expert Transformer is re-segmented into text features and speech features; After inputting speech features into a merging module, the output of the merging module is concatenated with the text features to form the input of the next layer of the modality mixing expert Transformer network. The merging module consists of an adaptive average pooling layer and a linear layer. When further capturing and fusing information from different modalities in multimodal fusion features using a first-layer modal mixing expert Transformer modal expert model, and when further capturing and fusing information from different modalities in multimodal fusion features using a second-layer modal mixing expert Transformer modal expert model, both include: Input multimodal fusion features into a multi-head attention mechanism; The output of the multi-head attention mechanism is segmented according to the order in which the multimodal fusion features are concatenated, and then reclassified into text features and speech features; Text features are input into the text modality expert model, and speech features are input into the speech modality expert model; The outputs of the text modality expert model and the speech modality expert model are concatenated in sequence to form the input of the next layer of modality mixing expert Transformer.

2. The multimodal emotion recognition method based on gated feature selection and expert modeling according to claim 1, characterized in that, When extracting speech features and text features at different levels from the original speech signal and original text respectively, the specific steps include: The original speech signal is input into a pre-trained speech feature extraction model to obtain speech features at all levels. The original text is input into a pre-trained text feature extraction model to obtain text features at all levels.

3. The multimodal emotion recognition method based on gated feature selection and expert modeling according to claim 1, characterized in that, The gating feature selection networks corresponding to speech and text respectively dynamically select the most suitable speech and text features from different levels of speech and text features, and when the selected speech and text features are weighted and fused, the following are included: The features of all levels are input into a linear layer, and then the output of the linear layer is input into a softmax layer to obtain the weight probability of the features at each level. In descending order of weight probability, a specified number of features are selected sequentially. The weight probability values ​​of the selected features are normalized to obtain the weighted probability of the selected features. The selected features are then weighted according to the weighted probability and input into a linear layer to adjust the feature dimension, thus obtaining the fused features.

4. The multimodal emotion recognition method based on gated feature selection and expert modeling according to claim 1, characterized in that, When processing new multimodal fusion features using the expert model best suited for handling new multimodal fusion features from the hybrid expert Transformer module, the following applies: Input multimodal fusion features into a multi-head attention mechanism; The output of the multi-head attention mechanism is input into a gating network to obtain the weight distribution of each expert model; Based on the weights assigned by the gating network, the most suitable expert model is selected from all expert models. The multimodal fusion features are used as the input data of the selected expert model, and the output results of the selected expert model are obtained. The results are then weighted and summed according to the weights assigned by the gating network to obtain the final prediction result.

5. The multimodal emotion recognition method based on gated feature selection and expert modeling according to claim 1, characterized in that, When predicting the emotional state based on the prediction results, the prediction results are specifically input into the emotional classification module to obtain the probability of different emotional categories. The emotional classification module consists of two linear layers and a ReLU activation function.

6. The multimodal emotion recognition method based on gated feature selection and expert modeling according to claim 1, characterized in that, After predicting the sentiment state based on the prediction results, the process also includes: designing a loss function, expressed as follows: L=L ce +0.001·L load Where L represents the loss function, L ce Let L be the cross-entropy loss function. load This represents the load balancing loss of the expert model.

7. The multimodal emotion recognition method based on gated feature selection and expert modeling according to claim 6, characterized in that, The cross-entropy loss function L ce And expert model load balancing loss L load The expression is as follows: C represents the number of emotion categories, y i Indicates the true label, p represents the probability of predicting different sentiment categories of sentiment state, N represents the number of expert models in the hybrid expert Transformer module, and p i This represents the probability that the i-th expert model is selected.

8. A multimodal emotion recognition system based on gated feature selection and expert modeling, characterized in that, It includes an interconnected microprocessor and a computer-readable storage medium, the microprocessor being programmed or configured to perform the multimodal emotion recognition method based on gated feature selection and expert modeling as described in any one of claims 1 to 7.

Images