Multi-modal emotion recognition method and system based on gating feature selection and expert modeling

Abstract

The invention discloses a multi-modal emotion recognition method and system based on gating feature selection and expert modeling, and the method comprises the following steps: obtaining an original voice signal and an original text, and then respectively extracting voice features and text features of different levels; dynamically selecting the most suitable speech features and text features from the speech features and text features of different levels through a corresponding gating feature selection network, carrying out weighted fusion, obtaining the fused speech features and the fused text features, and splicing the fused speech features and the fused text features to obtain multi-modal fusion features; the method comprises the following steps: further capturing information of different modals in multi-modal fusion features through a modal hybrid expert Transform module, fusing the information again to obtain new multi-modal fusion features, and then processing the new multi-modal fusion features through an expert model which is most suitable for processing the new multi-modal fusion features in the hybrid expert Transform module. And obtaining a final prediction result and predicting the emotional state based on the prediction result. According to the method, the emotion prediction accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer information technology, and in particular to a multimodal emotion recognition method and system based on gated feature selection and expert modeling. Background Art

[0002] Multimodal emotion recognition systems play a vital role in human-computer interaction systems by automatically perceiving the emotions expressed by humans through various signals such as speech, text, and visual signals. Due to its wide and valuable applications, multimodal emotion recognition systems have been an active research topic in the past few decades, especially by using Transformers. Typical works include early summation and early concatenation. Both methods are simple and easy to implement, but they largely ignore the information alignment between different heterogeneous signals.

[0003] To address this issue, more and more research is devoted to more complex and efficient multimodal emotion recognition designs to explore the intrinsic interactions and synergies between different signals. These methods usually involve cross-attention and hierarchical attention (also known as model / intermediate fusion strategies). Specifically, for cross-attention, it exchanges query embeddings between two independent Transformer streams (one for each modality). By doing so, each modality is conditioned on the other modalities, so it can implicitly learn cross-modal contextual information. However, this approach cannot extract intra-modal information because there is no self-attention in each modality. For hierarchical attention, multimodal inputs are encoded by independent Transformer streams, and then their outputs are concatenated and fused by subsequent Transformers to form a merged stream. This approach preserves the independence of unimodal representations and captures cross-modal interactions.

[0004] However, the above methods face challenges in dealing with cross-modal versatility and flexibility. Moreover, most methods also ignore the importance of feature selection. For example, extracting features from the fixed layers of the pre-trained model for subsequent multimodal fusion limits the adaptability and performance of the model on different datasets. The selection of fixed-layer features may not fully capture the key information in the task and dataset, thus affecting the overall effect of the model. Summary of the invention

[0005] The technical problem to be solved by the present invention is as follows: In view of the above-mentioned problems in the prior art, a multimodal emotion recognition method and system based on gated feature selection and expert modeling are provided, which dynamically selects and fuses the layer features of the pre-trained model through a gating mechanism, and combines the expert model for intra-modal and specific task modeling and fusion, thereby improving the accuracy of emotion prediction.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A multimodal emotion recognition method based on gated feature selection and expert modeling, comprising the following steps:

[0008] Obtaining original speech signals and original text;

[0009] Extracting speech features and text features at different levels from the original speech signal and the original text respectively, dynamically selecting the most appropriate speech features and text features from the speech features and text features at different levels through the gated feature selection networks corresponding to the speech and text respectively, and weightedly fusing the selected speech features and text features to obtain fused speech features and fused text features, and concatenating the fused speech features and fused text features to obtain multimodal fusion features;

[0010] The modal expert model of the modal mixture expert Transformer module further captures the information of different modalities in the multimodal fusion feature and fuses it again to obtain a new multimodal fusion feature. Then, the new multimodal fusion feature is processed by the expert model in the mixed expert Transformer module that is most suitable for processing the new multimodal fusion feature to obtain the final prediction result.

[0011] The emotional state is predicted according to the prediction result.

[0012] Furthermore, when extracting different levels of speech features and text features from the original speech signal and the original text respectively, it specifically includes:

[0013] Input the original speech signal into the pre-trained speech feature extraction model to obtain speech features of all levels;

[0014] Input the original text into the pre-trained text feature extraction model to obtain text features at all levels.

[0015] Furthermore, the most appropriate speech features and text features are dynamically selected from speech features and text features at different levels through the gated feature selection networks corresponding to speech and text, and the selected speech features and text features are weighted and fused respectively, including:

[0016] Input the features of all levels into a linear layer, and then input the output of the linear layer into a softmax layer to obtain the weighted probability of the features of each level;

[0017] In the order of weight probability from large to small, a specified number of features are selected in turn, and the weight probability values ​​of the selected features are normalized to obtain the weighted probabilities of the selected features. After the selected features are weighted according to the weighted probabilities, they are input into a linear layer to adjust the feature dimensions to obtain the fused features.

[0018] Furthermore, the modal mixing expert Transformer module includes three layers of modal mixing expert Transformer networks, each layer of modal mixing expert Transformer network includes two layers of mixed expert modal Transformer, and the modal expert model of the modal mixing expert Transformer module further captures the information of different modalities in the multimodal fusion features and fuses them again, specifically including:

[0019] The modal expert model of the first-layer hybrid expert modal Transformer further captures the information of different modalities in the multimodal fusion features and then fuses them again to obtain the output of the first-layer hybrid expert modal Transformer.

[0020] The output of the first-layer modal mixture expert Transformer is input to the second-layer modal mixture expert Transformer. The modal expert model of the second-layer mixed expert modal Transformer further captures the information of different modalities in the multimodal fusion features and then fuses them again to obtain the output of the second-layer mixed expert modal Transformer.

[0021] Re-segment the output of the second-layer modality mixture expert Transformer into text features and speech features;

[0022] After the speech features are input into a merging module, the output of the merging module is concatenated with the text features to form the input of the next layer of modal mixing expert Transformer network. The merging module is composed of an adaptive average pooling layer and a linear layer.

[0023] Furthermore, when the information of different modalities in the multimodal fusion features is further captured and then fused again by the modal expert model of the first layer hybrid expert modal Transformer, and when the information of different modalities in the multimodal fusion features is further captured and then fused again by the modal expert model of the second layer hybrid expert modal Transformer, both include:

[0024] Input multimodal fusion features into the multi-head attention mechanism;

[0025] The output of the multi-head attention mechanism is split according to the order of splicing multimodal fusion features, and re-divided into text features and speech features;

[0026] Input text features into the text modality expert model, and input speech features into the speech modality expert model;

[0027] The outputs of the text modality expert and the speech modality expert are concatenated together in sequence to form the input of the next layer of modality mixing expert Transformer.

[0028] Furthermore, when the new multimodal fusion features are processed by the expert model in the hybrid expert Transformer module that is most suitable for processing the new multimodal fusion features, the following steps are included:

[0029] Input multimodal fusion features into the multi-head attention mechanism;

[0030] The output of the multi-head attention mechanism is input into a gating network to obtain the weight distribution of each expert model;

[0031] According to the weights assigned by the gating network, the most suitable expert model is selected from all expert models, and the multimodal fusion features are used as the input data of the selected expert model. The output results of the selected expert model are obtained, and the weighted sum is performed according to the weights assigned by the gating network to obtain the final prediction result.

[0032] Furthermore, when predicting the emotional state according to the prediction result, the prediction result is specifically input into the emotion classification module to obtain the probabilities of different emotion categories. The emotion classification module is composed of two linear layers and a ReLU activation function.

[0033] Furthermore, after predicting the emotional state according to the prediction result, it also includes: designing a loss function, the expression of which is as follows:

[0034] L=L ce +0.001·L load

[0035] Among them, L represents the loss function, L ce is the cross entropy loss function, L load Load balancing loss for the expert model.

[0036] Furthermore, the cross entropy loss function L ce and the expert model load balancing loss L load The expression is as follows:

[0037]

[0038] C represents the number of sentiment categories, y i represents the true label, represents the probability of different emotion categories of the predicted emotional state, N represents the number of expert models in the hybrid expert Transformer module, and p i Represents the probability of the i-th expert model being selected.

[0039] The present invention also proposes a multimodal emotion recognition system based on gated feature selection and expert modeling, comprising a microprocessor and a computer-readable storage medium connected to each other, wherein the microprocessor is programmed or configured to execute any one of the multimodal emotion recognition methods based on gated feature selection and expert modeling.

[0040] Compared with the prior art, the advantages of the present invention are:

[0041] The present invention dynamically selects and fuses the layer features of the pre-trained model through a gating mechanism, and combines the modal hybrid expert Transformer module and the hybrid expert Transformer module for more detailed modeling and fusion, thereby improving the accuracy of emotion recognition. And by converting different modal experts, the architecture of single modality and multi-modality can be unified into one model, thereby improving the versatility of the model. In addition, the present invention also designs a hierarchical merging layer to reduce the information redundancy and computational load of the speech modality. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The figure is an overall flow chart of the method according to the embodiment of the present invention.

[0043] Figure 2 This is a detailed flow chart of the gated feature selection network in the method of an embodiment of the present invention.

[0044] Figure 3 Detailed flowchart of the modal mixing expert Transformer module in the method of an embodiment of the present invention.

[0045] Figure 4 Flow chart of the hybrid expert Transformer module in the method of an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The present invention is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present invention is not limited thereby.

[0047] Before introducing the specific embodiments of the present invention, related concepts or terms are first explained.

[0048] Features: Features are key information in data that describes a property or pattern. In machine learning and deep learning, features are the basis for models to identify and classify input data. Features can be manually designed or automatically extracted in a data-driven way, such as edges and color distribution in images, or pitch and spectrum in speech signals. Features consist of multiple vectors.

[0049] Feature extraction model: A feature extraction model is an algorithm or neural network specifically designed to extract useful features from raw data. It automatically learns patterns in raw data and converts them into feature representations suitable for downstream tasks (such as classification, regression, etc.). Common feature extraction models include self-supervised learning-based models (such as wav2vec 2.0, WavLM) for speech processing and (Bert, Roberta) for natural language processing.

[0050] All-layer features extracted from feature extraction models: Modern deep learning models are usually composed of multiple layers of neural networks (such as Transformer), and each layer learns different levels of representation of data. All-layer features of feature extraction models refer to the feature representations generated by each layer in the model. These representations are usually gradually enriched from low-level shallow features (such as local patterns or edges) to high-level deep features (such as global semantic information) as the number of network layers increases. Combining these hierarchical features helps enhance the generalization ability of the model.

[0051] Modality: Modality refers to different forms of data, such as images, speech, text, etc., which represent different sources of information and ways of perception. In multimodal learning, the model can more comprehensively understand and analyze complex tasks by processing data from multiple modalities.

[0052] Expert model: refers to a group of specialized sub-models or networks, each consisting of two linear layers, which perform better when processing specific types of inputs or specific tasks.

[0053] Weighted probability: In the feature fusion process, weighted probability is a way to calculate the relative importance of each feature in the fused feature by normalizing the probability values ​​of the three features with the highest probability. These weighted probabilities reflect the contribution of each feature, so that when performing feature fusion, the selected features can be weighted and summed according to these weights to obtain a more comprehensive and effective feature representation. This method can highlight the role of important features and improve the performance of the model in downstream tasks.

[0054] Embodiment 1

[0055] This embodiment proposes a multimodal emotion recognition method based on gated feature selection and expert modeling. The method makes full use of features at different levels and uses expert models to perform more detailed modeling and fusion of multimodal features, so as to make more effective use of information from different modalities, thereby improving the accuracy of emotion prediction.

[0056] like Figure 1 As shown, the method of this embodiment includes the following three stages:

[0057] S1) Feature extraction stage: In this stage, the pre-trained feature extraction model is used to extract features of all levels, and then the feature selection network is used to select the most appropriate layer features, and weighted with weighted probability to make them into fusion features. This stage specifically includes:

[0058] S11) obtaining an original speech signal and an original text, and extracting speech features and text features of different levels from the original speech signal and the original text respectively;

[0059] S12) dynamically selecting the most appropriate speech features and text features from speech features and text features at different levels through gated feature selection networks corresponding to speech and text, respectively, and weightedly fusing the selected speech features and text features to obtain fused speech features and fused text features, and splicing the fused speech features and fused text features to obtain multimodal fusion features;

[0060] S2) Expert modeling stage: In this stage, the feed-forward neural network in the original Transformer network is replaced with a different expert model, which can model and fuse information of different modalities more carefully. It can also assign a dedicated expert to each feature vector in the multimodal input, so that each expert can focus on the sentiment task and provide more effective information for the sentiment classification task. This stage specifically includes:

[0061] S21) The modal expert model of the modal mixing expert Transformer module further captures the information of different modalities in the multimodal fusion feature and fuses it again, so as to more carefully model and fuse the information of different modalities (such as text and voice) to obtain new multimodal fusion features; In this embodiment, in the modal mixing expert Transformer module, the modal expert model is trained to focus on a specific modality (such as vision, hearing, text, etc.). In this way, the model can make full use of information from different modalities and improve the overall performance;

[0062] S22) The new multimodal fusion features are processed by the expert model in the hybrid expert Transformer module that is most suitable for processing the new multimodal fusion features to obtain the final prediction result of the model; in this embodiment, in the hybrid expert Transformer module, the role of the expert model is to assign a special expert to each feature vector in the multimodal input in order to make full use of information from different modes. Each expert can focus on the processing of a specific mode or feature, and dynamically select the most suitable expert to respond through a gating mechanism. This method not only improves the performance of the model in each mode, but also enhances the comprehensive understanding of complex tasks. In addition, the collaboration of multiple experts can improve the robustness of the model, reduce the deviation caused by a single model, and optimize computational efficiency.

[0063] S3) Classification and loss calculation stage: In this stage, the output generated by the expert modeling stage is classified through the sentiment classification module, and the corresponding loss function is used to constrain the gradient update of the model, including:

[0064] S31) predicting the emotional state according to the prediction result, specifically inputting the output generated in the expert modeling stage, that is, the final prediction result of the model, into the emotion classification module, and the emotion classification module outputs the probabilities of different emotion categories;

[0065] S32) Establish classification loss and expert load balancing loss and use them to design a loss function to constrain the gradient update of the model during model training.

[0066] The following is a detailed description of each step.

[0067] Step S11 of this embodiment is to extract features of all levels of the pre-trained model and input them into the gated feature selection network. When extracting speech features and text features of different levels from the original speech signal and the original text respectively, the pre-trained speech feature extraction model and the text feature extraction model are used to perform feature extraction, which specifically includes the following steps:

[0068] S111) Input the original speech signal A into the pre-trained speech feature extraction model to obtain speech features of all levels Where n represents the number of speech features included;

[0069] S112) Input the original text T into the pre-trained text feature extraction model to obtain text features of all levels Where m represents the number of text features included.

[0070] In this embodiment, the speech feature extraction model uses the pre-trained speech encoder WavLM-large. How to use the speech encoder to extract speech features is well known to those skilled in the art, and this embodiment will not be repeated. Through the speech feature extraction process, a total of 24 levels of speech features are extracted, that is, n=24. The text feature extraction model uses the pre-trained text encoder Roberta-large. How to use the text encoder to extract text features is well known to those skilled in the art, and this embodiment will not be repeated. Through the text feature extraction process, a total of 24 levels of text features are extracted, that is, n=24.

[0071] Step S12 of this embodiment aims to dynamically select the most appropriate speech and text features through a gated feature selection network, and obtain the fused speech and text features through a weighted fusion method, wherein the most appropriate speech features and text features are dynamically selected from speech features and text features at different levels through gated feature selection networks corresponding to speech and text, and the selected speech features and text features are weightedly fused, such as Figure 2 As shown, the following steps are included:

[0072] S121) Input the features of all levels into a linear layer, and then input the output of the linear layer into a softmax layer to obtain the weighted probability of the features of each level; S122) Select a specified number of features (the specified number is 3 in this embodiment) in order of weight probability from large to small, normalize the weighted probability values ​​of the selected features to obtain the weighted probability of the selected features, weight the selected features according to the weighted probability, and then input them into a linear layer to adjust the feature dimension to obtain the fused features.

[0073] Specifically, the speech features of all levels are obtained Input a linear layer, and then input the output of the linear layer into a softmax layer to obtain the weighted probability of speech features at each level; according to the weighted probability, select the first three layers of features with the highest weighted probability, and normalize the weighted probabilities of these three layers of features to obtain the weighted probabilities of these features. After weighting the three layer features by weighted probability, adjust the feature dimension through a linear layer to obtain the fused speech feature F a ;

[0074] Similarly, the text features of all levels are obtained Input a linear layer, and then input the output of the linear layer into a softmax layer to obtain the weighted probability of text features at each level; according to the weighted probability, select the first three layers of features with the highest weighted probability, and normalize the weighted probabilities of these three layers of features to obtain the weighted probabilities of these features. After weighting the three layer features by weighted probability, adjust the feature dimension through a linear layer to obtain the fused text feature F t .

[0075] The above process expression is as follows:

[0076]

[0077] Top3=Selcet(P)

[0078]

[0079] F=W w F w +b w

[0080] Where W f With b f They are the weight and bias matrices of the linear layer, respectively, used to obtain the weight probability of each level feature. The Select() function selects the three probabilities with the highest probability and their corresponding layer features. Represents speech features or text features at all levels. w With b w The weight and bias matrices of the linear layer are also used to adjust the dimension of the features.

[0081] In step S12 of this embodiment, when the fused speech feature and the fused text feature are concatenated to obtain the multimodal fusion feature, the obtained speech fusion feature F a Fusion feature F with text t Splicing to obtain multimodal fusion features F ta =[F t ,F a ], where text features come first and speech features come second.

[0082] In this embodiment, the modal mixing expert Transformer module includes 3 layers of modal mixing expert Transformer networks, each layer of the modal mixing expert Transformer network includes 2 layers of mixed expert modal Transformers, and after each layer of the modal mixing expert Transformer network, a merging layer is added to gradually reduce the length of the speech features, reduce the amount of calculation, and gradually condense the semantic information of the speech. Step S21 is intended to input the fusion features into the modal mixing expert Transformer module to perform more detailed modeling of different modalities (such as text and speech). The modal expert model of the modal mixing expert Transformer module further captures the information of different modalities in the multimodal fusion features and then fuses them again, specifically including:

[0083] S211) further capturing the information of different modalities in the multimodal fusion features through the modal expert model of the first layer hybrid expert modal Transformer and fusing them again to obtain the output of the first layer hybrid expert modal Transformer;

[0084] S212) inputting the output of the first-layer modal hybrid expert Transformer to the second-layer modal hybrid expert Transformer, further capturing the information of different modalities in the multimodal fusion features through the modal expert model of the second-layer hybrid expert modal transformer, and then fusing them again to obtain the output of the second-layer hybrid expert modal transformer. Through the stacking results, the potential speech information in the multimodal fusion features can be further captured;

[0085] S213) re-segmenting the output of the second-layer modality mixture expert Transformer into text features and speech features;

[0086] S214) After the speech features are input into a merging module, the output of the merging module is concatenated with the text features to form the input of the next layer of modal mixing expert Transformer network, wherein the concatenation order is text features first and speech features second. The merging module is composed of an adaptive average pooling layer and a linear layer, which is used to shorten the length of the speech features, reduce the overall calculation amount of the model, condense the speech features, and reduce information redundancy.

[0087] The above merging process is expressed as follows:

[0088] F′ am =AvgPool2d(F a )

[0089] F am =Layernorom(W amF′ am +b am )

[0090] Among them, AvgPool2d represents 2D uniform pooling for reducing the length of speech features, and the linear layer added later is used to map the pooled features to enhance the expressiveness of the model. Its weight matrix and bias matrix are W respectively. am With b am ,Layernrom represents layer normalization, which is used to normalize features to speed up training, stabilize gradients, and improve the generalization ability of the model.

[0091] In this implementation, the modal expert model of the first layer hybrid expert modal Transformer further captures the information of different modalities in the multimodal fusion feature and then fuses it again, and the modal expert model of the second layer hybrid expert modal Transformer further captures the information of different modalities in the multimodal fusion feature and then fuses it again, as shown in FIG. Figure 3 As shown, the following steps are included:

[0092] S211) inputting the multimodal fusion features into the multi-head attention mechanism;

[0093] S212) dividing the output of the multi-head attention mechanism according to the order of splicing the multimodal fusion features, and re-dividing it into text features and speech features;

[0094] S213) inputting text features into a text modality expert model, and inputting speech features into a speech modality expert model;

[0095] S214) The outputs of the text modality expert and the speech modality expert are concatenated in order to form the input of the next layer of modality mixing expert Transformer, with the text features placed in front and the speech features placed in the back during concatenation.

[0096] The above process expression is as follows:

[0097]

[0098] MSA stands for multi-head attention mechanism, l stands for the modal mixture expert Transformer of the current layer, and l-1 stands for the modal mixture expert Transformer of the previous layer. represents the output of the previous modal mixture expert Transformer, and represents the output of the MSA module in the current layer of modal hybrid expert Transformer. n represents the length of the text feature and also represents the connection position between the text feature and the speech feature in the multimodal fusion feature. Therefore, the n position is split can represent text features, and Can represent speech features. and Represent the outputs of text modality experts and speech modality experts respectively. It represents the output of the modal mixture expert Transformer of the current layer, and is also the input of the modal mixture expert Transformer of the next layer. t With E a Represent text and speech modality expert models respectively.

[0099] Step S22 of this embodiment is to input the multimodal features output by the modal hybrid expert Transformer module into the hybrid expert Transformer module, dynamically adjust the weights of different expert models according to task requirements, flexibly adapt to different data requirements, and process the new multimodal fusion features through the expert model in the hybrid expert Transformer module that is most suitable for processing the new multimodal fusion features. Figure 4 As shown, including:

[0100] S221) inputting the multimodal fusion features into the multi-head attention mechanism;

[0101] S222) The output of the multi-head attention mechanism is input into a gating network. The gating network dynamically determines which expert model should be assigned to process the input sample according to its characteristics, thereby obtaining a weight distribution of each expert model. The gating network is usually a linear layer, and the input is mapped to the weight distribution of the expert model through a softmax function. Each expert model will obtain a weight, indicating its suitability for processing the current input;

[0102] S223) According to the weights assigned by the gating network, the most suitable expert model is selected from all expert models, and the multimodal fusion features are used as input data of the selected expert model. In this embodiment, according to the weights assigned by the gating network, the most suitable first two expert models are selected from the six expert models, and they will independently calculate the output according to the same input data, and each expert model focuses on a different data subspace or a specific task;

[0103] S224) obtains the output results of the selected expert model, and performs weighted summation according to the weights assigned by the gating network to obtain the final prediction result. This fusion mechanism ensures that the contribution of each expert matches its applicability to the task.

[0104] The above process expression is as follows:

[0105] p = Top2(Softmax(W h F′ ta ),2) (5)

[0107] p′=Top2(p) (6)

[0108]

[0109] Among them, W g represents the weight matrix of the gating network, F′ ta represents the output of the multi-head attention mechanism of the hybrid expert Transformer module, p represents the probability of the expert model corresponding to the multimodal feature vector, and Top2 represents the probability p′ of selecting the two expert models with the highest probability. p′ is a two-dimensional vector. k , p′ j are the elements in p′, F″ ta Represents the output multimodal feature vector after combining multiple expert models.

[0110] Step S31 of this embodiment aims to predict the speaker's emotional state based on the output of the hybrid expert Transformer module, specifically, the output of the hybrid expert Transformer module is input into the emotion classification module, and the emotion classification module outputs the probabilities of different emotion categories. The emotion classification module is composed of two linear layers and a ReLU activation function.

[0111] Step S32 of this embodiment is to design a corresponding loss function for model training. In the sentiment classification task, in order to better optimize the model performance, not only a single cross entropy loss function L is used ce , and also introduces a multi-task loss mechanism to work together on the gradient update of the model, thereby improving the generalization ability and robustness of the model. This mechanism introduces an additional expert model load balancing loss L load , ensuring that the load of each expert model in the hybrid expert Transformer module is as balanced as possible to avoid overuse of a few experts and improve the overall efficiency of the model. Therefore, the loss function expression designed is as follows:

[0112] L=L ce +0.001·L load (8)

[0113]

[0114] Among them, L represents the loss function, L ce is the cross entropy loss function, L load is the expert model load balancing loss, C represents the number of sentiment categories, y i represents the true label, represents the probability of different emotion categories of the predicted emotional state, N represents the number of expert models in the hybrid expert Transformer module, and p i Represents the probability of the i-th expert model being selected.

[0115] After calculating the loss function, the gradient of the loss function with respect to the network parameters can be calculated, and then the network parameters can be updated using an optimization algorithm based on the gradient. It should be noted that the specific process of how to calculate the gradient of the loss function with respect to the network parameters and how to use these gradients to update the parameters of the network is well known to those skilled in the art, and the method of this embodiment does not involve improvements in the related process, so the specific calculation process will not be repeated.

[0116] Embodiment 2

[0117] This embodiment also proposes a multimodal emotion recognition system based on gated feature selection and expert modeling, including a microprocessor and a computer-readable storage medium connected to each other, and the microprocessor is programmed or configured to execute the multimodal emotion recognition method based on gated feature selection and expert modeling described in Example 1.

[0118] In summary, the present invention proposes a multimodal emotion recognition method and system based on gated feature selection and expert modeling. This method makes full use of features at different levels, and uses modal expert models to model and fuse multimodal features in a more detailed manner, so that information from different modalities is more fully utilized, and further a dedicated expert is assigned to each feature vector in the multimodal input, so that each expert can focus on the emotion task. Compared with the prior art, the present invention makes full use of features at different levels, strengthens the modeling of multimodal information, and thus improves the accuracy of emotion prediction, and has the following advantages:

[0119] 1. The gated feature network can utilize information from different levels of features;

[0120] 2. The modal expert model can further capture specific modal information and perform more detailed modeling;

[0121] 3. The expert model focuses on sentiment modeling, further improving the accuracy of sentiment prediction;

[0122] 4. The experimental results are superior to other state-of-the-art methods in both qualitative and quantitative evaluation.

[0123] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A multimodal emotion recognition method based on gated feature selection and expert modeling, characterized in that: The following steps are involved: Obtaining original speech signals and original text; Extracting speech features and text features at different levels from the original speech signal and the original text respectively, dynamically selecting the most appropriate speech features and text features from the speech features and text features at different levels through the gated feature selection networks corresponding to the speech and text respectively, and weightedly fusing the selected speech features and text features to obtain fused speech features and fused text features, and concatenating the fused speech features and fused text features to obtain multimodal fusion features; The modal expert model of the modal mixture expert Transformer module further captures the information of different modalities in the multimodal fusion feature and fuses it again to obtain a new multimodal fusion feature. Then, the new multimodal fusion feature is processed by the expert model in the mixed expert Transformer module that is most suitable for processing the new multimodal fusion feature to obtain the final prediction result. The emotional state is predicted according to the prediction result.

2. The multimodal emotion recognition method based on gated feature selection and expert modeling according to claim 1, characterized in that: When extracting different levels of speech features and text features from the original speech signal and original text respectively, it specifically includes: Input the original speech signal into the pre-trained speech feature extraction model to obtain speech features of all levels; Input the original text into the pre-trained text feature extraction model to obtain text features at all levels.

3. The multimodal emotion recognition method based on gated feature selection and expert modeling according to claim 1, characterized in that: The most appropriate speech features and text features are dynamically selected from speech features and text features at different levels through the gated feature selection networks corresponding to speech and text, and the selected speech features and text features are weighted and fused respectively, including: Input the features of all levels into a linear layer, and then input the output of the linear layer into a softmax layer to obtain the weighted probability of the features of each level; In the order of weight probability from large to small, a specified number of features are selected in turn, and the weight probability values ​​of the selected features are normalized to obtain the weighted probabilities of the selected features. After the selected features are weighted according to the weighted probabilities, they are input into a linear layer to adjust the feature dimensions to obtain the fused features.

4. The multimodal emotion recognition method based on gated feature selection and expert modeling according to claim 1, characterized in that: The modal mixing expert Transformer module includes three layers of modal mixing expert Transformer networks, each layer of which includes two layers of mixed expert modal Transformers. When the modal expert model of the modal mixing expert Transformer module further captures the information of different modalities in the multimodal fusion features and fuses them again, it specifically includes: The modal expert model of the first-layer hybrid expert modal Transformer further captures the information of different modalities in the multimodal fusion features and then fuses them again to obtain the output of the first-layer hybrid expert modal Transformer. The output of the first-layer modal mixture expert Transformer is input to the second-layer modal mixture expert Transformer. The modal expert model of the second-layer mixed expert modal Transformer further captures the information of different modalities in the multimodal fusion features and then fuses them again to obtain the output of the second-layer mixed expert modal Transformer. Re-segment the output of the second-layer modality mixture expert Transformer into text features and speech features; After the speech features are input into a merging module, the output of the merging module is concatenated with the text features to form the input of the next layer of modal mixing expert Transformer network. The merging module is composed of an adaptive average pooling layer and a linear layer.

5. The multimodal emotion recognition method based on gated feature selection and expert modeling according to claim 4, characterized in that: When the information of different modalities in the multimodal fusion features is further captured and then fused again by the modal expert model of the first layer of hybrid expert modal Transformer, and when the information of different modalities in the multimodal fusion features is further captured and then fused again by the modal expert model of the second layer of hybrid expert modal Transformer, both include: Input multimodal fusion features into the multi-head attention mechanism; The output of the multi-head attention mechanism is split according to the order of splicing multimodal fusion features, and re-divided into text features and speech features; Input text features into the text modality expert model, and input speech features into the speech modality expert model; The outputs of the text modality expert and the speech modality expert are concatenated together in sequence to form the input of the next layer of modality mixing expert Transformer.

6. The multimodal emotion recognition method based on gated feature selection and expert modeling according to claim 1, characterized in that: When processing the new multimodal fusion features through the expert model in the hybrid expert Transformer module that is most suitable for processing the new multimodal fusion features, it includes: Input multimodal fusion features into the multi-head attention mechanism; The output of the multi-head attention mechanism is input into a gating network to obtain the weight distribution of each expert model; According to the weights assigned by the gating network, the most suitable expert model is selected from all expert models, and the multimodal fusion features are used as the input data of the selected expert model. The output results of the selected expert model are obtained, and the weighted sum is performed according to the weights assigned by the gating network to obtain the final prediction result.

7. The multimodal emotion recognition method based on gated feature selection and expert modeling according to claim 1, characterized in that: When predicting the emotional state according to the prediction result, specifically the prediction result is input into the emotion classification module to obtain the probabilities of different emotion categories, and the emotion classification module is composed of two linear layers and a ReLU activation function.

8. The multimodal emotion recognition method based on gated feature selection and expert modeling according to claim 1, characterized in that: After predicting the emotional state according to the prediction result, the method further includes: designing a loss function, which is expressed as follows: L=L ce +0.001·L load Among them, L represents the loss function, L ce is the cross entropy loss function, L load Load balancing loss for the expert model.

9. The multimodal emotion recognition method based on gated feature selection and expert modeling according to claim 8, characterized in that: The cross entropy loss function L ce and the expert model load balancing loss L load The expression is as follows: C represents the number of sentiment categories, y i represents the true label, represents the probability of different emotion categories for predicting the emotional state, N represents the number of expert models in the hybrid expert Transformer module, and p i Represents the probability of the i-th expert model being selected.

10. A multimodal emotion recognition system based on gated feature selection and expert modeling, characterized in that: It comprises a microprocessor and a computer-readable storage medium connected to each other, wherein the microprocessor is programmed or configured to execute the multimodal emotion recognition method based on gated feature selection and expert modeling as described in any one of claims 1 to 9.

Images