A hybrid modality expert emotion recognition method and system

Abstract

The present invention discloses a hybrid modality expert emotion recognition method and system, relating to the field of emotion recognition technology. The method comprises: utilizing pre-trained models corresponding to different modal data to perform feature extraction on the modal data to be recognized, thereby obtaining modal features corresponding to the modal data in the modal data to be recognized; wherein the modal data to be recognized includes at least one of audio modal data, text modal data, and visual modal data; utilizing a trained hybrid modality expert model to perform emotion recognition on the modal features, thereby obtaining an emotion recognition result for the modal data to be recognized; wherein the hybrid modality expert model is used to extract single modal representations and joint representations of the modal features, and dynamically fuses the single modal representations and joint representations to perform emotion recognition to obtain an emotion recognition result. The method of the present invention overcomes the problem of insufficient discriminative power of the extracted representation when only one modal data is available, thereby improving recognition performance in scenarios where multiple modalities are missing.

Description

Technical Field

[0001] The present invention belongs to the technical field of emotion recognition, and in particular relates to a mixed-modality expert emotion recognition method and system. Background Art

[0002] Emotions are fundamental human physiological and psychological needs, playing a crucial role in our daily lives, interpersonal relationships, and our perception of the world. Automatic emotion recognition systems have found widespread application in areas such as social media analysis, service robot design, video games, and mental health analysis. In recent years, multimodal emotion recognition systems have gradually replaced single-modality emotion models due to their ability to leverage information from multiple modalities (such as vision, hearing, and language) to make integrated decisions and identify emotions.

[0003] However, current multimodal emotion recognition models suffer from significant degradation in reliability in real-world scenarios where some modal data is missing or unavailable (e.g., when all visual data is unavailable) due to factors such as sensor damage, speech recognition failures, and privacy concerns. This has drawn widespread attention. There are two main approaches to address this: generative methods and joint learning.

[0004] The generative method is to train the model to generate missing modal data using the non-missing modal data to complete the data of all modalities. Since the generation process of this type of method is often too time-consuming, it affects the real-time processing capability of the model and is rarely adopted for emotion recognition. The core idea of ​​the joint learning method is to train the model to extract modal information shared between different modalities, that is, the joint representation of modalities. The general implementation method is to reconstruct the missing modalities through the decoder during the training of the model to constrain the learned modal representation to contain information from all modalities, such as Figure 1 Figure 1 shows a schematic diagram of a joint learning method. This type of method can extract a joint modal representation for emotion recognition given arbitrary modal inputs and has been widely validated in multimodal emotion recognition tasks in modality-deficient scenarios. However, existing joint representation methods experience significant degradation in recognition performance in scenarios with severe modality deficiency, where only a single modality is available. Summary of the Invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a hybrid modality expert emotion recognition method and system. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] The present invention provides a hybrid modality expert emotion recognition method, comprising:

[0007] Extracting features from the modal data to be identified using pre-trained models corresponding to different modal data to obtain modal features corresponding to the modal data in the modal data to be identified; wherein the modal data to be identified includes at least one of audio modal data, text modal data, and visual modal data;

[0008] The trained hybrid modal expert model is used to perform emotion recognition on the modal features to obtain the emotion recognition result of the modal data to be identified; wherein, the hybrid modal expert model is used to extract the single modal representation and the joint representation of the modal features, and the single modal representation and the joint representation are dynamically fused to perform emotion recognition to obtain the emotion recognition result.

[0009] The present invention provides a hybrid modality expert emotion recognition system, applicable to the hybrid modality expert emotion recognition method described in any of the above embodiments, and the system comprises:

[0010] a data preprocessing module, configured to extract features from the modal data to be identified using pre-trained models corresponding to different modal data, thereby obtaining modal features corresponding to the modal data in the modal data to be identified, wherein the modal data to be identified includes at least one of audio modal data, text modal data, and visual modal data;

[0011] The emotion recognition module is used to use the trained hybrid modal expert model to perform emotion recognition on the input modal features to obtain the emotion recognition results of the modal data to be identified, wherein the hybrid modal expert model is used to extract the single modal representation and the joint representation of the modal features, and dynamically fuse the single modal representation and the joint representation to perform emotion recognition to obtain the emotion recognition results.

[0012] Compared with the prior art, the present invention has the following beneficial effects:

[0013] The hybrid modality expert emotion recognition method of the present invention utilizes pre-trained models corresponding to different modal data to perform feature extraction on the modal data to be identified, and obtains modal features corresponding to the modal data in the modal data to be identified; wherein, the modal data to be identified includes at least one of audio modal data, text modal data and visual modal data; utilizes the trained hybrid modality expert model to perform emotion recognition on the modal features, and obtains the emotion recognition result of the modal data to be identified; wherein, the hybrid modality expert model is used to extract the single modal representation and the joint representation of the modal features, and dynamically fuses the single modal representation and the joint representation to perform emotion recognition to obtain the emotion recognition result. The hybrid modality expert emotion recognition method of the present invention simultaneously extracts single modality representation and joint representation through a hybrid modality expert model, and then dynamically fuses all representations to obtain a complete modality representation, and uses the complete modality representation to realize emotion recognition. This method can clearly extract single modality representation and joint representation of available modality data at the same time in scenarios where modalities are severely missing, strengthens the learning of modality-specific information, namely single modality representation, overcomes the problem of insufficient discriminative power of the representation extracted when only one modality data is available, and improves recognition performance in scenarios where multiple modalities are missing.

[0014] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following preferred embodiments are specifically cited and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a schematic diagram of the joint learning method;

[0016] Figure 2 This is a flow chart of a hybrid modality expert emotion recognition method provided by an embodiment of the present invention;

[0017] Figure 3 is a schematic diagram of a hybrid modality expert emotion recognition method provided by an embodiment of the present invention;

[0018] Figure 4 Schematic diagram of a training process of a mixed modality expert model provided by an embodiment of the present invention;

[0019] Figure 5 Schematic diagram showing the comparison of predictions of an actual conversation by the hybrid modality expert model provided by an embodiment of the present invention and the weights assigned by the three modality experts;

[0020] Figure 6 This is a structural block diagram of a hybrid modality expert emotion recognition system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following is a detailed description of a hybrid modality expert emotion recognition method and system proposed in accordance with the present invention in combination with the accompanying drawings and specific implementation methods.

[0022] The aforementioned and other technical contents, features, and effects of the present invention are clearly presented in the following detailed description of the specific embodiments in conjunction with the accompanying drawings. Through the description of the specific embodiments, a deeper and more specific understanding of the technical means and effects adopted by the present invention to achieve the intended purpose can be obtained. However, the accompanying drawings are provided for reference and illustration purposes only and are not intended to limit the technical solutions of the present invention.

[0023] In the first aspect, the embodiment of the present invention provides a mixed modality expert emotion recognition method, see Figure 2 , Figure 2 This is a flow chart of a hybrid modality expert emotion recognition method provided by an embodiment of the present invention. Figure 2 As shown, the hybrid modality expert emotion recognition method of this embodiment includes:

[0024] Step 1: Use the pre-trained models corresponding to different modal data to extract features of the modal data to be identified, and obtain the modal features corresponding to the modal data in the modal data to be identified.

[0025] The modal data to be identified includes at least one of audio modal data, text modal data and visual modal data.

[0026] Exemplarily, the modal data to be identified may be video data, which may be complete video data including audio modal data, text modal data, and visual modal data, or incomplete video data including one or two of audio modal data, text modal data, and visual modal data, that is, video data in a modality-missing scenario.

[0027] Step 2: Use the trained hybrid modal expert model to perform emotion recognition on the modal features to obtain the emotion recognition results of the modal data to be recognized.

[0028] Among them, the hybrid modal expert model is used to extract the single modal representation and joint representation of modal features, and the single modal representation and joint representation are dynamically integrated to perform emotion recognition to obtain the emotion recognition result.

[0029] The hybrid modality expert emotion recognition method of the embodiment of the present invention simultaneously extracts single modality representation and joint representation through a hybrid modality expert model, and then dynamically fuses all representations to obtain a complete modality representation, and uses the complete modality representation to realize emotion recognition. This method can clearly extract single modality representation and joint representation of available modality data at the same time in scenarios where modalities are severely missing, strengthens the learning of modality-specific information, namely single modality representation, overcomes the problem of insufficient discriminative power of the extracted representation when only one modality data is available, and improves recognition performance in scenarios where multiple modalities are missing.

[0030] See Figure 3 , Figure 3 Schematic diagram of a hybrid modality expert emotion recognition method provided by an embodiment of the present invention. Figure 3 As shown, in an optional embodiment, the pre-trained models include: a pre-trained wav2vec-large model, a pre-trained DeBERTa-large model, and a pre-trained MA-Net model. The pre-trained wav2vec-large model is used to extract audio features from audio modal data; the pre-trained DeBERTa-large model is used to extract text features from text modal data; and the pre-trained MA-Net model is used to extract visual features from visual modal data.

[0031] In this embodiment, for the audio modal data in the video data, a 512-dimensional acoustic feature is extracted using the pre-trained wav2vec-large model as the audio feature of the audio modal data, denoted as X a For the text modality data in the video data, the pre-trained DeBERTa-large model is used to encode each sentence into a 1024-dimensional vocabulary feature as the text feature of the text modality data, denoted as X t For the visual modality data in the video data, the MTCNN face detection model is used to capture the facial images in the video, and then the pre-trained MA-Net model is used to extract facial features. The facial features of all frames in a sentence are compressed into a 1024-dimensional feature as the visual feature of the visual modality data, denoted as X v When a modality data in the video data is missing, its modality feature can be represented by adding a superscript ∧, for example Figure 3 middle and

[0032] In an optional embodiment, the hybrid modal expert model includes a single modal expert, a soft router, a splicing unit, and a classifier. The single modal expert is used to extract single modal representations and joint representations of input modal features. The soft router is used to dynamically fuse the single modal representations and joint representations of the modal features to obtain a fused modal representation of the modal features. The splicing unit is used to splice the fused modal representations of the modal features. The classifier is used to perform emotion recognition based on the spliced ​​fused modal representations to obtain an emotion recognition result.

[0033] In this embodiment, the unimodal experts include audio experts, text experts, and visual experts. Optionally, the audio experts, text experts, and visual experts all employ Transformer model structures. The audio experts are used to extract audio unimodal representations for audio features, text-audio joint representations for text features, and visual-audio joint representations for visual features. The text experts are used to extract text unimodal representations for text features, audio-text joint representations for audio features, and visual-text joint representations for visual features. The visual experts are used to extract visual unimodal representations for visual features, audio-visual joint representations for audio features, and text-visual joint representations for text features.

[0034] In this embodiment, the soft routing is specifically used to obtain the weights corresponding to the single modal representation and the joint representation of the modal features according to the input modal features, and dynamically fuse the single modal representation and the joint representation of the modal features according to the weights to obtain a fused modal representation of the modal features.

[0035] Optionally, the soft router is a multi-layer perceptron containing two hidden layers. The introduction of the soft router can dynamically adjust the expression weights of the single modality representation and the joint representation for different modal data to be identified.

[0036] In this embodiment, the weights corresponding to the unimodal representation and the joint representation of the audio features corresponding to the audio modality data are calculated according to the following formula:

[0037]

[0038] In the formula, a represents audio modality data, t represents text modality data, v represents visual modality data, and X a represents the audio features of audio modal data, r represents soft routing, MLP represents multi-layer perceptron, S a represents the modal score set sum of audio modal data, represents the audio modality score, represents the audio-text modality score, represents the audio-visual modality score, softmax represents the normalized exponential function, when i is a, Represents the weight of the audio unimodal representation. When i is t, Represents the weight of the audio-text joint representation. When i is v, represents the weight of the joint audio-visual representation.

[0039] The weights of the unimodal representation and joint representation of the text features corresponding to the text modal data are calculated according to the following formula:

[0040]

[0041] Where, X t Represents the text features of text modality data, S t represents the set sum of modal scores of text modal data, represents the text modality score, represents the text-audio modality score, represents the text-visual modality score, when i is t, Represents the weight of the text unimodal representation. When i is a, Represents the weight of the text-audio joint representation. When i is v, represents the weight of the joint text-visual representation.

[0042] The weights corresponding to the unimodal representation and joint representation of the visual features corresponding to the visual modality data are calculated according to the following formula:

[0043]

[0044] Where, X v Represents the visual features of visual modality data, S v represents the set sum of modal scores of visual modality data, represents the visual modality score, represents the visual-audio modality score, Represents the visual-text modality score. When i is v, Represents the weight of visual unimodal representation. When i is a, represents the weight of the visual-audio joint representation. When i is t, represents the weight of the joint visual-text representation.

[0045] In this embodiment, for the audio corresponding to the audio modal data

[0046] The fusion modal representation of the features, that is, the audio fusion modal representation, is calculated according to the following formula:

[0047]

[0048] Where R a is the audio fusion modal representation, when i is a, Represents the audio unimodal representation. When i is t, Represents the audio-text joint representation. When i is v, Representing joint audio-visual representation.

[0049] The fusion modality representation of the text features corresponding to the text modality data, that is, the text fusion modality representation, is calculated according to the following formula:

[0050]

[0051] Where R t is the text fusion modality representation, when i is t, Represents the unimodal representation of text. When i is a, Represents the joint representation of text and audio. When i is v, Representing joint text-visual representation.

[0052] The fusion modality representation of the visual features corresponding to the visual modality data, that is, the visual fusion modality representation, is calculated according to the following formula:

[0053]

[0054] Where R v is the visual fusion modality representation, when i is v, Represents visual unimodal representation, when i is a, represents the joint visual-audio representation, when i is t, Representing joint visual-text representation.

[0055] In this embodiment, after obtaining the fused modal representation of the modal features, the splicing unit splices the fused modal representation of the modal features in the feature dimension to obtain a representation, namely the spliced ​​fused modal representation, and the classifier then performs emotion recognition based on the spliced ​​fused modal representation to obtain an emotion recognition result.

[0056] It should be noted that for scenarios with missing modalities, the fused modal representation of the missing modal data can be padded with 0 to achieve unification of the feature dimension of the spliced ​​fused modal representation.

[0057] It is understandable that the classifier of this embodiment can realize the sentiment classification task or the sentiment regression task, and is specifically set according to the actual application scenario and is not limited here.

[0058] Furthermore, the training process of the hybrid modal expert model of this embodiment is described in detail. In order to better learn the unimodal representation and joint representation of modal features, the hybrid modal expert model adopts a two-stage training strategy, namely the unimodal expert pre-training stage and the hybrid modal expert model training stage. First, each modal data will first extract modal features through the pre-training model. In the unimodal expert pre-training stage, the modal features of each modal data are used to independently train to obtain pre-trained unimodal experts for each modality. In the hybrid modal expert model training stage, the pre-trained unimodal experts obtained in the first stage are continued to be trained for various modality-missing scenarios, and the unimodal representation and joint representation of the modal features of each modal data are extracted and dynamically fused through soft routing to finally obtain a trained hybrid modal expert model. The trained hybrid modal expert model is finally used for emotion recognition in real modality-missing scenarios.

[0059] See Figure 4 , Figure 4 FIG. 1 is a schematic diagram of a training process of a hybrid modality expert model provided by an embodiment of the present invention. Figure 4 As shown, the training process of the hybrid modality expert model in this embodiment may include the following steps:

[0060] S1: Obtain an audio modal data training set, a text modal data training set, and a visual modal data training set, wherein the audio modal data training set includes a plurality of audio modal data with true emotional labels, the text modal data training set includes a plurality of text modal data with true emotional labels, and the visual modal data training set includes a plurality of visual modal data with true emotional labels.

[0061] It can be understood that for the mixed modal expert model that realizes the sentiment classification task, the true sentiment label assigned to the modal data in the modal data training set is the category sentiment label; for the mixed modal expert model that realizes the sentiment regression task, the true sentiment label assigned to the modal data in the modal data training set is the sentiment score label.

[0062] S2: Use the pre-trained models corresponding to different modal data to extract features from the audio modal data training set, the text modal data training set, and the visual modal data training set to obtain an audio modal feature set, a text modal feature set, and a visual modal feature set.

[0063] In this embodiment, a pre-trained wav2vec-large model is used to perform feature extraction on all audio modal data in the audio modal data training set to obtain an audio modal feature set, which includes the audio features of all audio modal data. A pre-trained DeBERTa-large model is used to perform feature extraction on all text modal data in the text modal data training set to obtain a text modal feature set, which includes the text features of all text modal data. A pre-trained MA-Net model is used to perform feature extraction on all visual modal data in the visual modal data training set to obtain a visual modal feature set, which includes the visual features of all visual modal data.

[0064] S3: Construct three Transformer models with the same structure, and pre-train the three Transformer models using the audio modality feature set, text modality feature set, and visual modality feature set to obtain pre-trained single-modal experts.

[0065] In this embodiment, to address the problem that existing methods are insufficient in extracting unimodal representations, corresponding modal experts are trained using data from each modality. First, three Transformer models with the same structure are constructed as the network structure of audio experts, text experts, and visual experts. Then, the three Transformer models are trained using the audio modality feature set, text modality feature set, and visual modality feature set, respectively, to obtain pre-trained unimodal experts, namely, pre-trained audio experts, pre-trained text experts, and pre-trained visual experts. After the unimodal expert pre-training stage, the pre-trained unimodal experts have learned the corresponding modal knowledge to extract unimodal representations.

[0066] For example, taking the audio expert pre-trained by the audio modal feature set training as an example, the specific training process is as follows: first, the audio feature X of the audio modal data in the audio modal feature set is a Input into the Transformer structure and output audio unimodal representation Expressed as Output audio unimodal representation The single-modal predicted emotion label Y is calculated through an independent fully connected layer FC, that is, the classifier a' , expressed as Then predict the sentiment label Y based on the single modality a' The loss value between the true emotional label Y is used to optimize the parameters of the Transformer structure, and finally a pre-trained audio expert is obtained.

[0067] It is understandable that for the sentiment classification task, the single-modal prediction of the sentiment label Y a'The calculation formula of the loss value between the true emotional label Y is: Loss = CrossEntropy(Y,Y a' ), CrossEntropy is the cross entropy loss. For the sentiment regression task, the single modality predicts the sentiment label Y a' The calculation formula of the loss value between the true emotional label Y is: Loss = MSE (Y, Y a' ), MSE is the mean square error loss.

[0068] S4: Construct an initial mixed modality expert model based on the pre-trained single modality expert, and train the initial mixed modality expert model using the audio modality feature set, text modality feature set, and visual modality feature set to obtain a trained mixed modality expert model.

[0069] In this embodiment, an initial hybrid modality expert model is constructed based on pre-trained unimodal experts using pre-trained audio experts, pre-trained text experts, pre-trained visual experts, soft routing, a splicing unit, and a classifier. The audio modality feature set, text modality feature set, and visual modality feature set are input into the initial hybrid modality expert model for training. Unimodal and joint representations are then learned simultaneously and dynamically fused to adapt to various modality-missing scenarios, ultimately resulting in a fully trained hybrid modality expert model.

[0070] For example, the training process of the initial mixed modality expert model is described by taking the modality missing scenario with only audio modality data as an example. First, the audio feature X of the audio modality data in the audio modality feature set is converted into a They are input into pre-trained audio experts, pre-trained text experts and pre-trained visual experts respectively, and each pre-trained unimodal expert outputs audio features X a The unimodal representation and joint representation of the audio are obtained by pre-training audio experts to output the audio unimodal representation. Expressed as Pre-trained text experts output audio-text joint representation Expressed as Pre-trained visual experts output joint audio-visual representations Expressed as

[0071] Secondly, the audio feature X of the audio modal data in the audio modal feature set is a Input into the soft router, and use the soft router to generate the audio feature X of the audio modal data a Dynamically adjust the expression weights of single-modal representation and joint representation, and transform the audio feature X of audio modal data into a The single modal representation and joint representation are dynamically fused to obtain the audio fusion modal representation Ra .

[0072] Finally, the audio fusion modal representation R that integrates all modal expert knowledge is a After being spliced ​​with the fused modal representation of the modal data of the missing scene, it will pass through a new fully connected layer, i.e., the classifier, to obtain the corresponding predicted emotion label Y′. Finally, the loss value between the predicted emotion label Y′ and the true emotion label Y is calculated through the loss function corresponding to the task to optimize the structural parameters of the pre-trained audio expert, pre-trained text expert, pre-trained visual expert, and soft routing, and finally a trained hybrid modal expert model is obtained.

[0073] It should be noted that for scenes with missing modalities, in order to achieve the unification of the feature dimension of the fused modal representation after splicing, the fused modal representation of the modal data of the missing scene is padded with 0.

[0074] To better extract unimodal representations, the hybrid modal expert emotion recognition method of the embodiment of the present invention explicitly defines individual modal experts and initializes the modal experts through unimodal expert pre-training. On this basis, the hybrid expert model is further trained to simultaneously learn unimodal and joint representations of modal features. Compared to existing hybrid expert models that primarily focus on unimodal tasks, use randomly initialized experts, and cannot leverage knowledge from multiple modalities, the method of the embodiment of the present invention strengthens the learning of modality-specific information, namely unimodal representations, overcoming the problem of insufficient discriminative power of representations extracted when only one modal data is available, and improving recognition performance in scenarios where multiple modalities are missing.

[0075] Furthermore, the performance of the hybrid modality expert emotion recognition method of the embodiment of the present invention is verified through simulation comparison experiments.

[0076] The hybrid modality expert emotion recognition method of the present invention was verified on three of the most widely recognized datasets in the field of emotion recognition. The datasets are:

[0077] (1) IEMOCAP dataset: It consists of five binary dialogue sessions in which actors improvise or perform scripted scenes. Each dialogue is further segmented into many utterances, each of which is labeled with a categorical emotion label.

[0078] (2) CMU-MOSI dataset: A collection of 2,199 video clips collected from YouTube. Each video clip is labeled with a sentiment score ranging from -3 (strongly negative) to +3 (strongly positive).

[0079] (3) CMU-MOSEI dataset: Contains 22,856 video clips from over 1,000 online YouTube speakers. All utterances are randomly selected from a variety of topics and monologue videos and follow the [-3, 3] sentiment annotation scheme in CMU-MOSI.

[0080] For the IEMOCAP dataset, weighted accuracy (WA) and unweighted accuracy (UA) were used as evaluation metrics. For the CMU-MOSI and CMU-MOSEI datasets, ACC and F1 score were used as evaluation metrics. Table 1 shows the experimental results of weighted accuracy for each method in the severe modality-missing scenario for the IEMOCAP dataset. MCTN stands for Multimodal Recurrent Translation Network, MMIN for Missing Modality Imagination Network, IF-MMIN for Missing Modality Imagination Network with invariance, and MRAN for Multimodal Reconstruction and Alignment Network.

[0081] Table 1. Comparison of performance in severe modality loss scenarios

[0082]

[0083] Table 2 shows the experimental results of various methods for all datasets in various modality-missing scenarios. GCNet represents a graph completion network, and IMDer represents an incomplete multimodal diffusion emotion recognition network. Higher values ​​in the table indicate better performance, {} represents the available modal combinations, and a, t, and v represent the audio, text, and visual modalities, respectively. It can be observed that the hybrid-modality expert emotion recognition method of this embodiment surpasses existing methods in almost all test metrics, demonstrating strong performance in all modality-missing scenarios.

[0084] Table 2. Comparison of indicators of different methods on three sentiment datasets

[0085]

[0086] See Figure 5 , Figure 5 The figure shows a comparison of the predictions of an actual conversation by the hybrid modal expert model provided by an embodiment of the present invention and the weights assigned by the three modal experts. To intuitively demonstrate the role of modal experts, a conversation from the IEMOCAP dataset was selected and the prediction results for each sentence by the hybrid modal expert model using only text experts and three modal experts in a modality-missing scenario where only text modal data is available were plotted, as well as the weight scores assigned to each modal expert by the soft router in the hybrid modal expert model. Figure 5As shown in the figure, it can be observed that text experts who rely solely on textual knowledge cannot make accurate predictions on some samples. However, the mixed modality expert model can make more accurate judgments by giving more weight to experts in other modalities and associating the text with the speaker's tone and expression.

[0087] In the second aspect, the embodiment of the present invention provides a hybrid modality expert emotion recognition system, which is applicable to the hybrid modality expert emotion recognition method provided in the first aspect. Figure 6 , Figure 6 This is a structural block diagram of a hybrid modality expert emotion recognition system provided by an embodiment of the present invention. Figure 6 The hybrid modality expert emotion recognition system of this embodiment includes:

[0088] A data preprocessing module is used to extract features from the modal data to be identified using pre-trained models corresponding to different modal data, and obtain modal features corresponding to the modal data in the modal data to be identified, wherein the modal data to be identified includes at least one of audio modal data, text modal data and visual modal data.

[0089] The emotion recognition module is used to use the trained hybrid modal expert model to perform emotion recognition on the input modal features to obtain the emotion recognition results of the modal data to be identified. Among them, the hybrid modal expert model is used to extract the single modal representation and joint representation of the modal features, and dynamically fuse the single modal representation and the joint representation to perform emotion recognition to obtain the emotion recognition results.

[0090] For the specific content and corresponding beneficial effects of the hybrid modal expert emotion recognition system, please refer to the relevant content of the hybrid modal expert emotion recognition method provided in the first aspect, which will not be elaborated here.

[0091] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations are intended to cover non-exclusive inclusion, such that an article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the article or device comprising the element.

[0092] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A mixed modality expert emotion recognition method, characterized in that: include: Extracting features from the modal data to be identified using pre-trained models corresponding to different modal data to obtain modal features corresponding to the modal data in the modal data to be identified; wherein the modal data to be identified includes at least one of audio modal data, text modal data, and visual modal data; Using the trained hybrid modal expert model to perform emotion recognition on the modal features, obtaining an emotion recognition result of the modal data to be recognized; wherein the hybrid modal expert model is used to extract a single modal representation and a joint representation of the modal features, and dynamically fuse the single modal representation and the joint representation to perform emotion recognition to obtain the emotion recognition result; The hybrid modality expert model includes a single modality expert, a soft router, a splicing unit and a classifier, wherein: The unimodal expert is used to extract the unimodal representation and the joint representation of the input modal features; The soft router is used to dynamically fuse the single modal representation and the joint representation of the modal feature according to the weights corresponding to the weights to obtain the fused modal representation of the modal feature; The splicing unit is used to splice the fused modal representations of the modal features; The classifier is used to perform emotion recognition based on the spliced ​​fusion modality representation to obtain the emotion recognition result; The unimodal experts include audio experts, text experts and visual experts, wherein: The audio expert is used to extract audio unimodal representation of audio features, text-audio joint representation of text features, and visual-audio joint representation of visual features; The text expert is used to extract a text unimodal representation of the text features, extract an audio-text joint representation of the audio features, and extract a visual-text joint representation of the visual features; The visual expert is used to extract a visual unimodal representation of the visual features, an audio-visual joint representation of the audio features, and a text-visual joint representation of the text features; When there is a modality missing scenario, before splicing the fused modality representations of the modal features, the method further includes: Fill the missing fusion modality representation with 0; For the audio features, the weights of their unimodal representation and joint representation are determined by the modal score set and the audio modal data after normalization operation; the modal score set and the modal score set are calculated by the MLP-based soft routing using the audio features; the modal score set and the modal score include audio modal score, audio-text modal score and audio-visual modal score.

2. The hybrid modality expert emotion recognition method according to claim 1, characterized in that The audio expert, the text expert, and the visual expert all have Transformer model structures.

3. The hybrid modality expert emotion recognition method according to claim 1, characterized in that The soft router is a multi-layer perceptron containing two hidden layers.

4. The hybrid modality expert emotion recognition method according to claim 1, characterized in that The weights corresponding to the unimodal representation and the joint representation of the text features corresponding to the text modality data are calculated according to the following formula: ； ； Where, Text features representing text modality data, represents the set sum of modal scores of text modal data, represents the text modality score, represents the text-audio modality score, represents the text-visual modality score, when for hour, represents the weight of the unimodal representation of the text, when for hour, represents the weight of the text-audio joint representation, when for hour, The weights representing the joint text-visual representation; The weights corresponding to the unimodal representation and the joint representation of the visual features corresponding to the visual modality data are calculated according to the following formula: ； ； Where, represents the visual features of the visual modality data, represents the set sum of modal scores of visual modality data, represents the visual modality score, represents the visual-audio modality score, represents the visual-textual modality score, when for hour, represents the weight of visual unimodal representation, when for hour, represents the weight of the visual-audio joint representation, when for hour, represents the weight of the joint visual-text representation.

5. The hybrid modality expert emotion recognition method according to claim 1, characterized in that: The training process of the mixed modality expert model includes: Acquire an audio modality data training set, a text modality data training set, and a visual modality data training set, wherein the audio modality data training set includes a plurality of audio modality data assigned with true emotion labels, the text modality data training set includes a plurality of text modality data assigned with true emotion labels, and the visual modality data training set includes a plurality of visual modality data assigned with true emotion labels; Performing feature extraction on the audio modality data training set, the text modality data training set, and the visual modality data training set using pre-trained models corresponding to different modality data to obtain an audio modality feature set, a text modality feature set, and a visual modality feature set; Constructing three Transformer models with the same structure, and pre-training the three Transformer models using the audio modality feature set, the text modality feature set, and the visual modality feature set, respectively, to obtain pre-trained single-modality experts; An initial mixed modality expert model is constructed based on the pre-trained single modality expert, and the initial mixed modality expert model is trained using the audio modality feature set, the text modality feature set, and the visual modality feature set to obtain a trained mixed modality expert model.

6. A hybrid modality expert emotion recognition system, characterized in that: The hybrid modality expert emotion recognition method according to any one of claims 1 to 5 comprises: a data preprocessing module, configured to extract features from the modal data to be identified using pre-trained models corresponding to different modal data, thereby obtaining modal features corresponding to the modal data in the modal data to be identified, wherein the modal data to be identified includes at least one of audio modal data, text modal data, and visual modal data; An emotion recognition module is configured to perform emotion recognition on the input modal features using a trained hybrid modal expert model to obtain an emotion recognition result for the modal data to be recognized, wherein the hybrid modal expert model is configured to extract a unimodal representation and a joint representation of the modal features, and dynamically fuse the unimodal representation and the joint representation to perform emotion recognition to obtain the emotion recognition result; The hybrid modality expert model includes a single modality expert, a soft router, a splicing unit and a classifier, wherein: The unimodal expert is used to extract the unimodal representation and the joint representation of the input modal features; The soft router is used to dynamically fuse the single modal representation and the joint representation of the modal feature according to the weights corresponding to the weights to obtain the fused modal representation of the modal feature; The splicing unit is used to splice the fused modal representations of the modal features; The classifier is used to perform emotion recognition based on the spliced ​​fusion modality representation to obtain the emotion recognition result; The unimodal experts include audio experts, text experts and visual experts, wherein: The audio expert is used to extract audio unimodal representation of audio features, text-audio joint representation of text features, and visual-audio joint representation of visual features; The text expert is used to extract a text unimodal representation of the text features, extract an audio-text joint representation of the audio features, and extract a visual-text joint representation of the visual features; The visual expert is used to extract a visual unimodal representation of the visual features, an audio-visual joint representation of the audio features, and a text-visual joint representation of the text features; The system also includes a filling module; The filling module is configured to fill in the missing modal representations with 0 before splicing the fused modal representations of the modal features when there is a modality missing scenario; For the audio features, the weights of their unimodal representation and joint representation are determined by the modal score set and the audio modal data after normalization operation; the modal score set and the modal score set are calculated by the MLP-based soft routing using the audio features; the modal score set and the modal score include audio modal score, audio-text modal score and audio-visual modal score.

Images