Hybrid mode expert emotion recognition method and system

Through the mixed modal expert emotion recognition method, single-modal and joint representations in multimodal emotion recognition model are extracted and fused, the problem of insufficient recognition performance in modal missing scenarios is solved, and the accuracy of emotion recognition is improved.

CN119089259AActive Publication Date: 2024-12-06GUANGZHOU DAYOU NETWORK TECH CO LTD +2

Patent Information

Application Number
CN202411051787.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-12-06
Estimated Expiration
2044-08-01

AI Technical Summary

Technical Problem

In the existing multimodal emotion recognition model, the recognition performance of the existing multimodal emotion recognition model has significantly decreased in the scenario of severe mode absence, especially when only one modal data is available, the discriminant power of extracting and characterization is insufficient.

Method used

The mixed modal expert emotion recognition method is adopted to extract modal features through pre-training models, and single-modal representation and joint representation are extracted using the mixed modal expert model. After dynamic fusion, emotional recognition is performed, which overcomes the recognition in the modal missing scene. Inadequate performance problem.

Benefits of technology

Extracting single-modal representation and joint representation of available modal data in severe modal missing scenarios improves recognition performance, enhances the learning of modal-specific information, and improves the recognition performance in multiple modal-loss scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089259A_ABST
    Figure CN119089259A_ABST
Patent Text Reader

Abstract

The invention discloses a mixed modal expert emotion recognition method and system, and relates to the technical field of emotion recognition, and the method comprises the steps: carrying out the feature extraction of to-be-recognized modal data through employing pre-training models corresponding to different modal data, and obtaining the modal features corresponding to the modal data in the to-be-recognized modal data; wherein the modal data to be identified comprises at least one of audio modal data, text modal data and visual modal data; performing emotion recognition on the modal features by using the trained hybrid modal expert model to obtain an emotion recognition result of the modal data to be recognized; wherein the mixed mode expert model is used for extracting single-mode representation and joint representation of the mode features, and emotion recognition is carried out after the single-mode representation and the joint representation are dynamically fused to obtain an emotion recognition result. The method provided by the invention overcomes the problem of insufficient discrimination of the extracted representation when only one piece of modal data is available, and improves the recognition performance in a multi-modal missing scene.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of emotion recognition, and in particular relates to a mixed modality expert emotion recognition method and system. Background Art

[0002] Emotion is a basic physiological and psychological need of human beings. It plays an important role in people's daily life, interpersonal relationships and perception of the world. Automatic emotion recognition systems have been widely used in social media analysis, service robot design, video games and mental health analysis. In recent years, multimodal emotion recognition systems have gradually replaced single-modal emotion models due to their ability to use information from multiple modalities (such as vision, hearing and language) to make comprehensive decisions and identify emotions.

[0003] However, the current multimodal emotion recognition model has a significant decline in recognition reliability in real-world scenarios where some modal data is missing or unavailable (e.g., all visual data is unavailable) due to sensor damage, speech recognition failure, and personal privacy protection, which has attracted widespread attention. There are two mainstream solutions to this problem: generative methods and joint learning methods.

[0004] The generative method is to train the model to generate missing modal data using non-missing modal data to complete the data of all modalities. Since the generation process of this type of method is often too time-consuming, it affects the real-time processing capability of the model and is rarely used for emotion recognition. The core idea of ​​the joint learning method is to train the model to extract modal information shared between different modalities, that is, the joint representation of the modalities. The general implementation method is to reconstruct the missing modalities through the decoder during the training of the model to constrain the learned modal representation to contain information from all modalities, such as Figure 1 Schematic diagram of the joint learning method shown in Figure 1. This type of method can extract modal joint representation for emotion recognition when given any modal input, and has been widely verified in multimodal emotion recognition tasks in modality-missing scenarios. However, the recognition performance of existing joint representation methods is significantly reduced in scenarios with severe modality loss, that is, when only one modality data is available. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a hybrid modality expert emotion recognition method and system. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] The present invention provides a mixed modality expert emotion recognition method, comprising:

[0007] Using pre-trained models corresponding to different modal data to extract features from the modal data to be identified, and obtaining modal features corresponding to the modal data in the modal data to be identified; wherein the modal data to be identified includes at least one of audio modal data, text modal data, and visual modal data;

[0008] The modal features are subjected to emotion recognition using the trained hybrid modal expert model to obtain the emotion recognition result of the modal data to be recognized; wherein the hybrid modal expert model is used to extract the unimodal representation and the joint representation of the modal features, and the unimodal representation and the joint representation are dynamically fused to perform emotion recognition to obtain the emotion recognition result.

[0009] The present invention provides a hybrid modality expert emotion recognition system, which is applicable to the hybrid modality expert emotion recognition method described in any one of the above embodiments, and the system comprises:

[0010] A data preprocessing module, used for extracting features of the modal data to be identified by using pre-trained models corresponding to different modal data, and obtaining modal features corresponding to the modal data in the modal data to be identified, wherein the modal data to be identified includes at least one of audio modal data, text modal data and visual modal data;

[0011] The emotion recognition module is used to use the trained hybrid modal expert model to perform emotion recognition on the input modal features to obtain the emotion recognition result of the modal data to be recognized, wherein the hybrid modal expert model is used to extract the single modal representation and the joint representation of the modal features, and dynamically fuse the single modal representation and the joint representation to perform emotion recognition to obtain the emotion recognition result.

[0012] Compared with the prior art, the present invention has the following beneficial effects:

[0013] The hybrid modality expert emotion recognition method of the present invention utilizes pre-trained models corresponding to different modality data to perform feature extraction on the modality data to be recognized, and obtains modal features corresponding to the modal data in the modality data to be recognized; wherein, the modality data to be recognized includes at least one of audio modality data, text modality data and visual modality data; utilizes the trained hybrid modality expert model to perform emotion recognition on the modality features, and obtains the emotion recognition result of the modality data to be recognized; wherein, the hybrid modality expert model is used to extract single modality representation and joint representation of the modality features, and dynamically fuses the single modality representation and the joint representation to perform emotion recognition to obtain the emotion recognition result. The hybrid modality expert emotion recognition method of the present invention simultaneously extracts unimodal representation and joint representation through a hybrid modality expert model, and then dynamically fuses all representations to obtain a complete modal representation, and uses the complete modal representation to realize emotion recognition. This method can simultaneously and clearly extract unimodal representation and joint representation of available modal data in scenarios where the modality is severely missing, strengthens the learning of modality-specific information, namely unimodal representation, overcomes the problem of insufficient discriminative power of the extracted representation when only one modality data is available, and improves recognition performance in scenarios where multiple modalities are missing.

[0014] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following specifically cites a preferred embodiment and describes it in detail with the accompanying drawings as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a schematic diagram of the joint learning method;

[0016] Figure 2 is a flow chart of a mixed modality expert emotion recognition method provided by an embodiment of the present invention;

[0017] Figure 3 is a schematic diagram of a mixed modality expert emotion recognition method provided by an embodiment of the present invention;

[0018] Figure 4 is a schematic diagram of a training process of a mixed modality expert model provided by an embodiment of the present invention;

[0019] Figure 5 It is a schematic diagram of the prediction comparison of an actual conversation by the hybrid modality expert model provided by an embodiment of the present invention and the weights assigned by the three modality experts;

[0020] Figure 6 It is a structural block diagram of a hybrid modality expert emotion recognition system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, a hybrid modality expert emotion recognition method and system proposed in accordance with the present invention are described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0022] The above and other technical contents, features and effects of the present invention are clearly presented in the following detailed description of the specific implementation modes in conjunction with the accompanying drawings. Through the description of the specific implementation modes, the technical means and effects adopted by the present invention to achieve the predetermined purpose can be more deeply and specifically understood. However, the attached drawings are only for reference and explanation purposes and are not used to limit the technical solutions of the present invention.

[0023] In the first aspect, the embodiment of the present invention provides a mixed modality expert emotion recognition method, see Figure 2 , Figure 2 is a flow chart of a hybrid modality expert emotion recognition method provided by an embodiment of the present invention. Figure 2 As shown, the hybrid modality expert emotion recognition method of this embodiment includes:

[0024] Step 1: Use the pre-trained models corresponding to different modal data to extract features of the modal data to be identified, and obtain the modal features corresponding to the modal data in the modal data to be identified.

[0025] The modal data to be identified includes at least one of audio modal data, text modal data and visual modal data.

[0026] Exemplarily, the modal data to be identified may be video data, which may be complete video data including audio modal data, text modal data, and visual modal data, or incomplete video data including one or two of audio modal data, text modal data, and visual modal data, i.e., video data in a modality-missing scenario.

[0027] Step 2: Use the trained hybrid modal expert model to perform emotion recognition on the modal features to obtain the emotion recognition results of the modal data to be recognized.

[0028] Among them, the hybrid modal expert model is used to extract the unimodal representation and joint representation of modal features, and the unimodal representation and the joint representation are dynamically fused to perform emotion recognition to obtain the emotion recognition result.

[0029] The hybrid modality expert emotion recognition method of the embodiment of the present invention simultaneously extracts unimodal representation and joint representation through a hybrid modality expert model, and then dynamically fuses all the representations to obtain a complete modal representation, and uses the complete modal representation to realize emotion recognition. This method can simultaneously and clearly extract unimodal representation and joint representation of available modal data in scenarios where the modality is severely missing, strengthens the learning of modality-specific information, namely unimodal representation, overcomes the problem of insufficient discriminability of the extracted representation when only one modality data is available, and improves recognition performance in scenarios where multiple modalities are missing.

[0030] See also Figure 3 , Figure 3 is a schematic diagram of a hybrid modality expert emotion recognition method provided by an embodiment of the present invention. Figure 3 As shown, in an optional embodiment, the pre-trained model includes: a pre-trained wav2vec-large model, a pre-trained DeBERTa-large model and a pre-trained MA-Net model. Among them, the pre-trained wav2vec-large model is used to extract audio features of audio modal data; the pre-trained DeBERTa-large model is used to extract text features of text modal data; and the pre-trained MA-Net model is used to extract visual features of visual modal data.

[0031] In this embodiment, for the audio modality data in the video data, a 512-dimensional acoustic feature is extracted using the pre-trained wav2vec-large model as the audio feature of the audio modality data, denoted by X a For the text modality data in the video data, the pre-trained DeBERTa-large model is used to encode each sentence into a 1024-dimensional vocabulary feature as the text feature of the text modality data, denoted as X t For the visual modality data in the video data, the MTCNN face detection model is used to capture the facial images in the video, and then the pre-trained MA-Net model is used to extract facial features. The facial features of all frames in a sentence are compressed into a 1024-dimensional feature as the visual feature of the visual modality data, denoted as X v When a modality data in the video data is missing, its modality feature can be represented by adding a superscript ∧, for example Figure 3 middle and

[0032] In an optional embodiment, the hybrid modal expert model includes a single modal expert, a soft router, a splicing unit and a classifier. The single modal expert is used to extract the single modal representation and the joint representation of the input modal features. The soft router is used to dynamically fuse the single modal representation and the joint representation of the modal features to obtain a fused modal representation of the modal features. The splicing unit is used to splice the fused modal representation of the modal features. The classifier is used to perform emotion recognition based on the spliced ​​fused modal representation to obtain an emotion recognition result.

[0033] In this embodiment, the unimodal experts include audio experts, text experts and visual experts. Optionally, the audio experts, text experts and visual experts are all Transformer model structures. The audio expert is used to extract audio unimodal representations of audio features, text-audio joint representations of text features, and visual-audio joint representations of visual features; the text expert is used to extract text unimodal representations of text features, audio-text joint representations of audio features, and visual-text joint representations of visual features; the visual expert is used to extract visual unimodal representations of visual features, audio-visual joint representations of audio features, and text-visual joint representations of text features.

[0034] In this embodiment, the soft routing is specifically used to obtain weights corresponding to the single modal representation and the joint representation of the modal features according to the input modal features, and dynamically fuse the single modal representation and the joint representation of the modal features according to the weights to obtain a fused modal representation of the modal features.

[0035] Optionally, the soft router is a multi-layer perceptron with two hidden layers. The introduction of the soft router can dynamically adjust the expression weights of the single modal representation and the joint representation for different modal data to be identified.

[0036] In this embodiment, the weights corresponding to the unimodal representation and the joint representation of the audio features corresponding to the audio modality data are calculated according to the following formula:

[0037]

[0038] In the formula, a represents audio modality data, t represents text modality data, v represents visual modality data, and X a represents the audio features of audio modal data, r represents soft routing, MLP represents multi-layer perceptron, S a represents the modal score set sum of audio modal data, represents the audio modality score, represents the audio-text modality score, represents the audio-visual modality score, softmax represents the normalized exponential function, when i is a, represents the weight of the audio unimodal representation. When i is t, represents the weight of the audio-text joint representation. When i is v, represents the weight of the joint audio-visual representation.

[0039] The weights corresponding to the unimodal representation and joint representation of the text features corresponding to the text modal data are calculated according to the following formula:

[0040]

[0041] Where, X t Represents the text features of text modality data, S t represents the set sum of modal scores for text modal data, represents the text modality score, represents the text-audio modality score, represents the text-visual modality score. When i is t, Represents the weight of the unimodal representation of the text. When i is a, represents the weight of the text-audio joint representation. When i is v, represents the weight of the joint text-visual representation.

[0042] The weights corresponding to the unimodal representation and joint representation of the visual features corresponding to the visual modality data are calculated according to the following formula:

[0043]

[0044] Where, X v Represents the visual features of visual modality data, S v represents the set sum of modal scores for visual modality data, represents the visual modality score, represents the visual-audio modality score, represents the visual-textual modality score. When i is v, represents the weight of visual unimodal representation. When i is a, represents the weight of the visual-audio joint representation. When i is t, represents the weight of the joint visual-text representation.

[0045] In this embodiment, for the audio corresponding to the audio modality data

[0046] The fusion modal representation of the feature, that is, the audio fusion modal representation, is calculated according to the following formula:

[0047]

[0048] In the formula, R a is the audio fusion modal representation. When i is a, Represents the audio unimodal representation. When i is t, represents the audio-text joint representation. When i is v, Representing audio-visual joint representation.

[0049] The fused modal representation of the text features corresponding to the text modal data, that is, the text fused modal representation, is calculated according to the following formula:

[0050]

[0051] In the formula, R t is the text fusion modality representation. When i is t, Represents the unimodal representation of text. When i is a, represents the joint representation of text and audio. When i is v, Representing joint text-visual representation.

[0052] The fused modal representation of the visual features corresponding to the visual modal data, that is, the visual fusion modal representation, is calculated according to the following formula:

[0053]

[0054] In the formula, R v is the visual fusion modality representation. When i is v, Represents visual unimodal representation. When i is a, represents the joint visual-audio representation. When i is t, Representing joint visual-text representation.

[0055] In this embodiment, after obtaining the fused modal representation of the modal features, the splicing unit splices the fused modal representation of the modal features in the feature dimension to obtain a representation, namely the spliced ​​fused modal representation, and the classifier then performs emotion recognition based on the spliced ​​fused modal representation to obtain an emotion recognition result.

[0056] It should be noted that, for the scenario where the modality is missing, the fused modal representation of the missing modal data can be padded with 0 to achieve the unification of the fused modal representation after splicing in the feature dimension.

[0057] It is understandable that the classifier of this embodiment can realize the sentiment classification task or the sentiment regression task, and is specifically set according to the actual application scenario and is not limited here.

[0058] Furthermore, the training process of the hybrid modal expert model of the present embodiment is described in detail. In order to better learn the unimodal representation and joint representation of modal features, the hybrid modal expert model adopts a two-stage training strategy, namely the unimodal expert pre-training stage and the hybrid modal expert model training stage. First, the modal features of each modal data will be extracted through the pre-training model. In the unimodal expert pre-training stage, the modal features of each modal data are independently trained to obtain pre-trained unimodal experts of each modality. In the hybrid modal expert model training stage, the pre-trained unimodal experts obtained in the first stage continue to be trained for various modality-missing scenarios, and the unimodal representation and joint representation of the modal features of each modal data are extracted and dynamically fused through soft routing, and finally a trained hybrid modal expert model is obtained. The trained hybrid modal expert model is finally used for emotion recognition in real modality-missing scenarios.

[0059] See also Figure 4 , Figure 4 FIG. 1 is a schematic diagram of a training process of a hybrid modality expert model provided by an embodiment of the present invention. Figure 4 As shown, the training process of the mixed modality expert model in this embodiment may include the following steps:

[0060] S1: Obtain an audio modal data training set, a text modal data training set and a visual modal data training set, wherein the audio modal data training set includes a plurality of audio modal data with true emotion labels, the text modal data training set includes a plurality of text modal data with true emotion labels, and the visual modal data training set includes a plurality of visual modal data with true emotion labels.

[0061] It can be understood that, for the mixed modal expert model that realizes the sentiment classification task, the true sentiment label assigned to the modal data in the modal data training set is the category sentiment label, and for the mixed modal expert model that realizes the sentiment regression task, the true sentiment label assigned to the modal data in the modal data training set is the sentiment score label.

[0062] S2: Use pre-trained models corresponding to different modal data to perform feature extraction on the audio modal data training set, the text modal data training set, and the visual modal data training set to obtain an audio modal feature set, a text modal feature set, and a visual modal feature set.

[0063] In this embodiment, the pre-trained wav2vec-large model is used to perform feature extraction on all audio modal data in the audio modal data training set to obtain an audio modal feature set, which includes the audio features of all audio modal data. The pre-trained DeBERTa-large model is used to perform feature extraction on all text modal data in the text modal data training set to obtain a text modal feature set, which includes the text features of all text modal data. The pre-trained MA-Net model is used to perform feature extraction on all visual modal data in the visual modal data training set to obtain a visual modal feature set, which includes the visual features of all visual modal data.

[0064] S3: Construct three Transformer models with the same structure, and use the audio modality feature set, text modality feature set, and visual modality feature set to pre-train the three Transformer models respectively to obtain pre-trained single-modal experts.

[0065] In this embodiment, in order to solve the problem that the existing methods are insufficient in extracting unimodal representations, the corresponding modal experts are trained using the data of each modality. First, three Transformer models with the same structure are constructed as the network structure of the audio expert, text expert and visual expert, and then the three Transformer models are trained respectively using the audio modality feature set, text modality feature set and visual modality feature set to obtain pre-trained unimodal experts, namely, pre-trained audio experts, pre-trained text experts and pre-trained visual experts. After the unimodal expert pre-training stage, the pre-trained unimodal experts have learned the corresponding modal knowledge to extract unimodal representations.

[0066] For example, taking the audio expert pre-trained by training the audio modal feature set as an example, the specific training process is to firstly transform the audio feature X of the audio modal data in the audio modal feature set into a Input into the Transformer structure and output audio unimodal representation Expressed as Audio unimodal characterization of the output The single-modal predicted sentiment label Y is calculated through an independent fully connected layer FC, i.e., the classifier a' , expressed as Then predict the sentiment label Y based on the single modality a' The loss value between the true emotional label Y is used to optimize the parameters of the Transformer structure, and finally a pre-trained audio expert is obtained.

[0067] It is understandable that for the sentiment classification task, the single-modal prediction of the sentiment label Y a'The calculation formula of the loss value between the true emotional label Y is: Loss = CrossEntropy(Y,Y a' ), CrossEntropy is the cross entropy loss. For the sentiment regression task, the single-modal prediction sentiment label Y a' The calculation formula of the loss value between the true emotional label Y is: Loss = MSE (Y, Y a' ), MSE is the mean square error loss.

[0068] S4: Construct an initial mixed modality expert model based on the pre-trained single modality expert, and train the initial mixed modality expert model using the audio modality feature set, the text modality feature set, and the visual modality feature set to obtain a trained mixed modality expert model.

[0069] In this embodiment, based on the pre-trained unimodal experts, the initial hybrid modal expert model is constructed using pre-trained audio experts, pre-trained text experts, pre-trained visual experts, soft routing, splicing units and classifiers. The audio modal feature set, text modal feature set and visual modal feature set are input into the initial hybrid modal expert model for training, and the unimodal representation and joint representation are further learned simultaneously, and they are dynamically fused to adapt to various modality missing scenarios, and finally a trained hybrid modal expert model is obtained.

[0070] For example, the training process of the initial mixed modality expert model is described by taking the modality missing scenario with only audio modality data as an example. First, the audio feature X of the audio modality data in the audio modality feature set is a The pre-trained audio experts, pre-trained text experts and pre-trained visual experts are input respectively, and each pre-trained unimodal expert outputs the audio feature X a The unimodal representation and joint representation of the pre-trained audio expert output audio unimodal representation Expressed as Pre-trained text experts output audio-text joint representations Expressed as Pre-trained visual experts output joint audio-visual representations Expressed as

[0071] Secondly, the audio feature X of the audio modal data in the audio modal feature set a Input into the soft router, and use the soft router to generate audio features X for the audio modal data a Dynamically adjust the expression weights of the single modal representation and the joint representation, and convert the audio feature X of the audio modal data into a The single modal representation and joint representation of the audio are dynamically fused to obtain the audio fusion modal representation Ra .

[0072] Finally, the audio fusion modal representation R that integrates all modal expert knowledge is a After being spliced ​​with the fused modal representation of the modal data of the missing scene, it will pass through a new fully connected layer, i.e., the classifier, to obtain the corresponding predicted emotion label Y′. Finally, the loss value between the predicted emotion label Y′ and the true emotion label Y is calculated through the loss function corresponding to the task to optimize the structural parameters of the pre-trained audio expert, pre-trained text expert, pre-trained visual expert, and soft routing, and finally a trained hybrid modal expert model is obtained.

[0073] It should be noted that for scenes with missing modalities, in order to achieve the unification of the fused modal representation after splicing in the feature dimension, the fused modal representation of the modal data of the missing scene is padded with 0.

[0074] In order to better extract unimodal representations, the hybrid modal expert emotion recognition method of the embodiment of the present invention explicitly defines each modal expert and initializes the modal expert through unimodal expert pre-training, and further learns the unimodal representation and joint representation of modal features through hybrid expert model training on this basis. Compared with the existing hybrid expert model that basically focuses on unimodal tasks, uses randomly initialized experts, and cannot utilize knowledge of multiple modalities, the method of the embodiment of the present invention strengthens the learning of modality-specific information, namely unimodal representation, overcomes the problem of insufficient discriminative power of the extracted representation when only one modal data is available, and improves the recognition performance in scenarios where multiple modalities are missing.

[0075] Furthermore, the performance of the hybrid modality expert emotion recognition method of the embodiment of the present invention is verified through simulation comparison experiments.

[0076] The mixed modality expert emotion recognition method of the present invention is verified on three most widely recognized datasets in the field of emotion recognition, which are:

[0077] (1) IEMOCAP dataset: It consists of 5 binary dialogue sessions in which actors improvise or perform scripted scenes. Each dialogue is further segmented into many utterances, each of which is labeled with a classified emotion label.

[0078] (2) CMU-MOSI dataset: A collection of 2199 video clips collected from YouTube. Each video clip is labeled with a sentiment score ranging from -3 (strongly negative) to +3 (strongly positive).

[0079] (3) CMU-MOSEI dataset: Contains 22,856 video clips from more than 1,000 online YouTube speakers. All utterances are randomly selected from various topics and monologue videos and follow the [-3, 3] sentiment annotation scheme in CMU-MOSI.

[0080] For the IEMOCAP dataset, weighted accuracy (WA) and unweighted accuracy (UA) are used as evaluation indicators. For the CMU-MOSI and CMU-MOSEI datasets, ACC and F1 scores are used as evaluation indicators. For the IEMOCAP dataset, the experimental results of the weighted accuracy of each method in the severe modality missing scenario are shown in Table 1, where MCTN is a multimodal recurrent translation network, MMIN is a missing modality imagination network, IF-MMIN is a missing modality imagination network with invariance characteristics, and MRAN is a multimodal reconstruction and alignment network.

[0081] Table 1. Comparison of performance in severe modality loss scenarios

[0082]

[0083] The experimental results of various methods for all datasets in various modality missing scenarios are shown in Table 2, where GCNet is a graph completion network, IMDer is an incomplete multimodal diffusion emotion recognition network, the higher the value in the table, the better the performance, {} represents the available modality combination, a, t and v represent the audio modality, text modality and visual modality respectively. It can be observed that the hybrid modality expert emotion recognition method of the embodiment of the present invention exceeds the existing methods in almost all test indicators and shows strong performance in all modality missing scenarios.

[0084] Table 2. Comparison of indicators of different methods on three sentiment datasets

[0085]

[0086] See also Figure 5 , Figure 5 The figure is a comparison of the predictions of a real conversation by the hybrid modality expert model provided by the embodiment of the present invention and a schematic diagram of the weights assigned by the three modality experts. In order to intuitively demonstrate the role of modality experts, a conversation in the IEMOCAP dataset is selected, and the prediction results of each sentence by the hybrid modality expert model using only text experts and three modality experts in the modality-missing scenario where only text modality data is available are plotted, as well as the weight scores assigned to each modality expert by the soft router in the hybrid modality expert model, such as Figure 5As shown in the figure, it can be observed that text experts who only rely on text knowledge cannot make accurate predictions on some samples, while the mixed modality expert model can make more accurate judgments by giving other modality experts more weight to associate the text with the speaker's tone and expression.

[0087] In a second aspect, an embodiment of the present invention provides a hybrid modality expert emotion recognition system, which is applicable to the hybrid modality expert emotion recognition method provided in the first aspect, see Figure 6 , Figure 6 is a structural block diagram of a hybrid modality expert emotion recognition system provided by an embodiment of the present invention. Figure 6 The hybrid modality expert emotion recognition system of this embodiment includes:

[0088] The data preprocessing module is used to extract features of the modal data to be identified using pre-trained models corresponding to different modal data, so as to obtain modal features corresponding to the modal data in the modal data to be identified, wherein the modal data to be identified includes at least one of audio modal data, text modal data and visual modal data.

[0089] The emotion recognition module is used to use the trained hybrid modal expert model to perform emotion recognition on the input modal features to obtain the emotion recognition results of the modal data to be recognized, wherein the hybrid modal expert model is used to extract the single modal representation and joint representation of the modal features, and dynamically fuse the single modal representation and the joint representation to perform emotion recognition to obtain the emotion recognition results.

[0090] For the specific content and corresponding beneficial effects of the hybrid modal expert emotion recognition system, please refer to the relevant content of the hybrid modal expert emotion recognition method provided in the first aspect, which will not be elaborated here.

[0091] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants are intended to cover non-exclusive inclusion, so that an article or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed. In the absence of more restrictions, the elements defined by the statement "including one..." do not exclude the existence of other identical elements in the article or device including the elements.

[0092] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A hybrid modality expert emotion recognition method, It is characterized in that include: Using pre-trained models corresponding to different modal data to extract features from the modal data to be identified, and obtaining modal features corresponding to the modal data in the modal data to be identified; wherein the modal data to be identified includes at least one of audio modal data, text modal data, and visual modal data; The modal features are subjected to emotion recognition using the trained hybrid modal expert model to obtain the emotion recognition result of the modal data to be recognized; wherein the hybrid modal expert model is used to extract the unimodal representation and the joint representation of the modal features, and the unimodal representation and the joint representation are dynamically fused to perform emotion recognition to obtain the emotion recognition result.

2. The hybrid modality expert emotion recognition method according to claim 1, It is characterized in that The pre-trained models include: a pre-trained wav2vec-large model, a pre-trained DeBERTa-large model and a pre-trained MA-Net model; Among them, the pre-trained wav2vec-large model is used to extract audio features of the audio modality data; the pre-trained DeBERTa-large model is used to extract text features of the text modality data; and the pre-trained MA-Net model is used to extract visual features of the visual modality data.

3. The hybrid modality expert emotion recognition method according to claim 2, It is characterized in that The hybrid modality expert model includes a single modality expert, a soft router, a splicing unit and a classifier, wherein: The unimodal expert is used to extract the unimodal representation and the joint representation of the input modal features; The soft routing is used to dynamically fuse the single modality representation and the joint representation of the modality feature to obtain a fused modality representation of the modality feature; The splicing unit is used to splice the fused modal representations of the modal features; The classifier is used to perform emotion recognition based on the spliced ​​fusion modal representation to obtain the emotion recognition result.

4. The hybrid modality expert emotion recognition method according to claim 3, It is characterized in that The unimodal experts include audio experts, text experts and visual experts, wherein: The audio expert is used to extract the audio unimodal representation of the audio feature, extract the text-audio joint representation of the text feature, and extract the visual-audio joint representation of the visual feature; The text expert is used to extract a text unimodal representation of the text feature, extract an audio-text joint representation of the audio feature, and extract a visual-text joint representation of the visual feature; The visual expert is used to extract the visual unimodal representation of the visual features, extract the audio-visual joint representation of the audio features, and extract the text-visual joint representation of the text features.

5. The hybrid modality expert emotion recognition method according to claim 4, It is characterized in that The audio expert, the text expert and the visual expert are all Transformer model structures.

6. The hybrid modality expert emotion recognition method according to claim 3, It is characterized in that The soft router is a multi-layer perceptron containing two hidden layers.

7. The hybrid modality expert emotion recognition method according to claim 3, It is characterized in that The soft routing is used to dynamically fuse the single modality representation and the joint representation of the modality feature to obtain a fused modality representation of the modality feature, including: The soft routing is used to obtain weights corresponding to the single modal representation and the joint representation of the modal feature according to the input modal feature, and dynamically fuse the single modal representation and the joint representation of the modal feature according to the weight to obtain a fused modal representation of the modal feature.

8. The hybrid modality expert emotion recognition method according to claim 7, It is characterized in that The weights corresponding to the unimodal representation and the joint representation of the audio features corresponding to the audio modality data are calculated according to the following formula: In the formula, a represents audio modality data, t represents text modality data, v represents visual modality data, and X a represents the audio features of audio modal data, r represents soft routing, MLP represents multi-layer perceptron, S a represents the modal score set sum of audio modal data, represents the audio modality score, represents the audio-text modality score, represents the audio-visual modality score, softmax represents the normalized exponential function, when i is a, represents the weight of the audio unimodal representation. When i is t, represents the weight of the audio-text joint representation. When i is v, weights representing the audio-visual joint representation; The weights corresponding to the unimodal representation and the joint representation of the text features corresponding to the text modality data are calculated according to the following formula: Where, X t Represents the text features of text modality data, S t represents the set sum of modal scores for text modal data, represents the text modality score, represents the text-audio modality score, represents the text-visual modality score. When i is t, Represents the weight of the unimodal representation of the text. When i is a, represents the weight of the text-audio joint representation. When i is v, The weights representing the joint text-visual representation; The weights corresponding to the unimodal representation and the joint representation of the visual features corresponding to the visual modality data are calculated according to the following formula: Where, X v Represents the visual features of visual modality data, S v represents the set sum of modal scores for visual modality data, represents the visual modality score, represents the visual-audio modality score, represents the visual-textual modality score. When i is v, represents the weight of visual unimodal representation. When i is a, represents the weight of the visual-audio joint representation. When i is t, represents the weight of the joint visual-text representation.

9. The hybrid modality expert emotion recognition method according to claim 3, It is characterized in that The training process of the mixed modality expert model includes: Acquire an audio modality data training set, a text modality data training set, and a visual modality data training set, wherein the audio modality data training set includes a plurality of audio modality data assigned with true emotion labels, the text modality data training set includes a plurality of text modality data assigned with true emotion labels, and the visual modality data training set includes a plurality of visual modality data assigned with true emotion labels; Using pre-trained models corresponding to different modal data, feature extraction is performed on the audio modal data training set, the text modal data training set, and the visual modal data training set to obtain an audio modal feature set, a text modal feature set, and a visual modal feature set; Constructing three Transformer models with the same structure, and pre-training the three Transformer models respectively using the audio modality feature set, the text modality feature set, and the visual modality feature set to obtain pre-trained single-modality experts; An initial mixed modality expert model is constructed based on the pre-trained single modality expert, and the initial mixed modality expert model is trained using the audio modality feature set, the text modality feature set and the visual modality feature set to obtain a trained mixed modality expert model.

10. A hybrid modality expert emotion recognition system, It is characterized in that The hybrid modality expert emotion recognition method according to any one of claims 1 to 9 comprises: A data preprocessing module, used for extracting features of the modal data to be identified by using pre-trained models corresponding to different modal data, and obtaining modal features corresponding to the modal data in the modal data to be identified, wherein the modal data to be identified includes at least one of audio modal data, text modal data and visual modal data; The emotion recognition module is used to use the trained hybrid modal expert model to perform emotion recognition on the input modal features to obtain the emotion recognition result of the modal data to be recognized, wherein the hybrid modal expert model is used to extract the single modal representation and the joint representation of the modal features, and dynamically fuse the single modal representation and the joint representation to perform emotion recognition to obtain the emotion recognition result.

Citation Information

Patent Citations

  • Feature fusion and decision fusion mixed multi-modal emotion recognition method

    CN112800875A

  • Speech emotion recognition method based on multi-classifier interactive learning

    CN112837701A

  • Cross-modal matching method of cross-modal attention screening network based on dynamic routing

    CN114676228A

  • Multi-modal sentiment analysis method fusing multiple features and attention mechanism

    CN116028846A

  • Emotion analysis method and system based on multi-modal feature fusion

    CN117113223A

Cited By

  • Multi-modal data prediction method and device based on hybrid expert attention network

    CN120974259A

  • Multimodal data prediction method and device based on hybrid expert attention network

    CN120974259B

  • Emotional feature visualization method and system based on modal mixing and dynamic evidence, terminal and storage medium

    CN122332930A

  • Emotion feature visualization method and system based on modal mixing and dynamic evidence, terminal and storage medium

    CN122332930B