Multi-modal emotion recognition model training system

Abstract

The invention discloses a multi-modal emotion recognition model training system. The system comprises a data preprocessing module which is responsible for cleaning, aligning and normalizing multi-modal data; the data enhancement module is used for generating an adversarial network by utilizing the multi-mode StarGANv2 and generating voice with target emotional intonations and corresponding facial expressions; the feature extraction module is used for extracting feature representations of texts, voices and videos; the multi-modal fusion module is used for carrying out multi-modal fusion; the emotion semantic space mapping module introduces a psychological emotion roulette theory, constructs an emotion concept map, maps fused features to an emotion semantic space by utilizing GATConv, and meanwhile, jointly optimizes classification loss and comparison loss; the knowledge distillation module adopts a multi-teacher heterogeneous distillation framework, and transmits knowledge to the student network by introducing a plurality of heterogeneous teacher models; and the adaptive training engine is used for dynamically matching the training difficulty with the model capability by calculating the gradient of the support set and updating meta-parameters.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion recognition, and particularly to an emotion recognition model training system for multi-modalities. Background Art

[0002] Emotion is an important part of human communication, involving subjective experiences, physiological reactions, and behavioral responses. Everyone has their own subjective feelings, and the body will have a series of physiological reactions, and emotions are expressed through behaviors such as expressions, speech, and body movements. These emotion expression ways constitute multi-modal information, providing a rich data source for emotion recognition. In the field of human-computer interaction, accurately recognizing and understanding the emotional state of users is the key to achieving natural and user-friendly interaction. Through the multi-modal emotion recognition model training system, a more intelligent and sensitive user emotion analysis system can be constructed to provide a more considerate and personalized service experience for users.

[0003] There are often complex associations and hierarchical structures among emotions. Existing technologies are difficult to effectively represent these high-order emotion concept associations, resulting in difficulties in distinguishing similar emotions, problems such as low emotion recognition accuracy and easy confusion of different emotion categories. Therefore, an emotion recognition model training system for multi-modalities is proposed. Summary of the Invention

[0004] The purpose of the present invention is to solve the deficiencies existing in the prior art, and an emotion recognition model training system for multi-modalities is proposed.

[0005] To achieve the above purpose, the present invention adopts the following technical solutions:

[0006] An emotion recognition model training system for multi-modalities, comprising:

[0007] A data preprocessing layer: responsible for collecting, cleaning, integrating, and standardizing medical and health data;

[0008] A data preprocessing module: responsible for cleaning, aligning, and normalizing multi-modal data;

[0009] A data augmentation module: using the multi-modal StarGANv2 generative adversarial network, by inputting real text and target emotions, generating speech with the target emotion intonation and corresponding facial expressions, and at the same time, ensuring the alignment of multi-modal data through consistency verification to solve the problem of scarce emotion data;

[0010] A feature extraction module: using pre-trained deep learning models (such as BERT, Wav2Vec2, 3D-ResNet, etc.) to extract the feature representations of text, speech, and video respectively;

[0011] Multi-modal Fusion Module: It adopts a dynamic fusion network to perform multi-modal fusion by calculating the weights of different modal features. The weight calculation is based on the attention mechanism and is dynamically adjusted according to the importance of different modalities;

[0012] Emotional Semantic Space Mapping Module: It introduces the psychological emotion wheel theory to construct an emotional concept graph, and uses GATConv to map the fused features to the emotional semantic space. At the same time, it jointly optimizes the classification loss and the contrastive loss to reduce emotional confusion;

[0013] Knowledge Distillation Module: It adopts a multi-teacher heterogeneous distillation architecture. By introducing multiple heterogeneous teacher models to transfer knowledge to the student network, the number of parameters of the student model is reduced while maintaining high accuracy. These teacher models each have different advantages and characteristics, and can capture different features and information in multi-modal data. By fusing and transferring this knowledge to the student network, the knowledge distillation module helps the student model learn a more comprehensive and accurate emotion recognition ability;

[0014] Adaptive Training Engine: It integrates the meta-learning adaptive curriculum and the modal missing robust training strategy, designs the MetaCurriculum class, and makes the dynamic matching of the training difficulty and the model ability by calculating the support set gradient and updating the meta-parameters, and enhances the robustness of the model to modal missing. At the same time, it automatically selects the training sample difficulty level based on the meta-loss to improve the convergence speed of the model.

[0015] The above technical solutions further include:

[0016] Furthermore, the data augmentation module uses the multi-modal StarGANv2 generative adversarial network. By inputting real text and target emotion, it generates speech with the target emotional intonation and corresponding facial expressions, including the following steps:

[0017] Model Selection and Initialization: Select the multi-modal StarGANv2 as the basic model of the generative adversarial network, and initialize the parameters of the generator G and the discriminator D;

[0018] Data Preparation: Collect real-world text data, and the text data should contain rich emotional expressions. Set labels for the target emotion, such as "happy", "sad", "angry", etc.;

[0019] Input Processing: Input the real text data into the generator G, and at the same time specify the target emotion label. The generator G generates speech with the target emotional intonation and corresponding facial expressions according to the input text and the target emotion;

[0020] Generative adversarial training: The discriminator D receives the generated speech and facial expression data, as well as the corresponding real-world data, and attempts to distinguish between them. The generator G attempts to deceive the discriminator D so that it cannot accurately distinguish between the generated data and the real data. By alternately optimizing the parameters of the generator G and the discriminator D, the quality and authenticity of the generated data are gradually improved;

[0021] Consistency check: After generating the data, a consistency check is performed by calculating the distance between the generated data and the real data in the emotional feature space to ensure that the generated speech, facial expressions, and the input text are consistent in emotional expression;

[0022] Data augmentation: The generated data that has passed the consistency check is added to the dataset.

[0023] Furthermore, for text data, the BERT (Bidirectional Encoder Representations from Transformers) model is used to extract features; for speech data, the Wav2Vec2 model is used to extract features; for video data, the 3D-ResNet model is used to extract features.

[0024] Furthermore, the multi-modal fusion module performs multi-modal fusion, including the following steps;

[0025] Feature extraction: Obtain the feature representations of the three modalities of text, speech, and video from the feature extraction module. The feature representations correspond to the feature vector representations f text , f audio , f video ;

[0026] Attention weight calculation: Based on the attention mechanism, calculate the weights of the features of each modality. By performing weighted summation on the feature vectors, important information is highlighted and unimportant information is suppressed. The specific steps of the weight calculation are as follows:

[0027] Calculate the attention score: For the feature vector f of each modality i , where i represents the modality, calculate an attention score s i , and the calculation formula is expressed as Among them, W i , b i are learnable parameters, s i = softmax(w i T tanh(W i f i + b i )) is a non-linear activation function, and the softmax function is used to ensure that the sum of the attention scores of all modalities is 1;

[0028] Calculate the weight: the attention score s i Directly use it as the weight w of the modal feature i , that is, w i = s i ;

[0029] Multimodal fusion: Use the calculated weight w i To perform weighted summation on the feature vectors of different modalities, so as to obtain the fused feature vector.

[0030] Furthermore, the specific steps of the emotional semantic space mapping module to map the fused features to the emotional semantic space using GATConv:

[0031] Introduce the psychological emotion wheel theory: According to the emotion wheel theory, define a multi-dimensional emotional space;

[0032] Construct an emotion concept graph: Based on the psychological emotion wheel theory, construct an emotion concept graph. The nodes in the emotion concept graph represent different emotions, and the edges represent the relationships between emotions. For example, pleasure and sadness are two opposing emotions on the emotion wheel, and they will be represented as two nodes connected by edges with negative weights in the graph;

[0033] Feature fusion: Receive the fused feature vector provided by the multimodal fusion module;

[0034] Map to the emotional semantic space using GATConv: Adopt the graph attention network (GATConv) to map the fused feature vector to the emotional semantic space. GATConv uses the node and edge information in the graph to learn the mapping relationship between the feature vector and the emotional space, that is, GATConv calculates the attention weights between nodes, performs weighted summation on the feature vector, and obtains the mapped emotional feature vector;

[0035] Jointly optimize the classification loss and the contrast loss: The classification loss is used to measure the difference between the predicted emotion label of the model and the true emotion label, and the contrast loss is used to measure the distance between different emotion feature vectors.

[0036] Furthermore, the specific steps of the multi-teacher heterogeneous distillation architecture:

[0037] Select teacher models: Select multiple heterogeneous teacher models. The teacher models are pre-trained models that perform excellently on different modalities, including a text teacher model (such as BERT), a speech teacher model (such as WavLM), and a visual teacher model (such as CLIP), which are used to capture different features and information in the multimodal data;

[0038] Extract teacher model features: For the given multi-modal input data (such as text, speech, and images), use each teacher model to extract its feature representation respectively;

[0039] Design the loss function: Guide the student model into the knowledge space of the teacher model. Use the Kullback-Leibler Divergence (KL divergence) to measure the difference between the student model and the teacher model in the softened output. At the same time, use L2 regularization to constrain the consistency of the intermediate layer feature representation of the student model with that of the teacher model. The loss function is expressed as where and represent the softened output probability distributions of the i-th teacher model and the student model respectively, and represent the intermediate layer feature representations of the i-th teacher model and the student model respectively. λ i is the weight coefficient of the i-th teacher model, used to balance the influence of different teacher models. γ is the weight coefficient of L2 regularization;

[0040] Train the student model: During the training process, use the multi-modal input data and its corresponding teacher model feature representations as supervision information, and update the parameters of the student model by minimizing the loss function. Since there are multiple teacher models providing supervision information, the student model can learn more comprehensive and accurate emotion recognition knowledge;

[0041] Evaluate and optimize: After the training is completed, use the validation set or test set to evaluate the performance of the student model. If the performance does not meet the requirements, adjust the hyperparameters such as the weight coefficient of the teacher model, the weight coefficient of L2 regularization, or the optimizer, and retrain the student model.

[0042] Furthermore, the MetaCurriculum class is responsible for dynamically adjusting the difficulty of training samples according to the current ability of the model, including the following steps:

[0043] Initialize the MetaCurriculum class, including setting initial meta-parameters, defining the support set and the query set;

[0044] Generate a local linear model: Use the LIME tool to generate a local linear model for the selected case. The local linear model will attempt to approximate the decision boundary of the X-CNN model near the selected case;

[0045] Update the meta-parameters according to the gradients on the support set;

[0046] Automatically select training samples of different difficulties based on the meta-loss.

[0047] Furthermore, the specific steps for updating the meta-parameters according to the gradients on the support set:

[0048] Calculate the gradient of the computational model: The support set is represented as where x i is the input sample, y i is the corresponding label, and the loss function of the model is L(f(x i ; θ), y i ), where f is the model and θ is the model parameter. The gradient on the support set is represented as

[0049] Update the meta-parameters: Use the gradient on the support set to update the meta-parameters. Let the meta-parameters be φ, and the update rule is represented as where L Q (φ) is the loss on the query set, α is the learning rate, and the query set is used to evaluate the performance of the model after updating the meta-parameters.

[0050] Furthermore, the specific steps of automatically selecting training samples of different difficulties based on the meta-loss are as follows:

[0051] By monitoring the change of the meta-loss, adjust the difficulty level of the training samples. The meta-loss is defined as the difference between the loss on the query set and a certain baseline loss, that is, L meta (φ) = L Q (φ) - L baseline , where L baseline represents a preset baseline loss value. When the meta-loss is large, it indicates that the current model has poor performance on the query set, and the difficulty of the training samples needs to be increased; when the meta-loss is small, it means that the model can already handle the samples of the current difficulty well, and the difficulty of the training samples can be appropriately reduced;

[0052] Randomly discard the input of certain modalities during training. For example, in the text-speech-visual multi-modal emotion recognition task, we can randomly discard one or more of the text, speech, or visual modalities. In this way, the model learns the emotion recognition ability in the absence of different modalities;

[0053] In each iteration, automatically select training samples of different difficulties according to the size of the meta-loss. Set a difficulty threshold T. When the meta-loss is greater than T, select difficult training samples; when the meta-loss is less than T, select easy training samples. The difficulty level is achieved by adjusting the sample distribution in the support set and the query set.

[0054] The present invention has the following beneficial effects:

[0055] In the present invention, a multi-modal StarGANv2 generative adversarial network is used to generate speech with a target emotional intonation and corresponding facial expressions, enhancing the generalization ability of the model, especially in the small-sample scenario. The accuracy of model training is improved. By introducing the psychological emotion wheel theory, hyperedges are constructed to represent high-order emotion concept associations, enhancing the expressive ability of the emotion semantic space. In addition, by jointly optimizing the classification loss and the contrast loss, different emotion categories can be more accurately distinguished, reducing emotion confusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 FIG. is a system block diagram of a system for training an emotion recognition model for multi-modal applications proposed by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0058] Please refer to Figure 1 As shown, the present invention is a system for training an emotion recognition model for multi-modal applications, including:

[0059] Data preprocessing module: responsible for cleaning, aligning, and normalizing multi-modal data;

[0060] Data augmentation module: Using a multi-modal StarGANv2 generative adversarial network, by inputting real text and target emotions, speech with a target emotional intonation and corresponding facial expressions are generated. At the same time, the alignment of multi-modal data is ensured through consistency verification to solve the problem of scarce emotion data;

[0061] Feature extraction module: Using pre-trained deep learning models (such as BERT, Wav2Vec2, 3D-ResNet, etc.) to extract feature representations of text, speech, and video respectively;

[0062] Multi-modal fusion module: Adopting a dynamic fusion network, by calculating the weights of different modal features, multi-modal fusion is performed. The weight calculation is based on the attention mechanism and is dynamically adjusted according to the importance of different modalities;

[0063] Emotion semantic space mapping module: Introducing the psychological emotion wheel theory, constructing an emotion concept graph, and using GATConv to map the fused features to the emotion semantic space. At the same time, the classification loss and the contrast loss are jointly optimized to reduce emotion confusion;

[0064] Knowledge Distillation Module: Adopts a multi-teacher heterogeneous distillation architecture. By introducing multiple heterogeneous teacher models, knowledge is transferred to the student network to reduce the number of parameters of the student model while maintaining high accuracy. These teacher models each have different advantages and characteristics, and can capture different features and information in multi-modal data. By fusing and transferring this knowledge to the student network, the knowledge distillation module helps the student model learn a more comprehensive and accurate emotion recognition ability;

[0065] Adaptive Training Engine: Integrates meta-learning adaptive curriculum and modality-missing robust training strategies, designs the MetaCurriculum class. By calculating the support set gradient and updating the meta-parameters, it enables the dynamic matching of training difficulty and model ability, and enhances the model's robustness to modality missing. At the same time, based on the meta-loss, it automatically selects the training sample difficulty level to improve the model's convergence speed.

[0066] In one embodiment, the data augmentation module utilizes the multi-modal StarGANv2 generative adversarial network. By inputting real text and target emotion, it generates speech with the target emotion intonation and corresponding facial expressions, including the following steps:

[0067] Model Selection and Initialization: Selects the multi-modal StarGANv2 as the basic model of the generative adversarial network, and initializes the parameters of the generator G and the discriminator D;

[0068] Data Preparation: Collects real-world text data, and the text data should contain rich emotional expressions. Sets labels for the target emotion, such as "happy", "sad", "angry", etc.;

[0069] Input Processing: Inputs the real text data into the generator G, and at the same time specifies the target emotion label. The generator G generates speech with the target emotion intonation and corresponding facial expressions according to the input text and target emotion;

[0070] Generative Adversarial Training: The discriminator D receives the generated speech and facial expression data, as well as the corresponding real-world data, and tries to distinguish them. The generator G tries to deceive the discriminator D so that it cannot accurately distinguish the generated data from the real data. By alternately optimizing the parameters of the generator G and the discriminator D, the quality and authenticity of the generated data are gradually improved;

[0071] Consistency Verification: After generating the data, calculates the distance between the generated data and the real data in the emotion feature space for consistency verification to ensure that the generated speech, facial expressions, and input text are consistent in emotional expression;

[0072] Data Augmentation: Adds the generated data that has passed the consistency verification to the data set.

[0073] In one embodiment, for text data, a BERT (Bidirectional Encoder Representations from Transformers) model is used to extract features; for speech data, a Wav2Vec2 model is used to extract features; and for video data, a 3D-ResNet model is used to extract features.

[0074] Text feature extraction:

[0075] Input processing: Convert the text data into the input format required by the BERT model, including word segmentation, adding special tokens (such as [CLS] and [SEP]), and converting to an ID sequence;

[0076] Model forward propagation: Input the processed input sequence into the BERT model for forward propagation calculation;

[0077] Feature extraction: Extract the feature representation from the output layer of the BERT model. Select the output vector corresponding to the [CLS] token as the overall feature representation of the text, or select the output vectors of specific layers or specific positions as finer-grained features;

[0078] Speech feature extraction:

[0079] Audio preprocessing: Preprocess the original audio signal, including sample rate conversion, normalization, etc.;

[0080] Model forward propagation: Input the preprocessed audio signal into the Wav2Vec2 model for forward propagation calculation;

[0081] Feature extraction: Extract the feature representation from the output layer of the Wav2Vec2 model. Select the last hidden state or the output vector after a specific transformation as the speech feature;

[0082] Video feature extraction:

[0083] Video preprocessing: Preprocess the original video, including frame sampling, size adjustment, normalization, etc.;

[0084] Model forward propagation: Input the preprocessed video frame sequence into the 3D-ResNet model for forward propagation calculation;

[0085] Feature extraction: Extract the feature representation from the output layer of the 3D-ResNet model. Select the output vector after the global average pooling layer as the overall feature representation of the video.

[0086] In one embodiment, the multimodal fusion module performs multimodal fusion, including the following steps;

[0087] Feature extraction: Obtain the feature representations of three modalities, namely text, speech, and video, from the feature extraction module. The feature representations correspond to the feature vector representations f text , f audio , f video ;

[0088] Attention weight calculation: Calculate the weights of each modality feature based on the attention mechanism. By performing weighted summation on the feature vectors, important information is highlighted and unimportant information is suppressed. The specific steps of the weight calculation are as follows:

[0089] Calculate the attention score: For the feature vector f of each modality i , where i represents the modality, calculate an attention score s i . The calculation formula is expressed as where W i , b i are learnable parameters, is a non-linear activation function, and the softmax function is used to ensure that the sum of the attention scores of all modalities is 1;

[0090] Calculate the weight: The attention score s i is directly used as the weight w of the modality feature i , that is, w i = s i ;

[0091] Multi-modal fusion: Use the calculated weight w i to perform weighted summation on the feature vectors of different modalities, thereby obtaining the fused feature vector.

[0092] In one embodiment, the specific steps of the emotional semantic space mapping module using GATConv to map the fused feature to the emotional semantic space are as follows:

[0093] Introduce the psychological emotion wheel theory: According to the emotion wheel theory, define a multi-dimensional emotional space;

[0094] Construct an emotional concept graph: Based on the psychological emotion wheel theory, construct an emotional concept graph. The nodes in the emotional concept graph represent different emotions, and the edges represent the relationships between emotions. For example, joy and sadness are two opposing emotions on the emotion wheel, and they will be represented as two nodes connected by edges with negative weights in the graph;

[0095] Feature fusion: Receive the fused feature vector provided by the multi-modal fusion module;

[0096] Mapping to the emotional semantic space using GATConv: The graph attention network (GATConv) is adopted to map the fused feature vectors to the emotional semantic space. GATConv utilizes the node and edge information in the graph spectrum to learn the mapping relationship between the feature vectors and the emotional space. That is, GATConv calculates the attention weights between nodes, performs weighted summation on the feature vectors, and obtains the mapped emotional feature vectors.

[0097] Jointly optimizing the classification loss and the contrastive loss: The classification loss is used to measure the difference between the predicted emotional labels and the true emotional labels of the model, and the contrastive loss is used to measure the distance between different emotional feature vectors.

[0098] In one embodiment, the specific steps of the multi-teacher heterogeneous distillation architecture are as follows:

[0099] Selecting teacher models: Multiple heterogeneous teacher models are selected. The teacher models are pre-trained models that perform excellently in different modalities, including a text teacher model (such as BERT), a speech teacher model (such as WavLM), and a visual teacher model (such as CLIP), which are used to capture different features and information in the multi-modal data.

[0100] Extracting teacher model features: For the given multi-modal input data (such as text, speech, and images), their feature representations are respectively extracted using each teacher model.

[0101] Designing the loss function: The student model is guided into the knowledge space of the teacher models. The Kullback-Leibler Divergence is used to measure the difference between the student model and the teacher models in the softened outputs. At the same time, L2 regularization is used to constrain the consistency of the intermediate layer feature representations of the student model and the teacher models. The loss function is expressed as where and respectively represent the softened output probability distributions of the i-th teacher model and the student model, and respectively represent the intermediate layer feature representations of the i-th teacher model and the student model, and λ i is the weight coefficient of the i-th teacher model, which is used to balance the influence of different teacher models, and γ is the weight coefficient of L2 regularization.

[0102] Training the student model: During the training process, the multi-modal input data and their corresponding teacher model feature representations are used as supervision information, and the parameters of the student model are updated by minimizing the loss function. Since there are multiple teacher models providing supervision information, the student model can learn more comprehensive and accurate emotion recognition knowledge.

[0103] Evaluation and Optimization: After training is completed, use the validation set or test set to evaluate the performance of the student model. If the performance does not meet the requirements, adjust hyperparameters such as the weight coefficients of the teacher model, the weight coefficients of L2 regularization, or the optimizer, and retrain the student model.

[0104] In one embodiment, the MetaCurriculum class is responsible for dynamically adjusting the difficulty of training samples according to the current ability of the model, including the following steps:

[0105] Initialize the MetaCurriculum class, including setting initial meta-parameters, defining the support set and the query set;

[0106] Generate a local linear model: Use the LIME tool to generate a local linear model for the selected cases, and the local linear model will attempt to approximate the decision boundary of the X-CNN model near the selected cases;

[0107] Update the meta-parameters according to the gradients on the support set;

[0108] Automatically select training samples of different difficulties based on the meta-loss.

[0109] In one embodiment, the specific steps of updating the meta-parameters according to the gradients on the support set:

[0110] Calculate the gradients of the model: The support set is represented as where x i is the input sample, y i is the corresponding label, and the loss function of the model is L(f(x i ; θ), y i ), where f is the model and θ is the model parameters. The gradients on the support set are represented as

[0111] Update the meta-parameters: Use the gradients on the support set to update the meta-parameters. Let the meta-parameters be φ, and the update rule is represented as where L Q (φ) is the loss on the query set, α is the learning rate, and the query set is used to evaluate the performance of the model after updating the meta-parameters.

[0112] In one embodiment, the specific steps of automatically selecting training samples of different difficulties based on the meta-loss:

[0113] By monitoring the change of the meta-loss, adjust the difficulty level of the training samples. The meta-loss is defined as the difference between the loss on the query set and a certain reference loss, that is, L meta (φ) = L Q (φ) - L baseline , where L baselineDenote a preset baseline loss value. When the meta-loss is large, it indicates that the current model has poor performance on the query set and the difficulty of training samples needs to be increased; when the meta-loss is small, it means that the model can already handle samples of the current difficulty well and the difficulty of training samples can be appropriately reduced.

[0114] During the training process, randomly discard the inputs of some modalities. For example, in the text-speech-vision multi-modal emotion recognition task, we can randomly discard one or more of the text, speech, or vision modalities. In this way, the model learns the emotion recognition ability in the absence of different modalities.

[0115] In each iteration, automatically select training samples of different difficulties according to the size of the meta-loss. Set a difficulty threshold T. When the meta-loss is greater than T, select difficult training samples; when the meta-loss is less than T, select easy training samples. The difficulty level is achieved by adjusting the sample distribution in the support set and the query set.

[0116] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An emotion recognition model training system for multimodality, characterized in that, Including: Data preprocessing module: responsible for cleaning, aligning, and normalizing multimodal data; Data augmentation module: Utilize the multimodal StarGANv2 generative adversarial network. By inputting real text and target emotion, generate speech with the target emotional intonation and corresponding facial expressions. Meanwhile, ensure the alignment of multimodal data through consistency verification; Feature extraction module: Use pre-trained deep learning models to extract feature representations of text, speech, and video respectively; Multimodal fusion module: Adopt a dynamic fusion network. Through calculating the weights of different modal features, perform multimodal fusion. The weight calculation is based on the attention mechanism and is dynamically adjusted according to the importance of different modalities; Emotional semantic space mapping module: Introduce the psychological emotion wheel theory, construct an emotional concept graph, and use GATConv to map the fused features to the emotional semantic space. Meanwhile, jointly optimize the classification loss and contrast loss; Knowledge distillation module: Adopt a multi-teacher heterogeneous distillation architecture. By introducing multiple heterogeneous teacher models, transfer knowledge to the student network; Adaptive training engine: Integrate the meta-learning adaptive curriculum and the modality-missing robust training strategy, design the MetaCurriculum class. By calculating the support set gradient and updating the meta-parameters, enable the dynamic matching of the training difficulty and the model ability. Meanwhile, automatically select the training sample difficulty level based on the meta-loss.

2. The emotion recognition model training system for multi-modal according to claim 1, wherein The data augmentation module utilizes the multimodal StarGANv2 generative adversarial network. By inputting real text and target emotion, generates speech with the target emotional intonation and corresponding facial expressions, including the following steps: Model selection and initialization: Select multimodal StarGANv2 as the basic model of the generative adversarial network, and initialize the parameters of the generator G and the discriminator D; Data preparation: Collect real-world text data and set labels for the target emotion; Input processing: Input the real text data into the generator G, and at the same time specify the target emotion label. The generator G generates speech with the target emotional intonation and corresponding facial expressions according to the input text and target emotion; Generative adversarial training: The discriminator D receives the generated speech and facial expression data, as well as the corresponding real-world data, and tries to distinguish them. The generator G tries to deceive the discriminator D so that it cannot accurately distinguish the generated data from the real data; Consistency verification: After generating the data, perform consistency verification by calculating the distance between the generated data and the real data in the emotional feature space; Data augmentation: Add the generated data that has passed the consistency verification to the dataset.

3. The emotion recognition model training system for multi-modal according to claim 1, characterized in that, For text data, use the BERT (Bidirectional Encoder Representations from Transformers) model to extract features; for speech data, use the Wav2Vec2 model to extract features; for video data, use the 3D-ResNet model to extract features.

4. The emotion recognition model training system for multi-modal according to claim 1, characterized in that The multimodal fusion module performs multimodal fusion, including the following steps; Feature extraction: Obtain feature representations of three modalities, namely text, speech, and video, from the feature extraction module. The feature representations correspond to the feature vector representations f text , f audio , f video ; Attention weight calculation: Calculate the weights of each modal feature based on the attention mechanism. By performing weighted summation on the feature vectors, important information is highlighted and secondary information is suppressed. The specific steps of the weight calculation are as follows: Calculate the attention score: For the feature vector f of each modality i , where i represents the modality, calculate an attention score s i , and the calculation formula is expressed as where W i , b i are learnable parameters, is a non-linear activation function, and the softmax function is used to ensure that the sum of the attention scores of all modalities is 1; Calculate the weight: the attention score s i Directly use it as the weight w of the modal feature i , that is, w i = s i ; Multimodal fusion: Using the calculated weight w i Perform weighted summation on the feature vectors of different modalities to obtain the fused feature vector.

5. The emotion recognition model training system for multi-modal according to claim 1, wherein The specific steps of the emotion semantic space mapping module using GATConv to map the fused features to the emotion semantic space are as follows: Introduce the psychological emotion wheel theory: According to the emotion wheel theory, define a multi-dimensional emotion space. Construct an emotion concept graph: Based on the psychological emotion wheel theory, construct an emotion concept graph. The nodes in the emotion concept graph represent different emotions, and the edges represent the relationships between emotions. Feature fusion: Receive the fused feature vectors provided by the multi-modal fusion module. Map to the emotion semantic space using GATConv: Use the Graph Attention Network (GATConv) to map the fused feature vectors to the emotion semantic space. GATConv utilizes the node and edge information in the graph to learn the mapping relationship between the feature vectors and the emotion space. That is, GATConv calculates the attention weights between nodes, performs weighted summation on the feature vectors, and obtains the mapped emotion feature vectors. Jointly optimize the classification loss and the contrast loss: The classification loss is used to measure the difference between the emotion labels predicted by the model and the true emotion labels, and the contrast loss is used to measure the distance between different emotion feature vectors.

6. The emotion recognition model training system for multi-modal according to claim 1, characterized in that The specific steps of the multi-teacher heterogeneous distillation architecture are as follows: Select teacher models: Select multiple heterogeneous teacher models. The teacher models are pre-trained models that perform excellently in different modalities, including a text teacher model, a speech teacher model, and a visual teacher model, which are used to capture different features and information in the multi-modal data. Extract teacher model features: For the given multi-modal input data, use each teacher model to extract its feature representations respectively. Design loss function: Guide the student model to the knowledge space of the teacher model. The Kullback-Leibler Divergence is used to measure the difference between the softened outputs of the student model and the teacher model. At the same time, L2 regularization is used to constrain the consistency of the intermediate layer feature representations of the student model and the teacher model. The loss function is expressed as where and represent the softened output probability distributions of the i-th teacher model and the student model respectively, and represent the intermediate layer feature representations of the i-th teacher model and the student model respectively. λ i is the weight coefficient of the i-th teacher model, used to balance the influence of different teacher models, and γ is the weight coefficient of L2 regularization; Train the student model: During the training process, use the multi-modal input data and its corresponding teacher model feature representations as supervision information, and update the parameters of the student model by minimizing the loss function. Evaluation and optimization: After training, use the validation set or test set to evaluate the performance of the student model.

7. The emotion recognition model training system for multi-modal according to claim 1, wherein The MetaCurriculum class is responsible for dynamically adjusting the difficulty of training samples according to the current capabilities of the model, including the following steps: Initialize the MetaCurriculum class, including setting initial meta-parameters, defining the support set and the query set. Generate a local linear model: Use the LIME tool to generate a local linear model for the selected case. The local linear model will attempt to approximate the decision boundary of the X-CNN model near the selected case. Update the meta-parameters according to the gradients on the support set. Automatically select training samples of different difficulties based on the meta-loss.

8. The emotion recognition model training system for multi-modal according to claim 7, characterized in that The specific steps of updating the meta-parameters according to the gradients on the support set are as follows: Gradient of the computational model: The support set is represented as where x i is the input sample, y i is the corresponding label, and the loss function of the model is L(f(x i ; θ), y i ), where f is the model and θ is the model parameter. The gradient on the support set is represented as Update meta-parameters: Use the gradients on the support set to update the meta-parameters. Let the meta-parameters be φ, and the update rule is expressed as where L Q (φ) is the loss on the query set, α is the learning rate, and the query set is used to evaluate the performance of the model after updating the meta-parameters.

9. The emotion recognition model training system for multi-modal according to claim 7, wherein The specific steps of automatically selecting training samples of different difficulties based on the meta-loss are as follows: By monitoring the change of the meta-loss, the difficulty level of the training samples is adjusted. The meta-loss is defined as the difference between the loss on the query set and a certain baseline loss, that is, L meta (φ) = L Q (φ) - L baseline , where L baseline represents a preset baseline loss value; Randomly discard the inputs of certain modalities during the training process. In this way, the model learns the emotion recognition ability in the case of missing different modalities. In each iteration, training samples of different difficulties are automatically selected according to the magnitude of the meta-loss. A difficulty threshold T is set. When the meta-loss is greater than T, difficult training samples are selected; when the meta-loss is less than T, easy training samples are selected. The difficulty level is achieved by adjusting the sample distributions in the support set and the query set.

Images