Multimodal emotion recognition model training method and device, electronic equipment, storage medium and program product

Abstract

The application provides a multi-modal emotion recognition model training method and device, electronic equipment, storage medium and program product. Training samples including audio signals, video frame sequences and real emotion category labels are obtained. Audio features and multi-scale visual features are extracted from the training samples by the model, and respective corresponding emotion features and style features are decoupled. Emotion classification loss is calculated by using the emotion features and the real emotion category labels, and style classification loss is calculated by using the style features and pre-generated pseudo style labels. Cross-modal distribution alignment loss and feature diversity regularization loss are calculated based on the emotion features. After all the losses are combined as a total loss value, the parameters of the multi-modal emotion recognition model are optimized by back propagation until the preset convergence condition is met, and a trained emotion recognition model is obtained. The application can improve the generalization performance and stability of the emotion recognition task in a cross-cultural environment, and achieve the goal of cross-cultural robust emotion recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of emotion recognition, in particular to a multi-modal emotion recognition model training method and device, electronic equipment, storage medium and program product. BACKGROUND

[0002] Automatic emotion recognition (also known as affect recognition) refers to a technology for automatically perceiving and analyzing human emotional states from multi-source signals such as speech, facial expressions, body movements, and text language through computer technology. This technology has wide application prospects in virtual intelligent assistants, mental health monitoring, human-computer interaction systems, and other fields, and is an important research direction at the intersection of artificial intelligence and cognitive science.

[0003] With the development of deep learning technology, recent research has generally adopted a multi-modal fusion method to improve the accuracy of emotion recognition. Multi-modal refers to the joint modeling of multiple information sources such as audio, video, and text. For example, in the audio modality, Mel Frequency Cepstral Coefficients (MFCCs) or log Mel frequency spectrum features are often used as acoustic representations; in the visual modality, dynamic features such as facial movements, eye movements, and micro-expressions are extracted from video frame sequences using convolutional neural networks (CNNs), visual transformers (ViTs), and other models. Through effective fusion of multi-modal features, existing methods have achieved high recognition accuracy on multiple public datasets.

[0004] In fact, human emotional expression is highly diverse and subjective. Taking "anger" as an example, in some cultural backgrounds, individuals may directly express dissatisfaction through raising their voices and making vigorous gestures; in other cultures, especially in societies that emphasize restraint and etiquette, the same emotion may be expressed as silence, avoiding eye contact, or slight frowning. This cross-cultural expression difference makes the distribution of the same emotion category in the feature space significantly heterogeneous. In addition, different emotions may also exhibit similar behavioral characteristics, causing confusion within and across modalities. For example, "crying" can be a manifestation of sadness, but it can also occur in situations of extreme anger or excitement, making it difficult for classifiers based on visual or acoustic features to accurately distinguish between them. This type of high confusion further compresses the clarity of the classification decision boundary, increasing the risk of misjudgment.

[0005] Therefore, how to provide an effective modeling mechanism to solve the problem of large representation bias and weak generalization ability in the practical application of cross-culture and cross-individuals in the prior art. SUMMARY

[0006] In view of this, the purpose of the present application is to provide a multi-modal emotion recognition model training method, device, electronic equipment, storage medium and program product, which can improve the generalization performance and stability of the emotion recognition task in a cross-cultural environment, and ultimately achieve cross-cultural robust emotion recognition.

[0007] To achieve the above object, the technical scheme adopted by the embodiments of the present application is as follows:

[0008] In a first aspect, the present application provides a multi-modal emotion recognition model training method, which comprises: obtaining training samples; wherein the training samples comprise audio signals corresponding to videos, video frame sequences and real emotion category labels; extracting audio features and multi-scale visual features from the training samples by a multi-modal emotion recognition model and decoupling the audio features and the multi-scale visual features to obtain emotion features and style features corresponding to the audio features and the multi-scale visual features respectively; calculating emotion classification loss using the emotion features and the real emotion category labels and calculating style classification loss using the style features and pre-generated pseudo style labels, and calculating cross-modal distribution alignment loss and feature diversity regularization loss based on the emotion features; after weighting and combining the emotion classification loss, the style classification loss, the cross-modal distribution alignment loss and the feature diversity regularization loss into a total loss value, optimizing the parameters of the multi-modal emotion recognition model through back propagation until a preset convergence condition is met, and obtaining a trained emotion recognition model.

[0009] In a second aspect, the present application provides a multi-modal emotion recognition model training device, which comprises: an obtaining module for obtaining training samples; wherein the training samples comprise audio signals corresponding to videos, video frame sequences and real emotion category labels; a feature extraction module for extracting audio features and multi-scale visual features from the training samples by a multi-modal emotion recognition model; a feature decoupling module for decoupling the audio features and the multi-scale visual features to obtain emotion features and style features corresponding to the audio features and the multi-scale visual features respectively; a loss calculation module for calculating emotion classification loss using the emotion features and the real emotion category labels and calculating style classification loss using the style features and pre-generated pseudo style labels, and calculating cross-modal distribution alignment loss and feature diversity regularization loss based on the emotion features; and a training module for weighting and combining the emotion classification loss, the style classification loss, the cross-modal distribution alignment loss and the feature diversity regularization loss into a total loss value, optimizing the parameters of the multi-modal emotion recognition model through back propagation until a preset convergence condition is met, and obtaining a trained emotion recognition model.

[0010] In a third aspect, the present application provides an electronic device comprising a processor and a memory, the memory storing machine executable instructions capable of being executed by the processor, and the processor being capable of executing the machine executable instructions to implement the multi-modal sentiment recognition model training method according to the first aspect.

[0011] In a fourth aspect, the present application provides a storage medium having stored thereon machine executable instructions, the machine executable instructions, when executed by a processor, implementing the multi-modal sentiment recognition model training method according to the first aspect.

[0012] In a fifth aspect, the present application provides a program product having stored thereon machine executable instructions, the machine executable instructions, when executed, implementing the multi-modal sentiment recognition model training method according to the first aspect.

[0013] The multi-modal sentiment recognition model training method, device, electronic device, storage medium and program product provided by the embodiments of the present application, the method comprises: first acquiring training samples containing audio signals, video frame sequences and real emotion category labels, then extracting features from audio and visual modalities by a multi-modal sentiment recognition model and decoupling each modality into emotion features and style features, this decoupling mechanism effectively separates the emotion essence information and the culture-related bias, which is helpful to the subsequent robustness of the model to cultural specificity noise; subsequently, calculating an emotion classification loss using the emotion features and the real emotion labels to ensure that the model has accurate emotion discrimination ability, and calculating a style classification loss using the style features and the pre-generated pseudo style labels to prompt the model to explicitly model and distinguish different expression styles, further weakening the interference of cultural style on emotion judgment; on this basis, introducing a cross-modal distribution alignment loss to make the emotion features of audio and visual modalities consistent in a shared semantic space, especially when the cultural differences lead to inconsistent expressions between modalities, the reliable reasoning can still be maintained; at the same time, applying a feature diversity regularization loss to encourage the emotion features to maintain sufficient discrimination between different samples, and to guarantee the sensitivity of the model to subtle emotional changes; finally, weighting and combining the above four losses into a total loss function, continuously optimizing the model parameters through back propagation, and obtaining an emotion recognition model after convergence, which not only learns to ignore cultural style interference and focus on cross-cultural consistent emotional core representation in the training process, but also realizes robust fusion of multi-modal information and high generalization of classification decision, thereby improving the generalization performance and stability of the emotion recognition task in the cross-cultural environment, and achieving the goal of cross-cultural robust emotion recognition.

[0014] In order to make the above objectives, characteristics and advantages of the present application more apparent, comprehensible and easy to understand, the following preferred embodiments are specifically described below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as limiting the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0016] Figure 1 A schematic flow chart of the multi-modal emotion recognition model training method provided by the embodiments of the present application is shown;

[0017] Figure 2 An example diagram of the bidirectional semantic alignment provided by the embodiments of the present application is shown;

[0018] Figure 3 An example diagram of the hybrid attention mechanism provided by the embodiments of the present application is shown;

[0019] Figure 4 A functional module diagram of the multi-modal emotion recognition model provided by the embodiments of the present application is shown;

[0020] Figure 5 A functional module diagram of the multi-modal emotion recognition model training device provided by the embodiments of the present application is shown;

[0021] Figure 6 A structural block diagram of the electronic device provided by the embodiments of the present application is shown. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0023] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of the present application.

[0024] It is to be understood that the terms "first" and "second" and similar relating terms are used merely to distinguish one entity or action from another, and do not necessarily require or imply any such actual relationship or order between such entities or actions. Also, the terms "comprising", "containing", or any other similar words are intended to encompass non-exclusive inclusion, so that a process, method, article, or apparatus including a list of elements includes not only those elements, but also other elements not explicitly listed, or inherent to such process, method, article, or apparatus. Without more limitations, the element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus including the element.

[0025] As the emotion recognition task gradually moves towards refinement and practicality, its model not only needs to be able to perceive the complex subtle dynamic characteristics in the expression, but also must maintain the consistency and stability of the recognition results in the actual application of cross-culture and multi-scene. However, the inventors found in the research process that: the existing emotion recognition model has significant deficiencies in diversified feature mining, cross-emotion consistency representation, and hierarchical feature refinement, etc. Specifically:

[0026] On the problem of modality separation of emotion representation, the traditional method often models the video emotion features and the audio emotion features independently: the video part is responsible for capturing visual clues such as facial expressions and body movements, and the audio part focuses on acoustic features such as tone and speed, and then the two types of features are aligned and spliced. The result of lacking a unified cross-modal modeling framework is that the model is difficult to form a consistent semantic representation that integrates audio-visual information, resulting in that in complex and variable real scene tasks, the information between modalities cannot be fully complementary or even interfere with each other. This modality split phenomenon is particularly evident in cross-cultural and high confusion emotion recognition tasks, often causing the performance of the discrimination model to decrease or the decision confidence to decrease.

[0027] Secondly, the existing method generally lacks a task target driven explicit diversification constraint mechanism. In the actual recognition process, the model needs to continuously optimize its internal feature extraction process to ensure that the emotion classification decision always evolves towards the direction with the strongest discriminative ability. However, such optimization often leads the model to rely too much on a salient feature (such as relying only on the shape of the mouth to judge joy), and strongly over-fits to this local pattern. When facing greater environmental interference, or the salient feature cannot fully represent the target emotion (such as occlusion, light change or cross-cultural expression difference), the model is prone to problems such as decreased discriminative ability, weakened generalization performance and bias amplification, thereby significantly reducing the recognition accuracy and robustness of the overall task.

[0028] Furthermore, there are still shortcomings in multi-scale feature modeling. Although some hierarchical methods attempt to establish connections between high-level semantics and low-level features, due to the lack of fine extraction of observational information and multi-scale fusion and adaptive modeling of feature representations, they often only provide coarse-grained or insufficiently discriminative emotional representations. This causes the models to exhibit insufficient recognition accuracy and generalization stability when dealing with subtle discriminative details and long-sequence cross-modal interactions. Especially in real-world environments, emotional expressions often change dynamically with context and are subject to a large amount of interference, while existing models lack sufficient perception of multi-scale semantics and dynamic evolution mechanisms, resulting in insufficient temporal consistency and limited scene adaptability.

[0029] In summary, it can be seen that existing emotion recognition methods have significant shortcomings in terms of diverse feature mining, cross-emotional consistency representation, and hierarchical feature refinement, making it difficult to support high-precision emotion perception tasks in complex and dynamic environments.

[0030] Therefore, this invention provides a multimodal emotion recognition model training method that can effectively integrate multimodal signals and remove irrelevant style interference. By introducing explicit and diverse constraint mechanisms, the model can balance discriminability and generalization during training, avoiding over-reliance on local features. In addition, this invention can also realize hierarchical and multi-scale feature mining mechanisms to enhance the ability to capture and structurally represent subtle emotional semantics, thereby comprehensively improving the model's generalization ability and recognition accuracy in cross-scene and cross-cultural practical applications, overcoming the limitations of existing technologies in complex real-world environments.

[0031] Please see Figure 1 , Figure 1 A schematic flowchart of a multimodal emotion recognition model training method provided by an embodiment of the present invention is shown. The method may include steps S101 to S104, as described below:

[0032] S101: Obtain training samples.

[0033] In this embodiment of the invention, the training samples include the audio signal corresponding to the video, the video frame sequence, and the real emotion category label;

[0034] S102: Extract audio features and multi-scale visual features from training samples using a multimodal emotion recognition model and decouple them to obtain the emotion features and style features corresponding to the audio features and multi-scale visual features respectively;

[0035] S103: Calculate sentiment classification loss using sentiment features and real sentiment labels, and calculate style classification loss using style features and pre-generated pseudo-style labels. Also calculate cross-modal distribution alignment loss and feature diversity regularization loss based on sentiment features.

[0036] S104: After weighting and combining the emotion classification loss, the style classification loss, the cross-modal distribution alignment loss, and the feature diversity regularization loss into a total loss value, the parameters of the multi-modal emotion recognition model are optimized in a reverse propagation manner until a preset convergence condition is met, and a trained emotion recognition model is obtained.

[0037] Different from the prior art, the embodiment of the present application acquires training samples containing audio signals, video frame sequences, and real emotion category labels, then extracts features from audio and visual modalities respectively through a multi-modal emotion recognition model and decouples each modality into emotion features and style features, which effectively separates the emotion essence information and the culture-related bias, and helps to improve the robustness of the subsequent enhanced model to cultural specificity noise; subsequently, the emotion classification loss is calculated using the emotion features and the real emotion labels to ensure that the model has accurate emotion discrimination ability, and the style classification loss is calculated using the style features and the pre-generated pseudo style labels to prompt the model to explicitly model and distinguish different expression styles, further weakening the interference of cultural style on emotion judgment; on this basis, the cross-modal distribution alignment loss is introduced to make the emotion features of audio and visual modalities consistent in the shared semantic space, especially when the cultural differences lead to inconsistent expression between modalities, the reliable reasoning can still be maintained; at the same time, the feature diversity regularization loss is applied to encourage the emotion features to maintain sufficient discrimination between different samples, and to ensure the sensitivity of the model to subtle emotional changes; finally, the above four losses are weighted and combined into a total loss, and the model parameters are continuously optimized through reverse propagation, and the emotion recognition model obtained after the training not only learns to ignore the cultural style interference and focus on the cross-cultural consistent emotional core representation during the training process, but also realizes the robust fusion of multi-modal information, thereby improving the generalization performance and stability of the emotion recognition task in the cross-cultural environment, and achieving the goal of cross-cultural robust emotion recognition.

[0038] Next, the embodiment of the present application will combine related drawings to Figure 1 The multi-modal emotion recognition model training process is described.

[0039] In step S101, the audio signal is used to represent the acoustic information in the speech content, the video frame sequence is used to capture the visual dynamics such as facial expressions and body movements, and the real emotion category label is used as a supervision signal to guide the model to learn the correct emotion classification result.

[0040] In the embodiment of the present application, the data source of the training sample can be a public emotion database or a self-collected and labeled data set, which is in the form of synchronous audio and video data stream and is attached with emotion categories labeled by artificial labeling or determined through experimental paradigm, such as joy, sadness, anger, and other basic emotion categories or more fine-grained emotion states.

[0041] In optional embodiments, the embodiments of the present application can also perform inter-modal time alignment on the training samples to ensure that the audio signals and the video frames are accurately matched in the time axis, thereby providing a consistent data basis for subsequent multi-modal feature extraction and fusion.

[0042] In step S102, first, the model extracts features from the audio signal to obtain audio features that can reflect emotional semantics, denoted as Meanwhile, the video sequence can be extracted with multi-scale visual features according to different sampling frequencies, denoted as to capture multi-level visual information such as local micro-expression changes and global action trends; specifically, the and can be extracted according to step a1, as follows:

[0043] Step a1: the audio encoder in the multi-modal emotion recognition model extracts audio features from the audio, and the video encoder extracts multi-scale visual features from the video according to the preset different sampling frequencies;

[0044] In the embodiments of the present application, the multi-modal emotion recognition model includes an encoding structure that can process different modal inputs. The audio encoder is used to process the input audio signal and extract audio features that can represent speech content, tone changes, and sound production patterns. The video encoder is used to process the input video frame sequence to capture dynamic visual information of different granularities in the time dimension, thereby obtaining multi-scale visual features covering local instantaneous actions and global expression evolution rules.

[0045] For example, a 25-millisecond Hamming window is used, and then the audio signal is intercepted at a step size of 10 milliseconds. The frequency spectrum of each intercepted audio is analyzed, and the frequency spectrum energy is projected onto 128 triangular filter banks based on the Mel scale distribution to obtain the energy value of each frequency band. Finally, the energy value is taken as the natural logarithm to form a set of 128-dimensional logarithmic Mel filter bank feature vectors, which constitute the audio features arranged in time sequence.

[0046] For the video frame sequence, denoted as where n is the number of frames, each frame is an RGB image with a size of HxW. Three sampling frequencies, denoted as T H , T M and T L , can be set, and then a set of time visual features can be extracted by the video encoder based on the three sampling frequencies, defined as: , , N is the number of sampling frames, and C is the feature dimension (768 by default).

[0047] It can be understood that the design of different sampling frequencies enables the video encoder to capture subtle facial muscle movements at high frequencies and model long-term emotional evolution trends at low frequencies, thereby forming a multi-scale visual feature with hierarchical time perception capability. This multi-scale feature constitutes a time feature pyramid, enhancing the robustness of the model to different expression durations.

[0048] Further, after completing the extraction of audio features and multi-scale visual features , and , the system further decomposes the two types of features by a preset decoupling function, aiming to separate the emotional features (called emotional primitives) carrying emotional semantics in each modality and style features (called style primitives) reflecting non-emotional related factors such as speaker individual differences, culture, expression habits, etc. .

[0049] In the embodiment of the present application, the emotional features are used to encode cross-cultural universal emotional semantics (e.g., physiological response patterns of anger). The style features are used to encode culture-specific expression patterns (e.g., micro-expression changes in East Asian populations). The decoupling process is shown in step a2:

[0050] Step a2: decoupling the audio features and the multi-scale visual features by a preset decoupling function to obtain the corresponding emotional features and style features.

[0051] In the embodiment of the present application, the decoupling function is defined as a feature decomposition mechanism based on linear projection, which includes a feature extractor and two mutually orthogonal decoupling projection matrices; wherein the feature extractor is responsible for receiving the original modality features as input and outputting an intermediate representation that can be used for projection operation, and the two orthogonal decoupling projection matrices act on the intermediate representation respectively, and map it to independent and non-overlapping emotional subspace and style subspace through linear transformation. It can be understood that the orthogonal constraint ensures the statistical independence of the emotional features and the style features in the vector space, thereby effectively avoiding the coupling interference between the two types of information and improving the accuracy of subsequent classification and alignment tasks.

[0052] For ease of understanding, the decoupling process in the embodiment of the present application can be expressed as:

[0053]

[0054] wherein, denotes a decoupling function, refers to audio features , multi-scale visual features , and , denotes a feature extractor; , is a decoupling projection matrix, and satisfies , and is a constant term, refers to audio features corresponding emotional features , multi-scale visual features , and corresponding emotional features , and , refers to audio features corresponding style features , multi-scale visual features , and corresponding style features , and .

[0055] It should be understood that the decoupling process aims to achieve a decoupled representation of emotional content and style interference, so that emotional features focus on cross-modal shareable emotional semantic expression, while style features retain modality-specific or speaker-related individual information, thereby improving the generalization ability of the model in complex scenarios.

[0056] In optional embodiments, the self-alignment of by the emotional classifier can be achieved, forcing the semantic consistency of emotional states between different samples within the same modality. At the same time, the self-alignment of by the style classifier can be achieved, ensuring the stability of culture or individual-specific style features. This dual-classifier design enables the model to decouple and align representations related to emotion and style, thereby reducing intra-modal ambiguity and enhancing the robustness of the representation.

[0057] Next, based on the decoupled emotional features and style features, step S103 in the embodiments of the present application is performed.

[0058] As can be seen from step S103, before training the multi-modal sentiment recognition model, the embodiment of the present application designs a training loss function, which is composed of a sentiment classification loss, a style classification loss, a cross-modal distribution alignment loss and a feature diversity regularization loss. The following will give a detailed description of these loss functions.

[0059] In the embodiment of the present application, the sentiment classification loss can be measured by the cross-entropy error between the prediction result output by the sentiment classifier after the sentiment feature is input into the sentiment classifier and the real sentiment category label, so as to drive the model to accurately identify the sentiment category of the input sample. At the same time, the style classification loss can be measured by the cross-entropy error between the prediction result output by the style classifier after the style feature is input into the style classifier and the pre-generated pseudo style label. Specifically, it can be realized by the following steps b1 to b3, which are described as follows:

[0060] Step b1: clustering all training samples to obtain a plurality of clusters, and taking the cluster number of the cluster where each training sample is located as the pseudo style label of the style feature corresponding to the training sample; wherein each cluster represents a style mode;

[0061] It should be understood that, in the embodiment of the present application, since not all data sets used provide style annotations, an optional pseudo label generation method is proposed for the style feature to make up for this limitation, that is, clustering analysis is performed on the entire training set, for example, using the K-Means algorithm to divide it into several groups (such as K clusters). Each cluster is considered to represent a typical expression style mode, which may correspond to a certain cultural tendency, regional characteristics or individual habit. Then, a "pseudo style label" generated by the clustering result is assigned to each sample, that is, the cluster number to which the sample belongs, and the cluster number to which each training sample belongs is taken as the pseudo style label of the style feature corresponding to the training sample, and then the style classification loss is calculated to enhance the discriminability and structural property of the style feature.

[0062] Step b2: determining the sentiment prediction result of the sentiment feature by the sentiment classifier in the multi-modal sentiment recognition model, and determining the style prediction result of the style feature by the style classifier;

[0063] In the embodiment of the present application, the sentiment classifier in the multi-modal sentiment recognition model receives the sentiment feature as input and outputs the sentiment prediction result based on the features; at the same time, the style classifier receives the style feature as input and outputs the style prediction result. The two classifiers will be iteratively optimized in the training process.

[0064] After obtaining the sentiment prediction result and the style prediction result, the two types of classification losses are further calculated, that is, step b3 is performed.

[0065] Step b3: determine the emotion classification loss using the emotion prediction result and the real emotion label, and determine the style classification loss using the style prediction result and the pseudo style label.

[0066] In the embodiments of the present application, the emotion classification loss is determined by comparing the difference between the emotion prediction result output by the emotion classifier and the real emotion label of the training sample. This difference is usually quantified by a cross-entropy loss function or the like, so as to measure the performance of the model in the emotion classification task. As for the style classification loss, it is determined by comparing the difference between the style prediction result output by the style classifier and the aforementioned generated pseudo style label. Similarly, this difference is also quantified by a suitable loss function to evaluate the accuracy of the model in the style classification task.

[0067] For the convenience of understanding and description, the emotion classification loss and the style classification loss are calculated as follows:

[0068]

[0069]

[0070] wherein, represents the emotion classifier, represents the real emotion label; represents the style classifier; represents the pseudo style label.

[0071] In this way, not only the performance of the model in emotion recognition can be optimized, but also the recognition ability of different style patterns can be enhanced, so as to improve the robustness and generalization ability of the overall model.

[0072] Further, in order to ensure the consistency and alignment of these emotion features between different modalities, the present embodiments compare the distribution of the same type of features under two modalities. For example, the emotion feature distribution in the audio should be as close as possible to the emotion feature distribution in the video; similarly, the style feature in the audio should also be consistent with the style feature in the video. Based on the above consideration, the present embodiments propose a cross-modal distribution alignment loss for measuring the consistency of the emotion features extracted from the audio modality and the visual modality in the probability distribution level, so as to promote the emotion representation of different modalities to tend to a unified semantic space.

[0073] In an implementation, the embodiment of the present application selects the divergence between the probability distribution corresponding to the emotional features of the audio features and the multi-scale visual features respectively to measure the difference between the two. This divergence can be quantified in various ways, such as KL divergence (Kullback-Leibler divergence) or JS divergence (Jensen-Shannon divergence), which is not limited here. In this way, the system can evaluate and optimize the alignment of emotional features between different modalities, thereby improving the cross-modal consistency of the model.

[0074] For example, taking KL divergence as an example, the calculation process of the cross-modal distribution alignment loss is shown in the following formula:

[0075]

[0076] In the formula, the includes , and . The KL divergence measures the similarity between the probability distribution of the emotional features corresponding to the audio features and the probability distribution of the multi-scale visual features . The smaller the KL divergence, the more similar the two distributions are.

[0077] Next, the embodiment of the present application also designs a feature diversity regularization loss to constrain the expressiveness of the multi-scale visual features and the audio features after fusion, prevent feature degradation or redundancy, and encourage the model to learn emotional expressions with differentiated semantics at multiple scales, thereby encouraging the model to learn complementary, scale-specific information.

[0078] In the embodiment of the present application, the feature diversity regularization loss is determined on the basis of semantic alignment and feature fusion of the emotional features corresponding to the multi-scale visual features and the audio features respectively. As shown in steps c1 to c3, the following is explained:

[0079] Step c1: respectively aligning the emotional features of the multi-scale visual features to the emotional features of the audio features and aligning the emotional features of the audio features to the emotional features of the multi-scale visual features;

[0080] In the embodiment of the present application, the semantic alignment of the emotional features of the multi-scale visual features to the emotional features of the audio features and the semantic alignment of the emotional features of the audio features to the emotional features of the multi-scale visual features are bidirectional, aiming to ensure the semantic consistency of the emotional features of different modalities. In simple terms, the audio features are allowed to focus on which part of the video best supports the current emotional judgment, and at the same time the video features are allowed to refer to which segment of the audio is most representative of the emotion. For example, when a person's voice trembles, the model can automatically focus on the area of the picture where his facial muscles twitch slightly through this mechanism, thereby confirming that it is a tense or fearful emotion. This bidirectional information interaction helps the model establish a more complete and accurate emotional understanding.

[0081] In the embodiment of the present application, the specific implementation can be to adjust the feature representation by introducing an alignment layer or using a specific alignment function, so that the emotional features of the multi-scale visual features and the emotional features of the audio features are more consistent in the common semantic space.

[0082] For example, in one embodiment, to obtain more accurate feature representation, the bidirectional semantic alignment described above can be completed by a cross-attention mechanism. Please refer to Figure 2 , Figure 2 An example diagram of bidirectional semantic alignment provided by the embodiment of the present application is shown, and the implementation process is as follows:

[0083]

[0084]

[0085] Among them, refers to a cross-attention mechanism, refers to using as a query, is supervised to organize and compress the relevant information within its modality, and then shares with , thereby realizing the semantic alignment from the visual modality to the audio modality. Similarly, realizes the semantic alignment from the audio modality to the visual modality.

[0086] Step c2: performing feature fusion on the emotional features of the multi-scale visual features and the audio features after semantic alignment respectively, to obtain the fused emotional features of each scale;

[0087] In the embodiment of the present application, the purpose of feature fusion is to combine the emotional features from different modalities together to generate a more comprehensive and rich fused feature representation. The specific fusion method can be simple weighted summation, concatenation or more complex fusion mechanism such as attention mechanism. Through these methods, the system can generate fused multi-scale emotional features, so as to better capture the complementarity and consistency of cross-modal information.

[0088] In order to further optimize the features after cross-modal fusion, in an embodiment of the present application, before performing step b3, the fused multi-scale emotional features can also be enhanced by hybrid attention mechanism.

[0089] For the convenience of intuitive understanding, please refer to Figure 3 , Figure 3 An example diagram of the hybrid attention mechanism provided by the embodiment of the present application is shown. As shown in Figure 3 , the hybrid attention mechanism includes a spatial attention mechanism and a scale attention mechanism. The spatial attention mechanism is used to capture temporal dynamics at different granularities, and the implementation process is shown in the following formula:

[0090]

[0091] Among them, represents the spatial attention mechanism, indicates the fused emotional features, including , and ; ReLU represents the activation function. The importance of the spatial position is dynamically adjusted by the average pooling (AvgPool) and the fully connected layer (FC), so that the model can focus on the task-related area.

[0092] In the embodiment of the present application, the role of the scale attention mechanism is to focus on the emotional features of different scales, and to enhance the representation of multi-scale features by assigning different weights to the emotional features of each scale, and the implementation process is shown in the following formula:

[0093]

[0094] Among them, represents the scale attention mechanism, and Conv represents the time convolution with a fixed size.

[0095] In the optional implementation, the embodiment of the present application can apply 1D deep separable convolution with multiple kernel sizes, which expands the receptive field without increasing the parameters.

[0096] In actual operation, firstly, the spatial attention mechanism is applied to the fused multi-scale sentiment features to generate a spatially enhanced feature representation. Then, the scale attention mechanism is applied to the spatially enhanced feature representation for further scale enhancement. In this way, the system can more effectively capture the relevance between sentiment features at different spatial positions and different scales, thereby improving the quality of the feature representation.

[0097] Step c3: calculating the feature diversity regularization loss between the multi-scale sentiment features.

[0098] In the embodiments of the present application, the feature diversity regularization loss between the fused multi-scale sentiment features is calculated. The feature diversity regularization loss aims to ensure that the fused multi-scale sentiment features have a certain diversity, avoiding the decline of the model generalization ability due to too similar features. Specifically, the distance or similarity between the multi-scale sentiment features can be calculated and added as a regularization term to the total loss function. This regularization mechanism helps to improve the robustness and generalization performance of the model when processing multi-scale sentiment features.

[0099] In the embodiments of the present application, and , the feature diversity regularization loss calculation formula between them is as follows:

[0100]

[0101] It should be understood that and , and and , the feature diversity regularization loss calculation formula is as above, which will not be described here.

[0102] In optional embodiments, to ensure effective representation at different scales, the embodiments of the present application can also set a classifier for each scale sentiment feature in the model, and use the real sentiment label to constrain each scale sentiment feature.

[0103] In addition, the embodiments of the present application can also apply a cosine similarity constraint to force the features at different scales to maintain orthogonality, thereby encouraging the model to learn complementary and scale-specific information. This design guides each classifier to focus on discriminative features at its respective scale, and the fused representation collectively improves the overall classification performance.

[0104] Through the above steps, the embodiments of the present application not only optimize the model's ability to align cross-modal sentiment features, but also enhance the model's diversity and robustness when processing multi-scale sentiment features, thereby improving the overall performance of the model.

[0105] In summary, the embodiment of the present application designs a sentiment classification loss before training the model to ensure that the model can accurately identify the sentiment category corresponding to the input data; a style classification loss to guide the model to effectively learn and distinguish the expression style differences between different cultural backgrounds or individuals, which is one of the core supervision signals for decoupling sentiment and style; a cross-modal distribution alignment loss , which is to make the extracted sentiment and style features of audio and visual two different modalities consistent in statistical distribution, so as to realize cross-modal information alignment; a feature diversification regularization loss to encourage the model to learn complementary and non-redundant sentiment features at different time scales.

[0106] It should be understood that includes , , and corresponding classification losses respectively, includes , , and corresponding losses respectively, in order to facilitate description, the embodiments of the present application will be described as and . includes respectively with , and corresponding losses respectively, includes the losses between the fused sentiment features of each scale , and two by two, finally, in order to facilitate understanding, the total loss function in the embodiments of the present application can be represented as:

[0107]

[0108] wherein, is the total loss function, and are hyperparameters.

[0109] ​The embodiment of the present application combines the emotion classification loss, the style classification loss, the cross-modal distribution alignment loss and the feature diversity regularization loss into a total loss value, which means that the above four loss terms are linearly weighted and summed according to the preset weight coefficient to form a comprehensive optimization objective function. The total loss value comprehensively reflects the comprehensive performance of the model in emotion recognition accuracy, style feature distinguishability, cross-modal semantic consistency and feature expression diversity. Based on the above total loss function The embodiment of the present application can perform model training according to the following steps S104 until the training is completed.

[0110] During the execution of step S104, the gradient is calculated layer by layer based on the total loss value of the back propagation algorithm, and the trainable parameters of each component in the multi-modal emotion recognition model are updated, including but not limited to the weights of the encoder, the decoupling module, the classifier and the related attention mechanism. The training process continues to iterate until the preset convergence condition is met, for example, the loss value change is lower than the threshold, the performance of the verification set no longer improves or reaches the maximum number of training rounds, and finally the emotion recognition model is obtained. The model has the ability to robustly extract and decouple emotion and style features from audio and video inputs, and can accurately output emotion category judgment results.

[0111] Through the above training method, the embodiment of the present application strengthens the consistency and expression diversity of cross-modal emotional semantics while preserving individual style differences, and improves the accuracy and generalization performance of multi-modal emotion recognition in complex environments.

[0112] In some optional embodiments, for a multi-modal emotion recognition model that has been trained by the above training method, the input in the actual application process can be a piece of original text (such as product reviews or social media statements) or an original video to be analyzed. The original text needs to be preprocessed by word segmentation and vectorization consistent with the training stage, and then input into the model after being converted into a numerical tensor. The output of the model is the prediction result for the preset emotion categories (such as "positive", "negative", "neutral", or more granular "joy", "anger", "sadness", etc.).

[0113] Next, the model introduces a special style classifier, which takes the decoupled style features as input to predict the corresponding pseudo-style label. By comparing the prediction result with the pseudo-label generated by clustering, the style classification loss is calculated using the cross-entropy method.

[0114] Therefore, this loss requires two key pieces of information: one is the style feature extracted from the model, and the other is the pseudo-style label generated in advance by clustering. Its role is to force the style feature to form a distinguishable and stable distribution structure in the hidden space, thereby effectively encoding the cultural or individual expression characteristics unrelated to emotion.

[0115] For the convenience of the whole understanding of the model training method provided by the embodiments of the present application, please refer to Figure 4 , Figure 4 The functional module diagram of the multi-modal emotion recognition model provided by the embodiments of the present application is shown. It should be understood that Figure 4 Some main functional modules of the model are displayed, and some necessary structures of the model are omitted, but this does not affect the understanding of the embodiments of the present application. In combination with Figure 4 , the working process of the multi-modal emotion recognition model is as follows:

[0116] First, in order to effectively deal with the representation distribution shift problem caused by cross-cultural and individual expression differences, the embodiments of the present application set up an emotion decoupling representation module in the model, which can strip the redundant information related to the cultural, group or individual unique expression style from the original multi-modal data, and extract the essential features highly related to the emotion semantics itself. By explicitly modeling the independence relationship between expression style and emotion semantics, this module can effectively reduce the interference of cultural factors on emotion representation, and avoid the model from over-relying on the expression, voice or body language features in a specific cultural background. Finally, the emotion decoupling representation module enables the model to learn more general and transferable emotion features, which fundamentally alleviates the representation distribution inconsistency problem in cross-cultural scenarios.

[0117] Secondly, on the basis of obtaining the preliminarily decoupled emotion features, the embodiments of the present application further introduce a fine-grained feature alignment module to realize the double alignment of cross-modal and cross-cultural. Specifically, this module not only performs fine-grained feature alignment among voice, text, vision and other multi-modalities, but also explicitly models the representation distribution difference between different cultural domains in the hidden space, and realizes the culture-independent emotion semantic alignment by minimizing the cross-cultural feature distribution distance. This method can effectively improve the adaptability and robustness of the model to emotion expression in different cultural backgrounds, avoid the generalization degradation problem caused by over-reliance on single cultural features, and thus significantly enhance the reliability and stability of multi-modal emotion understanding in cross-domain applications.

[0118] Finally, the embodiments of the present application also set up a latent feature mining module to fully explore and utilize the structured emotion semantic information in the hidden space. This module captures the deep associations and subtle differences between emotion categories through hierarchical modeling and semantic aggregation of latent representations, thereby enhancing the reasoning and perception ability of the model in the high-dimensional emotion semantic space. At the same time, this module can dynamically select the latent features that contribute most to emotion discrimination, and realize the efficient fusion of multi-modal semantic information. Through this process, the model can exhibit stronger generalization ability and robustness when facing expression diversity and context diversity, and achieve more stable and accurate prediction performance in cross-cultural multi-modal emotion understanding tasks.

[0119] In summary, the multi-modal training model realizes cross-cultural consistent and highly robust emotion representation learning, which can significantly improve the generalization performance and stability of the emotion recognition system in a cross-cultural environment, and provides a new idea and method for constructing an emotion computing system that adapts to the diversity of the real world.

[0120] To perform the corresponding steps in the above embodiments and various possible manners, an implementation of a multi-modal emotion recognition model training apparatus 50 is given below. Please refer to Figure 5 , Figure 5 A functional module diagram of the multi-modal emotion recognition model training apparatus provided by the embodiment of the application is shown. The multi-modal emotion recognition model training apparatus 50 comprises an acquisition module 501, a feature extraction module 502, a feature decoupling module 503, a loss calculation module 504, and a training module 505.

[0121] The acquisition module 501 is configured to acquire training samples; wherein the training samples comprise an audio signal corresponding to a video, a video frame sequence, and a real emotion category label;

[0122] The feature extraction module 502 is configured to extract audio features and multi-scale visual features from the training samples by a multi-modal emotion recognition model;

[0123] The feature decoupling module 503 is configured to decouple the audio features and the multi-scale visual features to obtain emotion features and style features corresponding to the audio features and the multi-scale visual features, respectively;

[0124] The loss calculation module 504 is further configured to calculate an emotion classification loss using the emotion features and the real emotion category label, and a style classification loss using the style features and a pre-generated pseudo style label, and calculate a cross-modal distribution alignment loss and a feature diversity regularization loss based on the emotion features;

[0125] The training module 505 is further configured to combine the emotion classification loss, the style classification loss, the cross-modal distribution alignment loss, and the feature diversity regularization loss as a total loss value after weighting, and then optimize the parameters of the multi-modal emotion recognition model through back propagation until a preset convergence condition is met, to obtain a trained emotion recognition model.

[0126] It can be understood that the acquisition module 501, the feature extraction module 502, the feature decoupling module 503, the loss calculation module 504, and the training module 505 can cooperatively perform each step in the above method to achieve the corresponding technical effects. Figure 1

[0127] ​It should be noted that the multi-modal emotion recognition model training apparatus 50 provided by the embodiment of the present application can be specific hardware on a device or software or firmware installed on the device, etc. The apparatus provided by the embodiment of the present application has the same implementation principle and generated technical effects as the foregoing method embodiments, and for brief description, the part not mentioned in the device embodiment can refer to the corresponding content in the foregoing method embodiments. The person skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, apparatus and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0128] Optionally, the above-mentioned modules can be stored in the memory shown in the form of software or firmware (Firmware) or solidified in the operating system (Operating System, OS) of the electronic device 60, and can be executed by the processor in the electronic device 60. Figure 6 Figure 6 Meanwhile, the data, program code and the like required for executing the above-mentioned modules can be stored in the memory.

[0129] Please refer to Figure 6 , Figure 6 The structure block diagram of the electronic device provided by the embodiment of the present application is shown, which includes a memory 601, a processor 602 and a communication interface 603. The memory 601, the processor 602 and the communication interface 603 are directly or indirectly electrically connected with each other to realize the transmission or interaction of data. For example, these elements can realize the electrical connection through one or more communication buses or signal lines.

[0130] Optionally, the bus can be a peripheral component interconnect (Peripheral Component Interconnect, PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 6 In the foregoing description, only one thick line is used to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0131] ​In the embodiments of the present application, the processor 602 can be a general processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general processor can be a microprocessor or any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as hardware processor execution or executed by a combination of hardware and software modules in the processor. The software module can be in the memory 601, and the processor 602 reads the program instruction in the memory 601 to complete the steps of the above method in combination with the hardware.

[0132] In the embodiments of the present application, the memory 601 can be a non-volatile memory such as a hard disk (HDD) or a solid-state disk (SSD), and can also be a volatile memory (Volatile Memory) such as a RAM. The memory can also be any other medium capable of carrying or storing desired program executable code in the form of instructions or data structures and capable of being accessed by a computer, but is not limited to this. The memory in the embodiments of the present application can also be a circuit or any other device capable of realizing a storage function, used for storing instructions and / or data.

[0133] The memory 601 can be used to store software programs and modules, such as the instructions / modules of the multi-modal emotion recognition model training device 50 provided by the embodiments of the present application, which can be stored in the memory 601 in the form of software or firmware or solidified in the operating system (OS) of the electronic device 60. The processor 602 executes the software programs and modules stored in the memory 601 to perform various functional applications and data processing. The communication interface 603 can be used for signaling or data communication with other node devices.

[0134] It can be understood that Figure 6 The structure shown is only schematic, and the electronic device 60 can further include more or fewer components than those shown in Figure 6 or have a different configuration than Figure 6 The components shown can be implemented in hardware, software or a combination thereof. Figure 6

[0135] Based on the above embodiments, the present application further provides a readable storage medium, the readable storage medium stores a computer program, and the computer program is executed by a computer to make the computer execute the multi-modal emotion recognition model training method provided by the above embodiments. For details, please refer to the method embodiments, which will not be repeated here. ​

[0136] Based on the above embodiments, the application further provides a program product containing a computer program, wherein the processor can execute the computer program to implement the multi-modal sentiment recognition model training method provided by the embodiments of the application. For details, refer to the method embodiments, which are not described here again.

[0137] In the embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0138] In addition, the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0139] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0140] It should be noted that when the functions are realized in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various media that can store program codes.

[0141] The above merely provides the preferred embodiments of the present application, and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the principles and technical scope of the present application shall fall into the scope of the present application.

Claims

1. A method for training a multimodal emotion recognition model, characterized in that, The method includes: Obtain training samples; wherein, the training samples include audio signals corresponding to the video, video frame sequences, and real emotion category labels; The audio features and multi-scale visual features are extracted from the training samples by a multimodal emotion recognition model and then decoupled to obtain the emotion features and style features corresponding to the audio features and the multi-scale visual features, respectively. The sentiment classification loss is calculated using the sentiment features and the real sentiment category labels, and the style classification loss is calculated using the style features and the pre-generated pseudo style labels. The cross-modal distribution alignment loss and feature diversity regularization loss are also calculated based on the sentiment features. The emotion classification loss, style classification loss, cross-modal distribution alignment loss, and feature diversity regularization loss are weighted and combined into a total loss value. Then, the parameters of the multimodal emotion recognition model are optimized by backpropagation until the preset convergence condition is met, and the trained emotion recognition model is obtained. The cross-modal distribution alignment loss and feature diversity regularization loss are calculated based on the emotional features, including: semantically aligning the emotional features of the multi-scale visual features to the emotional features of the audio features, and semantically aligning the emotional features of the audio features to the emotional features of the multi-scale visual features; performing feature fusion on the emotional features of the semantically aligned multi-scale visual features and audio features to obtain fused emotional features at each scale; and calculating the feature diversity regularization loss between the emotional features at each scale.

2. The multimodal emotion recognition model training method according to claim 1, characterized in that, Audio features and multi-scale visual features are extracted from the training samples using a multimodal emotion recognition model and then decoupled to obtain the emotion features and style features corresponding to the audio features and the multi-scale visual features, respectively, including: The audio features are extracted from the audio by the audio encoder in the multimodal emotion recognition model, and the multi-scale visual features are extracted from the video by the video encoder according to different preset sampling frequencies; The audio features and the multi-scale visual features are decoupled by a preset decoupling function to obtain their respective emotional features and style features; wherein the decoupling function is composed of a feature extractor and two orthogonal decoupling projection matrices.

3. The multimodal emotion recognition model training method according to claim 1, characterized in that, The process of calculating sentiment classification loss using the sentiment features and the true sentiment category labels, and calculating style classification loss using the style features and pre-generated pseudo-style labels, includes: All training samples are clustered to obtain multiple clusters, and the cluster number of each training sample is used as the pseudo-style label of the style feature corresponding to the training sample; wherein, each cluster represents a style pattern. The emotion prediction result of the emotion feature is determined by the emotion classifier in the multimodal emotion recognition model, and the style prediction result of the style feature is determined by the style classifier. The sentiment classification loss is determined using the sentiment prediction result and the true sentiment category label, and the style classification loss is determined using the style prediction result and the pseudo style label.

4. The multimodal emotion recognition model training method according to claim 1, characterized in that, Based on the aforementioned sentiment features, cross-modal distribution alignment loss and feature diversity regularization loss are calculated, and the method further includes: The divergence between the probability distributions of the emotional features corresponding to the audio features and the multi-scale visual features is used as the cross-modal distribution alignment loss.

5. The multimodal emotion recognition model training method according to claim 1, characterized in that, The method further includes: Before determining the feature diversification regularization loss, feature enhancement is performed on the fused sentiment features at each scale through a hybrid attention mechanism; wherein, the hybrid attention mechanism includes a spatial attention mechanism and a scale attention mechanism.

6. A multimodal emotion recognition model training device, characterized in that, include: An acquisition module is used to acquire training samples; wherein, the training samples include audio signals corresponding to the video, video frame sequences, and real emotion category labels; The feature extraction module is used to extract audio features and multi-scale visual features from the training samples through a multimodal emotion recognition model; The feature decoupling module is used to decouple audio features and multi-scale visual features to obtain the emotion features and style features corresponding to the audio features and the multi-scale visual features respectively. The loss calculation module is used to calculate the sentiment classification loss using the sentiment features and the real sentiment category labels, and to calculate the style classification loss using the style features and the pre-generated pseudo style labels. It also calculates the cross-modal distribution alignment loss and the feature diversity regularization loss based on the sentiment features. The training module is used to weight and combine the emotion classification loss, the style classification loss, the cross-modal distribution alignment loss and the feature diversity regularization loss into a total loss value, and then backpropagate to optimize the parameters of the multimodal emotion recognition model until the preset convergence condition is met, thereby obtaining the trained emotion recognition model. The loss calculation module is specifically used to semantically align the sentiment features of the multi-scale visual features to the sentiment features of the audio features, and to semantically align the sentiment features of the audio features to the sentiment features of the multi-scale visual features; to perform feature fusion on the sentiment features of the semantically aligned multi-scale visual features and audio features respectively, to obtain fused sentiment features at each scale; and to calculate the feature diversification regularization loss between the sentiment features at each scale.

7. An electronic device, characterized in that, It includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the multimodal emotion recognition model training method according to any one of claims 1-5.

8. A storage medium having machine-executable instructions stored thereon, characterized in that, When the machine-executable instructions are executed by the processor, they implement the multimodal emotion recognition model training method as described in any one of claims 1-5.

9. A program product, characterized in that, The program product runs machine-executable instructions, which, when executed, implement the multimodal emotion recognition model training method as described in any one of claims 1-5.

Images