A fine-grained emotion recognition model construction method for decoupled learning

Abstract

This invention belongs to the fields of multimodal emotion recognition and representation learning, and more specifically relates to a method for constructing an emotion recognition model using fine-grained representation decoupling learning. First, a modality-shared encoder and a modality-private encoder are used to extract shared and private representations of a modality. Second, a fine-grained alignment method is used to constrain the learning of the modality-shared representations, thereby capturing modality consistency. A fine-grained differentiation method is used to learn the modality-private representations and enhance their diversity. Subsequently, a fine-grained prediction method is used to ensure that the labels of the encoder output representations remain unchanged. Finally, a richer representation is constructed using a cross-modal fusion method for the emotion recognition task.

Description

Technical Field

[0001] This invention belongs to the fields of multimodal emotion recognition and representation learning, and more specifically relates to a method for constructing an emotion recognition model based on fine-grained representation decoupling learning. Background Technology

[0002] Multimodal emotion recognition is an important subfield of affective computing, aiming to identify and understand human emotional states using information from speech and other perceptual modalities, such as textual expression and body language. In recent years, with the rise of deep learning technology and relying on large amounts of labeled data, multimodal emotion recognition technology has developed rapidly. However, in practical applications, the inherent heterogeneity between modalities leads to inconsistencies and imbalances in information, increasing the difficulty of multimodal representation learning and fusion. Therefore, how to mine complementary information between modalities, reduce redundant information, and improve the performance of emotion recognition systems has become a hot research topic in recent years.

[0003] Research has found that designing sophisticated fusion strategies and mining the correlations between different modalities can yield effective multimodal representations and more robust modal semantics, thereby improving the accuracy of emotion recognition tasks. Based on this fact, many studies have focused on representation decoupling and representation fusion strategies, such as domain adversarial learning techniques, attention mechanisms, and ensemble learning.

[0004] Nevertheless, current multimodal emotion recognition technologies are not yet able to effectively mitigate the heterogeneity between different modalities. How to learn good emotion representations and further improve model performance has become a key challenge in current multimodal emotion recognition research. Summary of the Invention

[0005] Existing methods simply decouple sentiment representations to enhance the complementarity of modal information and reduce information redundancy, but they primarily focus on global modality alignment and differences, neglecting fine-grained representation separation. This oversight limits the model's ability to capture subtle differences in each modality. To address the limitations of existing methods and promote fine-grained representation decomposition, this invention proposes a method for constructing sentiment recognition models based on fine-grained representation decoupling learning. This method, based on learning decoupled representations, employs fine-grained alignment and difference components to balance global and local distribution alignment and constrain private spaces, thereby capturing modality consistency and diversity. The method utilizes a fine-grained prediction component to ensure that the label information remains unchanged during the decoupling process.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A method for constructing an emotion recognition model based on fine-grained representation decoupling learning, comprising the following steps:

[0008] S101, Feature extraction: Input the text data and speech data into the feature encoder and extract the intermediate feature representations corresponding to each part;

[0009] S102, Extraction of shared and private representations: Input all intermediate feature representations corresponding to the text modality and the speech modality into the shared encoder and the private encoder. The shared encoder and the private encoder project the intermediate feature representations into the shared and private feature subspaces to obtain the shared representation and the private representation, thus obtaining the text private representation, the speech private representation, the speech shared representation, and the text shared representation.

[0010] S103. Fine-grained alignment: Utilizing the shared representations of text and speech modalities obtained in S102, the shared representations are processed to achieve global alignment of the shared feature distributions of text and speech, and the global alignment loss is calculated. The formula is as follows:

[0011]

[0012] in, Represents cross-entropy loss, Indicates shared features, Indicates modal label, Represents the global discriminator. , They represent the speech modality and the text modality, respectively. ,

[0013] Category-associated local discriminators are used to align the local distributions of speech and text modalities. One local discriminator corresponds to each category. The loss for local alignment is... The calculation is as follows:

[0014]

[0015] in, Indicates the number of emotion categories, This represents a category-dependent local discriminator. This represents the cross-entropy loss related to the category. Indicates shared features, Indicates modal label, Indicates belonging to a category of The probability distribution, , They represent the speech modality and the text modality, respectively. ,

[0016] Finally, use dynamic factors The relative importance of global and local distributions is dynamically estimated, calculated as follows:

[0017]

[0018] in:

[0019]

[0020] in, and Representing speech and text modalities, This represents the distance between the global distributions of two modes. This represents the distance between the local distributions of two modes. Indicates the number of emotion categories, Indicates the global alignment loss. Indicates belonging to a category Local alignment loss;

[0021] S104, Fine-grained differentiation: Using a global modality discriminator to distinguish the source of modes, global difference loss. The calculation is as follows:

[0022]

[0023] in, Private representations of modalities Represents the global modality classifier. This represents the trainable parameters of the global modality classifier.

[0024] A difference loss is used to encourage orthogonalization of shared and private features; local difference loss. The calculation is as follows:

[0025]

[0026] in, Squared Frobenius norm;

[0027] S105, Fine-grained prediction, requires the fine-grained prediction component to constrain the encoder's output representation so that it does not change its corresponding sentiment label, and calculates the label information invariance loss. The formula is as follows:

[0028]

[0029] in, express and The sum of, Represents a sentiment classifier. This represents the trainable parameters of the sentiment classifier. Indicates emotional tags;

[0030] S106. Cross-modal fusion: The shared and private representations obtained in S102 are stacked into a matrix, and then a multi-head self-attention mechanism is used to obtain the final cross-modal representation. Calculate classification loss The formula is as follows:

[0031]

[0032] S107. A collaborative optimization model is used to jointly optimize multimodal emotion recognition. This involves shared representation alignment, private representation differentiation, and unchanged label information. Four loss functions are applied until the loss converges, yielding a high-performing multimodal emotion recognition model loss. as follows:

[0033]

[0034] in , and For hyperparameters, balance the four losses.

[0035] In a further optimization of this technical solution, the feature encoder is a Wav2vec2.0 or Bert encoder.

[0036] In a further optimization of this technical solution, step 103 uses a data distribution fitting method or a central moment loss function to globally align the shared feature distributions of text and speech.

[0037] In a further optimization of this technical solution, the global discriminator consists of two fully connected layers and a ReLU activation function.

[0038] In a further optimization of this technical solution, the local discriminator consists of two fully connected layers and a ReLU activation function.

[0039] In a further optimization of this technical solution, the global modality classifier consists of two fully connected layers.

[0040] Unlike existing technologies, the beneficial effect of the above technical solution is that the multimodal emotion recognition method with fine-grained representation decoupling learning proposed in this invention can capture the consistency and diversity of modalities at the global and category levels, thereby enhancing the ability of the multimodal emotion recognition model to alleviate modal heterogeneity and improving the performance of emotion recognition. Attached Figure Description

[0041] Figure 1 A flowchart illustrating the method for constructing an emotion recognition model that decouples learning from fine-grained representations. Detailed Implementation

[0042] To explain in detail the technical content, structural features, objectives, and effects of the technical solution, the following description is provided in conjunction with specific embodiments and accompanying drawings.

[0043] Please see Figure 1 This is a flowchart illustrating a method for constructing an emotion recognition model using fine-grained representation decoupling learning. A preferred embodiment of this invention provides a method for constructing an emotion recognition model using fine-grained representation decoupling learning, which specifically includes the following steps:

[0044] S101. Feature Extraction. Input the text data and speech data into the feature encoder, such as Wav2vec2.0 and Bert encoder, to extract the intermediate feature representations corresponding to each part.

[0045] S102. Extraction of Shared and Private Representations. All intermediate feature representations corresponding to the text and speech modalities are input into the shared encoder and private encoder. The shared encoder and private encoder project the intermediate feature representations into the shared and private representation subspaces to obtain the shared representation and private representation, i.e., the text private representation, the speech private representation, the speech shared representation, and the text shared representation.

[0046] S103, Fine-grained Alignment. To capture consistency across different modalities and reduce intra-class variance, the shared representations of text and speech modalities obtained in S102 are utilized. Furthermore, a data distribution fitting method (domain adversarial network) is employed to globally align the distributions of the shared text and speech representations, and the global alignment loss is calculated. The formula is as follows.

[0047]

[0048] in, Represents cross-entropy loss, Indicates shared representation, Indicates modal label, This represents the global discriminator, which consists of two fully connected layers and a ReLU activation function. , They represent the speech modality and the text modality, respectively. .

[0049] A category-associated local discriminator, used to align the local distributions of speech and text modalities, consists of two fully connected layers and a ReLU activation function. One local discriminator corresponds to each category. The loss function for local alignment is described. The calculation is as follows.

[0050]

[0051] in, This indicates the number of emotion categories. This represents a local discriminator that is related to the category. This represents the cross-entropy loss associated with the category. This indicates a shared representation. This indicates a modal label. Indicates belonging to a category of The probability distribution. , They represent the speech modality and the text modality, respectively. .

[0052] Finally, use dynamic factors The relative importance of global and local distributions is estimated dynamically, and the calculation formula is as follows.

[0053]

[0054] in:

[0055]

[0056] in, and It represents speech and text modalities. This represents the distance between the global distributions of two modes. This represents the distance between the local distributions of two modes. This indicates the number of emotion categories. This represents the global alignment loss. Indicates belonging to a category Local alignment loss.

[0057] S104, Fine-grained Differentiation. To ensure that private representations can model different aspects of multimodal data and reduce information redundancy between different modalities, a global modality classifier is used to distinguish the source of the modality. This classifier consists of two fully connected layers. Global Differentiation Loss. The calculation is as follows.

[0058]

[0059] in, Represents the private representation of a modality. Represents the global modality classifier. This represents the trainable parameters of the global modality classifier.

[0060] A difference loss is employed to encourage the orthogonalization of shared and private representations. Local difference loss. The calculation is as follows.

[0061]

[0062] in, Squared Frobenius norm.

[0063] S105. Fine-grained prediction. To ensure that the sentiment attributes of the decoupled modal representations remain unchanged, the fine-grained prediction component is required to restrict the encoder's output representation from altering its corresponding sentiment label, calculating the label information invariance loss. The formula is as follows.

[0064]

[0065] in, express and The sum of. This represents a sentiment classifier, which consists of two fully connected layers. This represents the trainable parameters of the sentiment classifier. Indicates emotional tags.

[0066] S106, Cross-modal fusion. To better understand multimodal information and construct richer representations, the shared and private representations obtained in S102 are stacked into a matrix. Then, a multi-head self-attention mechanism is used to obtain the final cross-modal representation. Calculate the classification loss. The formula is as follows.

[0067]

[0068] S107. Co-optimization Model. Co-optimization of multimodal sentiment recognition involves aligning shared representations, differentiating private representations, and preserving label information. Four loss functions are used until the loss converges, resulting in a better-performing multimodal sentiment recognition model. Loss as follows:

[0069]

[0070] in , and For hyperparameters, balance the four losses.

[0071] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Unless otherwise specified, an element defined by the phrase "comprising..." or "including..." does not exclude the presence of additional elements in the process, method, article, or terminal device that includes said element. Additionally, in this document, "greater than," "less than," "exceeding," etc., are understood to exclude the stated number; "above," "below," "within," etc., are understood to include the stated number.

[0072] Although the above embodiments have been described, those skilled in the art, once they understand the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the above descriptions are merely embodiments of the present invention and do not limit the scope of patent protection of the present invention. Any equivalent structural or procedural transformations made using the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for constructing an emotion recognition model based on fine-grained representation decoupling learning, characterized in that, The method includes the following steps: S101, Feature extraction: Input the text data and speech data into the feature encoder and extract the intermediate feature representations corresponding to each part; S102, Extraction of shared and private representations: Input all intermediate feature representations corresponding to the text modality and the speech modality into the shared encoder and the private encoder. The shared encoder and the private encoder project the intermediate feature representations into the shared and private feature subspaces to obtain the shared representation and the private representation, thus obtaining the text private representation, the speech private representation, the speech shared representation, and the text shared representation. S103. Fine-grained alignment: Utilizing the shared representations of text and speech modalities obtained in S102, the shared representations are processed to achieve global alignment of the shared feature distributions of text and speech, and the global alignment loss is calculated. The formula is as follows: ， in, Represents cross-entropy loss, Indicates shared features, Indicates modal label, Represents the global discriminator. They represent the speech modality and the text modality, respectively. , Category-associated local discriminators are used to align the local distributions of speech and text modalities. One local discriminator corresponds to each category. The loss for local alignment is... The calculation is as follows: ， in, Indicates the number of emotion categories, This represents a category-dependent local discriminator. This represents the cross-entropy loss related to the category. Indicates shared features, Indicates modal label, Indicates belonging to a category of The probability distribution, They represent the speech modality and the text modality, respectively. , Finally, use dynamic factors The relative importance of global and local distributions is dynamically estimated, calculated as follows: ， in: , ， in, and Representing speech and text modalities, This represents the distance between the global distributions of two modes. This represents the distance between the local distributions of two modes. Indicates the number of emotion categories, Indicates the global alignment loss. Indicates belonging to a category Local alignment loss; S104, Fine-grained differentiation: Using a global modality discriminator to distinguish the source of modes, global difference loss. The calculation is as follows: ， in, Private representations of modalities Represents the global modality classifier. This represents the trainable parameters of the global modality classifier. A difference loss is used to encourage orthogonalization of shared and private features; local difference loss. The calculation is as follows: ， in, Squared Frobenius norm; S105, Fine-grained prediction, requires the fine-grained prediction component to constrain the encoder's output representation so that it does not change its corresponding sentiment label, and calculates the label information invariance loss. The formula is as follows: ， in, express and The sum of, Represents a sentiment classifier. This represents the trainable parameters of the sentiment classifier. Indicates emotional tags; S106. Cross-modal fusion: The shared and private representations obtained in S102 are stacked into a matrix, and then a multi-head self-attention mechanism is used to obtain the final cross-modal representation. Calculate classification loss The formula is as follows: ， S107. A collaborative optimization model is used to jointly optimize multimodal emotion recognition. This involves shared representation alignment, private representation differentiation, and unchanged label information. Four loss functions are applied until the loss converges, yielding a high-performing multimodal emotion recognition model loss. as follows: in , and For hyperparameters, balance the four losses.

2. The method for constructing an emotion recognition model based on fine-grained representation decoupling learning as described in claim 1, characterized in that, The feature encoder is, for example, a Wav2vec2.0 or a Bert encoder.

3. The method for constructing an emotion recognition model based on fine-grained representation decoupling learning as described in claim 1, characterized in that, Step 103 uses a data distribution fitting method or a central moment loss function to globally align the shared feature distributions of text and speech.

4. The method for constructing an emotion recognition model based on fine-grained representation decoupling learning as described in claim 1, characterized in that, The global discriminator consists of two fully connected layers and a ReLU activation function.

5. The method for constructing an emotion recognition model based on fine-grained representation decoupling learning as described in claim 1, characterized in that, The local discriminator consists of two fully connected layers and a ReLU activation function.

6. The method for constructing an emotion recognition model based on fine-grained representation decoupling learning as described in claim 1, characterized in that, The global modality classifier consists of two fully connected layers.

Images