A method and apparatus for cross-database bimodal speech emotion recognition targeting unlabeled target domains

Abstract

This invention discloses a method and apparatus for cross-database bimodal speech emotion recognition targeting unlabeled target domains. The method includes: acquiring speech signals and their corresponding text information; extracting audio features and text features using an audio encoder and a text encoder, respectively; using a cross-attention mechanism to perform information interaction and alignment between the two modalities to obtain a fused emotion representation; and inputting the fused representation into an emotion classifier to output the emotion category. This invention can effectively alleviate domain bias when the target domain lacks annotations, improve the recognition accuracy and generalization ability across corpora and scenarios, and has good engineering application value.

Description

Technical Field

[0001] This invention belongs to the field of speech emotion recognition and cross-domain transfer learning technology, specifically relating to a cross-library bimodal speech emotion recognition method and apparatus for unlabeled target domains. Background Technology

[0002] Speech emotion recognition technology aims to automatically identify the speaker's emotional state from speech signals and is an important research direction in the fields of affective computing and human-computer interaction. With the development of intelligent customer service, intelligent companion systems, and affective computing applications, higher demands are being placed on the generalization ability of speech emotion recognition models in different application scenarios.

[0003] Existing speech emotion recognition methods are mainly based on single-modal speech modeling, extracting acoustic features and constructing classification models to predict emotions. However, single-modal methods are easily affected by recording devices, environmental noise, speaking styles, and differences in corpus distribution in cross-database applications, leading to a significant decrease in model performance in the target domain.

[0004] To improve recognition performance, some studies have incorporated textual information, achieving multimodal emotion recognition by fusing speech and text modalities. Existing multimodal methods typically employ feature concatenation or attention weighting for fusion, achieving some success within the same database. However, in cross-database scenarios, significant differences exist in text distribution, semantic expression, and corpus composition across different databases, making it difficult for simple fusion methods to guarantee consistent feature distribution across different domains.

[0005] On the other hand, domain adaptation methods, by constructing a domain discriminator and introducing adversarial training mechanisms, enable the model to learn domain-insensitive feature representations, achieving some progress in cross-domain tasks. However, most existing domain adversarial methods focus on aligning single-modal features, failing to fully consider the impact of fused representations resulting from multimodal feature interactions on domain differences.

[0006] Furthermore, in practical cross-database multimodal speech emotion recognition scenarios, when the target domain data contains fixed text, repetitive text, or highly concentrated text distribution, the fused features tend to cluster according to text semantics rather than according to emotion categories. This semantic-driven clustering phenomenon weakens the separability between emotion categories, thereby reducing the model's recognition accuracy in the target domain.

[0007] Based on this, it is necessary to propose a new cross-database speech emotion recognition method to achieve effective interaction and fusion of multimodal features, learn domain-invariant feature representations through domain adversarial mechanisms, and adjust the text modality contribution in specific scenarios, thereby improving the performance of cross-database emotion recognition. Summary of the Invention

[0008] Therefore, the technical problem to be solved by this invention is to overcome the problem that existing multimodal sentiment recognition methods have difficulty learning domain-invariant features and thus have reduced generalization ability in cross-corpus application scenarios due to large differences in the distribution between different data domains, especially when the source domain has sentiment labels but the target domain lacks labels; at the same time, it solves the problem that when there are fixed texts or highly repetitive text distributions in the target domain, the fused features tend to cluster according to text semantics rather than sentiment categories, thereby reducing the sentiment discrimination ability.

[0009] To address the aforementioned technical problems, this invention proposes a cross-database bimodal speech emotion recognition method for unlabeled target domains, comprising: Step 1: Obtain sentiment-labeled samples from the source domain and unlabeled samples from the target domain, where each sample includes speech data and corresponding text data; Step 2: Input the speech data and corresponding text data into a pre-constructed multimodal emotion recognition model. Input the speech data into an audio feature encoder to obtain audio feature representation; input the text data into a text feature encoder to obtain text feature representation. Step 3: Input the audio sequence features and text sequence features into the cross-modal bidirectional cross-attention module to obtain the audio-guided cross-modal enhanced feature sequence and the text-guided cross-modal enhanced feature sequence; Step 4: Concatenate the two cross-modal enhanced feature sequences in the time dimension and input them into the fusion self-attention module to model the global dependency relationship to obtain the fused refined feature sequence; Step 5: Perform masked pooling on the fused and refined feature sequence to obtain the fused summary vector; Step 6: Simultaneously perform masked pooling on the audio-side cross-modal enhancement feature sequence and the text-side cross-modal enhancement feature sequence to obtain the audio summary vector and the text summary vector; Step 7: Generate a dimension-wise gating weight through the gating fusion module, and perform a weighted summation on the audio summary vector and the text summary vector to obtain the gating fusion vector; Step 8: Perform residual fusion and normalization on the gated fusion vector and the fusion summary vector to obtain the final multimodal sentiment representation vector; Step 9: Input the multimodal sentiment representation vector into the sentiment classifier, output the sentiment prediction result, and calculate the sentiment classification loss based on the source domain samples; Step 10: Input the multimodal sentiment representation vector into the domain discriminator through the gradient inversion layer, output the domain prediction result, and calculate the domain discrimination loss based on the source domain samples and the target domain samples; Step 11: Jointly train the emotion classification loss and the domain discrimination loss to make the multimodal emotion representation vector insensitive to the difference in distribution between the source domain and the target domain; Step 12: In the inference phase, input the target domain samples into the trained model and output the sentiment category of the target domain samples.

[0010] In the above technical solution, the audio feature encoder outputs an audio feature sequence. The text feature encoder outputs a text feature sequence. , The cross-modal cross-attention includes two branches: text query audio and audio query text. in, , , , , , This is a learnable linear mapping matrix used to map input features into query vectors, key vectors, and value vectors, respectively. The feature dimension of a single attention head is used to scale the dot product result.

[0011] In the above technical solution, the sentiment classifier outputs the sentiment category probability distribution through a fully connected mapping and a Softmax function; The emotion classification loss uses cross-entropy loss, which is calculated only for labeled samples in the source domain.

[0012] In the above technical solution, the gradient reversal layer satisfies: Forward propagation: Backpropagation: in, To counteract strength coefficient.

[0013] In the above technical solution, the joint training is implemented based on a joint training objective function, which is a weighted sum of sentiment classification loss and domain discrimination loss; Samples in the target domain without sentiment labels only participate in the calculation of domain discrimination loss, and do not participate in the calculation of sentiment classification loss.

[0014] In the above technical solution, the audio feature encoder adopts a pre-trained acoustic representation model structure based on self-supervised learning; The text feature encoder adopts a pre-trained language model structure based on a self-attention mechanism; The encoders all employ a local unfreezing strategy, updating only high-level parameters.

[0015] In the above technical solution, during the training or inference phase, the target domain text data is subjected to a repetition determination. When the target domain text is determined to meet the preset repetition condition, a text contribution suppression strategy is enabled to reduce the weight of the text modality in the fusion feature. When the repetition condition is not met, the text contribution suppression strategy is disabled and the fusion feature is generated according to the default fusion strategy.

[0016] This invention also proposes a cross-database bimodal speech emotion recognition device for unlabeled target domains, comprising: The data acquisition module is used to acquire samples with sentiment labels from the source domain and samples without sentiment labels from the target domain, where each sample includes speech data and corresponding text data; An audio feature encoding module is used to extract features from the speech data and output an audio feature sequence. The text feature encoding module is used to extract features from the text data and output a text feature sequence; A bidirectional cross-modal cross-attention module is used to construct text query audio branches and audio query text branches respectively, realize intermodal information interaction, and generate text-guided cross-modal enhanced feature sequences and audio-guided cross-modal enhanced feature sequences. A self-attention module is integrated to model the global dependency relationship after concatenating the two cross-modal enhanced feature sequences; The mask pooling module is used to perform mask pooling on the fused feature sequence and the two-modal enhanced feature sequence to generate fused summary vector, text summary vector and audio summary vector; The gated fusion module is used to generate dimension-wise gated weights, perform weighted summation of the text summary vector and the audio summary vector, and perform residual fusion with the fused summary vector to obtain the final multimodal sentiment representation vector. The text duplication determination module is used to perform duplication detection or distribution difference analysis on target domain text, and outputs a trigger signal when preset conditions are met. The text contribution adjustment module is used to scale or constrain the text modality weights generated in the gated fusion module when the trigger signal is received, so as to reduce the contribution ratio of the text modality to the fusion representation. The sentiment classification module is used to input the multimodal sentiment representation vector into the sentiment classifier, output the sentiment prediction result, and calculate the sentiment classification loss based on the source domain samples. The domain adversarial module includes a gradient inversion unit and a domain discrimination unit, which are used to input the multimodal sentiment representation vector into the domain discrimination unit via the gradient inversion unit, output the domain prediction result, and calculate the domain discrimination loss based on the source domain samples and the target domain samples. The joint optimization module is used to jointly train each module based on the sentiment classification loss and the domain discrimination loss, so that the multimodal sentiment representation vector is insensitive to the difference in distribution between the source domain and the target domain. The sentiment output module is used to output the sentiment category of the target domain samples during the inference phase.

[0017] The advantages of this invention are: This invention can effectively alleviate domain bias when the target domain lacks annotations, improve the recognition accuracy and generalization ability across corpora and scenarios, and has good engineering application value. Attached Figure Description

[0018] The present invention will be further described below with reference to the accompanying drawings and embodiments: Figure 1 This is a schematic diagram of the overall process of the method of the present invention.

[0019] Figure 2 This is a block diagram of the overall structure of the multimodal emotion recognition model of the present invention.

[0020] Figure 3 This is a schematic diagram of the bidirectional cross-modal cross-attention structure of the present invention.

[0021] Figure 4 This is a schematic diagram of the structure of the gated fusion and residual fusion module of the present invention.

[0022] Figure 5 This is a schematic diagram of the domain adversarial training structure of the present invention.

[0023] Figure 6 is a schematic diagram of the t-SNE visualization results of the feature distribution across the IEMOCAP MSP-IMPROV task, where: Figure 6(a) is a schematic diagram of the feature distribution without domain adversarial training; Figure 6(b) is a schematic diagram of the feature distribution using domain adversarial training.

[0024] Figure 7 is a schematic diagram of the t-SNE visualization results of the feature distribution across the IEMOCAP CREMA-D task, where: Figure 7(a) shows the feature distribution without domain adversarial training; Figure 7(b) shows the feature distribution of domain adversarial training. Detailed Implementation

[0025] Example: Reference Figure 1As shown, this invention provides a cross-library bimodal speech emotion recognition method for unlabeled target domains, named BiCross-EMO-DA (Bidirectional Cross-Attention Emotion Network with Domain Adaptation). Figure 2 This is a block diagram of the overall structure of the multimodal emotion recognition model of the present invention. Specifically, it includes the following steps: Step 1: Perform unified preprocessing on different corpora and construct the samples into triplet form: in, For the audio path of a certain sample, Here is the text path of the sample. This is the sentiment label for this sample.

[0026] The corpus is preprocessed. For the IEMOCAP dataset, this invention sequentially traverses the Session1 to Session5 folders to read the audio path, text path, and tag path. This invention adopts a four-class sentiment classification system and establishes the following mapping relationship: At the same time, the "exc" category in the original tag is mapped to the "hap" category, that is: After mapping is completed, sample triples are constructed.

[0027] For the CREMA-D dataset, this invention reads audio files from a folder and extracts sentiment tags. Since this dataset lacks text modalities, the corresponding text information is obtained through speech transcription. The following mapping relationship is established for this dataset: Finally, construct the sample triples: For the MSP-IMPROV dataset, the original sentiment labels contain multiple sentiment categories. This invention retains only four sentiment categories and determines the final sentiment category based on multi-annotator perceived labels, establishing the following mapping relationship: Finally, construct the sample triples: After completing the preprocessing of data from each corpus, this invention constructs two data partitioning methods according to different application scenarios: one is a single corpus training mode, which divides the training set and test set within the same corpus to evaluate the emotion recognition performance of the model under the same distribution condition; the other is a cross-corpus training mode, which selects one corpus as the source domain for labeled training and selects another corpus as the target domain for testing or domain alignment training to evaluate the generalization ability of the model in cross-corpus scenarios.

[0028] For training and testing on a single corpus, separate training and testing sets are constructed for each corpus: In a single corpus scenario, the model is trained on the training set of the corpus and its performance is evaluated on the test set of the same corpus to verify the model's sentiment recognition ability under the same distribution condition.

[0029] For cross-corpus training and testing, in a cross-corpus scenario, one corpus is selected as the source dataset: Another corpus was selected as the target domain dataset: Supervised training using only the source domain dataset: Performance was evaluated on the target domain test set. Used to verify the model's generalization ability under conditions of distributional differences.

[0030] In the cross-database augmentation training phase, a domain adversarial mechanism is introduced to construct a joint dataset: The source domain samples are labeled with sentiment, while the target domain samples do not participate in sentiment loss calculation but only in domain discrimination training. Furthermore, the source and target domains come from different databases.

[0031] Through joint optimization: Achieve cross-domain feature distribution alignment.

[0032] Step 2: Perform feature encoding processing on the input text information and speech signal respectively to obtain text sequence features and audio frame-level feature sequences, providing input for subsequent cross-modal interaction and fusion computing.

[0033] The input text is segmented into tokens, and the text sequence is represented as a token sequence: in, Indicates the length of the text sequence.

[0034] Simultaneously construct a text attention mask: Among them, when the first When each token is a valid character ,otherwise .

[0035] Input the text token sequence and attention mask into the text feature encoder to obtain the text hidden state sequence: in, For batch size, Indicates the text dimension.

[0036] In one embodiment, the text feature encoder employs a pre-trained language model based on a Transformer encoder structure with a multi-head self-attention mechanism. To enhance cross-corpus generalization ability and control the scale of training parameters, a local unfreezing strategy is adopted for the text feature encoder, updating only the high-level encoding parameters while keeping the remaining parameters frozen.

[0037] In one embodiment, text feature dimensions It can be set to an integer between 256 and 1024; in one specific embodiment, the text feature dimension is set to 768.

[0038] The input speech signal is encoded. Let the speech waveform be represented as: in, Indicates the length of the speech sample.

[0039] The speech waveform is input into an audio feature encoder to obtain an audio frame-level hidden state sequence: in, The number of frames after encoding. This indicates the dimension of audio features.

[0040] In one embodiment, the audio feature encoder employs a self-supervised pre-trained acoustic representation model structure. To enhance cross-corpus generalization capability, a local unfreezing strategy is used for the audio feature encoder, updating only the high-level encoding parameters while keeping the remaining parameters frozen.

[0041] To avoid padding frames affecting subsequent attention calculations, an audio mask is constructed: Among them, when the first When the frame is a valid frame ,otherwise .

[0042] In one embodiment, audio feature dimensions It can be set to an integer between 256 and 1024; in one specific embodiment, the audio feature dimension is set to 768.

[0043] In one embodiment, the text feature dimension With audio feature dimensions Set to the same value so that subsequent cross-modal attention mechanisms can perform dimension alignment calculations.

[0044] After the above processing, the text sequence features are obtained. and audio sequence features And corresponding to text masks respectively. With audio mask This serves as the input for the cross-modal attention calculation in step 3.

[0045] Step 3: Input the text sequence features and audio sequence features obtained in Step 2 into the cross-modal bidirectional cross-attention module to achieve bidirectional information interaction between text and audio, and obtain the cross-modal enhanced feature sequence. The specific structure is as follows: Figure 3 As shown.

[0046] In the implementation, to be consistent with the key padding mask convention of the attention module, the mask is converted to: in, This position is considered padding and should be ignored during attention calculation. This is a logical negation.

[0047] A cross-attention computation is constructed using text sequence features as the query and audio sequence features as the key and value. For the... An attention head is defined, and a linear mapping is defined: in, , , These are learnable parameters.

[0048] Calculate the scaled dot product attention: in, For single-head dimension, For the number of attention heads, To apply negative infinity to the masked position so that its weight approaches 0.

[0049] The outputs of each head are concatenated and linearly mapped to obtain the cross-modal enhanced feature sequence of the text after audio enhancement: in, The output mapping matrix of multi-head attention.

[0050] In one embodiment, to map cross-modal outputs to a fusion space and enhance stability, for Further linear mapping, layer normalization, and random deactivation processing are performed: in, To integrate feature dimensions.

[0051] Using audio sequence features as queries and text sequence features as keys and values, calculate audio-guided cross-modal augmentation features. For the first... One point of attention is: Attention is calculated as follows: By concatenating the outputs of each head and linearly mapping them, we obtain the cross-modal enhanced feature sequence of the audio after text enhancement: Similarly, linear mapping, layer normalization, and random deactivation are applied to it: In one embodiment, the number of heads in a multi-head attention mechanism It can be set to 4 to 16; in one specific embodiment, Fusion feature dimensions It can be set to 256 to 1024; in one specific embodiment, .

[0052] Step 4: Concatenate the text-guided cross-modal enhanced feature sequence obtained in Step 3 with the audio-guided cross-modal enhanced feature sequence to construct a fused feature sequence, and model the global dependency relationship through a fused self-attention module; Step 5: Subsequently, masked pooling is performed on the fused sequence and each modal sequence to obtain the summary vector. Then, the dimension-wise gated weights are generated through the gated fusion module to obtain the final multimodal fusion representation. The specific structure is as follows: Figure 4 As shown.

[0053] Text enhancement sequence With audio enhancement sequence Concatenating the sequences along the time dimension yields the fused feature sequence: Simultaneously, the text mask and audio mask are concatenated to obtain the merged mask: To ensure semantic consistency with the key padding mask semantics of the fusion self-attention module, the following is constructed: Fuse feature sequences Input the fusion self-attention module, and perform global modeling of the fusion sequence through multi-head self-attention to obtain the fusion refinement sequence: in, For multi-head self-attention computation, Used to mask the fill position.

[0054] To obtain sequence-level representations, masked average pooling is performed on the fused and refined sequences to obtain a global summary vector for the fused sequences: in, express The a time step vector For fusion mask The Bit, To prevent division by zero of small constants.

[0055] Step 6: Similarly, perform masked pooling on the text enhancement sequence and the audio enhancement sequence respectively to obtain the two-modal summary vector: Step 7: Input the text summarization vector and audio summarization vector into the gating fusion module to generate dimension-wise gating logits: in, For learnable parameters, This is a bias term.

[0056] The two gated logits are concatenated in the modal dimension, and then Softmax is performed in the modal dimension to obtain the dimension-wise gated weights: The two modality summary vectors are weighted and summed dimension-wise according to the gating weights to obtain the gated fusion vector: in, For text-gated weights, For audio gating weights, This indicates element-wise multiplication.

[0057] Step 8: To fuse global sequence information and gated fusion information, the gated fusion vector and the fused sequence summary vector are subjected to residual fusion and normalization processing to obtain the final multimodal representation vector: in, It is a linear mapping matrix. For layer normalization, This is the activation function.

[0058] To mitigate the problem of text semantics dominating fusion features when there is fixed text or highly repetitive text distribution in the target domain, this invention introduces a text repetition determination module and a text contribution adjustment module after gating fusion. The text repetition determination module is used to perform repetition detection or distribution difference analysis on the text in the target domain, and outputs a trigger signal when the detection result meets preset conditions. When the text contribution adjustment module receives the trigger signal, it scales or constrains the gating weights generated by the text summary vector in the gating fusion, thereby reducing the dominant effect of text modality on the fusion representation.

[0059] Perform statistical analysis on the text sequences of the target domain samples to construct a text set. In one embodiment, after normalizing the text, a redundancy index for the target domain text is calculated: in, The number of unique entries after deduplication of the target field text. This represents the total number of text entries in the target domain. The smaller the value, the higher the text repetition rate.

[0060] Compare repeatability metrics with preset thresholds Comparison, when satisfied Then output trigger signal ,otherwise .

[0061] In one embodiment, the threshold Triggering conditions can be set according to the actual situation of the target domain to represent the "fixed text / highly repetitive text" scenario.

[0062] When the trigger signal At that time, scale the text-gated logits: in, The scaling factor contributes to the text, satisfying: .

[0063] When the trigger signal hour, .

[0064] The scaled logits input is gated and normalized to obtain the dimension-wise gated weights: And the adjusted gated fusion vector is obtained: Furthermore, the adjusted gated fusion vector and the fusion sequence summary vector are residually fused to obtain the final multimodal representation: Step 9: Input the multimodal sentiment representation vector output in Step 8 into the sentiment classification module, and obtain the prediction result of sentiment category through fully connected mapping.

[0065] In one embodiment, the sentiment classification module includes at least one fully connected mapping layer for mapping multimodal representations to a sentiment category space. Let the number of sentiment categories be... Then the classifier outputs logits as: in, These are learnable parameters.

[0066] In one embodiment, the emotion classification module may include a multilayer perceptron architecture. Let the... Layer hiding is represented as Then we have: in, To hide the number of layers, As a non-linear activation function, in one specific embodiment, The LeakyReLU function can be used.

[0067] In one embodiment, to improve the model's generalization performance and suppress overfitting, a random dropout process can be introduced before and after the fully connected mapping, which randomly masks the input features to enhance robustness.

[0068] The logits are converted into a sentiment probability distribution using the Softmax function: in, For the first The sample belongs to the first The probability of predicting sentiment.

[0069] Use the category with the highest probability as the target sentiment output: In the source domain supervision phase of single-corpus training or cross-corpus training, based on labeled samples from the source domain... The sentiment classification module is optimized using cross-entropy loss: in, This represents the number of samples in the source domain batch.

[0070] Output sentiment prediction results and the probability distribution of emotions In cross-corpus scenarios, the multimodal representation vectors It will also serve as input for the domain adversarial training in step 7 to further enhance cross-domain generalization capabilities.

[0071] Step 10: To align the feature distributions of the source and target domains, a domain adversarial training mechanism is introduced during the cross-corpus training phase. Labeled samples from the source domain and unlabeled samples from the target domain are input into the multimodal sentiment recognition model described in steps 2-9 to obtain the corresponding multimodal sentiment representation vectors. These vectors are then connected to the domain discrimination module via a gradient inversion layer to learn domain-invariant feature representations. The specific structure is as follows: Figure 5 As shown.

[0072] Let the source domain sample set be The target domain sample set is The source domain samples contain sentiment labels. The target domain samples do not contain sentiment labels.

[0073] By inputting both source and target domain samples into the multimodal emotion recognition model, the corresponding multimodal representation vector is obtained: in, This refers to the feature extraction and fusion network described in steps 2 to 9.

[0074] The multimodal representation vector is then input into the Gradient Reversal Layer (GRL): During forward propagation, GRL satisfies the following: Multiply the gradient by a negative coefficient during backpropagation: in, To counteract strength coefficient.

[0075] The representation vector processed by GRL is input into the neighborhood discriminator to obtain the neighborhood prediction logits: in, , These are the parameters for the domain discriminator, with 2 domain categories (source domain / target domain).

[0076] The neighborhood prediction probability is obtained through Softmax: in, This represents the probability that a sample belongs to the source domain or the target domain.

[0077] Let the domain label be: The domain discrimination loss is expressed as cross-entropy: in, This represents the total number of samples in the source and target domains.

[0078] Step 11: During the cross-corpus training phase, calculate the sentiment classification loss only for source domain samples. Simultaneously, the neighborhood discrimination loss is calculated jointly for samples from both the source and target domains. .

[0079] The joint training objective function is: Under the influence of GRL, the feature extraction network parameters The parameters of the fusion network were optimized as follows: in, For feature extraction and fusion network parameters, For the parameters of the sentiment classifier, These are the parameters for the neighborhood discriminator.

[0080] By implementing adversarial learning between the feature extraction network and the domain discriminator through a gradient inversion layer, the multimodal sentiment representation can maintain the separability of sentiment categories while being insensitive to the distribution differences between the source and target domains, thereby improving the model's cross-library generalization ability in unlabeled target domains.

[0081] Step 12: In order to evaluate the multimodal emotion recognition method proposed in this invention, this embodiment selected three public speech emotion recognition datasets: IEMOCAP, MSP-IMPROV, and CREMA-D, and conducted cross-corpus experiments of IEMOCAP across MSP-IMPROV and IEMOCAP across CREMA-D.

[0082] The IEMOCAP dataset contains audiovisual data recorded by 10 actors, totaling approximately 12 hours in length, along with corresponding sentiment annotations. The dataset contains 10,039 samples, with an average speech duration of approximately 4.5 seconds, and the longest speech segment exceeding 30 seconds.

[0083] The MSP-IMPROV dataset contains audio and text data recorded by 12 actors. The data collection process combined improvisation and scripted performance, and provides corresponding sentiment annotation information. The dataset contains a total of 7798 audio samples, with an average audio duration of approximately 3.2 seconds. The sentiment categories mainly include happiness, sadness, anger, and neutrality.

[0084] The CREMA-D dataset contains audio recordings by 91 actors, along with corresponding emotion annotations. The dataset contains 7442 audio samples, each corresponding to a short audio segment, with an average duration of approximately 2.5 seconds. It includes various emotion categories such as anger, happiness, sadness, and neutrality. The dataset consists of 12 fixed text sentences, with different actors expressing the same text with different emotions.

[0085] In this embodiment, four emotion categories—"anger," "happiness," "sadness," and "neutrality"—were uniformly selected for the experiment. Specifically, in the IEMOCAP dataset, samples originally labeled "Happy" and "Excited" were merged into a single "Happy" category.

[0086] This embodiment constructs a training and experimental environment for a multimodal emotion recognition model based on the PyTorch platform, uniformly setting the data batch size to 32, employing the Adam optimizer for model training, and using five-fold cross-validation for experimental evaluation. The average weighted accuracy (WA) and unweighted accuracy (UA) of the five-fold cross-validation results are ultimately used as the evaluation metrics for model performance.

[0087] Tables 1 and 2 present the experimental results for the IEMOCAP→CREMA-D and IEMOCAP→MSP-IMPROV cross-corpus sentiment recognition tasks, respectively. BiCross-EMO (Bidirectional Cross-Attention Emotion Network) represents the basic version of the proposed model, i.e., the structure without the inclusion of a domain adversarial training module.

[0088] Table 1. Comparison of sentiment recognition results across IEMOCAP and CREMA-D corpus Table 2. Comparison of sentiment recognition results across IEMOCAP and MSP-IMPROV corpus. Tables 1 and 2 present the experimental results of the proposed model in two cross-corpus sentiment recognition tasks: IEMOCAP→CREMA-D and IEMOCAP→MSP-IMPROV. Table 1 shows the performance comparison of different methods in the IEMOCAP cross-CREMA-D task. The results show that the proposed BiCross-EMO model, without the introduction of a domain adversarial training module, achieved a weighted accuracy (WA) of 51.13% and an unweighted accuracy (UA) of 50.30%. Further introducing the domain adversarial training module further improved the performance of the BiCross-EMO-DA model, with WA and UA reaching 55.87% and 55.83%, respectively, representing improvements of 4.74% and 5.53% compared to the model without the domain adversarial training module. This indicates that domain adversarial training can mitigate the impact of differences in cross-corpus data distribution to a certain extent, thereby improving the model's cross-domain sentiment recognition capability.

[0089] Table 2 presents the experimental results of IEMOCAP across the MSP-IMPROV task. It can be seen that without the domain adversarial training module, the WA and UA of the BiCross-EMO model are 40.9% and 37.4%, respectively. After introducing the domain adversarial training module, the performance of the BiCross-EMO-DA model is further improved, with its WA and UA reaching 45.04% and 41.23%, respectively, representing improvements of 4.14% and 3.83% compared to the base model. This result further validates the effectiveness of domain adversarial training in cross-corpus sentiment recognition tasks, which can reduce the feature distribution differences between the source and target domains to a certain extent, thereby improving the model's recognition performance in the target domain.

[0090] The experimental results in Tables 1 and 2 show that the model proposed in this invention exhibits stable performance improvements in both cross-corpus sentiment recognition tasks. This indicates that introducing a domain adversarial training mechanism can effectively enhance the model's cross-domain generalization ability and improve the adaptability of the multimodal sentiment recognition model across different corpora.

[0091] Furthermore, in order to analyze the feature distribution of the model in the cross-corpus sentiment recognition task, this embodiment uses the t-SNE method to visualize and analyze the multimodal fusion features learned by the model. By mapping the high-dimensional features to a two-dimensional space, the distribution of source and target domain samples in the feature space can be observed.

[0092] As shown in Figure 6(a), in the IEMOCAP cross-MSP-IMPROV task, without domain adversarial training, the source domain samples and target domain samples exhibit significant distribution differences in the feature space. The overall data distribution across different domains shows a certain degree of separation, indicating that the features learned by the model still contain strong domain-related information. Figure 6(b) shows the feature distribution after introducing domain adversarial training. It can be seen that the distribution difference between the source domain samples and target domain samples has decreased, and the overlap between samples from different domains in the feature space has significantly increased. This indicates that domain adversarial training can, to some extent, align the feature distributions of the source and target domains, thereby mitigating the impact of cross-corpus data distribution differences.

[0093] As shown in Figure 7(a), in the IEMOCAP cross-CREMA-D task, when domain adversarial training is not used, the samples exhibit a relatively obvious clustering phenomenon in the feature space, and the distribution boundaries between samples of different categories are not clear enough. At the same time, some samples show strong text-related clustering features. Further considering the characteristics of the CREMA-D dataset, it can be found that this dataset consists of a small number of fixed texts, and different actors express the same text with different emotions. Therefore, in multimodal fusion features, textual semantic information may have a strong influence on feature distribution.

[0094] As shown in Figure 7(b), after introducing domain adversarial training, the distribution difference between source domain samples and target domain samples is reduced, and the feature distributions between some sentiment categories are more similar, indicating that domain adversarial training can improve the feature alignment ability of cross-corpus sentiment recognition models to a certain extent. However, due to the strong repetition of target domain text content, the influence of text modality on fused features is still quite significant, which to some extent limits the model's ability to further improve cross-domain sentiment discrimination.

[0095] Based on the above analysis, it can be seen that in cross-corpus multimodal sentiment recognition tasks, while relying solely on domain adversarial training can alleviate some of the inter-domain distribution differences, the text modality may still occupy a large weight in the fused features when the target domain text has strong repetition, thus affecting the effective expression of sentiment information. Therefore, this invention proposes a text contribution adjustment mechanism in step 8. By adaptively adjusting the contribution ratio of text modalities in the fusion process, the influence of fixed text on feature distribution is reduced, allowing the model to pay more attention to acoustic information related to sentiment expression, thereby further improving the stability and generalization ability of cross-corpus sentiment recognition.

[0096] In summary, this invention proposes a cross-corpus bimodal speech emotion recognition method and apparatus for unlabeled target domains. By constructing a bidirectional cross-modal attention mechanism, it achieves deep information interaction and emotion alignment between the text and audio modalities. Furthermore, by combining a fusion self-attention mechanism and a gated fusion structure, it enhances the expressive power of the multimodal joint representation while preserving the emotion feature discrimination capabilities of each modality. Simultaneously, in cross-corpus scenarios, a domain adversarial training mechanism is introduced, using a gradient inversion layer to align feature distributions, making the learned multimodal emotion representations insensitive to distribution differences between the source and target domains. Moreover, by introducing text repetition judgment and text contribution adjustment mechanisms, it effectively suppresses the semantic dominance effect caused by fixed or highly repetitive text in the target domain, enhancing the separability between emotion categories. Therefore, this invention can achieve stable and accurate cross-corpus emotion recognition even when the target domain lacks emotion annotations, significantly improving the model's generalization ability and engineering application value.

Claims

1. A method for cross-database bimodal speech emotion recognition oriented towards unlabeled target domains, characterized in that, include: Step 1: Obtain sentiment-labeled samples from the source domain and unlabeled samples from the target domain, where each sample includes speech data and corresponding text data; Step 2: Input the speech data and corresponding text data into a pre-constructed multimodal emotion recognition model. Input the speech data into an audio feature encoder to obtain audio feature representation; input the text data into a text feature encoder to obtain text feature representation. Step 3: Input the audio sequence features and text sequence features into the cross-modal bidirectional cross-attention module to obtain the audio-guided cross-modal enhanced feature sequence and the text-guided cross-modal enhanced feature sequence; Step 4: Concatenate the two cross-modal enhanced feature sequences in the time dimension and input them into the fusion self-attention module to model the global dependency relationship to obtain the fused refined feature sequence; Step 5: Perform masked pooling on the fused and refined feature sequence to obtain the fused summary vector; Step 6: Simultaneously perform masked pooling on the audio-side cross-modal enhancement feature sequence and the text-side cross-modal enhancement feature sequence to obtain the audio summary vector and the text summary vector; Step 7: Generate a dimension-wise gating weight through the gating fusion module, and perform a weighted summation on the audio summary vector and the text summary vector to obtain the gating fusion vector; Step 8: Perform residual fusion and normalization on the gated fusion vector and the fusion summary vector to obtain the final multimodal sentiment representation vector; Step 9: Input the multimodal sentiment representation vector into the sentiment classifier, output the sentiment prediction result, and calculate the sentiment classification loss based on the source domain samples; Step 10: Input the multimodal sentiment representation vector into the domain discriminator through the gradient inversion layer, output the domain prediction result, and calculate the domain discrimination loss based on the source domain samples and the target domain samples; Step 11: Jointly train the emotion classification loss and the domain discrimination loss to make the multimodal emotion representation vector insensitive to the difference in distribution between the source domain and the target domain; Step 12: In the inference phase, input the target domain samples into the trained model and output the sentiment category of the target domain samples.

2. The method for cross-database bimodal speech emotion recognition oriented towards unlabeled target domains according to claim 1, characterized in that, The audio feature encoder outputs an audio feature sequence. The text feature encoder outputs a text feature sequence. , The cross-modal cross-attention includes two branches: text query audio and audio query text. in, , , , , , This is a learnable linear mapping matrix used to map input features into query vectors, key vectors, and value vectors, respectively. The feature dimension of a single attention head is used to scale the dot product result.

3. The method for cross-database bimodal speech emotion recognition oriented towards unlabeled target domains according to claim 1, characterized in that, The sentiment classifier outputs the sentiment category probability distribution through a fully connected mapping and a Softmax function. The emotion classification loss uses cross-entropy loss, which is calculated only for labeled samples in the source domain.

4. The method for cross-database bimodal speech emotion recognition oriented towards unlabeled target domains according to claim 1, characterized in that, The gradient inversion layer satisfies: Forward propagation: Backpropagation: in, To counteract strength coefficient.

5. The method for cross-database bimodal speech emotion recognition oriented towards unlabeled target domains according to claim 1, characterized in that, The joint training is implemented based on a joint training objective function, which is a weighted sum of sentiment classification loss and domain discrimination loss. Samples in the target domain without sentiment labels only participate in the calculation of domain discrimination loss, and do not participate in the calculation of sentiment classification loss.

6. The method for cross-database bimodal speech emotion recognition oriented towards unlabeled target domains according to claim 1, characterized in that, The audio feature encoder adopts a pre-trained acoustic representation model structure based on self-supervised learning; The text feature encoder adopts a pre-trained language model structure based on a self-attention mechanism; The encoders all employ a local unfreezing strategy, updating only high-level parameters.

7. The method for cross-database bimodal speech emotion recognition oriented towards unlabeled target domains according to claim 1, characterized in that, During the training or inference phase, the target domain text data is subjected to a repetition determination. When the target domain text is determined to meet the preset repetition condition, a text contribution suppression strategy is enabled to reduce the weight of the text modality in the fused features. When the repetition condition is not met, the text contribution suppression strategy is disabled and fused features are generated according to the default fusion strategy.

8. A cross-database bimodal speech emotion recognition device for unlabeled target domains, characterized in that, include: The data acquisition module is used to acquire samples with sentiment labels from the source domain and samples without sentiment labels from the target domain, where each sample includes speech data and corresponding text data; An audio feature encoding module is used to extract features from the speech data and output an audio feature sequence. The text feature encoding module is used to extract features from the text data and output a text feature sequence; A bidirectional cross-modal cross-attention module is used to construct text query audio branches and audio query text branches respectively, realize intermodal information interaction, and generate text-guided cross-modal enhanced feature sequences and audio-guided cross-modal enhanced feature sequences. A self-attention module is integrated to model the global dependency relationship after concatenating the two cross-modal enhanced feature sequences; The mask pooling module is used to perform mask pooling on the fused feature sequence and the two-modal enhanced feature sequence to generate fused summary vector, text summary vector and audio summary vector; The gated fusion module is used to generate dimension-wise gated weights, perform weighted summation of the text summary vector and the audio summary vector, and perform residual fusion with the fused summary vector to obtain the final multimodal sentiment representation vector. The text duplication determination module is used to perform duplication detection or distribution difference analysis on target domain text, and outputs a trigger signal when preset conditions are met. The text contribution adjustment module is used to scale or constrain the text modality weights generated in the gated fusion module when the trigger signal is received, so as to reduce the contribution ratio of the text modality to the fusion representation. The sentiment classification module is used to input the multimodal sentiment representation vector into the sentiment classifier, output the sentiment prediction result, and calculate the sentiment classification loss based on the source domain samples. The domain adversarial module includes a gradient inversion unit and a domain discrimination unit, which are used to input the multimodal sentiment representation vector into the domain discrimination unit via the gradient inversion unit, output the domain prediction result, and calculate the domain discrimination loss based on the source domain samples and the target domain samples. The joint optimization module is used to jointly train each module based on the sentiment classification loss and the domain discrimination loss, so that the multimodal sentiment representation vector is insensitive to the difference in distribution between the source domain and the target domain. The sentiment output module is used to output the sentiment category of the target domain samples during the inference phase.

Images