A cross-language speech emotion recognition method based on structural bias alignment

Abstract

The application discloses a cross-language speech emotion recognition method based on structural bias alignment, and belongs to the technical field of speech signal processing and emotion computing. The application aims to solve the problems that the existing cross-language speech emotion recognition method ignores emotion category structural bias, intra-class distribution inconsistency and weak generalization ability. The method comprises the following steps: obtaining source language and target language speech data and extracting deep emotion features; learning and unifying cross-language emotion category topological relationship through an inter-class structure alignment module; adaptively maintaining the compactness of the distribution of each category through an intra-class shape alignment module; introducing sample reliability supervision to suppress noise interference; and weighting and fusing classification loss, inter-class structure loss, intra-class shape loss and sample reliability loss to realize multi-objective joint optimization training. The application significantly reduces cross-language emotion structure difference through a double-structure alignment mechanism, improves emotion clustering boundary definition and intra-class distribution consistency, has higher recognition accuracy and robustness in a multi-language unlabeled migration scene, and can be widely applied to multi-language human-computer interaction and cross-cultural emotion computing, intelligent vehicle systems and the like.

Description

Technical Field

[0001] This invention relates to the fields of speech emotion recognition, cross-language transfer learning, and emotion computing, specifically to a cross-language speech emotion recognition method using a structural bias alignment network for multilingual human-computer interaction. Background Technology

[0002] Cross-lingual speech emotion recognition (CLSER) refers to training a model on labeled data in one language (source language) and transferring emotional knowledge to another unseen language (target language) to complete the emotion recognition task. It has significant application value in scenarios such as multilingual intelligent assistants, cross-cultural human-computer interaction, in-vehicle voice interaction, and intelligent customer service. Most existing cross-lingual speech emotion recognition methods focus on feature distribution alignment, narrowing the feature differences between the source and target languages ​​through domain adaptation and contrastive learning. However, these methods have significant shortcomings in practical applications.

[0003] 1. Ignoring biases in the structure of emotion categories: Due to cultural and expressive differences, the semantic topology of emotions varies across languages. For example, in Chinese, anger and fear, happiness and surprise are semantically similar, while in English, the boundaries are clearer; Chinese expressions of sadness are compact, while English expressions of sadness are more dispersed. Existing methods force a uniform structure of emotion space, leading to semantic distortion and category confusion.

[0004] 2. Focusing only on feature distribution alignment without modeling structural relationships: Mainstream methods use methods such as maximum mean difference (MMD), adversarial domain adaptation, and contrastive learning to align global feature distributions, without explicitly modeling and aligning the relative relationships between sentiment categories.

[0005] 3. Intra-class distribution patterns cannot be adaptively maintained: Uniform intra-class constraints will over-compress compact categories, incorrectly aggregate scattered categories, and lose fine-grained sentiment information.

[0006] 4. Noise samples interfere with structural learning: Without considering the reliability of the samples, outliers and noise samples will disrupt cross-linguistic structural consistency and reduce the model's generalization ability.

[0007] Therefore, there is an urgent need for a cross-lingual speech emotion recognition method that can simultaneously model the inter-class topological relationships and intra-class distribution patterns of cross-lingual emotion classes and has sample reliability supervision, in order to solve the performance degradation problem caused by structural bias. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide a cross-lingual speech emotion recognition method based on structural deviation alignment. By inter-class structural alignment, intra-class morphological alignment and sample reliability supervision, it achieves cross-lingual emotion structure consistency learning and improves the accuracy and robustness of emotion recognition in the target language.

[0009] To achieve the above objectives, this invention provides the following technical solution: a cross-lingual speech emotion recognition method based on structural deviation alignment, comprising the following steps: Step S1: Multilingual speech feature extraction; acquiring speech data of the source and target languages, and using a pre-trained speech encoder to convert the original speech into a fixed-dimensional emotion feature vector, providing input for subsequent structural learning. Step S2: Inter-class structural alignment; constructing a cross-lingual relationship matrix based on emotion category centers, aligning the relative relationships of emotion categories in the source and target languages, ensuring consistency in the emotional topology of different languages. Step S3: Intra-class morphological alignment; adaptively adjusting the aggregation strength of samples of the same category according to the distribution compactness of each language category, preserving the natural distribution morphology of emotions in different languages. Step S4: Sample reliability supervision; assigning high weights to valid samples and low weights to noisy samples to reduce the interference of abnormal samples on model training. Step S5: Joint optimization of multiple losses; weighted fusion of classification loss, inter-class structural loss, intra-class morphological loss, and sample reliability loss, uniformly training and updating the network to obtain a stable cross-lingual emotion recognition model.

[0010] The beneficial effects of this invention are as follows:

[0011] 1. Explicit modeling of cross-linguistic sentiment structure: By explicitly learning the relative relationships of sentiment categories through inter-class structural alignment, the problem of semantic topological inconsistency is solved, making the boundaries between similar and dissimilar sentiments in the source and target languages ​​clearer.

[0012] 2. Adaptive intra-class morphology preservation: Automatically adjusts aggregation strength based on the inherent distribution of language, avoiding the compression of compact categories and the erroneous aggregation of scattered categories, without destroying the emotional expression characteristics of the language itself, and preserving fine-grained emotional information.

[0013] 3. Strong robustness and anti-interference ability: Sample reliability supervision effectively reduces the impact of noise and improves the stability of the model in real-world scenarios.

[0014] 4. Superior generalization performance: It achieves leading results on multiple cross-language tasks, can be directly embedded into existing frameworks, and has strong versatility. Attached Figure Description

[0015] This invention has a total of appendices Figure 5 Zhang, of which:

[0016] Figure 1 This is a schematic diagram of the overall process of the method of the present invention.

[0017] Figure 2 This is a diagram illustrating the overall model architecture of the method of this invention.

[0018] Figure 3 This is a schematic diagram illustrating the principle of inter-class structural alignment in the method of this invention.

[0019] Figure 4 This is a schematic diagram illustrating the intra-class shape alignment principle in the method of this invention.

[0020] Figure 5 This is a flowchart illustrating the total loss in the method of the present invention. Detailed Implementation

[0021] Example 1: System Overall Architecture and Data Preprocessing

[0022] This embodiment is implemented based on the PyTorch deep learning framework and supports cross-language speech emotion recognition in German, English, and Chinese.

[0023] 1. Input data: Source language dataset and target language dataset, including publicly available emotional speech libraries such as EmoDB, eNTERFACE, CASIA, and IEMOCAP.

[0024] 2. Speech data preprocessing: All speech data is uniformly resampled to a 16kHz mono waveform without additional manual feature extraction.

[0025] 3. Feature Encoding: Wav2Vec 2.0-base is used as the pre-trained speech encoder. The original waveform is directly input, and the output is a 768-dimensional deep speech emotion feature vector, which serves as the input for the subsequent structure alignment module. This encoder has strong speech modeling capabilities and can effectively capture emotion-related information such as prosody and timbre.

[0026] 4. Batch construction: In each training round, small batches of data are sampled from the source language and the target language respectively to form paired batches for structure alignment.

[0027] Example 2: Inter-class structural alignment module

[0028] Inter-class structural alignment is used to unify the relative relationships of sentiment categories between the source and target languages, and to solve the problem of semantic topological inconsistency. The specific implementation steps are as follows:

[0029] 1. Calculate category centers: For the source and target language samples in the current training batch, calculate the feature center for each sentiment category. The average of all features within the same category is used to obtain the center position of that category in the feature space.

[0030] 2. Constructing the sentiment relation matrix: Based on the category centers of any language corpus, calculate the cosine similarity between any two category centers, and fill the matrix with the similarity values ​​to obtain the source language relation matrix and the target language relation matrix. Each element in the matrix represents the degree of correlation between the two sentiment categories.

[0031] 3. Calculate the inter-class structural alignment loss: The difference between two relation matrices is measured using the Frobenius norm. This difference is used as the loss function, and backpropagation reduces the error, achieving bidirectional alignment of cross-linguistic sentiment structures. The loss formula is defined as follows:

[0032]

[0033] in, The source language relation matrix, This is the target language relation matrix.

[0034] Example 3: Intra-class Shape Alignment Module

[0035] Intra-class shape alignment is used to adaptively maintain the intra-class distribution shape of different languages ​​and emotions, avoiding forced compression or excessive dispersion. The specific implementation steps are as follows:

[0036] 1. Calculate intra-class compactness: For each sentiment category in each language, calculate the average of the squared Euclidean distances from all samples in that category to the category center, and then take the reciprocal to obtain the intra-class compactness index. The higher the compactness, the more regular the expression of that sentiment in that language and the tighter the clustering.

[0037] 2. Adaptive weight allocation: Compactness is used as the adaptive weight, with higher aggregation constraints on compact categories and lower constraints on loose categories, thus preserving the emotional expression characteristics of the language itself.

[0038] 3. Calculate the intra-class morphological alignment loss: Use the compactness-weighted distance from the sample to the center as the loss to constrain the distribution morphology of samples within the same class. The formula is as follows:

[0039]

[0040] in, For adaptive weights, For sample features, This is the center of the corresponding category.

[0041] Example 4: Sample Reliability Supervision and Joint Optimization of Multiple Losses

[0042] 1. Sample Reliability Supervision: To suppress the interference of noisy samples and outliers on structure learning, sample reliability weights are introduced. High weights are assigned to valid samples with high classification confidence, and low weights are assigned to noisy samples, thus constructing a weighted classification loss:

[0043]

[0044] in, For sample reliability weights, This is the single-sample classification loss.

[0045] 2. Weighted Fusion of Multiple Losses: The standard cross-entropy classification loss, inter-class structural alignment loss, intra-class morphological alignment loss, and sample reliability loss are weighted by learnable coefficients to obtain the total loss function:

[0046]

[0047] in, For sentiment classification, cross-entropy loss, and These represent the losses for inter-class structural alignment and intra-class morphological alignment after supervision by a supervisory signal that suffers from sample reliability loss, respectively. and Hyperparameters, which can be learned or manually set, are used to balance the contributions of various loss terms.

Claims

1. A cross-lingual speech emotion recognition method based on structural deviation alignment, characterized in that, Includes the following steps: Step S1: Multilingual speech feature extraction; Obtain the source language speech datasets respectively With target language speech dataset The original speech waveform x is input into the pre-trained speech feature encoder. Mapping yields fixed-dimensional speech emotion feature vectors. ; Step S2: Cross-linguistic sentiment category structural alignment; Based on the current training batch Calculate the category center for each emotion category in the source language and the target language respectively. Construct an emotion relationship matrix between the source language and the target language based on category centers. By minimizing the error loss between the two relation matrices To achieve consistent alignment of relative topological structures across language emotion categories; Step S3: Cross-language sentiment category morphological alignment; Calculate the intra-class compactness index for each emotion category in both the source and target languages. Intra-class compactness is used as an adaptive weight. Using intra-class alignment loss Dynamically constrain the distance from similar samples to the corresponding category center, apply strong aggregation constraints to emotion categories with high compactness, and apply weak aggregation constraints to emotion categories with low compactness, so as to maintain the consistent distribution pattern of similar emotions across languages; Step S4: Sample reliability supervision; introducing sample reliability weights. By assigning low weights to noisy samples and high weights to valid samples, a sample reliability loss is constructed. To suppress the interference of noisy samples on structural alignment and model training; Step S5: Joint optimization of multiple losses; the total loss function is obtained by weighted summation of the emotion classification loss, inter-class structure alignment loss, intra-class morphological alignment loss, and sample reliability loss. The network parameters are updated by backpropagation based on the total loss function to complete the training and inference of the cross-language speech emotion recognition model.

2. The cross-lingual speech emotion recognition method based on structural deviation alignment according to claim 1, characterized in that, The pre-trained speech feature encoder in step S1 uses the Wav2Vec 2.0 model, which directly takes the original speech waveform with a sampling rate of 16kHz as input and outputs a fixed-dimensional deep emotion feature vector as input to the subsequent structure alignment module.

3. The cross-lingual speech emotion recognition method based on structural deviation alignment according to claim 1, characterized in that, The cross-linguistic sentiment class structural alignment mentioned in step S2 specifically includes: For the current batch of source language data Calculate the first Category center of each emotion category For the target language data in the current batch Calculate the first Category center of each emotion category Based on the cosine similarity of sentiment category centers, sentiment relation matrices for the source language and the target language are constructed respectively. ; The square of the Frobenius norm between the two relation matrices is used as the inter-class structural alignment loss. : This loss achieves bidirectional cross-linguistic sentiment topology alignment by minimizing the loss.

4. The cross-lingual speech emotion recognition method based on structural deviation alignment according to claim 1, characterized in that, The cross-linguistic sentiment intra-category morphological alignment mentioned in step S3 specifically includes: For each sentiment category in both the source and target languages, calculate the mean squared Euclidean distance of all samples in that category to the category center, and derive the intra-class compactness index based on this mean. ; Using intra-class compactness index as adaptive weight We calculate the weighted distance loss from samples of the same class to the class center, and obtain the intra-class shape alignment loss as follows: This loss function adaptively adjusts the aggregation intensity of different languages ​​and emotion categories, preserving the inherent distribution pattern of emotion expression in each language.

5. The cross-lingual speech emotion recognition method based on structural deviation alignment according to claim 1, characterized in that, The sample reliability loss function described in step S4 The specific definition is: ,in This indicates the inter-class structural alignment stage or the intra-class morphological alignment stage for samples. The loss contribution. When the model is in the inter-class structure alignment stage. ,in For the sample Loss of global inter-class structural alignment The gradient contribution. When the model is in the intra-class morphological alignment stage. ,in For the sample Explicitly weighted loss contribution. This loss weights the classification loss with sample weights, reducing the impact of noisy samples and outliers on model training and improving the stability of structure learning and sentiment classification.

6. The cross-lingual speech emotion recognition method based on structural deviation alignment according to claim 1, characterized in that, The total loss function described in step S5 is specifically expressed as follows: ,in, For sentiment classification, cross-entropy loss, and These represent the losses for inter-class structural alignment and intra-class morphological alignment after supervision by a supervisory signal that suffers from sample reliability loss, respectively. and Hyperparameters, which can be learned or manually set, are used to balance the contributions of various loss terms.

7. A voice emotion recognition system for performing the method according to any one of claims 1 to 6, characterized in that, include: Speech feature extraction module: configured to preprocess and encode source and target language speech signals, and output deep speech emotion feature vectors; Dual-structure alignment module: includes inter-class structure alignment unit and intra-class form alignment unit, configured to learn and align inter-class topological relationships and intra-class distribution forms of cross-language sentiment; Sample reliability supervision module: configured to evaluate sample reliability and construct a weighted loss to suppress interference from noisy samples; Joint training optimization module: configured to integrate classification loss, inter-class structure loss, intra-class morphological loss, and sample reliability loss, update the network end-to-end and output emotion recognition results.

8. The system according to claim 7, characterized in that, The inter-class structural alignment unit is further configured as follows: We calculate the sentiment category centers within a batch, construct a cross-linguistic relation matrix, and use Frobenius norm loss to achieve bidirectional alignment of the relation matrix, thus clarifying the classification boundaries of adjacent sentiment categories.

9. The system according to claim 7, characterized in that, The intra-class morphological alignment unit is further configured as follows: The compactness index of categories is calculated, and the compactness is used as an adaptive weight to constrain the intra-class distribution. It strongly aggregates compact categories and weakly constrains dispersed categories, thus maintaining the natural form of cross-linguistic sentiment distribution.

10. The system according to claim 7, characterized in that, The system supports cross-language emotion recognition in German, English, and Chinese, and can run stably in transfer learning scenarios without labeled target language data.

Images