Multi-mode signal processing method for audio emotion recognizer

Abstract

The invention provides a multi-modal signal processing method for an audio emotion recognizer, and belongs to the technical field of artificial intelligence, natural language processing and large models.The method comprises the steps that audio features and text features are extracted through a long-short-term memory network and a pre-trained BERT model respectively and divided into a modal invariant part and a modal specific part; the core of the method is that a modal binding mechanism is designed: for modal invariant features, the consistency information of the modal invariant features is enhanced by calculating a cross modal weight matrix; for modal specific features, a cross attention mechanism is adopted to capture a dynamic complementary relationship, after the bound features are combined with classification and position embedding, the features are sent to a Transform encoder for deep interactive learning, and finally, a classifier outputs an emotion recognition result. According to the method, optimization is carried out through a composite loss function including task loss, similarity loss and difference loss, accurate distinguishing and deep fusion of multi-modal features are achieved, and high efficiency and robustness are achieved.

Description

Technical Field

[0001] This invention belongs to the fields of multimodal information processing, artificial intelligence and emotion computing, and specifically relates to a multimodal signal processing method for audio emotion recognition. Background Technology

[0002] Multimodal sentiment analysis aims to accurately identify human emotional states by comprehensively analyzing data from multiple modalities, including text and speech. The core research question in this field is how to design efficient multimodal fusion strategies to fully integrate the consistent and dissimilar information contained in different modalities.

[0003] Early fusion immediately concatenates or jointly represents features after extraction before feeding them into a classifier for sentiment prediction. Poria et al. used a multi-kernel learning algorithm to train the combined feature vector after extracting multimodal features. Late fusion weights or votes the analysis results of each single modality to arrive at a final conclusion. Zadeh et al. proposed a multi-attention recurrent network. Both of these strategies struggle to capture fine-grained dynamic interactions between modalities, limiting the model's expressive power. To address this limitation, researchers have proposed various more refined fusion schemes, such as tensor-based methods, attention-based methods, and cross-modal translation methods. Although these methods have made significant progress, their complex structural designs often rely on domain knowledge, limiting their versatility in practical applications.

[0004] In recent years, the Transformer model, due to its powerful self-attention mechanism, has demonstrated great potential in processing data from different modalities and is considered a highly promising multimodal fusion framework. Its self-attention mechanism can not only capture global information but also naturally capture the interaction relationships between multimodal data. Currently, various Transformer-based multimodal sentiment analysis models have emerged. Some models enhance the interaction between paired modalities by designing cross-modal attention modules, but fewer consider the synchronous coupling of multiple modalities. Furthermore, a key challenge in current research is how to effectively distinguish and extract modality-invariant features from modality-specific features. While existing models can learn invariant features using shared network structures and are constrained by similarity loss functions, they often fail to adequately distinguish between the two, potentially leading to model bias towards a particular modality during learning. Summary of the Invention

[0005] The technical problem this invention aims to solve is that existing audio emotion recognition methods, when fusing multimodal information such as speech and text, struggle to capture fine-grained dynamic interactions between modalities and cannot effectively distinguish and utilize modality-invariant features (emotional information shared in speech and text) and modality-specific features (emotional information unique to tone or word choice). This invention provides a novel multimodal signal processing method that aims to improve the model's ability to distinguish and integrate multimodal features through a designed modality binding mechanism and a specialized loss function, thereby enhancing the accuracy of emotion recognition.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A multimodal signal processing method for audio emotion recognition, based on the Transformer architecture, includes the following steps:

[0008] Step 1: Extract audio modal features and text modal features respectively to obtain audio feature sequences and text feature sequences;

[0009] Step 2: Divide the audio feature sequence and text feature sequence into modality-invariant feature parts and modality-specific feature parts;

[0010] Step 3: Design a modality binding mechanism to process the features: fuse the modality-specific feature parts of the audio modality features and text modality features using a cross-attention mechanism to capture their dynamic interaction relationship; perform linear projection on the modality-invariant feature parts of the audio modality features and text modality features and calculate the cross-modality weight matrix, then multiply it with the original modality-invariant feature parts to enhance modal interaction;

[0011] Step 4: Add the bound modality-invariant features and modality-specific features to the learnable classification embedding and position embedding, respectively, to form the final feature matrix;

[0012] Step 5: Input the final feature matrix into a Transformer encoder for processing, and learn the deep interaction relationships between features through its self-attention mechanism;

[0013] Step 6: Feed the output of the Transformer encoder into the classifier to obtain the final emotion classification result.

[0014] Preferably, the feature extraction process in step 1 is as follows: the audio signal is processed using a Long Short-Term Memory (LSTM) network to extract audio modal features, and its parameters are updated as the model is trained; at the same time, a pre-trained BERT model is used to process the audio transcribed text to extract text modal features, and its parameters are kept frozen during model training.

[0015] Preferably, the binding mechanism for modality-invariant features in step 3 is as follows: the modality-invariant feature parts of the audio modality features and the text modality features are mapped to a preset N-dimensional hidden space through independent linear projection layers. Then, matrix multiplication is performed on the two projected features to obtain a cross-modality weight matrix. The obtained cross-modality weight matrix is ​​used as weights and combined with the original modality-invariant feature parts in a weighted manner, so that the model can adaptively learn the consistency information between the two modalities.

[0016] Preferably, the cross-attention mechanism described in step 3 is as follows: taking the modality-specific feature part of one modality as the query and the modality-specific feature part of another modality as the key and value, calculating a feature representation that integrates the information of the other modality, and performing this process alternately on the two modalities to achieve bidirectional information enhancement.

[0017] Preferably, the process of constructing the final feature matrix in step 4 is as follows: combining the bound modality-invariant feature part and the fused modality-specific feature part together, adding a learnable classification identifier at the beginning of the sequence, and adding a position embedding to each element in the sequence to preserve its position information in the sequence.

[0018] Preferably, the core composite loss function in the model training phase described in steps 5 and 6 This function is calculated after the final classification result is output and is used to update all network parameters (including audio LSTM, modality binding projection layer, Transformer encoder, and various embedding vectors) through backpropagation, except for the frozen BERT. The formula is as follows:

[0019] ;

[0020] ;

[0021] ;

[0022] ;

[0023] Among them, task loss (L task ) Calculate the predicted value (pred) in the batch of samples N. i ) and real labels (y i To optimize the performance of the main task, the error of similarity loss (L) is calculated. sim By maximizing the audio and text modality invariant features (O) I Cosine similarity (sim) between modalities is used to extract cross-modal commonalities; difference loss (L) diff Using the Frobenius norm (||·||)F Forced mode invariance feature (O) I ) and modality-specific features (O S The features are orthogonal to each other, and the specific features of different modes are orthogonal to each other to ensure the complementarity and uniqueness of the features. The three are weighted and balanced by hyperparameters (λ1 and λ2).

[0024] Compared with the prior art, the beneficial effects of the present invention are:

[0025] 1. Precise feature differentiation and binding: Through an innovative modality binding mechanism and a three-stage composite loss function, the model can be explicitly guided to learn and distinguish between modality-invariant features and modality-specific features, overcoming the problems of feature confusion and information redundancy in traditional fusion methods.

[0026] 2. Deep cross-modal interaction: The interaction between modality-invariant features is enhanced by using a cross-modal weight matrix, the interaction between modality-specific features is enhanced by using a cross-attention mechanism, and the global dependencies between all features are captured by the Transformer architecture, achieving a deeper and more granular information fusion than simple splicing or late fusion.

[0027] 3. Efficient end-to-end learning: The entire processing flow is an end-to-end deep learning model, which reduces the reliance on complex manual feature engineering and fusion rule design in traditional methods, and improves the versatility and deployment efficiency of the method. Attached Figure Description

[0028] Figure 1 This is a system overview diagram of a self-powered wearable audio emotion recognition device involved in this invention;

[0029] Figure 2 This is a diagram of the audio emotion recognition model architecture used in this invention. Detailed Implementation

[0030] To facilitate understanding of the technical content of this invention by those skilled in the art, the invention will be further described in detail below with reference to the accompanying drawings and specific examples. It should be understood that the specific examples described herein are merely illustrative and not intended to limit the scope of the invention.

[0031] This example describes the specific implementation and performance evaluation of a multimodal signal processing method for an audio emotion recognition device. The algorithm is integrated into a self-powered wearable audio emotion recognition system. At the hardware level, the system uses a flexible organic photovoltaic (FOPV) device as the core power supply unit to provide energy for the entire system. Audio signals are acquired by a digital microphone (model SPH0645), and the acquired data is processed by the main control MCU unit (model ESP32-WROOM-32E) and uploaded to a cloud server via a 2.4GHz WiFi module for storage and subsequent emotion recognition analysis.

[0032] like Figure 1 As shown, Figure 1 In this context, 'a' represents the device's mounting and layered structure. Figure 1 In the diagram, 'b' represents the system hardware and data flow. Figure 1 In this context, 'c' represents the UI (User Interface).

[0033] Users access cloud data via a mobile app, triggering the signal processing flow disclosed in this invention. Upon receiving the instruction, the server executes the following algorithm steps: first, features are extracted from the audio signal and its transcribed text; then, features are fused using a modality binding mechanism; finally, the fused features are input into the Transformer model for calculation, and the final emotion classification result is fed back to the user's app interface, such as... Figure 2 As shown, specifically:

[0034] 1. Multimodal feature extraction;

[0035] In the feature extraction stage, this method employs a dual-track approach to process audio and text data. For the audio modality, a Long Short-Term Memory (LSTM) network is used to capture its temporal information and contextual dependencies. The parameters of the LSTM are learnable and continuously optimized as the overall model is trained. For the text modality (transcribed from the audio signal), a pre-trained BERT model is used to extract deep semantic features. To preserve BERT's powerful language representation capabilities, its parameters are frozen during model training and not updated, as shown in the following formula:

[0036] ;

[0037] ;

[0038] Among them, text features (F t ) and audio features (F a ) respectively from input data (U t and A t ) is generated, the former through parameter (θ) BERTThe frozen pre-trained BERT model extracts semantics to preserve meaning, which is achieved through parameters (θ). LSTM Updatable LSTM network extraction.

[0039] 2. Modal binding and fusion;

[0040] Extracted audio features (F a ) and text features (F t The modality is further divided into modality-specific and modality-invariant components. The modality binding mechanism designed in this method is as follows:

[0041] For modality-specific features, a cross-attention mechanism is used for interaction. Text-specific features are used as queries to focus on relevant parts of audio-specific features (as keys and values), thereby generating an audio feature rich in textual contextual information. The reverse is also true. This allows for the dynamic capture of complementary information between the two modalities.

[0042] For modality-invariant features—that is, the parts expressing common sentiment in the two modalities—a method similar to that in the CLIP paper is adopted. First, they are projected onto the same N-dimensional space through their respective linear layers. Then, a cross-modal weight matrix is ​​calculated using matrix multiplication. This weight matrix is ​​subsequently used to weight the original modality-invariant features, thereby enhancing the information interaction between the two modalities, as shown in the following formula:

[0043] ;

[0044] ;

[0045] The linear projection layer (Pro) maps features to the same dimension and processes them through layer normalization (LN). It uses matrix multiplication (@) to calculate the cross-modal weight matrix and then uses element-wise multiplication (∗) to generate binding features (Ft′) and (Fa′) that enhance consistency information.

[0046] 3. Model training and classification;

[0047] After modality binding, the modality-specific and modality-invariant features from the Transformer encoder output are respectively augmented with classification embeddings (CLS tokens) and positional embeddings, resulting in a unified feature sequence. This sequence is then fed into a classifier to obtain the emotion recognition result. The entire model is optimized during training using a composite loss function that includes task loss, similarity loss, and dissimilarity loss, as shown in the following formula:

[0048] ;

[0049] The final feature matrix OF is constructed by concatenating the modality-invariant features O through a concatenation operation (⊕). I With modality-specific features O S Together, learnable classification identifiers (CLS) are embedded in the sequence to enhance the model's ability to distinguish modal types and aggregate context. Finally, positional embeddings (PE) are superimposed to preserve the temporal position information of the sequence, thus forming a unified feature sequence for the Transformer encoder to perform deep interactive learning.

[0050] 4. Effectiveness evaluation;

[0051] To evaluate the performance of this method, the publicly available MOSI dataset was used for training and testing, with the dataset divided into a 60% training set, 10% validation set, and 30% test set. In an experimental system powered by a FOPV (Follicular Unit Transmission Vehicle), the method was used to perform emotion recognition on audio signals from the test set. The prediction accuracies for the three rounds of testing were 94.6%, 95.7%, and 95.7%, respectively. In contrast, under the same conditions using traditional dry-cell batteries, the accuracies for the three rounds of testing were 96.8%, 97.8%, and 95.7%, respectively. The experimental results demonstrate that this method maintains extremely high recognition accuracy even on a self-powered system, with an average performance difference of only about 1.5% compared to a stable power supply, validating the robustness and efficiency of the algorithm.

[0052] The present invention has been described through the above embodiments. However, it should be understood that the above embodiments are for illustrative purposes only and are not intended to limit the invention to the scope of the described embodiments. Furthermore, those skilled in the art will understand that the present invention is not limited to the above embodiments, and many more variations and modifications can be made based on the teachings of the present invention, all of which fall within the scope of protection claimed by the present invention. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal signal processing method for an audio emotion recognizer, the method being based on the Transformer architecture, characterized in that, Includes the following steps: Step 1: Extract audio modal features and text modal features respectively to obtain audio feature sequences and text feature sequences; Step 2: Divide the audio feature sequence and text feature sequence into modality-invariant feature parts and modality-specific feature parts; Step 3: Design a modality binding mechanism to process the features: fuse the modality-specific feature parts of the audio modality features and text modality features using a cross-attention mechanism to capture their dynamic interaction relationship; perform linear projection on the modality-invariant feature parts of the audio modality features and text modality features and calculate the cross-modality weight matrix, then multiply it with the original modality-invariant feature parts to enhance modal interaction; Step 4: Add the bound modality-invariant features and modality-specific features to the learnable classification embedding and position embedding, respectively, to form the final feature matrix; Step 5: Input the final feature matrix into a Transformer encoder for processing, and learn the deep interaction relationships between features through its self-attention mechanism; Step 6: Feed the output of the Transformer encoder into the classifier to obtain the final emotion classification result.

2. The multimodal signal processing method for an audio emotion recognition device according to claim 1, characterized in that, The feature extraction process described in step 1 is as follows: the audio signal is processed using a Long Short-Term Memory (LSTM) network to extract audio modal features, and its parameters are updated as the model is trained; at the same time, a pre-trained BERT model is used to process the audio transcribed text to extract text modal features, and its parameters are kept frozen during model training.

3. The multimodal signal processing method for an audio emotion recognition device according to claim 1, characterized in that, The binding mechanism for modality-invariant features described in step 3 is as follows: the modality-invariant feature parts of the audio modality features and the text modality features are mapped to a preset N-dimensional hidden space through independent linear projection layers. Then, matrix multiplication is performed on the two projected features to obtain a cross-modality weight matrix. The obtained cross-modality weight matrix is ​​used as weights and combined with the original modality-invariant feature parts in a weighted manner, so that the model can adaptively learn the consistency information between the two modalities.

4. The multimodal signal processing method for an audio emotion recognizer according to claim 1, characterized in that, The cross-attention mechanism described in step 3 is as follows: the modality-specific feature part of one modality is used as the query, and the modality-specific feature part of another modality is used as the key and value. A feature representation that integrates the information of the other modality is calculated, and the process is performed alternately on the two modalities to achieve bidirectional information enhancement.

5. A multimodal signal processing method for an audio emotion recognition device according to claim 1, characterized in that, The process of constructing the final feature matrix in step 4 is as follows: the bound modality-invariant feature part and the fused modality-specific feature part are combined together, and a learnable classification identifier is added to the front of the sequence. At the same time, a position embedding is added to each element in the sequence to preserve its position information in the sequence.

6. A multimodal signal processing method for an audio emotion recognition device according to claim 1, characterized in that, The core composite loss function described in steps 5 and 6 during the model training phase This function is calculated after the final classification result is output. It is used to update all network parameters (including audio LSTM, modality binding projection layer, Transformer encoder, and various embedding vectors) through backpropagation, except for the frozen BERT. The formula is as follows: ； ； ； ； Among them, task loss (L task ) Calculate the predicted value (pred) in the batch of samples N. i ) and real labels (y i To optimize the performance of the main task, the error of similarity loss (L) is calculated. sim By maximizing the audio and text modality invariant features (O) I Cosine similarity (sim) between modalities is used to extract cross-modal commonalities; difference loss (L) diff Using the Frobenius norm (||·||) F Forced mode invariance feature (O) I ) and modality-specific features (O S The features are orthogonal to each other, and the specific features of different modes are orthogonal to each other to ensure the complementarity and uniqueness of the features. The three are weighted and balanced by hyperparameters (λ1 and λ2).

Images