Speech Emotion Recognition Enhancement System Using Self-Attention-Based Masked Spectrogram Generation and Self-Supervised Learning

Abstract

The present invention relates to the field of speech signal processing and artificial intelligence (AI)-based emotion recognition technology, and to a system that improves speech emotion recognition performance by utilizing self-attention-based mask spectrogram generation and self-supervised learning. According to the present invention, a voice emotion classification model including a multilayer self-attention structure is trained using a spectrogram as input, and an attention map collection unit (100) collects attention maps produced from each attention head and each layer; a mask spectrogram generation unit (200) accumulates the attention maps collected by the attention map collection unit (100) layer by layer to identify a masking target area in the spectrogram and invalidates the spectrogram value corresponding to the area to generate a mask spectrogram; and a mask-based learning unit (300) performs self-supervised learning using the mask spectrogram and classifies voice emotions based on the results of the self-supervised learning. By combining self-attention-based mask spectrogram generation and self-supervised learning, various emotion characteristics can be efficiently learned and the reliability and accuracy of emotion recognition can be improved.

Description

Technology Field

[0001] The present invention relates to the field of speech signal processing and artificial intelligence (AI)-based emotion recognition technology, and more specifically, to a system that improves speech emotion recognition performance by utilizing self-attention-based mask spectrogram generation and self-supervised learning. Background Technology

[0002] Speech Emotion Recognition (SER) is attracting attention as a core technology in various fields, such as human-computer interaction, counseling support systems, and emotion analysis services. Conventional SER technology primarily utilizes deep learning-based models that classify emotions by extracting basic acoustic characteristics of speech, such as pitch, energy, and spectrograms.

[0003] However, in real-world environments, there are many limited datasets where the same sentence is repeatedly uttered, and various external factors exist, such as background noise, differences in speaker pronunciation, and changes in the acoustic environment. Under these conditions, existing deep learning-based SER models rely excessively on the unique acoustic patterns of the sentences, leading to frequent overfitting problems.

[0004] Furthermore, modes of emotional expression vary from speaker to speaker and exhibit diverse patterns depending on psychological and cultural factors. Consequently, existing learning methods tend to acquire only patterns specialized for specific datasets, making it difficult to secure the generalization performance required in real-world environments. In particular, when datasets are small or emotion classes are imbalanced, there is a high likelihood that the model will be trained with a bias toward certain classes.

[0005] Therefore, in the field of speech emotion recognition, there is an urgent need for training methods that enable models to learn various emotional characteristics even in limited data environments and generalize to new environments without performance degradation. Prior art literature

[0006] Registered Patent No. 2382191 (March 30, 2022) The problem to be solved

[0007] The present invention aims to solve the aforementioned problem by combining self-attention-based mask spectrogram generation with self-supervised learning, with the objective of efficiently learning various emotional characteristics and improving the reliability and accuracy of emotion recognition. means of solving the problem

[0008] The present invention relates to a voice emotion recognition enhancement system using self-attention-based mask spectrogram generation and self-supervised learning, comprising: an attention map collection unit (100) that learns a voice emotion classification model including a multilayer self-attention structure with a spectrogram as input and collects attention maps produced from each attention head and each layer; a mask spectrogram generation unit (200) that accumulates the attention maps collected by the attention map collection unit (100) layer by layer to identify a masking target area in the spectrogram and invalidates the spectrogram value corresponding to the area to generate a mask spectrogram; and a mask-based learning unit (300) that performs self-supervised learning using the mask spectrogram and classifies voice emotions based on the results of the self-supervised learning.

[0009] The attention map collection unit (100) receives a Log-Mel spectrogram as input and collects an attention map containing attention values ​​calculated through a multi-head self-attention operation and interaction relationships between each input token.

[0010] The mask spectrogram generation unit (200) generates a mask spectrogram by replacing the spectrogram value corresponding to the masking target area with 0.

[0011] The mask spectrogram generation unit (200) extracts tokens that are time-frequency domains greater than or equal to a preset threshold from the attention map, and then selectively additionally masks only those tokens to generate a partially masked spectrogram.

[0012] The mask-based learning unit (300) performs final emotion classification using a single linear layer and a token representing the entire input while the weights of the model learned through self-supervised learning are frozen.

[0013] Meanwhile, a method using a voice emotion recognition enhancement system utilizing self-attention-based mask spectrogram generation and self-supervised learning comprises: (a) a step in which the system learns a voice emotion classification model including a multilayer self-attention structure with a spectrogram as input, and collects attention maps produced from each attention head and each layer; (b) a step in which the system accumulates the collected attention maps layer by layer to identify a masking target region in the spectrogram, and invalidates the spectrogram value corresponding to the region to generate a mask spectrogram; (c) a step in which the system performs self-supervised learning using the mask spectrogram and classifies voice emotions based on the results of the self-supervised learning.

[0014] Step (a) receives a Log-Mel spectrogram as input and collects an attention map containing attention values ​​calculated through multi-head self-attention operations and interaction relationships between each input token.

[0015] (b) Step generates a mask spectrogram by replacing the spectrogram value corresponding to the masking target area with 0.

[0016] (b) Step (b) extracts tokens in the time-frequency domain that are above a preset threshold from the above attention map, and then selectively additionally masks only those tokens to generate a partially masked spectrogram.

[0017] (c) In this step, the final emotion classification is performed using a single linear layer and a token representing the entire input while the weights of the model trained through self-supervised learning are frozen. Effects of the invention

[0018] According to the present invention, by combining self-attention-based mask spectrogram generation with self-supervised learning, it is possible to efficiently learn various emotional characteristics and improve the reliability and accuracy of emotion recognition. Brief explanation of the drawing

[0019] Figure 1 shows the configuration of a voice emotion recognition enhancement system using self-attention-based mask spectrogram generation and self-supervised learning according to one embodiment of the present invention. FIG. 2 is a flowchart illustrating a method using a voice emotion recognition enhancement system utilizing self-attention-based mask spectrogram generation and self-supervised learning according to one embodiment of the present invention. FIG. 3 is a diagram illustrating an overview of a voice emotion recognition enhancement system using self-attention-based mask spectrogram generation and self-supervised learning proposed in one embodiment of the present invention. Figure 4 shows the process of calculating the final attention distribution accumulated through the attention rollout according to the present embodiment. Figure 5 illustrates the extraction of important regions and masking according to the present embodiment. Specific details for implementing the invention

[0020] The present invention relates to a technology for improving the performance of a Speech Emotion Recognition (SER) system by combining self-attention-based mask spectrogram generation and a self-supervised learning technique.

[0021] Figure 1 shows the configuration of a voice emotion recognition enhancement system using self-attention-based mask spectrogram generation and self-supervised learning according to one embodiment of the present invention.

[0022] As illustrated in FIG. 1, according to one embodiment of the present invention, it includes an attention map collection unit (100), a mask spectrogram generation unit (200), and a mask-based learning unit (300).

[0023] The attention map collection unit (100) learns a speech emotion classification model including a multilayer self-attention structure with a spectrogram as input, and collects attention maps produced from each attention head and each layer.

[0024] Here, a 'multilayer self-attention structure' refers to a network structure configured to process input spectrograms through multiple layers of self-attention operations, and to learn different levels of features and dependencies between tokens at each layer.

[0025] In addition, the term 'speech emotion classification model including a multi-layer self-attention structure' refers to a model in which the self-attention layers are arranged sequentially to stepwise extract features necessary for emotion classification from an input spectrogram and finally produce an emotion class.

[0026] The attention map collection unit (100) receives a Log-Mel spectrogram as input and collects an attention map containing attention values ​​calculated through a multi-head self-attention operation and interaction relationships between each input token.

[0027] Here, multi-head self-attention operation refers to an operation that extracts various features by applying different weight matrices to input spectrogram patch tokens to compute multiple attention heads in parallel, and allowing each head to pay attention to different parts of the input tokens.

[0028] Furthermore, the interaction relationship between each input token refers to information that numerically represents the importance (weight) and influence between tokens, expressed as an attention value indicating which location of information a specific token should focus on more compared to other tokens.

[0029] In this embodiment, the generalization performance of the model is improved by generating a self-attention map based on a data-efficient image transformer (DeiT) structure for the log-mel spectrogram of the input speech and selectively masking emotionally important regions using this.

[0030] For reference, the Data-efficient Image Transformer (DeiT) architecture refers to a transformer-based model proposed for image classification. It is designed to achieve high classification performance with relatively small training data by applying knowledge distillation and optimized training strategies to address the limitations of the existing Vision Transformer (ViT), which requires large datasets and long training times.

[0031] DeiT divides input data into patches, converts them into tokens, and extracts features by learning the relationships between tokens through a multi-head self-attention mechanism.

[0032] Regarding this attention map collection unit (100), the procedure is described as follows.

[0033] It consists of a learning phase and an attention map collection phase.

[0034] The training phase performs emotion recognition by training a DeiT-base model using Log-Mel spectrograms as input. In this embodiment, the DeiT base384 model pre-trained on ImageNet 1k is used as is.

[0035] However, this is merely one example, and it can be applied equally to any model capable of performing emotion recognition by receiving a spectrogram input using a self-attention-based transformer structure, regardless of its type or configuration.

[0036] In other words, although a DeiT-base model was used in this embodiment, it is not limited thereto, and any self-attention-based transformer model can be applied in the same way.

[0037] In the attention map collection step, self-attention maps generated from each transformer layer of the trained DeiT model are collected. These maps represent the interaction relationships between multiple heads and input tokens.

[0038] The mask spectrogram generation unit (200) accumulates the attention map collected by the attention map collection unit (100) layer by layer to identify the masking target area in the spectrogram, and generates a mask spectrogram by invalidating the spectrogram value corresponding to the area.

[0039] This mask spectrogram generation unit generates a mask spectrogram by replacing the spectrogram value corresponding to the masking target area with 0.

[0040] In addition, tokens in the time-frequency domain that are above a preset threshold are extracted from the attention map, and only those tokens are selectively additionally masked to generate a partially masked spectrogram.

[0041] The procedure of the mask spectrogram generation unit (200) for the present embodiment consists of an attention rollout step, an important region extraction step, and a masking step, and is described as follows.

[0042] In the attention rollout stage, the mask spectrogram generator (200) accumulates the collected attention maps layer by layer to produce a final attention map for all input tokens. At this time, the attention matrix of each layer is normalized through a softmax and weighted averaged with the identity matrix as follows to reflect residual connections.

[0043]

[0044] represents the attention matrix ultimately used in the l-th layer, and represents the raw attention value obtained through the self-attention operation of the l-th layer. I is the identity matrix, which is a factor that ensures the token maintains minimal influence on itself, and α represents the weight (a scalar value between 0 and 1) that controls the contribution of the raw attention value and the identity matrix.

[0045] In other words, the above equation represents a process of adding the value of the identity matrix to the raw attention value at a certain ratio to prevent attention between tokens from being excessively skewed in one direction and to obtain a stable attention value.

[0046] Here, α is the scaling parameter of the attention weights and has a value of 0.9.

[0047] The final flow of attention is defined as follows.

[0048]

[0049] represents the final attention matrix accumulated by passing through all self-attention layers, and represents the self-attention matrices from the 1st layer to the Lth layer, respectively. Attention is propagated hierarchically through matrix multiplication, and This represents the process of calculating the entire attention flow from the input token to the final output token.

[0050] In other words, the above mathematical formula is a process that sequentially accumulates attention information calculated in each layer to produce a final attention map indicating which region of the input spectrogram the model progressively focuses on.

[0051] Here, L represents the total number of self-attention layers.

[0052] The important region extraction step extracts the areas where the model focuses by selecting tokens (patches) that are greater than or equal to a threshold (e.g., 0.5) from the final rollout attention map.

[0053] The masking step removes part of the spectrogram by replacing the extracted important regions with zeros. This encourages the model to learn emotion-related features from a wider range rather than relying solely on specific regions.

[0054] Here, based on the generated self-attention map, the model removes unimportant regions and creates a mask by inverting the attention values. Subsequently, additional threshold filtering is performed to precisely distinguish between the regions to be used for actual training and those to be excluded. This resulting mask is applied to the original spectrogram to mask regions where the model paid excessive attention, thereby inducing the learning of relatively less prominent but generalizable sentiment cues.

[0055] In this embodiment, the identification and masking of important areas based on self-attention are described exemplarily as follows.

[0056] First, the log-Mel spectrogram input to the ViT series (self-attention-based) emotion recognition model is embedded in patch units (e.g., 16×16, overlap stride=10), and the self-attention values ​​generated in each transformer layer of the model are collected to obtain the final importance map through the attention rollout.

[0057] At this time, to prevent the model from relying excessively on a specific time-frequency range, high-importance tokens in the importance map are selectively removed (masked) to generate a mask spectrogram with some regions removed.

[0058] These mask spectrograms are input into self-supervised learning, which suppresses overfitting to specific patterns by removing regions that the model deems important, and mitigates the bias of the model focusing on specific tokens, thereby enabling additional learning of subtle time-frequency emotional cues.

[0059] As a result, the self-attention-based masking method according to the present invention can mitigate overfitting and improve the generalization performance of an emotion recognition model without external large-scale data or separate prior training.

[0060] The mask-based learning unit (300) performs self-supervised learning using the mask spectrogram generated by the mask spectrogram generation unit and classifies voice emotions based on the self-supervised learning results.

[0061] With the weights of the model trained through this self-supervised learning frozen, the final emotion classification is performed using a token representing the entire input and a single linear layer.

[0062] The mask-based learning unit according to the present embodiment consists of a self-supervised learning stage and a downstream learning stage.

[0063] In the self-supervised learning stage, the mask-based learning unit performs self-supervised learning using masked spectrograms as input. The output consists of two heads: a classification head for emotion classification and a restoration head for masking region restoration. Both heads consist of a single linear layer. Through this, the model simultaneously performs two types of learning: emotion recognition and spectrogram restoration.

[0064] In the downstream learning stage, the mask-based learning unit performs final sentiment classification by additionally training only CLS tokens and a single linear layer while keeping the learned weights frozen after completing self-supervised learning.

[0065] For reference, CLS token stands for Classification Token. It is a special token inserted at the beginning of a sequence when an input spectrogram is split into patches and converted into a sequence in a Transformer-based self-attention structure. This token is trained to collect summary information of the entire input and serves as a representative embedding vector used in the model's final output, such as sentiment classification.

[0066] Here, a single linear layer refers to a fully connected layer that produces an output value by applying a weight matrix and biases to an input feature vector, and performs only linear transformations without a non-linear activation function.

[0067] FIG. 2 is a flowchart illustrating a method using a voice emotion recognition enhancement system utilizing self-attention-based mask spectrogram generation and self-supervised learning according to one embodiment of the present invention.

[0068] As illustrated in FIG. 2, the system according to the present invention consists of the following steps.

[0069] A voice emotion recognition enhancement system using self-attention-based mask spectrogram generation and self-supervised learning (hereinafter referred to as the "system") learns a voice emotion classification model including a multilayer self-attention structure with a spectrogram as input, and collects attention maps produced from each attention head and each layer (a).

[0070] Next, the system accumulates the collected attention map layer by layer to identify the masking target area in the spectrogram, and generates a mask spectrogram by invalidating the spectrogram value corresponding to the area (b).

[0071] Next, the system performs self-supervised learning using a mask spectrogram and classifies speech emotions based on the results of the self-supervised learning (c).

[0072] Here, in step (a), the system receives a Log-Mel spectrogram as input and collects an attention map containing attention values ​​calculated through multi-head self-attention operations and interaction relationships between each input token.

[0073] In step (b), the system generates a mask spectrogram by replacing the spectrogram values ​​corresponding to the masking target area with 0. At this time, in step (b), tokens that are time-frequency domains greater than or equal to a preset threshold are extracted from the attention map, and then only those tokens are selectively additionally masked to generate a partially masked spectrogram.

[0074] In step (c), the system performs final emotion classification using a single linear layer and a token representing the entire input, while freezing the weights of the model learned through self-supervised learning.

[0075] FIG. 3 is a diagram illustrating an overview of a voice emotion recognition enhancement system using self-attention-based mask spectrogram generation and self-supervised learning proposed in one embodiment of the present invention.

[0076] FIG. 3 illustrates an overview of a speech emotion recognition enhancement system (SAMSG) using self-attention-based masked spectrogram generation and self-supervised learning proposed in one embodiment of the present invention, comprising (i) attention map collection, (ii) masked spectrogram-based self-supervised learning, and (iii) fine-tuning steps.

[0077] Here, (i) of FIG. 3 illustrates the attention map collection process, which includes dividing the input spectrogram into patches, inputting them into a self-attention-based model, and collecting the attention maps produced from the self-attention operation within the model.

[0078] In the present invention, the attention map is a value automatically generated within the model, and

[0079] First, (i) in the attention map collection process, input voice data is converted into a Log-Mel spectrogram, divided into patches of a certain size (e.g., 16×16), subjected to a linear projection process, and then input into a Transformer Encoder.

[0080] Inside the Transformer encoder, the interrelationships between each patch are learned through a self-attention mechanism, and through this, an attention map is generated that numerically represents the area within the time-frequency domain of the input spectrogram where the model focuses on emotion classification.

[0081] This map visualizes the model's attention focus and serves as foundational data for identifying areas important for emotion recognition.

[0082] For reference, the Linear Projection shown in Fig. 3 is a module for dividing an input Log-Mel spectrogram into patches of a certain size (e.g., 16×16) and then converting each patch into a vector of a fixed dimension.

[0083] That is, by converting the 2D form (time × frequency) of the original spectrogram into a 1D embedding vector sequence, it performs the role of mapping to an input form that the transformer encoder can process.

[0084] This process compresses the spatial (time-frequency) distribution of the spectrogram into a vector embedding (token embedding), allowing subsequent networks to learn this in token units while each patch retains feature information corresponding to specific time intervals and frequency bands of the speech.

[0085] Therefore, Linear Projection plays an important role in solving the problem of input format mismatch between spectrograms and deep learning models, and enabling efficient feature learning while minimizing time-frequency information loss.

[0086] Subsequently, the Transformer Encoder takes linearly projected patch vectors as input and performs self-attention operations.

[0087] The Transformer encoder dynamically calculates the correlation between input tokens to determine which region within the entire spectrogram each patch is closely related to.

[0088] To this end, the Query (Q), Key (K), and Value (V) vectors are calculated, and the interaction between patches is quantified through the following self-attention equation.

[0089]

[0090] Here, represents the dimensionality of the Key vector and probabilistically normalizes the importance between each patch through Softmax operation.

[0091] This enables the encoder to simultaneously learn global patterns and local sentiment cues of the entire spectrogram.

[0092] In this invention, an attention map is formed based on the results of such self-attention calculations to visually confirm the area the model focuses on during actual emotion classification and to identify time-frequency intervals important for emotion recognition.

[0093] In other words, the Transformer Encoder functions not as a simple feature extractor, but as an intelligent pattern analyzer that calculates attention regions directly associated with emotion recognition.

[0094] Through such a combined structure of Linear Projection and Transformer Encoder, the present invention can efficiently analyze the complex frequency-time correlation of a spectrogram and quantitatively calculate the attention region key to emotion recognition.

[0095] Next, (ii) in the self-supervised learning process (Self-Supervised Learning using SAMSG), high-attention regions above a threshold (Threshold > 0.5) are masked based on the previously generated attention map to create a self-attention-based masked spectrogram.

[0096] This mask removes or weakens parts of the input spectrogram on which the model relies excessively, thereby inducing the model to learn relatively less prominent emotional signals (intonation, intensity, subtle frequency changes, etc.).

[0097] For reference, the term 'threshold' here refers to a quantitative reference value used to distinguish the level at which the model is judged to be significantly focused on emotion recognition among the attention weights of each time-frequency patch (token) on the attention map.

[0098] Through the self-attention operation of the Transformer encoder, an attention weight between 0 and 1 is calculated for each patch of the input spectrogram, and a higher value indicates that the model recognizes that area as more important.

[0099] In the present invention, an area having a value of 0.5 or higher among these attention weights is defined as a 'high-attention area' and is set as a masking target area.

[0100] in other words, It is defined in the form of,

[0101] A(x,y) represents the attention value (attention weight) at time x and frequency y.

[0102] The purpose of setting these thresholds is to selectively remove specific segments (e.g., specific syllables, specific frequency bands) that the model over-attends during the emotion recognition process, thereby inducing the model to learn other secondary emotional cues—namely, changes in intonation, subtle energy fluctuations, the rhythm of the speaker's speech, etc.

[0103] The threshold is an experimentally adjustable hyperparameter; if set too low, the model masks almost all areas, causing the loss of sentiment features, while if set too high, key attention regions remain intact, preventing the assurance of learning variability.

[0104] Therefore, in this invention, through verification on several datasets (SAVEE, EmoDB, CREMA-D), it was confirmed that a threshold of 0.5 provides the most stable results in terms of emotion recognition performance and generalization.

[0105] Consequently, the threshold numerically distinguishes the importance of the region the model pays attention to, performs selective masking based on the distinction value, and acts as a key control variable to prevent the emotion recognition model from overfitting to a specific utterance pattern.

[0106] Subsequently, the model performs parallel learning through two training heads.

[0107] The first head is a Classification Head that predicts emotional states, and the second head is a Reconstruction Head that performs reconstruction learning to reconstruct masked regions.

[0108] By performing these two learning processes in parallel, the model can simultaneously achieve the complementary learning goals of emotion recognition and reconstruction, and form richer and more generalized latent representations.

[0109] Finally, in the (iii) Fine-Tuning step, additional training is performed on the downstream emotion recognition task with the weights pre-trained through self-supervised learning frozen.

[0110] At this point, the final emotion classification is performed by additionally training the CLS (Classification) token and a single linear layer.

[0111] The entire process of Figure 3 consists of a series of steps to identify key emotion-related regions based on self-attention, selectively mask important regions to suppress overfitting, learn generalized emotion expressions through self-supervised learning, and optimize actual emotion classification performance through fine-tuning.

[0112] Therefore, the system of the present invention can improve emotion recognition performance through its own self-attention-based feedback loop without external large-scale unsupervised data or pre-trained models, and can ensure high reliability and reproducibility across various speakers, languages, and environments.

[0113] To elaborate, first, the model is trained for emotion recognition to learn self-attention and identify key regions for speech emotion recognition within the spectrogram. In this process, each token represents a specific time-frequency range of the speech signal, and the self-attention map numerically indicates which areas the model focuses its attention on.

[0114] Through these self-attention maps, it is possible to visually identify repetitive acoustic patterns unique to sentences or regions with biases specific to the dataset.

[0115] Second, based on the generated self-attention map, the model removes unimportant regions and generates a mask through the inversion of attention values.

[0116] Subsequently, additional threshold filtering is performed to precisely distinguish between the regions to be used for actual training and those to be excluded. The resulting mask is applied to the original spectrogram to mask areas where the model paid excessive attention, thereby guiding the model to learn relatively less prominent but generalizable sentiment cues.

[0117] This process can be performed using only fully labeled datasets, without the need for large-scale unsupervised pre-training.

[0118] Third, the model performs fine-tuning for the downstream task of speech sentiment recognition. In this step, it predicts the final sentiment class using a single linear layer with the same structure as the classification head.

[0119] Representations learned through the proposed self-attention-based mask spectrogram generation are directly utilized for sentiment classification, enabling the recognition of more generalized sentiment patterns while relying less on repetitive sentence structures or dataset-specific biases.

[0120] Figure 4 shows the process of calculating the final attention distribution accumulated through the attention rollout according to the present embodiment.

[0121] Figure 4 illustrates the process of calculating the final attention distribution accumulated through an attention rollout according to an embodiment of the present invention, showing the result of visualizing the global attention flow of the model by dividing the input spectrogram into patches, performing self-attention calculations after linear projection, and weightedly accumulating the attention maps for each layer.

[0122] First, as shown above, the input is a log-Mel spectrogram of size 128×1001.

[0123] This spectrogram represents information on the time axis and frequency axis in a two-dimensional form, allowing the model to visually recognize acoustic features (intonation, intensity, frequency band, etc.).

[0124] Subsequently, the input spectrogram is divided into 16×16 patches, with the stride between patches set to 10, and consists of a total of 1,188 patch tokens.

[0125] Each patch is the smallest unit representing a specific time-frequency interval of speech, and is subsequently converted into a fixed-dimensional vector form through a linear projection process.

[0126] These vectors are provided as input to the Transformer Encoder, and self-attention operations are performed to learn the interrelationships between each patch.

[0127] The formula indicated in the center of the drawing This is the core of the self-attention operation, a step of calculating attention weights that indicate how much attention each patch should pay to other patches by normalizing the inner product between the Query (Q) and Key (K) vectors.

[0128] Subsequently, the generated attention matrix is ​​accumulated layer by layer through the Attention Rollout process.

[0129] Caution map for each floor silver It is normalized into the form of , reflecting the influence of residual connections, where α is the scaling parameter of attention weights and has a value of 0.9.

[0130] Through this, layer-by-layer attention information is accumulated without loss, ultimately forming the model's global attention flow.

[0131] The attention map shown at the bottom is the final attention distribution obtained through this cumulative process, visually indicating which part of the time-frequency domain of the input speech the model is focusing on.

[0132] In this invention, the attention map is used to identify regions on which the model relies excessively during the emotion recognition process, and by masking these regions in a subsequent step, the model's attention bias is mitigated and generalization performance is improved.

[0133] Figure 5 illustrates the extraction of important regions and masking according to the present embodiment.

[0134] FIG. 5 illustrates a process of self-attention-based important region extraction and masking according to an embodiment of the present invention, wherein an attention map is calculated from an input spectrogram, a high-attention region greater than or equal to a threshold value (0.5) is designated as a mask, and then removed from the input spectrogram, thereby suppressing the region on which the model relies excessively and inducing generalized emotion expression learning.

[0135] The present invention includes a process of converting input voice data into a Log-Mel spectrogram, then generating a self-attention map using a pre-trained Transformer-based model, and thereby identifying and masking key time-frequency regions that affect emotion recognition.

[0136] First, as shown above, the input spectrogram is input data in the form of a two-dimensional matrix that visualizes the temporal variation of speech and the energy distribution by frequency band.

[0137] This spectrogram is divided into fixed-length tokens through a linear projection and provided as input to a Transformer Encoder.

[0138] Each token represents a specific time-frequency patch of the speech signal, and subsequently, the model learns the correlations between each token through Multi-Head Self-Attention operations.

[0139] An attention map is generated from the transformer encoder, which numerically represents which regions (time and frequency intervals) the model focuses high attention on.

[0140] In the present invention, attention values ​​of all layers are rolled out to form a final attention map, and through this map, areas that play a major role in emotion recognition are extracted.

[0141] The Mask (threshold > 0.5) shown in the center right of Fig. 5 represents the result of selectively extracting high-attention regions in the final attention map where the model relies excessively based on a threshold of 0.5.

[0142] That is, only token patches with an attention value of 0.5 or higher are selected and designated as a mask.

[0143] Subsequently, as illustrated below, this mask controls the model so that it does not over-concentrate in a corresponding time-frequency region by subtracting or zeroing out the corresponding time-frequency region from the input spectrogram.

[0144] Through this process, the model becomes able to learn secondary emotional cues that were previously overlooked, such as subtle changes in intonation and tension in the high-frequency range.

[0145] That is, the present invention combines attention-based salient region extraction and selective masking as shown in FIG. 5, thereby mitigating the problem of overfitting to sentence-specific patterns or characteristics specific to a particular dataset, and enabling the learning of generalized emotional expressions even under various speaker and environmental conditions.

[0146] The experimental environment and dataset according to an embodiment of the present invention are described as follows.

[0147] In the present invention, all speech data is normalized to 16 kHz, the length is limited to a maximum of 4 seconds, and zero padding is applied to samples shorter than 4 seconds. Subsequently, a Short-Time Fourier Transform (STFT) is applied to convert the data into a log-Mel spectrogram, using an FFT size N=1024, a hop size H=64, a window size 512, and a Hamming window. Additionally, to improve model generalization performance, data augmentation techniques such as noise addition, time shifting, and speed perturbation are applied. The final log-Mel spectrogram size is 128×1001.

[0148] The Adam optimizer is used to train the DeiT model. The initial learning rate is set to 1e-5, and the learning rate is reduced by 0.5 times every 5 epochs to induce stable convergence. The total training epochs are 25, and the batch size is 4. Cross-entropy loss is used to improve classification performance, and mean squared error (MSE) loss is additionally applied, similar to the Audio Spectrometer Transformer (Gong et al.), to reconstruct the masking region.

[0149] All experiments were conducted under the same learning settings to enable a fair comparison between various attention mechanisms and structural variations. Gradual decay of the learning rate contributes to preventing overfitting and ensuring learning stability even in limited data environments.

[0150] All experiments of the present invention were performed on a Windows Subsystem for Linux 2 (WSL2) Docker container on a Windows workstation. The specifications of the Windows workstation are as follows: CPU: AMD Ryzen 5 7600X 6-Core Processor (4.70 GHz), RAM: 32GB, GPU: Nvidia RTX 4090 GPU (24GB VRAM, CUDA Cores: 16384).

[0151] The development tools used are Visual Studio Code, Pytorch 2.0.1, CUDA 11.8, and cuDNN 8.2.1.

[0152] The CREMA-D dataset consists of 7,442 acting videos of 91 actors (48 men and 43 women) of various races, and is a dataset in which actors perform emotions by using 12 fixed sentences corresponding to 6 emotions: anger, fear, disgust, happiness, neutrality, and sadness.

[0153] The SAVEE dataset consists of a total of 480 English sentences performed by 4 male actors and is classified into 7 emotions: anger, fear, disgust, happiness, neutrality, sadness, and surprise.

[0154] The EmoDB dataset consists of a total of 535 voice data recorded by 10 German voice actors (5 men and 5 women) and includes 7 emotion labels: anger, anxiety, boredom, disgust, happiness, neutrality, and sadness.

[0155] In the present invention, voice data is divided into a training set, a validation set, and a test set in a ratio of 70%:15%:15%, and each is used for training, validation, and performance evaluation of a deep learning model.

[0156] The effect of the speech emotion recognition enhancement method (SAMSG) using self-attention-based mask spectrogram generation and self-supervised learning proposed in this invention on speech emotion recognition accuracy was quantitatively analyzed. By comparing the accuracy based on whether the proposed technique is applied, it was evaluated whether self-attention-based masking contributes to the generalization performance and accuracy improvement of the model. The DeiT model used for comparison receives a log-Mel spectrogram as input and is trained without separate masking.

[0157] In contrast, the proposed SAMSG pre-trains on DeiT-base, masks important regions through attention rollout, pre-trains the masked spectrograms again, and performs fine-tuning on downstream emotion recognition tasks. For objective performance comparison, accuracy and F1-score performance against state-of-the-art methods were measured on three datasets (SAVEE, EmoDB, CREMA-D). All models were trained using root mean square error for reconstruction in self-supervised learning and cross-entropy loss for emotion classification. [Table 1] shows the results of comparing speech emotion recognition accuracy with state-of-the-art methods on the three datasets.

[0158] [Table 1]

[0159]

[0160] In addition, the Macro-averaged F1-score was used to account for class imbalance issues, which is calculated by averaging the harmony of Precision and Recall for all sentiment classes.

[0161] The formula for the F1 score is as follows.

[0162]

[0163] TP_i : True Positives (TP) - Number of samples correctly predicted as class i

[0164] FP_i : False Positives (FP) - The number of actual samples of the other class that were incorrectly predicted as class i.

[0165] FN_i : False Negatives (FN) - Number of actual class i samples incorrectly predicted as another class

[0166] TN_i : True Negatives (TN) - Number of samples that do not belong to class i and are not predicted for class i.

[0167] Macro-F1 is the average of F1-scores calculated individually for each emotion class, allowing for a fair evaluation of model performance even if there is data imbalance between emotion classes.

[0168] Precision_i refers to the proportion of samples that actually belong to a specific sentiment class i among the samples predicted by the model to belong to that class.

[0169] Recall_i represents the proportion of samples (TP_i) that the model correctly predicted as that class out of the total samples (TP_i + FN_i) that actually belong to that class for a specific sentiment class i.

[0170] In relation to, is the harmonic mean of Precision_i and Recall_i, and is a value used to evaluate the balance between the two performance metrics.

[0171] is for each class i It is defined as the average value of, where C represents the total number of emotion classes. Therefore, It is an indicator that can evaluate the performance of each class with equal weight, even if the data distribution of each emotion class is unbalanced.

[0172] [Table 2]

[0173]

[0174] [Table 2] shows the results of comparing speech sentiment recognition macro-averaged F1-scores with the latest methods for three datasets.

[0175] It was confirmed that the speech emotion recognition enhancement method using self-attention-based mask spectrogram generation and self-supervised learning proposed in this invention is effective in improving the generalization performance of speech emotion recognition models. Existing CNN, LSTM, and Transformer-based models have mainly focused on expanding representational capacity or improving performance through multimodal fusion. However, these approaches tend to over-adapt to the repetitive sentence structures of specific datasets or speaker characteristics, which becomes a factor that limits applicability in real-world environments.

[0176] The proposed method utilizes self-attention maps during the training process to identify areas where the model focuses its attention, and masks parts that are unnecessary for emotion recognition or could cause overfitting, such as sentence-specific acoustic features. Through this, the model focuses on less distinct but generalizable emotional signals, resulting in stable emotion recognition performance across various environments. As confirmed by experimental results, the present invention demonstrated stable and superior performance compared to existing methods on datasets containing inherent structural bias, such as SAVEE, EmoDB, and CREMA-D.

Claims

Claim 1 A system for enhancing speech emotion recognition using self-attention-based mask spectrogram generation and self-supervised learning comprises: an attention map collection unit (100) that learns a speech emotion classification model including a multilayer self-attention structure with a spectrogram as input and collects attention maps produced from each attention head and each layer; a mask spectrogram generation unit (200) that accumulates the attention maps collected by the attention map collection unit (100) layer by layer to quantitatively identify time-frequency regions important for emotion recognition and selectively masks the identified regions based on a threshold value to generate a mask spectrogram; and a mask-based learning unit (300) that performs self-supervised learning using the mask spectrogram, predicts an emotion state through a classification head, performs parallel learning to reconstruct the masked regions through a restoration head, and performs final emotion classification by performing additional learning on a downstream emotion recognition task using a token representing the entire input and a single linear layer while fixing the weights of the model learned through self-supervised learning. Voice emotion recognition enhancement system using learning. Claim 2 In claim 1, the attention map collection unit (100) is characterized by receiving a Log-Mel spectrogram as input and collecting an attention map that includes attention values ​​calculated through multi-head self-attention operations and interaction relationships between each input token, thereby enhancing speech emotion recognition using self-attention-based mask spectrogram generation and self-supervised learning. Claim 3 A voice emotion recognition enhancement system using self-attention-based mask spectrogram generation and self-supervised learning, wherein, in claim 1, the mask spectrogram generation unit (200) generates a mask spectrogram by replacing the spectrogram value corresponding to the masking target area with 0. Claim 4 A voice emotion recognition enhancement system using self-attention-based mask spectrogram generation and self-supervised learning, wherein, in claim 1, the mask spectrogram generation unit (200) extracts tokens that are time-frequency domains greater than or equal to a preset threshold from the attention map, and then selectively additionally masks only the corresponding tokens to generate a partially masked spectrogram. Claim 5 delete Claim 6 A method using a system for enhancing speech emotion recognition using self-attention-based mask spectrogram generation and self-supervised learning, comprising: (a) a step in which the system learns a speech emotion classification model including a multilayer self-attention structure with a spectrogram as input, and collects attention maps generated from each attention head and each layer; (b) a step in which the system accumulates the collected attention maps layer by layer to quantitatively identify time-frequency regions important for emotion recognition, and selectively masks the identified regions based on a threshold value to generate a mask spectrogram; (c) a step in which the system performs self-supervised learning using the mask spectrogram and classifies speech emotions based on the results of the self-supervised learning; wherein step (c) includes a step of performing parallel learning to predict an emotion state through a classification head and reconstruct a masked region through a restoration head, and performing final emotion classification by performing additional learning on a downstream emotion recognition task using a token representing the entire input and a single linear layer while fixing the weights of the model learned through self-supervised learning. Method for improving speech emotion recognition using self-supervised learning. Claim 7 In claim 6, the above step (a) is characterized by receiving a Log-Mel spectrogram as input and collecting an attention map containing attention values ​​calculated through multi-head self-attention operations and interaction relationships between each input token. A method for enhancing speech emotion recognition using self-attention-based mask spectrogram generation and self-supervised learning. Claim 8 A method for improving speech emotion recognition using self-attention-based mask spectrogram generation and self-supervised learning, wherein, in claim 6, the above step (b) generates a mask spectrogram by replacing the spectrogram value corresponding to the masking target area with 0. Claim 9 A method for enhancing speech emotion recognition using self-attention-based mask spectrogram generation and self-supervised learning, wherein in claim 6, step (b) extracts tokens that are in a time-frequency domain greater than or equal to a preset threshold from the attention map, and then selectively additionally masks only those tokens to generate a partially masked spectrogram. Claim 10 delete

Images