Emotion estimation device using multi-modal model and learning method thereof

Abstract

An emotion estimation device according to the present invention comprises: a speech emotion expression model processing unit for inputting acquired speaker identity features and acquired speaker speech data features to a prescribed speech emotion expression model and outputting a speech feature query, a speech feature key, and a speech feature value of the speaker through a prescribed process; a text emotion expression model processing unit for outputting a text feature query, a text feature key, and a text feature value of text features from transcription text corresponding to the speaker identity features and the speech data features of the speaker through a prescribed process; and an expression fusion model processing unit for inputting the output speech feature query, speech feature key, and speech feature value of the speaker and the output text feature query, text feature key, and text feature value to a prescribed expression fusion model, and outputting an emotion classification-specific probability of the speaker by applying a cross-attention mechanism.

Description

Emotion estimation device using a multimodal model and its learning method

[0001] The present invention relates to a device for estimating emotions, and more particularly, to a device for estimating emotions by effectively fusing multiple modalities corresponding to a user's voice information and text information transcribed from the voice, a method therefor, and a learning method therefor.

[0002] The present invention is the result of the project on the development of a strategic emotional-emotional evaluation model for non-face-to-face environment users and commercialization of two-way digital content application (project number 1711195712) of the Ministry of Science and ICT's realistic content core technology development project.

[0003] Emotions are a crucial tool for human expression. Expressing emotions not only enriches communication but also enhances the meaning of human existence. Humans express their emotions through facial expressions, gestures, voice, and language. Furthermore, emotions can be expressed and recognized primarily through communication processes. Because we typically communicate through conversation, many studies attempting to recognize emotions in conversation primarily utilize voice or text data. Recognizing emotions in conversation is a crucial and challenging task in applications such as natural human-machine interaction, intelligent educational tutoring, and mental health analysis. In daily life, people engage in multi-turn conversations that convey their emotional states in various ways. This occurs through both verbal and nonverbal means (e.g., facial expressions and body language), and each modality carries emotional information. Methods for efficiently and effectively integrating these diverse inputs are a key research focus.

[0004] In everyday life, text modality is often used in conjunction with audio modality. The emotional content embedded in text features is complemented by various information, such as pitch, loudness, and other frequency-related metrics. Effective interaction between text and audio data can provide a more comprehensive context and provide more emotional information. However, effectively integrating various modalities remains a significant challenge. While many previous single-modality emotion recognition techniques have been developed, multimodal emotion recognition still presents a key challenge. Different modalities rely on independent preprocessing and feature extraction designs due to their heterogeneous spatial distributions. To create a model that is applicable and generalizable to both individual modalities and fused models, intra- and cross-modal interactions must be learned to reveal distinct emotional content.

[0005] Most prior emotion recognition research has focused on unimodal emotion recognition, such as audio / speech, facial expressions, and text. Despite significant progress in unimodal emotion recognition, unimodal emotion recognition suffers from the following challenges:

[0006] First, a single-modality emotion model cannot accurately characterize people's emotional states. This may be because people express emotions in a variety of ways.

[0007] Second, because multimodal data from different modalities may be interrelated or complementary in the ultimate emotion recognition task, single-modal data cannot contain complete emotional semantic information. To address this issue, researchers have been working to extend relatively simple unimodal emotion recognition to complex multimodal emotion recognition (MER). Multimodal emotion recognition has the advantage of leveraging knowledge from diverse sources.

[0008] Compared to single-modality systems, emotion recognition is related to human emotional expression. In particular, human language primarily consists of auditory and textual expressions. In this case, audio and textual modalities can provide crucial emotional information for identifying human emotional states in a complementary manner. Therefore, integrating audio and textual cues is known to have great potential for building effective multimodal emotion recognition models.

[0009] However, the highly heterogeneous nature of multimodal data leads to significant discrepancies between audio and text data, posing significant challenges when fusion of audio and text modalities in multimodal emotion recognition processes. Therefore, effective multimodal data fusion strategies that integrate heterogeneous audio and text data are crucial for fundamental multimodal emotion recognition systems. To alleviate this issue in early multimodal emotion recognition research, three typical modal data fusion approaches, such as multi-feature-level fusion (e.g., early fusion), decision-level fusion (e.g., late fusion), and model-level fusion, were widely used to fuse audio and textual cues.

[0010] Feature-level fusion is the simplest fusion method, directly concatenating different features extracted from multimodal modalities into a long feature vector that serves as input to the latter's classifier. However, this simple feature concatenation has the disadvantage of not accounting for inter-modal dynamics over time. Furthermore, feature-level fusion can easily suffer from the curse of dimensionality due to the high-dimensional concatenated feature vectors.

[0011] In contrast, decision-level fusion aims to model different modalities independently, using specific calculation rules such as average and maximum. Nevertheless, decision-level fusion fails to accurately reflect the interrelationships between modalities.

[0012] Model-level fusion aims to explicitly exploit correlations between different modalities and thus represents a compromise between feature-level and decision-level fusion methods. Representative model-level fusion methods include multi-kernel learning and deep learning techniques. Recently, various attention-based deep learning models can be broadly categorized into two groups: inter-modal attention-based methods and intra-modal attention-based methods, which have been applied to multimodal emotion recognition.

[0013] Cross-modal attention-based methods have been criticized for focusing on the dynamic interactions between different modalities and ignoring the inherent connections between feature elements within a single modality. Conversely, intra-modal attention-based methods focus on extracting emotionally salient feature representations within a single modality, thereby neglecting the relationships between different modalities. In this sense, intra-modal relationships within a single modality and inter-modal relationships between different modalities can enhance each other to some extent. However, most prior studies have not sufficiently considered inter-modal and intra-modal attention mechanisms, making it difficult to expect potential performance enhancements in multimodal emotion recognition by jointly learning intra- and inter-modal emotional salience information.

[0014] Feature extraction and emotion classification are two key steps for single-model emotion recognition. Feature extraction involves extracting feature representations associated with emotional expressions from a correlated single model. Emotion classification aims to learn the mapping relationship between the extracted feature representations and emotion labels, and then use an appropriate classifier to obtain the final emotion recognition result. Existing machine learning methods such as hidden Markov models (HMMs), support vector machines (SVMs), Bayesian networks (BNs), k-nearest neighbors (KNNs), multilayer perceptrons (MLPs), and decision trees have been widely used as emotion classifiers for single-mode emotion recognition.

[0015] Extracting emotional acoustic features is crucial for audio emotion recognition. Early, commonly used acoustic features were hand-crafted acoustic low-level descriptors (LLDs), such as prosodic features, voice quality features, and spectral features. Common prosodic features include pitch, loudness, and duration. Common voice quality features include formants, spectral energy distribution, glottal features, and voice source parameters. A representative spectral feature is Mel-Frequency Cepstral Coefficients (MFCCs). While these hand-crafted acoustic LLDs have shown promising performance in audio emotion recognition tasks, their low quality still leaves a significant emotional gap between hand-crafted features and subjective emotions.

[0016] Recently developed deep learning techniques are widely applied to audio emotion recognition due to their powerful feature learning capabilities. Commonly used deep learning methods for audio emotion recognition include deep convolutional neural networks (CNNs), recurrent neural networks (RNNs), and their variants, such as long-short-term memory (LSTMs).

[0017] Text sentiment recognition is a technology that automatically recognizes emotional states in text expressions using text features. Early feature extraction methods commonly used in text sentiment recognition research included handcrafted bag-of-words (BoW) models, latent Dirichlet allocation (LDA), and latent semantic analysis (LSA). However, these handcrafted models have limitations in capturing the high-level semantic information hidden in text data. To alleviate the aforementioned issues, deep learning techniques have been employed in recent years to improve the performance of text sentiment recognition. Several representative pre-trained word embedding methods, such as Word2vec and GloVe, have also been used for text sentiment analysis. Recently, various pre-trained language models in natural language processing (NLP), such as contextualized word vectors (CoVe), embeddings from language models (ELMo), and bidirectional encoder representations from transformers (BERT), have been adopted for text sentiment classification.

[0018] This technology aims to develop a multimodal emotion recognition model that simultaneously utilizes voice and text data in this regard.

[0019] The technical problem to be achieved in the present invention is to provide a device capable of estimating a user's emotions.

[0020] Another technical task to be achieved in the present invention is to provide a method for estimating a user's emotions.

[0021] Another technical problem to be achieved by the present invention is to provide a computer-readable recording medium or a computer-readable recording medium having recorded thereon a computer program for performing a method for estimating user emotions.

[0022] Another technical task to be achieved in the present invention is to provide a learning method for a device capable of estimating a user's emotions.

[0023] Another technical problem to be achieved in the present invention is to provide a computer-readable recording medium or a computer-readable recording medium having recorded thereon a computer program for performing a learning method for outputting an emotion classification probability of a device capable of estimating an emotion to a user.

[0024] The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0025] In order to achieve the above technical task, a device capable of estimating a user's emotion according to the present invention comprises: a speech emotion representation model processing unit that inputs acquired speaker identity features and acquired speaker's voice data features into a predetermined speech emotion representation (SER) model, encodes the speaker identity features and the speaker's voice data features through a speech encoder in the speech emotion representation model, and outputs the speaker's voice feature query, voice feature key, and voice feature value by encoding the encoded speaker identity features and the speaker's voice data features through a disentangled feature encoder; a text emotion representation model processing unit that inputs text features acquired from a transcription text corresponding to the speaker identity features and the speaker's voice data features into a predetermined text emotion representation (TER) model, encodes the text features through a text encoder in the text emotion representation model, and outputs the text feature query, text feature key, and text feature value of the text features; and The method may include a representation fusion model processing unit that inputs a voice feature query, a voice feature key, and a voice feature value and the output text feature query, text feature key, and text feature value into a predetermined representation fusion (RF) model and applies a cross attention mechanism to output a probability for each emotion classification of the speaker.

[0026] In the above-mentioned voice emotion expression model processing unit, the speaker identity features and the speaker's voice data features encoded by the voice encoder may correspond to the encoded voice expression before downsampling is applied.

[0027] In the above expression fusion model processing unit, the cross-attention mechanism may include at least one of a first cross-attention mechanism that applies the acquired voice feature query, text feature key, and text feature value, and a second cross-attention mechanism that applies the acquired text feature query, voice feature key, and voice feature value.

[0028] The above expression fusion model processing unit can output to the user the emotion classification with the highest probability among the probability output values ​​for each emotion classification of the speaker.

[0029] In the above voice emotion expression model processing unit, the speaker's voice data features may be wav2vec features or wav2vec2.0 features extracted from a predetermined voice recognition pre-learning model.

[0030] In order to achieve the above technical task, a method for estimating a user's emotion according to the present invention comprises the steps of: inputting acquired speaker identity features and acquired speaker's voice data features into a predetermined speech emotion representation (SER) model; encoding the speaker identity features and the speaker's voice data features through a voice encoder in the speech emotion representation model; encoding the encoded speaker identity features and the speaker's voice data features through a disentangled feature encoder to output the speaker's voice feature query, voice feature key, and voice feature value; inputting text features acquired from a transcription text corresponding to the speaker identity features and the speaker's voice data features into a predetermined text emotion representation (TER) model; encoding the text features in the text emotion representation model through a text encoder to output the text feature query, text feature key, and text feature value of the text feature; and outputting the speaker's voice feature query, voice feature key, and voice feature value. The step of inputting the above-mentioned output text feature query, text feature key, and text feature value into a predetermined representation fusion (RF) model and applying a cross attention mechanism to output the probability for each emotion classification of the speaker may be included.

[0031] In the step of encoding the speaker identity features and the voice data features through a voice encoder, the speaker identity features and the speaker's voice data features encoded by the voice encoder may correspond to an encoded voice expression before downsampling is applied.

[0032] The step of outputting the probability by emotion classification of the speaker may include a step of applying at least one of a first cross-attention mechanism applying the acquired voice feature query, text feature key, and text feature value, and a second cross-attention mechanism applying the acquired text feature query, voice feature key, and voice feature value.

[0033] The step of outputting the probability of each emotion classification of the speaker may include a step of outputting to the user the emotion classification with the highest probability among the probability output values ​​of each emotion classification of the speaker.

[0034] In order to achieve the above technical task, a learning method of a device capable of estimating a user's emotion according to the present invention may include a step of inputting acquired speaker identity features, acquired speaker's voice data features, acquired phonetic features, and acquired Mel Spectrogram into a predetermined voice emotion expression model, a step of encoding the speaker identity features and the voice data features in the voice emotion expression model with a voice encoder, a step of encoding the acquired phonetic features in the voice emotion expression model with a phonetic feature encoder, a step of decoding the speaker identity features, the encoded speaker identity features and voice data features, and the encoded phonetic features in the voice emotion expression model with a voice decoder, and a step of converting the output of the voice decoder into a Mel Spectrogram and learning by designating the acquired Mel Spectrogram as a reference truth label.

[0035] The step of encoding with the above voice encoder may include a step of downsampling a voice expression including the acquired speaker identity features and the acquired speaker's voice data features, and the step of decoding with the above voice decoder may include a step of upsampling the downsampled voice expression.

[0036] The step of decoding with the above voice decoder may include a step of concatenating at least one of the speaker identity feature and the encoded voice feature with a result of upsampling the speaker identity feature and the voice data feature.

[0037] In order to achieve the above technical task, a learning method of a device capable of estimating a user's emotion according to the present invention comprises the steps of: inputting acquired speaker identity features and acquired speaker's voice data features into a predetermined speech emotion representation (SER) model; encoding the speaker identity features and the voice data features using a voice encoder in the speech emotion representation model; encoding the encoded speaker identity features and the voice data features using a disentangled feature encoder to output a voice feature query, a voice feature key, and a voice feature value of the speaker; inputting text features acquired from a transcribed text corresponding to the speaker identity features and the speaker's voice data features into a predetermined text emotion representation (TER) model; encoding the text features in the text emotion representation model using a text encoder to output a text feature query, a text feature key, and a text feature value of the text feature; and encoding the output speaker's voice feature query, voice feature key, and voice feature value and the output text. The method may include a step of inputting a feature query, a text feature key, and a text feature value into a predetermined representation fusing (RF) model, applying a cross attention mechanism to output the probability of each emotion classification of the speaker, and a step of learning the output probability of each emotion classification of the speaker as the probability of each emotion classification designated as a reference truth label.

[0038] It may include a step of training with the emotion classification having the highest probability among the emotion classification probabilities of the output speaker output through the above-described representation fusion (RF) model and the emotion classification designated as the reference truth label.

[0039] In the step of encoding the acquired speaker identity features and acquired voice data features through a voice encoder, the speaker identity features and voice data features encoded by the voice encoder may be encoded voice expressions before downsampling is applied.

[0040] The step of outputting the probability by emotion classification of the speaker may include a step of applying at least one of a first cross-attention mechanism applying the acquired voice feature query, text feature key, and text feature value, and a second cross-attention mechanism applying the acquired text feature query, voice feature key, and voice feature value.

[0041] In order to achieve the above technical task, a learning performing device for estimating a user's emotion according to the present invention may include a voice emotion expression model processing unit that inputs acquired speaker identity features, acquired speaker's voice data features, acquired phonetic features, and acquired Mel Spectrogram into a predetermined voice emotion expression model, encodes the speaker identity features and the voice data features in the voice emotion expression model with a voice encoder, encodes the acquired phonetic features in the voice emotion expression model with a phonetic feature encoder, decodes the speaker identity features, the encoded speaker identity features and voice data features, and the encoded phonetic features in the voice emotion expression model with a voice decoder, and converts the output of the voice decoder into a Mel Spectrogram for learning by designating the acquired Mel Spectrogram as a reference truth label.

[0042] The above voice encoder can perform downsampling after the encoding, and the above voice decoder can perform upsampling before the decoding on the encoded speaker identity features and voice data features.

[0043] In order to achieve the above technical task, a learning performing device for estimating a user's emotion according to the present invention comprises: a speech emotion representation model processing unit that inputs acquired speaker identity features and acquired speaker's voice data features into a predetermined speech emotion representation (SER) model, encodes the speaker identity features and the voice data features using a voice encoder in the speech emotion representation model, and encodes the encoded speaker identity features and the voice data features using a disentangled feature encoder to output a voice feature query, a voice feature key, and a voice feature value of the speaker; a text emotion representation model processing unit that inputs text features acquired from a transcribed text corresponding to the speaker identity features and the speaker's voice data features into a predetermined text emotion representation (TER) model, encodes the text features using a text encoder in the text emotion representation model, and outputs a text feature query, a text feature key, and a text feature value of the text features; And the output speaker's voice feature query, voice feature key and voice feature value and the output text feature query, text feature key and text feature value may be input into a predetermined representation fusing (RF) model, and a cross attention mechanism may be applied to output the probability of the speaker's emotion classification, and the expression fusion model processing unit may be included to learn the output speaker's emotion classification probability as the emotion classification probability designated as the reference truth label.

[0044] The above expression fusion model processing unit can learn the emotion classification with the highest probability among the emotion classification probabilities of the speaker output through the above-described expression fusion model and the emotion classification designated as the reference truth label.

[0045] The effects of the device, method, and learning method of the emotion estimation device according to one embodiment of the present invention are as follows.

[0046] The multimodal emotion recognition proposed in this invention enhances the accuracy of emotion analysis by combining various data sources. For example, it simultaneously analyzes voice tone and language content, providing a more accurate understanding of emotions.

[0047] Furthermore, according to one embodiment of the present invention, it is possible to capture subtle emotional nuances that are difficult to detect with a single modality, which makes it easy to understand the user's emotions, especially in complex cases where there are delicate emotional states or conflicting emotional signals.

[0048] Furthermore, the multimodal system according to the present invention better understands the context of a situation by integrating information provided from various modalities. This has the advantage of making emotion recognition more accurate and in-depth, thereby enhancing contextual understanding.

[0049] Furthermore, the multimodal approach according to the present invention can maintain or improve the accuracy of emotion recognition even in situations where a single modality may fail. For example, when speech data is unclear, it can be supplemented with text data.

[0050] Furthermore, the multimodal emotion recognition according to the present invention can be applied to a wide range of applications, including healthcare, education, customer service, and entertainment. This enhances the user experience and makes human-machine interaction more natural.

[0051] Furthermore, multimodal emotion recognition according to the present invention can help us better understand how people interact and communicate. This provides valuable insights into the study of human behavior.

[0052] The effects that can be obtained from the invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0053] The accompanying drawings, which are included as part of the detailed description to aid in understanding the present invention, provide embodiments of the present invention and, together with the detailed description, explain the technical idea of ​​the present invention.

[0054] Figure 1 is a diagram illustrating the layer structure of an artificial neural network.

[0055] Figure 2 is a diagram illustrating an example of a deep neural network.

[0056] FIG. 3 is a block diagram illustrating the function of a device capable of estimating a user's emotions according to the present invention.

[0057] FIG. 4 is a diagram schematically illustrating the probability of each emotion classification, which is an output result of a device capable of estimating a user's emotion according to one embodiment of the present invention.

[0058] FIG. 5 is a diagram schematically illustrating a method for estimating a user's emotions and a method for learning to estimate emotions according to one embodiment of the present invention.

[0059] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The detailed description set forth below, together with the accompanying drawings, is intended to illustrate exemplary embodiments of the present invention and is not intended to represent the only embodiments in which the present invention may be practiced. The following detailed description includes specific details to provide a thorough understanding of the present invention. However, one of ordinary skill in the art will appreciate that the present invention may be practiced without these specific details.

[0060] The present invention is susceptible to various modifications and embodiments. Specific embodiments are illustrated and described in detail in the drawings. However, this is not intended to limit the present invention to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present invention.

[0061] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0062] Terms such as first, second, etc. may be used to describe various components, but these components should not be limited by these terms. These terms are used solely to distinguish one component from another.

[0063] The terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, it should be understood that the terms "comprises" or "has" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0064] Additionally, terms such as “part,” “unit,” “module,” and “device” described in the specification mean a unit that processes at least one function or operation, which may be implemented by hardware, software, or a combination of hardware and software.

[0065] In some cases, to avoid obscuring the concepts of the present invention, well-known structures and devices may be omitted or illustrated in block diagram form focusing on the core functions of each structure and device. Furthermore, the same components are described using the same reference numerals throughout this specification.

[0066] The present invention proposes a device, method, and learning method for estimating user emotions using multimodal data of different modalities, corresponding to speech and text. The proposed algorithm model effectively fuses the acquired speech and text modalities to more accurately estimate user emotions.

[0067] Before explaining the present invention, let's explain artificial intelligence (AI), machine learning, and deep learning. The easiest way to understand the relationship between these three concepts is to imagine three concentric circles. AI is the largest circle, followed by machine learning, and deep learning, which is driving the current AI boom, is the smallest circle.

[0068] The concept of artificial intelligence first emerged in 1956 at the Dartmouth Conference hosted by Professor John McCarthy at Dartmouth College, and has experienced explosive growth in recent years. This growth has been particularly accelerated since 2015 by the introduction of GPUs, which offer fast and powerful parallel processing capabilities. The advent of the big data era, with its explosive growth in storage capacity and the resulting overflow of data across all domains, including images, text, and mapping data, has also significantly influenced this growth.

[0069] Artificial Intelligence - Implementing human intelligence in machines

[0070] Back in 1956, the pioneers of artificial intelligence dreamed of ultimately creating a complex computer with characteristics similar to human intelligence. While this type of AI possesses human senses, reasoning, and human-like thinking, it's called "general AI." However, the AI ​​currently achievable with technological advancement falls under the umbrella of "narrow AI." Narrow AI is characterized by its ability to perform specific tasks, such as image classification services for social media or facial recognition, with superior human capabilities.

[0071] Machine Learning - A Specific Approach to Implementing Artificial Intelligence

[0072] Machine learning automatically filters spam from inboxes. Meanwhile, machine learning fundamentally uses algorithms to analyze data, learn from the analysis, and then make judgments or predictions based on what it learns. Therefore, the ultimate goal is not to directly code specific decision-making guidelines into software, but to "learn" the computer itself through massive amounts of data and algorithms, allowing it to learn how to perform tasks. Machine learning originated from concepts pioneered by early AI researchers, and algorithmic approaches include decision tree learning, inductive logic programming, clustering, reinforcement learning, and Bayesian networks. However, none of these achieved the ultimate goal of general AI, and it's true that early machine learning approaches often struggled to achieve even narrow AI.

[0073] While machine learning has achieved significant results in fields like computer vision, it faces the limitation of requiring a certain amount of coding throughout the entire process of implementing artificial intelligence, even without specific guidelines. For example, when recognizing an image of a stop sign using a machine learning system, developers must manually code edge detection filters that programmatically identify the object's start and end, shape detection to identify the object's faces, and classifiers that recognize characters like "STO-P." In this way, machine learning recognizes images from "coded" classifiers and "learns" about stop signs through algorithms.

[0074] Machine learning's learning method adjusts model parameters to minimize the error between the target value and the predicted value, thereby finding the most appropriate model. Here, the predicted value refers to the value produced when the input value is input to the model, i.e., the output value. For example, a model composed of an arbitrary number of convolutional layers, bidirectional LSTMs, and feedforward layers will, as learning progresses, change each convolutional layer, bidirectional LSTM, and feedforward layer to create a model with the smallest error from the target value.

[0075] While machine learning's image recognition rate is sufficient for commercial use, it can sometimes be poor in certain situations, such as fog or trees obscuring signs. Until recently, computer vision and image recognition have struggled to reach human-level performance due to these recognition rate issues and frequent errors.

[0076] Deep Learning - The Technology That Realizes Full Machine Learning

[0077] Artificial neural networks, another algorithm developed by early machine learning researchers, were inspired by the biological properties of the human brain, specifically the interconnected structure of its neurons. However, unlike the brain, where any physically adjacent neurons can be interconnected, artificial neural networks have a fixed layer connection and data propagation direction.

[0078] For example, if an image is sliced ​​into numerous tiles and input into the first layer of a neural network, the neurons in that tile will repeatedly pass the data on to the next layer until the final output is generated in the last layer. Each neuron is then assigned a weight representing the accuracy of the input based on the task it performs, and the final output is then determined by adding all the weights together. In the case of a stop sign, the characteristics of the image, such as the octagonal shape, the red color, the characters displayed, the size, and the presence of movement, are sliced ​​and "examined" by the neurons, and the neural network's task is to identify whether it is a stop sign. Here, a "probability vector" is utilized, which predicts the outcome based on the weights based on sufficient data.

[0079] Deep learning is a form of artificial intelligence developed from artificial neural networks. It learns data by utilizing information input / output layers similar to neurons in the brain. However, even basic neural networks require enormous computational power, hindering the commercialization of deep learning from the beginning. Despite this, researchers continued their research and, using supercomputers, successfully parallelized algorithms that proved the concept of deep learning. The advent of GPUs, optimized for parallel computing, dramatically accelerated neural network computation, ushering in the emergence of true deep learning-based artificial intelligence.

[0080] Neural networks are likely to make numerous errors during the "learning" process. Returning to the stop sign example, adjusting the weights of neuron inputs precisely enough to consistently produce the correct answer regardless of weather conditions or day / night changes might require learning hundreds, thousands, or even millions of images. Only when this level of accuracy is achieved can the neural network be considered to have properly learned the stop sign classification. In 2012, Google and Stanford University Professor Andrew Ng implemented a "deep neural network" consisting of over a billion neural networks on 16,000 computers. Using this network, they analyzed 10 million images from YouTube and successfully taught the computer to classify photos of people and cats. They taught the computer to recognize and judge the shape and appearance of cats in the videos on its own.

[0081] Systems trained with deep learning already have image recognition capabilities that surpass those of humans. Deep learning also encompasses the ability to identify cancer cells in blood and tumors in MRI scans. Google's AlphaGo learned the fundamentals of Go and further strengthened its neural network through repeated matches against AI systems like itself. The advent of deep learning has enhanced the practicality of machine learning and expanded the scope of artificial intelligence. Deep learning subdivides tasks into every possible way a computer system can support. Technologies based on deep learning, such as driverless cars, better preventative medicine, and more accurate movie recommendations, are already in use or are on the verge of becoming a reality in our daily lives. Deep learning is considered both the present and the future of artificial intelligence, with the potential to realize general AI, once a fantasy of science fiction.

[0082] Below, we will look at deep learning in more detail.

[0083] Deep learning is a type of artificial neural network (ANN) that utilizes the theory of the human neural network (Neural Network). It is a set of machine learning models or algorithms that refer to a deep neural network (DNN) that is structured in a layer structure and has one or more hidden layers (hereinafter referred to as intermediate layers) between the input layer and the output layer. Simply put, deep learning can be said to be an artificial neural network with a deep layer.

[0084] The human brain is estimated to be composed of 25 billion nerve cells. The brain is made up of nerve cells, and each nerve cell (neuron) refers to a single nerve cell that forms a neural network. A nerve cell contains a cell body, an axon (or nucleus), and usually multiple dendrites (or protoplasmic processes). Information is transmitted between these nerve cells through synapses, the connections between nerve cells. While a single nerve cell appears very simple, when these nerve cells come together, they are capable of human intelligence. Dendrites are the part that receives signals from other nerve cells (input), and the axon is the very long part that extends from the cell body and transmits signals to other nerve cells (output). Synapses, the connections between axons and dendrites that transmit signals between nerve cells, do not transmit signals unconditionally. Instead, they only transmit signals if the signal strength exceeds a certain value (threshold). In other words, not only does each synapse have a different connection strength, but it also determines whether or not a signal will be transmitted.

[0085] Artificial neural networks (ANNs), a branch of artificial intelligence, are mathematical models modeled after the structure of the biological (typically human) brain. In other words, ANNs mimic the information processing and transmission processes of biological neurons. Similar to how the human brain solves problems, ANNs exhibit excellent parallelism because each neuron operates independently. Furthermore, because information is distributed across numerous connections, problems in a few neurons do not significantly impact the overall system. Consequently, ANNs are robust to a certain level of error and possess the ability to learn from a given environment.

[0086] Deep neural networks (DNNs) can be considered descendants of artificial neural networks (ANNs). They transcend existing limitations and achieve success in areas where numerous AI technologies have failed in the past. They are the latest version of ANNs. Looking at the modeling of ANNs based on biological neural networks, the processing units are modeled as nodes, and the connections are modeled as synapses, which are weights, as shown in Table 1.

[0087] Biological neural network Artificial neural network Cell body Node Dendrite Input Axon Output Synapse Weight

[0088] Figure 1 illustrates the layer structure of an artificial neural network. Just as human biological neurons are interconnected in multiple layers to perform meaningful tasks, individual neurons in an artificial neural network are interconnected through synapses. Multiple layers are interconnected, and the connection strengths between each layer can be updated using weights. This multilayer structure and connection strengths are utilized in fields such as learning and cognition.

[0089] Each node is connected by weighted links, and the entire model learns by repeatedly adjusting the weights. Weights are the basic means of long-term memory and express the importance of each node. Simply put, an artificial neural network trains the entire model by initializing these weights and updating and adjusting them with the training data set. After training is complete, when a new input value is received, the appropriate output value is inferred. The learning principle of an artificial neural network can be viewed as a process in which intelligence is formed through the generalization of experience and is performed in a bottom-up manner. In Figure 1, when there are two or more intermediate layers (i.e., 5 to 10), it is considered deep and is called a deep neural network. Learning and inference models achieved through such a deep neural network can be referred to as deep learning.

[0090] While artificial neural networks can perform to some extent with a single intermediate layer (commonly referred to as a hidden layer) beyond the input and output layers, as the problem complexity increases, the number of nodes or layers must be increased. While increasing the number of layers to achieve a multilayered model is effective, its application is limited due to the impossibility of efficient learning and the large computational load required to train the network.

[0091] However, by overcoming these limitations, artificial neural networks have been able to achieve deep structures. This has enabled the construction of complex, highly expressive models, leading to groundbreaking results in diverse fields such as speech recognition, facial recognition, object recognition, and character recognition. Figure 2 illustrates an example of a deep neural network.

[0092] A deep neural network (DNN) is an artificial neural network (ANN) with multiple hidden layers between the input layer and the output layer. It is a collection of machine learning models or algorithms that refer to deep neural networks (DNNs) with one or more hidden layers between the input layer and the output layer. The connections in the neural network are made from the input layer to the hidden layer, and from the hidden layer to the output layer.

[0093] Deep neural networks, like typical artificial neural networks, can model complex non-linear relationships. For example, in a deep neural network architecture for object recognition, each object can be represented as a hierarchical structure of basic image elements. Additional layers can then gradually aggregate features from lower layers. This characteristic of deep neural networks allows them to model complex data with a smaller number of units (nodes) compared to similarly implemented artificial neural networks.

[0094] While previous deep neural networks were typically designed as feed-forward neural networks, recent research has successfully applied deep learning structures to recurrent neural networks (RNNs). For example, deep neural network architectures have been applied to language modeling. Convolutional neural networks (CNNs) have been successfully applied to computer vision, with each successful application well documented. More recently, CNNs have been applied to acoustic modeling for Automatic Speech Recognition (ASR), and are considered more successful than existing models. Deep neural networks can be trained using the standard error backpropagation algorithm, where weights are updated using stochastic gradient descent using equations.

[0095] Various signals from the surrounding environment that humans receive through their sense organs can be expressed through computers in the form of text, audio, images, and videos and stored as data in the computer's internal storage devices.

[0096] High-dimensional data corresponding to the text, audio, images and videos stored in a computer are data consisting of a combination of continuous '0' and '1' from a low-dimensional perspective, and from a slightly higher-dimensional perspective of a computer program, they are various structures, objects or class instances defined in the programming language used by each program.

[0097] In order for artificial intelligence technology to learn, it is necessary to extract features, which are data that can effectively express high-dimensional data such as text, audio, images, and videos that humans can accept through computers. In addition, the implementation methods and terms for this feature data are different in various artificial intelligence models and various programming languages ​​that can implement artificial intelligence models.

[0098] The present invention is not limited to a specific conventional artificial intelligence model or a specific programming language capable of implementing an artificial intelligence model, and data capable of effectively representing the high-dimensional data are collectively referred to as feature vectors or expressions, and in particular, in the case of voice-related data, they can be referred to as voice expressions.

[0099] FIG. 3 is a block diagram illustrating the function of a device capable of estimating a user's emotions according to the present invention.

[0100] Referring to FIG. 3, a device (300) capable of estimating a user's emotions may include a processor (310) and a memory (320). In the present invention, the processor (310) may include a voice emotion expression model processing unit (330), a text emotion expression model processing unit (340), and an expression fusion model processing unit (350), and may perform various necessary operations, such as storing intermediate or final result output information generated by each processing unit (340, 350) through a series of processing processes in the memory (320).

[0101] In the present invention, the memory (320) may store information on a voice emotion expression model, information on a text emotion expression model, information on an expression fusion model, and information necessary for the classifier and processor (310) to estimate emotions. The memory (320) may store information on intermediate or final results produced by a voice emotion expression model processing unit (330), a text emotion expression model processing unit (340), and an expression fusion model processing unit (350) through a series of processing steps.

[0102] In the present invention, the voice emotion expression model processing unit (330) is a component that processes the voice emotion expression model stored in the memory (320) on the processor (310), and can input speaker identity features and the speaker's voice data features into a predetermined voice emotion expression model stored in the memory (320).

[0103] Speaker identity features and speaker voice data features are key data that contain information that can determine emotions from a typical speaker's voice signal, and in the present invention, when determining the speaker's emotions, data that directly influences emotions (e.g., speaker identity features and voice data features) and data that do not (e.g., phonetic features) are separated, and only data that directly influences emotions are selected and input into the voice encoder (331) of the voice emotion expression model, thereby improving the accuracy of estimating the user's emotions.

[0104] The voice emotion expression model processing unit (330) encodes the acquired speaker identity features and voice data features in the voice emotion expression model through a voice encoder (331), and can encode data before downsampling of the voice encoder (331) that generally includes a downsampling layer (Downsampler) through a disentangled feature encoder (333).

[0105] In artificial neural network models, which are generally composed of an encoder and a decoder, a bottleneck refers to a performance limitation of the neural network. A bottleneck can occur in the encoder or decoder during the model's training or inference process. An encoder bottleneck occurs when the encoder does not adequately represent the input sequence during the downsampling process, resulting in insufficient information being transmitted to the decoder, which can limit its performance. A decoder bottleneck occurs when the decoder does not properly interpret the encoded information during the upsampling process, resulting in inaccurate final output.

[0106] In the present invention, the voice emotion expression model processing unit (330) operates to avoid a bottleneck caused by downsampling of the voice encoder (331) by using data before downsampling of the voice encoder (331), thereby fully reflecting the emotion information included in the speaker's voice signal during the inference process.

[0107] In the field of artificial intelligence and machine learning, disentangled and entangled states are terms that describe the properties of features or representations in artificial intelligence and machine learning.

[0108] Disentangled features or representations mean that each dimension is independent and separate from the others. Because each dimension represents a different aspect of the system and does not interfere with one another, a model with disentangled features reflects only its specific properties without affecting other dimensions. Because each feature changes independently and without affecting the others, this state of clear separation across the entire feature space is beneficial for data representation and understanding. It allows for a clearer understanding of inter-feature relationships and enhances model performance by enabling the model to generalize more robustly to new environments or data. Furthermore, the separation of features increases adaptability to various situations, and the model's robustness to noise is enhanced by reducing the impact of noise on other features due to the separation of features. Furthermore, because features are learned independently, data utilization is efficient, allowing for the extraction of useful features from even small amounts of data.

[0109] When features or representations are entangled, it refers to a state in which each dimension interacts with or is highly dependent on the other. In other words, changes in one dimension affect other dimensions, and specific attributes may appear intertwined. Therefore, in models with entangled features, each dimension can be organically interconnected. Furthermore, because it's difficult to identify relationships between features, and generalization ability, robustness to noise, and data efficiency are low, the fields of artificial intelligence and machine learning favor disentangled representations and are researching methods to achieve this state.

[0110] In the present invention, the voice emotion expression model processing unit (330) can output a voice expression combined with speaker identity features and voice data features input to the voice emotion expression model in an unlocked state by encoding the output of the voice encoder (331) through the unlocked feature encoder (333), as a voice feature query, a voice feature key, and a voice feature value. The voice data features encoded through the unlocked feature encoder (333) can exclude data that are not related to emotion from the speaker identity features and voice data features, and select only data that are related to emotion. By selecting only data related to such emotion through the unlocked feature encoder (333), the model of the present invention can be expected to have the aforementioned effects of strengthening generalization ability, robustness to noise, and data efficiency when inferring the user's emotion thereafter.

[0111] In the present invention, the text emotion expression model processing unit (340) inputs text features obtained from a transcription text corresponding to speaker identity features and speaker voice data features into a predetermined text emotion expression model stored in a memory (320), and outputs the text features as a text feature query, a text feature key, and a text feature value through a text encoder (341).

[0112] In the present invention, the expression fusion model processing unit (350) can input the voice feature query, voice feature key and voice feature value output from the voice emotion expression model processing unit (330), and the text feature query, text feature key and text feature value output from the text emotion expression model processing unit (340) into a predetermined expression fusion model stored in the memory (320).

[0113] The expression fusion model processing unit (350) may apply a cross-attention mechanism in the expression fusion model as a suitable embodiment of the present invention. Here, the cross-attention mechanism may include at least one of a first cross-attention mechanism that applies a speech feature query, a text feature key, and a text feature value, and a second cross-attention mechanism that applies a text feature query, a speech feature key, and a speech feature value. FIG. 5 illustrates a configuration in which both the first cross-attention mechanism and the second cross-attention mechanism are applied to the expression fusion model.

[0114] The expression fusion model processing unit (350) can perform a function of outputting the probability for each emotion classification through a predetermined classifier (353) stored in the memory (320) for the results output through the expression fusion model. The expression fusion model processing unit (350) can output to the user the emotion classification with the highest probability among the probability values ​​for each emotion classification output through the classifier (353).

[0115] The expression fusion model processing unit (350) can perform learning by comparing the probability values ​​for each emotion classification output through the classifier (353) with the probability values ​​for each emotion classification designated as the standard truth label.

[0116] The voice emotion expression model processing unit (330) can input the acquired speaker identity features, the acquired speaker's voice data features, the acquired phonetic features, and the acquired Mel spectrogram into a predetermined voice emotion expression model stored in the memory (320). Here, the acquired speaker identity features, the acquired speaker's voice data features, the acquired phonetic features, and the acquired Mel spectrogram can be acquired from the same speech data of the same speaker.

[0117] The voice emotion expression model processing unit (330) can encode the acquired speaker identity features and voice data features in the voice emotion expression model through a voice encoder (331), and can encode the acquired voice features in the voice emotion expression model through a voice feature encoder (335).

[0118] The voice emotion expression model processing unit (330) can input the result encoded through the voice encoder (331) in the voice emotion expression model, the speaker identity feature, and the voice feature encoded through the voice feature encoder (335) into the voice decoder (337).

[0119] The voice emotion expression model processing unit (330) may, as an embodiment of the present invention, perform upsampling by applying the result encoded through the voice encoder (331) in the voice emotion expression model to the voice decoder (337), and concatenate the upsampling result with the speaker identity feature and the encoded voice feature. Here, the feature vector that has passed through the voice decoder (337) may experience a bottleneck phenomenon due to the voice encoder (331) or the downsampler and upsampler included in the voice decoder (337) in the voice emotion expression model. However, since the feature vector that has passed through the voice encoder (331) and the feature vector that has passed through the speaker identity feature and the voice feature encoder (335) are concatenated, the feature vector that has passed through the voice decoder (337) can sufficiently compensate for the loss and deformation of information due to the bottleneck phenomenon.

[0120] The voice emotion expression model processing unit (330) can reconstruct the feature vector, which is the final result of the voice decoder (337), into a mel spectrogram through the mel spectrogram reconstruction unit (339).

[0121] The Mel Spectrogram Reconstruction Unit (339) may include a process of decoding the feature vector, which is the final result of the voice decoder (337), into an audio signal, a process of converting the decoded audio signal into a spectrogram, and a process of changing the converted spectrogram into a Mel Spectrogram.

[0122] The voice emotion expression model processing unit (330) can perform learning by comparing the Mel Spectrogram obtained from the speaker's voice utterance with the Mel Spectrogram output through the Mel Spectrogram reconstruction unit (339).

[0123] FIG. 4 is a diagram schematically illustrating the probability of each emotion classification, which is an output result of a device (300) capable of estimating a user's emotion according to one embodiment of the present invention.

[0124] Referring to FIG. 4, a device (300) capable of estimating a user's emotion according to the present invention can provide probabilities for six emotion categories classified as anger, disgust, fear, joy, sadness, and surprise, for example, and can also select and output or provide only the anger emotion showing the highest emotion probability among these.

[0125] FIG. 5 is a diagram schematically illustrating a method by which a device (300) according to one embodiment of the present invention can estimate a user's emotions and a method of learning to estimate emotions.

[0126] First, a method for estimating user emotions of a device (300) according to an embodiment illustrated in FIG. 5 will be described.

[0127] Referring to FIG. 5, a processor (310) in a device (300) capable of estimating a user's emotions may include a voice emotion expression model processing unit (330), a text emotion expression model processing unit (340), and an expression fusion model processing unit (350).

[0128] Referring to FIG. 5, the process of learning by comparing the mel spectrogram output through the voice emotion expression model (SER Model) and the mel spectrogram reconstruction unit (339) with the mel spectrogram obtained from the speaker's voice signal is processed in the voice emotion expression model processing unit (330), the text emotion expression model (TER Model) is processed in the text emotion expression model processing unit (340), and the process of learning by comparing the emotion specified as the expression fusion model (RF Model), the classifier (353), and the reference truth label and the estimated emotion is processed in the expression fusion model processing unit (350).

[0129] In Fig. 5, the voice emotion expression model processing unit (330) obtains speaker identity features, speaker's voice data features, voice features, and mel spectrogram, all of which can be extracted to include a segment of a predetermined frame in the speaker's voice utterance.

[0130] The voice emotion expression model processing unit (330) can set the speaker identity features to include an embedding of a predetermined dimension for each utterance, and can obtain them through Resemblyzer, for example, and the speaker's voice data features can be obtained through a pre-trained Wav2vec2.0 model, for example, and can input the obtained speaker identity features and the speaker's voice data features into the voice emotion expression model.

[0131] Referring to FIG. 5, the voice encoder (331) of the voice emotion expression model may generally include a configuration of a weighted average layer with hyperparameters adjusted to more effectively extract voice data features of Wav2vec2.0, a concatenation layer that combines the results of the weighted average layer and the acquired speaker identity features, a one-dimensional convolution normalization layer consisting of three filters, two bidirectional LSTM layers, and a downsampler layer.

[0132] The voice emotion expression model processing unit (330) can input encoded voice data features prior to the downsampling step of the voice encoder (331) into the unpacked feature encoder (333) and output a feature query, a voice feature key, and a voice feature value. In addition, the unpacked feature encoder (333) can include a configuration of first, second, and third one-dimensional convolution layers each consisting of one filter.

[0133] Referring to FIG. 5, the text sentiment expression model processing unit (340) can input text features acquired from transcribed text corresponding to speaker identity features and speaker voice data features into the text sentiment expression model. Here, the transcribed text can have a maximum token length set to a predetermined size, and the text features can be features extracted through a pre-trained Bert-Large version.

[0134] Referring to FIG. 5, the text encoder (341) of the text sentiment expression model may include a configuration of a one-dimensional convolution layer composed of two filters, and a fourth and fifth one-dimensional convolution layers composed of one filter.

[0135] The text emotion expression model processing unit (340) can input text features into the text encoder (341) and output text feature queries, text feature keys, and text feature values.

[0136] Referring to FIG. 5, the expression fusion model may include a configuration of a first cross-attention mechanism, a second cross-attention mechanism, and a first add-normalization layer, one feedforward layer, a second add-normalization layer, and one max pooling layer that can be applied to the results of each cross-attention mechanism, respectively.

[0137] Referring to FIG. 5, the classifier (353) may be connected by applying a combination layer to the final maximum pooled result of the results of the first cross-attention mechanism and the second cross-attention mechanism of the expression fusion model, and may include configurations of one feedforward layer, one dropout layer, and one feedforward layer.

[0138] According to FIG. 5, the expression fusion model processing unit (350) may include a configuration of a cross-attention mechanism applied to the expression fusion model, including a first cross-attention mechanism applying a voice feature query, a text feature key, and a text feature value, and a second cross-attention mechanism applying a text feature query, a voice feature key, and a voice feature value.

[0139] That is, voice and text expression Queries, keys and values ​​are linearly projected. Each modality can be expressed as in mathematical formula 1.

[0140]

[0141]

[0142]

[0143] The interest importance of the cross-attention mechanism can be applied together with the softmax function to derive the interest score probability. This probability is reflected in the value representing the information propagated in the audio. Represents information that is transmitted from audio to text or from text to audio, reflected in the value. is a single-head attention, and 8 multi-head attentions can be used, and single-head attentions can be connected to each other, and this process can be calculated as in mathematical expression 2.

[0144]

[0145]

[0146]

[0147] The information calculated through the above process is applied to the features of the query form, and the first joint-normalization layer applied to each cross-attention mechanism can calculate the relationship between the forms as in mathematical expression 3.

[0148]

[0149]

[0150] The two expressions can be respectively global-max-pooled and concatenated for input to the classifier (353), and the first feedforward layer, the second concatenation-normalization layer, the max-pooling layer, and the subsequent concatenation layer applied to the cross-attention mechanism can be calculated as in Equation 4.

[0151]

[0152]

[0153]

[0154] As a result, the expression is the final output of the fusion representation model, and the representation contains information about the relationship between the two modalities for emotion recognition.

[0155] According to FIG. 5, the expression fusion model processing unit (350) can perform a function of outputting the probability for each emotion classification through a classifier (353) based on the output results through the expression fusion model. In addition, the expression fusion model processing unit (350) can output to the user the emotion classification with the highest probability among the probability output values ​​for each emotion classification through the classifier.

[0156] Referring to FIG. 5, the voice decoder (337) of the voice emotion expression model may include a configuration of an up-sampler layer, a concatenation layer that combines the output of the up-sampler layer with acquired speaker identity features and encoded voice features, one unidirectional LSTM layer, a one-dimensional convolutional normalization layer composed of three filters, two unidirectional LSTM layers, one linear normalization layer, a one-dimensional convolutional normalization layer composed of four filters, and a one-dimensional convolutional normalization layer composed of one filter.

[0157] Referring to FIG. 5, the voice feature encoder (335) of the voice emotion expression model may include a configuration of a one-dimensional convolutional normalization layer composed of three filters and two bidirectional LSTM layers.

[0158] According to FIG. 5, the voice emotion expression model processing unit (330) can apply the encoded result through the voice encoder (331) in the voice emotion expression model to the voice decoder (337) to perform an upsampling step and concatenate the upsampled result with the speaker identity feature and the encoded voice feature. The concatenated feature vector includes all of the feature vector of the upsampled voice encoder (331), the speaker identity feature vector, and the encoded voice feature vector, and the number of dimensions is the sum of all dimensions of the concatenated feature vector. The result, which may cause a bottleneck phenomenon through the downsampling step and the upsampling step included in the voice encoder (331) and the voice decoder (337) of the voice emotion expression model, is combined with the speaker identity feature and the encoded voice feature in the concatenation layer of the voice decoder (337), thereby supplementing the accuracy of the final output of the voice decoder (337).

[0159] A method of learning to estimate user emotions of a device (300) according to an embodiment illustrated in FIG. 5 is described.

[0160] According to FIG. 5, the voice emotion expression model processing unit (330) may include a step of designating a mel spectrogram obtained from a speaker's voice utterance as a reference truth label and training the mel spectrogram output through the mel spectrogram reconstruction unit (339) in a supervised learning manner.

[0161] The Mel Spectrogram output through the Mel Spectrogram Reconstruction Unit (339) is sufficiently compensated for the loss and deformation of information due to the bottleneck phenomenon in the feature vector that has passed through the voice decoder (337), so the learning model for estimating the user's emotion according to the present invention, which performs learning by designating the Mel Spectrogram obtained from the speaker's voice signal as a reference truth label, has very high robustness to noise and accuracy of emotion inference.

[0162] According to FIG. 5, the expression fusion model processing unit (350) may include a step of training the probability values ​​for each emotion classification outputted through the classifier (353) with probability values ​​for each emotion classification designated as a reference truth label, and may also include a step of training one emotion classification with the highest probability among the probability values ​​for each emotion classification outputted through the classifier (353) and one emotion classification designated as a reference truth label.

[0163] Referring to FIG. 5, the difference between the mel spectrogram obtained from the speaker's speech specified as the reference truth label in the voice emotion expression model processing unit (330) and the mel spectrogram output through the mel spectrogram reconstruction unit (339) can be defined as the first loss, and the difference between the emotion classification data (including probability values ​​for each emotion classification and one emotion classification with the highest probability) specified as the reference truth label in the expression fusion model processing unit (350) and the emotion data (including probability values ​​for each emotion classification and one emotion classification with the highest probability) output through the classifier (353) can be defined as the second loss. In addition, the sum of the first loss and the second loss can be defined as the total loss of the model according to one embodiment of the present invention, and a predetermined loss function can be applied and trained to minimize the total loss.

[0164] Table 2 below is a table illustrating the accuracy of the emotion estimation device according to the present invention.

[0165] ModelUA(unweighted accuracy)WA(weighted accuracy)ASR-SER69.768.6Learning alignment70.972.5STSER72.0571.06CMA+Raw waveform72.8-Model according to the present invention73.8374.65

[0166] Referring to Table 2, the data used is the result of using The interactive emotional dyadic motion capture database (IEMOCAP), which is currently the most actively used dataset in emotion recognition research using voice, and the evaluation index is the result of performing 5-fold cross-validation on a total of 4 emotions (happiness, sadness, anger, and neutral). ASR-SER, Learning alignment, STSER, and CMA+Raw waveform are the results of estimating emotion recognition using conventional techniques, respectively, and the model according to the present invention is the result obtained by applying the dataset to the present invention. As shown in Table 2, the emotion estimation result of the device applying the present invention shows significantly higher accuracy estimation results than the conventional technique in both UA (Unweighted Accuracy) and WA (Weighted Accuracy).

[0167] The high performance of the artificial intelligence model according to the present invention can be interpreted as an improvement in performance as a result of selectively using features that directly affect emotions when judging the emotions of a speaker, as explained above, applying the results before the down-sampling step in the voice emotion expression model to the unpacked feature encoder (333) and using data to which unpacked features are applied, and combining the speaker identity features and voice features acquired after the up-sampling step in the voice emotion expression model to avoid and supplement bottlenecks that may occur in the voice encoder (331) and voice decoder (337) in machine learning, and effectively fusing heterogeneous modalities corresponding to voice and text by utilizing unpacked features.

[0168] The embodiments described above are combinations of components and features of the present invention in a predetermined form. Each component or feature should be considered optional unless explicitly stated otherwise. Each component or feature may be implemented without being combined with other components or features. Furthermore, it is also possible to form an embodiment of the present invention by combining some components and / or features. The order of operations described in the embodiments of the present invention may be changed. Some components or features of one embodiment may be included in another embodiment or may be replaced with corresponding components or features of another embodiment. It is self-evident that claims that do not have an explicit citation relationship in the patent claims may be combined to form an embodiment or may be incorporated as a new claim through a post-application amendment.

[0169] In the present invention, the processor (310) may be implemented by hardware, firmware, software, or a combination thereof. When implementing an embodiment of the present invention using hardware, application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), etc. configured to perform the present invention may be provided in the processor (310). The method according to the present invention may also be implemented as a computer-readable recording medium recording a program for executing the method on a computer.

[0170] It will be apparent to those skilled in the art that the present invention can be embodied in other specific forms without departing from the essential characteristics thereof. Therefore, the above detailed description should not be construed as limiting in any respect, but rather as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the scope of equivalents of the present invention are intended to be included within the scope of the present invention.

[0171] An emotion estimation device using a multimodal model and a learning method thereof can be industrially utilized to monitor emotional changes in patients in the healthcare industry.

Claims

1. Input the acquired speaker identity features and acquired speaker voice data features into a given speech emotion representation (SER) model, In the above voice emotion expression model, the speaker identity features and the speaker's voice data features are encoded through a voice encoder, A voice emotion expression model processing unit that encodes the encoded speaker identity features and the speaker's voice data features through a disentangled feature encoder and outputs the speaker's voice feature query, voice feature key, and voice feature value; Inputting text features obtained from the transcription text corresponding to the speaker identity features and the speaker's voice data features into a predetermined Text Emotion Representation (TER) model, A text sentiment expression model processing unit that encodes the text feature in the above text sentiment expression model through a text encoder and outputs a text feature query, a text feature key, and a text feature value of the text feature; and An emotion estimation device, comprising a representation fusion model processing unit that inputs the output speaker's voice feature query, voice feature key and voice feature value and the output text feature query, text feature key and text feature value into a predetermined representation fusion (RF) model and applies a cross attention mechanism to output the probability for each emotion classification of the speaker.

2. In paragraph 1, An emotion estimation device in which the speaker identity features and the speaker's voice data features encoded by the voice encoder in the above-mentioned voice emotion expression model processing unit correspond to the encoded voice expression before downsampling is applied.

3. In paragraph 1, In the above expression fusion model processing unit, the cross-attention mechanism is An emotion estimation device comprising at least one of a first cross-attention mechanism applying the acquired voice feature query, text feature key, and text feature value, and a second cross-attention mechanism applying the acquired text feature query, voice feature key, and voice feature value.

4. In paragraph 1, The above expression fusion model processing unit is, An emotion estimation device that outputs to the user the emotion classification with the highest probability among the probability output values ​​for each emotion classification of the speaker.

5. In paragraph 1, An emotion estimation device in which the speaker's voice data features in the above-mentioned voice emotion expression model processing unit are wav2vec features or wav2vec2.0 features extracted from a predetermined voice recognition pre-learning model.

6. In the method of estimating the user's emotions, A step of inputting the acquired speaker identity features and the acquired speaker's voice data features into a predetermined speech emotion representation (SER) model; A step of encoding the speaker identity features and the speaker's voice data features through a voice encoder in the above voice emotion expression model; A step of encoding the encoded speaker identity features and the speaker's voice data features through a disentangled feature encoder to output the speaker's voice feature query, voice feature key, and voice feature value; A step of inputting text features obtained from a transcription text corresponding to the speaker identity features and the speaker's voice data features into a predetermined Text Emotion Representation (TER) model; A step of encoding the text feature through a text encoder in the text sentiment expression model to output a text feature query, a text feature key, and a text feature value of the text feature; An emotion estimation method, comprising the step of inputting the output speaker's voice feature query, voice feature key and voice feature value and the output text feature query, text feature key and text feature value into a predetermined representation fusion (RF) model and applying a cross attention mechanism to output the probability for each emotion classification of the speaker.

7. In paragraph 6, A method for estimating emotions, wherein in a step of encoding the acquired speaker identity features and acquired voice data features through a voice encoder, the speaker identity features and the speaker's voice data features encoded by the voice encoder correspond to the encoded voice expression before downsampling is applied.

8. In paragraph 6, The step of outputting the probability by emotion classification of the speaker above is: A method for estimating emotion, comprising a step of applying at least one of a first cross-attention mechanism applying the acquired voice feature query, text feature key, and text feature value, and a second cross-attention mechanism applying the acquired text feature query, voice feature key, and voice feature value.

9. In paragraph 6, The step of outputting the probability by emotion classification of the speaker above is: An emotion estimation method, comprising a step of outputting to a user an emotion classification having the highest probability among the probability output values ​​for each emotion classification of the speaker.

10. In a learning method of a device capable of estimating a user's emotions, A step of inputting the acquired speaker identity features, acquired speaker voice data features, acquired phonetic features, and acquired Mel Spectrogram into a predetermined voice emotion expression model; A step of encoding the speaker identity features and the voice data features using a voice encoder in the above voice emotion expression model; A step of encoding the acquired voice features in the above voice emotion expression model using a phonetic feature encoder; A step of decoding the speaker identity feature, the encoded speaker identity feature and the voice data feature and the encoded voice feature in the voice emotion expression model using a voice decoder; and A learning method for an emotion estimation device, comprising a step of converting the output of the speech decoder into a Mel Spectrogram and training it by designating the obtained Mel Spectrogram as a reference truth label.

11. In paragraph 10, The step of encoding with the above voice encoder includes the step of downsampling a voice representation including the acquired speaker identity features and the acquired speaker's voice data features, A learning method of an emotion estimation device, wherein the step of decoding with the above voice decoder includes the step of upsampling the downsampled voice expression.

12. In paragraph 10, A learning method of an emotion estimation device, wherein the step of decoding with the voice decoder includes a step of concatenating at least one of the speaker identity feature and the encoded voice feature with the result of upsampling the speaker identity feature and the voice data feature.

13. In a learning method of a device capable of estimating a user's emotions, A step of inputting the acquired speaker identity features and the acquired speaker's voice data features into a predetermined speech emotion representation (SER) model; A step of encoding the speaker identity features and the voice data features using a voice encoder in the above voice emotion expression model; A step of encoding the encoded speaker identity features and voice data features through a disentangled feature encoder to output the speaker's voice feature query, voice feature key, and voice feature value; A step of inputting text features obtained from a transcription text corresponding to the speaker identity features and the speaker's voice data features into a predetermined Text Emotion Representation (TER) model; A step of encoding the text feature through a text encoder in the text sentiment expression model to output a text feature query, a text feature key, and a text feature value of the text feature; A step of inputting the output speaker's voice feature query, voice feature key and voice feature value and the output text feature query, text feature key and text feature value into a predetermined representation fusion (RF) model and applying a cross attention mechanism to output the probability for each emotion classification of the speaker; and A learning method of an emotion estimation device, comprising a step of learning the emotion classification probability of the speaker output above as the emotion classification probability designated as the reference truth label.

14. In paragraph 13, A learning method for an emotion estimation device, comprising a step of learning using an emotion classification having the highest probability among the emotion classification probabilities of the speaker output through the above-described representation fusion (RF) model and an emotion classification designated as a reference truth label.

15. In paragraph 13, A learning method of an emotion estimation device, wherein, in a step of encoding the acquired speaker identity features and acquired voice data features through a voice encoder, the speaker identity features and the voice data features encoded by the voice encoder correspond to encoded voice expressions before downsampling is applied.

16. In paragraph 13, The step of outputting the probability by emotion classification of the speaker above is: A learning method of an emotion estimation device, comprising a step of applying at least one of a first cross-attention mechanism applying the acquired voice feature query, text feature key, and text feature value, and a second cross-attention mechanism applying the acquired text feature query, voice feature key, and voice feature value.

17. Input the acquired speaker identity features, acquired speaker voice data features, acquired phonetic features, and acquired Mel Spectrogram into a given voice emotion expression model. In the above voice emotion expression model, the speaker identity feature and the voice data feature are encoded using a voice encoder, In the above voice emotion expression model, the acquired voice features are encoded using a phonetic feature encoder, In the above voice emotion expression model, the speaker identity feature, the encoded speaker identity feature and voice data feature, and the encoded voice feature are decoded by a voice decoder. A learning performing device for estimating a user's emotion, including a voice emotion expression model processing unit that converts the output of the voice decoder into a Mel Spectrogram and trains it by designating the obtained Mel Spectrogram as a reference truth label.

18. In paragraph 17, The above voice encoder performs downsampling after the encoding, The above voice decoder is a learning performing device for estimating a user's emotion, which performs upsampling on the encoded speaker identity features and voice data features before decoding.

19. Input the acquired speaker identity features and acquired speaker voice data features into a given speech emotion representation (SER) model, In the above voice emotion expression model, the speaker identity feature and the voice data feature are encoded using a voice encoder, A voice emotion expression model processing unit that encodes the encoded speaker identity features and voice data features through a disentangled feature encoder and outputs the speaker's voice feature query, voice feature key, and voice feature value; Inputting text features obtained from the transcription text corresponding to the speaker identity features and the speaker's voice data features into a predetermined Text Emotion Representation (TER) model, A text sentiment expression model processing unit that encodes the text feature in the above text sentiment expression model through a text encoder and outputs a text feature query, a text feature key, and a text feature value of the text feature; and The speaker's voice feature query, voice feature key and voice feature value outputted above and the text feature query, text feature key and text feature value outputted above are inputted into a predetermined representation fusion (RF) model, and a cross-attention mechanism is applied to output the probability for each emotion classification of the speaker. A learning performing device for estimating a user's emotion, including an expression fusion model processing unit that learns the probability of each emotion classification of the speaker outputted above as the probability of each emotion classification designated as a reference truth label.

20. In paragraph 19, The above expression fusion model processing unit is a learning performing device for estimating the user's emotions by learning the emotion classification with the highest probability among the emotion classification probabilities of the speaker output through the above-mentioned expression fusion model and the emotion classification designated as the reference truth label.

Images