FACS-Based Emotion Recognition Method and Apparatus Reflecting Korean Facial Expression Characteristics

Abstract

A FACS-based emotion recognition method and apparatus reflecting Korean facial expression characteristics are disclosed. The FACS-based emotion recognition method reflecting Korean facial expression characteristics comprises: (a) inputting each frame of a Korean face image dataset into a FACS-based analysis model to calculate an AU intensity value and obtaining an emotion label corresponding to the face of each frame; (b) generating an emotion-specific AU preset corresponding to each emotion class using the emotion label and the AU intensity value; (c) calculating a cosine similarity between the emotion-specific AU presets to calculate a similarity value between emotion pairs; (d) configuring a plurality of emotion transition sequences using the similarity values ​​between emotion pairs; and (e) inputting a current emotion state and emotion conditions into a CVAE (Conditional Variational AutoEncoder)-based model, and the model generating an expression including a time-series emotion expression in real time by reflecting temporally continuous emotion state prediction and facial expression change according to the transition flow based on the emotion transition sequences.

Description

Technology Field

[0001] The present invention relates to a FACS-based emotion recognition method and apparatus that reflects the facial expression characteristics of Koreans. Background Technology

[0003] Existing emotion recognition technologies have been implemented by extracting features from human facial expressions and classifying the emotions represented by those expressions. A representative technology among these is the Facial Action Coding System (FACS)-based emotion analysis method. FACS defines specific facial muscle movements as basic units called Action Units (AUs) and utilizes specific AU combinations for each emotion as classification criteria. Previous studies have proposed methods such as matching AU strings with specific emotions or training emotion classifiers based on AU intensity.

[0004] Korean Patent No. 10-2015-0139985 proposed a method for extracting emotional expression information based on AU to classify emotions by matching AU strings with emotion labels, and proposed a device and storage medium structure corresponding to this. In addition, academic studies such as IEEE and CVPR have proposed Dynamic Bayesian Networks (DBN), Boosting-based Classifiers, or Transformer-based Attention structures to model dynamic relationships or temporal continuity between AUs.

[0005] Conventional FACS-based emotion recognition technology mechanically classifies emotions by associating them with specific AU combinations, but this does not adequately reflect the fact that emotion, a subjective experience, is a complex response of an organism reacting to internal and external stimuli. Emotional expression consists of elements beyond quantifiable muscle movements, and there are limitations in explaining a complete emotional state using only simple AU values. The problem to be solved

[0007] The present invention is intended to provide a FACS-based emotion recognition method and apparatus that reflect Korean facial expression characteristics.

[0008] In addition, the present invention aims to provide a FACS-based emotion recognition method and apparatus that reflect Korean facial expression characteristics capable of accurate emotion recognition reflecting socio-cultural characteristics by learning the relationship between emotion and AU using a Korean face image dataset.

[0009] Furthermore, the present invention aims to provide a FACS-based emotion recognition method and apparatus that reflect Korean facial expression characteristics capable of emotion recognition by reflecting the flow of emotion transfer that continuously shifts over time and considering the diversity of facial expressions according to conditional characteristics such as gender and age group. means of solving the problem

[0011] According to one aspect of the present invention, a FACS-based emotion recognition method reflecting Korean facial expression characteristics is provided.

[0012] According to one embodiment of the present invention, an emotion recognition method may be provided comprising: (a) inputting each frame of a Korean face image dataset into a FACS-based analysis model to calculate an AU intensity value and obtaining an emotion label corresponding to the face of each frame; (b) generating an emotion-specific AU preset corresponding to each emotion class using the emotion label and the AU intensity value; (c) calculating a cosine similarity between the emotion-specific AU presets to calculate a similarity value between emotion pairs; (d) configuring a plurality of emotion transition sequences using the similarity values ​​between emotion pairs; and (e) inputting a current emotion state and an emotion condition into a CVAE (Conditional Variational AutoEncoder)-based model, and the model generating an expression including a time-series emotion expression in real time by reflecting temporally continuous emotion state prediction and facial expression change according to the transition flow based on the emotion transition sequences.

[0013] Step (b) above can be performed by setting each emotion label as an independent variable (X) and setting the AU intensity value corresponding to each emotion label as a dependent variable (Y) to derive an emotion-specific regression coefficient vector, and by reflecting the emotion-specific regression coefficient vector in the AU intensity value to generate an emotion-specific AU preset.

[0014] Step (c) above can generate an emotion transfer sequence for emotion pairs in which the similarity value between the emotion pairs is greater than or equal to a predefined threshold.

[0015] The above emotional transition sequence can be configured to express a dynamic emotional flow in which emotional states change sequentially along a time axis.

[0016] The above emotion transfer sequence is composed of a directed graph, wherein step (d) selects emotion pairs whose similarity between emotion pairs is greater than or equal to a threshold value as emotion transfer candidates, sets emotion classes as each node, and constructs a directed graph by connecting the emotion pairs with directed edges, wherein the direction of each edge of the directed graph is set based on a predefined psychological order for emotion flow or similarity between emotions, and the weight of each edge may be configured to be proportional to the similarity value between the emotion pairs.

[0017] The above emotion transfer sequence can be generated by dynamically configuring a directed graph, with the emotion transfer probability, emotion transfer path direction, and transfer priority adjusted according to user conditions including gender and age group.

[0018] Prior to step (d) above, the method further includes a step of classifying the AU intensity values ​​for each emotion label according to gender and age group, and then analyzing the difference value of the average AU intensity values ​​according to gender and age group within the same emotion, wherein the difference value according to gender and age group can be used as a correction coefficient to adjust the edge weights of the directed graph.

[0019] The above AU presets for each emotion can be generated by performing regression analysis with one-hot encoded emotion vectors as independent variables for each emotion class.

[0021] According to another aspect of the present invention, an apparatus for performing a FACS-based emotion recognition method that reflects Korean facial expression characteristics is provided.

[0022] According to one embodiment of the present invention, a computing device may be provided comprising: an AU analysis unit that inputs each frame of a Korean face image dataset into a FACS-based analysis model to calculate an AU intensity value and obtains an emotion label corresponding to the face of each frame; a preset generation unit that generates an emotion-specific AU preset corresponding to each emotion class using the emotion label and the AU intensity value; a transition sequence generation unit that calculates a similarity value between emotion pairs by calculating the cosine similarity between the emotion-specific AU presets and constructs a plurality of emotion transition sequences using the similarity value between emotion pairs; and an emotion prediction and expression generation unit that inputs a current emotion state and emotion conditions into a CVAE (Conditional Variational AutoEncoder)-based model, and the model generates an expression including a time-series emotion expression in real time by reflecting temporally continuous emotion state prediction and expression change according to the transition flow based on the emotion transition sequences.

[0023] The preset generation unit can generate an AU preset for each emotion by setting each emotion label as an independent variable (X), setting the AU intensity value corresponding to each emotion label as a dependent variable (Y), performing Ridge regression analysis to derive an emotion-specific regression coefficient vector, and reflecting the emotion-specific regression coefficient vector in the AU intensity value.

[0024] The above transition sequence generation unit can generate an emotion transition sequence for emotion pairs in which the similarity value between the emotion pairs is greater than or equal to a predefined threshold.

[0025] The above AU analysis unit classifies the AU intensity values ​​for each emotion label according to gender and age group, and then further analyzes the difference values ​​of the average AU intensity values ​​according to gender and age group within the same emotion, wherein the difference values ​​according to gender and age group can be used as correction coefficients for adjusting the edge weights of the directed graph. Effects of the invention

[0027] By providing a FACS-based emotion recognition method and apparatus that reflect Korean facial expression characteristics according to one embodiment of the present invention, there is an advantage in that emotion recognition optimized for the unique emotional expression characteristics of Koreans, which reflect socio-cultural characteristics, is possible by learning the relationship between emotion and AU using a Korean face image dataset.

[0028] In addition, the present invention can reflect an emotional flow that changes naturally over time by configuring an emotional transition sequence based on AU presets for each emotion.

[0029] In addition, the present invention reflects an emotional transition flow that continuously shifts over time and can dynamically configure a personalized emotional flow by adjusting the emotional transition probability, transition path direction, and transition priority according to user conditions including gender and age group. Therefore, it has the advantage of enabling the generation of personalized emotional responses and the provision of advanced user experiences in various application fields such as digital humans, emotional response prediction, and real-time interfaces.

[0030] In addition, the present invention has the advantage of enabling flexible expression generation that reflects the diversity of expressions and individual differences even in a single emotional state by probabilistically generating AU vectors according to emotional conditions using a generative model based on CVAE (Conditional Variational AutoEncoder). Brief explanation of the drawing

[0032] FIG. 1 is a flowchart illustrating a FACS-based emotion recognition method reflecting Korean facial expression characteristics according to an embodiment of the present invention. FIG. 2 is a diagram illustrating a heatmap visualizing the relative importance of AU for each emotion according to an embodiment of the present invention. FIG. 3 is a diagram visualizing the results of calculating cosine similarity between AU presets by emotion according to an embodiment of the present invention in the form of an emotion similarity matrix. FIG. 4 is a diagram visualizing the results of calculating cosine similarity between AU presets by emotion according to an embodiment of the present invention in the form of an emotion similarity matrix. FIG. 5 is a diagram illustrating an emotion transfer sequence according to an embodiment of the present invention. FIG. 6 is a diagram illustrating an emotion condition-based AU vector diversity generation interface according to an embodiment of the present invention. FIG. 7 is a block diagram illustrating the internal configuration of a computing device that performs a FACS-based emotion recognition method reflecting Korean facial expression characteristics according to an embodiment of the present invention. Specific details for implementing the invention

[0033] As used in this specification, singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "composed" or "comprising" should not be interpreted as necessarily including all of the various components or steps described in the specification, and should be interpreted as meaning that some of the components or steps may be excluded, or that additional components or steps may be included. Furthermore, terms such as "...part," "module," etc., as used in the specification refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0034] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0036] FIG. 1 is a flowchart illustrating a FACS-based emotion recognition method reflecting Korean facial expression characteristics according to an embodiment of the present invention; FIG. 2 is a diagram illustrating a heatmap visualizing the relative importance of AUs for each emotion according to an embodiment of the present invention; FIG. 3 is a diagram visualizing the results of calculating cosine similarity between AU presets for each emotion in the form of an emotion similarity matrix according to an embodiment of the present invention; FIG. 4 is a diagram visualizing the results of calculating cosine similarity between AU presets for each emotion in the form of an emotion similarity matrix according to an embodiment of the present invention; FIG. 5 is a diagram illustrating an emotion transition sequence according to an embodiment of the present invention; and FIG. 6 is a diagram illustrating an emotion condition-based AU vector diversity generation interface according to an embodiment of the present invention.

[0037] In step 110, the computing device (100) inputs each frame of the Korean face image dataset into a FACS-based analysis model to calculate an AU intensity value and obtains an emotion label corresponding to each frame.

[0038] It is assumed that the Korean face image dataset has a predefined emotion label (e.g., happy, surprised, sad, fear, anger, disgust, neutral) assigned to each image or frame.

[0039] For each frame constituting the Korean face image dataset, an Action Unit (AU) analysis tool based on Facial Action Coding System (FACS) can be used to extract an intensity score for each AU, convert it into a numerical vector, and use it as an input feature for an emotion recognition model.

[0040] These AU intensity values ​​are values ​​that quantify the intensity of facial muscle movements as continuous values ​​within a defined range, and these numerical vectors can be used as basic data for analyzing the emotion-AU relationship corresponding to each emotion label and generating AU presets.

[0041] In step 120, the computing device (100) generates an emotion-specific face action unit (AU: Action Unit) preset corresponding to each emotion class using emotion labels and AU intensity values.

[0042] AU presets should be understood as multidimensional vectors composed of static feature vectors that quantitatively express the intensity values ​​of multiple AUs corresponding to a specific emotion class, and sets of AU intensity values ​​or regression coefficient values ​​derived for each emotion. This will be more clearly understood through the following explanation.

[0043] A computing device (100) may apply a one-hot encoding method to convert emotion labels assigned to each frame of a face dataset into a learnable format. For example, if the emotion label is 'sad' and there are 7 emotion classification criteria, it may be converted as [0, 0, 1, 0, 0, 0, 0]. If the total number of emotion classifications increases, one-hot encoding may be performed based on the number of corresponding emotion classes.

[0044] Subsequently, the computing device can perform Ridge regression analysis by setting the one-hot encoded emotion label as the independent variable (X) and the AU intensity value vector corresponding to the emotion label as the dependent variable (Y).

[0045] Ridge regression is a linear regression method that includes an L2 regularization term and has the characteristic of reducing the instability of regression coefficients and preventing overfitting even when there is a correlation between AUs. The regression coefficient (weight) vector for each emotion derived from the regression analysis results numerically represents the intensity of AU expression that acts relatively importantly in the corresponding emotional state. Accordingly, the computing device (100) can generate an emotion-specific AU preset that quantitatively expresses the AU expression pattern specialized for each emotion by reflecting the emotion-specific regression coefficient vector into the AU-specific intensity value vector.

[0046] FIG. 2 illustrates a diagram showing an AU Preset for each emotion derived by regression analysis according to an embodiment of the present invention, and a heatmap visualizing the relative importance of the AU for each emotion.

[0047] In Figure 2, the horizontal axis represents the emotion class, and the vertical axis represents the major AU. Red colors indicate positive coefficients (i.e., the importance of the corresponding AU in the corresponding emotion is high), and blue colors indicate negative coefficients (the corresponding AU is relatively unimportant in the corresponding emotion).

[0048] Referring to Figure 2, for the happiness emotion class, the regression coefficient for AU12 (corner of the mouth pulling) was the highest, similar to the existing FACS criteria, indicating that the corresponding AU plays a key role in the expression of happiness. Along with this, relatively high regression coefficients were also derived for AU6 (cheek lifting), AU1 (eyebrow lifting), and AU20 (lip pulling horizontally), indicating that the expression of happiness by Koreans involves a complex interplay of fine movements around the eyes and lips.

[0049] In addition, for the surprise emotion class, significant positive regression coefficients were identified for AU5 (upper eyelid) and AU9 (nose squint), in addition to AU1 (eyebrow lift) and AU2 (outer eyebrow lift). This implies that AU combinations different from the surprise expression defined by Western standards were observed, indicating that the surprise expression of Koreans tends to be accompanied by characteristic movements around the eyelids and nose.

[0050] In addition, for the sadness emotion class, the regression coefficient of AU5 (upward eyelid elevation) was found to be higher than that of AU15 (downward corner of the mouth), which traditionally represents sadness, indicating that the expression of sadness in Koreans may be closely related to eyelid movements, and the high importance of AU20 (horizontal lip pulling) indicates that horizontal tension of the lips can be a major characteristic of a sad expression.

[0051] In addition, for the anger emotion class, high regression coefficients were derived for AU9 (nose twitching) and AU17 (chin lifting) in addition to the traditionally known AU4 (brow furrowing), indicating that the expression of anger among Koreans is closely related not only to the eyebrows but also to muscle activation in the nose and jaw areas.

[0052] As described above, by reflecting the emotion-specific regression coefficient (weight) vector to the AU-specific intensity value vector extracted through FACS-based AU analysis, emotion-specific AU presets specialized for each emotion can be quantitatively configured. Through this, the computing device (100) according to one embodiment of the present invention can numerically model and analyze cultural and physiological emotional expression differences inherent in the unique emotional expression style of Koreans.

[0053] In addition, the computing device (100) according to one embodiment of the present invention is not limited to regression analysis based on a single average value when configuring AU presets by emotion, and may generate AU presets by reflecting differences in AU expression characteristics by condition according to user information (e.g., gender and age group) within a Korean face image dataset.

[0054] For example, by performing Ridge regression analysis on samples with the same emotion label separated by gender and age group, you can construct emotion-gender AU presets, emotion-age AU presets, or AU presets that consider both emotion, gender, and age.

[0055] In step 130, the computing device (100) calculates the similarity between each emotion-specific AU preset.

[0056] I will explain this in more detail.

[0057] The computing device (100) can calculate the cosine similarity between each pair of emotions based on an AU preset for each emotion.

[0058] More specifically, the computing device (100) can calculate the cosine similarity between AU presets corresponding to emotion classes i and j to calculate the similarity values ​​between emotions. For example, the AU preset for the happy emotion class can have its cosine similarity calculated with the AU preset for other emotion classes, such as the sad emotion class, the surprise emotion class, and the fear emotion class, respectively, and through this, the similarity values ​​between the happy emotion and each emotion can be individually derived.

[0059] Cosine similarity can be calculated using the following mathematical formula.

[0060]

[0061] Here, i and j represent the emotion class index, respectively, and AU_Preset i , AU_Preset j represents the AU presets for each emotion class i, j, and represents the L2 norm of each vector, and represents the inner product operation between two vectors.

[0062] Figure 4 is a diagram visualizing the results of calculating cosine similarity between AU presets by emotion in the form of an emotion similarity matrix.

[0063] The colors shown in Figure 3 represent the similarity values ​​between the AU presets for emotion A and emotion B, and the closer to +1, the higher the facial similarity, and the closer to -1, the opposite expression pattern.

[0064] This similarity matrix is ​​not limited to simple numerical comparison but can be utilized for the analysis of transfer structures that reflect the similarity in behavioral expressions and psychological associations between emotions.

[0065] For example, the emotion transfer sequence of none → surprise → fear shows high positive correlations in terms of AU similarity, with cosine similarity between each emotion being 0.77 and 0.86, respectively. This is consistent with the emotion flow of no stimulus → unexpected stimulus → threat perception described by Ekman’s basic emotion theory and Lazarus’s cognitive evaluation theory.

[0066] Furthermore, the emotional transference sequence of happy → sad → disgust captures a flow of gradual emotional change with a similarity of +0.84 between happiness and sadness and +0.52 between sadness and disgust; this aligns with the emotional transference flow of a sense of loss and antipathy toward the external world following the breakdown of positive emotions, as explained by Contrast Theory and Izard's Differential Emotion Theory.

[0067] Furthermore, the anger → surprise transition sequence exhibits high similarity with a cosine similarity of +0.65, indicating an emotional context transition akin to the abrupt reversal or reset of emotional states described by Reversal Theory.

[0068] As such, the AU preset similarity between emotions shown in Figure 4 demonstrates consistency with existing psychological theories of emotion and can be utilized as a data-based basis that provides quantitative and structural criteria for determining the possibility of emotion transference.

[0069] In addition, the existence of an edge between each emotion on the transition graph is determined based on whether there is a possibility of transition between emotions, and the thickness or weight of the edge is determined based on the relative magnitude of the AU preset similarity value, thereby enabling the modeling of an accurate emotion transition sequence that reflects both psychological flow and facial structure.

[0070] In addition, when calculating the similarity between emotion-specific AU presets, the computing device may, in addition to using a single preset based on the entire dataset, separately configure emotion-specific AU presets classified according to gender and age group conditions, and then calculate the cosine similarity between emotion-specific AU presets according to each condition.

[0071] Through this, even for the same pair of emotions, different similarity values ​​can be calculated depending on gender or age conditions, and this enables the configuration of an emotion transfer sequence optimized for each user condition.

[0072] In step 140, the computing device (100) constructs a directed graph-based emotion transfer sequence using similarity derived between emotion-specific AU presets.

[0073] A computing device (100) can determine that pairs of emotions have a similarity value greater than or equal to a predefined threshold (e.g., 0.5) based on the cosine similarity value between AU presets for each emotion as candidates for emotion transition, set the emotion class of each pair of emotions as a node, and construct a directed graph by connecting the pairs of emotions with directed edges to construct an emotion transition sequence (see FIG. 5). At this time, the direction of each edge of the directed graph can be set based on a predefined psychological order or similarity between emotions for the emotion flow, and the weight of each edge can be assigned in proportion to the similarity value between the pairs of emotions.

[0074] For example, the similarity between the emotions of happiness (exp_happy) and sadness (exp_sad) was found to be high at +0.84, which means that some of the AU presets between the two emotions are similar. More specifically, AU12 (corner of the mouth pull) appearing in the happy state may reflect a structural similarity that can be weakly expressed in the sad state.

[0075] On the other hand, the similarity between the emotions of happiness (exp_happy) and fear (exp_fear) was -0.97, showing a very low negative correlation, which means that the AU presets for these two emotions consist of completely different AU combinations.

[0076] The emotions of surprise (exp_surprise) and sadness (exp_sad) were also calculated to have a similarity of -0.97, indicating that the facial expression patterns are contrasting.

[0077] Therefore, based on the similarity between AU presets for each emotion, an emotion transition sequence leading from none → surprise → fear → sad, or an emotion transition sequence leading from happy → calm → surprise → fear, can be generated.

[0078] In addition, according to another embodiment of the present invention, the emotion transfer sequence may be configured as a structure that implements, based on data, a psychological theory such as Russell's Circumplex Model of Affect, which states that emotions are distributed in a continuous emotional space and are mutually transferable.

[0079] The direction and probability of transference between emotions can be regulated by the temporal stimulus context, changes in emotional intensity, or sequences of behavioral responses.

[0080] For example, none → surprise → fear

[0081] It is possible to generate a tension-escalating flow sequence leading from a state of no stimulus to the occurrence of an external stimulus → surprise → perception of threat → fear. This is based on the premise in Ekman’s basic emotion theory that surprise is the fastest emotional response, and reflects the dual-stage structure in Lazarus’s cognitive evaluation theory where surprise acts as the primary evaluation and fear as the secondary evaluation.

[0082] Happiness → Sadness → Disgust

[0083] It is an emotional transference sequence of positive emotion breakdown → sense of loss → aversion or avoidance, which may be based on the theory of emotional contrast, where negative emotions can appear as a reaction after positive emotions, and Izard's theory of differential emotion, where sadness is transferred to aversion.

[0084] Anger → Surprise

[0085] It is an emotional transference sequence in which there is a rapid shift to surprise upon an unexpected change in situation from a state of anger; based on Reversal Theory, which explains that emotional states can rapidly reverse depending on stimuli or changes in context, it can function as a process of emotional reset or reality re-evaluation.

[0086] Such emotional transfer sequences have the advantage of reflecting the actual flow of emotions more naturally and precisely than transfer models that rely simply on facial expression-based similarity, by reflecting the psychological associations and causal relationships between emotional states.

[0087] Such emotion transfer sequences are not fixed as a single path and can branch into multiple transfer paths depending on the multi-connection structure between nodes within the emotion transfer graph.

[0088] Therefore, multiple emotional transition sequences starting from a single emotional state can be constructed, which enables situation-specific customized transition flow modeling based on the similarity between emotional states, psychological flow, user conditions, etc.

[0089] In addition, according to one embodiment of the present invention, when configuring an emotion transfer sequence, the emotion transfer probability, the direction of the emotion transfer path, and the transfer priority may vary depending on user conditions including gender and age group, and as a result, the edge connection structure of the directed graph is dynamically changed, thereby generating a user-customized emotion transfer sequence.

[0090] This configuration enables more precise emotion recognition and prediction that reflects the individual characteristics of actual emotional responses by allowing the flow of transitions between emotions to be configured differently depending on user conditions, even if the emotional state is the same.

[0091] For example, let us explain with reference to Fig. 3. Fig. 3 is a diagram showing a heatmap visualizing the difference in average AU intensity values ​​according to gender within the same emotion.

[0092] The vertical axis represents the emotion classes (happy, surprise, none, disgust, anger, fear, sad), and the horizontal axis represents the AUs to be analyzed, such as AU1, AU2, AU4, AU5, ..., AU26.

[0093] Each cell represents the difference in average intensity values ​​(female - male) for each AU in the corresponding emotion using numbers and colors, and positive (+) values ​​indicate that women express the corresponding AU more strongly than men, while negative (-) values ​​indicate the opposite.

[0094] For example, in the emotion of disgust, the value of AU5 (upper eyelid elevation) was significantly higher for women than for men, indicating that Korean women tend to open their eyes wider when expressing disgust.

[0095] In addition, regarding the emotion of happiness, the value of AU17 (chin lifting) was higher for men than for women, showing a negative difference, indicating that the use of jaw muscles is more emphasized in the expression of happiness by Korean men. Regarding the emotion of sadness, the value of AU20 (lip pulling horizontally) was higher for women than for men, confirming that Korean women tend to pull their lips horizontally more when expressing sadness.

[0096] These differences in AU intensity values ​​between genders indicate that even for the same emotion, the AU that is primarily activated may differ depending on gender, and this can be used as a quantitative indicator reflecting physiological and cultural differences in emotional expression.

[0097] Accordingly, in one embodiment of the present invention, when configuring an emotion transfer sequence, user conditions such as gender and age group are considered, and different AU presets are applied according to the conditions to adjust the emotion transfer sequence based on similarity between the presets.

[0098] That is, the transition sequence configured based on AU similarity between emotions is not limited to a single fixed structure, and can be configured as a user-customized emotion transition sequence in which the direction, priority, or transition probability of the transition path is adjusted according to conditions when the AU representation method of each emotion class changes depending on gender and age conditions. Accordingly, in one embodiment of the present invention, when configuring the emotion transition sequence, the flow of emotion transition can be adjusted by considering gender and age group.

[0099] In step 150, the computing device (100) inputs emotional conditions into a CVAE (Conditional Variation AutoEncoder) based model according to the current emotional state based on emotional transition sequences, and generates an expression that reflects emotional prediction and expression change according to the transition flow.

[0100] Through this, the present invention enables emotional expressions to be output in the form of a sequence having a temporal flow rather than as a one-off event. That is, it is a structure that is adjusted and transitioned in real time according to input emotional conditions (e.g., text, voice, biosignals, etc.) or changes in the user's state. Through this, the present invention allows emotional expressions to be expressed as continuous emotional changes, such as sadness → neutral → joy, moving stepwise over a temporal flow. A CVAE-based model can be configured to receive both an emotional condition (c) and an AU vector (x) as input to probabilistically generate an expression (AU vector) suitable for the emotional condition. The encoder receives the emotional condition (c) and the AU vector (x) as input and the probability distribution of the latent vector z that changes according to the emotional condition Estimating , the decoder can be trained to output a reconstructed AU vector (′) based on the sampled and sentiment conditions.

[0101] During training, the final loss function is constructed by combining the reconstruction loss between the AU vector and the reconstructed AU vector (′) with the KL divergence (Kullback-Leibler divergence) loss to induce the latent space to follow a normal distribution, thereby enabling training into a probabilistic generative model that meets emotional conditions while ensuring facial expression diversity.

[0102] Through this learning structure, CVAE-based models can generate AU vectors suitable for a given emotion in multiple diverse sampled forms rather than just one when a specific emotion condition is given, and can provide flexibility to reflect subtle differences in emotion expression, intensity of expression, and individual differences.

[0103] In addition, after training is completed, by sequentially inputting emotional conditions at each time point along the flow of the emotion transition sequence, it is possible to generate a sequence of AU vectors that change continuously according to the temporal flow of emotion. That is, by generating facial expressions corresponding to the corresponding emotion at each step along the time-series emotion transition structure, the CVAE model enables natural and continuous emotional response prediction and real-time facial expression generation that corresponds to the actual flow of human emotion.

[0104] Compared to existing fixed emotion-expression mapping methods, this structure offers the advantage of being able to reflect temporally changing emotional transition flows, as well as the diversity and individuality of emotional expressions.

[0105] In addition, by sequentially predicting the next emotional state based on the chronological order of the emotional transition sequence when emotional conditions are input, it is possible to predict time-series emotional responses reflecting the emotional flow and generate facial expressions in real time.

[0106] Figure 6 is a diagram illustrating an interface for generating AU vector diversity based on emotional conditions.

[0107] When a user selects a specific emotional state (e.g., anger), the emotion transfer-based CVAE model of the present invention receives the corresponding emotional condition as input and generates AU vectors sampled from various probability distributions.

[0108] Each generated AU vector is utilized as a quantitative representation to construct a facial expression suitable for the corresponding emotional condition, thereby securing the possibility of various expressions (AU diversity) even within a single emotional state. This structure can quantitatively reflect individual differences in emotional expression, contextual differences, or even subtle changes in expression over time, making it possible to generate expressions that are much more natural and realistic compared to fixed emotion-expression mapping.

[0110] FIG. 7 is a block diagram illustrating the internal configuration of a computing device that performs a FACS-based emotion recognition method reflecting Korean facial expression characteristics according to one embodiment of the present invention.

[0111] FIG. 7 shows that a computing device (100) according to one embodiment of the present invention comprises an AU analysis unit (710), a preset generation unit (720), a transition sequence generation unit (730), an emotion prediction and facial expression generation unit (740), a memory (750), and a processor (760).

[0112] The AU analysis unit (710) is a means for inputting each frame of a face image into a FACS-based analysis model to calculate an intensity value for each AU and to obtain an emotion label corresponding to the face in the frame.

[0113] Additionally, the AU analysis unit (710) can extract gender and age group information from the input user profile and analyze the difference in average values ​​of AU intensity values ​​within the same emotion.

[0114] The preset generation unit (720) can generate an emotion-specific AU preset corresponding to each emotion class by learning the relationship between emotion labels and AU intensity values ​​using methods such as Ridge regression analysis.

[0115] The transition sequence generation unit (730) can generate an emotion transition sequence based on Russell's emotion prototype model or the cosine similarity between emotion-specific AU presets, construct a directed graph by connecting emotion pairs that are greater than or equal to a predefined threshold with directed edges, and generate an emotion transition sequence based on the cosine similarity between emotion-specific AU presets. Additionally, when generating an emotion transition sequence, the transition sequence generation unit (730) can generate an emotion transition sequence by considering gender and age group information as user conditions, reflecting the difference in average values ​​of AU intensity values ​​within the same emotion, and dynamically configuring a directed graph so that the emotion transition probability, the direction of the transition path, and the transition priority are adjusted according to the conditions.

[0116] The emotion prediction and expression generation unit (740) inputs emotion conditions into a CVAE-based model based on the current emotion state and emotion transition sequence, and probabilistically generates an AU vector that reflects emotion prediction and expression changes according to the transition flow corresponding to the emotion conditions, thereby generating continuous expression changes according to the temporal flow of emotion.

[0117] The detailed operation of the AU analysis unit (710), preset generation unit (720), transition sequence generation unit (730), and emotion prediction and expression generation unit (740) is as described above with reference to FIG. 1, so a redundant description will be omitted.

[0118] The memory (750) can store commands for performing a FACS-based emotion recognition method reflecting Korean facial expression characteristics according to one embodiment of the present invention and various data derived from the process.

[0119] The processor (760) is a means for controlling internal components of a computing device (100) according to one embodiment of the present invention (e.g., AU analysis unit (710), preset generation unit (720), transition sequence generation unit (730), emotion prediction and expression generation unit (740), memory (750), etc.).

[0121] An apparatus and method according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the computer-readable medium may be those specifically designed and configured for the present invention, or they may be those known and available to a person skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0122] The hardware device described above may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

[0123] The present invention has been described above with reference to its embodiments. Those skilled in the art will understand that the present invention may be implemented in modified forms without departing from the essential characteristics of the invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the invention is defined by the claims, not by the foregoing description, and all variations within the scope of the claims should be interpreted as being included in the invention.

Claims

Claim 1 (a) inputting each frame of a Korean face image dataset into a FACS-based analysis model to calculate an AU intensity value and obtaining an emotion label corresponding to the face in each frame; (b) generating an emotion-specific AU preset corresponding to each emotion class using the emotion label and the AU intensity value; (c) calculating a cosine similarity between the emotion-specific AU presets to calculate a similarity value between emotion pairs; (d) constructing a plurality of emotion transition sequences using the similarity values ​​between emotion pairs; and (e) inputting a current emotion state and emotion conditions into a CVAE (Conditional Variational AutoEncoder)-based model, and the model generating an expression including a time-series emotion expression in real time by reflecting temporally continuous emotion state prediction and facial expression change according to the transition flow based on the emotion transition sequences. Claim 2 An emotion recognition method according to claim 1, wherein step (b) is characterized by setting each emotion label as an independent variable (X), setting the AU intensity value corresponding to each emotion label as a dependent variable (Y), performing Ridge regression analysis to derive an emotion-specific regression coefficient vector, and generating an emotion-specific AU preset by reflecting the emotion-specific regression coefficient vector in the AU intensity value. Claim 3 An emotion recognition method according to claim 1, wherein step (c) is characterized by generating an emotion transfer sequence for emotion pairs in which the similarity value between the emotion pairs is greater than or equal to a predefined threshold. Claim 4 An emotion recognition method according to claim 1, characterized in that the emotion transfer sequence is configured to express a dynamic emotional flow in which emotional states change sequentially along a time axis. Claim 5 An emotion recognition method according to claim 1, wherein the emotion transfer sequence is composed of a directed graph, and step (d) comprises selecting emotion pairs whose similarity between emotion pairs is greater than or equal to a threshold value as emotion transfer candidates, setting emotion classes as each node, and constructing a directed graph by connecting the emotion pairs with directed edges, wherein the direction of each edge of the directed graph is set based on a predefined psychological order or similarity between emotions regarding the emotion flow, and the weight of each edge is configured to be proportional to the similarity value between the emotion pairs. Claim 6 An emotion recognition method according to claim 5, wherein the emotion transfer sequence is characterized by dynamically configuring and generating a directional graph by adjusting the emotion transfer probability, emotion transfer path direction, and transfer priority according to user conditions including gender and age group. Claim 7 An emotion recognition method according to claim 5, further comprising, prior to step (d), a step of classifying the AU intensity values ​​for each emotion label according to gender and age group, and then analyzing the difference values ​​of the average values ​​of the AU intensity values ​​according to gender and age group within the same emotion, wherein the difference values ​​according to gender and age group are used as correction coefficients for adjusting the edge weights of the directed graph. Claim 8 An emotion recognition method according to claim 1, characterized in that the emotion-specific AU preset is generated by performing regression analysis with one-hot encoded emotion vectors for each emotion class as independent variables. Claim 9 A computer-readable recording medium having a program for performing a method according to any one of claims 1 through 8. Claim 10 A computing device comprising: an AU analysis unit that inputs each frame of a Korean face image dataset into a FACS-based analysis model to calculate an AU intensity value and obtains an emotion label corresponding to the face of each frame; a preset generation unit that generates an emotion-specific AU preset corresponding to each emotion class using the emotion label and the AU intensity value; a transition sequence generation unit that calculates cosine similarity between the emotion-specific AU presets to calculate similarity values ​​between emotion pairs, and constructs a plurality of emotion transition sequences using the similarity values ​​between emotion pairs; and an emotion prediction and expression generation unit that inputs a current emotion state and emotion conditions into a CVAE (Conditional Variational AutoEncoder)-based model, and generates an expression including a time-series emotion expression in real time by reflecting temporally continuous emotion state prediction and expression change according to the transition flow based on the emotion transition sequences. Claim 11 A computing device according to claim 10, wherein the preset generation unit sets each emotion label as an independent variable (X), sets the AU intensity value corresponding to each emotion label as a dependent variable (Y), performs Ridge regression to derive an emotion-specific regression coefficient vector, and generates an emotion-specific AU preset by reflecting the emotion-specific regression coefficient vector in the AU intensity value. Claim 12 A computing device according to claim 10, wherein the transition sequence generating unit generates an emotion transition sequence for emotion pairs in which the similarity value between the emotion pairs is greater than or equal to a predefined threshold. Claim 13 A computing device according to claim 10, wherein the emotion transition sequence is configured to express a dynamic emotion flow in which the emotional state changes sequentially along a time axis. Claim 14 A computing device according to claim 10, wherein the emotion transfer sequence is composed of a directed graph, and the transfer sequence generation unit selects emotion pairs whose similarity between emotion pairs is greater than or equal to a threshold value as emotion transfer candidates, sets an emotion class as each node, and constructs a directed graph by connecting the emotion pairs with directed edges, wherein the direction of each edge of the directed graph is set based on a predefined psychological order or similarity between emotions for the emotion flow, and the weight of each edge is configured to be proportional to the similarity value between the emotion pairs. Claim 15 A computing device according to claim 14, wherein the emotion transfer sequence is characterized by dynamically configuring and generating a directed graph by adjusting the emotion transfer probability, emotion transfer path direction, and transfer priority according to user conditions including gender and age group. Claim 16 A computing device according to claim 14, wherein the AU analysis unit classifies the AU intensity values ​​for each emotion label according to gender and age group, and then further analyzes the difference values ​​of the average values ​​of the AU intensity values ​​according to gender and age group within the same emotion, wherein the difference values ​​according to gender and age group are used as correction coefficients for adjusting the edge weights of the directed graph. Claim 17 A computing device according to claim 10, wherein the emotion-specific AU preset is generated by performing regression analysis with one-hot encoded emotion vectors as independent variables for each emotion class.

Images