Method and system for generating emotional talking face from audio perspective for emotional face expression

Abstract

Proposed are a method and system for generating an emotional talking face from an audio perspective for emotional face expression. The system for generating an emotional talking face from an audio perspective for emotional face expression, as proposed in the present invention, comprises: a neutrality conversion unit that receives inputs of a reference image and audio to transmit only identity information in the reference image and generates a neutral talking face image for the reference image; and an emotional expression unit that encodes emotion information from the input of the audio and generates an emotional talking face by using separate expression learning of emotional expression and emotional intensity to reflect the emotional talking face in the generated neutral talking face image.

Description

Method and System for Generating Emotional Speaking Faces from an Audio Perspective for Emotional Facial Expressions

[0001] The present invention relates to a method and system for generating an emotional speaking face from an audio perspective for emotional facial expression.

[0002] Recently, thanks to neural network-based generative models, multimedia applications such as film production, the metaverse, and virtual humans have advanced significantly. In particular, audio-based speaking face generation is receiving great attention. The task of generating speaking face animations using a single facial identity image and audio aims to realistically reproduce lip movements that match the given audio.

[0003] However, with the exception of a few studies, most research has not sufficiently explored the importance of realistic facial expressions as much as lip movements in relation to the other person's face in terms of emotional interaction. As a result, most conventional technologies have synthesized expressions that rely on the emotion of a given facial identity or generated expressions where specific emotions were overly exaggerated.

[0004] Most existing technologies for generating speaking emotional face images have adopted a method of extracting emotion vectors from emotional reference images or emotion labels to convey emotion to the generated images. These vectors were used as conditional inputs to implement speaking face image generation in existing generators. Although conventional technologies have made significant contributions, three major limitations still exist.

[0005] First, when extracting identity features, the emotions of the identity image cannot be completely separated, resulting in the mixing of the identity image's emotional expressions with the elicited emotional expressions, causing the generation of unwanted emotions.

[0006] Second, since emotions are not extracted from the audio, an emotional video is generated that is unrelated to the emotional flow of the audio.

[0007] Third, unnatural video playback occurs because only one emotional intensity is expressed in most frames.

[0008] The technical problem that the present invention aims to solve is to analyze the shortcomings of conventional technologies that synthesize expressions dependent on the emotions of a given facial identity or generate similar emotional levels across all frames, and to ultimately generate natural speaking face images by analyzing new vector representations and learning methods to provide an audio-perspective method and system for generating emotional speaking faces that considers not only lip movements but also the perspective of emotional interaction.

[0009] In one aspect, the audio-perspective emotional speaking face generation system for emotional face expression proposed in the present invention includes a neutral conversion unit that receives a reference image and audio to transmit only identity information from the reference image and generates a neutral speaking face image for the reference image, and an emotional expression unit that generates an emotional speaking face using separate expression learning of emotional expression and emotional intensity to encode emotional information from the audio input and reflect it in the generated neutral speaking face image.

[0010] The above neutral transformation unit defines a neutral class with minimized emotional expression as the target domain and learns through contrastive goal-based learning to move away from classes containing emotional expression and closer to the neutral class in the metric space, thereby removing emotional features from the facial features of the input reference image and retaining only identity information in the expression.

[0011] The above-mentioned emotion expression unit inputs two features embedded in the emotion expression space and the emotion intensity space, respectively, for the separate expression learning of emotion expression and emotional intensity, into the audio emotion encoder and the audio emotion intensity regression encoder to normalize each encoder and perform continuous emotion analysis.

[0012] The above emotion expression unit trains an audio emotion encoder to extract continuous emotions from the audio, and subsequently trains an audio intensity regression encoder to refine the emotion intensity per frame.

[0013] In another aspect, the method for generating an emotional speaking face from an audio perspective for emotional face expression proposed in the present invention comprises the steps of receiving a reference image and audio to transmit only identity information from the reference image, generating a neutral speaking face image for the reference image from the audio input through a neutral conversion unit, and generating an emotional speaking face using separate expression learning of emotional expression and emotional intensity to encode emotional information from the audio input through an emotional expression unit and reflect it in the generated neutral speaking face image.

[0014] According to embodiments of the present invention, by analyzing new vector representations and learning methods, an audio-perspective emotional speaking face generation method and system that considers not only lip movements but also the perspective of emotional interaction can be used to analyze the disadvantages of conventional technology, such as synthesizing expressions dependent on the emotion of a given face identity or generating similar emotional levels across all frames, and ultimately generate natural speaking face images.

[0015] FIG. 1 is a schematic diagram illustrating an audio perspective emotional speaking face generation framework for emotional face expression according to one embodiment of the present invention.

[0016] FIG. 2 is a flowchart illustrating a method for generating an emotional speaking face from an audio perspective for emotional face expression according to an embodiment of the present invention.

[0017] FIG. 3 is a diagram showing the configuration of an audio perspective emotional speaking face generation system for emotional face expression according to one embodiment of the present invention.

[0018] FIG. 4 is a diagram illustrating a process for generating a neutral speaking face image according to an embodiment of the present invention.

[0019] FIG. 5 is a diagram illustrating the ANT and ERA learning process according to an embodiment of the present invention.

[0020] FIG. 6 is a diagram comparing the result of visualizing a bounding box according to one embodiment of the present invention with the prior art.

[0021] In order to resolve the input bias problem of the prior art, the present invention proposes a new neutralization method, thereby achieving impressive results in generating a neutral speaking face from an emotionally expressed face. Furthermore, through 2D continuous emotion label-based regression learning, various emotional intensities are effectively generated on a frame-by-frame basis from audio rather than emotion labeling. Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0022]

[0023] FIG. 1 is a schematic diagram illustrating an audio perspective emotional speaking face generation framework for emotional face expression according to one embodiment of the present invention.

[0024] The proposed algorithm of the present invention consists of two parts. The first is Audio-to-Neutral Translation (ANT) (110). It takes audio (130) as input and aims to generate a neutral speaking face (112) for any emotional face input (111). A neutral (emotional) class with minimized emotional expression is defined as the target domain, and through contrastive goal-based learning, emotional face features are brought closer to face identity features, thereby leaving only the identity components in the expression.

[0025] The second is Emotional Representation from Audio (ERA) (120). To enhance emotional representation, separate representation learning of facial expression and emotional intensity is used. Two features embedded in the expression and intensity spaces are input into an emotion recognition and intensity regression encoder, respectively, to normalize each encoder. This learning method is the first instance of applying continuous emotion analysis in the field of generating speaking face images, and it generates an emotional speaking face (121) capable of expressing emotional intensity close to the actual sequence at the frame level.

[0026]

[0027] FIG. 2 is a flowchart illustrating a method for generating an emotional speaking face from an audio perspective for emotional face expression according to an embodiment of the present invention.

[0028] A method for generating an emotional speaking face from an audio perspective for emotional face expression according to an embodiment of the present invention comprises: a step (210) of receiving a reference image and audio to transmit only identity information from a reference image and generating a neutral speaking face image for the reference image from the audio input through a neutral conversion unit; and a step (220) of encoding emotional information from the audio input through an emotional expression unit and generating an emotional speaking face using separated expression learning of emotional expression and emotional intensity to reflect it in the generated neutral speaking face image.

[0029] In step (210), a reference image and audio are input to transmit only identity information from the reference image, and a neutral speaking face image is generated from the audio input through a neutral conversion unit.

[0030] According to an embodiment of the present invention, a neutral class with minimized emotional expression is defined as the target domain, and training is performed through contrastive goal-based learning with the goal of moving away from the class containing emotional expression and closer to the neutral class in the metric space, thereby removing emotional features from the facial features of an input reference image and retaining only identity information in the expression.

[0031] In step (220), emotional speaking face is generated using separate expression learning of emotional expression and emotional intensity to encode emotional information from audio input through an emotional expression unit and reflect it in the generated neutral speaking face image.

[0032] According to an embodiment of the present invention, for learning separate expressions of emotion expression and emotion intensity, two features embedded in an emotion expression space and an emotion intensity space are input to an audio emotion encoder and an audio emotion intensity regression encoder, respectively, to normalize each encoder and perform continuous emotion analysis.

[0033] According to an embodiment of the present invention, an audio emotion encoder is trained to extract continuous emotions from audio, and then an audio intensity regression encoder is further trained to refine the emotion intensity per frame.

[0034]

[0035] FIG. 3 is a diagram showing the configuration of an audio perspective emotional speaking face generation system for emotional face expression according to one embodiment of the present invention.

[0036] An audio-perspective emotional speaking face generation system (300) according to the present embodiment may include a processor (310), a bus (320), a network interface (330), memory (340), and a database (350). The memory (340) may include an operating system (341) and an audio-perspective emotional speaking face generation routine (342) for emotional face expression. The processor (310) may include a neutral conversion unit (311) and an emotion expression unit (312). In other embodiments, the audio-perspective emotional speaking face generation system (300) may include more components than those of FIG. 3. However, it is not necessary to clearly illustrate most prior art components. For example, the audio-perspective emotional speaking face generation system (300) may include other components such as a display or a transceiver.

[0037] Memory (340) is a computer-readable recording medium and may include a non-perishable mass storage device such as RAM (random access memory), ROM (read only memory), and a disk drive. Additionally, program code for an operating system (341) and an audio-perspective emotional speaking face generation routine (342) for emotional face expression may be stored in memory (340). These software components may be loaded from a computer-readable recording medium separate from memory (340) using a drive mechanism (not shown). This separate computer-readable recording medium may include computer-readable recording media (not shown), such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. In another embodiment, software components may be loaded into memory (340) via a network interface (330) rather than a computer-readable recording medium.

[0038] The bus (320) can enable communication and data transmission between components of the audio-perspective emotional speaking face generation system (300). The bus (320) can be configured using a high-speed serial bus, a parallel bus, a Storage Area Network (SAN), and / or other suitable communication technology.

[0039] The network interface (330) may be a computer hardware component for connecting the audio-perspective emotional speaking face generation system (300) to a computer network. The network interface (330) may connect the audio-perspective emotional speaking face generation system (300) to a computer network via a wireless or wired connection.

[0040] The database (350) can serve to store and maintain all information necessary for generating an emotional speaking face from an audio perspective for emotional facial expression. In FIG. 3, the database (350) is shown built and included inside the audio-perspective emotional speaking face generation system (300), but it is not limited thereto and may be omitted depending on the system implementation method or environment, or it is also possible for all or part of the database to exist as an external database built on a separate system.

[0041] The processor (310) may be configured to process instructions of a computer program by performing input / output operations of the emotional speaking face generation system (300) of basic arithmetic, logic, and audio perspectives. Instructions may be provided to the processor (310) via memory (340) or a network interface (330) and via a bus (320). The processor (310) may be configured to execute program code for a neutral conversion unit (311) and an emotion expression unit (312). Such program code may be stored in a recording device such as memory (340).

[0042] The neutral conversion unit (311) and the emotion expression unit (312) can be configured to perform the steps (210–220) of FIG. 2.

[0043] An audio perspective emotional speaking face generation system (300) may include a neutral conversion unit (311) and an emotion expression unit (312).

[0044] A neutral conversion unit (311) according to an embodiment of the present invention receives a reference image and audio to transmit only identity information from a reference image, and generates a neutral speaking face image for the reference image from the audio input.

[0045] A neutral transformation unit (311) according to an embodiment of the present invention defines a neutral class with minimized emotional expression as a target domain and learns to move away from a class containing emotional expression and closer to a neutral class in a metric space through contrastive goal-based learning, thereby removing emotional features from the facial features of an input reference image and leaving only identity information in the expression.

[0046] An emotion expression unit (312) according to an embodiment of the present invention generates an emotional speaking face by using separate expression learning of emotion expression and emotion intensity to encode emotional information from an audio input and reflect it in the generated neutral speaking face image.

[0047] According to an embodiment of the present invention, the emotion expression unit (312) inputs two features embedded in the emotion expression space and the emotion intensity space for learning separate expressions of emotion expression and emotion intensity into an audio emotion encoder and an audio emotion intensity regression encoder, respectively, to normalize each encoder and perform continuous emotion analysis.

[0048] An emotion expression unit (312) according to an embodiment of the present invention learns an audio emotion encoder to extract continuous emotions from audio, and subsequently trains an audio intensity regression encoder to refine the emotion intensity per frame. An audio-perspective emotional speaking face generation system according to an embodiment of the present invention will be described in more detail below with reference to FIGS. 4 and 5.

[0049]

[0050] FIG. 4 is a diagram illustrating a process for generating a neutral speaking face image according to an embodiment of the present invention.

[0051] The present invention proposes an Audio-to-Neutral Translation (ANT) for transmitting only identity information from a reference image (410) along with audio. The proposed ANT is first trained on an in-the-wild face video dataset so that the speaking video has robust performance. It is then fine-tuned for neutralization. The generated neutral speaking face video (420) aims to move away from other emotions (431) in the metric space and closer to neutral emotions (432).

[0052] We propose a neutralization goal based on N-pair loss to map the identity face image of a reference image with strong emotions into a neutral subspace. This goal aims to move the reference image away from other emotions and closer to neutral emotions in the metric space during the initial learning stage (441). The table in the final learning stage (442) is an anchor. (Goal Neutral Emotion) and Positive Samples Minimize the distance between, and negative samples The distance between them is maximized to allow for the normalization of difficult examples. In this way, the input reference image (451) can finally be converted into a neutral speaking face image (452).

[0053]

[0054] FIG. 5 is a diagram illustrating the ANT and ERA learning process according to an embodiment of the present invention.

[0055] To apply the algorithm proposed in the present invention, ANT and ERA are trained respectively.

[0056] ANT (510) enables the generator to produce high-quality videos using adversarial loss and cognitive loss functions. It also uses a loss function that compares the mouth shape landmarks of the target image with the mouth shape landmarks of the generated image to match audio and mouth shape. Finally, it uses a metric loss function for emotion neutralization of the face.

[0057] ERA (520) is trained using a modified adversarial loss function and a class loss function for contextual emotion recognition. It also trains frame-by-frame emotion intensity regression through an intensity loss function.

[0058] Pre-training is performed on a real-world face video dataset to achieve the basic performance of face generation as defined by ANT. The purpose of this pre-training is to ensure the generation of realistic face videos by synchronizing lip movements with audio. This is pursued through the following three objectives. The first objective is the lip landmark loss L lip It relies on, which minimizes the difference between the lip landmarks extracted from the target video and the video generated by the face landmark estimator. This allows lip movements to be synchronized with the audio, and the audio encoder It helps capture features that appropriately represent lip movements. The second goal, namely perceptual loss L per It improves visual quality by minimizing semantic differences between the generated video and the target video. The final loss is adversarial loss L through a discriminator to further enhance visual quality. adv is. Inspired by the analysis that adversarial loss contributes significantly to the learning of fine textures and patterns, L adv It helps reduce blur and enhance edge details. As a result, L per and L adv is a generator It stimulates to generate high-quality video. Based on the three goals above, the trained ANT undergoes fine-tuning to neutralize arbitrary emotional faces.

[0059] While ANT's pre-training improves the overall performance of audio-based speaking face generation, it alone cannot completely solve the problem. A separate training process is required to mitigate this issue. Based on the results showing that neutral faces best represent the subject's identity, fine-tuning focused on neutral emotions is performed using the MEAD dataset.

[0060] The present invention proposes an Audio-to-Neutral Translation (ANT) for transmitting only identity information from a reference image along with audio. The proposed ANT is first trained on an in-the-wild face video dataset to ensure robust performance for speaking videos. Subsequently, it undergoes fine-tuning for neutralization. The generated neutral speaking face video aims to move away from other emotions in the metric space and closer to neutral emotions.

[0061] We propose an N-pair loss-based neutralization goal to map the identity face image of a reference image with strong emotions into a neutral subspace. This goal aims to move the reference image away from other emotions and closer to neutral emotions in the metric space during the early stages of training. The table in the final training stage is the anchor. (Goal Neutral Emotion) and Positive Samples Minimize the distance between, and negative samples Maximizing the distance between them allows for the normalization of difficult examples. The loss function for learning this can be expressed as follows:

[0062]

[0063] Here, C represents the number of emotion classes, and f represents the average of the features per frame. Specifically, The proposed method quantifies sentiment-based semantic distinctions using MEAD when encoding reference image I. Therefore, this learning framework can lead to significant performance improvements in neutralizing difficult examples. The overall goal of ANT fine-tuning is as follows:

[0064]

[0065] Here, λ lip , λ per , λ adv and λ metric are L respectively lip , L per , L adv and L metric It is the scale factor of.

[0066] Next, according to an embodiment of the present invention, the generation of Emotional Representation from Audio (ERA) is proposed to effectively encode emotional information from audio and reflect it in the generated video. In the present invention, emotional intensity, which has often been overlooked in existing research, is defined independently along with the contextual emotion of the audio. To extract the contextual emotion of the audio, an additional audio emotion encoder not used in ANT is introduced. Furthermore, an audio encoder is additionally trained to refine the emotional intensity per frame. Since the encoder is initially trained on a neutral domain, it may lack emotional information. Therefore, by training the audio encoder using intensity regression based on Russell's 2D continuous emotion system, it is ensured that emotional intensity is included in the audio specifics.

[0067] According to an embodiment of the present invention, two task branches (indicated as θ and φ in FIG. 5) are introduced for contextual emotion classification and frame-by-frame emotion intensity regression. An additional audio emotion encoder not utilized in ANT is used to extract contextual emotions from audio. Introduces. Also, an audio encoder to refine emotional intensity per frame Train additionally. Since it is initially trained on a neutral domain, it may lack sentiment information. Therefore, using intensity regression based on Russell's 2D continuous sentiment system Audio features by training Ensure that emotional intensity is included.

[0068] It takes the audio Mel spectrogram as input and an emotion feature sequence Process as. Is It must be trained to include only sentiment information. Therefore, classification branches are trained based on categorical labels (e.g., happiness, surprise). The LSTM-based classifier θ is trained using the following loss function:

[0069]

[0070] Here is the sentiment label and B is the batch size. The above equation uses cross-entropy to improve sentiment prediction using audio functions. Ensure that sufficient contextual emotional information is included. Also, L e-adv L indicated by adv The modified version evaluates the accuracy of the generated video in expressing the target emotion. This is achieved through class-specific MinMax learning based on emotion conditions (branch in Fig. 5 (Explained in ). Through learning with these two loss terms It is used as an emotional function of audio.

[0071] To represent the flow of emotional intensity, we present a Valence and Arousal (VA)-based intensity regression. Here, VA represents the positive / negative and activation levels of emotion, and the magnitude of the VA vector represents emotional intensity without loss of generality. Generally, VA labels exist per frame in video and can appropriately match frame-level labels in audio. However, the MEAD dataset according to the embodiments of the present invention does not contain such continuous VA labels. Therefore, for VA regression, we adopt a strategy of obtaining so-called virtual labels using SOTA technology. The virtual label r of per-frame emotional intensity is defined as follows:

[0072]

[0073] Here and are each VA values. In the present invention Through the VA regression backbone indicated by The goal is to predict intensity flow and enhance expression in terms of emotional intensity through an MLP-based regressor φ. For this purpose, intensity loss is defined as follows:

[0074]

[0075] Here, B and N represent the batch size and sequence length, respectively. Based on the above equation, the model learns frame-by-frame changes according to emotional intensity to better represent emotional intensity. Unlike existing techniques that define a single emotional expression for all frames, the method proposed in this invention instructs the model regarding frame-by-frame changes in emotional intensity. It should be noted that this is the first instance of this being applied to an ETF task. The above equation plays a crucial role in extending visual domain knowledge to the audio domain by leveraging audio-visual data correlations. Finally, the overall goal of ERA is defined as follows:

[0076]

[0077] Here, λ lip , λ e-adv , λ cls and λ inten are L respectively lip , L e-adv , L cls and L inten It is the scale factor of.

[0078]

[0079] FIG. 6 is a diagram comparing the result of visualizing a bounding box according to one embodiment of the present invention with the prior art.

[0080] To demonstrate the effectiveness of the present invention, the performance of speech videos was evaluated on representative speech datasets, LRW and MEAD. The speech face generator to which the present invention is applied achieved higher performance than representative speech face generators ATVG [1], Wav2Lip [2], MakeItTalk [3], AVCT [4], PC-AVS [5], EAMM [6], and EAT [7], and had performance improved by 2.2 points and 1.2 points, respectively, compared to before knowledge was distilled.

[0081] To compare the qualitative evaluation of the present invention, an experiment was conducted on identity reference images with strong emotions. Referring to the results according to the embodiments of the present invention, it can be confirmed that the emotions of the reference photos were successfully excluded. Furthermore, it is possible to generate more natural images with varying emotional intensities for each frame.

[0082] According to an embodiment of the present invention, the output of the model must satisfy the following two conditions for any emotional face input: 1) The original emotion of the identity reference image is not reflected. 2) The emotional intensity varies frame by frame corresponding to the emotional flow of the audio.

[0083] Referring to Fig. 6, it can be seen that the EAMM outputs a video that relies heavily on the emotions of IDs #1 and #2. The EAT also generates unwanted emotions for ID #2. Furthermore, all frames express emotions at nearly the same intensity level, indicating a degradation in naturalness.

[0084] On the other hand, the EAP according to an embodiment of the present invention generates emotions and intensity corresponding to the audio even for highly expressive inputs. Specifically, when compared to the emotion labels of the audio, the EAP generates the desired emotion for both ID #1 and #2. When compared to GT frames for evaluation purposes, the EAP exhibits varying intensity levels per frame.

[0085] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include a plurality of processing elements and / or a plurality of types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0086] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0087] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0088] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0089] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

1. A neutral converter that receives a reference image and audio as input to transmit only identity information from the reference image and generates a neutral speaking face image for the reference image; and An emotion expression unit that generates an emotional speaking face using separated expression learning of emotion expression and emotion intensity to encode emotional information from audio input and reflect it in the generated neutral speaking face image. An emotional speaking face generation system including 2. In Paragraph 1, The above neutral conversion unit is, Defines a neutral class with minimized emotional expression as the target domain, and trains through contrastive goal-based learning to move away from classes containing emotional expression and closer to the neutral class in the metric space, thereby removing emotional features from the facial features of input reference images and learning to retain only identity information in the expression. Emotional speaking face generation system.

3. In Paragraph 1, The above-mentioned emotion expression unit is, To learn the separate representations of emotion expression and emotional intensity, two features embedded in the emotion expression space and the emotion intensity space are input into the audio emotion encoder and the audio emotion intensity regression encoder, respectively, to normalize each encoder and perform continuous emotion analysis. Emotional speaking face generation system.

4. In Paragraph 3, The above-mentioned emotion expression unit is, To extract continuous emotions from audio, train an audio emotion encoder, and Subsequently, to refine the emotional intensity per frame, an audio intensity regression encoder is additionally trained. Emotional speaking face generation system.

5. A step of receiving a reference image and audio as input to transmit only identity information from the reference image, and generating a neutral speaking face image for the reference image from the audio input through a neutral conversion unit; and A step of generating an emotional speaking face using separated expression learning of emotional expression and emotional intensity to encode emotional information from audio input through an emotion expression unit and reflect it in the generated neutral speaking face image A method for generating emotional speaking faces that includes 6. In Paragraph 5, The step of receiving a reference image and audio as input to transmit only identity information from the above reference image, and generating a neutral speaking face image for the reference image from the audio input through a neutral conversion unit, Defines a neutral class with minimized emotional expression as the target domain, and trains through contrastive goal-based learning to move away from classes containing emotional expression and closer to the neutral class in the metric space, thereby removing emotional features from the facial features of input reference images and learning to retain only identity information in the expression. How to generate an emotional speaking face.

7. In Paragraph 5, The step of generating an emotional speaking face using separated expression learning of emotional expression and emotional intensity to encode emotional information from an audio input through the emotional expression unit and reflect it in the generated neutral speaking face image is To learn the separate representations of emotion expression and emotional intensity, two features embedded in the emotion expression space and the emotion intensity space are input into the audio emotion encoder and the audio emotion intensity regression encoder, respectively, to normalize each encoder and perform continuous emotion analysis. How to generate an emotional speaking face.

8. In Paragraph 7, The step of generating an emotional speaking face using separated expression learning of emotional expression and emotional intensity to encode emotional information from an audio input through the emotional expression unit and reflect it in the generated neutral speaking face image is To extract continuous emotions from audio, train an audio emotion encoder, and Subsequently, to refine the emotional intensity per frame, an audio intensity regression encoder is additionally trained. How to generate an emotional speaking face.

Images